Nsd4 not detecting changed included files when rebuilding database

Hi,

nsd4 is trying to be too smart and thus when a 'reload' or even a
'restart' is issued the database is not being rebuild properly when one
uses includes.

It seems (I did for once not read the code :wink: that nsd checks the
filetime of the files as configured in a 'zone' statement, and based on
the changetime of that file it then decided to rebuild the zone or not.

Hence, if the primary file is not changed, but an included file has
changed, the zone will not be rebuild.

Quick work-around: touch *, and then a nsd-control reload

It would great if the rebuilding logic would either always rebuild zones
that contain includes, or possibly better, keep a 'include' list and
then based on the actual files did a rebuild or not.

I am not sure if it does that now or if it is possinle, but having a
per-file 'compiled' version could work here too, then if there is a
small include and another big include, only the small one will be
reparsed/compiled while the big non-changed one does not have to be handled.

Greets,
Jeroen

Hi Jeroen,

Hi,

nsd4 is trying to be too smart and thus when a 'reload' or even a
'restart' is issued the database is not being rebuild properly when
one uses includes.

It seems (I did for once not read the code :wink: that nsd checks the
filetime of the files as configured in a 'zone' statement, and
based on the changetime of that file it then decided to rebuild the
zone or not.

Hence, if the primary file is not changed, but an included file
has changed, the zone will not be rebuild.

Quick work-around: touch *, and then a nsd-control reload

It would great if the rebuilding logic would either always rebuild
zones that contain includes, or possibly better, keep a 'include'
list and then based on the actual files did a rebuild or not.

Thank you for the description, that is certainly a feature that looks
like a good idea. Not sure if we can keep track of the included-files
in memory, but perhaps we can do that in a simple manner.

Best regards,
   Wouter

Hi Jeroen,

Hi,

nsd4 is trying to be too smart and thus when a 'reload' or even a
'restart' is issued the database is not being rebuild properly when
one uses includes.

It seems (I did for once not read the code :wink: that nsd checks the
filetime of the files as configured in a 'zone' statement, and
based on the changetime of that file it then decided to rebuild the
zone or not.

Hence, if the primary file is not changed, but an included file
has changed, the zone will not be rebuild.

Quick work-around: touch *, and then a nsd-control reload

Another quick work-around would be an option ala 'ignore file times for
determining newness' then everything would be rebuild every time, but it
avoids the above 'touch'.

It would great if the rebuilding logic would either always rebuild
zones that contain includes, or possibly better, keep a 'include'
list and then based on the actual files did a rebuild or not.

Thank you for the description, that is certainly a feature that looks
like a good idea. Not sure if we can keep track of the included-files
in memory, but perhaps we can do that in a simple manner.

I think it should be meta-data of the zone compiler (but I have no clue
how that thing works or what you store there :wink:

Just a list of filenames with a date + include, json style description
of the array should do the trick, and then do recursive lookups:

{
"example.com": { "2014-01-01 01:01", {"file.inc" }
"file.inc" => { "2014-05-05 05:05" }
};

That would explode on you though and require scans to find which file
includes what, thus an "included by" notion would be better:

{
"example.com": { "2014-01-01 01:01", {} }
"file.inc" => { "2014-05-05 05:05", {"example.com"} }
};

Then if file.inc changes, you know to update example.com (in mem, just
use pointers for the list of course...

Should not be too heavyweight, even with millions of files.

Greets,
Jeroen