I've observed something strange on some of our busy servers. Logrotate
reports something like this:
/etc/cron.daily/logrotate:
gzip: stdin: file size changed while zipping
So here's what logrotate does:
1. mv nsd.log nsd.log.old
2. runs "nsd-control log_reopen"
3. gzip nsd.log.old
Occasionally, when performing step 3, gzip complains that the input file
size changed, meaning that NSD was still writing to it.
Is it possible that NSD delays re-opening of the log file?
It communicates the reopen internally over the pipes. It could be race
condition, where the gzip starts before the reopen has finished. sleep
1 or sleep 2 before calling gzip would solve that.
Or maybe make log_reopen wait until the reopen has finished, not sure
how to accomplish that...
It communicates the reopen internally over the pipes. It could be race
condition, where the gzip starts before the reopen has finished. sleep
1 or sleep 2 before calling gzip would solve that.
This is probably right. When looking at the log files around this time,
it looks like NSD was still busy applying an XFR and reloading a zone,
so it still keep writing to the old log file. But logrotate doesn't know
this, so it starts compressing the old log file.
Logrotate has an option called "delaycompress" that will not compress
the rotated log file. It will only compress it the following day. I
could use this, as disk space isn't a problem for us.
I could add a delay to the compress program, but it would apply to all
logs, so it's not ideal.
I could also add a delay after the call to "nsd-control log_reopen".
I'll think about this, and see what the best solution is.
Or maybe make log_reopen wait until the reopen has finished, not sure
how to accomplish that...
Well, if a process receives a command, such as "log_reopen" can it not
report back when it has actually done that?