Permission error after upgrade to Debian Buster (10.2)

NSD
1

Dear colleagues,

after upgrading from Stretch to Debian Buster (10.2) I get following error message which blocks NSD (4.1.26) from starting:

Nov 24 16:18:40 ns2 nsd[989]: [2019-11-24 16:18:40.030] nsd[989]: error: could not open zone list /var/lib/nsd/zone.list: Permission denied
Nov 24 16:18:40 ns2 nsd[989]: [2019-11-24 16:18:40.032] nsd[989]: error: could not read zonelist file /var/lib/nsd/zone.list

However, the permissions are all fine; they did not change during the update.

ls -l /var/lib/nsd/zone.list

-rw-r–r-- 1 nsd nsd 1195 Nov 4 17:33 /var/lib/nsd/zone.list

I had a look into /lib/systemd/system/nsd.service:

[Unit]
Description=Name Server Daemon
Documentation=man:nsd(8)
After=network.target

[Service]
Type=notify
Restart=always
ExecStart=/usr/sbin/nsd -d
ExecReload=+/bin/kill -HUP $MAINPID
CapabilityBoundingSet=CAP_CHOWN CAP_IPC_LOCK CAP_NET_BIND_SERVICE CAP_SETGID CAP_SETUID CAP_SYS_CHROOT
MemoryDenyWriteExecute=true
NoNewPrivileges=true
PrivateDevices=true
PrivateTmp=true
ProtectHome=true
ProtectControlGroups=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectSystem=strict
ReadWritePaths=/var/lib/nsd /etc/nsd /run
RuntimeDirectory=nsd
RestrictRealtime=true
SystemCallArchitectures=native
SystemCallFilter=~@clock @cpu-emulation @debug @keyring @module mount @obsolete @resources

[Install]
WantedBy=multi-user.target

Once I remove following line,

CapabilityBoundingSet=CAP_CHOWN CAP_IPC_LOCK CAP_NET_BIND_SERVICE CAP_SETGID CAP_SETUID CAP_SYS_CHROOT

while it does not help to only remove params from it (I need to remove the full line), the error message changes to:

Nov 24 16:37:57 ns2 systemd[1]: Starting Name Server Daemon…
Nov 24 16:37:57 ns2 nsd[1607]: [2019-11-24 16:37:57.144] nsd[1607]: error: Cannot open /var/log/nsd.log for appending (Read-only file system), logging to stderr
Nov 24 16:37:57 ns2 nsd[1607]: [2019-11-24 16:37:57.145] nsd[1607]: notice: nsd starting (NSD 4.1.26)
Nov 24 16:37:57 ns2 nsd[1607]: [2019-11-24 16:37:57.252] nsd[1608]: notice: nsd started (NSD 4.1.26), pid 1607
Nov 24 16:37:57 ns2 systemd[1]: Started Name Server Daemon.

Since nsd-control zonestatus now works, NSD now can read the /var/lib/nsd/zone.list. However, it is still not running fine because now NSD says it cannot open the /var/log/nsd.log.

Does anyone know how to fix that?

Kind Regards,
Kaulkwappe

2

Hi Kaulkwappe,

Dear colleagues,

after upgrading from Stretch to Debian Buster (10.2) I get following error
message which blocks NSD (4.1.26) from starting:

> Nov 24 16:18:40 ns2 nsd[989]: [2019-11-24 16:18:40.030] nsd[989]: error:
could not open zone list /var/lib/nsd/zone.list: Permission denied
> Nov 24 16:18:40 ns2 nsd[989]: [2019-11-24 16:18:40.032] nsd[989]: error:
could not read zonelist file /var/lib/nsd/zone.list

However, the permissions are all fine; they did not change during the update.

ls -l /var/lib/nsd/zone.list
> -rw-r--r-- 1 nsd nsd 1195 Nov 4 17:33 /var/lib/nsd/zone.list

I had a look into /lib/systemd/system/nsd.service:

> [Unit]
> Description=Name Server Daemon
> Documentation=man:nsd(8)
> After=network.target

> [Service]
> Type=notify
> Restart=always
> ExecStart=/usr/sbin/nsd -d
> ExecReload=+/bin/kill -HUP $MAINPID
> CapabilityBoundingSet=CAP_CHOWN CAP_IPC_LOCK CAP_NET_BIND_SERVICE CAP_SETGID
CAP_SETUID CAP_SYS_CHROOT
> MemoryDenyWriteExecute=true
> NoNewPrivileges=true
> PrivateDevices=true
> PrivateTmp=true
> ProtectHome=true
> ProtectControlGroups=true
> ProtectKernelModules=true
> ProtectKernelTunables=true
> ProtectSystem=strict
> ReadWritePaths=/var/lib/nsd /etc/nsd /run
> RuntimeDirectory=nsd
> RestrictRealtime=true
> SystemCallArchitectures=native
> SystemCallFilter=~@clock @cpu-emulation @debug @keyring @module mount
@obsolete @resources

> [Install]
> WantedBy=multi-user.target

Once I remove following line,

> CapabilityBoundingSet=CAP_CHOWN CAP_IPC_LOCK CAP_NET_BIND_SERVICE CAP_SETGID
CAP_SETUID CAP_SYS_CHROOT

while it does not help to only remove params from it (I need to remove the full
line), the error message changes to:

To override a setting you need to reset it and then set it to what you
want, otherwise similar directives accumulate. Like this:

CapabilityBoundingSet=
CapabilityBoundingSet=MY_NEW_SET_OF_CAPS

> Nov 24 16:37:57 ns2 systemd[1]: Starting Name Server Daemon...
> Nov 24 16:37:57 ns2 nsd[1607]: [2019-11-24 16:37:57.144] nsd[1607]: error:
Cannot open /var/log/nsd.log for appending (Read-only file system), logging to
stderr

With the IMO good ProtectSystem=strict, nsd can only ever write to
directories listed in ReadWritePaths=.

> Nov 24 16:37:57 ns2 nsd[1607]: [2019-11-24 16:37:57.145] nsd[1607]: notice:
nsd starting (NSD 4.1.26)
> Nov 24 16:37:57 ns2 nsd[1607]: [2019-11-24 16:37:57.252] nsd[1608]: notice:
nsd started (NSD 4.1.26), pid 1607
> Nov 24 16:37:57 ns2 systemd[1]: Started Name Server Daemon.

Since nsd-control zonestatus now works, NSD now can read
the /var/lib/nsd/zone.list.

It's a wild guess but maybe /var/lib/nsd/zone.list is opened by nsd
while still running as root. When you have reduced caps
(CapabilityBoundingSet), you don't have CAP_FOWNER, CAP_DAC_READ_SEARCH
and/or CAP_DAC_OVERRIDE so might get a permission error.

I'm not sure if the right thing to do would be to have nsd open the file
post SETUID/GID or if more caps should be added.

However, it is still not running fine because now
NSD says it cannot open the /var/log/nsd.log.

Does anyone know how to fix that?

If you want nsd to do logging to a file directly, you'll need to add it
to the ReadWritePaths. I'd suggest "sudo systemctl edit nsd" then enter
and save the following:

  [Service]
  ReadWritePaths=/var/log

Then "sudo systemctl restart nsd". You can also throw in the
CapabilityBoundingSet tweaks you want in there.

HTH,
Simon

3

Hi Simon,

thanks for your fast answer.

It seems that you’re right that NSD tries to open the files as root user – which seems is blocked by the restrictive nsd.service configuration. See also:

So, I changed the owner of all the files to ‘root:root’ and added ‘/var/log’ to ‘ReadWritePaths’. Then NSD starts without any problems.

However, on the next startup I see that NSD always changes back the ownership of ‘/var/log/nsd.log’ from ‘root:root’ back to the nsd user. This leads to following error message:

Nov 24 18:48:05 ns2 nsd[1959]: [2019-11-24 18:48:05.896] nsd[1959]: error: Cannot open /var/log/nsd.log for appending (Read-only file system), logging to stderr

When I stop NSD, I get following messages:

Nov 24 21:01:22 ns2 nsd[2168]: [2019-11-24 21:01:22.109] nsd[2169]: warning: signal received, shutting down…
Nov 24 21:01:22 ns2 nsd[2168]: [2019-11-24 21:01:22.112] nsd[2169]: warning: failed to unlink pidfile /run/nsd/nsd.pid: Permission denied
Nov 24 21:01:22 ns2 nsd[2168]: [2019-11-24 21:01:22.117] nsd[2168]: error: xfrd: Could not open file /var/lib/nsd/xfrd.state for writing: Permission denied

This is very confusing since /var/lib/nsd/xfrd.state still has root:root, while NSD created the /run/nsd/nsd.pid using nsd:nsd.

Kind Regards,
Kaulkwappe

4

Hi Simon,

thanks for your fast answer.

It seems that you're right that NSD tries to open the files as root user – which
seems is blocked by the restrictive nsd.service configuration. See also:
I Challenge Thee

So, I changed the owner of all the files to 'root:root' and added '/var/log' to
'ReadWritePaths'. Then NSD starts without any problems.

However, on the next startup I see that NSD always changes back the ownership of
'/var/log/nsd.log' from 'root:root' back to the nsd user. This leads to
following error message:
> Nov 24 18:48:05 ns2 nsd[1959]: [2019-11-24 18:48:05.896] nsd[1959]: error:
Cannot open /var/log/nsd.log for appending (Read-only file system), logging to
stderr

I would have expect a permission error instead of a "read-only" one. It
looks as if /var/log was not properly added to be ReadWritePaths set.

When I stop NSD, I get following messages:
> Nov 24 21:01:22 ns2 nsd[2168]: [2019-11-24 21:01:22.109] nsd[2169]: warning:
signal received, shutting down...
> Nov 24 21:01:22 ns2 nsd[2168]: [2019-11-24 21:01:22.112] nsd[2169]: warning:
failed to unlink pidfile /run/nsd/nsd.pid: Permission denied

This unlink failure is expected and AFAICT harmless.

> Nov 24 21:01:22 ns2 nsd[2168]: [2019-11-24 21:01:22.117] nsd[2168]: error:
xfrd: Could not open file /var/lib/nsd/xfrd.state for writing: Permission denied

This is very confusing since /var/lib/nsd/xfrd.state still has root:root, while
NSD created the /run/nsd/nsd.pid using nsd:nsd.

I believe that xfrd.state should be owned by nsd:nsd as the daemon needs
to write to that file.

For reference, here's what it looks on my local slave:

root@ns0:~# ll /var/lib/nsd/xfrd.state /run/nsd/nsd.*
srwxr-xr-x 1 nsd nsd 0 Nov 24 19:41 /run/nsd/nsd.ctl=
-rw-r--r-- 1 nsd nsd 4 Nov 24 19:41 /run/nsd/nsd.pid
-rw-r--r-- 1 nsd nsd 2702 Nov 24 19:39 /var/lib/nsd/xfrd.state

Regards,
Simon

5

Hi Simon,

I would have expect a permission error instead of a “read-only” one. It
looks as if /var/log was not properly added to be ReadWritePaths set.

That is what I have used:

ReadWritePaths=/var/lib/nsd /var/log /etc/nsd /run

This unlink failure is expected and AFAICT harmless.
It should be harmless, but it doesn’t look nice. I would consider this as a bug.

I believe that xfrd.state should be owned by nsd:nsd as the daemon needs
to write to that file.

After changing the owner to nsd:nsd I believe this problem is fixed. Thanks!

Kind Regards,
Kaulkwappe

6

Hi Simon,

> I would have expect a permission error instead of a "read-only" one. It
> looks as if /var/log was not properly added to be ReadWritePaths set.

That is what I have used:
> ReadWritePaths=/var/lib/nsd /var/log /etc/nsd /run

Not sure what would explain the read-only error then. I'd double check
if it's indeed effective with "systemctl show nsd | grep ReadWritePaths"

> This unlink failure is expected and AFAICT harmless.
It should be harmless, but it doesn't look nice. I would consider this as a bug.

Agreed. Interestingly, unbound accepts "-p" to skip managing its own
PID. If nsd could get this, it would be handy when managing the daemon
with systemd.

> I believe that xfrd.state should be owned by nsd:nsd as the daemon needs
> to write to that file.
After changing the owner to nsd:nsd I believe this problem is fixed. Thanks!

Glad to hear that!

Regards,
Simon

7

[…] I’d double check if it’s indeed effective with “systemctl show nsd | grep ReadWritePaths”

Seems to be effective:

systemctl show nsd | grep ReadWritePaths

ReadWritePaths=/var/lib/nsd /var/log /etc/nsd /run

The problem with the log file will never stop the NSD service from working (I believe) but the log file is quite important, so, of course, NSD should be able to append to it.

Does anyone already had this problem after an upgrade?

Kind Regards,
Kaulkwappe

8

Hi Kaulkwappe,

> [...] I'd double check if it's indeed effective with "systemctl
show nsd | grep ReadWritePaths"

Seems to be effective:
> # systemctl show nsd | grep ReadWritePaths
> ReadWritePaths=/var/lib/nsd /var/log /etc/nsd /run

The problem with the log file will never stop the NSD service from
working (I believe) but the log file is quite important, so, of
course, NSD should be able to append to it.

Does anyone already had this problem after an upgrade?

Kind Regards,
Kaulkwappe

My knowledge on this subject is very limited, but since you ask I give
you my recent experience. I have also upgraded from Debian 9 to Debian
10, two ways, starting from Debian 9 and also from scratch. In both
cases I have not got NSD to write the log file. I have tested changes
of permissions and/or routes.
However, I have not had problems with the start of NSD, but I clarify
that I use NSD with a very elementary configuration and without
/var/lib/nsd/zone.list defined.
A cordial greeting.
José Luis

9

Hi Simon,

Hi Simon,

> I would have expect a permission error instead of a "read-only" one. It
> looks as if /var/log was not properly added to be ReadWritePaths set.

That is what I have used:
> ReadWritePaths=/var/lib/nsd /var/log /etc/nsd /run

Not sure what would explain the read-only error then. I'd double check
if it's indeed effective with "systemctl show nsd | grep ReadWritePaths"

> This unlink failure is expected and AFAICT harmless.
It should be harmless, but it doesn't look nice. I would consider this as a bug.

Agreed. Interestingly, unbound accepts "-p" to skip managing its own
PID. If nsd could get this, it would be handy when managing the daemon
with systemd.

When trying to add the option for you, I saw the code should accept -P
"" on the commandline or pidfile: "" in nsd.conf omits creation of the
pidfile.

It should already work! I could still create a convenience option or
perhaps a description for it? Perhaps in nsd's usage printout something
to say that '-P "" stop creation of the pidfile' or something along
those lines. If this also works, of course.

Best regards, Wouter

10

Unfortunately I still get this errors in NSD 4.1.26 on Debian Buster 10.2:

  1. Log file:

error: Cannot open /var/log/nsd.log for appending (Permission denied), logging to std

When it se the owner of nsd.log to root:root, I don’t get an error message on start. However, after this start, NSD will change the owner to nsd:nsd and on the next start I will get this error message.

  1. PID file:

warning: failed to unlink pidfile /run/nsd/nsd.pid: Permission denied
It seems that NSD needs a PID file, because if I change pidfile: “/run/nsd/nsd.pid” to pidfile: “” I get:

11

Hi Kaulkwappe,

Unfortunately I still get this errors in NSD 4.1.26 on Debian Buster
10.2:

1) Log file:
> error: Cannot open /var/log/nsd.log for appending (Permission
denied), logging to std

When it se the owner of nsd.log to root:root, I don't get an error
message on start. However, after this start, NSD will change the
owner to nsd:nsd and on the next start I will get this error message.

2) PID file:
> warning: failed to unlink pidfile /run/nsd/nsd.pid: Permission
denied
It seems that NSD needs a PID file, because if I change pidfile:
"/run/nsd/nsd.pid" to pidfile: "" I get:

> error: cannot open pidfile : No such file or directory
> error: cannot overwrite the pidfile : No such file or directory

---> Debian 10.2
---> NSD 4.1.26

About LOG and PID, I don't know if what I'm doing is correct, but
apparently NSD works correctly.

In the NSD configuration file I considered:

sudo nano /etc/nsd/nsd.conf
...
        logfile: "/var/run/log/nsd.log"
...
        pidfile: "/var/lib/nsd/nsd.pid"
...

With those routes I got the LOG written (I didn't get it written in
other places):

sudo mc
...
/var/run/log
  /journal
  nsd.log <--- !!

But the LOG reported that the PID could not be written in
/var/lib/nsd due to permission issues ...

Because I think *pid.nsd* must be written by *root*, I modified owner
and permissions like this:

sudo chown root:nsd /var/lib/nsd
sudo chmod 755 /var/lib/nsd

With this, the PID is now written:

sudo mc
...
/var/lib/nsd
  nsd.db
  nsd.pid <--- !!
  xfrd.state

Regards.
José Luis

12

Again ...

Hi Kaulkwappe,

> Unfortunately I still get this errors in NSD 4.1.26 on Debian
> Buster
> 10.2:
>
> 1) Log file:
> > error: Cannot open /var/log/nsd.log for appending (Permission
> denied), logging to std
>
> When it se the owner of nsd.log to root:root, I don't get an error
> message on start. However, after this start, NSD will change the
> owner to nsd:nsd and on the next start I will get this error
> message.
>
> 2) PID file:
> > warning: failed to unlink pidfile /run/nsd/nsd.pid: Permission
> denied
> It seems that NSD needs a PID file, because if I change pidfile:
> "/run/nsd/nsd.pid" to pidfile: "" I get:
>
> > error: cannot open pidfile : No such file or directory
> > error: cannot overwrite the pidfile : No such file or directory

---> Debian 10.2
---> NSD 4.1.26

About LOG and PID, I don't know if what I'm doing is correct, but
apparently NSD works correctly.

In the NSD configuration file I considered:

sudo nano /etc/nsd/nsd.conf
...
        logfile: "/var/run/log/nsd.log"
...
        pidfile: "/var/lib/nsd/nsd.pid"
...

With those routes I got the LOG written (I didn't get it written in
other places):

sudo mc
...
/var/run/log
  /journal
  nsd.log <--- !!

But the LOG reported that the PID could not be written in
/var/lib/nsd due to permission issues ...

Because I think *pid.nsd* must be written by *root*, I modified owner
and permissions like this:

sudo chown root:nsd /var/lib/nsd
sudo chmod 755 /var/lib/nsd

With this, the PID is now written:

sudo mc
...
/var/lib/nsd
  nsd.db
  nsd.pid <--- !!
  xfrd.state

Regards.
José Luis

After several tests, restarting the server and restarting only NSD, the
PID was truncated but stopped writing. So I believe that it is not
*root* who writes it ...

Then, I modified owner and permissions in this way:

sudo chown nsd:nsd /var/lib/nsd
sudo chmod 777 /var/lib/nsd

Now PID is always written and the LOG does not report errors.

13

Hello guys,

I don't run Debian, so I can't offer a solution now, but I am worried
that you're all just stumbling in the dark here, and randomly changing
permissions on directories and files. A well-built package should not
require any of this, and should just work. Has any one of you approached
the maintainer of the Debian package? Perhaps it has been built
incorrectly, and needs to be fixed.

Regards,
Anand

14

Hi Anand,

Hello guys,

I don't run Debian, so I can't offer a solution now, but I am worried
that you're all just stumbling in the dark here, and randomly
changing
permissions on directories and files. A well-built package should not
require any of this, and should just work. Has any one of you
approached
the maintainer of the Debian package? Perhaps it has been built
incorrectly, and needs to be fixed.

Regards,
Anand

No, at least I have not contacted the NSD package maintainer in Debian.
Thank you so much for your advice.
Regards.
José Luis

15

Actually with sudo chmod 666 /var/run/log/nsd.log works fine !

16

Gosh!
Well, chmod -R 1777 / will work too. But it is NOT, repeat: NOT
the way to go..

/mjt

17

> Actually with sudo chmod 666 /var/run/log/nsd.log works fine !

Gosh!
Well, chmod -R 1777 / will work too. But it is NOT, repeat: NOT
the way to go..

Agree :wink: