Getting SERVFAIL when trying to reach .co.il domains

Gil_Levy · December 31, 2020, 11:22am

Using Unbound 1.9.0 on Raspberry Pi with Pihole.

Since two days ago I cannot access .co.il domains, such as hwzone.co.il or ynet.co.il.
When I issue# sudo systemctl stop unbound and then #sudo systemctl start unbound, I can access these sites for a few seconds and then they become unavailable again.

I tried to isolate the case by using Quad9 DNS instead of Unbound and the issue resolved.
When I revert back to Unbound, I encounter the same problem.

I also run Unbound on my Oracle cloud Ubuntu instance and I experience the same issue there.

I use Unbound as the authoritative server, I don’t have it set up with upstream servers.

The tail log of the SERVFAIL can be found here: https://textuploader.com/185f0

Appreciate any help in troubleshooting this.

Thanks,
Gil

Joe_Abley · December 31, 2020, 12:33pm

Hi Gil,

heidnes · December 31, 2020, 12:39pm

Using Unbound 1.9.0 on Raspberry Pi with Pihole.

Since two days ago I cannot access .co.il domains, such as hwzone.co.il or
ynet.co.il.

The analysis tool at https://dnsviz.net/ seems to indicate there's a
problem with the DNSSEC setup for both .IL and .CO.IL, ref.

https://dnsviz.net/d/hwzone.co.il/dnssec/

The recurring message seems to be that e.g. the DNSKEY RRset for .IL
includes a key with algorithm 13 (ECDSAP256SHA256), but no
corresponding RRSIG can be found, and the same for the .CO.IL domain.

Whether that should be a fatal error is another matter, it probably
should not, as long as there exists other keys where there exists a
matching RRSIG. Newer unbound (e.g. 1.12.0) does not make this a
fatal error, and resolves those names just fine.

Regards,

- Håvard

Gil_Levy · December 31, 2020, 12:40pm

Hi Joe,

Thanks for the reply!
The verbosity of Unbound is set to 3 and all I get is:

pi@raspberrypi:/var/log $ tail -f syslog
Dec 31 23:13:31 raspberrypi avahi-daemon[363]: Registering new address record for 192.168.1.2 on eth0.IPv4.
Dec 31 23:13:31 raspberrypi dbus-daemon[300]: [system] Successfully activated service ‘org.freedesktop.hostname1’
Dec 31 23:13:31 raspberrypi systemd[1]: Started Hostname Service.
Dec 31 23:13:35 raspberrypi systemd[1]: systemd-rfkill.service: Succeeded.
Dec 31 23:13:39 raspberrypi dhcpcd[380]: eth0: no IPv6 Routers available
Dec 31 23:13:52 raspberrypi systemd[1]: systemd-fsckd.service: Succeeded.
Dec 31 23:14:04 raspberrypi systemd-timesyncd[284]: Synchronized to time server for the first time 217.147.208.1:123 (2.debian.pool.ntp.org).
Dec 31 23:14:11 raspberrypi systemd[1]: systemd-hostnamed.service: Succeeded.
Dec 31 23:14:51 raspberrypi systemd[1]: Started Session 6 of user pi.
Dec 31 23:15:14 raspberrypi systemd[1]: Started Session 7 of user pi.
Dec 31 23:16:05 raspberrypi systemd[1]: Started Session 8 of user pi.
Dec 31 23:16:17 raspberrypi systemd[1]: Stopping Unbound DNS server via resolvconf…
Dec 31 23:16:17 raspberrypi package-helper[978]: Too few arguments.
Dec 31 23:16:17 raspberrypi systemd[1]: unbound-resolvconf.service: Succeeded.
Dec 31 23:16:17 raspberrypi systemd[1]: Stopped Unbound DNS server via resolvconf.
Dec 31 23:16:17 raspberrypi systemd[1]: Stopping Unbound DNS server…
Dec 31 23:16:17 raspberrypi systemd[1]: unbound.service: Succeeded.
Dec 31 23:16:17 raspberrypi systemd[1]: Stopped Unbound DNS server.
Dec 31 23:16:17 raspberrypi systemd[1]: Starting Unbound DNS server…
Dec 31 23:16:17 raspberrypi unbound[1028]: [1609416977] unbound[1028:0] debug: chdir to /etc/unbound
Dec 31 23:16:17 raspberrypi unbound[1028]: [1609416977] unbound[1028:0] debug: drop user privileges, run as unbound
Dec 31 23:16:17 raspberrypi unbound[1028]: [1609416977] unbound[1028:0] debug: switching log to /dev/null
Dec 31 23:16:17 raspberrypi systemd[1]: Started Unbound DNS server.
Dec 31 23:16:18 raspberrypi systemd[1]: Started Unbound DNS server via resolvconf.
Dec 31 23:16:18 raspberrypi package-helper[1029]: Too few arguments.

If I was to guess I’d say that you have a full set of servers for some zone or other that is causing trouble

Sorry, I don’t know what it means. I didn’t set up anything like that.
I also don’t use ipv6.

I don’t know why it worked well 2 days ago and now I have this issue.

How do I proceed from here?

Thanks and happy new year!

Gil_Levy · December 31, 2020, 12:41pm

Does that mean that the problem is not with my network?

I don’t know how to compile the latest build of unbound for Pihole using RaspberryOS (Debian)

Thanks for the info.

hdais · January 1, 2021, 9:28am

Hi,

".co.il" and ".il" (seemingly under DNSSEC algorithm rollover) have
several errors. Current versions of Unbound in default configuration
tolerate them, but in a specific configuration Unbound could make
fatal errors.

Assuming [1] is your configuration file, the offending line is:

harden-algo-downgrade: yes

"harden-algo-downgrade: no" (this is the current default value) makes
Unbound tolerant.

[1] https://pastebin.com/ZAUVFVEF

Gil_Levy · January 1, 2021, 11:06am

Thanks, Daisuke.

However, I’m past that line. While I will change the settings as you kindly suggested (thank you for that), I’m encountering other issues which disable me from using Unbound.

I shot an email earlier today with the following:

Cannot open log file (despite it’s configured in unbound.conf)

Cannot use the unbound-checkconf utility
I provided a link to my config file at the bottom.
Appreciate your help!

Gil

pi@raspberrypi:/etc/unbound $ sudo systemctl status unbound
● unbound.service - Unbound DNS resolver
Loaded: loaded (/lib/systemd/system/unbound.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2021-01-01 10:44:56 AEDT; 19min ago
Process: 456 ExecStartPre=/usr/sbin/unbound-anchor -r /etc/unbound/root.hints -a /etc/unbound/root.key (code=exited, status=0/SUCCESS)
Main PID: 481 (unbound)
Tasks: 1 (limit: 2063)
CGroup: /system.slice/unbound.service
└─481 /usr/sbin/unbound -c /etc/unbound/unbound.conf -d

Jan 01 10:44:56 raspberrypi unbound-anchor[456]: [1609458296] libunbound[456:0] error: udp connect failed: Network is unreachable for 198.41.0.4 port 53
Jan 01 10:44:56 raspberrypi unbound-anchor[456]: [1609458296] libunbound[456:0] error: udp connect failed: Network is unreachable for 192.33.4.12 port 53
Jan 01 10:44:56 raspberrypi unbound-anchor[456]: [1609458296] libunbound[456:0] error: udp connect failed: Network is unreachable for 2001:dc3::35 port 53
Jan 01 10:44:56 raspberrypi unbound-anchor[456]: [1609458296] libunbound[456:0] error: udp connect failed: Network is unreachable for 2001:500:1::53 port 53
Jan 01 10:44:56 raspberrypi unbound-anchor[456]: [1609458296] libunbound[456:0] error: udp connect failed: Network is unreachable for 2001:500:9f::42 port 53
Jan 01 10:44:56 raspberrypi unbound-anchor[456]: [1609458296] libunbound[456:0] error: udp connect failed: Network is unreachable for 199.7.91.13 port 53
Jan 01 10:44:56 raspberrypi unbound[481]: [1609458296] unbound[481:0] error: Could not open logfile /var/log/unbound/unbound.log: No such file or directory
Jan 01 10:44:57 raspberrypi unbound[481]: [1609458297] unbound[481:0] notice: init module 0: validator
Jan 01 10:44:57 raspberrypi unbound[481]: [1609458297] unbound[481:0] notice: init module 1: iterator
Jan 01 10:44:57 raspberrypi unbound[481]: [1609458297] unbound[481:0] info: start of service (unbound 1.13.0).

pi@raspberrypi:/var/log/unbound $ ls
unbound.log

pi@raspberrypi:/etc/unbound $ unbound-checkconf /etc/unbound/unbound.conf
/etc/unbound/var/log/unbound: No such file or directory
[1609459551] unbound-checkconf[1316:0] fatal error: logfile directory does not exist

pi@raspberrypi:/etc/unbound $ ls
root.hints root.key root.zone unbound.conf unbound_control.key unbound_control.pem unbound.log unbound.pid unbound_server.key unbound_server.pem

unbound.conf here → https://pastebin.com/ZAUVFVEF

Any ideas what should I do? I’m really lost here and would like to keep using unbound.

Thanks in advance.

Unbound · January 1, 2021, 11:23am

Thanks, Daisuke.

However, I'm past that line. While I will change the settings as you kindly
suggested (thank you for that), I'm encountering other issues which disable
me from using Unbound.
I shot an email earlier today with the following:

   1. Cannot open log file (despite it's configured in unbound.conf)
   2. Cannot use the unbound-checkconf utility

I provided a link to my config file at the bottom.
Appreciate your help!

Gil

*pi@raspberrypi:/etc/unbound $ sudo systemctl status unbound*
● unbound.service - Unbound DNS resolver
   Loaded: loaded (/lib/systemd/system/unbound.service; enabled; vendor
preset: enabled)
   Active: active (running) since Fri 2021-01-01 10:44:56 AEDT; 19min ago
  Process: 456 ExecStartPre=/usr/sbin/unbound-anchor -r
/etc/unbound/root.hints -a /etc/unbound/root.key (code=exited,
status=0/SUCCESS)
Main PID: 481 (unbound)
    Tasks: 1 (limit: 2063)
   CGroup: /system.slice/unbound.service
           └─481 /usr/sbin/unbound -c /etc/unbound/unbound.conf -d

Jan 01 10:44:56 raspberrypi unbound-anchor[456]: [1609458296]
libunbound[456:0] error: udp connect failed: Network is unreachable for
198.41.0.4 port 53
Jan 01 10:44:56 raspberrypi unbound-anchor[456]: [1609458296]
libunbound[456:0] error: udp connect failed: Network is unreachable for
192.33.4.12 port 53
Jan 01 10:44:56 raspberrypi unbound-anchor[456]: [1609458296]
libunbound[456:0] error: udp connect failed: Network is unreachable for
2001:dc3::35 port 53
Jan 01 10:44:56 raspberrypi unbound-anchor[456]: [1609458296]
libunbound[456:0] error: udp connect failed: Network is unreachable for
2001:500:1::53 port 53
Jan 01 10:44:56 raspberrypi unbound-anchor[456]: [1609458296]
libunbound[456:0] error: udp connect failed: Network is unreachable for
2001:500:9f::42 port 53
Jan 01 10:44:56 raspberrypi unbound-anchor[456]: [1609458296]
libunbound[456:0] error: udp connect failed: Network is unreachable for
199.7.91.13 port 53
Jan 01 10:44:56 raspberrypi unbound[481]: [1609458296] unbound[481:0] *error:
Could not open logfile /var/log/unbound/unbound.log: No such file or
directory*
Jan 01 10:44:57 raspberrypi unbound[481]: [1609458297] unbound[481:0]
notice: init module 0: validator
Jan 01 10:44:57 raspberrypi unbound[481]: [1609458297] unbound[481:0]
notice: init module 1: iterator
Jan 01 10:44:57 raspberrypi unbound[481]: [1609458297] unbound[481:0]
info: start of service (unbound 1.13.0).

pi@raspberrypi:/var/log/unbound $ ls
unbound.log

pi@raspberrypi:/etc/unbound $ unbound-checkconf /etc/unbound/unbound.conf
/etc/unbound/var/log/unbound: *No such file or directory*

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
I won't speak for all your woes. But this line (above) says it all.
On one hand you indicate your log file is located here:

pi@raspberrypi:/var/log/unbound $ ls
unbound.log

But apparently your unbound.conf file indicates it's here:

/etc/unbound/var/log/unbound

See the difference?
Are you running unbound in a chroot(8)?

Gil_Levy · January 1, 2021, 1:14pm

But apparently your unbound.conf file indicates it’s here:

/etc/unbound/var/log/unbound

This has already been fixed in my unbound.conf file (see here: unbound.conf), but it still errors: error: Could not open logfile /var/log/unbound/unbound.log: No such file or directory

See the difference?
Are you running unbound in a chroot(8)?

I don’t know how to check that.

Joe_Abley · January 1, 2021, 1:53pm

man chroot

for a better description of what chroot does, and how the interpretation of absolute pathnames differs inside and outside the chroot namespace.

man man

if you’re unfamiliar with how manual pages are organised. If you don’t have manual pages installed and can’t add them as a package, it should not be hard to find collections of manual pages for your particular distribution if you search for them.

grep chroot unbound.conf

seems like a reasonable place to start to find configuration options in your environment that relate to chroot. You might also refer to the unbound documentation to understand the defaults and the specific meaning of individual parameters.

Another common error is to try and write log files to places where the process generating them does not have the necessary permissions. Determine the user that unbound is running as and check the permissions in the filesystem.

If you don’t know how unix filesystem permissions work, I would invest the time in finding out. This information is easy to find.

Joe

Jaap · January 1, 2021, 2:05pm

Joe Abley via Unbound-users writes:

>
>
> >> Are you running unbound in a chroot(8)?
> > I don't know how to check that.
>
> man chroot
>
> for a better description of what chroot does, and how the interpretation of
> absolute pathnames differs inside and outside the chroot namespace.
>
> man man
>
> if you're unfamiliar with how manual pages are organised. If you don't have
> manual pages installed and can't add them as a package, it should not be hard
> to find collections of manual pages for your particular distribution if you
> search for them.
>
> grep chroot unbound.conf

For a running unbound, do

unbound-control get_option chroot

to get the value it is using.

> seems like a reasonable place to start to find configuration options in your
> environment that relate to chroot. You might also refer to the unbound
> documentation to understand the defaults and the specific meaning of individual
> parameters.

Especially take notice what

man unbound.conf

tells you about the interaction between chroot and absolute path names.

>
> Another common error is to try and write log files to places where the process
> generating them does not have the necessary permissions. Determine the
> user that unbound is running as and check the permissions in the filesystem.

Or the directories are missing after the chroot took place...

jaap

Gil_Levy · January 1, 2021, 2:29pm

Thanks, guys!
I’m running chroot on /etc/unbound.

I followed this guide to compile unbound on my machine: https://pastebin.com/UUjss5aY
Some initial values there made use of /etc/unbound instead of /var/log/unbound so after I compiled unbound-1.13.0, I changed the paths to point to /var/log/unbound

The log file user is set to unbound with write permissions, but seems it’s not aware of its location (?)
The unbound-checkconf command is failing as well. It feels like the solution is not complicated, yet I’m unsure how to fix it or if I should try to compile all over again.
I’d rather try to fix it, if it’s ok to ask for such type of help over this thread.

pi@raspberrypi:/etc/unbound $ grep chroot unbound.conf
chroot: “/etc/unbound”

pi@raspberrypi:/etc/unbound $ ls -l /var/log/unbound/unbound.log
-rw-r–r-- 1 unbound unbound 5553 Oct 21 00:16 /var/log/unbound/unbound.log

pi@raspberrypi:/etc/unbound $ unbound-checkconf
/etc/unbound/var/log/unbound: No such file or directory
[1609510296] unbound-checkconf[2288:0] fatal error: logfile directory does not exist

pi@raspberrypi:/etc/unbound $ sudo systemctl status unbound
● unbound.service - Unbound DNS resolver
Loaded: loaded (/lib/systemd/system/unbound.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2021-01-02 00:46:44 AEDT; 25min ago
Process: 457 ExecStartPre=/usr/sbin/unbound-anchor -r /etc/unbound/root.hints -a /etc/unbound/root.key (code=exited, status=0/SUCCESS)
Main PID: 483 (unbound)
Tasks: 1 (limit: 2063)
CGroup: /system.slice/unbound.service
└─483 /usr/sbin/unbound -c /etc/unbound/unbound.conf -d

Jan 02 00:46:49 raspberrypi unbound[483]: [1609508809] unbound[483:0] error: udp connect failed: Network is unreachable for 199.7.83.42 port 53
Jan 02 00:46:49 raspberrypi unbound[483]: [1609508809] unbound[483:0] error: udp connect failed: Network is unreachable for 198.41.0.4 port 53
Jan 02 00:46:49 raspberrypi unbound[483]: [1609508809] unbound[483:0] error: udp connect failed: Network is unreachable for 199.7.91.13 port 53
Jan 02 00:46:49 raspberrypi unbound[483]: [1609508809] unbound[483:0] error: udp connect failed: Network is unreachable for 198.97.190.53 port 53
Jan 02 00:46:49 raspberrypi unbound[483]: [1609508809] unbound[483:0] error: udp connect failed: Network is unreachable for 199.7.91.13 port 53
Jan 02 00:46:49 raspberrypi unbound[483]: [1609508809] unbound[483:0] error: udp connect failed: Network is unreachable for 193.0.14.129 port 53
Jan 02 00:46:49 raspberrypi unbound[483]: [1609508809] unbound[483:0] error: udp connect failed: Network is unreachable for 199.7.91.13 port 53
Jan 02 00:46:49 raspberrypi unbound[483]: [1609508809] unbound[483:0] error: udp connect failed: Network is unreachable for 192.33.4.12 port 53
Jan 02 00:46:49 raspberrypi unbound[483]: [1609508809] unbound[483:0] error: udp connect failed: Network is unreachable for 192.58.128.30 port 53
Jan 02 00:46:50 raspberrypi unbound[483]: [1609508810] unbound[483:0] info: generate keytag query _ta-4f66. NULL IN

pi@raspberrypi:/etc/unbound $ sudo lsof -i :53
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
pihole-FT 829 pihole 4u IPv4 27749 0t0 UDP *:domain
pihole-FT 829 pihole 5u IPv4 27750 0t0 TCP *:domain (LISTEN)
pihole-FT 829 pihole 6u IPv6 27751 0t0 UDP *:domain
pihole-FT 829 pihole 7u IPv6 27752 0t0 TCP *:domain (LISTEN)

Unbound · January 1, 2021, 10:43pm

OK based on what I was able to ascertain from the somewhat jumbled
info in this thread. I'm going to stick my neck out and suggest this
is probably a *system* thing, more than an unbound thing.
That said. Let's try and sort this for you.
If you perform the following, do you get output?

$ cat /etc/unbound.conf

If you get output.
SO. I can see, given your pastebin link, you can see the contents of
at least one of your unbound.conf files, and that you *are* running
unbound in a chroot(8).
Given the errors that I've seen in this thread, and your comments.
It appears that you're unfamiliar with chroot(8). Simply put, it
reroots the environment into a new (directory) tree. Your system
appears to think that's /etc/unbound
IMHO this is a poor choice of locations. As /etc is usually owned
by root, and is *intended* for initial configuration of your system
services.
Let's try this (based upon my own setup on a large server farm)
chroot unbound into /var -- or more accurately /var/unbound
DO NOTE: your init(8) script *must* reference this location
for (unbound) start|status|stop|...
Copy your current /etc/unbound.conf to /etc/unbound.conf.last

$ cp /etc/unbound.conf /etc/unbound.conf.last

empty the entire /etc/unbound.conf, then add ONLY the following:

include: "/var/unbound/unbound.conf"

save /etc/unbound.conf
Make the initial unbound chroot and populate it

$ mkdir /var/unbound
$ cd /var/unbound

I've created an unbound.conf on your pastebin copy:
https://internethell.org/var-unbound-unbound.conf
Grab it, and place this file in /var/unbound as
unbound.conf
Ensure that unbound owns this chroot directory.
While in /var/unbound do:

$ chown -Rh unbound:unbound .

After ensuring that your init(8) script correctly references
your unbound chroot tree. Start unbound

service unbound start

Do note; you will likely need to preface all the commands
indicated above with: sudo

Give this a try, and indicate the status.

Best wishes, and Happy New Year!

--- trimmed for brevity ----