Unbound 1.26.1 times out on specific domains with DNSSEC enabled

Unbound
1

Hello,

I am experiencing intermittent/specific-domain DNS resolution failures with Unbound 1.26.1 on a Raspberry Pi running Pi-hole.

System:

  • Raspberry Pi, ARM64
  • Debian 13 (Trixie)
  • Unbound 1.26.1-0+deb13u1
  • Pi-hole FTL 6.7.1
  • dig 9.20.29-1~deb13u1-Debian
  • Unbound listening on 127.0.0.1:5335
  • DNSSEC enabled
  • auto-trust-anchor-file: “/var/lib/unbound/root.key”

Unbound and DNSSEC otherwise appear to be working correctly.

For example:

dig cloudflare.com @127.0.0.1 -p 5335 +dnssec

returns:

flags: qr rd ra ad

and an RRSIG record.

Therefore DNSSEC validation is definitely working through Unbound.

CURRENT REPRODUCIBLE PROBLEM

The current reproducible problem is:

my-account.spectator.co.uk

Running:

dig my-account.spectator.co.uk @127.0.0.1 -p 5335 +dnssec

results in repeated timeouts:

communications error to 127.0.0.1#5335: timed out

no servers could be reached

However, the same query through Cloudflare succeeds immediately:

dig my-account.spectator.co.uk @1.1.1.1 +dnssec

Cloudflare returns:

my-account.spectator.co.uk. CNAME my-account.coeditor.io.
my-account.coeditor.io. CNAME d35ystitntcfyq.cloudfront.net.

followed by four CloudFront A records.

I can also query the authoritative DNS server directly:

dig my-account.coeditor.io @gabe.ns.cloudflare.com +dnssec

which returns immediately:

my-account.coeditor.io. 300 IN CNAME d35ystitntcfyq.cloudfront.net.

The final CloudFront hostname also resolves successfully through Unbound:

dig d35ystitntcfyq.cloudfront.net @127.0.0.1 -p 5335 +dnssec

returns NOERROR immediately with four A records.

I also checked the delegation:

dig NS coeditor.io @1.1.1.1 +dnssec

which returns:

gabe.ns.cloudflare.com.
kay.ns.cloudflare.com.

There is no DS record for coeditor.io:

dig DS coeditor.io @1.1.1.1 +dnssec

returns ANSWER: 0.

I flushed the Unbound cache for the zone:

sudo unbound-control flush_zone coeditor.io

which returned:

ok removed 2 rrsets, 1 messages and 0 key entries

but the query continued to time out.

OTHER DOMAINS

I previously saw similar intermittent behaviour with:

spectator.com

and:

archive.today

The spectator.com problem subsequently resolved, so my-account.spectator.co.uk is currently the reproducible example.

I am not claiming these necessarily have the same cause, but the common feature is that the domains work through public recursive resolvers while Unbound can time out during resolution.

The problem is particularly noticeable on an iPhone using WireGuard with Pi-hole as its DNS server. Pi-hole forwards the requests to Unbound, but receives no response for my-account.spectator.co.uk. Other devices using the same network can work normally.

I have not changed the Unbound source or made any unusual configuration changes.

My main question is:

Could this be related to DNSSEC, EDNS, CNAME processing, iterator behaviour, or another change/issue in Unbound 1.26.1?

I would be interested to know what additional diagnostic information would be useful.

Thanks.

2

We manage 300+ Debian systems and encounter the exact same problem on ~12 systems in resolving DNS queries for spamassassin.org and docker.io (and the domains mentioned by @theengine).

# dig @127.0.0.2 docker.io. A
;; communications error to 127.0.0.2#53: timed out
;; communications error to 127.0.0.2#53: timed out
;; communications error to 127.0.0.2#53: timed out

; <<>> DiG 9.20.29-1~deb13u1-Debian <<>> docker.io. A @127.0.0.2
; (1 server found)
;; global options: +cmd
;; no servers could be reached

Querying without DNSSEC validation (dig ... +cd) works.

All of the systems use:

  • Different hardware
  • Debian 13 (Trixie) amd64
  • Unbound 1.26.1-0+deb13u1

The Unbound configuration on these systems:

server:
    interface: 127.0.0.2
    do-ip4: yes
    do-ip6: no
    do-udp: yes
    do-tcp: yes
    root-hints: "/usr/share/dns/root.hints"
    hide-identity: yes
    hide-version: yes
    cache-min-ttl: 0
    cache-max-ttl: 86400
    prefetch: yes
    access-control: 127.0.0.0/8 allow
    access-control: 192.168.0.0/19 allow
    access-control: 172.31.0.0/16 allow
    verbosity: 1
    auto-trust-anchor-file: "/var/lib/unbound/root.key"

The contents of /usr/share/dns/root.hints and /var/lib/unbound/root.key appear to be valid compared to all the other systems.

The problem occurred the day Unbound was updated to version 1.26.1-0+deb13u1.
However, interestingly, a test downgrade back to version 1.22.0-2+deb13u3 (with cache flush) does not resolve the problem.

3

Unbound is working, the because it find the domains broken. As example, see DNSSEC Debugger - docker.io

4

Thank you for your reply.

That’s a fair point. However, I’m unable to explain why the other 278+ systems don’t experience this issue. Their Unbound configurations are identical.
I’ve attempted to reproduce the problem on an unaffected system by clearing the cache, but without success.

5

Thanks, br1a. Your report is very interesting because it closely matches what I am seeing.

In my case, the problem originally appeared while running Unbound 1.17.1, which had worked reliably for many years. I upgraded to 1.26.1 only after the problem had already started, so I don’t believe 1.26.1 was the original trigger.

I can reproduce the problem directly against Unbound, independently of Pi-hole. For example, my-account.spectator.co.uk times out with DNSSEC validation enabled, whereas the same name resolves immediately through Cloudflare. archive.today showed similar behaviour, and disabling DNSSEC made it resolve.

DNSSEC itself is working correctly on my system — for example, signed domains such as cloudflare.com return the ADflag and RRSIG records.

I also find it interesting that your downgrade to 1.22.0 did not resolve the problem. Taken together with my experience on 1.17.1, this makes me wonder whether we are seeing an interoperability issue triggered by changes in particular authoritative DNS infrastructure, rather than a regression specific to 1.26.1.

Hopefully these additional cases help narrow it down.

6

I understand that these timeout cases are for a cold cache (or a cache that does not have the required data at least).
Do consecutive queries eventually work for these? I expect that they eventually work.

7

In my case, they do not eventually work reliably.

my-account.spectator.co.uk repeatedly times out when queried directly against Unbound with DNSSEC enabled. I also tested the underlying my-account.coeditor.io name directly against Unbound, and repeated queries continued to time out.

I flushed the Unbound cache for coeditor.io:

unbound-control flush_zone coeditor.io

which reported that 2 RRsets, 1 message and 0 key entries were removed. Retrying the query afterwards still resulted in repeated timeouts.

By comparison, the same names resolve immediately through Cloudflare (1.1.1.1), and the authoritative Cloudflare nameserver for coeditor.io also responds immediately.

So at least for my case, this doesn’t appear to be simply a cold-cache delay where a subsequent query eventually succeeds.

8

How about asking d35ystitntcfyq.cloudfront.net. directly to Unbound?
This is the last node in the CNAME chain for my-account.spectator.co.uk..

9

According to the dnssec analyzer, the domain is broken: DNSSEC Debugger - my-account.spectator.co.uk

10

Hi Yorgos,

I have already tested the final CNAME target directly against Unbound:

dig d35ystitntcfyq.cloudfront.net @127.0.0.1 -p 5335 +dnssec

This resolves successfully and immediately, returning NOERROR and four A records.

The failure occurs when resolving the preceding names:

my-account.spectator.co.uk → timeout
my-account.coeditor.io → timeout

I also queried the authoritative Cloudflare nameserver for coeditor.io directly, and my-account.coeditor.io returns immediately.

So the CloudFront endpoint itself appears not to be the problem; the failure seems to occur while Unbound is processing the preceding part of the chain/delegation/DNSSEC validation.

Jaap’s observation about the DNSSEC analyser showing my-account.spectator.co.uk as broken is therefore particularly interesting.

11

I don’t think those domains are broken.
It seems that dnssec debugger chooses to use red and 'X’es for insecure domains. Maybe dnssec debugger is only meant for DNSSEC protected domains.
Or at least this is what I understand from the output.

What I do see though is that “some” (one?) io nameservers do not respond over UDP. This triggers timeout logic from Unbound and could lead to longer resolution times.

I also see that for the io. DNSKEY answer, Unbound needs to reconnect over TCP.

Locally it works here for my-account.spectator.co.uk, but given which nameservers are contacted, it may require more time.

In your case I get the feeling that Unbound maybe cannot fetch the io. DNSKEY?

12

Hi Yorgos,

I tested the .io DNSKEY lookup directly against my Unbound:



$ dig io. DNSKEY @127.0.0.1 -p 5335 +dnssec

;; communications error to 127.0.0.1#5335: timed out
;; communications error to 127.0.0.1#5335: timed out
;; communications error to 127.0.0.1#5335: timed out

;; no servers could be reached

So yes, in my case Unbound is unable to retrieve the io. DNSKEY and the query times out.

This seems consistent with your observation that the .io DNSKEY lookup may be where my resolution is getting stuck.

13

I don’t suppose you have TCP turned off (do-tcp: no).
Also currently you need TCP to fetch the root key which seems to work for other domains where DNSSEC succeeds there.
Can you try both:
dig -4 @a.root-servers.net . dnskey +dnssec +norecurse +bufsize=1232
and
dig -4 @a2.nic.io io dnskey +dnssec +norecurse +bufsize=1232
from the server Unbound is running from?

I would expect the first to work and the second not.

14

Hi Yorgos,

I ran the first test from the Unbound server:



$ dig -4 @a.root-servers.net . dnskey +dnssec +norecurse +bufsize=1232

;; Truncated, retrying in TCP mode.

;; communications error to 198.41.0.4#53: timed out
;; communications error to 198.41.0.4#53: timed out
;; communications error to 198.41.0.4#53: timed out

;; no servers could be reached

So the root DNSKEY query also fails when it falls back to TCP.

This wasn’t the result you expected, so presumably this gives us another clue.

15

I have found another unrelated iPhone app that exhibits the same behaviour: the Shell fuel/rewards app.

The phone is connected to my Pi via WireGuard.

With Pi-hole using Unbound, the Shell app does not work.

If I change Pi-hole back to Cloudflare DNS, the Shell app works immediately over the same WireGuard connection.

If I disable WireGuard and use the mobile provider’s DNS, it also works.

So this appears to be another independent application showing the same Unbound-specific behaviour.

This seems consistent with the root DNSKEY test we just performed, where UDP truncates and the subsequent TCP connection to the root server times out.

16

I am getting the feeling that something upstream of your Pi(Unbound) is blocking DNS on TCP which is bad as you can see; and violates the standard
Can you try the two dig commands for DNSKEY from another machine in the same network as the Pi?
If you still get timeouts, maybe some middleware/firewall or your ISP?

17

Hi Yorgos,

I ran both tests from my iMac, which is on the same network as the Pi but is not using Unbound:



$ dig -4 @a.root-servers.net . dnskey +dnssec +norecurse +bufsize=1232

;; Truncated, retrying in TCP mode.
...
;; connection timed out; no servers could be reached

And:



$ dig -4 @a2.nic.io io dnskey +dnssec +norecurse +bufsize=1232

;; Truncated, retrying in TCP mode.
...
;; connection timed out; no servers could be reached

So both DNSKEY queries also fail when the TCP fallback is attempted from the iMac.

This seems to confirm that TCP DNS is being blocked somewhere upstream of the Pi, rather than being an Unbound-specific problem.

18

I’m unable to test with another system in my environment, but I’ve run the tests locally.

$ dig -4 @65.22.163.17 io. dnskey +dnssec +norecurse +bufsize=1232
;; Truncated, retrying in TCP mode.
;; communications error to 65.22.163.17#53: timed out
;; communications error to 65.22.163.17#53: timed out
;; communications error to 65.22.163.17#53: timed out
[...]
;; no servers could be reached

$ dig -4 @8.8.8.8 io. dnskey +dnssec +norecurse +bufsize=1232
;; Truncated, retrying in TCP mode.
;; communications error to 8.8.8.8#53: timed out
;; communications error to 8.8.8.8#53: timed out
;; communications error to 8.8.8.8#53: timed out
[...]
;; no servers could be reached

These queries never reach Unbound (confirmed via strace), which indeed indicate this is a network-level issue upstream, rather than an Unbound-specific problem.
Since this is outside our control, we’ve implemented DNS-over-TLS with an open DNS resolver as a workaround.

Thank you all for your help!

19

I’ve experienced random, inexplicable time-outs with some domains as well, but w/o DNSSEC. My gut feeling was that it mostly involved cloudlfare dot com, but others were affected as well. For some domains it was very pervasive, notably geoip dot maxmind dot com which is behind cloudflare.com.

Dots . fill in for the stupid 2x link rule.