Trying to understand a SERVFAIL

Pim_van_Pelt · December 28, 2009, 6:25pm

Hoi,

I've loaded two zones, paphosting.net and l.paphosting.net (see [1])
on three nameservers (ns.paphosting.{net,nl.eu}). These nameservers
serve requests generally fine and authoritatively (for example,
http0.l.paphosting.net resolves to two A and two AAAA records).
However, on some resolvers (and some, but not all client),
intermittently, I get unexpected answers (SERVFAIL), and looking at
tcpdump output it seems that perhaps there is a bit of a weird answer
coming from NSD.

The client (on hispeed.ch, a caching nameserver running bind 9.4.2 on
OpenBSD 4.3) is speaking to an NSD (on bit.nl, an nsd authoritative
nameserver running 3.2.2 on Linux 2.6/Ubuntu LTS 8.04). Using host(1),
I can expose this issue, while using dig(1) I cannot. So running
"""host -t A www.paphosting.net 192.168.2.1""", here's the resulting
UDP conversation[2] - it shows SERVFAIL, and seems to try each of the
3 nameservers twice, in turn, before giving up.

I observed each NSD giving an odd answer:
http0.l.paphosting.net. A nlede01.paphosting.net.122.109.193.in-addr.arpa
http0.l.paphosting.net. A http.weirdnet.nl

the first record is not my intention, and perhaps a clue. There are
sibling domains on the NSD authoritative nameservers (paphosting.nl
and paphosting.eu) and for them, looking up www.paphosting.{nl,eu}
works fine and those lookups do not show the .arpa reply.

This is an intriguing problem to me, because a simple method exists
for populating the cache, see here:
$ host www.paphosting.net 192.168.2.1
Using domain server:
Name: 192.168.2.1
Address: 192.168.2.1#53
Aliases:

Host www.paphosting.net not found: 2(SERVFAIL)
$ dig @192.168.2.1 ANY www.paphosting.net

; <<>> DiG 9.4.2 <<>> @192.168.2.1 ANY www.paphosting.net
; (1 server found)
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44766
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 5

;; QUESTION SECTION:
;www.paphosting.net. IN ANY

;; ANSWER SECTION:
www.paphosting.net. 86400 IN CNAME http0.l.paphosting.net.

;; AUTHORITY SECTION:
paphosting.net. 84971 IN NS ns.paphosting.eu.
paphosting.net. 84971 IN NS ns.paphosting.net.
paphosting.net. 84971 IN NS ns.paphosting.nl.

;; ADDITIONAL SECTION:
ns.paphosting.nl. 84971 IN A 94.142.245.3
ns.paphosting.nl. 84971 IN AAAA 2a02:898:28::3
ns.paphosting.net. 84971 IN AAAA 2001:7b8:3:47:20d:b9ff:fe14:70d4
ns.paphosting.eu. 84970 IN A 62.220.146.194
ns.paphosting.eu. 84971 IN AAAA 2001:788:2:117::2

;; Query time: 25 msec
;; SERVER: 192.168.2.1#53(192.168.2.1)
;; WHEN: Mon Dec 28 19:13:39 2009
;; MSG SIZE rcvd: 251

$ host www.paphosting.net 192.168.2.1
Using domain server:
Name: 192.168.2.1
Address: 192.168.2.1#53
Aliases:

www.paphosting.net is an alias for http0.l.paphosting.net.
http0.l.paphosting.net has address 94.142.245.2
http0.l.paphosting.net has address 193.109.122.243
http0.l.paphosting.net has IPv6 address 2a02:898:28::2
http0.l.paphosting.net has IPv6 address 2001:7b8:3:4f:216:3eff:fe4b:ae79

and from then on, the cache understands what I mean. Does anybody
have any clues? Are there things I can do to better understand this
issue? This issue can be observed from the internet - I'd be happy to
facilitate any necessary debugging (for example, I can try to load this
data on another auth nameserver like bind or powerdns).

groet,
Pim

[1]
$ORIGIN paphosting.net.
$TTL 86400
@ IN SOA ns.paphosting.net. hostmaster.paphosting.net. (
2009122501
28800
7200
604800
86400
)

@ NS ns.paphosting.nl.
@ NS ns.paphosting.net.
@ NS ns.paphosting.eu.
@ MX 10 mx0.paphosting.net.
@ MX 20 mx1.paphosting.net.
@ MX 30 mx2.paphosting.net.
; DNS
ns AAAA 2001:7b8:3:47:20d:b9ff:fe14:70d4
ns A 213.154.229.21
l NS ns.paphosting.nl.
l NS ns.paphosting.net.
l NS ns.paphosting.eu.

www CNAME http0.l.paphosting.net.

$ORIGIN l.paphosting.net.
$TTL 300
@ IN SOA ns.paphosting.net. hostmaster.paphosting.net. (
2009121300
28800 ; Refresh
7200 ; Retry
604800 ; Expire
300 ; Minimum
)
@ IN NS ns.paphosting.nl.
@ IN NS ns.paphosting.net.
@ IN NS ns.paphosting.eu.
; TODO: Autogenerate this zonefile based on a configfile

; nl.ede, BIT, Ede, Netherlands
http0 IN A 193.109.122.243
http0 IN AAAA 2001:7b8:3:4f:216:3eff:fe4b:ae79

; nl.ams, Coloclue, Amsterdam, Netherlands
http0 IN A 94.142.245.2
http0 IN AAAA 2a02:898:28::2

[2] tcpdump while typing 'host www.paphosting.net 192.168.2.1' on an
empty bind 9.4.2 resolver
18:51:08.146580 00:0d:b9:14:70:70 00:17:10:00:c9:34 ip 89:
77-56-102-109.dclient.hispeed.ch.45465 > 62.220.146.194.domain: [udp
sum ok] 53168% [1au] A? www.paphosting.net. ar: . OPT UDPsize=4096
(47) (ttl 64, id 46989, len 75)
18:51:08.164377 00:17:10:00:c9:34 00:0d:b9:14:70:70 ip 324:
62.220.146.194.domain > 77-56-102-109.dclient.hispeed.ch.45465: [udp
sum ok] 53168*- q: A? www.paphosting.net. 3/6/3 www.paphosting.net.
CNAME http0.l.paphosting.net., http0.l.paphosting.net. A
nlede01.paphosting.net.122.109.193.in-addr.arpa,
http0.l.paphosting.net. A http.weirdnet.nl ns: l.paphosting.net. NS
ns.paphosting.nl., l.paphosting.net. NS ns.paphosting.net.,
l.paphosting.net. NS ns.paphosting.eu., paphosting.net. NS
ns.paphosting.nl., paphosting.net. NS ns.paphosting.net.,
paphosting.net. NS ns.paphosting.eu. ar: ns.paphosting.net. A
console.colo.bit.nl, ns.paphosting.net. AAAA
2001:7b8:3:47:20d:b9ff:fe14:70d4, . OPT UDPsize=4096 (282) [tos 0x20]
(ttl 53, id 8308, len 310)
18:51:08.165146 00:0d:b9:14:70:70 00:17:10:00:c9:34 ip 89:
77-56-102-109.dclient.hispeed.ch.45465 > console.colo.bit.nl.domain:
[udp sum ok] 64115% [1au] A? www.paphosting.net. ar: . OPT
UDPsize=4096 (47) (ttl 64, id 15411, len 75)
18:51:08.202528 00:17:10:00:c9:34 00:0d:b9:14:70:70 ip 324:
console.colo.bit.nl.domain > 77-56-102-109.dclient.hispeed.ch.45465:
[udp sum ok] 64115*- q: A? www.paphosting.net. 3/6/3
www.paphosting.net. CNAME http0.l.paphosting.net.,
http0.l.paphosting.net. A
nlede01.paphosting.net.122.109.193.in-addr.arpa,
http0.l.paphosting.net. A http.weirdnet.nl ns: l.paphosting.net. NS
ns.paphosting.nl., l.paphosting.net. NS ns.paphosting.net.,
l.paphosting.net. NS ns.paphosting.eu., paphosting.net. NS
ns.paphosting.nl., paphosting.net. NS ns.paphosting.net.,
paphosting.net. NS ns.paphosting.eu. ar: ns.paphosting.net. A
console.colo.bit.nl, ns.paphosting.net. AAAA
2001:7b8:3:47:20d:b9ff:fe14:70d4, . OPT UDPsize=4096 (282) (ttl 55, id
35079, len 310)
18:51:08.203184 00:0d:b9:14:70:70 00:17:10:00:c9:34 ip 89:
77-56-102-109.dclient.hispeed.ch.45465 > ns1.weirdnet.nl.domain: [udp
sum ok] 29356% [1au] A? www.paphosting.net. ar: . OPT UDPsize=4096
(47) (ttl 64, id 1335, len 75)
18:51:08.241402 00:17:10:00:c9:34 00:0d:b9:14:70:70 ip 324:
ns1.weirdnet.nl.domain > 77-56-102-109.dclient.hispeed.ch.45465: [udp
sum ok] 29356*- q: A? www.paphosting.net. 3/6/3 www.paphosting.net.
CNAME http0.l.paphosting.net., http0.l.paphosting.net. A
nlede01.paphosting.net.122.109.193.in-addr.arpa,
http0.l.paphosting.net. A http.weirdnet.nl ns: l.paphosting.net. NS
ns.paphosting.nl., l.paphosting.net. NS ns.paphosting.net.,
l.paphosting.net. NS ns.paphosting.eu., paphosting.net. NS
ns.paphosting.nl., paphosting.net. NS ns.paphosting.net.,
paphosting.net. NS ns.paphosting.eu. ar: ns.paphosting.net. A
console.colo.bit.nl, ns.paphosting.net. AAAA
2001:7b8:3:47:20d:b9ff:fe14:70d4, . OPT UDPsize=4096 (282) (ttl 54, id
57236, len 310)
18:51:08.249549 00:0d:b9:14:70:70 00:17:10:00:c9:34 ip 89:
77-56-102-109.dclient.hispeed.ch.45465 > 62.220.146.194.domain: [udp
sum ok] 64826% [1au] A? www.paphosting.net. ar: . OPT UDPsize=4096
(47) (ttl 64, id 7106, len 75)
18:51:08.266204 00:17:10:00:c9:34 00:0d:b9:14:70:70 ip 324:
62.220.146.194.domain > 77-56-102-109.dclient.hispeed.ch.45465: [udp
sum ok] 64826*- q: A? www.paphosting.net. 3/6/3 www.paphosting.net.
CNAME http0.l.paphosting.net., http0.l.paphosting.net. A
nlede01.paphosting.net.122.109.193.in-addr.arpa,
http0.l.paphosting.net. A http.weirdnet.nl ns: l.paphosting.net. NS
ns.paphosting.nl., l.paphosting.net. NS ns.paphosting.net.,
l.paphosting.net. NS ns.paphosting.eu., paphosting.net. NS
ns.paphosting.nl., paphosting.net. NS ns.paphosting.net.,
paphosting.net. NS ns.paphosting.eu. ar: ns.paphosting.net. A
console.colo.bit.nl, ns.paphosting.net. AAAA
2001:7b8:3:47:20d:b9ff:fe14:70d4, . OPT UDPsize=4096 (282) [tos 0x20]
(ttl 53, id 34156, len 310)
18:51:08.266877 00:0d:b9:14:70:70 00:17:10:00:c9:34 ip 89:
77-56-102-109.dclient.hispeed.ch.45465 > console.colo.bit.nl.domain:
[udp sum ok] 34265% [1au] A? www.paphosting.net. ar: . OPT
UDPsize=4096 (47) (ttl 64, id 56895, len 75)
18:51:08.304342 00:17:10:00:c9:34 00:0d:b9:14:70:70 ip 324:
console.colo.bit.nl.domain > 77-56-102-109.dclient.hispeed.ch.45465:
[udp sum ok] 34265*- q: A? www.paphosting.net. 3/6/3
www.paphosting.net. CNAME http0.l.paphosting.net.,
http0.l.paphosting.net. A
nlede01.paphosting.net.122.109.193.in-addr.arpa,
http0.l.paphosting.net. A http.weirdnet.nl ns: l.paphosting.net. NS
ns.paphosting.nl., l.paphosting.net. NS ns.paphosting.net.,
l.paphosting.net. NS ns.paphosting.eu., paphosting.net. NS
ns.paphosting.nl., paphosting.net. NS ns.paphosting.net.,
paphosting.net. NS ns.paphosting.eu. ar: ns.paphosting.net. A
console.colo.bit.nl, ns.paphosting.net. AAAA
2001:7b8:3:47:20d:b9ff:fe14:70d4, . OPT UDPsize=4096 (282) (ttl 55, id
62916, len 310)
18:51:08.304972 00:0d:b9:14:70:70 00:17:10:00:c9:34 ip 89:
77-56-102-109.dclient.hispeed.ch.45465 > ns1.weirdnet.nl.domain: [udp
sum ok] 43899% [1au] A? www.paphosting.net. ar: . OPT UDPsize=4096
(47) (ttl 64, id 36756, len 75)
18:51:08.343809 00:17:10:00:c9:34 00:0d:b9:14:70:70 ip 324:
ns1.weirdnet.nl.domain > 77-56-102-109.dclient.hispeed.ch.45465: [udp
sum ok] 43899*- q: A? www.paphosting.net. 3/6/3 www.paphosting.net.
CNAME http0.l.paphosting.net., http0.l.paphosting.net. A
nlede01.paphosting.net.122.109.193.in-addr.arpa,
http0.l.paphosting.net. A http.weirdnet.nl ns: l.paphosting.net. NS
ns.paphosting.nl., l.paphosting.net. NS ns.paphosting.net.,
l.paphosting.net. NS ns.paphosting.eu., paphosting.net. NS
ns.paphosting.nl., paphosting.net. NS ns.paphosting.net.,
paphosting.net. NS ns.paphosting.eu. ar: ns.paphosting.net. A
console.colo.bit.nl, ns.paphosting.net. AAAA
2001:7b8:3:47:20d:b9ff:fe14:70d4, . OPT UDPsize=4096 (282) (ttl 54, id
24040, len 310)

Pim_van_Pelt · December 28, 2009, 8:53pm

Hoi Jeremy,

multiple NS RRsets in authority section

This seems correct to me.

;; AUTHORITY SECTION:
l.paphosting.net. 300 IN NS ns.paphosting.nl.
l.paphosting.net. 300 IN NS ns.paphosting.net.
l.paphosting.net. 300 IN NS ns.paphosting.eu.
paphosting.net. 86400 IN NS ns.paphosting.nl.
paphosting.net. 86400 IN NS ns.paphosting.net.
paphosting.net. 86400 IN NS ns.paphosting.eu.

(I don't know why.)

this is because www.paphosting.net is a CNAME to
http0.l.paphosting.net (which is a different zone, the nameservers of
which are the same but their TTL is 300s).

groet,
Pim

Pim_van_Pelt · December 28, 2009, 10:03pm

Hoi,

Addendum, see inline:

Hoi,

I've loaded two zones, paphosting.net and l.paphosting.net (see [1])
on three nameservers (ns.paphosting.{net,nl.eu}). These nameservers
serve requests generally fine and authoritatively (for example,
http0.l.paphosting.net resolves to two A and two AAAA records).
However, on some resolvers (and some, but not all client),
intermittently, I get unexpected answers (SERVFAIL), and looking at
tcpdump output it seems that perhaps there is a bit of a weird answer
coming from NSD.

The client (on hispeed.ch, a caching nameserver running bind 9.4.2 on
OpenBSD 4.3) is speaking to an NSD (on bit.nl, an nsd authoritative
nameserver running 3.2.2 on Linux 2.6/Ubuntu LTS 8.04). Using host(1),
I can expose this issue, while using dig(1) I cannot. So running
"""host -t A www.paphosting.net 192.168.2.1""", here's the resulting
UDP conversation[2] - it shows SERVFAIL, and seems to try each of the
3 nameservers twice, in turn, before giving up.

I observed each NSD giving an odd answer:
http0.l.paphosting.net. A nlede01.paphosting.net.122.109.193.in-addr.arpa
http0.l.paphosting.net. A http.weirdnet.nl

the first record is not my intention, and perhaps a clue.

This is not a clue, because in 122.109.193.in-addr.arpa., also served
by the same nameservers, I had omitted the trailing '.' behind
nlede01.paphosting.net; which made it become that *.arpa address. This
is now fixed,
but the issue remains. The rest is simply an artifact from tcpdump,
which translates all IP addresses to names

The issue itself, as described in the original post, remains.

Wouter · January 1, 2010, 12:07am

Hi Pim, Jeremy,

This response looks like a corner case. I think it may trigger that bad
behaviour in some resolvers. This may be something that is caused by
new 'Kaminksy-era-paranoia' fixes in resolvers. I see that this
response triggers drastic cutting measures in unbound (but it does work
there), perhaps BIND does something as well.

One operational fix would be to integrate both zones - no zone cut, SOA
or NS records for l.paphosting.net. Simply put the information into the
paphosting.net zone. This is an operational fix, perhaps BIND and NSD
code need fixes.

http0.l IN A 193.109.122.243
http0.l IN AAAA 2001:7b8:3:4f:216:3eff:fe4b:ae79
http0.l IN A 94.142.245.2
http0.l IN AAAA 2a02:898:28::2

Since many resolvers won't use the information after the CNAME from the
first response anyway, perhaps NSD should not descend into the next zone.

Best regards,
Wouter

Pim_van_Pelt · January 31, 2010, 2:39pm

Hoi Wouter, Colleagues,

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Pim, Jeremy,

This response looks like a corner case. I think it may trigger that bad
behaviour in some resolvers. This may be something that is caused by
new 'Kaminksy-era-paranoia' fixes in resolvers.

I have seen the unwanted behavior in second zone that I loaded from a
bind9 authorative to an nsd one:
$ORIGIN sixxs.net.
m NS ns1 NS ns2 NS ns3
tic CNAME tic.m

and m.sixxs.net runs on the bind9 authoritative servers. A query
coming to tic.sixxs.net fails, when the NSD gets it, it serves out a
reply but it is not understood by all resolvers.

I think this is an issue that can likely be fixed in NSD even if it is
an issue also in bind (resolver). Where can I file a bug against it?
Should this discussion be brought broader (so the teams can hash it
out amongst themselves how to best fix it?). If so - can you help me
get the right people aligned? I've not posted to name droppers lists
since quite a few years

Wouter · February 1, 2010, 10:03am

Hi Pim, Jeremy,

Yes that would also create a parent and its child zone on the same
server with a CNAME between them triggering double 'additional section
processing', resulting in a message with two NS rrsets in the authority
section out of NSD, which the BIND resolver rejects.

Perhaps this can be fixed in NSD - since two NS sets is simply very
large and we can make the responses smaller in this case. Probably by
including only the 'first' NS set (for the source of the CNAME),
although many resolvers immediately query for the destination of the CNAME.

I believe isc has a mailing list, forum, bind9-bugs@isc.org ticket
system ... so it should not be too hard to tell them if you want. I am
not completely sure what the perfect fix is either.

Best regards,
Wouter