Validating failure against Comcast forwarders

Unbound
1

hi,

i'm trying to debug a validation failure for the name
"businessipv6.trials.comcast.net". it only occurs when i use comcast's
DNSSEC-enabled recursives as forwarders for unbound (75.75.75.75,
2001:558:feed::1). i see debug messages in syslog from unbound like
"CNAME response was wildcard expansion and did not prove original data
did not exist". is there a bug in unbound or in comcast's responses?
if the latter, i will report it to them.

here is some debug output: first query is to unbound operating in full
recursive mode, which successfully validates; second query is to unbound
operating in forwarding mode, which returns SERVFAIL; third query is
directly to one of comcast's validating recursive servers, which returns
a response with the 'AD' bit.

i've also attached data from a separate run demonstrating the issue.
(unbound-control dump_cache, unbound-control dump_infra, syslog with
verbosity 4, and packet capture.)

root@chase{0}:~# unbound-control forward
off (using root hints)
root@chase{0}:~# dig +dnssec @::1 businessipv6.trials.comcast.net

; <<>> DiG 9.9.3-P2 <<>> +dnssec @::1 businessipv6.trials.comcast.net
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10566
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 4, AUTHORITY: 8, ADDITIONAL: 21

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;businessipv6.trials.comcast.net. IN A

;; ANSWER SECTION:
businessipv6.trials.comcast.net. 7200 IN CNAME trials.comcast.net.
businessipv6.trials.comcast.net. 7200 IN RRSIG CNAME 5 3 7200 20130819220517 20130812190017 52026 comcast.net. Qf1+jCdKnul/LJLbNsXDCwa2gDAjFEFfpQ3p6AXjDcdean88D/GpiyqS ibXlGLwHNrCQtPdabAcScgcega1sOayFUTPcb7A1lJ1OBFazONWQZjZg kq8tA+51Sl7Gxik4bFhmeDob5pTfZz06IEKEbwi6cPq5lxZ7Xxzh/svt 3wk=
trials.comcast.net. 7200 IN A 69.241.25.127
trials.comcast.net. 7200 IN RRSIG A 5 3 7200 20130819220517 20130812190017 52026 comcast.net. 43ohDOeBaDWah1rKIKABxFEAwIAsKIPUVWLXJ9lp21m83ccxqzw0uQJv qhcxekcJFYEDUCJFwn2j8THZWlCKM+jro+0KOPqMsVGaWkND0EDxwXuE 5buknodCkn6q0fjHAnXW8cXZ68tmC8eCXYoZUJISzmspBYrcyynjunUo OZs=

;; AUTHORITY SECTION:
*.trials.comcast.net. 3600 IN NSEC troubleshooting.comcast.net. CNAME RRSIG NSEC
*.trials.comcast.net. 3600 IN RRSIG NSEC 5 3 3600 20130819220517 20130812190017 52026 comcast.net. uRrFYkj5tKT0eJCl93Jcw5g+Pf2sOrACse2VA/zwmEeEwj9D85lU8qo/ QwpCV+VHs533vXNNsiYYdCW54BhH68YGu7maNktf0l0yJqqmANg+4U26 A9Q5aqiKq0ZnTrjis3Uk0TRq0rIMPZreS6DsLro/GEgEWtDde5Gp9tUu n1s=
comcast.net. 7200 IN NS dns103.comcast.net.
comcast.net. 7200 IN NS dns104.comcast.net.
comcast.net. 7200 IN NS dns105.comcast.net.
comcast.net. 7200 IN NS dns102.comcast.net.
comcast.net. 7200 IN NS dns101.comcast.net.
comcast.net. 7200 IN RRSIG NS 5 2 7200 20130819220517 20130812190017 52026 comcast.net. A7eTXBXu4UuAhzaBSeRtcTAFsSP+GX9I9uyr3MF3KrWijVDQQW0pgCN6 S+TI+Otpi7C/mvjym3UP4qzM1n8/Xjifh8S/JmtE5h2kEqpNiHFB1Amc NKuSaTJlqN0b36B/Ux+9NoFomZsN2gJ1souTEiff0IaEu4g+2t9Df0W6 fQo=

;; ADDITIONAL SECTION:
dns103.comcast.net. 7200 IN AAAA 2001:558:1014:c:68:87:76:228
dns104.comcast.net. 7200 IN AAAA 2001:558:100a:5:68:87:68:244
dns105.comcast.net. 7200 IN AAAA 2001:558:100e:5:68:87:72:244
dns102.comcast.net. 7200 IN AAAA 2001:558:1004:7:68:87:85:132
dns101.comcast.net. 7200 IN AAAA 2001:558:fe23:8:69:252:250:103
dns103.comcast.net. 7200 IN A 68.87.76.228
dns104.comcast.net. 7200 IN A 68.87.68.244
dns105.comcast.net. 7200 IN A 68.87.72.244
dns102.comcast.net. 7200 IN A 68.87.85.132
dns101.comcast.net. 7200 IN A 69.252.250.103
dns103.comcast.net. 7200 IN RRSIG AAAA 5 3 7200 20130819220517 20130812190017 52026 comcast.net. R2otbBFPIrgSwRrUjgLOsXe3hLpjBhKJA1o3emUn9NZzR2LBvYE4uOiZ MnOyi06WkM/Yg2t0MxfGE4YV7E91IKvQj4AhWXyuy9FUl+eHDF8Ivu70 UVM3zm+VFz/xDolXxRiVoCO/Z/ai5eXp0Y5EhXZXXcuGzOmnKsFXgcmA qBY=
dns104.comcast.net. 7200 IN RRSIG AAAA 5 3 7200 20130819220517 20130812190017 52026 comcast.net. vbsLLYzuULtGjVprUSbsByJ7G9anDH7HmqGioiHFRG/b3lAqlCL7Gn06 65kF9JeAcjBEYuHDnc698jU5VahBoCS17dAty3RH4utzDWhRj5AW0sVS GY+844Do+al3PgK4D9CS9Re4DpjjNA+m1SyC6r3ihMyw/SBMeo7ZmFwz SGw=
dns105.comcast.net. 7200 IN RRSIG AAAA 5 3 7200 20130819220517 20130812190017 52026 comcast.net. QUcw5f4xKpdfOJJ0uXaJBnSjtRdpi0qiWNZbKR2kBBFuTzWlhenL9fon Gdn3ACtw5n7zKHFFHcyJgP+FuOJZt4gRPJRN9W4OpxlK6O+LEI/J5Jsw Y29Yt7sCJKcQnp81Stx8iUyXUzt6YgyVv/GZiuqUyuyjuq9rgoFT0TEp Kj8=
dns102.comcast.net. 7200 IN RRSIG AAAA 5 3 7200 20130819220517 20130812190017 52026 comcast.net. J1Lrk4fSw576t949j2KojwNjwQQxLt/qbjZP85JJeZ+8LPFVDfCi9aSs 2sETuWoBEyfyvB6wjrKiAjg4BrmgmB7vLV1/yuLvr/8YnPANe+bkIezi cRvhhYVodsNnj5u/xPCgNti1PRVsdVk7SgqrPjxRs6GHucn53+mvhsUI DFU=
dns101.comcast.net. 7200 IN RRSIG AAAA 5 3 7200 20130819220517 20130812190017 52026 comcast.net. on7EYhQAp0v7GmKHcLi+4V6ED4edYbLmnoP+BmJvLTkVDFkIPw6oGsip Cjl/sWzS6unrN8P9tt7HpYYjr9w9iZfOtjJ796Gp8o+ViQx8+QZmjnpc rnLZaHgUIUrSBMUni3XoxO63QGnzHWdlcpLf+cTOYhghFbGZsYy9zt6R JT8=
dns103.comcast.net. 7200 IN RRSIG A 5 3 7200 20130819220517 20130812190017 52026 comcast.net. zAZDI93qbalClGpRb4jAvMFvXt6sQPTesin743/M/5VgXOrLNXpKkCcW vSu0uR/slrSszl4yz1PbQN9TVoJKVs4f6F3iUOolesvZs5WTfYYJRzCy pEtWLZNsKXzA/x7IA371F0T+oGb8qtp6mENzTgbu89FHrcUr0Y/+vL6L hoM=
dns104.comcast.net. 7200 IN RRSIG A 5 3 7200 20130819220517 20130812190017 52026 comcast.net. cVMR/SVzbnP8Ut+b3LfztEj65OzgQQeNihhTLSLlPuOxuUOb2Iu6ryer pE9VaN8S/pxg6ftWxWttmitufJeabmWs+493jtTWssE6eM5CGCyJxOal 0XxjaGuj4f0iOnh763jzMGxPMSKRFQQEbLh499vVq4jMz/T+dXCoC0c2 lk8=
dns105.comcast.net. 7200 IN RRSIG A 5 3 7200 20130819220517 20130812190017 52026 comcast.net. TdB00q6wEZ9+o4rSeLtjkozdT0RKsTQglip7+hnfvrb7oZ98ZWd/Ldr5 XOzhsO+vDI0QGGL5HGYLvsMaXuKjpbCEAioIJ/RzzwCRuvXSCSa8/HMf 2cMD8dwZLE2YyRzgLaL+Om0xUbWl1KQ34c4czul+DOLFyQvmyPIGTWWH G3s=
dns102.comcast.net. 7200 IN RRSIG A 5 3 7200 20130819220517 20130812190017 52026 comcast.net. INMxmoy9jDfasHYJRrl/LXcWiOQJmDgE6bdJ8tT58R1rje9KTtNSlJ73 /6opL982HsN6UMOI14wszP+mL3ajBnAy67TY8Ssff7Vu4QZfHjsrJm/h NfK4SmJGp2puJvJnusxdD0XGwQYG/j+lsd/1nEbf6sXeJeOUDRbhGf1j rAY=
dns101.comcast.net. 7200 IN RRSIG A 5 3 7200 20130819220517 20130812190017 52026 comcast.net. DPOqgXwJxIGlSyDuXtgL0PVPlGUnZjifKY6V1YxRamUrxzGyksgyAYmD +7loyfIH39hrJB7mADOgtf8jOprs/P4uS6KJX96sDNzC9xWcxq7JOPGc RxX9/+RxHjl6lyNONgtmL7aNx+l9G676IsiudoS6/OJcmqs0gXANgjFq O/U=

;; Query time: 963 msec
;; SERVER: ::1#53(::1)
;; WHEN: Tue Aug 13 10:41:35 EDT 2013
;; MSG SIZE rcvd: 2860

root@chase{0}:~# unbound-control forward 75.75.75.75 2001:558:feed::1
ok
root@chase{0}:~# unbound-control flush_zone "."
ok removed 73 rrsets, 27 messages and 3 key entries
root@chase{0}:~# dig +dnssec @::1 businessipv6.trials.comcast.net

; <<>> DiG 9.9.3-P2 <<>> +dnssec @::1 businessipv6.trials.comcast.net
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 54096
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;businessipv6.trials.comcast.net. IN A

;; Query time: 318 msec
;; SERVER: ::1#53(::1)
;; WHEN: Tue Aug 13 10:42:15 EDT 2013
;; MSG SIZE rcvd: 60

root@chase{0}:~# dig +dnssec @2001:558:feed::1 businessipv6.trials.comcast.net

; <<>> DiG 9.9.3-P2 <<>> +dnssec @2001:558:feed::1 businessipv6.trials.comcast.net
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22099
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4000
;; QUESTION SECTION:
;businessipv6.trials.comcast.net. IN A

;; ANSWER SECTION:
businessipv6.trials.comcast.net. 1338 IN CNAME trials.comcast.net.
businessipv6.trials.comcast.net. 1338 IN RRSIG CNAME 5 3 7200 20130819220517 20130812190017 52026 comcast.net. Qf1+jCdKnul/LJLbNsXDCwa2gDAjFEFfpQ3p6AXjDcdean88D/GpiyqS ibXlGLwHNrCQtPdabAcScgcega1sOayFUTPcb7A1lJ1OBFazONWQZjZg kq8tA+51Sl7Gxik4bFhmeDob5pTfZz06IEKEbwi6cPq5lxZ7Xxzh/svt 3wk=
trials.comcast.net. 1338 IN A 69.241.25.127
trials.comcast.net. 1338 IN RRSIG A 5 3 7200 20130819220517 20130812190017 52026 comcast.net. 43ohDOeBaDWah1rKIKABxFEAwIAsKIPUVWLXJ9lp21m83ccxqzw0uQJv qhcxekcJFYEDUCJFwn2j8THZWlCKM+jro+0KOPqMsVGaWkND0EDxwXuE 5buknodCkn6q0fjHAnXW8cXZ68tmC8eCXYoZUJISzmspBYrcyynjunUo OZs=

;; Query time: 11 msec
;; SERVER: 2001:558:feed::1#53(2001:558:feed::1)
;; WHEN: Tue Aug 13 10:42:26 EDT 2013
;; MSG SIZE rcvd: 432

root@chase{0}:~#

(attachments)

businessipv6.trials.comcast.net.cache (5.51 KB)
businessipv6.trials.comcast.net.infra (267 Bytes)
businessipv6.trials.comcast.net.log (150 KB)
businessipv6.trials.comcast.net.pcap (11.8 KB)

2

Hi Robert,

hi,

i'm trying to debug a validation failure for the name
"businessipv6.trials.comcast.net". it only occurs when i use
comcast's DNSSEC-enabled recursives as forwarders for unbound
(75.75.75.75, 2001:558:feed::1). i see debug messages in syslog
from unbound like "CNAME response was wildcard expansion and did
not prove original data did not exist". is there a bug in unbound
or in comcast's responses? if the latter, i will report it to
them.

It seems to be a bug in comcast's, it looks like the BIND bug where it
omitted NSECs for wildcard expansion.

When you dig at comcast's servers you get the CNAME response but no
NSEC in the authority section. This is the problem, and causes
validation failure. That NSEC (or an NSEC3 for an NSEC3 domain, but
this domain has NSEC) has to be there to prove that the query name
does not exist and thus the wildcard expansion must be used.

In unbound's working response this is the NSEC:
*.trials.comcast.net. 3600 IN NSEC
troubleshooting.comcast.net. CNAME RRSIG NSEC
[ and an RRSIG over it ].

If this is the BIND bug, then it has been fixed already (I think), and
they simply need to upgrade.

Best regards,
   Wouter