Problems resolving akamai hosted domains with unbound?

I'm experiencing problems looking up "images-na.ssl-images-amazon.com"
(an akamai hosted domain) using unbound -- could someone take a look at
these traces?

I attach here four files --

  amazon-dnscache.pcap: the packets generated by a successful lookup
  using dnscache

  amazon-dnscache.txt: the dig output generated by a successful lookup
  using dnscache

  amazon-unbound-host.pcap: the packets generated by an unsuccessful
  (SERVFAIL) lookup using unbound-host

  amazon-unbound-host.txt: the output of `unbound-host -v -d -d -t a
  images-na.ssl-images-amazon.com`

any ideas? are one or more of akamai, dnscache, or unbound at fault?

(attachments)

amazon-dnscache.pcap (13.4 KB)
amazon-dnscache.txt (746 Bytes)
amazon-unbound-host.pcap (23.9 KB)
amazon-unbound-host.txt (74.7 KB)

Hi Robert,

An initial look, it seems there are no replies from akamai.
Looking at the pcap, I can see that akamai responds with weird answers,
that have a CNAME twice in there.

Answer section:
images-na.ssl-images-amazon.com. 300 IN CNAME
images-na.ssl-images-amazon.com.edgekey.net.
images-na.ssl-images-amazon.com. 300 IN CNAME
images-na.ssl-images-amazon.com.edgekey.net.

Currently, those akamai servers are responding with only one CNAME for me.

Unbound is dropping the return messages, as integrity checks are
failing. Only one CNAME record is allowed at a name (otherwise, which
one do you follow?).

It drops all messages that try to have multiple CNAMEs for one name. The
reason for the check is to protect the internal logic of the recursor.

If the problem persists for you, I could try to 'fix it up' by seeing if
the CNAMEs are identical, and then merging them.

Thank you very much for both the unbound-host output (although even more
- -d's would have been nice; then it prints packets) and the pcap.

Best regards,
~ Wouter

Robert Edmonds wrote:

W.C.A. Wijngaards wrote:

If the problem persists for you, I could try to 'fix it up' by seeing if
the CNAMEs are identical, and then merging them.

Thank you very much for both the unbound-host output (although even more
-d's would have been nice; then it prints packets) and the pcap.

In svn trunk for unbound r1109 is a fix for this issue. Extremely
lenient acceptance of CNAMEs, if a name has multiple CNAMEs for it, the
first is simply used and the rest is dropped.

Best regards,
~ Wouter

W.C.A. Wijngaards wrote:

W.C.A. Wijngaards wrote:
> If the problem persists for you, I could try to 'fix it up' by seeing if
> the CNAMEs are identical, and then merging them.
>
> Thank you very much for both the unbound-host output (although even more
> -d's would have been nice; then it prints packets) and the pcap.

In svn trunk for unbound r1109 is a fix for this issue. Extremely
lenient acceptance of CNAMEs, if a name has multiple CNAMEs for it, the
first is simply used and the rest is dropped.

thanks! of course, now akamai is sending me responses with only one
CNAME...

thanks! of course, now akamai is sending me responses with only one
    CNAME...

I noticed that there were on the nanog list reports of problems
with amazon.com. That might have been the real cause of the problems
you saw.

  jaap

Hi Wouter,

In svn trunk for unbound r1109 is a fix for this issue. Extremely
lenient acceptance of CNAMEs, if a name has multiple CNAMEs for it, the
first is simply used and the rest is dropped.

while as a customer and user I'd appreciate the quick and pragmatic
solution to the operational problem, there remain doubts that in this
case circumnavigating the bug was the best long term approach. Of
course, as a follower, the new kid on the block has a hard time making
friends by pointing the others to the rule book instead of playing
their way. That said, let's look into the scripture (which also means
this may better be discussed on namedroppers):

RFC 2181, section 10.1 explains that there must not be multiple CNAMEs
for any single owner (it doesn't use RFC 2119 language, but this is
basicall it). In section 5, introducing RRSets, there is a strong
recommendation against identical RRs in an RRSet. This is later clarified
in RFC 4034, section 6.3:

   [RFC2181] specifies that an RRset is not allowed to contain duplicate
   records (multiple RRs with the same owner name, class, type, and
   RDATA). Therefore, if an implementation detects duplicate RRs when
   putting the RRset in canonical form, it MUST treat this as a protocol
   error. If the implementation chooses to handle this protocol error
   in the spirit of the robustness principle (being liberal in what it
   accepts), it MUST remove all but one of the duplicate RR(s) for the
   purposes of calculating the canonical form of the RRset.

This is for DNSSEC purposes only, but that would hit Unbound sooner or
later anyway. The paragraph is tough prose: "... MUST treat this as a protocol
error." And later (bending the poor old robustness principle once more):
"... MUST remove all but one of the duplicate RR(s) for the purposes ...".

All in all I'd suggest with multiple CNAME RRs in parallel, the targets
being equal or not, the response is "weird" and Unbound's original
unwillingness to accept it, wasn't that bad. Out of curiosity, the other
system mentioned in the packet trace gives unusual responses, as well:

dig +norec @81.52.250.132 images-na.ssl-images-amazon.com.edgekey.net.

will give you random samples of eight out of thirteen root NS RRs in the
authority section. Nothing to worry about too much, but another
indication that this whole setup is "special".

Best regards,
   Peter

Er, Peter, those responses are normal. It is just the server chasing the CNAME and returning the closest set of NS records, which just happen to be the root hints. This behavior is a direct consequence of following the authoritative server algorithm in RFC 1034.

Hi David,

the CNAME and returning the closest set of NS records, which just
happen to be the root hints. This behavior is a direct consequence of
following the authoritative server algorithm in RFC 1034.

understood. There's been some discussion about out-of-zone CNAME chasing
on namedroppers already, but here I was unclear of the target of my note:
It was the 8 of 13 NS RRs I wanted to point out as "special". Of course,
teh recipient isn't supposed to actually use this information other
than recognizing the incomplete name resolution.

-Peter