Hi,
Today for me the .nl top level domain stopped to be valid.
I restarted unbound and the problem went away.
This is an installation of Unbound 1.4.7
A dump of the cache was made and I increased verbosity to 5 to log some information.
Any developer interrested in seeing the logs or the dump ?
Mail me directly if you do.
Have a nice day,
Leen.
a message of 20 lines which said:
Today for me the .nl top level domain stopped to be valid.
.nl added a new ZSK, 20331, around 2000 UTC. Could it be related?
Maybe, the error was:
verify rrset <sidn.nl. DS IN>
DS rrset in DS response did not verify
validator operate: query <www.sidn.nl. A IN>
Could not establish a chain of trust to keys for <sidn.nl. DNSKEY IN>
But I'm starting to think I should have logged some for .nl itself to be really useful.
I've seen similar outages. I experienced one too yesterday where my own
nohats.ca (but really almost all queries) failed to resolve. I ran a
verbosity 2 while the process was still running and it showed a massive
amount of ipv6 connection attempts (despite not having been on an ipv6
network in weeks)
A similar even seem to have happened on the Sunday of ICANN45 in Toronto,
where some important high up record stopped validating, causing everything
below it to fail.
Paul
>> a message of 20 lines which said:
>>
>>>Today for me the .nl top level domain stopped to be valid.
>>
>>.nl added a new ZSK, 20331, around 2000 UTC. Could it be related?
>>
>
>Maybe, the error was:
>
>verify rrset <sidn.nl. DS IN>
>DS rrset in DS response did not verify
>validator operate: query <www.sidn.nl. A IN>
>Could not establish a chain of trust to keys for <sidn.nl. DNSKEY IN>
>
>But I'm starting to think I should have logged some for .nl itself to be really useful.I've seen similar outages. I experienced one too yesterday where my own
nohats.ca (but really almost all queries) failed to resolve. I ran a
verbosity 2 while the process was still running and it showed a massive
amount of ipv6 connection attempts (despite not having been on an ipv6
network in weeks)A similar even seem to have happened on the Sunday of ICANN45 in Toronto,
where some important high up record stopped validating, causing everything
below it to fail.
I assume this is with a very recent version of ldns/Unbound ?
Don't think I had any IPv6 issues, it seems to still want to query over IPv6 too if I've
read the logs correctly.
Just to let you know we are aware of this and investigating in.
Nothing to report further yet, though...
>>> verify rrset <sidn.nl. DS IN>
>>> DS rrset in DS response did not verify
>>> validator operate: query <www.sidn.nl. A IN>
>>> Could not establish a chain of trust to keys for <sidn.nl. DNSKEY IN>Just to let you know we are aware of this and investigating in.
Nothing to report further yet, though...
Marco,
As I mentioned before this was with an old version of Unbound, the bug is probably fixed already.
And if you want a log and a cache-dump mail me directly, I'll send it to you.
Quoting Leen Besselink (leen@consolejunkie.net):
> >>> verify rrset <sidn.nl. DS IN>
> >>> DS rrset in DS response did not verify
> >>> validator operate: query <www.sidn.nl. A IN>
> >>> Could not establish a chain of trust to keys for <sidn.nl. DNSKEY IN>
> Just to let you know we are aware of this and investigating in.
> Nothing to report further yet, though...
As I mentioned before this was with an old version of Unbound, the bug
is probably fixed already. And if you want a log and a cache-dump
mail me directly, I'll send it to you.
The issue with the .nl validation we've seen yesterday evening are not
related to Unbound or Unbound versions. People using different resolver
software also reported problems with the .nl zone.
SIDN is looking in to it and will probably release some formal
communication about it in due time. 
Regards,
-Sndr.
FWIW, ISC DNSDB shows that the DNSKEY RRset prior to insertion of the new ZSK was seen as late as 2012-10-28 19:40:50, but the RRSIG covering sidn.nl/DS made by the new ZSK was seen as soon as 2012-10-28 19:55:50, only 15 minutes later. Looks like perhaps the new ZSK wasn’t pre-published long enough. Since the TTL of the nl/DNSKEY RRset is two hours, it is very possible that validators were attempting to validate RRSIGs made by the new ZSK having only a version of the nl/DNSKEY RRset without the new ZSK in cache.
;; last seen: 2012-10-28 19:40:50 -0000
nl. IN DNSKEY 256 3 8 AwEAAcCIZ6GTKCwV5fpNXuvSr6eOPDo0NRrCFjjmerK1UphiWCpoV5oX bCydxv3wyOPAhIRNSUOzT/o8WegaNy93jM+arLHi/4oYpasXDDcBSIjZ j8LpYzAP7fbUrkw8kSjmr+IA/mawpuQ8m/XTtgn7AIzL1eN38/iMTp6K fPWa9dHZ
nl. IN DNSKEY 257 3 8 AwEAAbgqMqYHpmZrqQd3zFNOzYv2lw8bWBnrtK9TjlwK/ZBYMwKGR6TN bmMuwdjebpIE2vFxTHGLQfb2PmUJpazAGkG0fUaqrjuIU99Qbe5hwLYX qyGe2Mm+ZNRsomBxhluR/ky/XX4V1TjTqeXYH4gkzEs7I6og5IE0tKyh hpU38XHtuFVj7uunIAWGn5g9tZ0ZNnv8CkwLE5hLmRf+AoNTd483ZBX4 FUT32KbF6XV3ikctXbsMe2GqGlIf0gMqJQbNvYf1NuNMbxauh9YavEQ0 yaavI1hz5eLMJRruq4wDTyRnMJHupxY69oZZ9IbIsEf0FurtaA7fXrAx qcfEfARr4b0=
;; first seen: 2012-10-28 19:55:50 -0000
;; last seen: 2012-10-29 14:14:43 -0000
sidn.nl. IN RRSIG DS 8 2 7200 1352664247 1351444502 20331 nl. aP/JmxOzE3nzDj7fgKq+T6/j9f2c4DKTyAF9wKckSukeDSfbXqO0Iias ZIl6kAn/7m4aE4nIoOsZr45GsiTmY49rquR7LNlcuxCv37SqFvwCTKsM 8ARyHfOXG+oG+DdbO2uYpIYDlJBN2gpBkFkgcepUZ3aiuXnnXN8OuBbI rdY=
Regards,
Casey
[ Quoting <casey@deccio.net> in "Re: [Unbound-users] DNSSEC validati..." ]
FWIW, ISC DNSDB shows that the DNSKEY RRset *prior* to insertion of the new ZSK
was seen as late as 2012-10-28 19:40:50, but the RRSIG covering sidn.nl/DS made
by the new ZSK was seen as soon as 2012-10-28 19:55:50, only 15 minutes later.
Looks like perhaps the new ZSK wasn't pre-published long enough. Since the TTL
of the nl/DNSKEY RRset is two hours, it is very possible that validators were
attempting to validate RRSIGs made by the new ZSK having only a version of the
nl/DNSKEY RRset without the new ZSK in cache.;; last seen: 2012-10-28 19:40:50 -0000
nl. IN DNSKEY 256 3 8 AwEAAcCIZ6GTKCwV5fpNXuvSr6eOPDo0NRrCFjjmerK1UphiWCpoV5oX
bCydxv3wyOPAhIRNSUOzT/o8WegaNy93jM+arLHi/4oYpasXDDcBSIjZ
j8LpYzAP7fbUrkw8kSjmr+IA/mawpuQ8m/XTtgn7AIzL1eN38/iMTp6K fPWa9dHZ
nl. IN DNSKEY 257 3 8 AwEAAbgqMqYHpmZrqQd3zFNOzYv2lw8bWBnrtK9TjlwK/ZBYMwKGR6TN
bmMuwdjebpIE2vFxTHGLQfb2PmUJpazAGkG0fUaqrjuIU99Qbe5hwLYX qyGe2Mm+ZNRsomBxhluR/
ky/XX4V1TjTqeXYH4gkzEs7I6og5IE0tKyh
hpU38XHtuFVj7uunIAWGn5g9tZ0ZNnv8CkwLE5hLmRf+AoNTd483ZBX4
FUT32KbF6XV3ikctXbsMe2GqGlIf0gMqJQbNvYf1NuNMbxauh9YavEQ0
yaavI1hz5eLMJRruq4wDTyRnMJHupxY69oZZ9IbIsEf0FurtaA7fXrAx qcfEfARr4b0=;; first seen: 2012-10-28 19:55:50 -0000
;; last seen: 2012-10-29 14:14:43 -0000
sidn.nl. IN RRSIG DS 8 2 7200 1352664247 1351444502 20331 nl. aP/
JmxOzE3nzDj7fgKq+T6/j9f2c4DKTyAF9wKckSukeDSfbXqO0Iias ZIl6kAn/
7m4aE4nIoOsZr45GsiTmY49rquR7LNlcuxCv37SqFvwCTKsM
8ARyHfOXG+oG+DdbO2uYpIYDlJBN2gpBkFkgcepUZ3aiuXnnXN8OuBbI rdY=
That's cool info! Note that day light saving was activated (de-activated? I
never know) the evening before...
Regards,
Hi Miek,
Hi,
Looks like perhaps the new ZSK wasn't pre-published long enough.
As promised a brief (informal) follow-up on what happened.
Indeed the new ZSK wasn't pre-published long enough. After OpenDNSSEC
generated it and just prior to publishing it in the DNS, the software
encountered a problem. As a result of that, the zonefile was never
published. In fact, we missed two zonefileupdates before we got all
the right people mobilised and where ready to restart the process.
When we published the new zonefile, OpenDNSSEC figured that the
pre-publication time was long enough and started to include new
RRSIg's, made by the new ZSK. This resulted in validation errors.
So, the observation of Casey was just right.
We will maintain to look into this issue further and we will implement
protective measures to prevent this from happening again.
Regards,
- --
Marco