DNSSEC validating resolver on machines without RTC or wrong date

Unbound
1

Hi unbound users,

I maintain unbound on Fedora and RHEL. I met some question on some Fedora channel about problems with NTP service. It turned out the problem of that user lied were in DNSSEC validating resolver and wrong time on his machine. Like significantly wrong date, which made DNSSEC validation fail because some timestamp on RRSIG did not fail.

I would like to have DNSSEC validation on a machine good default choice. Or at least possible. I consider it a problem that this wrong date will *not* fix automatically in otherwise default configuration.

Like many other systems, Fedora tries to use chrony service to use NTP to synchronize and correct the time. Problem is unless the user has configured fixed IP or NTP servers were provided by DHCP, it needs to do DNS resolution. Fedora uses 2.fedora.pool.ntp.org. ntp.org is not signed, but org. has to pass validation. It will never success if the date is wrong and the cache is already up, therefore the system relies on it.

I think it is a technical problem there is dependency loop. DNSSEC needs at least roughly correct time in for unbound (or any validating resolver) to deliver IP for NTP server. It would fix the system time, fixing the validation failures. But until someone fixes the system time to pass root and org. domains validation, it cannot contact configured ntp service to obtain current time.

I would like to ask opinions how this should be fixed to autocorrect auto-magically. I am aware unbound is more usually used on servers, which should keep time synced on boot and are not powered off for extended time. But I think it is a good choice also for workstations.

* should initial NTP service request names with CD flag set, requesting any validation ignored for its queries? The problem I see those services use getaddrinfo() calls, which do not offer a way to set CD flags only for some request. It seems to me DNS-only resolution library should not be required in time synchronization service

* should system resolver always disable DNSSEC validation on early start and enable it only after initial time synchronization passed? I think it makes it possible by insecure domain . defined on start. Then when time is successfully obtained, it can switch validation on by "unbound-control insecure_remove ." command. I think it would be risky, because initial cache should be revalidated on such switch and bogus answers should be flushed. Not sure it is possible with current code.

* possible workaround would be time synchronization service having fixed IP address instead of name. It might save last used addresses on shutdown and use it in case initial name query failed. That might work on raspberry-pi like devices, but does not solve the first boot after the installation. That should be good compromise between fixed address.

I think even if we choose to use DNS over TLS without DNSSEC, the problem with initial wrong time stays on its certificate check. Is there a way to fix it in that case?

Do you have other ideas how should DNSSEC validation fixed on end workstations? How it should obtain time in secure way and not dropping validation entirely? Is there already a best practice for it? Should unbound support mode where it would check just signatures, but not timestamps?

Please share any comments you may have, thank you.

Best Regards,
Petr

2

A question can be a good question, without being the right question...

[...] I consider it a problem that this wrong date will *not* fix automatically in otherwise default configuration.

Like many other systems, Fedora tries to use chrony service to use NTP to synchronize and correct the time. Problem is unless the user has configured fixed IP or NTP servers were provided by DHCP, it needs to do DNS resolution.

This is where it starts to go off the rails for me. I mean: where? Someplace which is neither configured a fixed address or provisioned with DHCP... and yet is connected to the internet: where is that?

So since when is mDNS protected by DNSSEC? Is mDNS supposed to even require internet?

Fedora uses 2.fedora.pool.ntp.org. ntp.org is not signed, but org. has to pass validation.

Is there an internet connection? How does that work without a fixed IP or DHCP or mDNS?

[...]
I would like to ask opinions how this should be fixed to autocorrect auto-magically. I am aware unbound is more usually used on servers, which should keep time synced on boot and are not powered off for extended time. But I think it is a good choice also for workstations.

This has been an issue with TSIG for forever. If something is that broken, maybe somebody should wake up and pay attention: what if the whole datacenter has come adrift of its time moorings? (DAMHIK!)

I really can't picture what network you're envisioning, and if it's DR or "internet in a box" then that entails forethought.

Convince me that this is a DNS problem...

3

Petr Menšík via Unbound-users writes:

> Hi unbound users,
>
> I maintain unbound on Fedora and RHEL. I met some question on some
> Fedora channel about problems with NTP service. It turned out the
> problem of that user lied were in DNSSEC validating resolver and wrong
> time on his machine. Like significantly wrong date, which made DNSSEC
> validation fail because some timestamp on RRSIG did not fail.

I'm used to run ntpdate pb t boottim, before stratng ntpd or unbound.
That seems to avpid the problem most of the time unless your machine
is really out of wack.

  jaap

4

Fred Morris via Unbound-users writes:

> This has been an issue with TSIG for forever. If something is that broken,
> maybe somebody should wake up and pay attention: what if the whole
> datacenter has come adrift of its time moorings? (DAMHIK!)
>
> I really can't picture what network you're envisioning, and if it's DR or
> "internet in a box" then that entails forethought.
>
> Convince me that this is a DNS problem...

I won't because it ain't. For a more in dept discussion see "The
Impact of Time on DNS Security ", https://eprint.iacr.org/2019/788.pdf.

  jaap

5

This is where it starts to go off the rails for me. I mean: where?
Someplace which is neither configured a fixed address or provisioned
with DHCP... and yet is connected to the internet: where is that?

he means a fixed ip for the ntp server, not for the client.

-JimC

6

Hi,
couldn't this be added to /etc/hosts?

Ciao,
Tito

7

Hello Petr,

this scenario is also mentioned in RFC 8027 [1] with the same options to solve that:

- DNSSEC depend on correct time. If the local time is wrong the system startup will fail -> to be fixed by a human
- disable DNSSEC validation until the system hat a correct time -> it's convenient for the user but hard for you as implementer.

I personally prefer the first option.

Andreas

[1] https://www.rfc-editor.org/rfc/rfc8027.html#section-6

8

For a small, "IoT" device without real-time clock, the first option is far from ideal. Typically those devices don't have a user to watch them boot. For those devices, the solution is obvious, at boot a 'ntpdate'-like program should run with a stub resolver that allows disabling DNSSEC validation.

9

Externalities. I generally eschew interviews where somebody asks "tell me what happens when $something boots": I mean, is it Systems-On-Chips all the way down or not? What about that keyboard, should I start there? How about the UPS?

Epistemologically, what role does The DNS play in the boot process? I say little to none and I'd like to keep it that way. Same with The Internet writ large: I don't see that an internet connection should be necessary to boot. Not everybody agrees with me. On the other hand, you want to do things on your own network with DNS? What warm-blooded meat puppet doesn't? I'm all for it and I applaud your efforts.

I think things which don't even have a vague sense of what time it is shouldn't be connected to The Internet and using The DNS writ large unless they're purpose-built not to need that capability. That doesn't mean that they can't use ARP, DHCP, DNS, UDP or TCP, ICMP inside a nice padded playpen while they learn to gird their loins and tie their shoes.

When they've learned that, then hopefully as part of that process they've learned enough to ask for the proper address for DNS services and the gateway address. This seems like the proper "order of battle" to me.

Things which come out of the box with enough smarts (which will never be updated) to hack their way to The Internet are indistinguishable from rogue devices because they ARE rogue devices. (And that goes for that keyboard I was talking about.) I've got an ASUS wifi repeater on my home network which periodically goes on a rampage and tries random doors, UPnP, you name it. Always has. I consider it a free pentest. It could be prepwned; how would I know? Anyway, it will never see The Internet. Anybody who finds themselves on my network uninvited will have to deal with it eventually.

> This is where it starts to go off the rails for me. I mean: where?
> Someplace which is neither configured a fixed address or provisioned
> with DHCP... and yet is connected to the internet: where is that?

he means a fixed ip for the ntp server, not for the client.

Yes. He means a fixed IP or resource name for the NTP server, /on/ the client. Actually he means the network, too.

If I configure DHCP for my segment and I don't configure gateway, DNS or NTP: what is my intention?

If I configure a fixed address (for the device) and I don't configure gateway, DNS or NTP: what is my intention?

If I don't configure anything, what is my intention?

Should the vendor's intention be imposed (shouldn't the intent be well known)? Should any network interface come up at all? Should an intent to connect this to The Internet be respected or should it be denounced? Should the vendor be explaining how they're going to prevent anything running this from becoming e-waste and a liability in our lifetime?

I'm sorry to have to ask (in the sense that it diminishes us all), but please explain for all of us, tell us: exactly what happens when this boots?

10

file and use something like

ExecStartPre=unbound-control insecure_add ntp.pool.org

ExecStartPost=dig ntp.pool.org
ExecStartPost=unbound-control insecure_del ntp.pool.org
ExecStartPost=unbound-control flush_zone ntp.pool.org

Paul

11

Run Unbound in “val-override-date: -1” mode at very short term after boot, and once your machine gets good datetime[1], restart Unbound in normal mode.

In this mode, Unbound performs DNSSEC validation without RRSIG expiration check. The only risk you must take here is the possibility of accepting expired signatures.

[1] The next problem is to get datetime by secure method. Your company should run DNS server publishing datetime in signed zone like:
time.redhat.com. IN TXT “1687842121”

12

I ran into a similar issue running Unbound as my external resolver on a Pi 3, which does not have an RTC.

I solved it by configuring chrony with a static IP address for the initial sync:

server 2a00:d78:0:712:94:198:159:10 iburst

The pool servers are configured as follows:

server 2.nl.pool.ntp.org iburst prefer

That means the pool servers will be used, overwriting the static IP, as soon as resolving the addresses works.

I also added -s and -r to the options in /etc/sysconfig/chronyd for mitigating the absence of an RTC.

While that is not auto-magic, it works well for me. The Pi's IP addresses are all statically configured. It does not use DHCP, but provides it to the network.

-- Sandro

13

If you add this into /etc/hosts, then you could instead just use fixed address(es) in NTP service instead of a name. The use of DNS is good, because you can change it on server only and clients will notice that soon.

If you hardcode IP address or address for the name, then there is no reason to use the name anymore. A comment above IP addresses would be just as good.

14

Yes, something like that should work. But I think we lack some systemd target, which would announce the time is already synchronized. Then some single-run service could have

After=time-set.target
After=unbound.service

ExecStartPost=unbound-control insecure_del ntp.pool.org
ExecStartPost=unbound-control flush_zone ntp.pool.org

Without guessing by dig it is already reachable. But yes, something like that should work.

I think it may make sense to have unbound-control flush_validate command. After I remove insecure flag to ntp.pool.org, I could just request revalidation of that that name. If anything under it were signed, but did not pass validation before or now, just flush such records. In most cases the data are already good, just have to be validated and marked secure. It could avoid innecessary new query for the same thing.

15

"Pulling yourself up by your bootstraps" is never going to be pretty, although it can be entertaining (I'm picturing Jerry Lewis or Dick Van Dyke on the Carol Burnett show).

If you add this into /etc/hosts, then you could instead just use fixed address(es) in NTP service instead of a name. The use of DNS is good, because you can change it on server only and clients will notice that soon.

If you hardcode IP address or address for the name, then there is no reason to use the name anymore. A comment above IP addresses would be just as good.

There are clearly options. 8)

"FMvU" == Fred Morris via Unbound-users
<unbound-users@lists.nlnetlabs.nl> writes:

> This is where it starts to go off the rails for me. I mean: where?
> Someplace which is neither configured a fixed address or > provisioned
> with DHCP... and yet is connected to the internet: where is that?

he means a fixed ip for the ntp server, not for the client.

-JimC

Hi,
couldn't this be added to /etc/hosts?

DNSSEC requires accurate time (as does TSIG). Without going into the sprawling, messy details (they're everywhere!) it's because The DNS is a global resource.

DNS the protocol, operating locally in a controlled environment, arguably doesn't need DNSSEC at all. Today. Not yesterday. Today; and tomorrow. (Not sure about the day after that.)

Bootstrapping is a messy thing, and it often requires doing things at one stage which are countermanded / replaced / nullified at a later stage. Like that keyboard, or a disk, or network card, needing a driver loaded before the "real" boot.

In a containerized environment, /etc/hosts could indeed be edited in the image by the host OS prior to booting the image. OTOH it could certainly have an initial value which points to a local resource to start with. Lots of options here, some much more complicated or sophisticated (not interested in saying anything here is a "problem" thereby in need of a patented "solution").

I helped build a malware sandbox which ran malware which was most def interested in learning as much as possible about its operating environment. Needless to say, we were successful. We did it with adversarial payloads, and you (generic traditional rhetorical plural) can't do it when presented with an environment which is purpose built to help you succeed? I find it puzzling... at least, excepting misfeasance and malfeasance.

I still want to understand more about "what boot environment does this?" but this is not a DNS question. I totally get that a device could boot a real OS without having a real clock. Why can't someone propose a real environment as a reference model to center and pin this discussion?

16

Oh great, I were not aware there is something similar. I would like to avoid the need for a restart. dnsmasq has similar special knob, but I were not aware unbound has something like it too.

I have tested it even on older unbound, it is possible to manipulate this properly via unbound-control set_option “val-override-date: -1”

And indeed, if I set the clock to wrong value and flush org zone, it fails until I call above unbound-control command. It is a bit concerning in that case it still has ad bit, even though it were not full validated. And I expect no other indication to client hints not full validation were done. But yes, starting with val-override-date: -1 in configuration and setting “val-override-date: 0” with cache flush once time is set initially after boot would be the best solution I can find.

The cache flush could be avoided if it were possible to flush just records, which do not pass only time checks. Since we have already done crypto validation, we have already the most CPU intensive work done. Better to not waste it by full cache flush again.

17

I have a feeling you have something personal against me, but cannot remember we ever discussed. Your responses seem to me a bit over-reacted and I do not understand why. More below.

"Pulling yourself up by your bootstraps" is never going to be pretty, although it can be entertaining (I'm picturing Jerry Lewis or Dick Van Dyke on the Carol Burnett show).

I don't follow, do not care about actors name in TV shows anyway.

If you add this into /etc/hosts, then you could instead just use fixed address(es) in NTP service instead of a name. The use of DNS is good, because you can change it on server only and clients will notice that soon.

If you hardcode IP address or address for the name, then there is no reason to use the name anymore. A comment above IP addresses would be just as good.

There are clearly options. 8)

There always are.

"FMvU" == Fred Morris via Unbound-users
<unbound-users@lists.nlnetlabs.nl> writes:

> This is where it starts to go off the rails for me. I mean: where?
> Someplace which is neither configured a fixed address or > provisioned
> with DHCP... and yet is connected to the internet: where is that?

he means a fixed ip for the ntp server, not for the client.

-JimC

Hi,
couldn't this be added to /etc/hosts?

DNSSEC requires accurate time (as does TSIG). Without going into the sprawling, messy details (they're everywhere!) it's because The DNS is a global resource.

DNS the protocol, operating locally in a controlled environment, arguably doesn't need DNSSEC at all. Today. Not yesterday. Today; and tomorrow. (Not sure about the day after that.)

I do not agree with your result. I think DNSSEC can be useful even on end devices like laptops, maybe even phones. Already, today. But I admit preparing system to have DNSSec enabled by default has its challenges. Especially broken forwarders are still not rare enough. I think DANE can still be useful on common end devices. Therefore I am looking for way to make it possible. Do not want to enforce it anywhere, just possible.

Bootstrapping is a messy thing, and it often requires doing things at one stage which are countermanded / replaced / nullified at a later stage. Like that keyboard, or a disk, or network card, needing a driver loaded before the "real" boot.

In a containerized environment, /etc/hosts could indeed be edited in the image by the host OS prior to booting the image. OTOH it could certainly have an initial value which points to a local resource to start with. Lots of options here, some much more complicated or sophisticated (not interested in saying anything here is a "problem" thereby in need of a patented "solution").

I doubt containerized environment need to solve time setting, because the host is responsible to provide it. Makes sense it has done it well enough before starting any containers. Also the host should be doing dns cache, not every container, IMHO. It seems to me you are referring more to server world, where I am looking more at end user devices systems.

I helped build a malware sandbox which ran malware which was most def interested in learning as much as possible about its operating environment. Needless to say, we were successful. We did it with adversarial payloads, and you (generic traditional rhetorical plural) can't do it when presented with an environment which is purpose built to help you succeed? I find it puzzling... at least, excepting misfeasance and malfeasance.

I am afraid I do not follow here.

I still want to understand more about "what boot environment does this?" but this is not a DNS question. I totally get that a device could boot a real OS without having a real clock. Why can't someone propose a real environment as a reference model to center and pin this discussion?

--

Fred Morris

Take an example of Fedora distribution image prepared to run on Raspberry PI device. Let's say I would like to use that device as a ssh terminal and I would like to have SSHFP records validated (where possible). Instead of systemd-resolved I would like unbound as a system cache, but with booting race conditions solved from the vendor already. So there is just minimal steps to do on my side as an user. Ideally it would boot from live DVD alternative without me changing anything.

Similarly when I boot live DVD on a fresh bought laptop, where lets imagine DNSSEC validation is enabled by default. I want to boot into graphical interface without having to ever visit BIOS to set the date, I expect it can fix it itself. All I need to do is plug in the network cable.

Regards,
Petr

18

I have a feeling you have something personal against me, but cannot remember we ever discussed. Your responses seem to me a bit over-reacted and I do not understand why. More below.

Sorry that you feel that way. I feel that there are solutions already for the commonly encountered use cases (you can't always get what you want!), so wanted to make sure there wasn't some use case which wasn't accounted for. Granted, there may be a better solution out there for the common cases.

"Pulling yourself up by your bootstraps" is never going to be pretty,
[...]

There are clearly options. 8)

There always are.

[...]
Take an example of Fedora distribution image prepared to run on Raspberry PI device. Let's say I would like to use that device as a ssh terminal and I would like to have SSHFP records validated (where possible). Instead of systemd-resolved I would like unbound as a system cache, but with booting race conditions solved from the vendor already. So there is just minimal steps to do on my side as an user. Ideally it would boot from live DVD alternative without me changing anything.

Similarly when I boot live DVD on a fresh bought laptop, where lets imagine DNSSEC validation is enabled by default. I want to boot into graphical interface without having to ever visit BIOS to set the date, I expect it can fix it itself. All I need to do is plug in the network cable.

Thanks.