Resolve failures when using forwarders that do recursion

Unbound
1

Hi list,
consider the following configuration:
1. Unbound is set up to use 2-3 DNS servers as forwarders
2. These servers are doing recursive resolving
3. Most of resolve requests are for several domain names,
  but there are certainly requests for other domains too.

Now look at this tcpdump log:

15:59:41.599014 IP unbound.host.13453 > forwarder-1.com.domain: 48359+% [1au] MX? some.domain. (41)
15:59:42.246023 IP unbound.host.29253 > forwarder-2.com.domain: 35322+% [1au] MX? some.domain. (41)
15:59:42.277969 IP forwarder-1.com.domain > unbound.host.13453: 48359 2/0/2 MX mx1.some.domain. 5, MX mx2.some.domain. 10 (102)
15:59:42.278009 IP unbound.host > forwarder-1.com: ICMP unbound.host udp port 13453 unreachable, length 36
15:59:43.254411 IP unbound.host.20082 > forwarder-3.com.domain: 18647+% [1au] MX? some.domain. (41)
15:59:43.575827 IP forwarder-2.com.domain > unbound.host.29253: 35322 2/0/2 MX mx2.some.domain. 10, MX mx1.some.domain. 5 (102)
15:59:43.575933 IP unbound.host > forwarder-2.com: ICMP unbound.host udp port 29253 unreachable, length 36
15:59:43.943166 IP forwarder-3.com.domain > unbound.host.20082: 18647 2/0/2 MX mx1.some.domain. 5, MX mx2.some.domain. 10 (102)
15:59:43.943304 IP unbound.host > forwarder-3.com: ICMP unbound.host udp port 20082 unreachable, length 36

We see that Unbound tries to resolve "some.domain" using all three upstream forwarders.
Every upstream server has to do a recursive resolving because "some.domain" is not in cache.

Now let's reorder the dump so that it is grouped by upstream server:

15:59:41.599014 IP unbound.host.13453 > forwarder-1.com.domain: 48359+% [1au] MX? some.domain. (41)
15:59:42.277969 IP forwarder-1.com.domain > unbound.host.13453: 48359 2/0/2 MX mx1.some.domain. 5, MX mx2.some.domain. 10 (102)
15:59:42.278009 IP unbound.host > forwarder-1.com: ICMP unbound.host udp port 13453 unreachable, length 36

So the answer came in >600ms, unbound has closed its socket -> system answers with ICMP unreach

15:59:42.246023 IP unbound.host.29253 > forwarder-2.com.domain: 35322+% [1au] MX? some.domain. (41)
15:59:43.575827 IP forwarder-2.com.domain > unbound.host.29253: 35322 2/0/2 MX mx2.some.domain. 10, MX mx1.some.domain. 5 (102)
15:59:43.575933 IP unbound.host > forwarder-2.com: ICMP unbound.host udp port 29253 unreachable, length 36

Here answer came in >1s, the same reaction with ICMP unreach

15:59:43.254411 IP unbound.host.20082 > forwarder-3.com.domain: 18647+% [1au] MX? some.domain. (41)
15:59:43.943166 IP forwarder-3.com.domain > unbound.host.20082: 18647 2/0/2 MX mx1.some.domain. 5, MX mx2.some.domain. 10 (102)
15:59:43.943304 IP unbound.host > forwarder-3.com: ICMP unbound.host udp port 20082 unreachable, length 36
Upstream answered in >700ms, => ICMP unreach.

So, each of upstream servers has done hard job with recursive resolving,
Unbound hasn't accepted any of the answers, returned SERVFAIL to the mail server,
mail server hasn't sent a mail, the sender is in disaster.

Unbound uses an algorithm described at [1] to set timeouts when
sending queries. This works well when Unbound is used as a recursive resolver
because Internet is a complex wild network full of crappy overloaded DNS servers
and one has to take changing conditions and failing servers in account.

But when Unbound uses forwarders that in turn should deal with that
wild Internet outside, it doesn't forgive its forwarders when they
deliver the answer a bit late. It thinks that those servers should be
FAST just because most answers come from their cache and Unbound uses its
infra-cache to remember this.

If we turn the infra-cache off, Unbound will use its standard 376ms timeout and
the situation may get even worse.

Does it maybe make sense to add a new configuration parameter that allows to set
a custom timeout value when using forwarders? In the case of that poor unbound.host
we would set that timeout to be ~ 1500ms or something like that.
It may be per-server or global value, and should be used only for requests
to "upstream" servers.

[1] http://www.unbound.net/documentation/info_timeout.html

2

Zitat von Ilya Bakulin <Ilya_Bakulin@genua.de>:

Hi list,
consider the following configuration:
1. Unbound is set up to use 2-3 DNS servers as forwarders
2. These servers are doing recursive resolving
3. Most of resolve requests are for several domain names,
  but there are certainly requests for other domains too.

Now look at this tcpdump log:

15:59:41.599014 IP unbound.host.13453 > forwarder-1.com.domain: 48359+% [1au] MX? some.domain. (41)
15:59:42.246023 IP unbound.host.29253 > forwarder-2.com.domain: 35322+% [1au] MX? some.domain. (41)
15:59:42.277969 IP forwarder-1.com.domain > unbound.host.13453: 48359 2/0/2 MX mx1.some.domain. 5, MX mx2.some.domain. 10 (102)
15:59:42.278009 IP unbound.host > forwarder-1.com: ICMP unbound.host udp port 13453 unreachable, length 36
15:59:43.254411 IP unbound.host.20082 > forwarder-3.com.domain: 18647+% [1au] MX? some.domain. (41)
15:59:43.575827 IP forwarder-2.com.domain > unbound.host.29253: 35322 2/0/2 MX mx2.some.domain. 10, MX mx1.some.domain. 5 (102)
15:59:43.575933 IP unbound.host > forwarder-2.com: ICMP unbound.host udp port 29253 unreachable, length 36
15:59:43.943166 IP forwarder-3.com.domain > unbound.host.20082: 18647 2/0/2 MX mx1.some.domain. 5, MX mx2.some.domain. 10 (102)
15:59:43.943304 IP unbound.host > forwarder-3.com: ICMP unbound.host udp port 20082 unreachable, length 36

We see that Unbound tries to resolve "some.domain" using all three upstream forwarders.
Every upstream server has to do a recursive resolving because "some.domain" is not in cache.

Now let's reorder the dump so that it is grouped by upstream server:

15:59:41.599014 IP unbound.host.13453 > forwarder-1.com.domain: 48359+% [1au] MX? some.domain. (41)
15:59:42.277969 IP forwarder-1.com.domain > unbound.host.13453: 48359 2/0/2 MX mx1.some.domain. 5, MX mx2.some.domain. 10 (102)
15:59:42.278009 IP unbound.host > forwarder-1.com: ICMP unbound.host udp port 13453 unreachable, length 36

So the answer came in >600ms, unbound has closed its socket -> system answers with ICMP unreach

15:59:42.246023 IP unbound.host.29253 > forwarder-2.com.domain: 35322+% [1au] MX? some.domain. (41)
15:59:43.575827 IP forwarder-2.com.domain > unbound.host.29253: 35322 2/0/2 MX mx2.some.domain. 10, MX mx1.some.domain. 5 (102)
15:59:43.575933 IP unbound.host > forwarder-2.com: ICMP unbound.host udp port 29253 unreachable, length 36

Here answer came in >1s, the same reaction with ICMP unreach

15:59:43.254411 IP unbound.host.20082 > forwarder-3.com.domain: 18647+% [1au] MX? some.domain. (41)
15:59:43.943166 IP forwarder-3.com.domain > unbound.host.20082: 18647 2/0/2 MX mx1.some.domain. 5, MX mx2.some.domain. 10 (102)
15:59:43.943304 IP unbound.host > forwarder-3.com: ICMP unbound.host udp port 20082 unreachable, length 36
Upstream answered in >700ms, => ICMP unreach.

So, each of upstream servers has done hard job with recursive resolving,
Unbound hasn't accepted any of the answers, returned SERVFAIL to the mail server,
mail server hasn't sent a mail, the sender is in disaster.

Unbound uses an algorithm described at [1] to set timeouts when
sending queries. This works well when Unbound is used as a recursive resolver
because Internet is a complex wild network full of crappy overloaded DNS servers
and one has to take changing conditions and failing servers in account.

But when Unbound uses forwarders that in turn should deal with that
wild Internet outside, it doesn't forgive its forwarders when they
deliver the answer a bit late. It thinks that those servers should be
FAST just because most answers come from their cache and Unbound uses its
infra-cache to remember this.

If we turn the infra-cache off, Unbound will use its standard 376ms timeout and
the situation may get even worse.

Does it maybe make sense to add a new configuration parameter that allows to set
a custom timeout value when using forwarders? In the case of that poor unbound.host
we would set that timeout to be ~ 1500ms or something like that.
It may be per-server or global value, and should be used only for requests
to "upstream" servers.

[1] http://www.unbound.net/documentation/info_timeout.html

--
Ilya

Hello

If you have Unbound > 1.4.14 you can try

tcp-upstream: <yes or no>
               Enable or disable whether the upstream queries use TCP only for
               transport. Default is no. Useful in tunneling scenarios.

Not sure if the (short) timeout also apply to tcp and if the tcp connection is used for multiple outstanding queries...

Regards

Andreas

3

Hi,

I have been using unbound for a couple of months now and I have also trouble
with the timeouts. I tried the workarounds discussed on the mailinglist, but I
haven't found a proper solution yet.

Actually Andreas' idea sounds good and I tried it, but I was surprised about
the high load produced by unbound:
On my dns server at home tcp-upstream = yes works pretty good because I have a
moderate number of dns request.
On the company firewall the system load increased by 1. That seems to be to
much.

In any case I think disabling udp should only be a workaround. Unbound should
be usable as dns forwarder with udp.

In my opinion the idea of Ilya sounds good. If we had a config option to set
a lower limit for timeouts, it would be easily possible to use unbound with
forwarders and udp.

Regards

Florian

4

Hi,

Please have a look to the attached patch.
It adds a new config option 'infra-cache-min-rtt' which makes the former
constant value of RTT_MIN_TIMEOUT adjustable. This gives the user the
opportunity to choose a reasonable retransmit timeout value.

I have tested different scenarios with forwarders inside my local network and
with google dns forwarders. Values about 1000 ms seems to be fine.
Inside lan 1500 ms works a little bit better. I haven't seen timeouts anymore
with this value.
If using google dns it could be neccessary to decrease the value to 750 ms
to compensate packet loss.

I would be glad if you could commit my patch.

Thanks.

Florian

Index: doc/unbound.conf.5.in

5

Hi,

Please have a look to the attached patch.
It adds a new config option 'infra-cache-min-rtt' which makes the former
constant value of RTT_MIN_TIMEOUT adjustable. This gives the user the
opportunity to choose a reasonable retransmit timeout value.

Hi Wouter,

I'm still thinking about the problem with the infra cache timeouts with
forwarders. I would like to ask you about your opinion of a 'right'
solution for the problem.
I guess adding a config option (see my patch) is kinda hack, but I don't see
any other solution at the moment.

Actually I was thinking about this idea:
After a timeout unbound could reuse port and query id in the second query.
Then we could accept the first reply still after the second query was sent.
Reuse port and query id will avoid security problems with the kaminsky attack.
But this solution works only if the second query gets send to the same server
as the first. In most cases people use >1 global forwarders, so it won't work.
So I guess it's to much work to implement this behavior if it doesn't fix the
problem in all cases.

Have you any other suggestions how we could fix this problem?
Have you any considerations about my patch with the infra-cache-min-rtt option?

Thanks.

Florian

6

Hi Florian,

Hi,

Please have a look to the attached patch. It adds a new config
option 'infra-cache-min-rtt' which makes the former constant
value of RTT_MIN_TIMEOUT adjustable. This gives the user the
opportunity to choose a reasonable retransmit timeout value.

Hi Wouter,

I'm still thinking about the problem with the infra cache timeouts
with forwarders. I would like to ask you about your opinion of a
'right' solution for the problem. I guess adding a config option
(see my patch) is kinda hack, but I don't see any other solution at
the moment.

Actually I was thinking about this idea: After a timeout unbound
could reuse port and query id in the second query. Then we could
accept the first reply still after the second query was sent. Reuse
port and query id will avoid security problems with the kaminsky
attack. But this solution works only if the second query gets send
to the same server as the first. In most cases people use >1 global
forwarders, so it won't work. So I guess it's to much work to
implement this behavior if it doesn't fix the problem in all
cases.

Have you any other suggestions how we could fix this problem? Have
you any considerations about my patch with the infra-cache-min-rtt
option?

So, the same fix as the min-rtt option, but then conditional on the
recursiveness of the target. So, if unbound sends a packet to a
destination that is recursive, it uses the timeout of 1000 msec for
it. This gives the recursive destionation the time to perform the
recursion before a retry.

However this conflicts with the desire for unbound to poll a second
recursive server, just to see if this query is in cache for that
server. And come back to the first one later (on a later reprobe),
(this is the current behaviour).

Best regards,
   Wouter

7

Hi Florian,

I have implemented a completely different option, does that meet your
needs? It is called delay-close: msec. If you set eg. delay-close:
1500, then when a UDP socket timeouts that port is kept open for 1500
msec afterwards. Meanwhile unbound continues (but a socket is still
in use) as normal.

Only the right ID, IPaddr is accepted on that port; bad packets are
added to the unwanted_replies counter. The right ID,IP also closes
the port.

This keeps ports open for a little while longer, without impacting the
rest of unbound.

Do you like this option, or do you (also-) want me to accept your patch?

Best regards,
   Wouter

8

Zitat von "W.C.A. Wijngaards" <wouter@nlnetlabs.nl>:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Florian,

I have implemented a completely different option, does that meet your
needs? It is called delay-close: msec. If you set eg. delay-close:
1500, then when a UDP socket timeouts that port is kept open for 1500
msec afterwards. Meanwhile unbound continues (but a socket is still
in use) as normal.

Only the right ID, IPaddr is accepted on that port; bad packets are
added to the unwanted_replies counter. The right ID,IP also closes
the port.

This keeps ports open for a little while longer, without impacting the
rest of unbound.

Do you like this option, or do you (also-) want me to accept your patch?

Best regards,
   Wouter

Hello,

will this be available in Unbound 1.4.22?? It also might solve our problem with cascaded Unbound and slow host resolving like the esta.cbp.dhs.gov.

Regards

Andreas

9

Hi Andreas,

10

Hi Wouter,

sorry for the late answer. I have missed your reply in my pile of mails.

I like the new delay-close option very much. After a first test it solves
an other problem I had also. It was described by Ilya Bakulin a couple of
months ago. We have seen a bunch of ICMP-Port-Unreachable packets in some
cases and we were using a workaround in our firewall to block them. Your
solution is much better. Thanks for it!

I think (not tested yet) the delay-close option can not solve the problem
described in this thread. My problem here were not the ICMP packets. The problem
is that unbound can't resolve if usually fast answering forwarders
need a lot of time to do recursive resolution for a specific domain.
With delay-close I can suppress the ICMP Packets to the forwarders but I still
can't resolve the name. In that case the answers get dropped too early
because of a small retransmit timeout.

I'm running unbound with my patch for a couple of months now and it seems
to work pretty good. I'm not sure how useful the patch is for the other users.
Maybe a higher minimum retransmit timeout only for forwarder would be an
option too and no special configuration would be required. On the other hand,
with my patch everybody can fix such problems without recompiling unbound.

Best regards,

Florian