Unbound not resolving after ISC.org attack

Unbound
1

Hi,

I am trying to find the cause of an issue we have been experiencing last Thursday. We are running multiple Unbound servers

In order to provide internet to our users. I would say we were under attack with a couple of IPs trying to request as many as possible records for “ANY? isc.org.”. The DNS were resolving until a certain period of time where it became just impossible to resolve anything. My first try was to restart the unbound service and it worked for a couple of second, then it failed again. Next step was to block these IPs with iptables, but still it wasn’t resolving, even after a second restart of the unbound service.

What resolved the issue was to route the trafic to a different NAT ip so the unbound servers were seen as a different public ip when going to internet. At that point I thought I could have been throttled or blacklisted by the roots servers. I wrote to them and they explain to me that they don’t have such a mean to limite the rate of queries or throttle any of our request.

So I am turning to you guys to ask some question what could be slowing me down or blocking me from my local unbound server to resolve any name? Is there any configuration I need to change?

How would you prevent these kind of attack in the future?

Unbound version: 1.4.18

OS: Debian Squeeze 6.0.6

Best regards.

Dominick Rivard,
Solutions Architect

image001

5275 Queen Mary
Montréal, Qc
H3W 1Y3
Tel: 514-385-4448 ext 126
Fax: 514-385-6660

Notice: This message is confidential and privileged. If you are not the addressee, please inform the sender by return e-mail immediately and delete this message and destroy all copies.

Avis : Ce message est confidentiel et protégé par le secret professionnel. Si vous n’êtes pas le destinataire, veuillez informer l’expéditeur par courrier électronique immédiatement et effacer ce message et en détruire toute copie.

2

Hi Dominick,

Likely, unbound itself is not the issue here. Unbound, after a quick
lookup of isc.org, answers from cache. And thus there is not much
traffic from unbound to upstream. The isc.org requests should not
really interfere with other unbound lookups (until the CPU is overloaded).

If the firewall has connection tracking, which, for UDP, and with an
attack, could run out of space and deny lookups. Then, unbound, when
it tries to make connections is denied connectivity to the internet
and starts failing? What I suggest here is that simply your network
kit has overloaded and thus unbound cannot make connections any
longer. This would also fit with a different IP (which may have
different treatment by your own firewall) that makes things work?

You can increase verbosity on unbound (level 2 may already log enough
for you to know it is trying to contact upstream servers). But that
is likely to simply tell you that unbound is indeed trying to send out
traffic, but that this traffic fails.

If somehow unbound is the problem here, please, let me know. We think
performance under high load is important.

Best regards,
   Wouter

3

Hi Wouter,

Thanks for your answer, I will be honest we haven't checked the firewall at
that moment to look if we were denying ourselves.
Another info I can tell is that the servers resources were in no point
overloaded, we balance the port 53 to many servers and they were all fine.

14:09:07 up 147 days, 3:47, 2 users, load average: 0.00, 0.01, 0.00 that
almost like this on each server.
That day the ram was used at about 60% and cpu at 20%.

How much space do you think I need to activate the verbosity 2 for let say a
week? I get between 15 and 20 million requests a day.

Best regards.
Dominick