Why are unbound-control local_zone_remove/local_zone/local_data so incredibly slow ?

Unbound
1

Hello list,

I'm running Unbound 1.5.9 on OpenBSD 6.

I have the need to periodically add certain local data to unbound as
part of a dynamic security block list (i.e. redirect to null). These
lists can be 30k+ items in length.

Originally, I had this scripted as follows :
   - Get and parse data into unbound config format
   - Reload unbound

Obviously the problem with that approach is that reloading unbound
causes cache to be flushed .... which is not cool. However on the
plus side the script runs super-quick as its just a config reload !

So I looked at my options and came up with a second script :
   - Get and parse data into two groups : remove and add
   - On "remove" set run unbound-control local_zone_remove
   - On "add" set run unbound-control local_zone and unbound-control local_data

The problem is its taking forever to process that because calls to
unbound-control are so slow ! e.g. its been well over an hour now and
it *still* hasn't finished processing 30k items. The server is not
slow, the server is not heavily loaded, and my script is simple ....
the problem is very much the slowness of the calls to unbound-control.

Help !!

Thanks

Tim

2

Hi,

Unbound-control sets up a TLS connection for every command. Setting up
30k TLS connections one after another does take some time.

If your unbound daemon and unbound-control are on the same machine you
could try to communicate over a secured local socket and disable
control-use-cert. This might speed it up a little.

I just added some code to Unbound to bulk add and remove local-zones and
local-data, by reading input from stdin. This gives the possibility to
do the update over a single connection using local_zones,
local_zones_remove, local_datas and local_datas_remove. This code will
be in the next Unbound release.

Regards,
-- Ralph

3

Hi Ralph,

Thanks for your reply.

Interesting idea to try a local socket, will look into that some more.

Good to hear you added bulk add/remove ... that was on my wish list !

On 30 November 2016 at 13:36, Ralph Dolmans via Unbound-users

4

Hi Ralph,

Hmm... bad news ...have just looked into it and OpenBSD default config is :
remote-control:
        control-enable: yes
        control-use-cert: no
        control-interface: /var/run/unbound.sock

So guess I'll have to live with it until the next version.....

Thanks again.

5

Or you go back to the reload and do a dump_cache before and a
load_cache after that...
Yeah, I know, it's not entirely clean (after all, there might have been
redirects loaded which are also present in the cache. Should be alright
after the cache data times out, though.)

regards, j.

6

Interesting idea, nice bit of thinking outside the box ! But in
relation to "should be alright after cache data times out" , does
that mean that the idea might not work too well if I've got "prefetch:
yes" in my config ?

7

In the situation where a cached resolution has been prefetched before
the dump, and that site is supposed to be blacklisted/nullrouted, you'd
add >=90% of the ttl (up to cache-max-ttl) to the time until it blacks
out for good ...

... *unless* unbound either checks local data before the cache, or the
load_cache checks against local data and doesn't taint the cache with
the entries loaded, I'd assume. But essentially, that's beyond my
knowledge of the execution flow... j.

8

Local data is handled before the cache lookup. So the cache entries will
be ignored as soon as there is matching local data.

Regards,
-- Ralph

9

Fab ! Thanks !