Unbound swapping

Unbound
1

I'm trying to understand why unbound is swapping ~900MB:

root@dns:~# top -b -n 1 | grep "SWAP\|unbound"
  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
SWAP COMMAND
11991 unbound 20 0 9751036 8.189g 3824 S 81.2 69.9 3192:03
840364 unbound

root@dns:~# cat /etc/unbound/unbound.conf | grep '\-size'
        ratelimit-size: 16m
    rrset-cache-size: 1536m
    msg-cache-size: 768m

This configuration doesn't hurt the "2.5x" tip from documentation.

root@dns:~# free -m
              total used free shared buff/cache available
Mem: 12004 8814 1296 237 1893 1449
Swap: 3395 907 2488

root@dns:~# unbound-control stats_noreset | grep mem
mem.cache.rrset=110262793
mem.cache.message=92861937
mem.mod.iterator=16548
mem.mod.validator=5767649
mem.mod.respip=0
mem.mod.subnet=872423102

root@dns:~# unbound-control stats_noreset | grep time
time.now=1505172298.707575
time.up=265711.841724
time.elapsed=279.009376

root@dns:~# sysctl vm.swappiness
vm.swappiness = 10

Some clue?

Thank you.

Regards,

2

Hi Eduardo,

I have no real good idea. But looking at your numbers, I see that you
are running a network heavy application, unbound, and it uses about 10G
on 12G memory. The buff/cache is 2G. Adds up to 12G. And it is
swapping. Sounds reasonable, it is maxed out on memory, this is where
swap is supposed to make space, right?

These options change the buffer space allocated by unbound: so-rcvbuf,
so-sndbuf, maybe also so-reuseport, or more tcp connections (that use
buffer space) or more network interfaces, or simply a different kernel
version that uses (slightly) more memory or something along those lines.

What is the limit on the subnet cache size? What does the memory max
out on when the subnetmodule is not enabled?

Best regards, Wouter

3

Hi Eduardo,

Hi Wouter!
Thank you for the answer.

I have no real good idea. But looking at your numbers, I see that you
are running a network heavy application, unbound, and it uses about 10G
on 12G memory. The buff/cache is 2G. Adds up to 12G. And it is
swapping. Sounds reasonable, it is maxed out on memory, this is where
swap is supposed to make space, right?

Yes, our application is really heavy.

Indeed about swap, but when unbound is swapping, performance degrades a lot.
If possible, we expect to not swap at all.

These options change the buffer space allocated by unbound: so-rcvbuf,
so-sndbuf, maybe also so-reuseport, or more tcp connections (that use
buffer space) or more network interfaces, or simply a different kernel
version that uses (slightly) more memory or something along those lines.

root@dns:~# cat /etc/unbound/unbound.conf | grep 'so-'
    so-rcvbuf: 16m
    so-sndbuf: 16m
    so-reuseport: yes

root@dns:~# cat /etc/unbound/unbound.conf | grep 'tcp'
    do-tcp: yes
    incoming-num-tcp: 1024
    outgoing-num-tcp: 1024

root@dns:~# uname -a
Linux dns 4.9.26-040926-generic #201705031231 SMP Wed May 3 16:34:12
UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

What is the limit on the subnet cache size? What does the memory max
out on when the subnetmodule is not enabled?

It's a public DNS resolver project in Brasil (like Google DNS), so...
That's why we need to use subnetcache module.

# cat /etc/unbound/unbound.conf | grep 'subnet'
    module-config: "subnetcache validator iterator"
    send-client-subnet: 0.0.0.0/0
    send-client-subnet: 2000::/3
    client-subnet-always-forward: no
    max-client-subnet-ipv6: 48
    max-client-subnet-ipv4: 24

In subnetcache module manual says:

"The maximum size of the ECS cache is controlled by 'msg-cache-size' in
       the configuration file. On top of that, for each query only 100 differ-
       ent subnets are allowed to be stored for each address family. Exceeding
       that number, older entries will be purged from cache."

This section "100-different subnets ... and then purged", how to know
if it's doing exactly this?
My guessing there is a memory leak, because the large memory it uses.

Best regards,

4

Wouter,

Something I just noticed:

root@dns:/opt# tail -f unbound.log
[1505432173] unbound[53221:4] error: internal error: looping module stopped
[1505432173] unbound[53221:4] error: internal error: looping module stopped
[1505432173] unbound[53221:4] error: internal error: looping module stopped
[1505432252] unbound[53221:2] error: internal error: looping module stopped
[1505432252] unbound[53221:2] error: internal error: looping module stopped
[1505432252] unbound[53221:2] error: internal error: looping module stopped

Maybe it's a problem?

Regards,