High Latency During Peak Periods on Unbound Recursive DNS

Hi Everyone

I am running a recursive anycast DNS for over 15m subscribers in a telecom/isp environment. I have 7 HP servers running Freebsd 13.2, Unbound DNS 1.17.1 & Quagga.

I have tuned the OS and applied Unbound DNS optimization recommendations for busy servers.
NLnet Labs Documentation - Unbound - Howto Optimise

additions to /etc/sysctl.conf

set to at least 25MB for 10GE hosts

kern.ipc.maxsockbuf=26214400

set autotuning maximum to at least 25MB too

net.inet.tcp.sendbuf_max=26214400
net.inet.tcp.recvbuf_max=26214400

enable send/recv autotuning

net.inet.tcp.sendbuf_auto=1
net.inet.tcp.recvbuf_auto=1

increase autotuning step size

net.inet.tcp.sendbuf_inc=16384

Servers performed very well (peak/off peak) with latency under 50ms until recently when I enabled dnssec validation. https://nlnetlabs.nl/documentation/unbound/howto-anchor/

Since then performance has been erratic with latency spiking to over 600ms during peak hours.

I have disabled dnnsec validation and restarted unbound service / server several times. But the performance is not as good as before.

Can anyone suggest any tuning/optimization parameter to implement?

Hardware specs

hw.machine: amd64
hw.model: Intel(R) Xeon(R) Gold 6334 CPU @ 3.60GHz
hw.ncpu: 32
hw.byteorder: 1234
hw.physmem: 549395005440
hw.usermem: 546043224064
hw.pagesize: 4096

Regards,
Isaac

Hello

Anyone faced this problem? High latency is usually between 7pm - 9:30PM. Cache hit rate is low within this period too.

Regards,
Isaac