error: outgoing tcp: bind: Address already in use

Nick_Urbanik · July 6, 2017, 8:13am

Dear Folks,

A DNS server running unbound 1.6.3 has these messages; any suggestions
on what is happening?

error: serviced_tcp_initiate: failed to send tcp query
error: outgoing tcp: bind: Address already in use

Wouter · July 6, 2017, 8:30am

Hi Nick,

The config number for outgoing tcp is likely too low. outgoing-num-tcp:
1000

Also, you may be running out of port numbers, perhaps this causes the
'bind a tcp socket returns errno Address already in use', because the
choice of port number was left to the kernel? Unless you force a
particular port number in the config, but that is unlikely. People
usually want to change the kernel handling of timewait with port reuse
when that happens.

Best regards, Wouter

Quirin_Scheitle · July 6, 2017, 8:32am

Hi Nick,

IIRC, unbound does a “bind before connect” on sockets, which requires a local port to be completely unused [1] to be used by bind.

Unbound 1.6.4 should fix this by setting SO_REUSEADDR on outgoing ports - please give it a try.

Kind regards
Quirin

[1] https://idea.popcount.org/2014-04-03-bind-before-connect/

Nick_Urbanik · July 12, 2017, 4:37am

Dear Wouter,

Hi Nick,

The config number for outgoing tcp is likely too low. outgoing-num-tcp:
1000

We had it set to 3000. The machine is a resolver, with four machines
forwarding to it. It has normal traffic of about 8,000 queries per
second. The caches have up to 45,000 queries per second.

Also, you may be running out of port numbers, perhaps this causes the
'bind a tcp socket returns errno Address already in use', because the
choice of port number was left to the kernel? Unless you force a
particular port number in the config, but that is unlikely.

Please can you suggest how I might do that?

People usually want to change the kernel handling of timewait with
port reuse when that happens.

So that is through a sysctl setting, I expect.

Best regards, Wouter

Dear Folks,

A DNS server running unbound 1.6.3 has these messages; any suggestions
on what is happening?
error: serviced_tcp_initiate: failed to send tcp query
error: outgoing tcp: bind: Address already in use

Here is the configuration, which is complete, except for the
access-control statements, which are included from
/etc/unbound/local.d/*.conf. No other statements are included.

Can you suggest any possible changes to better support TCP queries?

include: /etc/unbound/conf.d/*.conf
remote-control:
    control-cert-file: /etc/unbound/unbound_control.pem
    control-enable: yes
    control-interface: 127.0.0.1
    control-key-file: /etc/unbound/unbound_control.key
    server-cert-file: /etc/unbound/unbound_server.pem
    server-key-file: /etc/unbound/unbound_server.key
server:
    auto-trust-anchor-file: /var/lib/unbound/root.key
    chroot: ""
    directory: /etc/unbound
    dlv-anchor-file: /etc/unbound/dlv.isc.org.key
    do-ip6: no
    extended-statistics: yes
    harden-below-nxdomain: yes
    harden-glue: yes
    harden-referral-path: yes
    hide-identity: yes
    hide-version: yes
    include: /etc/unbound/local.d/*.conf
    incoming-num-tcp: 3000
    infra-cache-numhosts: 40000
    infra-cache-slabs: 16
    interface: 127.0.0.1
    interface: 130.232.53.173
    interface: 20.81.34.106
    interface: 24.219.107.46
    interface-automatic: no
    key-cache-size: 128m
    key-cache-slabs: 16
    log-time-ascii: yes
    logfile: /var/log/unbound/unbound.log
    max-udp-size: 3072
    minimal-responses: yes
    msg-cache-size: 800m
    msg-cache-slabs: 16
    neg-cache-size: 128m
    num-queries-per-thread: 16384
    num-threads: 8
    outgoing-interface: 24.219.107.46
    outgoing-num-tcp: 3000
    outgoing-port-avoid: 0-2767
    outgoing-port-permit: 2768-65535
    outgoing-range: 49152
    pidfile: /var/run/unbound/unbound.pid
    prefetch: yes
    prefetch-key: yes
    ratelimit: 1000
    ratelimit-factor: 10
    ratelimit-for-domain: . 10000
    ratelimit-for-domain: com. 10000
    ratelimit-for-domain: dlv.isc.org. 10000
    ratelimit-for-domain: edu. 5000
    ratelimit-for-domain: gov. 5000
    ratelimit-for-domain: net. 5000
    ratelimit-for-domain: org. 5000
    ratelimit-size: 128m
    ratelimit-slabs: 8
    rrset-cache-size: 1600m
    rrset-cache-slabs: 16
    rrset-roundrobin: yes
    so-rcvbuf: 8m
    so-reuseport: yes
    so-sndbuf: 8m
    statistics-cumulative: yes
    trusted-keys-file: /etc/unbound/keys.d/*.key
    unwanted-reply-threshold: 10000000
    username: unbound

Nick_Urbanik · July 21, 2017, 3:29am

Dear Folks,

Pete_Wright · July 21, 2017, 4:32pm

not likely since this is a TCP error not UDP error. Are you sure there are no other processes binding to the port that your unbound wants to listen on? try running sockstat or netstat to see if any procs are already listening that you may need to kill.

-pete

Wouter · July 24, 2017, 10:57am

Hi Nick,

Nick_Urbanik · July 25, 2017, 6:43am

A DNS server running unbound 1.6.3 has these messages; any suggestions
on what is happening?

error: serviced_tcp_initiate: failed to send tcp query
error: outgoing tcp: bind: Address already in use

Could our setting of
max-udp-size: 3072
be too low? Any guidelines for tuning this?

I think unbound 1.6.4 port reuse option may solve it for you as Quirin
suggests.

I will use 1.6.4 as soon as permitted

The max-udp-size is not important for this error you are
seeing in logs.

But if the reply will not fit in 3072 bytes, then will unbound not
then need to use TCP? And if there are a very large number of
responses required, could that not cause the TCP ports to run out if
they cannot easily be reused?

This problem arises on our resolvers, which are separate from the
caches; the caches forward to the resolvers.