Can't Bind Socket Error

Hi there…

We just started migrating some of our customer base over to Unbound.

Server is VM instance (VmWare) with 8 Gig RAM, 4 vCPU’s running CentOS 7 (64 bit). Unbound version 1.4.20 via YUM package install.

We started noticing the following errors in the logs this evening:

Oct 8 19:48:51 dns2-pppoe unbound: [1110:0] error: can’t bind socket: Permission denied

Oct 8 19:49:24 dns2-pppoe unbound: [1110:0] error: can’t bind socket: Permission denied

Oct 8 19:50:46 dns2-pppoe unbound: [1110:0] error: can’t bind socket: Permission denied

Oct 8 19:56:40 dns2-pppoe unbound: [1110:0] error: can’t bind socket: Permission denied

Oct 8 19:59:24 dns2-pppoe unbound: [1110:0] error: can’t bind socket: Permission denied

When looking at stats, the number of lookups and cache hits for that specific instance keep incrementing. Just wondering if anyone has any ideas on what is causing these errors?

Thanks,

Paul

Hi Paul,

Hi there…

We just started migrating some of our customer base over to
Unbound.

Server is VM instance (VmWare) with 8 Gig RAM, 4 vCPU’s running
CentOS 7 (64 bit). Unbound version 1.4.20 via YUM package
install.

The socket bind this late is for queries to upstream authority
servers, it is trying to bind one of the outgoing-interface IP
addresses. If you configured it, one of those addresses is not
actually served by your system (or the VM). Another option is that
IPv6 is broken in the VM and binding the ipv6 fd to ::0 fails.

Best regards,
   Wouter

Thank you for the response. There is only one interface configured in the
CentOS and IPv6 is disabled (for now - it will be turned up shortly).

In the configuration of Unbound, I specified the IP address (there is only
one) - should I perhaps change the IP to 0.0.0.0? It doesn't appear that
the error in the log file is causing any serious issues ....

Cheers,
Paul

We just started migrating some of our customer base over to Unbound.

Server is VM instance (VmWare) with 8 Gig RAM, 4 vCPU's running CentOS
7 (64 bit). Unbound version 1.4.20 via YUM package install.

The socket bind this late is for queries to upstream authority servers, it
is trying to bind one of the outgoing-interface IP addresses. If you
configured it, one of those addresses is not actually served by your system
(or the VM). Another option is that
IPv6 is broken in the VM and binding the ipv6 fd to ::0 fails.

Best regards,
   Wouter

Hi Paul,

Did you configure the interface or the outgoing-interface? In any
case, 0.0.0.0 or your IP giving permission denied once in a while is
bad. Something is wrong with your system somehow, and it is
intermittent. Are there port numbers that are not allowed by system
policy that you have enabled in unbound.conf (by default it skips
<1024 ports) ?

Best regards,
   Wouter

Hi Wouter and thanks again for the prompt responses....

Here is the configuration to hopefully answer your questions. I read
through the performance documentation and made several adjustments so
perhaps one of those changes were done in error.

Much appreciated,

Paul

Hi Paul,

You have ipv6 enabled (#do-ipv6: yes at the default) and you have tcp
disabled. You should consider do-tcp: yes since the DNS occasionally
needs to use TCP, but that is not the cause of your socket errors.

Best regards,
   Wouter

Hi Wouter and thank you for spotting that. I completely missed that I had
Ipv6 still enabled in the config.

I have also enabled TCP as per your suggestion.

Cheers,
Paul

Zitat von Paul Stewart <paul@paulstewart.org>:

Hi Wouter and thanks again for the prompt responses....

Here is the configuration to hopefully answer your questions. I read
through the performance documentation and made several adjustments so
perhaps one of those changes were done in error.

Much appreciated,

Paul

#
# See unbound.conf(5) man page.
#
# this is a comment.

  # buffer size for UDP port 53 incoming (SO_RCVBUF socket option).
  # 0 is system default. Use 4m to catch query spikes for busy
servers.
  so-rcvbuf: 8m

  # buffer size for UDP port 53 outgoing (SO_SNDBUF socket option).
  # 0 is system default. Use 4m to handle spikes on very busy
servers.
  so-sndbuf: 8m

Not sure what happens if these are higher than the OS is willing to accept. Could this be logged as a can't bind error?

Regards

Andreas

Paul Stewart:

We just started migrating some of our customer base over to Unbound.

I assume you have an internal instance up and running
and have good reasons no not use default values.

Otherwise I suggest to start with an /empty/ unbound.conf and set only necessary values.
Unbound has good defaults...
Or check, you are using most of them until you really need to change defaults.

At first I would remove any interface / outgoing-interface setting on a host with only
one ip-address.

Andreas

Thanks Andreas....

We wanted to ensure that the system was "tuned" due to the amount of DNS
clients that would be using it. Roughly 8,000 endpoints are querying the
pair of Unbound instances at this point.

I wasn't sure how "robust" the defaults were out of the box as our
experience with other types of software have always been that the default
configurations are for much smaller usage than what we need.

Take care,
Paul

Paul Stewart:

I wasn't sure how "robust" the defaults were out of the box as our
experience with other types of software have always been that the default
configurations are for much smaller usage than what we need.

yes, it's difficult to say.
I started by simply adding memory ( could send my settings tomorrow ... )
on my cache for customers.

Just following up on this socket error - we are still seeing it including a
couple of new servers I turned up today. It does not appear to be
operationally impacting (at least anything obvious).

All of these servers are running CentOS7 64 bit under VmWare 5.5 - just
wondering if anyone else running CentOS7 ?

Thanks,
Paul

Are you still getting "error: can't bind socket: Permission denied
for..." errors in your logs ?
(and do you have selinux enabled ?)

I just started testing unbound on RHEL7 and noticed that selinux
policy denies unbound to use port 5546:
type=AVC msg=audit(1416495730.557:3528): avc: denied { name_bind } for
pid=15009 comm="unbound" src=5546 scontext=system_u:system_r:named_t:s0
tcontext=system_u:object_r:dhcpc_port_t:s0 tclass=udp_socket

(I think these are in dontaudit rules so to get these in logs you
have to: semodule --disable_dontaudit --build
(and to get silent dontaudit rules back: semodule --build))
(8.3.2. Possible Causes of Silent Denials | Security-Enhanced Linux | Red Hat Enterprise Linux | 6 | Red Hat Documentation)

So maybe this in config will help: outgoing-port-avoid: 5546

(BTW what's correct way to define multiple
outgoing-port-avoid (without using ranges):
outgoing-port-avoid: 5546
outgoing-port-avoid: 8853
or outgoing-port-avoid: 5546,8853 ?)

-Jarno

Hi Jarno,

Just following up on this socket error - we are still seeing it
including a couple of new servers I turned up today. It does not
appear to be operationally impacting (at least anything
obvious).

All of these servers are running CentOS7 64 bit under VmWare 5.5
- just wondering if anyone else running CentOS7 ?

Are you still getting "error: can't bind socket: Permission denied
for..." errors in your logs ? (and do you have selinux enabled ?)

I just started testing unbound on RHEL7 and noticed that selinux
policy denies unbound to use port 5546: type=AVC
msg=audit(1416495730.557:3528): avc: denied { name_bind } for
pid=15009 comm="unbound" src=5546
scontext=system_u:system_r:named_t:s0
tcontext=system_u:object_r:dhcpc_port_t:s0 tclass=udp_socket

(I think these are in dontaudit rules so to get these in logs you
have to: semodule --disable_dontaudit --build (and to get silent
dontaudit rules back: semodule --build))
(8.3.2. Possible Causes of Silent Denials | Security-Enhanced Linux | Red Hat Enterprise Linux | 6 | Red Hat Documentation)

So maybe this in config will help: outgoing-port-avoid: 5546

(BTW what's correct way to define multiple outgoing-port-avoid
(without using ranges):

outgoing-port-avoid: 5546 outgoing-port-avoid: 8853

This is correct. Multiple lines. (also for outgoing-port-permit)

The lines are processed in the order encountered, in case you mix
permit and avoid lines in a weird way. An easy way to write them is
to first permit some ranges and then after the permit lines, write
avoid lines with some specific cases you want to avoid.

Best regards,
   Wouter