FreeBSD + Unbound 1.0.2 error: too many file descriptors requested

Hello unbound-users!

I'm runing unbound-1.0.2 installed from ports, under FreeBSD 7.0
Unbound is installed without libevent. Yesterday I have done portupgade from
unbound-1.0.1 to unbound-1.0.2

Now I met the following problem - Unbound do not starts. I found the reasone -
it is because of value of num-threads set. Old Unbound 1.0.1 configuration
has been set to use num-threads: 8 , but Unbound 1.0.2 can be started only
with setting num-threads: to 3 or less.

When I try to start Unbound with num-threads: 4 or more, unbound says to me
that:

# /usr/local/etc/rc.d/unbound start
Starting unbound.
[1218219064] unbound[3358:0] error: too many file descriptors requested. The
builtinmini-event cannot handle more than 1024. Config for less fds or
compile with libevent
[1218219064] unbound[3358:0] fatal error: configuration needs too many file
descriptors

Hi Dmitriy,

Unbound 1.0.2 calculates the maximum number of open files. And checks
if that is going to work. So that it does not start OK, but later on,
when it is busy, fails. This error is because the select system call
has a hardcoded limit of 1024 (limited by the system).

Libevent does not work for you?

OK an alternative.
Compile unbound with ./configure --without-pthreads

Then, you can even set the outgoing-range to 256, or 512 or 768
File descriptors are less of a problem.

The downside is unbound will fork processes, and every process has it's
own cache memory. So it uses more memory, 8x more memory with 8
'num-threads:' setting.

On the plus side, its 10-20% faster :slight_smile:

Best regards,
~ Wouter

Dmitriy Demidov wrote:

Hello unbound-users!

I'm runing unbound-1.0.2 installed from ports, under FreeBSD 7.0
Unbound is installed without libevent. Yesterday I have done

portupgade from

unbound-1.0.1 to unbound-1.0.2

Now I met the following problem - Unbound do not starts. I found the

reasone -

it is because of value of num-threads set. Old Unbound 1.0.1

configuration

has been set to use num-threads: 8 , but Unbound 1.0.2 can be started

only

with setting num-threads: to 3 or less.

When I try to start Unbound with num-threads: 4 or more, unbound says

to me

that:

# /usr/local/etc/rc.d/unbound start
Starting unbound.
[1218219064] unbound[3358:0] error: too many file descriptors

requested. The

builtinmini-event cannot handle more than 1024. Config for less fds or
compile with libevent
[1218219064] unbound[3358:0] fatal error: configuration needs too many

file

descriptors

---

My system is configured this way:

ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) 524288
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 20095
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 65536
cpu time (seconds, -t) unlimited
max user processes (-u) 5547
virtual memory (kbytes, -v) unlimited

---

My unbound.conf:

server:
        verbosity: 0
        #statistics-interval: 3600
        #num-threads: 8
        num-threads: 3
        interface: 0.0.0.0
        port: 53
        outgoing-range: 256
        msg-cache-size: 16m
        msg-cache-slabs: 4
        num-queries-per-thread: 1024
        rrset-cache-size: 32m
        rrset-cache-slabs: 4
        cache-max-ttl: 86400
        infra-host-ttl: 900
        infra-lame-ttl: 900
        infra-cache-slabs: 4
        infra-cache-numhosts: 10000
        infra-cache-lame-size: 10k
        do-ip4: yes
        do-ip6: no
        do-udp: yes
        do-tcp: yes
        do-daemonize: yes

        access-control: 0.0.0.0/0 refuse
        access-control: 127.0.0.0/8 allow
        access-control: ::0/0 refuse
        access-control: ::1 allow
        access-control: ::ffff:127.0.0.1 allow

        chroot: "/usr/local/etc/unbound"
        username: "unbound"
        directory: "/usr/local/etc/unbound"
        #logfile: "/usr/local/etc/unbound/unbound.log"
        logfile: ""
        use-syslog: no
        pidfile: "/usr/local/etc/unbound/unbound.pid"
        root-hints: "/usr/local/etc/unbound/named.cache"

        identity: "DNS"
        version: "1.0"
        hide-identity: yes
        hide-version: yes
        harden-glue: yes
        do-not-query-address: 127.0.0.1/8
        do-not-query-address: ::1
        do-not-query-localhost: yes
        module-config: "iterator"

---

AFAIK my unbound configuration with num-threads: 4 will requests only

256x4 =

1024 file descriptors + 20 for TCP. In the same time there is system wide
ilimit for 20095 file descriptors...

Do this behaviour is because of hard coded threshold in Unbound?
If file descriptors count is >= 1024 and no libevent is used, then

Unbound

will stops with this error message?

You see, right now, under FreeBSD there is only libevent 1.3e

available in

ports, and I do not want to use it because of problems that it

produced to me

Unbound 1.0.2 calculates the maximum number of open files. And checks
if that is going to work. So that it does not start OK, but later on,
when it is busy, fails. This error is because the select system call
has a hardcoded limit of 1024 (limited by the system).

For what it's worth - I'm running 1.0.2 on a FreeBSD 7.0-STABLE single
processor box, with num-threads = 1. No apparent problems so far.

The 0x20 stuff gives me lots of interesting stuff in the logs, though
:slight_smile:

Steinar Haug, Nethelp consulting, sthaug@nethelp.no

Hi Wouter. Thanks for you answer!

Hi Dmitriy,
Unbound 1.0.2 calculates the maximum number of open files. And checks
if that is going to work. So that it does not start OK, but later on,
when it is busy, fails. This error is because the select system call
has a hardcoded limit of 1024 (limited by the system).

Sorry. Can you please explain it more detailed - "hardcoded limit of 1024" -
is it FreeBSD limit or Unbound limit? Unbound 1.0.2 now checks for that, but
earlier versions didn't performed that checks?

Libevent does not work for you?
OK an alternative.
Compile unbound with ./configure --without-pthreads

OK, I will take it into consideration.

It would be nice to be able to do it right - via threading, not via process
forking... May be, my problems will be solved then new libevent version will
be available in ports.

Abount unbound+libevent 1.3e problems. When unbound is linked with it, is
freezes randomly, it may do core dumps, and, if for example, it is configured
with 8 threads - only 2 or 3 of them are used and remaining threads are not
used at all (unbound statistics shows that). :frowning:

Thank one more time for you answer.

limit on number of descriptors for select is FD_SETSIZE

/usr/include/sys/select.h:#define FD_SETSIZE 1024U

If you need more than 1024 descriptors better to build Unbound with libevent.

Other option is use one thread - for not heavy loaded server it will be enough.

Anton Yuzhaninov wrote:

Hi Wouter. Thanks for you answer!

Hi Dmitriy,
Unbound 1.0.2 calculates the maximum number of open files. And checks
if that is going to work. So that it does not start OK, but later on,
when it is busy, fails. This error is because the select system call
has a hardcoded limit of 1024 (limited by the system).

Sorry. Can you please explain it more detailed - "hardcoded limit of
1024" - is it FreeBSD limit or Unbound limit? Unbound 1.0.2 now checks
for that, but earlier versions didn't performed that checks?

limit on number of descriptors for select is FD_SETSIZE

/usr/include/sys/select.h:#define FD_SETSIZE 1024U

If you need more than 1024 descriptors better to build Unbound with
libevent.

Other option is use one thread - for not heavy loaded server it will be
enough.

The above is correct.
And Yes, the limit was there in previous versions, but 1.0.2 does
checking for it.

That is why I suggested forking processes, because then every process
has a 1024 limit - thus 8*1024 - 8192 limit in total.

Unbound uses num-threads equal to your number of cpus really.
It could also be a scheduling problem in your OS (like a raging herd
problem).

Best regards,
~ Wouter

Dmitriy Demidov wrote:

Abount unbound+libevent 1.3e problems. When unbound is linked with
it, is freezes randomly, it may do core dumps, and, if for example, it
is configured with 8 threads - only 2 or 3 of them are used and
remaining threads are not used at all (unbound statistics shows
that). :frowning:

I saw the same problem on Linux 2.6, using unbound 1.0.2 with libevent
1.3e. Upgrading libevent to 1.4.6 seems to have fixed the problem.

  Norbert

Unfortunately, I was wrong. Using libevent 1.4.6 does not fix the
problem, unbound still freezes randomly, uses 100% CPU and does not
answer queries.

  Norbert

Norbert Tretkowski wrote:

Dmitriy Demidov wrote:

Abount unbound+libevent 1.3e problems. When unbound is linked with
it, is freezes randomly, it may do core dumps, and, if for example, it
is configured with 8 threads - only 2 or 3 of them are used and
remaining threads are not used at all (unbound statistics shows
that). :frowning:

I saw the same problem on Linux 2.6, using unbound 1.0.2 with libevent
1.3e. Upgrading libevent to 1.4.6 seems to have fixed the problem.

Unfortunately, I was wrong. Using libevent 1.4.6 does not fix the
problem, unbound still freezes randomly, uses 100% CPU and does not
answer queries.

  Norbert

Which backend is libevent using? unbound at high verbosity(4) will print
the backend that libevent is using at startup, like:

[1219063086] unbound[21633:0] debug: libevent 1.4.5 uses select method.

On linux it defaults to epoll. That must have the bug. You can disable
the libevent EPOLL backend with:
         EVENT_NOEPOLL=yes; export EVENT_NOEPOLL
set this environment variable in the shell before you start unbound.
Then it will use poll(2), which is slower, but much simpler. That could
make your bugs go away.

If this is a libevent problem, I am not sure how I can help?

For the record (and for Solaris users) the other libevent backends can
be disabled as well (you can define multiple environment variables)
         EVENT_NOKQUEUE=yes; export EVENT_NOKQUEUE
         EVENT_NODEVPOLL=yes; export EVENT_NODEVPOLL
         EVENT_NOPOLL=yes; export EVENT_NOPOLL
         EVENT_NOSELECT=yes; export EVENT_NOSELECT
         EVENT_NOEPOLL=yes; export EVENT_NOEPOLL
         EVENT_NOEVPORT=yes; export EVENT_NOEVPORT

There is also libev (http://dist.schmorp.de/libev/libev-3.43.tar.gz),
that aims to provide an alternative libevent-API implementation (author
says it is faster). Quick compile test on a linux machine: ./configure;
make on libev-3.43, then ./configure --with-libevent=<thelibevbuilddir>
and that seems to work fine. It can be an alternative.

Best regards,
   Wouter

You are right, it uses epoll. I'll try with poll method instead, thanks.

  Norbert

Wouter Wijngaards wrote:

There is also libev (http://dist.schmorp.de/libev/libev-3.43.tar.gz),
that aims to provide an alternative libevent-API implementation (author
says it is faster). Quick compile test on a linux machine: ./configure;
make on libev-3.43, then ./configure --with-libevent=<thelibevbuilddir>
and that seems to work fine. It can be an alternative.

hi, Wouter and Norbert:

I happen to be the debian maintainer responsible for both unbound and
libev. I'd be interested in hearing if I should configure unbound with
libev by default, or if I should provide multiple unbound packages
configured with different event libs a la the apache2-mpm-* packages.

Robert Edmonds wrote:

Wouter Wijngaards wrote:

There is also libev (http://dist.schmorp.de/libev/libev-3.43.tar.gz),
that aims to provide an alternative libevent-API implementation (author
says it is faster). Quick compile test on a linux machine: ./configure;
make on libev-3.43, then ./configure --with-libevent=<thelibevbuilddir>
and that seems to work fine. It can be an alternative.

hi, Wouter and Norbert:

I happen to be the debian maintainer responsible for both unbound and
libev. I'd be interested in hearing if I should configure unbound with
libev by default, or if I should provide multiple unbound packages
configured with different event libs a la the apache2-mpm-* packages.

If they both crash, without any libev/libevent, but using the builtin.

The builtin select has a 1024 filedescriptor limit, which is Norbert's
problem ; he wants more. And the available choices for that are:

o compile without pthreads, no shared memory, but the forked processes
each have 1024 file descriptors. (set num-threads: 4 as usual, but
instead of threads unbound will fork processes).
o compile with libevent. Choice of backends (system-specific or poll).
o compile with libev. Also multiple backends.

Best regards,
   Wouter

Using libevent's poll method does not fix the problem.

  Norbert