Nsd utilize 100% cpu

Hi there.

Can anybody help me or explain what happen?

I run nsd with 40к domain zones.
And very often nsd utilize 100%cpu for a long time.
For example, when I run nsdc patch, command have already finished but process nsd still running and use 100% cpu.
When I do strace I see:
gettimeofday({1260459192, 450604}, NULL) = 0
pselect6(8, [7], [7], [], {4, 549396000}, {NULL, 8}) = 1 (out [7], left {4, 549396000})
write(7, "\10\0\0\0\0\17\1\t\3\10\5\0\4nw-r\2ru\0", 21) = 21
gettimeofday({1260459192, 461029}, NULL) = 0
pselect6(8, [7], [7], [], {4, 538971000}, {NULL, 8}) = 1 (out [7], left {4, 538971000})
write(7, "\10\0\0\0\0\26\1\20\3\17\f\0\vnw-normativ\2ru\0", 28) = 28
gettimeofday({1260459192, 471457}, NULL) = 0

And even after 20 minutes after "nsd patch" have finished, nsd process still use 100% cpu.

Can anybody help me?

Vadim wrote:

Hi there.

Can anybody help me or explain what happen?

I run nsd with 40к domain zones.
And very often nsd utilize 100%cpu for a long time.
For example, when I run nsdc patch, command have already finished but process nsd still running and use 100% cpu.

Can you please specify at least version of nsd you're running?

When I do strace I see:
gettimeofday({1260459192, 450604}, NULL) = 0
pselect6(8, [7], [7], [], {4, 549396000}, {NULL, 8}) = 1 (out [7], left {4, 549396000})
write(7, "\10\0\0\0\0\17\1\t\3\10\5\0\4nw-r\2ru\0", 21) = 21
gettimeofday({1260459192, 461029}, NULL) = 0
pselect6(8, [7], [7], [], {4, 538971000}, {NULL, 8}) = 1 (out [7], left {4, 538971000})
write(7, "\10\0\0\0\0\26\1\20\3\17\f\0\vnw-normativ\2ru\0", 28) = 28
gettimeofday({1260459192, 471457}, NULL) = 0

mchost? :slight_smile:

And what is the fd#7? Is it the database file? What does lsof say?
How large the resulting db file is?

Apparently nsd writes each domain label separately, without any attempt
to batch writes. It's much better than writing by single char as some
other programs are doing but worse still than batching writes in at least
512byte blocks like stdio is doing.

And even after 20 minutes after "nsd patch" have finished, nsd process still use 100% cpu.

How long it takes total?

/mjt

We were also playing around with NSD. Loading about 80,000 domains. Using an older machine. Dual Xenon, 2GB ram. One thread would be at 100% cpu for hours, sometimes it would never get out of that state. Memory usage wasn't very high.

Dan Durrer
No-IP.com

Quoting Michael Tokarev <mjt@tls.msk.ru>:

Michael Tokarev wrote:

Vadim wrote:

Hi there.

Can anybody help me or explain what happen?

I run nsd with 40к domain zones.
And very often nsd utilize 100%cpu for a long time.
For example, when I run nsdc patch, command have already finished but process nsd still running and use 100% cpu.

Can you please specify at least version of nsd you're running?

Sorry. nsd-3.2.2-2.el5

When I do strace I see:
gettimeofday({1260459192, 450604}, NULL) = 0
pselect6(8, [7], [7], [], {4, 549396000}, {NULL, 8}) = 1 (out [7], left {4, 549396000})
write(7, "\10\0\0\0\0\17\1\t\3\10\5\0\4nw-r\2ru\0", 21) = 21
gettimeofday({1260459192, 461029}, NULL) = 0
pselect6(8, [7], [7], [], {4, 538971000}, {NULL, 8}) = 1 (out [7], left {4, 538971000})
write(7, "\10\0\0\0\0\26\1\20\3\17\f\0\vnw-normativ\2ru\0", 28) = 28
gettimeofday({1260459192, 471457}, NULL) = 0

mchost? :slight_smile:

And what is the fd#7?

nsd 13142 nsd 7u unix 0xffff880011084900 5504109 socket

Is it the database file?

no

What does lsof say?
How large the resulting db file is?

19M /var/lib/nsd/nsd.db

Apparently nsd writes each domain label separately, without any attempt
to batch writes. It's much better than writing by single char as some
other programs are doing but worse still than batching writes in at least
512byte blocks like stdio is doing.

And even after 20 minutes after "nsd patch" have finished, nsd process still use 100% cpu.

How long it takes total?

I ran again nsdc patch and nsd utilized 100% cpu about 10 minutes

/mjt

Thanks.

You mean nsdc rebuild? I wouldn't expect nsd patch to take that long. Though
AFAIK, nsd patch should not affect the nameserver operations going on.

Paul

Paul Wouters wrote:

Apparently nsd writes each domain label separately, without any attempt
to batch writes. It's much better than writing by single char as some
other programs are doing but worse still than batching writes in at least
512byte blocks like stdio is doing.

And even after 20 minutes after "nsd patch" have finished, nsd process still use 100% cpu.

How long it takes total?

I ran again nsdc patch and nsd utilized 100% cpu about 10 minutes

You mean nsdc rebuild? I wouldn't expect nsd patch to take that long. Though
AFAIK, nsd patch should not affect the nameserver operations going on.

No. I mean nsdc patch. I don't know what nsd do at this time, because command nsd patch works:
time nsdc patch > /dev/null 2>&1

real 0m43.057s
user 0m15.181s
sys 0m1.404s

But after that, nsd process begin utilize 100% cpu for a long time. I showed strace above.

Look what I see in top:
  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

26044 nsd 25 0 204m 193m 356 R 99.9 37.8 32:05.96 nsd

for 20 minutes now. Also I have upgraded nsd to 3.2.3 version but I see the same behaviour.