I've been contacted to evaluate NSD performance and I've identified a
little stangeness in the TCP chatter with NSD. NSD always sends the
two byte response size as a separate TCP packet (causing the requestor
to send a separate ACK) to the main body of the request.
You might Expect a TCP DNS request (omitting the possible UDP request
that fails) to go something like:
SYN -->
<-- SYN,ACK
ACK -->
DNS QRY -->
<-- ACK
<-- DNS RSP
ACK -->
<-- FIN, ACK
FIN, ACK -->
... plus or minus the optimization of the ACK's with data (which seems
to require that you send(2) before your host receives the first ACK).
But NSD always does the following:
SYN -->
<-- SYN,ACK
ACK -->
DNS QRY -->
<-- ACK
<-- DNS length (2 bytes)
ACK -->
<-- DNS RSP
ACK -->
<-- FIN, ACK
FIN, ACK -->
Here's a very brief binary dump of the above conversation:
I reported this in May 2006. Found in nsd-3.0.6/doc/TODO:
- From Aaron Hopkins: write tcp length and tcp data in one write operation,
instead of multiple calls to write. Avoids Nagle algo delay in this case.
preallocate 2 bytes in front of buffer to put them into.
There are a few other performance-related things I reported at the same time
still in the TODO that you might be interested in.
A quick and simple, not very portable, fix in the meantime might be to use the
TCP_CORK/TCP_NOPUSH socket option (depending on platform) on the socket.
NSD always sends the two byte response size as a separate TCP packet
(causing the requestor to send a separate ACK) to the main body of the
request.
I reported this in May 2006. Found in nsd-3.0.6/doc/TODO:
- From Aaron Hopkins: write tcp length and tcp data in one write operation,
instead of multiple calls to write. Avoids Nagle algo delay in this case.
preallocate 2 bytes in front of buffer to put them into.
There are a few other performance-related things I reported at the same time
still in the TODO that you might be interested in.
A quick and simple, not very portable, fix in the meantime might be to use the
TCP_CORK/TCP_NOPUSH socket option (depending on platform) on the socket.
Yes, this is what NSD does. Simple code.
TCP performance is not considered important, as it should rarely be used
with DNS. UDP performance is the goal. Therefore the (otherwise
excellent) patches from Aaron did not get applied, they introduce extra
code, portability attention, where it is not needed.
The non-blocking patch is (in some form) in NSD now, Aaron, because that
fixed some race-conditions for secondary zones.
NSD always sends the two byte response size as a separate TCP
packet (causing the requestor to send a separate ACK) to the main
body of the request.
I reported this in May 2006. Found in nsd-3.0.6/doc/TODO:
- From Aaron Hopkins: write tcp length and tcp data in one write
operation, instead of multiple calls to write. Avoids Nagle algo
delay in this case. preallocate 2 bytes in front of buffer to put
them into.
Yes, this is what NSD does. Simple code.
TCP performance is not considered important, as it should rarely be
used with DNS. UDP performance is the goal. Therefore the
(otherwise excellent) patches from Aaron did not get applied, they
introduce extra code, portability attention, where it is not
needed.
I'd like to have a look at this patch. Maybe the patch can be worked
ina more acceptable manner. My client is very concerned about TCP
performance because of DNSSEC being on the horizon.
Of course, TCP isn't maybe the only way DNSSEC will get responses, but
it's a concern for sure. It also seems that this TCP issue is a DoS
waiting to happen, since it imposes rather more overhead, AFAICT.
Unless I've misunderstood something (and it wouldn't be the first time).
Hrm. Hadn't thought of this, but by my count, I can send three
packets (SYN, ACK, and the request) and get the host to occupy a TCP
slot and send at least 5 packets ... possibly more (SYN-ACK, ACK, 2
byte size, Reply, FIN (probably multiple FIN if I ignore them)).
That's not a large multiplier of packets, but it does occupy resources
while this is happening.
The byte multiplier is not bad (TCP has a lot of overhead). Under
IPv4, I send 120 bytes + request and I cause the host to send 200
bytes plus the reply (or more). Under IPv6, I send 180 bytes +
request and I cause the host to send 300 bytes plus the reply (or
more).
Fixing this bug takes the attack from 5+ packets to 4+ packets. It
probably doesn't change the length of time that the TCP stack keeps
the slot open (for FINs) in any measurable way.
But from a byte count perspective it reduces the overhead transmission
from 200 bytes to 160 bytes (vs. 120 bytes from the attacker) for v4
and from 300 bytes to 240 (vs. 180 bytes from the attacker) for v6.
Is this in something newer than 3.0.6? At least in the 3.0.6 source
grepping for CORK or NOPUSH results in no hits. Using them would solve the
problem of an extra round-trip to answer every TCP query at the expense of
two extra system calls per response.
- The use of blocking sockets in a nsd had some funky race conditions that
could've resulted in the server freezing or the extra servers started by the
-N flag not being used. This has been fixed in newer NSDs.
- Only one request is processed per socket per select(). And select() is
kind of expensive, particularly when you start handling more sockets (like
lots of TCP sockets open). Wrapping a loop around the UDP recvfrom() allows
us to amortize the cost of the select() over many requests, leading to a 23%
throughput improvement in my tests. (This will obviously vary with the
hardware, OS, etc.) The downside is that if it concentrates on one socket,
it can starve others at least briefly. The current version of the patch
limits this to 100 requests, which at 50000 requests/sec works out to up to
2ms of extra latency for the other sockets.
Is this in something newer than 3.0.6? At least in the 3.0.6 source
grepping for CORK or NOPUSH results in no hits. Using them would solve the
problem of an extra round-trip to answer every TCP query at the expense of
two extra system calls per response.
I am sorry you misunderstand my post.
I mean, the 5+ packet sequence is what NSD does, this includes the extra
round trip. It is simpler code. NSD does not use TCP_CORK/NOPUSH.
EDNS0 is a better way to handle DNSSEC sizes than TCP. TCP is not used
much in DNS, mostly for zone transfers or 'very large replies'.