Fix compat b64_{pton,ntop} handling in nsd

NSD
1

Currently NSD's compat functions for b64_pton and b64_ntop are always
used. Worse, they're the old versions which abort(). This is because
the library symbols are prefixed by __ (not just in OpenBSD) and are
only accessible as b64_xx when resolv.h is included.

First attempt at fixing this involved borrowing tmux's autoconf checks
to find these functions. This works on OpenBSD, but fails on OS with
non-ancient resolv.h/nameser.h files which have #defines for T_xxx
resource records which conflict with those in NSD.

So the diff below uses the __ versions directly (OpenSSH does similar).
I've tested this on OpenBSD and Linux (SL6), including with the autoconf
check artificially broken to force using the compat versions.
The autoconf check is borrowed/adapted from the tmux one.

OK for OpenBSD?

Would this be acceptable for NSD upstream?

Index: configparser.y

2

Stuart Henderson wrote:

Currently NSD's compat functions for b64_pton and b64_ntop are always
used. Worse, they're the old versions which abort(). This is because
the library symbols are prefixed by __ (not just in OpenBSD) and are
only accessible as b64_xx when resolv.h is included.

First attempt at fixing this involved borrowing tmux's autoconf checks
to find these functions. This works on OpenBSD, but fails on OS with
non-ancient resolv.h/nameser.h files which have #defines for T_xxx
resource records which conflict with those in NSD.

So the diff below uses the __ versions directly (OpenSSH does similar).
I've tested this on OpenBSD and Linux (SL6), including with the autoconf
check artificially broken to force using the compat versions.
The autoconf check is borrowed/adapted from the tmux one.

OK for OpenBSD?

Would this be acceptable for NSD upstream?

This seems suspicious. Where are the prototypes for these function coming
from?

3

They're from the existing prototypes in nsd's configure.ac, the parameters
are the same size as the prototypes in resolv.h at least here and on Linux,
(written differently though, uint8_t / unsigned char).

These libc functions are a complete mess anyway, do you have any ideas
on how to do it better?