Version 1.15.0 compatible with libunbound.so.2, is it good idea?

Unbound
1

Hi,

We have request [1] to rebase unbound to recent version on RHEL8. So I
dug into the code and found relatively simple way to keep ABI unchanged
and compatible with version 1.7.3, which is still there. It allows
upgrading to unbound 1.15.0 without the need to recompile depending
binaries. Or at least it seems so.

Would you review the change [2], whether it creates possible
regressions? Do you think it might be good idea?

I have already found libreswan does not expect such change and would not
build with such version. Do you know about other users of unbound
library, which might be affected? Do you know about any other attempt to
keep ABI but update bugs or features? Has any other distribution solved
such problem?

Cheers,
Petr

1. https://bugzilla.redhat.com/show_bug.cgi?id=2027735
2.
https://github.com/InfrastructureServices/unbound/commit/190bd437598aafc02f6a9f044b00a0106ebe28e2

2

We have request [1] to rebase unbound to recent version on RHEL8.

yay :slight_smile:

dug into the code and found relatively simple way to keep ABI unchanged
and compatible with version 1.7.3, which is still there. It allows
upgrading to unbound 1.15.0 without the need to recompile depending
binaries. Or at least it seems so.

Would you review the change [2], whether it creates possible
regressions? Do you think it might be good idea?

This looks good to me. The only change I would make is instead of using
0x100 RCODE to signal this, to use a value in the 3841-4095 range as
technically 256 can become a valid RCODE:

https://www.iana.org/assignments/dns-parameters/dns-parameters.xhtml#dns-parameters-6

I have already found libreswan does not expect such change and would not
build with such version.

You mean to say that with the patch, no recompile is needed right ?

Thanks for looking at modernizing rhel8 unbound !

Paul

3

...

I have already found libreswan does not expect such change and would not
build with such version. Do you know about other users of unbound
library, which might be affected? Do you know about any other attempt to
keep ABI but update bugs or features? Has any other distribution solved
such problem?

It looks like by now, (some) users of libunbound already expect the new
interface with extra parameter (maybe with an ifdef based on version),
and this change will break it on source level.

In debian we have everything building with the new API for quite some time, --
many packages use libunbound8 these days.

/mjt

4

Ah, true. I have hit this with libreswan package in copr. Made another
change, which should solve this. I fake version definitions in unbound.h
to 1.7.x, where x is concatenated real major and minor version. That
should work with most of source codes.

When I tried recompilation at copr
(https://copr.fedorainfracloud.org/coprs/pemensik/unbound/builds/), only
libreswan failed before. I hope this change would make it transparent
for it.

Check out this addition:
https://github.com/InfrastructureServices/unbound/commit/4e66707e8ce18c7f7b8e4ed954b7298c830eea40

I think it could work well.

5

07.05.2022 12:59, Petr Menšík wrote:

Ah, true. I have hit this with libreswan package in copr. Made another
change, which should solve this. I fake version definitions in unbound.h
to 1.7.x, where x is concatenated real major and minor version. That
should work with most of source codes.

When I tried recompilation at copr
(https://copr.fedorainfracloud.org/coprs/pemensik/unbound/builds/), only
libreswan failed before. I hope this change would make it transparent
for it.

Check out this addition:
https://github.com/InfrastructureServices/unbound/commit/4e66707e8ce18c7f7b8e4ed954b7298c830eea40

I think it could work well.

Yes, that should work.

The only prob is what we do now :slight_smile:
Especially once some new features are available in libunbound and new software
will try to use UNBOUND_VERSION_* macros to find out if it is available :slight_smile:

I understand the resistence against rebuilding other software to update
soname, especially in cases like this which could have been avoided easily
(we'd all live much better life if we knew everything beforehand :wink: ).
I don't remember how it is on redhat, - on debian we have this done almost
automatically, the transitions from one soname to another. Stuff just being
rebuilt. The probs happen and requires manual intervention if things goes
wrong ofcourse, but in many cases it just works.

It is even possible for upstream unbound to switch from libunbound8 back to
libunbound2 (if it was 2 before, I don't remember already) with this change
in API (not only ABI but also API).

I think we should stay more compatible between each other and upstream. I
really hated it in the past when redhat API was entirely different than
anything else including upstream :slight_smile:

In debian I know only once piece of software (besides unbound itself) who
use callbacks - this is libreswan.

/mjt

6

Yes, that should work.

The only prob is what we do now :slight_smile:
Especially once some new features are available in libunbound and new
software
will try to use UNBOUND_VERSION_* macros to find out if it is
available :slight_smile:

When someone needs new features of libunbound, he has two choices

a) Detect RHEL8 compatibility macros and use UNBOUD_VERSION_MINOR_REAL
b) Modify macros detection to detect the feature presence itself, not by
unbound version

I understand the resistence against rebuilding other software to update
soname, especially in cases like this which could have been avoided
easily
(we'd all live much better life if we knew everything beforehand :wink: ).
I don't remember how it is on redhat, - on debian we have this done
almost
automatically, the transitions from one soname to another. Stuff just
being
rebuilt. The probs happen and requires manual intervention if things goes
wrong ofcourse, but in many cases it just works.

It would not be a problem on Fedora. We can rebuild dependencies too.
But RHEL is stable distribution not only for its own packages. We
provide API and ABI even for any customer's projects. If they would use
unbound for anything, they would have to rebuild in minor release and
support one version before and one version after. That is what I would
like to prevent.

It is even possible for upstream unbound to switch from libunbound8
back to
libunbound2 (if it was 2 before, I don't remember already) with this
change
in API (not only ABI but also API).

I think we should stay more compatible between each other and upstream. I
really hated it in the past when redhat API was entirely different than
anything else including upstream :slight_smile:

We have a rule 'upstream first' for almost every change we do. If
upstream is willing to make it different way than our proposal, we would
use it that way. If it can satisfy our demands of course. But this
change is compatibility change only. Intended to upgrade unbound daemon
itself, but do not break other dependencies. It would also allow not
shipping unbound-libs 1.7.x and unbound-1.15.x as separate packages
conflicting between themselves.

In debian I know only once piece of software (besides unbound itself) who
use callbacks - this is libreswan.

/mjt

It is different for us in RHEL. We support only part of packages
ourselves. For others we provide building block, but do not have build
or maintain them ourselves. Primary examples are in EPEL repository.
Some other could be private packages of other companies. I think unbound
is a great library and I would like to make it as stable as possible. We
prevent also possible breakages of software not in our distribution.

If someone relies on newer version, there would be RHEL 9 soon for them.
It never had older ABI. But I admit more projects use unbound, but only
libreswan failed to rebuild without the second commit. I guess that
means asynchronous resolution is not very common.

7

On 5/8/22 12:28, Michael Tokarev wrote:

Yes, that should work.

The only prob is what we do now :slight_smile:
Especially once some new features are available in libunbound and new
software
will try to use UNBOUND_VERSION_* macros to find out if it is
available :slight_smile:

When someone needs new features of libunbound, he has two choices

a) Detect RHEL8 compatibility macros and use UNBOUD_VERSION_MINOR_REAL

Please ensure a simple define is present, preferably with REDHAT in the name ?

b) Modify macros detection to detect the feature presence itself, not by
unbound version

Please don’t assume autoconf. Without autoconf, detecting outside of ifdef’s is really hard and hacky.

Paul

IIt would not be a problem on Fedora. We can rebuild dependencies too.

But RHEL is stable distribution not only for its own packages. We
provide API and ABI even for any customer's projects. If they would use
unbound for anything, they would have to rebuild in minor release and
support one version before and one version after. That is what I would
like to prevent.

Package old as libunboundXX, similar to like bind8/bind(9) in the past?

Paul

8

On 5/8/22 12:28, Michael Tokarev wrote:

Yes, that should work.

The only prob is what we do now :slight_smile:
Especially once some new features are available in libunbound and new
software
will try to use UNBOUND_VERSION_* macros to find out if it is
available :slight_smile:

When someone needs new features of libunbound, he has two choices

a) Detect RHEL8 compatibility macros and use UNBOUD_VERSION_MINOR_REAL

Please ensure a simple define is present, preferably with REDHAT in the name ?

would UNBOUND_VERSION_MINOR_REDHAT work better? Or should I define something like UNBOUND_REDHAT_CALLBACK? I am not sure

b) Modify macros detection to detect the feature presence itself, not by
unbound version

Please don’t assume autoconf. Without autoconf, detecting outside of ifdef’s is really hard and hacky.

Paul

It is still reasonably simple even with cmake. Do you have any specific
build environment on mind? It only compilation tests are used, it would
work fine even when cross compiling. But I admit I don't know simple
test for callback, which would fail with error. Not only with warning.

IIt would not be a problem on Fedora. We can rebuild dependencies too.

But RHEL is stable distribution not only for its own packages. We
provide API and ABI even for any customer's projects. If they would use
unbound for anything, they would have to rebuild in minor release and
support one version before and one version after. That is what I would
like to prevent.

Package old as libunboundXX, similar to like bind8/bind(9) in the past?

Paul

The thing is unbound-libs package contains also unbound-anchor.service,
which uses unbound-anchor to keep /var/lib/unbound/root.key up-to-date
automagically even if the key changes. Shipping another library package
would be possible, but it would have to solve conflict of those services
and who should maintain that key validity. It gets unnecessary complicated.

I think suggested changes make it simple enough and backward compatible
while adding just self-contained changes.

But all packages I checked on Fedora do not use ub_resolve_event
function with just one exception: libreswan. It seems no one else
adopted asynchronous callback.

I have just verified it still passes ratelimit test, even after my change.

9

09.05.2022 18:04, Petr Menšík wrote:
..

The thing is unbound-libs package contains also unbound-anchor.service,
which uses unbound-anchor to keep /var/lib/unbound/root.key up-to-date
automagically even if the key changes. Shipping another library package
would be possible, but it would have to solve conflict of those services
and who should maintain that key validity. It gets unnecessary complicated.

How do you run unbound-anchor? From a cron job?

unbound itself manages root trust anchor automatically these days
(before, unbound-anchor were needed to keep it up to date iirc).

In debian we decided to provide a separate package, dns-root-data, which
contains the root.key and root.hints, distributed using the usual way.
I dunno myself how reliable that will be in practice.

I think suggested changes make it simple enough and backward compatible
while adding just self-contained changes.

But all packages I checked on Fedora do not use ub_resolve_event
function with just one exception: libreswan. It seems no one else
adopted asynchronous callback.

Yes, this is exactly why it is failing, - this is the only known
software which actually uses this functionality... :slight_smile:

/mjt

10

For what it's worth, this is pretty much what we had in mind when I was at ICANN working on the team that deployed DNSSEC in the root zone.

We imagined that software vendors might use their established code- and package-signing crypto infrastructure to distribute verifies copies of the root zone trust anchor, which is why the root key (amongst other formats) has been made available in the form of a CSR, as described in RFC 7958.

The vendors we had in mind were those who maintained operating systems as well as DNS-specific software.

We thought vendors might make arrangements with ICANN to establish processes to validate the authenticity of each new root anchor, sign the corresponding CSR and either distribute the resulting certificate themselves or ask ICANN to do it. None of this really happened as we thought, but Debian distributing a trust anchor that has been verified as accurate to the satisfaction of whoever maintains the package seems like pretty much the same thing.

If you can trust environments to keep packages up to date, and you have a trustworthy package distribution system, this still seems like a good option to me, and likely still worth doing even if other mechanisms also exist, e.g. support for RFC 5011 in validating resolvers.

Joe

11

09.05.2022 18:04, Petr Menšík wrote:
..

The thing is unbound-libs package contains also unbound-anchor.service,
which uses unbound-anchor to keep /var/lib/unbound/root.key up-to-date
automagically even if the key changes. Shipping another library package
would be possible, but it would have to solve conflict of those services
and who should maintain that key validity. It gets unnecessary
complicated.

How do you run unbound-anchor? From a cron job?

It is actually triggered by systemd unit unbound-anchor.timer. It allows
not precise time daily job. But yes, very similar to cron job.

unbound itself manages root trust anchor automatically these days
(before, unbound-anchor were needed to keep it up to date iirc).

I know it does. But this is part of unbound-libs, because it targets
unbound library users. I think all servers except dnsmasq can do RFC
5011 rollover without additional configuration.

In debian we decided to provide a separate package, dns-root-data, which
contains the root.key and root.hints, distributed using the usual way.
I dunno myself how reliable that will be in practice.

We do not have separate trust anchor package for this. Main reason is
not all software consumes it in the same format. dnsmasq requires
different trust anchor format and bind needs that too. But we modified
most of software using unbound library for verification to use that
trust anchor. That way even openvswitch or libreswan should keep trust
anchor as accurate as possible. Even after eventual keyroll they should
be able to validate, even without the package update.

I think the only package which does not depend on unbound-libs but can
use its trust anchor is ldns-utils package. It will use unbound's trust
anchor too if it is installed, but does not have it as a dependency. I
thought about similar package as Debian has, but its reuse would be
limited anyway.

But I admit I don't know how trust anchor and hints are solved in
packages not inside RHEL, like pdns or knot. They might be able to reuse
some files if we made a shared package too.