Rewrite manpages to mdoc(7)

Dear authors of NSD,

currently, the manpages that come with NSD
are written in the traditional man(7) markup language.
I am proposing to rewrite them into the semantic markup
of the mdoc(7) language. I am willing to do the work.
See a version of nsd-checkzone.8 below as an example.

Both the man(7) and mdoc(7) languages have been around for decades,
and are supported by the prevalent formatters: groff(1) on most Linuxes
and mandoc(1) on the *BSDs and some others. In particular,
there is nothing to install or reconfigure on most systems
- both formats can be both rendered with existing man(1)
implemnetations and processed into plaintext, html, pdf, ps
and possibly other formats.

The main point is that mdoc(7) allows for constructs like

  .Op Fl f Ar arg

meaning

  there is an optional 'f' flag
  which takes an 'arg' argument

as opposed to

  switch to italics, type a bracket, a dash, "f",
  then switch to boldface and type "arg"

in the physical roff markup of man(7).
This leads to much better readability and maintainability.

Please let me know if you are interested.

  Jan

.Dd December 31, 2022
.Dt NSD-CHECKZONE 8
.Os
.Sh NAME
.Nm nsd-checkzone
.Nd check NSD zone file syntax
.Sh SYNOPSIS
.Nm
.Op Fl hp
.Op Fl i Ar oldfile
.Op Fl n Ar number
.Op Fl s Ar size
.Ar zonename
.Ar zonefile
.Sh DESCRIPTION
.Nm
reads a DNS zone file and checks it for errors.
It prints errors to stderr.
On failure it exits with nonzero exit status.
This is used to check files before feeding them to the
.Xr nsd 8
daemon.
The
.Ar zonename
is the zone to check,
the
.Ar zonefile
is the file to read.
.Pp
The options are as follows:
.Bl -tag -width Ds
.It Fl h
Print usage help information and exit.
.It Fl p
Print the zone contents to stdout if the zone is ok.
This prints the contents as it has been parsed,
not literally a copy of the input,
but as printed by the formatting routines in NSD,
much like the
.Xr nsd-control 8
command write does.
.It Fl i Ar oldfile
Create an IXFR from the differences between the old zone file
and the new zone file.
The
.Ar oldfile
argument to the
.Fl i
option is the old zonefile, the
.Ar zonefile
argument passed to
.Nm
is the new zonefile.
The difference is computed between the two zonefiles
by keeping one version of the zone in memory,
and another version in a temporary file.
The temporary file is located in the zonefile directory.
This is also where the result is written
in a file with the zonefile name, ending with
.Sq .ixfr .
This is also where NSD reads it when IXFRs are configured for the zone.
If other ixfr files exist for the zone,
they are renamed to become older IXFR contents for the zone.
If the output file already exists with the correct contents,
as determined by checking its header,
no new file is created.
.It Fl n Ar number
The number of IXFR versions to store at the most, 5 by default.
This is the number of ixfr files that get created for the zone.
Older ixfr files are deleted when the number is exceeded.
.It Fl s Ar size
The number of bytes of storage to use for IXFRs for the zone;
1048576 by default.
If an IXFR is bigger it is not created.
If the sum of IXFR file sizes exceeds this number,
older versions are deleted.
.El
.Sh SEE ALSO
.Xr nsd 8 ,
.Xr nsd-checkconf 8
.Sh AUTHORS
NSD was written by NLnet Labs and RIPE NCC joint team.
Please see CREDITS file in the distribution for further details.

Why not go to xml and use xml2man / pandoc ?

That’s what we use for libreswan. xml is easier to write and read than man/mdoc.

Paul

Sent using a virtual keyboard on a phone

Hi Jan, Paul,

The others will have to chip in as well. I don't have a strong opinion
about either, but we do have to build on all supported platforms. I
think the most strict build is OpenBSD. As we're part of base, we need
to ensure we don't depend on packages that are not in base(?) I think
mdoc is in OpenBSD, but I'm not sure.

I agree with Paul that neiter man/mdoc is easy to write. Though that's
probably because I don't do it on a regular base...

What I'd prefer is have the documentation in reStructuredText so we can
easily include it in the online documentation too. There's probably
some nifty way to convert that to manpages, but I haven't looked into
that in great detail.

- Jeroen

Actually, perhaps the other way is possible too if that easier(?) So,
mdoc to rst. But also, for that xml2man/pandoc maybe a better fit
because it's structured?

That being said, maybe mdoc is just a very nice first step. If it's
available on all supported systems and you're excited about it/willing
to do the work. Then feel free to do so, any improvement is always
appreciated!

Before you start though, I'd check with Wouter and Willem. There may be
a reason we're using what we're currently using and they'd know.

Thanks!

- Jeroen

My experience with several projects are to use RST and Sphinx. It’s human readable, the output is really nice, and you can publish the results to RTD.

Ondrej

This was discussed in 2015:

https://lists.nlnetlabs.nl/pipermail/nsd-users/2015-August/002167.html

Jeroen Koekkoek via nsd-users writes:

> Before you start though, I'd check with Wouter and Willem. There may
> be a reason we're using what we're currently using and they'd know.

As far as I know, nsd is older then mdoc.

  jaap

Why not go to xml and use xml2man / pandoc ?

Because it is a terrible manpage format
and the toolchain produces broken crap.

xml is easier to write and read than man/mdoc.

That's entirely a matter of opinion.
Which is easier to write, read, understand and maintain?
Which has less outside dependencies and produces better output?

(1)

.Sh NAME
.Nm nsd-checkzone
.Nd check NSD zone file syntax

(2)

<names>
<name>nsd-checkzone-conf<desc> checks DNS file syntax</desc></name>
</names>

Jan

mdoc is as old as 4.4BSD; it's not something new
that needs to be accommodated, it has been around for decades.

Ah, thanks for reminding me :slight_smile:

I just read the whole thread (please do as well)
- AFAICT it led nowhere for lack of interest.

The only valid point I see there is that Solaris
does not have a formatter for the mdoc(7) language,
which I find hard to beliave (doesn't Solaris have groff?).
But surely that's a moot point now.

  Jan

I have no specific knowledge on this, but would just like to "me too"
to this thread that whatever format is used, it MUSTN'T impact on the
quality of any resultant man/mdoc conversion.

</bikeshed>

Indeed, thanks Florian!

Thanks for all other suggestions too!

Haven't read the whole thread yet, but I do like to point out it's not
lack of interest.

Wouter makes an excellent point that I hadn't considered yet.
Portability and compatibility. I don't know if we still have Solaris
users, so I don't know if it'd make their lives harder.

One thing I did think of after my response is that if dependencies are
changed, we probably want to wait for the next minor (major?) version?
I feel we shouldn't make the lives of package maintainers harder by
swapping dependencies in a patch version.

I'll need to read up on possible tooling etc first to determine what
makes sense.

- Jeroen

> > Why not go to xml and use xml2man / pandoc ?
>
> Because it is a terrible manpage format
> and the toolchain produces broken crap.

I have no specific knowledge on this, but would just like to "me too"
to this thread that whatever format is used, it MUSTN'T impact on the
quality of any resultant man/mdoc conversion.

What I am proposing is to _write_ the mdoc(7) manpages,
not have them produced by some toolchain or another.

</bikeshed>

I am concerned that you consider the quality of documentation bikeshiding.

Hi Jan,

Was typing this on GitHub, but this is fine too :slight_smile:

It's really very nice of you that you want to do some of the work!

There's other angles to consider here too though. mdoc does not
automatically improve all things and that we can easily include the man
page from the online documentation is very compelling to us. Your
message simply brought that subject up too.

I can see why you want to have good manpages and maybe rst2man is not
ideal for that(?) I wouldn't know personally, but I can see that having
a dependency on Python is not ideal for all builds.

So, I was thinking, why not man2rst (other way around). I found this
article which uses mandoc to convert man pages written in mdoc to rst
(https://undeadly.org/cgi?action=article&sid=20170304230520).

I have to test it to see if it's viable. The added benefit would be
that we have "proper" or "plain" man pages for the basic builds and
have a dependency on Python, Sphinx, etc only for developers and online
documentation builds.

If my suggestions work, personally, I'd be more than happy to take you
up on your offer :slight_smile:

Of course, this too is subject to discussion, it's merely an idea for
now, but I'll run some tests.

Thanks!

- Jeroen

Hi Jeroen,

I would recommend to think about the “producers” and “consumers” here.

The consumers are users, so ideally you want output that’s easy to read - both man pages and the documentation (HTML, PDF, EPUB…)

The producers are developers and contributors - ideally you want a format that doesn’t have steep learning curve to promote people contributing fixes to the documentation.

Personally, I don’t think there’s much to gain by converting from a format only a handful people can read and write to another format that only handful people can read and write.

If you already have in-house experience with Sphinx, I would strongly suggest to stick with that. There are options how to make the dependency optional - either stick the pre-generated man pages to the git repository as I suggested in the GH issue (I’ve done this for both Knot DNS and BIND 9, so I can provide some hints on how to do that), or just skip the man page generation if Sphinx is not available.

The integration with the documentation is something that you cannot easily achieve with neither man or medic formats.

Python is pretty much ubiquitous nowadays and installing pypi packages is a common skill. I don’t consider this to be really a blocker.

And just another data point - BIND 9 used to have documentation in DocBook (XML) and it wasn’t that terrible, but we haven’t looked back since we converted the documentation to Sphinx. The only hiccups we hit from time to time are the old Sphinx versions in the older distributions.

The other thing to consider is the shared wisdom - BIND 9, Knot and PowerDNS projects are already using Sphinx, and you know where to find us :wink:

Ondrej

I would recommend to think about the “producers” and “consumers” here.

The consumers are users, so ideally you want output that’s easy to read
- both man pages and the documentation (HTML, PDF, EPUB…)

To be clear: I am only talking about the manpages.
While the mdoc(7) language could be (ab?)used to annotate
other kinds of texts (such as lengthy structured prose),
it is intended as a semantic annotation of manpages.

The producers are developers and contributors - ideally you want
a format that doesn’t have steep learning curve to promote people
contributing fixes to the documentation.

I might be misunderstanding "promote" here, but sure:
you want a simple, well-established, standardized language;
such as mdoc(7), the language of manpages since the 70's.

Personally, I don’t think there’s much to gain by converting
from a format only a handful people can read and write to another
format that only handful people can read and write.

While true, this misses the point of my proposal entirely.
The benefit is (1) the semantic markup, (2) the ease of
reading, writing and maintaining the mapages (meaning source).

  .Xr nsd 8
  => see also the manpage of nsd in section 8

as opposed to

  \fInsd\fR(8)
  => switch to italics, type "nsd",
     switch back to roman, and type 8 in parenthesis
  

If you already have in-house experience with Sphinx,

Apparently, the content of
https://nsd.docs.nlnetlabs.nl/en/latest/
is produce by Sphinx.

I would strongly suggest to stick with that. There are options
how to make the dependency optional - either stick the pre-generated
man pages to the git repository as I suggested in the GH issue

Having _generated_ code in the repository
is just asking for unsynced versions.

Manpage is a manpage. It should be written by hand and maintained,
just like any other source file.

Maybe that is where our views differ: manpage is the source, to be
formatted with man(1) (i.e. eventualy by groff or mandoc or whatever
behind the curtains), or processed into html od pdf or whatever else,
not something that gets created automaticaly;
it's source code, just like the C files.

or just skip the man page generation if Sphinx is not available.

That's a joke, right?
Or are you seriously proposing to _not_ have manpages?

The integration with the documentation is something
that you cannot easily achieve with neither man or medic formats.

I don't know what medic is, but html integration of mdoc(7) source
is trivial with mandoc -Thtml, and perfectly doable with groff -Thtml,
and has been for a long time. Stop the fud.

Python is pretty much ubiquitous nowadays and installing pypi
packages is a common skill. I don’t consider this to be really a blocker.

For comparison, man(7) and mdoc(7) formatters have been around
since the seventies.

And just another data point - BIND 9 used to
have documentation in DocBook (XML) and it wasn’t that terrible

See here for why DocBook would be a terrible choice:
https://undeadly.org/cgi?action=article&amp;sid=20190419101505

The other thing to consider is the shared wisdom - BIND 9,
Knot and PowerDNS projects are already using Sphinx,

Yes, and entire operating system, including macOS and all the BSDs
have their manpages in mdoc(7), for comparison.

  Jan

> The integration with the documentation is something
> that you cannot easily achieve with neither man or medic formats.

I don't know what medic is, but html integration of mdoc(7) source
is trivial with mandoc -Thtml, and perfectly doable with groff -Thtml,
and has been for a long time. Stop the fud.

For example, the entirety of manpages.debian.org is done with that.

For comparison here’s one of the bind9 man pages — I for one much prefer the resulting output.

https://bind9.readthedocs.io/en/v9_18_10/manpages.html#delv-dns-lookup-and-validation-utility

Some of the output formats have to be generated (man pages, too, I guess!). I’m not sure why it’s important that the man pages are the “source” as long as they get generated, too.

Ask

If you're comparing the same manual between bind9.readthedocs.io (rst
straight to html) and manpages.debian.org (first rst is converted to
man, and that is then converted to html) it's obvious which is going
to look best, simply because it has more information available
(semantic markup rather than basic formatting information).

A better comparison would be between html pages generated from a
"proper" mdoc conversion and from the rst source, e.g.

https://bind9.readthedocs.io/en/v9_18_10/manpages.html#dig-dns-lookup-utility
https://man.openbsd.org/dig

- the content is a bit different between the two as the version of dig
in openbsd is from an old fork and cut-down compared to current ISC
releases, but ignoring that, you get a better picture of what high
quality output looks like from each of the source formats.

Both look pretty much ok to me; the semantic information in mdoc is
pretty good for searches, and mandoc's linter can be helpful to maintain
quality, on the other hand rst has a slightly lower barrier for users to
edit at the expense of heavier dependencies. The key thing is to use
something that whoever does most work to maintain the documentation is
happy with really imho.

> For example, the entirety of manpages.debian.org is done with that.

For comparison here’s one of the bind9 man pages
https://bind9.readthedocs.io/en/v9_18_10/manpages.html#delv-dns-lookup-and-validation-utility
— I for one much prefer the resulting output.

This "resulting output" is a html rendering of
https://raw.githubusercontent.com/isc-projects/bind9/main/bin/delv/delv.rst
apparently produced with http://docutils.sourceforge.net/

Compare the above to e.g.

  http://man.openbsd.org/ls
  https://manpages.debian.org/bullseye/coreutils/ls.1.en.html

That's three different html layouts.
That's not what I'm talking about; these are just examples
to show that having manpages in mdoc(7), or man(7) for that matter,
is no obstacle to having html versions too, as FUDed here previously.

The html content of
https://manpages.debian.org/bullseye/coreutils/ls.1.en.html
is in fact a rendering of
https://manpages.debian.org/bullseye/coreutils/ls.1.en.gz

Similarly, the html content of
http://man.openbsd.org/ls
is a rendering of
http://cvsweb.openbsd.org/cgi-bin/cvsweb/~checkout~/src/bin/ls/ls.1

You are comparing the html outputs;
that's not what I'm talking about.

Some of the output formats have to be generated (man pages, too, I guess!).

Currently, nsd* has a set of *.8 manpages in man(7)
(in fact, *.8.in run through some sed replacements).
What I'm proposing is to have a set of *.8 manpges in mdoc(7),
because mdoc(7) is a better manpage annotation language then man(7).

Formats such as html (and pdf and ps and ...) can be generated from both,
which has been the case for decades.

I’m not sure why it’s important that the man pages are the “source”
as long as they get generated, too.

Generated from what? That's the issue here: the source format
of the manpages. Currently it's man(7); I say mdoc(7),
and am offering to do the work.

Compare https://manpages.debian.org/bullseye/coreutils/ls.1.en.gz
to http://cvsweb.openbsd.org/cgi-bin/cvsweb/~checkout~/src/bin/ls/ls.1
and compare nsd's nsd-checkzone.8, written in man(7), to the one I sent,
written in mdoc(7). That's what I'm proposing.

  Jan