Running Cascade using containers

Cascade
1

Good morning all,

I’m posting this both to collect my thoughts in one place for something that concerns multiple different source code repositories (and so has no single convenient Git repository to which I could push it), and to get community feedback from any potential Cascade operators out there who would want to run it inside a container on their needs and any opinions and/or issues with what I describe here.

This is not a ready-to-run recipe for running Cascade using containers, please don’t ask for support on how to use this, though of course we do value feedback on anything you think is missing, incorrect or questionable.

Background & inspiration

These thoughts arose from some manual testing I did with Nitrokey NetHSM while reviewing remove obsolete note about NetHSM by jans23 · Pull Request #621 · NLnetLabs/cascade · GitHub . The Nitrokey NetHSM is offered in container form for testing and prior to the issue described in PR #621 @jpmens kindly contributed documentation on Integrating (Cascade) with a Nitrokey NetHSM. However, that documentation was written for a Debian 12 host while my work development machine runs Fedora.

Rather than run yet another instance of Cascade from sources or RPMs on my work machine, given that NetHSM is already avaiable in containerized form, I decided this would be a good excuse to create a Docker Compose YAML description of a private test network running Cascade, Cascade-HSM-Bridge and Nitrokey NetHSM as separate containers, as we know that there are customers that want to run Cascade in containerized form.

I also knew that there are some existing resources out there I could use to help me automate the setup of the Nitrokey NetHSM, including a provisioning script by Nitrokey themselves and Niek Willems of SIDN also has projects online based around NetHSM from which I could take inspiration from the “Stiab: DNSSEC Signer street-in-a-box” NetHSM provisioning script and CENTR tech 52 HSM Workshop.

So, let’s think a moment about what it means to run Cascade inside a container, e.g. using Docker (with which I have by far the most experience).

Typical architecture when using containers

Some things to keep in mind with Docker containers is that they are not complete operating systems, they don’t have systemd for example, or syslog, and the general advice is to run one process per container. Of course you can run multiple processes including syslog and systemd in a single container if you really want to, but it’s not what I’m going to try doing here.

Which components do we need to run Cascade when using containers?

For the Cascade daemon itself we need a Docker image that provides and runs the cascaded daemon binary and makes available to it its configuration file, data directories, and enables logging by Cascade to standard out/err (as is typical for containers).

This image will also need a few more things:

  • The dnst binary which Cascade invokes.

  • Optional: a way for users to mount their own review hook scripts into the container for Cascade to invoke.

  • The above imply that we also need the tini binary to run as PID 0 (for proper child process reaping and signal handling), or for operators to know that they must invoke Docker or other container host with --init (which also runs tini as PID 0 but without including tini in the container image).

  • Support for mounting files into the container from the host, e.g. for:

    • Supplying and editing policy files in order for cascade policy reload to work.
    • Enabling cascade zone add --source path/to/file to work (if wanted, e.g. instead of using A/IXFR to receive the input zone).

  • Expose some network UDP/TCP ports:

    • The Cascade publication server port (UDP+TCP): for DNS queries to be able to reach Cascade from outside the container, e.g. from upstream primaries, downstream secondaries, monitoring tools and operators.
    • The Cascade control port (TCP):
      • Prometheus metric gathering.
      • Health checking.
      • Control by a Cascade client binary running outside the container.

How will we configure and control the Cascade daemon?

The Cascade client binary needs to be run somewhere so that operators can configure and control Cascade. We could:

  • Require operators to exec a shell inside the Cascade daemon container and run the cascade client binary there, OR

  • Expect operators to separately have a copy of the cascade client binary on an operator machine and connect to the Cascade control port of the container running Cascade, OR

  • Run an additional container that provisions other containers such as NetHSM and also offers the Cascade binary for operators to shell in and use.

How can we get dnst into the Cascade daemon image?

  • Build from sources. This would be slightly awkward as normally, e.g. how our Ploutos packaging framework expects, a single Rust project Git repo is built and generated binaries included in the final Docker image, but dnst sources are in a separate repo to that of Cascade.
  • Pre-build the dnst binary outside the container and include it during image building. This would require the host to be the same architecture and use the same LIBC dependency as the container image.
  • Install dnst using a DEB or RPM package that we publish. This would not match well with the fact that we typically use Alpine as our base image for Docker images that we publish (for its tiny size).
  • Use the Docker FROM --image directive to copy the dnst binary from an existing published dnst Docker image. This would require a Dockerfile in the dnst project as well. This might at first seem odd as dnst is not a service you leave running, but there are examples of one shot containers that run a command produce an output. We could offer such an image and we could then copy the binary from inside it. If it too used Alphine, as we usually do, it would be compatible with the Cascade container image.
  • Mount a dnst binary from the host into the container. This would require the binary to be compatible with the container.

The above would be enough to use Cascade with on-disk keys, but not to use Cascade with an HSM.

What extra bits are needed to communicate with an HSM?

  • An image from which to deploy a container running the cascade-hsm-bridge daemon binary, and associated configuration file and logging to stdout/stderr (as is typical for containers).
  • A means to provide the HSM PKCS#11 module to the bridge daemon. The module would need to be binary compatible for the O/S and architecture, and have any supporting libraries, config files, environment variables, etc. i.e. the dependencies needed by the specific HSM type to run which varies from HSM to HSM. Often those files are only available to paying customers, i.e. we couldn’t provide images for all HSM we support even if we wanted to and had the resources to maintain them all, but we could offer a base image and a demonstration derived image based on something open like NetHSM. (or SoftHSM, Krypotic and probably others)

This could be achieved by offering a “base” image for the bridge that operators would extend with their own actual image containing their specific HSM PKCS#11 module and dependencies.

Such a base image might need to be Debian/Ubuntu and also available as RedHat compatible, so that operators whose PKCS#11 module, and possibly also required O/S package dependencies, are packaged as DEB or RPM.

Alternatively we could decide NOT to offer an image for the bridge but only document what such an image would need to do and leave others to create HSM specific images.

We would also need a way to provision the HSM for first use, e.g. setup user accounts and passwords. This could be hard-coded into the HSM image on startup or it can be done (assuming remote control is possible) via a separate provisioning container.

Current status

There are other tasks that have higher priority than Dockerization, thus I decided to stop where I was and capture my notes here to make sure they are not lost and also to get insight from others, especially prospective customers.

Below are the files I created. They are NOT finished, but they worked well enough for me to establish why I was has having difficulty with NetHSM and Cascade.

Issues encountered

  • Default binding to 127.0.0.1 doesn’t work inside a container because exposing/publishing the port to the outside leaves clients unable to connect, for that to work one must listen inside the container on 0.0.0.0.
  • Secrets management is a work in progress. Cascade-HSM-Bridge only works properly with NetHSM if the users AND passwords are defined in the p11nethsm.conf NetHSM PKCS#11 config file, so some way of getting external passwords, e.g. from Docker /run/secrets/xxxfiles, into the p11nethsm.conf file is needed.
  • Name resolution inside Docker containers seems to vary from one process to another in whether or not they can resolve the name of another container, or the value of the Docker Compose hostname can be resolved, so sometimes keyfender as the hostname of the NetHSM container is what I need to configure and other times I needed to use nethsm which is the container name.

What isn’t covered here

  • Persistent storage of data, e.g. via Docker volume or host mount or within the container and snapshot’ing the container.
  • High availability, clustering, hot standby, etc topologies: we don’t have support for this yet in Cascade, though we think we see a way that dnst is used we can have a single centralized key manager container (which is perhaps yet another argument for having a dedicated dnst image) producing keys and signing instructions to be consumed by multiple running Cascade deamon instances. However, we really want to take the time to look at such architectures thoroughly and consider the edge cases and how we want to handle them.
  • Possibly related to the above point: replicated/shared/distributed storage among containers.
  • Monitoring of the containers, e.g. what metrics might it be useful to expose via the Cascade Prometheus endpoint and are there any that are specific to containerized deployment? And is the way Cascade currently reports “health” compatible with or sufficient for dockerized deployment?

Supporting files

See the first reply to this post as including the files here makes this text too long to post!

Closing out…

So, that was a long brain dump, maybe too long for most people or anyone to actually read!

However, it would be really useful if potential operators of Cascade in a containerized setting let us know of anything they’d really like Cascade to have or they need Cascade to have in order to work better/at all in such an environment, so ladies and gentlemen, here’s your chance to let us know!

Thanks,

Ximon

2

Now the supporting files:

docker-compose.yml

# Based on:
#   - https://github.com/Nitrokey/nethsm-pkcs11/blob/main/container/openbao/compose.yml
services:
  nethsm:
    image: docker.io/nitrokey/nethsm:testing
    networks:
      - cascade-net
    hostname: keyfender

  nethsm-provisioner:
    image: curlimages/curl:latest
    depends_on:
      nethsm:
        condition: service_started
    volumes:
      - ./provision.sh:/provision.sh:ro
    command: ["/bin/sh", "/provision.sh"]
    networks:
      - cascade-net
    environment:
      - NETHSM_HOST_PORT=keyfender:8443
    secrets:
      - nethsm-operator-password
      - nethsm-admin-password

  cascade-hsm-bridge:
    image: nlnetlabs/cascade-hsm-bridge:nethsm-latest
    depends_on:
      nethsm:
        condition: service_started
      nethsm-provisioner:
        condition: service_completed_successfully
    networks:
      - cascade-net
    # secrets:
    #   - nethsm-operator-password
    #   - nethsm-admin-password
    environment:
      - NETHSM_OPERATOR_PASSWORD=012345678
      - NETHSM_ADMIN_PASSWORD=012345678

  cascade:
    image: nlnetlabs/cascade:latest
    depends_on:
      nethsm-provisioner:
        condition: service_completed_successfully
      cascade-hsm-bridge:
        condition: service_started
    networks:
      - cascade-net
    ports:
      - "4539-4542:4539-4542"
    volumes:
      - ./policies:/etc/cascade/policies
      - ./zones:/etc/cascade/zones

networks:
  cascade-net:
    driver: bridge

secrets:
  nethsm-operator-password:
    environment: "NETHSM_OPERATOR_PASSWORD"
  nethsm-admin-password:
    environment: "NETHSM_ADMIN_PASSWORD"

provision.sh

#!/bin/bash
# Based on:
#   - https://github.com/Nitrokey/nethsm-pkcs11/blob/main/container/openbao/provision.sh
#   - https://github.com/niek-sidn/stiab/blob/main/files/hsm-client/provision_hsm.sh

OP_PASS=$(cat /run/secrets/nethsm-operator-password)
ADMIN_PASS=$(cat /run/secrets/nethsm-admin-password)

while ! curl -f -k -s https://${NETHSM_HOST_PORT}/api/v1/health/alive ; do echo "Waiting for NetHSM to be alive..."; sleep 1 ; done

echo "OP_PASS=${OP_PASS}"
echo "ADMIN_PASS=${ADMIN_PASS}"

if curl -k -s https://${NETHSM_HOST_PORT}/api/v1/health/state | grep -q "Unprovisioned"; then
    echo "NetHSM unprovisioned. Provisioning now!"
    curl -k -v "https://${NETHSM_HOST_PORT}/api/v1/provision" -d '{"unlockPassphrase":"UnlockPassphrase","adminPassphrase":"'${ADMIN_PASS}'","systemTime":"'$(date -u "+%Y-%m-%dT%H:%M:%SZ")'"}'
    curl -k -v "https://admin:${ADMIN_PASS}@${NETHSM_HOST_PORT}/api/v1/users/operator" -X PUT -d '{"role":"Operator", "passphrase":"'${OP_PASS}'","realName":"Operator"}' -H "Content-Type: application/json"
else
    echo "NetHSM already provisioned!"
    if [ "${NETHSM_AUTO_UNLOCK}" = "true" ]; then
        echo "Auto unlocking NetHSM now!"
        curl -k -v "https://admin:${ADMIN_PASS}@${NETHSM_HOST_PORT}/api/v1/unlock" -d '{ "passphrase": "UnlockPassphrase" }' -H "Content-Type: application/json"
    fi
    while ! curl -f -k -s https://${NETHSM_HOST_PORT}/api/v1/health/ready ; do echo "Waiting for NetHSM to be ready..."; sleep 1 ; done
fi
echo "Finished provisioning!"
exit 0

cascade/Dockerfile

# This is a multi-stage Dockerfile, with a selectable first stage. With this
# approach we get:
#
#   1. Separation of dependencies needed to build our app in the 'build' stage
#      and those needed to run our app in the 'final' stage, as we don't want
#      the build-time dependencies to be included in the final Docker image.
#
#   2. Support for either building our app for the architecture of the base
#      image using MODE=build (the default) or for externally built app
#      binaries (e.g. cross-compiled) using MODE=copy.
#
# In total there are four stages consisting of:
#   - Two possible first stages: 'build' or 'copy'.
#   - A special 'source' stage which selects either 'build' or 'copy' as the
#     source of binaries to be used by ...
#   - The 'final' stage.


###
### ARG DEFINITIONS ###########################################################
###

# This section defines arguments that can be overriden on the command line
# when invoking `docker build` using the argument form:
#
#   `--build-arg <ARGNAME>=<ARGVALUE>`.


# DNST_VERSION
# ============
# Which DNST Docker image should the `dnst` binary be copied from?
# Defaults to `nlnetlabs/dnst:latest`
# ARG DNST_IMAGE=nlnetlabs/dnst:latest
# Commented out due to: https://github.com/moby/buildkit/issues/2034#issuecomment-834203964

# MODE
# ====
# Supported values: build (default), copy
#
# By default this Dockerfile will build our app from sources. If the sources
# have already been (cross) compiled by some external process and you wish to
# use the resulting binaries from that process, then:
#
#   1. Create a directory on the host called 'dockerbin/$TARGETPLATFORM'
#      containing the already compiled app binaries (where $TARGETPLATFORM
#      is a special variable set by Docker BuiltKit).
#   2. Supply arguments `--build-arg MODE=copy` to `docker build`.
ARG MODE=build


# BASE_IMG
# ========
#
# Only used when MODE=build.
ARG BASE_IMG=alpine:3.24


# CARGO_ARGS
# ==========
#
# Only used when MODE=build.
#
# This ARG can be used to control the features enabled when compiling the app
# or other compilation settings as necessary.
ARG CARGO_ARGS


###
### BUILD STAGES ##############################################################
###


# -----------------------------------------------------------------------------
# Docker stage: build
# -----------------------------------------------------------------------------
#
# Builds our app binaries from sources.
FROM ${BASE_IMG} AS build
ARG CARGO_ARGS

RUN apk add --no-cache rust cargo openssl-dev

WORKDIR /tmp/build
COPY . .

# `CARGO_HTTP_MULTIPLEXING` forces Cargo to use HTTP/1.1 without pipelining
# instead of HTTP/2 with multiplexing. This seems to help with various
# "spurious network error" warnings when Cargo attempts to fetch from crates.io
# when building this image on Docker Hub and GitHub Actions build machines.
#
# `cargo install` is used instead of `cargo build` because it places just the
# binaries we need into a predictable output directory. We can't control this
# with arguments to cargo build as `--out-dir` is unstable and contentious and
# `--target-dir` still requires us to know which profile and target the
# binaries were built for. By using `cargo install` we can also avoid needing
# to hard-code the set of binary names to copy so that if we add or remove
# built binaries in future this will "just work". Note that `--root /tmp/out`
# actually causes the binaries to be placed in `/tmp/out/bin/`. `cargo install`
# will create the output directory for us.
RUN CARGO_HTTP_MULTIPLEXING=false cargo install \
  --locked \
  --path . \
  --root /tmp/out/ \
  ${CARGO_ARGS}


# -----------------------------------------------------------------------------
# Docker stage: copy
# -----------------------------------------------------------------------------
# Only used when MODE=copy.
#
# Copy binaries from the host directory 'dockerbin/$TARGETPLATFORM' directory
# into this build stage to the same predictable location that binaries would be
# in if MODE were 'build'.
#
# Requires that `docker build` be invoked with variable `DOCKER_BUILDKIT=1` set
# in the environment. This is necessary so that Docker will skip the unused
# 'build' stage and so that the magic $TARGETPLATFORM ARG will be set for us.
FROM ${BASE_IMG} AS copy
ARG TARGETPLATFORM
ONBUILD COPY dockerbin/$TARGETPLATFORM /tmp/out/bin/


# -----------------------------------------------------------------------------
# Docker stage: source
# -----------------------------------------------------------------------------
# This is a "magic" build stage that "labels" a chosen prior build stage as the
# one that the build stage after this one should copy application binaries
# from. It also causes the ONBUILD COPY command from the 'copy' stage to be run
# if needed. Finally, we ensure binaries have the executable flag set because
# when copied in from outside they may not have the flag set, especially if
# they were uploaded as a GH actions artifact then downloaded again which
# causes file permissions to be lost.
# See: https://github.com/actions/upload-artifact#permission-loss
FROM ${MODE} AS source
RUN chmod a+x /tmp/out/bin/*


# -----------------------------------------------------------------------------
# Docker stage: final
# -----------------------------------------------------------------------------
# Create an image containing just the binaries, configs & scripts needed to run
# our app, and not the things needed to build it.
#
# The previous build stage from which binaries are copied is controlled by the
# MODE ARG (see above).
FROM ${BASE_IMG} AS final

# Copy binaries from the 'source' build stage into the image we are building.
COPY --from=source /tmp/out/bin/* /usr/local/bin/

# Copy the dnst  binary into this image from the specified Docker image.
# COPY --from=${DNST_IMAGE} /usr/local/bin/dnst /usr/local/bin/
# Commented out due to: https://github.com/moby/buildkit/issues/2034#issuecomment-834203964
COPY --from=nlnetlabs/dnst:latest /usr/local/bin/dnst /usr/local/bin/

# Install a configuration file
COPY ./etc/config.template.toml /etc/cascade/config.toml
RUN sed -i -e 's|127.0.0.1|0.0.0.0|g' /etc/cascade/config.toml

# Build variables for uid and guid of user to run container.
ARG RUN_USER=cascade
ARG RUN_USER_UID=1012
ARG RUN_USER_GID=1012

# Install required runtime dependencies.
RUN apk add --no-cache tini libgcc

RUN addgroup -g ${RUN_USER_GID} ${RUN_USER} && \
    adduser -D -u ${RUN_USER_UID} -G ${RUN_USER} ${RUN_USER}

# Create the data directory.
RUN mkdir -p /var/lib/cascade && \
    chown -R ${RUN_USER_UID}:${RUN_USER_GID} /var/lib/cascade

# Switch to our application user.
USER $RUN_USER_UID

# Hint to operators the TCP port that the application in this image listens on
# (by default).
EXPOSE 4539/tcp
EXPOSE 4540/tcp
EXPOSE 4540/udp
EXPOSE 4541/tcp
EXPOSE 4541/udp
EXPOSE 4542/tcp
EXPOSE 4542/udp

# Use Tini to ensure that our application responds to CTRL-C when run in the
# foreground without the Docker argument "--init" (which is actually another
# way of activating Tini, but cannot be enabled from inside the Docker image).
ENTRYPOINT ["/sbin/tini", "--", "cascaded"]

# Load a config file that doesn't exist, the operator running this image must
# mount a suitable config file at /etc/cascade/config.toml.
CMD ["--config", "/etc/cascade/config.toml"]

dnst/Dockerfile

# This is a multi-stage Dockerfile, with a selectable first stage. With this
# approach we get:
#
#   1. Separation of dependencies needed to build our app in the 'build' stage
#      and those needed to run our app in the 'final' stage, as we don't want
#      the build-time dependencies to be included in the final Docker image.
#
#   2. Support for either building our app for the architecture of the base
#      image using MODE=build (the default) or for externally built app
#      binaries (e.g. cross-compiled) using MODE=copy.
#
# In total there are four stages consisting of:
#   - Two possible first stages: 'build' or 'copy'.
#   - A special 'source' stage which selects either 'build' or 'copy' as the
#     source of binaries to be used by ...
#   - The 'final' stage.


###
### ARG DEFINITIONS ###########################################################
###

# This section defines arguments that can be overriden on the command line
# when invoking `docker build` using the argument form:
#
#   `--build-arg <ARGNAME>=<ARGVALUE>`.


# DNST_VERSION
# ============
# Which DNST Docker image should the `dnst` binary be copied from?
# Defaults to `nlnetlabs/dnst:latest`
# ARG DNST_IMAGE=nlnetlabs/dnst:latest
# Commented out due to: https://github.com/moby/buildkit/issues/2034#issuecomment-834203964

# MODE
# ====
# Supported values: build (default), copy
#
# By default this Dockerfile will build our app from sources. If the sources
# have already been (cross) compiled by some external process and you wish to
# use the resulting binaries from that process, then:
#
#   1. Create a directory on the host called 'dockerbin/$TARGETPLATFORM'
#      containing the already compiled app binaries (where $TARGETPLATFORM
#      is a special variable set by Docker BuiltKit).
#   2. Supply arguments `--build-arg MODE=copy` to `docker build`.
ARG MODE=build


# BASE_IMG
# ========
#
# Only used when MODE=build.
ARG BASE_IMG=alpine:3.24


# CARGO_ARGS
# ==========
#
# Only used when MODE=build.
#
# This ARG can be used to control the features enabled when compiling the app
# or other compilation settings as necessary.
ARG CARGO_ARGS


###
### BUILD STAGES ##############################################################
###


# -----------------------------------------------------------------------------
# Docker stage: build
# -----------------------------------------------------------------------------
#
# Builds our app binaries from sources.
FROM ${BASE_IMG} AS build
ARG CARGO_ARGS

RUN apk add --no-cache rust cargo openssl-dev

WORKDIR /tmp/build
COPY . .

# `CARGO_HTTP_MULTIPLEXING` forces Cargo to use HTTP/1.1 without pipelining
# instead of HTTP/2 with multiplexing. This seems to help with various
# "spurious network error" warnings when Cargo attempts to fetch from crates.io
# when building this image on Docker Hub and GitHub Actions build machines.
#
# `cargo install` is used instead of `cargo build` because it places just the
# binaries we need into a predictable output directory. We can't control this
# with arguments to cargo build as `--out-dir` is unstable and contentious and
# `--target-dir` still requires us to know which profile and target the
# binaries were built for. By using `cargo install` we can also avoid needing
# to hard-code the set of binary names to copy so that if we add or remove
# built binaries in future this will "just work". Note that `--root /tmp/out`
# actually causes the binaries to be placed in `/tmp/out/bin/`. `cargo install`
# will create the output directory for us.
RUN CARGO_HTTP_MULTIPLEXING=false cargo install \
  --locked \
  --path . \
  --root /tmp/out/ \
  ${CARGO_ARGS}


# -----------------------------------------------------------------------------
# Docker stage: copy
# -----------------------------------------------------------------------------
# Only used when MODE=copy.
#
# Copy binaries from the host directory 'dockerbin/$TARGETPLATFORM' directory
# into this build stage to the same predictable location that binaries would be
# in if MODE were 'build'.
#
# Requires that `docker build` be invoked with variable `DOCKER_BUILDKIT=1` set
# in the environment. This is necessary so that Docker will skip the unused
# 'build' stage and so that the magic $TARGETPLATFORM ARG will be set for us.
FROM ${BASE_IMG} AS copy
ARG TARGETPLATFORM
ONBUILD COPY dockerbin/$TARGETPLATFORM /tmp/out/bin/


# -----------------------------------------------------------------------------
# Docker stage: source
# -----------------------------------------------------------------------------
# This is a "magic" build stage that "labels" a chosen prior build stage as the
# one that the build stage after this one should copy application binaries
# from. It also causes the ONBUILD COPY command from the 'copy' stage to be run
# if needed. Finally, we ensure binaries have the executable flag set because
# when copied in from outside they may not have the flag set, especially if
# they were uploaded as a GH actions artifact then downloaded again which
# causes file permissions to be lost.
# See: https://github.com/actions/upload-artifact#permission-loss
FROM ${MODE} AS source
RUN chmod a+x /tmp/out/bin/*


# -----------------------------------------------------------------------------
# Docker stage: final
# -----------------------------------------------------------------------------
# Create an image containing just the binaries, configs & scripts needed to run
# our app, and not the things needed to build it.
#
# The previous build stage from which binaries are copied is controlled by the
# MODE ARG (see above).
FROM ${BASE_IMG} AS final

# Copy binaries from the 'source' build stage into the image we are building.
COPY --from=source /tmp/out/bin/* /usr/local/bin/

# Copy the dnst  binary into this image from the specified Docker image.
# COPY --from=${DNST_IMAGE} /usr/local/bin/dnst /usr/local/bin/
# Commented out due to: https://github.com/moby/buildkit/issues/2034#issuecomment-834203964
COPY --from=nlnetlabs/dnst:latest /usr/local/bin/dnst /usr/local/bin/

# Install a configuration file
COPY ./etc/config.template.toml /etc/cascade/config.toml
RUN sed -i -e 's|127.0.0.1|0.0.0.0|g' /etc/cascade/config.toml

# Build variables for uid and guid of user to run container.
ARG RUN_USER=cascade
ARG RUN_USER_UID=1012
ARG RUN_USER_GID=1012

# Install required runtime dependencies.
RUN apk add --no-cache tini libgcc

RUN addgroup -g ${RUN_USER_GID} ${RUN_USER} && \
    adduser -D -u ${RUN_USER_UID} -G ${RUN_USER} ${RUN_USER}

# Create the data directory.
RUN mkdir -p /var/lib/cascade && \
    chown -R ${RUN_USER_UID}:${RUN_USER_GID} /var/lib/cascade

# Switch to our application user.
USER $RUN_USER_UID

# Hint to operators the TCP port that the application in this image listens on
# (by default).
EXPOSE 4539/tcp
EXPOSE 4540/tcp
EXPOSE 4540/udp
EXPOSE 4541/tcp
EXPOSE 4541/udp
EXPOSE 4542/tcp
EXPOSE 4542/udp

# Use Tini to ensure that our application responds to CTRL-C when run in the
# foreground without the Docker argument "--init" (which is actually another
# way of activating Tini, but cannot be enabled from inside the Docker image).
ENTRYPOINT ["/sbin/tini", "--", "cascaded"]

# Load a config file that doesn't exist, the operator running this image must
# mount a suitable config file at /etc/cascade/config.toml.
CMD ["--config", "/etc/cascade/config.toml"]

cascade-hsm-bridge/Dockerfile

# This is a multi-stage Dockerfile, with a selectable first stage. With this
# approach we get:
#
#   1. Separation of dependencies needed to build our app in the 'build' stage
#      and those needed to run our app in the 'final' stage, as we don't want
#      the build-time dependencies to be included in the final Docker image.
#
#   2. Support for either building our app for the architecture of the base
#      image using MODE=build (the default) or for externally built app
#      binaries (e.g. cross-compiled) using MODE=copy.
#
# In total there are four stages consisting of:
#   - Two possible first stages: 'build' or 'copy'.
#   - A special 'source' stage which selects either 'build' or 'copy' as the
#     source of binaries to be used by ...
#   - The 'final' stage.


###
### ARG DEFINITIONS ###########################################################
###

# This section defines arguments that can be overriden on the command line
# when invoking `docker build` using the argument form:
#
#   `--build-arg <ARGNAME>=<ARGVALUE>`.


# MODE
# ====
# Supported values: build (default), copy
#
# By default this Dockerfile will build our app from sources. If the sources
# have already been (cross) compiled by some external process and you wish to
# use the resulting binaries from that process, then:
#
#   1. Create a directory on the host called 'dockerbin/$TARGETPLATFORM'
#      containing the already compiled app binaries (where $TARGETPLATFORM
#      is a special variable set by Docker BuiltKit).
#   2. Supply arguments `--build-arg MODE=copy` to `docker build`.
ARG MODE=build


# BASE_IMG
# ========
#
# Only used when MODE=build.
ARG BASE_IMG=alpine:3.24


# CARGO_ARGS
# ==========
#
# Only used when MODE=build.
#
# This ARG can be used to control the features enabled when compiling the app
# or other compilation settings as necessary.
ARG CARGO_ARGS


###
### BUILD STAGES ##############################################################
###


# -----------------------------------------------------------------------------
# Docker stage: build
# -----------------------------------------------------------------------------
#
# Builds our app binaries from sources.
FROM ${BASE_IMG} AS build
ARG CARGO_ARGS

RUN apk add --no-cache rust cargo

WORKDIR /tmp/build
COPY . .

# `CARGO_HTTP_MULTIPLEXING` forces Cargo to use HTTP/1.1 without pipelining
# instead of HTTP/2 with multiplexing. This seems to help with various
# "spurious network error" warnings when Cargo attempts to fetch from crates.io
# when building this image on Docker Hub and GitHub Actions build machines.
#
# `cargo install` is used instead of `cargo build` because it places just the
# binaries we need into a predictable output directory. We can't control this
# with arguments to cargo build as `--out-dir` is unstable and contentious and
# `--target-dir` still requires us to know which profile and target the
# binaries were built for. By using `cargo install` we can also avoid needing
# to hard-code the set of binary names to copy so that if we add or remove
# built binaries in future this will "just work". Note that `--root /tmp/out`
# actually causes the binaries to be placed in `/tmp/out/bin/`. `cargo install`
# will create the output directory for us.
RUN CARGO_HTTP_MULTIPLEXING=false cargo install \
  --locked \
  --path . \
  --root /tmp/out/ \
  ${CARGO_ARGS}


# -----------------------------------------------------------------------------
# Docker stage: copy
# -----------------------------------------------------------------------------
# Only used when MODE=copy.
#
# Copy binaries from the host directory 'dockerbin/$TARGETPLATFORM' directory
# into this build stage to the same predictable location that binaries would be
# in if MODE were 'build'.
#
# Requires that `docker build` be invoked with variable `DOCKER_BUILDKIT=1` set
# in the environment. This is necessary so that Docker will skip the unused
# 'build' stage and so that the magic $TARGETPLATFORM ARG will be set for us.
FROM ${BASE_IMG} AS copy
ARG TARGETPLATFORM
ONBUILD COPY dockerbin/$TARGETPLATFORM /tmp/out/bin/


# -----------------------------------------------------------------------------
# Docker stage: source
# -----------------------------------------------------------------------------
# This is a "magic" build stage that "labels" a chosen prior build stage as the
# one that the build stage after this one should copy application binaries
# from. It also causes the ONBUILD COPY command from the 'copy' stage to be run
# if needed. Finally, we ensure binaries have the executable flag set because
# when copied in from outside they may not have the flag set, especially if
# they were uploaded as a GH actions artifact then downloaded again which
# causes file permissions to be lost.
# See: https://github.com/actions/upload-artifact#permission-loss
FROM ${MODE} AS source
RUN chmod a+x /tmp/out/bin/*


# -----------------------------------------------------------------------------
# Docker stage: final
# -----------------------------------------------------------------------------
# Create an image containing just the binaries, configs & scripts needed to run
# our app, and not the things needed to build it.
#
# The previous build stage from which binaries are copied is controlled by the
# MODE ARG (see above).
FROM ${BASE_IMG} AS final

# Copy binaries from the 'source' build stage into the image we are building.
COPY --from=source /tmp/out/bin/* /usr/local/bin/

# Build variables for uid and guid of user to run container.
ARG RUN_USER=cascade
ARG RUN_USER_UID=1012
ARG RUN_USER_GID=1012

# Install required runtime dependencies.
RUN apk add --no-cache tini libgcc

RUN addgroup -g ${RUN_USER_GID} ${RUN_USER} && \
    adduser -D -u ${RUN_USER_UID} -G ${RUN_USER} ${RUN_USER}

# Switch to our application user.
USER $RUN_USER_UID

# Hint to operators the TCP port that the application in this image listens on
# (by default).
EXPOSE 5696/tcp

# Define ONBUILD commands that will be run when building a Docker image that
# uses this image as the FROM image to build upon.
# 
# Use Tini to ensure that our application responds to CTRL-C when run in the
# foreground without the Docker argument "--init" (which is actually another
# way of activating Tini, but cannot be enabled from inside the Docker image).
ONBUILD ENTRYPOINT ["/sbin/tini", "--", "cascade-hsm-bridge", "--config", "/etc/cascade-hsm-bridge/config.toml"]

# Load a config file that doesn't exist, the operator running this image must
# mount a suitable config file at /etc/cascade/config.toml and also mount into
# the image filesystem the PKCS#11 module to load and any additional files
# needed by the PKCS#11 module.
# ONBUILD CMD ["--config", "/etc/cascade-hsm-bridge/config.toml"]

# Define a dummy entrypoint that tells the user that this image is not
# intended to be run directly.
ENTRYPOINT ["/bin/echo", "This image must be used as the base of another image\nIgnored inputs:"]

cascade-hsm-bridge/docker/nethsm/

Dockerfile:

FROM nlnetlabs/cascade-hsm-bridge:latest

COPY docker/nethsm/config.toml /etc/cascade-hsm-bridge/
COPY docker/nethsm/p11nethsm.conf /etc/nitrokey/

WORKDIR /usr/lib64/pkcs11
ADD --chmod=o+r https://github.com/Nitrokey/nethsm-pkcs11/releases/download/v2.2.0/nethsm-pkcs11-v2.2.0-x86_64-linux-musl.so /usr/lib64/pkcs11/

USER cascade
ENV LOCALHSMPASS=blah
ENV LOCALDMINHSMPASS=blah2

config.toml:

version = "v1"

[pkcs11]
lib-path = "/usr/lib64/pkcs11/nethsm-pkcs11-v2.2.0-x86_64-linux-musl.so"

[server]
addr = "0.0.0.0:5696"

[daemon]
log-level = "info"
log-target = { type = "stderr" }
daemonize = false

p11nethsm.conf:

enable_set_attribute_value: false
log_level: "Trace"
slots:
  - label: DockerizedNetHSM
    description: NetHSM running in a Docker container
    operator:
      username: "operator"
      password: "0123456789"
    administrator:
      username: "admin"
      password: "0123456789"
    instances:
      - url: "https://keyfender:8443/api/v1"
        max_idle_connections: 10
        danger_insecure_cert: true
    certificate_format: PEM
    retries:
      count: 3
      delay_seconds: 1
    tcp_keepalive:
      time_seconds: 600
      interval_seconds: 60
      retries: 3
    connections_max_idle_duration: 1800
    timeout_seconds: 10
3

Also see these issues which I raised while working on this:

4

thx for your detailed first steps here. I been working on a nearly similar setup yesterday and you beat me to posting it. i will need to write a summary of my findings still and will do that as soon i am done with my development setup

1 Like
5

Hi @j.waibel,

I look forward to reading your summary and applying any lessons learned from it to our own thinking on this topic!

Ximon

6

This is my first small stack to test the components.

just use the scripts i added to bootstrap all of it, add or delete some zones.

1 Like
7

@marcgweg shared with me that there is also marcgweg/cascade - Codeberg.org which looks to me like a great resource to learn from! Thanks @marcgweg :heart: !