Configuring and Verifying Rust‑Based conmon v3 with Podman

By the end of this guide, readers will be able to install the Rust‑implemented conmon v3 monitor, configure Podman to use it system‑wide or per‑user, verify the active monitor version, and understand the operational differences and trade‑offs compared with the previous C implementation.

Understanding conmon and Its Role in Podman

Podman delegates the low‑level container lifecycle to an OCI runtime such as crun or runc. Once the runtime has started the container process, a small helper called conmon remains attached to that process. Conmon’s responsibilities include:

• Relaying standard input, output, and error streams between the container and the invoking Podman command.
• Providing the endpoint for podman attach operations.
• Collecting and forwarding container logs to the configured logging driver.
• Recording the container’s exit status so that Podman can report it to the user.

Historically conmon was written in C (conmon v2). The new conmon v3 replaces that codebase with Rust, aiming to eliminate memory‑safety bugs common in low‑level C programs. Additionally, the Rust version introduces a unified logging‑backend interface, simplifying future extensions and testing of log drivers.

Installing conmon v3

The Rust implementation is packaged as conmon-v3. Installation uses the system package manager; on Fedora the command is:

sudo dnf install conmon-v3

This command adds the /usr/bin/conmon-v3 binary to the system and registers the package in the RPM database. To revert the installation, the same package manager can be used to remove the package:

sudo dnf remove conmon-v3

System‑Wide Configuration

Podman reads its configuration from /etc/containers/containers.conf and any drop‑in files located in /etc/containers/containers.conf.d/. To make conmon v3 the default monitor for all users, create a drop‑in file with the following content:

# /etc/containers/containers.conf.d/99-conmon-v3.conf
[engine]
conmon_path = ["/usr/bin/conmon-v3"]

The conmon_path key expects a JSON‑style array; the array allows future extensions for multiple monitors, but a single entry is sufficient for now. After creating the file, Podman will resolve the monitor path during the next container start. Existing containers continue to use the monitor that was active when they were launched; they are not retroactively switched.

To undo the change, simply delete the drop‑in file:

sudo rm /etc/containers/containers.conf.d/99-conmon-v3.conf

Podman will fall back to the monitor defined in the main configuration file or the compiled default, which is typically the C‑based conmon binary.

Per‑User Override

Individual users may prefer a different monitor without affecting the system default. User‑specific configuration resides in $HOME/.config/containers/containers.conf (or the containers.conf.d directory). To select conmon v3 for a single user, create the following drop‑in:

# $HOME/.config/containers/containers.conf.d/99-conmon-v3.conf
[engine]
conmon_path = ["/usr/bin/conmon-v3"]

The precedence order is: user drop‑ins → user main file → system drop‑ins → system main file. Removing the user drop‑in restores the previous behavior for that account.

Verifying the Active Monitor

Podman provides the podman info command, which reports the path and version of the monitor currently in use. Run the command as a regular user to see the user‑level configuration, and with sudo to inspect the rootful configuration:

podman info --format json | jq '.store.conmon.path, .store.conmon.version'

The expected output should show /usr/bin/conmon-v3 as the path and a version string beginning with “3”. For rootful verification:

sudo podman info --format json | jq '.store.conmon.path, .store.conmon.version'

If the path points to the older binary (commonly /usr/bin/conmon) or the version does not start with “3”, the configuration has not been applied correctly. Double‑check the drop‑in file syntax and ensure the file is readable by the Podman process (typically owned by root with mode 0644).

Testing Conmon v3 Behavior

After confirming that Podman is using conmon v3, start fresh containers to exercise the new monitor. The following checklist helps identify regressions or differences relative to conmon v2:

• Hangs or missing output: Run a container that produces continuous stdout (e.g., podman run --rm alpine sh -c "while true; do echo tick; sleep 1; done") and attach to it. Verify that output appears promptly and that Ctrl+C terminates cleanly.
• Incorrect exit status: Execute a container that exits with a non‑zero code (e.g., podman run --rm alpine false) and check that podman ps -a reports the same exit code.
• CPU usage spikes: Monitor the host’s CPU while a container runs a CPU‑intensive workload (e.g., podman run --rm alpine stress --cpu 2 --timeout 30s). Compare the usage to a baseline run with conmon v2 to spot abnormal consumption.
• Logging consistency: Enable a logging driver (e.g., journald) and confirm that logs appear in the expected destination without duplication or loss.
• Interactive sessions: Start an interactive shell (podman run -it --rm alpine sh) and test attach/detach cycles. Ensure that the session remains stable after multiple attach operations.

When a discrepancy is observed, capture the following data for reporting:

# Detailed Podman debug output
podman info --debug

# System journal entries for the container
journalctl -u podman -b | grep 

The collected logs, along with the Fedora version, package versions, and a concise reproduction script, constitute a useful bug report for the conmon v3 issue tracker.

Trade‑offs Between conmon v2 and v3

Switching to the Rust implementation offers clear safety benefits. Rust’s ownership model eliminates many classes of memory errors (buffer overflows, use‑after‑free, null dereferences) that have historically required patches in the C codebase. This reduces the attack surface for privilege‑escalation exploits that target the monitor process.

The unified logging backend also simplifies the addition of new log drivers. Projects that need custom log handling can now implement a single interface rather than adapting multiple code paths.

However, administrators should be aware of the following considerations:

• Binary size and start‑up latency: The Rust binary includes the standard library and may be slightly larger than the C counterpart. In most workloads the difference is negligible, but environments with extremely tight storage constraints should verify the footprint.
• Maturity of the codebase: Conmon v3 is newer and may still have edge‑case bugs not yet discovered. Maintaining the ability to revert to v2 (by removing the drop‑in) ensures continuity during troubleshooting.
• Compatibility with third‑party tools: Some external tools parse the monitor path from podman info and may have assumptions about the binary name. Verify that any automation scripts handle the new path gracefully.

Overall, the safety improvements outweigh the modest trade‑offs for most production and development environments. Administrators are encouraged to adopt conmon v3 on a test machine or VM before rolling it out fleet‑wide, using the verification steps described earlier.

Rolling Back to the Previous Monitor

If testing reveals incompatibilities, restoring the original monitor is straightforward. Remove the configuration file that points to conmon v3, then restart any affected containers so that they are launched with the default monitor. The steps are:

# System‑wide revert
sudo rm /etc/containers/containers.conf.d/99-conmon-v3.conf

# User‑specific revert (if applicable)
rm $HOME/.config/containers/containers.conf.d/99-conmon-v3.conf

After removal, new containers will automatically use the monitor defined by the remaining configuration (typically the C‑based conmon). Existing containers continue with the monitor they were started with; stopping and restarting them applies the change.

Source: fedoramagazine.org

Deploying Navidrome with Podman on Fedora – A Step‑by‑Step Guide

By the end of this article the reader will be able to install, configure, and run the Navidrome music server on a Fedora system using Podman, with SELinux contexts correctly applied and the service managed through podman‑compose.

Understanding Navidrome and Its Container Model

Navidrome is an open‑source music‑streaming server that serves a personal audio collection over HTTP. It is distributed as a set of multi‑architecture container images that include all runtime dependencies, eliminating the need for a host‑level package manager. The images are published under the deluan/navidrome repository and support the common architectures linux/amd64, linux/arm64, linux/arm/v6, and linux/arm/v7, making the same deployment process applicable to x86 servers as well as ARM‑based devices such as Raspberry Pi.

Podman is a daemonless container engine that implements the OCI container runtime specifications. It offers Docker‑compatible command‑line syntax while providing rootless execution and tighter integration with SELinux on Fedora. Because Podman does not require a central daemon, containers can be launched directly from a user session, and the podman unshare utility can be used to perform file‑ownership changes inside the user namespace, preserving the principle of least privilege.

Preparing the Host Environment

The first step is to ensure that the host system has the required tools installed. Fedora includes Podman in its default repositories, and the optional podman-compose package provides a Docker‑Compose‑compatible front‑end.

Install the necessary packages with the following command, which adds the Podman binaries and the compose helper:

sudo dnf install -y podman podman-compose

This command installs the podman engine, the podman-compose script, and any required SELinux policy modules. The installation can be reversed by removing the packages with sudo dnf remove podman podman-compose.

Creating the Directory Structure

Navidrome expects two persistent volumes: one for its internal database and configuration (/data) and one for the music collection (/music). Create a dedicated directory hierarchy in the user’s home directory to hold these files.

mkdir -p /home/nico/navidrome/{Data,Music}

The command creates Data and Music sub‑directories under /home/nico/navidrome. Ownership and SELinux labeling will be addressed in the next steps.

Applying SELinux Contexts

When SELinux is enforcing (the default on Fedora), container processes are confined by the container_file_t type. To allow the Navidrome container to read and write its data directories, the host directories must be labeled accordingly.

Add the SELinux file context for each directory and its contents:

sudo semanage fcontext -a -t container_file_t '/home/nico/navidrome/Data(/.*)?'
sudo semanage fcontext -a -t container_file_t '/home/nico/navidrome/Music(/.*)?'

These commands register a persistent mapping that tells SELinux to treat any file under the specified paths as container_file_t. Apply the mapping immediately with:

sudo restorecon -R /home/nico/navidrome/Data
sudo restorecon -R /home/nico/navidrome/Music

If later adjustments are required, the semanage fcontext -d command can delete the rule, and restorecon can be re‑run to revert the labeling.

Setting Correct Ownership

Navidrome runs inside the container as a non‑root user with UID 1000 and GID 1000. To avoid permission errors, the host directories must be owned by the same identifiers. Because the host user may have a different UID, the podman unshare command performs the ownership change inside a user namespace, ensuring the container sees the correct ownership without requiring root privileges on the host filesystem.

podman unshare chown 1000:1000 /home/nico/navidrome/Data
podman unshare chown 1000:1000 /home/nico/navidrome/Music

These commands change the owner and group of the two directories to UID 1000/GID 1000. To revert, run the same commands with the original UID/GID values.

Defining the Container with podman‑compose

Using podman-compose simplifies multi‑container orchestration and mirrors the familiar Docker‑Compose workflow. Create a podman-compose.yml file in the same directory as the data volumes.

cat > podman-compose.yml <<'EOF'
version: "3"
services:
  navidrome:
    image: deluan/navidrome:latest
    user: 1000:1000
    ports:
      - "4533:4533"
    restart: unless-stopped
    environment:
      ND_SCANSCHEDULE: 1h
      ND_LOGLEVEL: info
      ND_SESSIONTIMEOUT: 24h
      ND_BASEURL: ""
    volumes:
      - "/home/nico/navidrome/Data:/data"
      - "/home/nico/navidrome/Music:/music:ro"
EOF

The file declares a single service named navidrome. Key elements:

• image: pulls the latest official Navidrome image from Docker Hub.
• user: runs the container with UID/GID 1000, matching the host directory ownership.
• ports: maps host port 4533 to the container’s HTTP port.
• restart: ensures the container restarts automatically unless stopped manually.
• environment: optional variables that control scan frequency, log level, session timeout, and base URL. The values shown are the defaults; they can be omitted or customized as needed.
• volumes: mounts the persistent data directory as read/write (/data) and the music collection as read‑only (/music:ro) to prevent accidental modification from within the container.

Launching the Service

Start the Navidrome container in detached mode with:

podman-compose up -d

Podman pulls the image if it is not already cached, creates the container, and starts it. The -d flag runs the container in the background. To stop and remove the service, use:

podman-compose down

The command stops the container and removes the associated pod and network resources, leaving the host‑side data directories untouched.

Alternative One‑Shot Run Command

For environments where a compose file is unnecessary, the same container can be launched directly with a single podman run invocation. This approach is useful for quick tests or when integrating Navidrome into an existing script.

podman run -d \
  --name navidrome \
  --restart=unless-stopped \
  --user $(id -u):$(id -g) \
  -v /home/nico/navidrome/Music:/music:ro \
  -v /home/nico/navidrome/Data:/data \
  -p 4533:4533 \
  -e ND_LOGLEVEL=info \
  deluan/navidrome:latest

Explanation of the flags:

• -d: run in detached mode.
• --name: assign a stable container name for later management.
• --restart=unless-stopped: replicate the compose restart policy.
• --user $(id -u):$(id -g): dynamically set the UID/GID to match the invoking user, ensuring correct file permissions.
• -v: bind‑mount the host directories; the music mount is read‑only.
• -p: expose port 4533 on the host.
• -e: set environment variables; only ND_LOGLEVEL is shown as an example.

To stop this container, execute podman stop navidrome followed by podman rm navidrome. The data stored in the host directories remains intact.

Verification and Routine Operations

After the container is running, verify that Navidrome is reachable by requesting the HTTP endpoint:

curl -s http://localhost:4533/ | grep -i navidrome

A successful response contains the Navidrome HTML title, confirming that the service is listening on the expected port. For a more detailed health check, inspect the container logs:

podman logs navidrome

Typical log entries include the startup banner, the configured scan schedule, and any errors related to accessing the music directory. Because the music mount is read‑only, permission‑denied messages indicate an SELinux labeling issue; re‑run restorecon if necessary.

To trigger a manual library rescan without restarting the container, set the ND_SCANSCHEDULE environment variable to 0 and send a SIGHUP to the process. This can be done via podman exec:

podman exec navidrome pkill -HUP navidrome

The command sends a hang‑up signal to the Navidrome process, causing it to re‑read the music directory. The operation does not modify container configuration and can be reversed by sending a regular SIGTERM to stop the container if needed.

Trade‑offs and Operational Considerations

Deploying Navidrome with Podman on Fedora offers several advantages:

• Rootless operation: By running containers as an unprivileged user, the attack surface is reduced compared with traditional Docker daemons.
• SELinux integration: Proper labeling ensures that the container cannot escape its confined file‑system view, complying with Fedora’s default security posture.
• Architecture portability: The same compose file works on both x86_64 and ARM platforms, simplifying management of heterogeneous hardware.

However, there are practical trade‑offs to consider:

• Performance of bind mounts: Accessing a large music library over a bind mount may incur additional overhead compared with using a dedicated volume driver. Monitoring I/O latency on the host can reveal bottlenecks.
• Backup strategy: The /data directory contains the Navidrome SQLite database and configuration files. Regular backups of /home/nico/navidrome/Data are required to protect against data loss, as container restarts do not affect the host‑side files.
• Network exposure: Exposing port 4533 directly to the host makes the service reachable on any network interface. In production environments, consider placing a reverse proxy (e.g., Nginx) in front of Navidrome to enforce TLS and add authentication layers.
• Resource limits: Podman does not impose default CPU or memory limits. For high‑traffic deployments, define explicit --cpus and --memory constraints in the compose file to prevent a single container from exhausting host resources.

By following the steps outlined above, system administrators can maintain a reproducible, SELinux‑compliant Navidrome deployment that remains functional across hardware revisions and Fedora updates for the foreseeable future.

Source: fedoramagazine.org

How to Run Rootless Containers on Ubuntu 24.04 with Podman and systemd Quadlet (Auto-Updates Included)

Overview

Running containers without root is a big security win, and Ubuntu 24.04 LTS makes it easy with Podman and systemd Quadlet. Podman is a Docker-compatible container engine that works without a daemon, and Quadlet lets you define containers as systemd units for reliable startup, health checks, logging, and auto-updates. In this tutorial, you will set up a rootless container managed by systemd, enable automatic image updates, and learn how to monitor and troubleshoot the service.

Prerequisites

You will need an Ubuntu 24.04 LTS system, a regular user with sudo rights, outbound internet access to pull images, and an open application port above 1024 (rootless containers cannot bind to privileged ports). This guide assumes your username is $USER and that you will host a simple HTTP service on port 8080.

Step 1: Install Podman

Ubuntu 24.04 includes a recent Podman package. Install it via APT and verify the version:

sudo apt update
sudo apt install -y podman
podman --version
podman info

If podman info shows cgroup version 2 and a working network backend, you are good to go. No system-wide daemon is needed.

Step 2: Enable user lingering for systemd

To let your user services keep running after logout, enable lingering and confirm that the user systemd instance is active:

sudo loginctl enable-linger "$USER"
systemctl --user status

If the second command shows a status page, the per-user systemd is ready. Otherwise, log out and back in, or start it by running any user service.

Step 3: Create a Quadlet unit for a sample web app

Quadlet reads unit files from ~/.config/containers/systemd/ and generates corresponding systemd services. Create the directory and a container unit that runs a tiny HTTP server (traefik/whoami) on port 8080:

mkdir -p ~/.config/containers/systemd

cat > ~/.config/containers/systemd/whoami.container <<'EOF'
[Unit]
Description=Rootless whoami demo (Podman + Quadlet)
After=network-online.target
Wants=network-online.target

[Container]
# Image to run
Image=traefik/whoami:latest
# Name the container consistently
ContainerName=whoami
# Map host port 8080 to container port 80
PublishPort=8080:80
# Persist simple state/logs if needed
Volume=%h/containers/whoami:/data:Z
# Health check for systemd readiness
HealthCmd=curl -fsS http://127.0.0.1:80/ || exit 1
HealthInterval=30s
# Timezone for logs
Environment=TZ=UTC
# Auto update from registry when new image is available
AutoUpdate=registry
# Pass any extra runtime args if needed
# ContainerRuntimeArguments=--pids-limit=256

[Service]
# Restart on failure
Restart=on-failure
RestartSec=3

[Install]
WantedBy=default.target
EOF

The AutoUpdate=registry directive is the Quadlet-native way to enable image updates. The PublishPort setting maps host port 8080 to the container’s port 80. The HealthCmd allows systemd to consider the service healthy only after the endpoint responds.

Step 4: Generate and start the systemd service

Reload the user systemd manager to pick up the new Quadlet file, then enable and start the service:

systemctl --user daemon-reload
systemctl --user enable --now whoami.service
systemctl --user status whoami.service

You should now see the service active. Verify the container is running and accessible:

podman ps
curl -s http://127.0.0.1:8080/

The curl output returns headers that include your container’s hostname, confirming the service is up.

Step 5: Enable automatic image updates

Podman provides a built-in timer to auto-update containers based on Quadlet’s AutoUpdate=registry or the io.containers.autoupdate label. Enable the timer:

systemctl --user enable --now podman-auto-update.timer
systemctl --user list-timers | grep podman-auto-update

You can test an update cycle without applying it:

podman auto-update --dry-run

When a new image is available, Podman will pull it and restart only the affected containers, minimizing downtime.

Step 6: Logs, troubleshooting, and lifecycle

To review live logs and diagnose issues, use journalctl and Podman’s own logs. Here are the most useful commands:

journalctl --user-unit whoami.service -f
podman logs -f whoami
systemctl --user restart whoami.service
systemctl --user stop whoami.service

If you change the .container file, always run systemctl --user daemon-reload and then systemctl --user restart whoami.service to apply changes. To remove the service completely, stop and disable it, delete the Quadlet file, and reload:

systemctl --user disable --now whoami.service
rm ~/.config/containers/systemd/whoami.container
systemctl --user daemon-reload
podman rm -f whoami

Step 7: Exposing the service on your network

If you plan to expose the service beyond localhost, open the firewall and bind to the host’s address. With UFW, allow port 8080:

sudo ufw allow 8080/tcp

For production, consider putting this behind a reverse proxy such as Caddy, Nginx, or Traefik for HTTPS termination and rate limiting. Rootless containers cannot bind to ports below 1024, so use a reverse proxy on 443/80 or use auth/stream termination upstream.

Step 8: Tips for production use

Use bind mounts or volumes for persistent data, set resource limits with ContainerRuntimeArguments (CPU/memory/pids), and prefer pinned tags or digests for image stability. For private registries, log in with podman login <registry> and consider CredentialHelpers. Finally, keep Ubuntu patched (using unattended-upgrades), and monitor your services with systemd health checks and alerts.

Conclusion

You have created a secure, rootless container on Ubuntu 24.04 using Podman and systemd Quadlet, enabled automatic image updates, and learned how to manage the service lifecycle with systemd. This pattern makes container workloads feel like first-class system services, while keeping your host safer by avoiding root privileges.