Logging and Debugging

WendyOS uses systemd-journald for log management. All services write to the journal via stdout/stderr or logger. This document covers how journald is configured, how to enable persistent storage, and how to access logs.

Default behaviour

By default, journal logs are stored in volatile memory (/run/log/journal) and lost on reboot. The default for a given board tracks WENDYOS_MENDER: targets without a Mender-managed /data partition (QEMU, RPi, Thor Phase 1) use volatile logs; Orin (tegra234) uses persistent logs.

Persistent journal option

To persist logs across reboots, set WENDYOS_PERSIST_JOURNAL_LOGS=1 in your build configuration (local.conf or the distro conf). This causes two extra files to be installed:

/usr/lib/systemd/journald.conf.d/10-wendyos-persistent.conf

[Journal]
Storage=persistent
SystemMaxUse=512M
SystemKeepFree=1G
RuntimeMaxUse=64M
MaxRetentionSec=1month
Compress=yes
ForwardToSyslog=yes

/lib/systemd/system/var-log.mount

Bind-mounts /data/log onto /var/log before systemd-journald.service starts:

[Mount]
What=/data/log
Where=/var/log
Type=none
Options=bind,x-systemd.mkdir

The x-systemd.mkdir option automatically creates /data/log if it does not exist. The unit runs after data.mount and mender-systemd-growfs-data.service (if present) to ensure the data partition is fully expanded before the bind mount is established.

When Storage=persistent is set, journald creates /var/log/journal/<machine-id>/ and writes binary journal files there. These survive reboots and OTA updates (the /data partition is outside the update boundary on Tegra machines).

Enabling persistent logging at runtime (without a rebuild)

On a running device you can force persistence without a full image rebuild:

mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal
systemctl restart systemd-journald

This only persists until the next OTA update that does not include the persistent-logging configuration.

Accessing logs

Follow a service in real time

journalctl -fu wendyos-agent.service
journalctl -fu gadget-setup.service
journalctl -fu wendyos-etc-binds.service

View logs since last boot

journalctl -b

View logs from a previous boot (requires persistent storage)

# List available boots
journalctl --list-boots

# Show logs from boot index -1 (previous boot)
journalctl -b -1

Filter by systemd identifier (syslog tag)

journalctl -t wendy-agent-updater
journalctl -t rpi5-eeprom-update
journalctl -t gadget-setup.sh

View all WendyOS service logs together

journalctl -u "wendyos-*" -u gadget-setup.service

Check journal disk usage

journalctl --disk-usage

Vacuum old logs

# Keep only the last 7 days
journalctl --vacuum-time=7d

# Keep only 200 MB
journalctl --vacuum-size=200M

Log storage locations

Individual service log identifiers

Kernel command line and console

By default, WendyOS configures two kernel consoles:

• console=serial0,115200 — hardware UART (see uart.md)
• console=tty1 — HDMI framebuffer

Kernel messages and early boot output appear on both. Systemd redirects unit output to the journal after early boot.

To see kernel messages in the journal:

journalctl -k          # current boot
journalctl -k -b -1    # previous boot