systemd service restarts vs hardware watchdogs: which recovery layer?

One stuck app should not send the whole box back through its boot screen. Restart a failed process, use a service watchdog for missed application heartbeats, and keep the hardware watchdog for a host that stops feeding its watchdog device.

This guide follows systemd’s manuals checked on October 19, 2026; use your distribution’s matching version when applying settings.

Choose the smallest recovery scope

Start with the symptom you can actually observe. A process exiting, a missed notification and an incorrect application response are different events.

Failure observation	Candidate layer	Important boundary
Process exits with a failure	Service Restart= policy	Does not diagnose a running but stuck process
Application stops sending watchdog notifications	Service WatchdogSec= plus Restart=	Application must support meaningful notifications
Host stops feeding the watchdog device	Manager RuntimeWatchdogSec=	Requires a working hardware watchdog
Application gives wrong answers while heartbeats continue	Application-level health check	Neither heartbeat proves useful work

Define a successful application operation alongside the recovery action, so you can tell when useful work resumes.

For power restoration and boot planning, read the headless-boot guide. Use its boot baseline when you time your full application recovery.

Restart a failed process

The service manual recommends Restart=on-failure for long-running services. It covers non-zero exits, unclean signals, operation timeouts and configured watchdog failures, subject to documented overrides and start limits. A clean exit is different; Restart=always also covers clean exits but does not undo an explicit stop. It is rejected for Type=oneshot.

RestartSec= is the delay before a restart, not a recovery deadline. Restart= belongs in the service unit’s [Service] section. Neither setting alone identifies a process that remains alive while doing no useful work.

Before changing policy, record the effective unit and its overrides. Ask whether the program is supposed to exit successfully, whether restarting it is safe for its state, and what evidence you need to retain when it fails. Avoid turning an expected completion into an accidental restart loop.

Detect missed application heartbeats

WatchdogSec= in [Service] requires the application to send regular sd_notify notifications containing WATCHDOG=1 after startup. Missing the interval fails and terminates the service; a suitable restart policy can then restart it, as documented in the service watchdog section.

Adding the directive does not teach an arbitrary program to send meaningful heartbeats. Confirm application support before enabling it. A heartbeat can continue while the work a reader cares about is broken, so define a separate correctness check. Test the operation you actually need the application to perform.

Use a hardware watchdog for host recovery

RuntimeWatchdogSec= is a manager setting, separate from the service setting. It programs watchdog hardware to reboot the host if contact stops. It defaults to off and has no effect without a hardware watchdog. Supported timeout values depend on the device; systemd may select the closest available value. Use the hardware-watchdog requirements to identify a working driver and device on your board.

An application can fail while the system manager continues feeding that device. Host watchdog activity therefore cannot stand in for application supervision. RebootWatchdogSec= is different again: it protects the later reboot phase after regular services have terminated, rather than watching a running application.

Before enabling host resets, identify the driver and device, check effective settings, and plan access for recovery. Then test the effective timeout with your recovery path ready.

Keep start limits visible

StartLimitIntervalSec= and StartLimitBurst= belong in [Unit]. They limit starts within an interval, including manual starts. Once the limit is hit, automatic restarts stop. Interval expiry alone does not initiate another attempt; a later manual, timer or socket start can reactivate restart logic. systemctl reset-failed clears the rate counter, according to the unit manual.

If retries stop, investigate the original error and the limit together. Clearing a counter is not a repair. Preserve the earlier failure record before changing policy, and decide how a stopped retry loop will become visible to an operator. Do not disable limits merely to hide a recurring fault.

Plan and record separate tests

With expendable data and recovery access ready, validate process failure separately from missed notifications. Only plan the notification test for an application that supports them. Treat host-watchdog validation as a separate disruptive exercise with a recovery path and a plan for application data.

Record the distribution, installed systemd version, effective unit and overrides, restart policy, delay and start limits. For a service watchdog, record notification support and what it represents. For hardware, record the driver, device and effective timeout.

Keep failure detection, restart or reboot, application readiness, and a successful application operation as separate observations. Time the whole sequence to see where the outage actually spends its time. Preserve diagnostics and check rollback as part of the plan. If the same fault returns, investigate it before adding more retries.

For a virtualised host, the Proxmox versus direct Docker guide adds the guest and application restore boundaries. Test guest startup and application readiness as well as host recovery.