Headless homelab boot: Wake-on-LAN and power-loss recovery

Your headless box should come back without a treasure hunt for the power button. Start with AC-loss recovery, magic-packet Wake-on-LAN, a hardware watchdog and a panic timeout before shopping for remote-power gear.

Measured on our bench: boot timings on the Gigabyte B450M DS3H V2 (F62b, 2021-09-06) and Ryzen 5 5600G, with settings and kernel logs in the bench notes; recovery-hardware features below come from maker and Home Assistant docs.

Key takeaway: Firmware brings the box back after a power cut. Magic-packet WoL lets you wake it on demand. The hardware watchdog can reset a host that stops servicing it; the panic timeout restarts a crashed kernel. Skip any one layer and you’re back to walking over to press a button.

For your AM4 setup, plan installation and recovery display access before relying on headless boot. The 5600G versus 5600X platform guide explains integrated graphics and the board-specific boot checks.

Match every failure to the layer that fixes it

Each way a headless box dies needs its own fix. Nothing in software helps after a power cut, and firmware can’t see a kernel hang. Here is what our bench runs (bench notes):

Failure	Layer	Bench setting
AC power lost, then restored	Firmware	Power back on when AC returns
Box is off, you want it on	NIC / NetworkManager	WoL g (magic), persisted
Userspace or kernel hangs	systemd hardware watchdog	RuntimeWatchdogSec = 1 min
Kernel panic	sysctl	kernel.panic = 10
Box suspends itself	systemd targets	sleep and suspend masked

The front-panel power button on this bench isn’t even connected. That’s deliberate. If the box can’t come back without that button, the setup has failed.

Get the box back after a power cut with one firmware switch

Set the firmware to power on when AC returns, and do it first. It’s the only layer that works with the OS down. Our bench board has this enabled (bench notes).

We measured the cost of coming back with systemd-analyze-style boot phases. Firmware took 8.2 s, loader 1.0 s, kernel 2.7 s, initrd 2.1 s and userspace 10.6 s, for 24.6 s to fully up (bench notes). Firmware takes a third of that total. The board holds four mixed DIMMs (48 GB, all at 3200 MT/s with XMP); to investigate memory-training overhead, compare firmware time before and after a DIMM change.

For memory validation alongside these boot checks, use the Memtest86+ testing guide. Record its test settings and results alongside your boot and kernel logs.

Wake it remotely with magic-packet WoL that survives reboots

Use magic-packet WoL only, and persist it in NetworkManager. A one-off ethtool setting can be lost on reboot. Our bench NIC supports p/u/m/b/g, and we set g (bench notes).

NetworkManager lists phy, unicast, multicast, broadcast, arp and magic as allowed values. Magic means magic-packet wake (NetworkManager docs). We avoid the traffic-triggered modes because stray packets can wake the box.

sudo nmcli connection modify CONNECTION 802-3-ethernet.wake-on-lan magic
sudo ethtool IFACE | grep Wake-on

For utility interruptions, the UPS sizing and shutdown guide separates capacity, battery runtime, orderly halt and return-power behavior. Plan the shutdown and return-power sequence together.

Let the machine reboot itself after hangs and panics

For application failures, first distinguish a service restart from a host reset. The systemd restart and watchdog guide explains service heartbeats, hardware watchdogs and start limits; a responsive host can still have a failed application.

Turn on the hardware watchdog and a panic timeout. Out of the box, neither is set to save you.

RuntimeWatchdogSec drives the hardware watchdog and defaults to 0 (off). RebootWatchdogSec acts as a safety net so a requested reboot actually happens. Both settings do nothing without a hardware watchdog (systemd-system.conf). A positive kernel.panic reboots that many seconds after a panic, and a positive panic_on_oops escalates an oops to a panic (kernel sysctl docs).

# /etc/systemd/system.conf
RuntimeWatchdogSec=1min

# /etc/sysctl.d/90-panic.conf
kernel.panic = 10

sudo systemctl mask sleep.target suspend.target

Find the watchdog: Check ls /dev/watchdog* and your system logs. RuntimeWatchdogSec needs an available hardware watchdog; setting a timeout alone cannot create one.

Recovery estimate (illustrative): Add the 10 s panic timeout to our 24.6 s boot baseline: 34.6 s, or about 35 s. A full 60 s watchdog window plus that boot baseline is 84.6 s, or about 85 s. These are timing sums, not measured recovery runs; firmware, storage checks and guest startup can change your result.

The bench ran 3 days 13 hours under GPU and LLM load with zero machine-check or memory-error entries in the kernel log (bench notes). Use the tests below to exercise each recovery layer.

Recovery and energy budgeting answer different questions. If your box will remain powered continuously, use the monthly homelab electricity-cost guide to price measured outlet energy; its example wattages and workload hours are hypothetical.

Recovery hardware: a power switch, a remote screen and a wall meter

Buy for the failure you need to fix. A smart plug cuts electricity; a remote KVM lets you see what the machine is doing. The meter answers a third question: is the box drawing power at all?

Your problem	Pick	What it adds
A frozen host needs a last-resort power cycle	Shelly Plug US Gen4	Local switching and power readings through Home Assistant’s Shelly integration
You need the firmware or boot screen while the OS is down	GL.iNet Comet GL-RM1	Browser keyboard/video access over HDMI, USB and Ethernet
You want a standalone outlet reading	P3 Kill A Watt P4400	Watts and accumulated kWh on its own LCD

Shelly Plug US Gen4: the last-resort power switch

Choose the Shelly Plug US Gen4 when you need local outlet control. Home Assistant’s Shelly integration talks directly to the device without a cloud connection; the Gen4 maker guide also documents power metering and local actions. Our smart-plug guide explains the integration choices.

Try an orderly shutdown or restart first. Cutting power to a running server can lose active writes or corrupt application data, so use the outlet switch only when the host is stuck. Enable AC-loss power-on first, or switching electricity back on may leave you with a perfectly powered, perfectly off computer.

Keep the plug’s control path independent of the server you’re switching: if Home Assistant runs on that server, it disappears when you cut power. Keep your router and recovery tools powered too. With a UPS in the chain, switching the UPS input off only starts its battery; it doesn’t power-cycle the server. Use the plug within the maker’s load limits and plan the UPS shutdown sequence separately.

Comet GL-RM1: see the boot screen instead of guessing

Choose the GL.iNet Comet GL-RM1 when remote desktop is too late in the boot process. Its setup guide connects the host’s HDMI output and USB input to the Comet, then gives the Comet its own power and wired Ethernet. You control the console in a browser, separately from the host OS. Make sure the connected video output shows your firmware screen before relying on it for recovery.

Keyboard and video access aren’t the power button. GL.iNet’s optional ATX board adds power-button operations through a compatible motherboard front-panel header; plan it as a separate accessory, with the correct wiring for your board. A sealed mini PC may lack that header. Power the Comet independently of the switched host, and use a private remote-access route rather than publishing its management interface. The setup guide includes Tailscale as one option.

P4400: read watts at the wall

The Kill A Watt P4400 is the simple pick when you want an outlet reading without another integration. P3 lists watts and cumulative kWh on its local display. Record a normal idle reading, an active workload and the halted state so you have something useful to compare when recovery gets weird. The plug-in power-meter guide explains the meter choices and measurements.

Prove it with this recovery checklist before you walk away

1. Firmware: AC-loss restore is enabled. Test: shut down cleanly, disconnect AC, restore power, and time the box to fully up. This checks return-power behavior without interrupting active writes.
2. WoL: ethtool shows Wake-on: g after a reboot. Test: power off, send a magic packet, and confirm it boots.
3. Watchdog: /dev/watchdog* exists and RuntimeWatchdogSec is set. Check with systemctl show -p RuntimeWatchdogUSec.
4. Panic: sysctl kernel.panic returns 10. Test: echo c | sudo tee /proc/sysrq-trigger on a box with nothing important running, then time the return.
5. Sleep: systemctl status sleep.target suspend.target shows both masked.
6. Baseline: record your own boot phase times so you know what “normal” looks like.

Tip: Run step 4 last. It hard-crashes the kernel, so sync or stop anything writing to disk first.

A recovered physical host is only part of an application restore plan. The Proxmox versus direct Docker guide helps you plan host recovery, guest startup and application data coverage.

Boot recovery does not restore lost files or application data. The NAS backup decision guide explains which recovery layers need independent copies and how to test retrieval without overwriting the working source.

Plan access separately from boot recovery

A system recovering after a power interruption still needs an explicit access-control design. Our Tailscale vs Cloudflare Tunnel decision guide helps you choose private clients or protected browser access and understand which provider you trust. Taking the recovery kit on the road? Our Beryl travel-router comparison covers the travel-network side.

Before assigning a host display GPU to a guest, the GPU passthrough requirements guide explains why an independent, verified host recovery route belongs ahead of boot and driver changes.

Choosing the screen-and-keyboard recovery path? Our Comet vs JetKVM vs PiKVM guide compares video capture, power-control accessories, ISO mounting and local access.

Keep a boot-tested USB rescue and installer kit beside your console tools so recovery media is ready when the boot drive is not.