# Server deployment and recovery

The mediaserver's `update` and `update2` aliases, the router UI Apply action, and the 05:15 nightly update now use `fred-deploy`. The clients' build/copy/switch workflow and the migration backup script are unchanged. This change has not been activated on any machine.

## Transaction

Each deployment runs in its own transient systemd service. Losing the terminal or router UI apply process does not terminate that service. Logs go to the journal; the calling command prints the unit to follow and collects its journal after completion.

One lock, `/run/lock/fred-deploy.lock`, covers the entire server build, activation, verification, commit and recovery. Other deployments queue behind it. The WAN watchdog takes the same lock nonblockingly and skips its cycle when the lock is held, including conntrack flushes and network restarts.

The transaction captures the **running system** and **system-profile boot default** separately and retains both, the candidate and its own recovery code as GC roots. It builds exactly one candidate; committing never re-fetches a moving branch. A candidate must contain the boot recovery service.

After the build succeeds, a separate ten-minute systemd timer is armed. The candidate is activated with `switch-to-configuration test`. Essential router checks must pass before the system profile is changed and `switch-to-configuration boot` installs the new boot default. A final health check precedes the durable `committed` result and timer cancellation.

Activation, health-check or boot-commit errors restore the captured boot profile, reinstall its boot configuration, and reactivate the captured running system. Recovery attempts runtime restoration even if restoring the bootloader fails. Failed recovery retains the roots and records `rollback-failed`; it is never reported as a successful rollback. New deployments refuse to overwrite an unresolved transaction.

The independent timer stops a stuck deployment worker before taking the lock, so it does not deadlock behind that worker. Stale timers cannot undo a committed or newer transaction. Subprocess timeouts kill their process groups. Durable state and GC-root directory changes are fsynced.

Transient timers disappear on reboot. The boot recovery service checks for an unfinished transaction from a previous boot and runs the pinned recovery code. It uses `Type=exec` so its activation does not block completion of the boot targets that the switch script starts. A deployment arriving before boot recovery takes the lock recovers the old transaction first. Same-boot activation of this service does not cancel the deployment installing it.

## Health checks and limits

Checks cover networkd, dnsmasq, AdGuard Home, nftables, SSH, nginx and the router UI; the LAN address and WAN default route; IPv4 forwarding; default-drop input/forward chains; DNAT and masquerade rules; WAN ping through either of two targets; an actual LAN DNS lookup and SSH banner; the local UI HTTP endpoint and its TLS management proxy.

Media applications are not health probes. A nonzero Nix activation exit still causes rollback, including activation errors originating outside the router services. An ISP, public DNS or certificate outage can also fail verification and cause a conservative rollback. The local probes do not prove reachability from every external client.

A build is limited to 90 minutes; individual activation and bootloader commands to two minutes. Health checks retry for 90 seconds, with each probe also bounded. The timer's ten-minute deadline starts after building. The supervised deployment has a 100-minute runtime limit, including lock waits. Recovery has its own runtime limit.

The nightly job retains the existing conditional reboot behavior: after a successful transaction, it requests a reboot if the booted kernel, initrd or kernel modules differ from the candidate. This is **not** an automatic post-reboot kernel validation/boot-fallback system. Recovery cannot undo database migrations, change the currently running kernel, restore arbitrary application data, or fix a failed disk or unresponsive systemd/kernel.

## Commands

Normal server update:

```sh
update
```

Deploy a particular source or commit:

```sh
sudo fred-deploy --flake 'git+https://forg.gregersen.it/rope/nixos?rev=FULL_COMMIT_SHA'
```

Use `--reboot` to request the nightly job's conditional reboot behavior. The server aliases now accept `fred-deploy` options, rather than arbitrary `nixos-rebuild` flags. Calling `nixos-rebuild switch` or `boot` directly bypasses this transaction.

Inspect the latest transaction or retry interrupted/failed recovery:

```sh
sudo cat /var/lib/fred-deploy/last-result.json
sudo fred-deploy recover
journalctl -u fred-deploy-recover
```

`recover` handles an unfinished transaction; it is not a general rollback command after a successful commit. Manual recovery also runs in an independent systemd service. Keep the saved roots and state until recovery succeeds.

For the first installation, the live `update` alias still points to the old updater. Build the new command from a checkout on the server first:

Supervise this first rollout: older generations predate the shared watchdog lock and boot recovery service. In particular, booting an older generation can restore connectivity without automatically clearing the saved unfinished transaction; use the built command's `recover` action before retrying.

```sh
deploy_tool=$(nix build --no-link --print-out-paths path:.#nixosConfigurations.FredOS-Mediaserver.config.system.build.deploy)
```

When ready to activate that checkout, invoke the built command explicitly with `sudo "$deploy_tool/bin/fred-deploy" --flake "path:$PWD"`. Alternatively, pass the exact Forgejo commit containing these changes. Build and review the command before invoking it; none of these activation commands were run during implementation.

## Verification performed

`python3 scripts/deploy-check.py` exercises failure after partial runtime activation, health failure, partial profile/boot commit, failed boot restoration, failed recovery health, timer setup/cancellation failures, stale timers, boot recovery, a full state filesystem, and missing candidate recovery support. It also tests actual process-level lock exclusion, independent recovery stopping a hung fixture worker before taking its lock, and subprocess timeout cleanup of descendants. Health-probe failures use mocked services, networking and sockets.

The deployment wrapper is built with Nix's shell checks, and physical host toplevels are evaluated. These tests use temporary filesystem/system fixtures; they do not create live deployment timers, reboot machines, or activate a real generation. A controlled server rollout remains the final operational validation.