Harden deployment, authentication, firewall and media storage failure paths

docs/deployment.md

@@ -0,0 +1,71 @@
+# Server deployment and recovery
+
+The mediaserver's `update` and `update2` aliases, the router UI Apply action, and the 05:15 nightly update now use `fred-deploy`. The clients' build/copy/switch workflow and the migration backup script are unchanged. This change has not been activated on any machine.
+
+## Transaction
+
+Each deployment runs in its own transient systemd service. Losing the terminal or router UI apply process does not terminate that service. Logs go to the journal; the calling command prints the unit to follow and collects its journal after completion.
+
+One lock, `/run/lock/fred-deploy.lock`, covers the entire server build, activation, verification, commit and recovery. Other deployments queue behind it. The WAN watchdog takes the same lock nonblockingly and skips its cycle when the lock is held, including conntrack flushes and network restarts.
+
+The transaction captures the **running system** and **system-profile boot default** separately and retains both, the candidate and its own recovery code as GC roots. It builds exactly one candidate; committing never re-fetches a moving branch. A candidate must contain the boot recovery service.
+
+After the build succeeds, a separate ten-minute systemd timer is armed. The candidate is activated with `switch-to-configuration test`. Essential router checks must pass before the system profile is changed and `switch-to-configuration boot` installs the new boot default. A final health check precedes the durable `committed` result and timer cancellation.
+
+Activation, health-check or boot-commit errors restore the captured boot profile, reinstall its boot configuration, and reactivate the captured running system. Recovery attempts runtime restoration even if restoring the bootloader fails. Failed recovery retains the roots and records `rollback-failed`; it is never reported as a successful rollback. New deployments refuse to overwrite an unresolved transaction.
+
+The independent timer stops a stuck deployment worker before taking the lock, so it does not deadlock behind that worker. Stale timers cannot undo a committed or newer transaction. Subprocess timeouts kill their process groups. Durable state and GC-root directory changes are fsynced.
+
+Transient timers disappear on reboot. The boot recovery service checks for an unfinished transaction from a previous boot and runs the pinned recovery code. It uses `Type=exec` so its activation does not block completion of the boot targets that the switch script starts. A deployment arriving before boot recovery takes the lock recovers the old transaction first. Same-boot activation of this service does not cancel the deployment installing it.
+
+## Health checks and limits
+
+Checks cover networkd, dnsmasq, AdGuard Home, nftables, SSH, nginx and the router UI; the LAN address and WAN default route; IPv4 forwarding; default-drop input/forward chains; DNAT and masquerade rules; WAN ping through either of two targets; an actual LAN DNS lookup and SSH banner; the local UI HTTP endpoint and its TLS management proxy.
+
+Media applications are not health probes. A nonzero Nix activation exit still causes rollback, including activation errors originating outside the router services. An ISP, public DNS or certificate outage can also fail verification and cause a conservative rollback. The local probes do not prove reachability from every external client.
+
+A build is limited to 90 minutes; individual activation and bootloader commands to two minutes. Health checks retry for 90 seconds, with each probe also bounded. The timer's ten-minute deadline starts after building. The supervised deployment has a 100-minute runtime limit, including lock waits. Recovery has its own runtime limit.
+
+The nightly job retains the existing conditional reboot behavior: after a successful transaction, it requests a reboot if the booted kernel, initrd or kernel modules differ from the candidate. This is **not** an automatic post-reboot kernel validation/boot-fallback system. Recovery cannot undo database migrations, change the currently running kernel, restore arbitrary application data, or fix a failed disk or unresponsive systemd/kernel.
+
+## Commands
+
+Normal server update:
+
+```sh
+update
+```
+
+Deploy a particular source or commit:
+
+```sh
+sudo fred-deploy --flake 'git+https://forg.gregersen.it/rope/nixos?rev=FULL_COMMIT_SHA'
+```
+
+Use `--reboot` to request the nightly job's conditional reboot behavior. The server aliases now accept `fred-deploy` options, rather than arbitrary `nixos-rebuild` flags. Calling `nixos-rebuild switch` or `boot` directly bypasses this transaction.
+
+Inspect the latest transaction or retry interrupted/failed recovery:
+
+```sh
+sudo cat /var/lib/fred-deploy/last-result.json
+sudo fred-deploy recover
+journalctl -u fred-deploy-recover
+```
+
+`recover` handles an unfinished transaction; it is not a general rollback command after a successful commit. Manual recovery also runs in an independent systemd service. Keep the saved roots and state until recovery succeeds.
+
+For the first installation, the live `update` alias still points to the old updater. Build the new command from a checkout on the server first:
+
+Supervise this first rollout: older generations predate the shared watchdog lock and boot recovery service. In particular, booting an older generation can restore connectivity without automatically clearing the saved unfinished transaction; use the built command's `recover` action before retrying.
+
+```sh
+deploy_tool=$(nix build --no-link --print-out-paths path:.#nixosConfigurations.FredOS-Mediaserver.config.system.build.deploy)
+```
+
+When ready to activate that checkout, invoke the built command explicitly with `sudo "$deploy_tool/bin/fred-deploy" --flake "path:$PWD"`. Alternatively, pass the exact Forgejo commit containing these changes. Build and review the command before invoking it; none of these activation commands were run during implementation.
+
+## Verification performed
+
+`python3 scripts/deploy-check.py` exercises failure after partial runtime activation, health failure, partial profile/boot commit, failed boot restoration, failed recovery health, timer setup/cancellation failures, stale timers, boot recovery, a full state filesystem, and missing candidate recovery support. It also tests actual process-level lock exclusion, independent recovery stopping a hung fixture worker before taking its lock, and subprocess timeout cleanup of descendants. Health-probe failures use mocked services, networking and sockets.
+
+The deployment wrapper is built with Nix's shell checks, and physical host toplevels are evaluated. The existing `mediaserver-vm` root-filesystem/bootloader issue still prevents a clean whole-flake check. These tests use temporary filesystem/system fixtures; they do not create live deployment timers, reboot machines, or activate a real generation. A controlled server rollout remains the final operational validation.

docs/failure-path-fixes.md

@@ -0,0 +1,66 @@
+# Approved failure-path fixes
+
+This note covers proposal items 1, 2, 4 and 5. Item 3 is now implemented and documented in [deployment.md](deployment.md); the migration backup script remains last. Camera configuration is unchanged.
+
+- **Firewall startup:** shared configuration now uses NixOS's early firewall startup. The mediaserver continues to own its separate nftables firewall.
+- **WAN forwarding:** forwarded Docker traffic must match the original protocol/port in `ports.toml`. TCP and UDP 25565–25600 remain available automatically for Pelican allocations. SFTP binds to 10.0.0.1:2022. Forward names, LAN destinations, overlaps and SSH/HTTPS destinations are validated both by the UI and during Nix evaluation.
+- **Login and locking:** greetd runs the Quickshell greeter as `greeter` inside Cage and authenticates before creating Fred's session. The existing visual layout is shared with an independent Quickshell lock process. The greeter uses the default wallpaper and the desktop's GB/Norwegian keyboard layouts; the lock reads the current wallpaper. Greetd unlocks the login keyring before Spotify starts. The old boot marker and IPC unlock method are removed. `fred-lock` waits for compositor-confirmed screen coverage; failure or timeout fails the required pre-sleep service and prevents sleep.
+- **Media storage:** missing data disks no longer make the mounts mandatory for router boot. Device waits are bounded at 15 seconds. The pool requires every branch, and media consumers require a readiness service bound to all four mounts. Directory creation and the existing recursive permission rules run only after mount checks. These consumers stop when a required mount disappears. Routing and services outside the media pool remain independent.
+
+## Validation
+
+All four physical hosts were evaluated. The authentication bundle and media-readiness unit were built without activating a system. Tests cover:
+
+- Existing router UI round trips plus malformed destinations/names, overlaps and redirected management ports; equivalent Nix checks reject 13 invalid inventories.
+- Evaluated nftables rules in private user/network namespaces, with a second NAT table modeling Docker publications: WAN TCP/UDP at both Pelican range endpoints succeeds; unlisted ports fail; LAN SFTP succeeds. This is a packet-level test of the rules, not a running Pelican instance.
+- The actual greeter QML against a fake greetd socket: a rejected password never requests a session launch; successful retry requests the session with Wayland metadata. The pinned Quickshell logs an unexpected-response message for the cancellation acknowledgement; retry and launch still succeed.
+- The actual lock QML on a private headless Sway compositor: coverage is confirmed; the helper fails when IPC disappears or reports that coverage never became secure.
+- The generated media-readiness script in a private mount namespace: missing pool, missing branch and subsequently lost branch all fail; a complete mounted fixture receives its directories; the underlying root directory stays empty.
+
+The broader `nix flake check --no-build` still fails on the existing `mediaserver-vm` output's root-filesystem/bootloader assertions. The configured nixos MCP server did not complete initialization; options were checked against the pinned module sources and host evaluations instead.
+
+These checks do not substitute for a physical login, keyring unlock, lid-close/resume or disk-disconnection test. No configuration has been activated, no production service has been restarted, and no production firewall or mount has been changed.
+
+## Repeating the focused tests
+
+The Python tests document their required tools and arguments in their opening docstrings. Use packages from the flake's pinned package set. New files must be included in the source; `path:.` includes the working tree before they are committed.
+
+Build the authentication fixture:
+
+```sh
+nix build --no-link --print-out-paths path:.#nixosConfigurations.FredOS-Gaming.config.system.build.auth
+```
+
+Run `scripts/greeter-check.py` and `scripts/session-lock-check.py` with that output path. Put `qs` and `sway` on PATH and set `QML2_IMPORT_PATH` to the pinned `qt6.qt5compat` package's `lib/qt-6/qml` directory.
+
+Build the media-readiness fixture:
+
+```sh
+nix build --no-link --print-out-paths --impure --expr \
+  '(builtins.getFlake ("path:" + toString ./.)).nixosConfigurations.FredOS-Mediaserver.config.systemd.units."media-storage-ready.service".unit'
+```
+
+Pass its output path to `sudo python3 scripts/media-storage-check.py`. The script creates a private mount namespace before making any mount changes.
+
+Run the Nix forwarding checks:
+
+```sh
+nix eval --impure --expr \
+  'import ./scripts/router-forwards-check.nix { lib = (builtins.getFlake ("path:" + toString ./.)).inputs.nixpkgs.lib; }'
+```
+
+For packet tests, export the evaluated tables and pass the file to `python3 scripts/router-firewall-check.py`; that script creates its own private user/network namespace:
+
+```sh
+nix eval --raw path:.#nixosConfigurations.FredOS-Mediaserver.config.networking.nftables.tables \
+  --apply 'tables: builtins.concatStringsSep "\n" (builtins.attrValues (builtins.mapAttrs (name: table: "table ${table.family} ${name} {\n${table.content}\n}") tables))' \
+  > /tmp/router-rules.nft
+```
+
+## Operational behavior
+
+The new greeter takes effect on the next login or boot. It supports the configured password authentication; unusual PAM conversations such as forced password changes require a text console. The independent locker starts on the first lock request. Restarting that locker reacquires the lock, while ordinary bar reloads leave it alone.
+
+If media storage is unavailable, restore and verify the missing disks first, then start the affected media services. Their dependencies retry mounts and directory preparation. Services deliberately stay stopped after losing storage; they do not automatically resume merely because a disk returns. Inspect `journalctl -u media-storage-ready` and `systemctl status mnt-storage.mount` when diagnosing this.
+
+The shared deployment transaction is documented separately in [deployment.md](deployment.md).