nixos/docs/failure-path-fixes.md

# Approved failure-path fixes

This note covers proposal items 1, 2, 4 and 5. Item 3 is now implemented and documented in [deployment.md](deployment.md); the migration backup script remains last. Camera configuration is unchanged.

- **Firewall startup:** shared configuration now uses NixOS's early firewall startup. The mediaserver continues to own its separate nftables firewall.
- **WAN forwarding:** forwarded Docker traffic must match the original protocol/port in `ports.toml`. TCP and UDP 25565–25600 remain available automatically for Pelican allocations. SFTP binds to 10.0.0.1:2022. Forward names, LAN destinations, overlaps and SSH/HTTPS destinations are validated both by the UI and during Nix evaluation.
- **Login and locking:** greetd runs the Quickshell greeter as `greeter` inside Cage and authenticates before creating Fred's session. The existing visual layout is shared with an independent Quickshell lock process. The greeter uses the default wallpaper and the desktop's GB/Norwegian keyboard layouts; the lock reads the current wallpaper. Greetd unlocks the login keyring before Spotify starts. The old boot marker and IPC unlock method are removed. `fred-lock` waits for compositor-confirmed screen coverage; failure or timeout fails the required pre-sleep service and prevents sleep.
- **Media storage:** missing data disks no longer make the mounts mandatory for router boot. Device waits are bounded at 15 seconds. The pool requires every branch, and media consumers require a readiness service bound to all four mounts. Directory creation and the existing recursive permission rules run only after mount checks. These consumers stop when a required mount disappears. Routing and services outside the media pool remain independent.

## Validation

All four physical hosts were evaluated. The authentication bundle and media-readiness unit were built without activating a system. Tests cover:

- Existing router UI round trips plus malformed destinations/names, overlaps and redirected management ports; equivalent Nix checks reject 13 invalid inventories.
- Evaluated nftables rules in private user/network namespaces, with a second NAT table modeling Docker publications: WAN TCP/UDP at both Pelican range endpoints succeeds; unlisted ports fail; LAN SFTP succeeds. This is a packet-level test of the rules, not a running Pelican instance.
- The actual greeter QML against a fake greetd socket: a rejected password never requests a session launch; successful retry requests the session with Wayland metadata. The pinned Quickshell logs an unexpected-response message for the cancellation acknowledgement; retry and launch still succeed.
- The actual lock QML on a private headless Sway compositor: coverage is confirmed; the helper fails when IPC disappears or reports that coverage never became secure.
- The generated media-readiness script in a private mount namespace: missing pool, missing branch and subsequently lost branch all fail; a complete mounted fixture receives its directories; the underlying root directory stays empty.

The broader `nix flake check --no-build` still fails on the existing `mediaserver-vm` output's root-filesystem/bootloader assertions. The configured nixos MCP server did not complete initialization; options were checked against the pinned module sources and host evaluations instead.

These checks do not substitute for a physical login, keyring unlock, lid-close/resume or disk-disconnection test. No configuration has been activated, no production service has been restarted, and no production firewall or mount has been changed.

## Repeating the focused tests

The Python tests document their required tools and arguments in their opening docstrings. Use packages from the flake's pinned package set. New files must be included in the source; `path:.` includes the working tree before they are committed.

Build the authentication fixture:

```sh
nix build --no-link --print-out-paths path:.#nixosConfigurations.FredOS-Gaming.config.system.build.auth
```

Run `scripts/greeter-check.py` and `scripts/session-lock-check.py` with that output path. Put `qs` and `sway` on PATH and set `QML2_IMPORT_PATH` to the pinned `qt6.qt5compat` package's `lib/qt-6/qml` directory.

Build the media-readiness fixture:

```sh
nix build --no-link --print-out-paths --impure --expr \
  '(builtins.getFlake ("path:" + toString ./.)).nixosConfigurations.FredOS-Mediaserver.config.systemd.units."media-storage-ready.service".unit'
```

Pass its output path to `sudo python3 scripts/media-storage-check.py`. The script creates a private mount namespace before making any mount changes.

Run the Nix forwarding checks:

```sh
nix eval --impure --expr \
  'import ./scripts/router-forwards-check.nix { lib = (builtins.getFlake ("path:" + toString ./.)).inputs.nixpkgs.lib; }'
```

For packet tests, export the evaluated tables and pass the file to `python3 scripts/router-firewall-check.py`; that script creates its own private user/network namespace:

```sh
nix eval --raw path:.#nixosConfigurations.FredOS-Mediaserver.config.networking.nftables.tables \
  --apply 'tables: builtins.concatStringsSep "\n" (builtins.attrValues (builtins.mapAttrs (name: table: "table ${table.family} ${name} {\n${table.content}\n}") tables))' \
  > /tmp/router-rules.nft
```

## Operational behavior

The new greeter takes effect on the next login or boot. It supports the configured password authentication; unusual PAM conversations such as forced password changes require a text console. The independent locker starts on the first lock request. Restarting that locker reacquires the lock, while ordinary bar reloads leave it alone.

If media storage is unavailable, restore and verify the missing disks first, then start the affected media services. Their dependencies retry mounts and directory preparation. Services deliberately stay stopped after losing storage; they do not automatically resume merely because a disk returns. Inspect `journalctl -u media-storage-ready` and `systemctl status mnt-storage.mount` when diagnosing this.

The shared deployment transaction is documented separately in [deployment.md](deployment.md).