diff --git a/runbooks/phase5-incident-log.md b/runbooks/phase5-incident-log.md index e385349..c9bee30 100644 --- a/runbooks/phase5-incident-log.md +++ b/runbooks/phase5-incident-log.md @@ -307,6 +307,29 @@ Subscriber reminder: subscribe ntfy client to topic `health-digest` on 192.168.9 --- +### P5-13 — media-server Kernel Upgrade 6.8.0-124 → 6.8.0-134 (DONE 2026-07-01) + +**Host:** media-server (192.168.99.183 / VM 101) +**Rebooted:** 2026-07-01 23:54 CDT into `6.8.0-134-generic` (was running `-124`). + +Kernel was staged earlier that day (6.8.0-134 on disk; `nvidia-srv/580.159.03` DKMS module built + signed for -134). `GRUB_DEFAULT=0` → reboot booted -134; `-124` and `-117` remain DKMS-built in the GRUB Advanced menu as fallbacks. Rollback strategy = PBS live backup of VM 101 (nvmebak storage can't snapshot without shutdown). + +**Post-boot checks — all green:** +- `uname -r` = `6.8.0-134-generic` +- `nvidia-smi` — both Quadro P2200s present, driver 580.159.03 loaded; `dkms status` shows nvidia-srv built for -134 +- 54/54 containers running, none restarting/unhealthy +- 18/18 Prometheus targets up +- 0 failed systemd units +- `/mnt/media` (TrueNAS .29 NFS) mounted + +**⚠️ Gluetun caveat — a stale PIA WireGuard peer does NOT survive a host reboot.** After the reboot gluetun came back on the last-written `wg0.conf` (Jun-30 peer, endpoint `181.41.206.145`) which no longer passed traffic: tight VPN restart loop, healthchecks failing with `dial tcp4: lookup github.com/cloudflare.com: i/o timeout`, `public_ip` stuck at `""`. Even though that peer had been freshly registered Jun 30, it did not survive the cold start. Fixed by running the documented self-heal: +```bash +~/downloads/pia-wg/regen-and-up.sh # regen fresh peer → recreate gluetun → reconnect sab/transmission +``` +Result: fresh `us_denver` peer (endpoint `181.41.206.159:1337`), gluetun Healthy, egress verified as PIA Denver, Colorado. **Treat `regen-and-up.sh` as a required post-reboot step for the downloads stack — don't wait for the VpnTunnelDown alert.** + +--- + ## Known Config Drift (not incidents — track for cleanup) | Item | Location | Drift |