From 09a466b2c6dbec39761d537f1b9f6fc74d9ef380 Mon Sep 17 00:00:00 2001 From: tommy Date: Thu, 2 Jul 2026 00:01:48 -0500 Subject: [PATCH] runbooks: log P5-13 media-server kernel upgrade to 6.8.0-134 Rebooted 2026-07-01; post-boot checks green (nvidia/DKMS, 54 containers, 18 Prom targets, 0 failed units, /mnt/media). Documents the gluetun caveat: a stale PIA WG peer does not survive a reboot -> run pia-wg/regen-and-up.sh as a required post-reboot step. Co-Authored-By: Claude Opus 4.8 --- runbooks/phase5-incident-log.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/runbooks/phase5-incident-log.md b/runbooks/phase5-incident-log.md index e385349..c9bee30 100644 --- a/runbooks/phase5-incident-log.md +++ b/runbooks/phase5-incident-log.md @@ -307,6 +307,29 @@ Subscriber reminder: subscribe ntfy client to topic `health-digest` on 192.168.9 --- +### P5-13 — media-server Kernel Upgrade 6.8.0-124 → 6.8.0-134 (DONE 2026-07-01) + +**Host:** media-server (192.168.99.183 / VM 101) +**Rebooted:** 2026-07-01 23:54 CDT into `6.8.0-134-generic` (was running `-124`). + +Kernel was staged earlier that day (6.8.0-134 on disk; `nvidia-srv/580.159.03` DKMS module built + signed for -134). `GRUB_DEFAULT=0` → reboot booted -134; `-124` and `-117` remain DKMS-built in the GRUB Advanced menu as fallbacks. Rollback strategy = PBS live backup of VM 101 (nvmebak storage can't snapshot without shutdown). + +**Post-boot checks — all green:** +- `uname -r` = `6.8.0-134-generic` +- `nvidia-smi` — both Quadro P2200s present, driver 580.159.03 loaded; `dkms status` shows nvidia-srv built for -134 +- 54/54 containers running, none restarting/unhealthy +- 18/18 Prometheus targets up +- 0 failed systemd units +- `/mnt/media` (TrueNAS .29 NFS) mounted + +**⚠️ Gluetun caveat — a stale PIA WireGuard peer does NOT survive a host reboot.** After the reboot gluetun came back on the last-written `wg0.conf` (Jun-30 peer, endpoint `181.41.206.145`) which no longer passed traffic: tight VPN restart loop, healthchecks failing with `dial tcp4: lookup github.com/cloudflare.com: i/o timeout`, `public_ip` stuck at `""`. Even though that peer had been freshly registered Jun 30, it did not survive the cold start. Fixed by running the documented self-heal: +```bash +~/downloads/pia-wg/regen-and-up.sh # regen fresh peer → recreate gluetun → reconnect sab/transmission +``` +Result: fresh `us_denver` peer (endpoint `181.41.206.159:1337`), gluetun Healthy, egress verified as PIA Denver, Colorado. **Treat `regen-and-up.sh` as a required post-reboot step for the downloads stack — don't wait for the VpnTunnelDown alert.** + +--- + ## Known Config Drift (not incidents — track for cleanup) | Item | Location | Drift |