tommy/homelab-configs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

runbooks: log P5-13 media-server kernel upgrade to 6.8.0-134

Browse Source 
Rebooted 2026-07-01; post-boot checks green (nvidia/DKMS, 54 containers,
18 Prom targets, 0 failed units, /mnt/media). Documents the gluetun
caveat: a stale PIA WG peer does not survive a reboot -> run
pia-wg/regen-and-up.sh as a required post-reboot step.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
tommy
2026-07-02 00:01:48 -05:00
parent 3746f3fe7c
commit 09a466b2c6
1 changed files with 23 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
runbooks/phase5-incident-log.md
+23
View File
@@ -307,6 +307,29 @@ Subscriber reminder: subscribe ntfy client to topic `health-digest` on 192.168.9
---
### P5-13 — media-server Kernel Upgrade 6.8.0-124 → 6.8.0-134 (DONE 2026-07-01)
**Host:** media-server (192.168.99.183 / VM 101)
**Rebooted:** 2026-07-01 23:54 CDT into `6.8.0-134-generic` (was running `-124`).
Kernel was staged earlier that day (6.8.0-134 on disk; `nvidia-srv/580.159.03` DKMS module built + signed for -134). `GRUB_DEFAULT=0` → reboot booted -134; `-124` and `-117` remain DKMS-built in the GRUB Advanced menu as fallbacks. Rollback strategy = PBS live backup of VM 101 (nvmebak storage can't snapshot without shutdown).
**Post-boot checks — all green:**
- `uname -r` = `6.8.0-134-generic`
- `nvidia-smi` — both Quadro P2200s present, driver 580.159.03 loaded; `dkms status` shows nvidia-srv built for -134
- 54/54 containers running, none restarting/unhealthy
- 18/18 Prometheus targets up
- 0 failed systemd units
- `/mnt/media` (TrueNAS .29 NFS) mounted
**⚠️ Gluetun caveat — a stale PIA WireGuard peer does NOT survive a host reboot.** After the reboot gluetun came back on the last-written `wg0.conf` (Jun-30 peer, endpoint `181.41.206.145`) which no longer passed traffic: tight VPN restart loop, healthchecks failing with `dial tcp4: lookup github.com/cloudflare.com: i/o timeout`, `public_ip` stuck at `""`. Even though that peer had been freshly registered Jun 30, it did not survive the cold start. Fixed by running the documented self-heal:
```bash
~/downloads/pia-wg/regen-and-up.sh # regen fresh peer → recreate gluetun → reconnect sab/transmission
```
Result: fresh `us_denver` peer (endpoint `181.41.206.159:1337`), gluetun Healthy, egress verified as PIA Denver, Colorado. **Treat `regen-and-up.sh` as a required post-reboot step for the downloads stack — don't wait for the VpnTunnelDown alert.**
---
## Known Config Drift (not incidents — track for cleanup)
| Item | Location | Drift |