docs: add INC-006 (PBS upsmon DEADTIME), update INC-001 pool state

INC-006: PBS upsmon DEADTIME=15s caused autonomous shutdown when Beast NUT server was briefly unreachable during reboot. Fixed 2026-05-16 by setting DEADTIME 300 on PBS. INC-001: usb1-zfs and usb2-zfs manually imported 2026-05-16 09:14 CDT after May 14 reboot left both MISSING. Both ONLINE, no errors. Scrubs initiated. Cachefile updated at /etc/zfs/zpool.cache. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-16 16:49:30 -05:00
parent 2ee7c474f5
commit 4372080a3b
1 changed files with 34 additions and 4 deletions
@@ -1,6 +1,6 @@
 # Phase 5 — Incident Log / State Drift

-Last updated: 2026-05-16  
+Last updated: 2026-05-16 (INC-006 added, INC-001 pool state updated)
 Purpose: Consolidates real findings from the Phase 4 worksession. Inputs to Phase 5 planning and repair.

 ---
@@ -50,11 +50,41 @@ Battery quick test on cyberpower2 produced `OL DISCHRG` — no ONBATT event, no

 ---

+### INC-006 — PBS upsmon DEADTIME Autonomous Shutdown (FIXED 2026-05-16)
+
+**Discovered:** 2026-05-16 during health-check sweep root-cause analysis
+**Fixed:** 2026-05-16 09:05 CDT
+**Class:** Same as INC-005 — autonomous mechanism triggering unexpected node shutdown
+
+**What happened:**
+PBS (192.168.99.153) rebooted unexpectedly at 17:41 CDT on May 14, 2026 — 10 days after INC-005 was fixed on Beast. Journal analysis showed nut-monitor crash-looping from 16:22, then reconnecting at 17:08 and reporting "UPS unavailable / connection timed out" for the cyberpower1 UPS served by Beast. After 15 seconds of no response from Beast's upsd (during a period when Beast was rebooting or its NUT daemon was unavailable), PBS upsmon triggered an autonomous "power down" and initiated a clean shutdown at 17:41.
+
+**Root cause:**
+`/etc/nut/upsmon.conf` on PBS had `DEADTIME 15` — meaning upsmon would declare the UPS communication lost and shut the system down after only 15 seconds of no response from the master NUT server (Beast). During any Beast reboot, NUT config reload, or transient network blip, this 15-second window is trivially exceeded. PBS was configured as a NUT slave: `MONITOR cyberpower1@beast.goattw.net:3493 1 ... slave`.
+
+The default NUT DEADTIME of 15 seconds is appropriate for a standalone UPS monitor where a lost connection means the UPS itself is gone. For a slave node monitoring a remote master, it should be set to a value that survives normal master reboots (typically 120–300s).
+
+**Fix applied (PBS, 2026-05-16):**
+```bash
+# Backup: /etc/nut/upsmon.conf.bak.20260516
+# Change:
+DEADTIME 15   →   DEADTIME 300
+```
+`nut-monitor` restarted and confirmed `active`. No force flags needed. Config committed to `homelab-configs/monitoring/` (see P5-12 commit scope — add standalone commit if needed).
+
+**Downstream effects:**
+- Both PBS ZFS pools (usb1-zfs, usb2-zfs) were not auto-imported after the May 14 reboot (stale cachefile). Manually imported 2026-05-16; scrubs initiated.
+- INC-001 zfs-health-check.sh was alerting to ntfy/zfs-health on every 5-min poll since the reboot.
+
+**Check Beast's slave nodes:** If any other NUT slave (compute5, etc.) has `DEADTIME 15`, apply the same fix. Beast itself does not need this change (it is the NUT master / upsd host).
+
+---
+
 ### INC-001 — PBS USB Hub Failure (ACTIVE — hardware fix in progress)

 **First event:** March 12, 2026 (earliest confirmed crash in journal)
 **Crash count:** 34+ unclean reboots since April 21; ~50+ total since March 12
-**Current state:** PBS rebooted May 14 17:41 CDT. After reboot: usb1-zfs MISSING (not imported — was previously SUSPENDED). usb2-zfs also MISSING (NEW — was ONLINE on May 5). Both pools failing to import. zfs-health-check.sh alerting to ntfy/zfs-health every 5 min. Drives still detected by kernel (sdb-sdf on USB paths). Pools not auto-imported after boot; manual import attempt needed.
+**Current state (2026-05-16):** PBS rebooted May 14 17:41 CDT (INC-006 — upsmon DEADTIME triggered). After reboot both usb1-zfs and usb2-zfs were MISSING (stale cachefile + not auto-imported). Manually imported 2026-05-16 09:14 CDT — both ONLINE, no data errors. Scrubs started: usb1-zfs scrubbing (last scrub Apr 12, 613G), usb2-zfs completed immediately (last scrub May 10). Cachefile updated at `/etc/zfs/zpool.cache`. Cenmate enclosure dropped again at ~07:16 on current boot (DID_NO_CONNECT at 138697s uptime) — hub failure is ongoing. Hardware replacement remains the only durable fix.

 **Root cause (confirmed):**  
 A 4-port USB hub at xHCI Bus 2 Port 6 intermittently loses power or signal, dropping all 4 ASM1153E (174c:55aa) USB-to-SATA bridges simultaneously. ZFS detects uncorrectable I/O on usb1-zfs and suspends the pool. The PBS `agents` process enters D-state waiting for ZFS TXG sync on the suspended pool. After 737+ seconds, the kernel logs a hung task warning. Eventually the system hangs and watchdog-reboots.