docs: add P5-12 to phase5-incident-log, update INC-001 and P5-05 states

P5-12 (weekly-health-digest): documented scope, deployed location, cron
schedule, ntfy topic, break-test result. Removes duplicate
monitoring/PHASE5-incidents.md — P5-12 belongs in the canonical
runbooks/phase5-incident-log.md alongside all other phase items.

INC-001 state updated: PBS rebooted May 14; both usb1-zfs and usb2-zfs
now MISSING (usb2-zfs was previously ONLINE on May 5 — new degradation).
zfs-health-check alerting every 5 min to ntfy/zfs-health.

P5-05 marked RESOLVED: all certs renewed to Jul 22 2026, confirmed via
live openssl check on 2026-05-16.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
tommy
2026-05-16 08:58:16 -05:00
parent a4726a4da6
commit 2ee7c474f5
2 changed files with 23 additions and 61 deletions
+23 -4
View File
@@ -1,6 +1,6 @@
# Phase 5 — Incident Log / State Drift
Last updated: 2026-05-06
Last updated: 2026-05-16
Purpose: Consolidates real findings from the Phase 4 worksession. Inputs to Phase 5 planning and repair.
---
@@ -54,7 +54,7 @@ Battery quick test on cyberpower2 produced `OL DISCHRG` — no ONBATT event, no
**First event:** March 12, 2026 (earliest confirmed crash in journal)
**Crash count:** 34+ unclean reboots since April 21; ~50+ total since March 12
**Current state:** usb1-zfs SUSPENDED with 4 data errors / 24 write errors. PBS up ~24h on current boot. agents hung task cleared. usb2-zfs ONLINE.
**Current state:** PBS rebooted May 14 17:41 CDT. After reboot: usb1-zfs MISSING (not imported — was previously SUSPENDED). usb2-zfs also MISSING (NEW — was ONLINE on May 5). Both pools failing to import. zfs-health-check.sh alerting to ntfy/zfs-health every 5 min. Drives still detected by kernel (sdb-sdf on USB paths). Pools not auto-imported after boot; manual import attempt needed.
**Root cause (confirmed):**
A 4-port USB hub at xHCI Bus 2 Port 6 intermittently loses power or signal, dropping all 4 ASM1153E (174c:55aa) USB-to-SATA bridges simultaneously. ZFS detects uncorrectable I/O on usb1-zfs and suspends the pool. The PBS `agents` process enters D-state waiting for ZFS TXG sync on the suspended pool. After 737+ seconds, the kernel logs a hung task warning. Eventually the system hangs and watchdog-reboots.
@@ -179,11 +179,13 @@ The C3 issue (May 5 health check: i/o timeouts every 2 min under scanner load) i
- Add Prometheus scrape target for Authelia metrics
- Add alert: `AutheliaHighErrorRate``authelia_request_duration_seconds_count{status="408"}` rate elevated for >5m
### P5-05 — TLS Cert Renewal Verification (time-sensitive)
### P5-05 — TLS Cert Renewal Verification (RESOLVED 2026-05-16)
4 certs were in the 3035d expiry window during the cert break-test (~May 5).
Traefik should auto-renew via ACME when ≤30d remain.
**Action schedule:**
**RESOLVED:** All certs renewed to July 22 2026 (67 days remaining) — confirmed via live openssl check 2026-05-16. Traefik renewed automatically before expiry.
**Original action schedule (for reference):**
- **May 11:** SSH to active Traefik node (docker-node01), inspect acme.json, confirm the 4 certs have new `notAfter` dates.
```bash
ssh tommy@192.168.99.186 "cat ~/traefik/acme.json | python3 -c \"
@@ -229,6 +231,23 @@ Monitor: check `Unsafe Shutdowns` and `Power Cycles` in SMART at each health che
---
### P5-12 — Weekly Health Digest (DONE 2026-05-16)
**Deliverable:** `monitoring/weekly-health-digest.sh`
**Deployed:** media-server `~/ansible/scripts/weekly-health-digest.sh`
**Cron:** `/etc/cron.d/weekly-health-digest` — Monday 07:00 CDT
**ntfy topic:** `health-digest` on 192.168.99.198:8095 (dedicated topic, not grafana-alerts)
Lightweight weekly drift-catcher. Covers: PVE reachability, Prometheus target health, firing alerts, PBS backup status, PBS ZFS pool presence, filesystem >85%, ZFS pool state (das-mirror/compute3-zfs), Docker unhealthy containers, TLS cert expiry <21d.
**Does NOT replace the quarterly full health check.** Full sweeps documented in `~/ansible/reports/health-check-*.md`.
Break-tested 2026-05-16 08:54 CDT: correctly flagged PBS ZFS MISSING + Grafana firing alerts. Suppressed known-persistent issues (P5-10 .227, HAOS .100, immich_machine_learning). ntfy delivery confirmed.
Subscriber reminder: subscribe ntfy client to topic `health-digest` on 192.168.99.198:8095.
---
## Known Config Drift (not incidents — track for cleanup)
| Item | Location | Drift |