From 2ee7c474f57b3c1953d44c3129397e3fc0b65c50 Mon Sep 17 00:00:00 2001 From: tommy Date: Sat, 16 May 2026 08:58:16 -0500 Subject: [PATCH] docs: add P5-12 to phase5-incident-log, update INC-001 and P5-05 states MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit P5-12 (weekly-health-digest): documented scope, deployed location, cron schedule, ntfy topic, break-test result. Removes duplicate monitoring/PHASE5-incidents.md — P5-12 belongs in the canonical runbooks/phase5-incident-log.md alongside all other phase items. INC-001 state updated: PBS rebooted May 14; both usb1-zfs and usb2-zfs now MISSING (usb2-zfs was previously ONLINE on May 5 — new degradation). zfs-health-check alerting every 5 min to ntfy/zfs-health. P5-05 marked RESOLVED: all certs renewed to Jul 22 2026, confirmed via live openssl check on 2026-05-16. Co-Authored-By: Claude Sonnet 4.6 --- monitoring/PHASE5-incidents.md | 57 --------------------------------- runbooks/phase5-incident-log.md | 27 +++++++++++++--- 2 files changed, 23 insertions(+), 61 deletions(-) delete mode 100644 monitoring/PHASE5-incidents.md diff --git a/monitoring/PHASE5-incidents.md b/monitoring/PHASE5-incidents.md deleted file mode 100644 index 3733ef4..0000000 --- a/monitoring/PHASE5-incidents.md +++ /dev/null @@ -1,57 +0,0 @@ -# Phase 5 — Incident & Deliverable Log - -## P5-12 — Weekly Health Digest (2026-05-16) - -**Deliverable:** `monitoring/weekly-health-digest.sh` -**Deployed to:** media-server (192.168.99.183) at `~/ansible/scripts/weekly-health-digest.sh` -**Schedule:** `/etc/cron.d/weekly-health-digest` — Monday 07:00 CDT - -### What the digest covers - -Lightweight drift-catcher. Runs read-only checks each Monday and posts a single concise message to the **ntfy `health-digest` topic** (dedicated topic — not grafana-alerts, not prometheus-alerts). - -| Check | Method | -|-------|--------| -| PVE cluster reachable | Ping beast node-exporter | -| Prometheus targets all up | `up{job="node-exporter"}` query — known-persistent down (.227/.100) suppressed | -| Grafana/Prometheus firing alerts | `/api/v1/alerts` — PrometheusTargetDown/NodeDown suppressed (covered above) | -| PBS backup freshness | `backup_status` textfile metric | -| PBS ZFS pools imported | `zfs_pool_present` textfile metric (usb1-zfs, usb2-zfs) | -| Filesystem >85% (all nodes) | Prometheus node_filesystem query | -| ZFS pools ONLINE (cluster-wide) | `node_zfs_zpool_state` for degraded/faulted/suspended/unavail | -| Docker containers unhealthy/restarting | `docker ps` on media-server (known-unhealthy `immich_machine_learning` suppressed) | -| TLS cert expiry <21 days | Live openssl check on 6 representative domains | - -### What the digest does NOT do - -- Does NOT replace the quarterly full health check sweep. -- Does NOT check HAOS internals, Frigate camera counts, Coral inference times, or Authelia error rates. -- Does NOT check docker-node01/02 containers directly (no Docker API exposed). -- Does NOT verify Keepalived MASTER/BACKUP state. -- Does NOT check PBS datastore GC/verify schedules in detail. - -These areas require a human-run full sweep (see `~/ansible/reports/health-check-*.md`). - -### ntfy subscription - -Subscribe ntfy client to topic `health-digest` on `192.168.99.198:8095`. -Same procedure as the `zfs-health` topic added in Phase 4D. - -### Break-test result (2026-05-16 08:54 CDT) - -Script ran successfully. Delivery confirmed on `health-digest` topic. Correctly flagged: -- **PBS ZFS pools MISSING** (usb1-zfs + usb2-zfs — INC-001 + new usb2-zfs failure) ✓ -- **Grafana FIRING** (ContainerMemoryNearLimit, SwapHighWarning) ✓ -- All healthy checks reported OK ✓ -- Known-down nodes (.227 P5-10, .100 HAOS) correctly suppressed ✓ - ---- - -## Open Phase 5 Items - -| ID | Description | Status | -|----|-------------|--------| -| P5-07 | ansible-control disk at 79.2% (was 81.5%) | Ongoing — still approaching 85% | -| P5-10 | Pi4 node-exporter wrong-arch binary (.227) | Open | -| P5-12 | Weekly health digest | **DONE** (this entry) | - diff --git a/runbooks/phase5-incident-log.md b/runbooks/phase5-incident-log.md index fe9f79c..24a8bfe 100644 --- a/runbooks/phase5-incident-log.md +++ b/runbooks/phase5-incident-log.md @@ -1,6 +1,6 @@ # Phase 5 — Incident Log / State Drift -Last updated: 2026-05-06 +Last updated: 2026-05-16 Purpose: Consolidates real findings from the Phase 4 worksession. Inputs to Phase 5 planning and repair. --- @@ -54,7 +54,7 @@ Battery quick test on cyberpower2 produced `OL DISCHRG` — no ONBATT event, no **First event:** March 12, 2026 (earliest confirmed crash in journal) **Crash count:** 34+ unclean reboots since April 21; ~50+ total since March 12 -**Current state:** usb1-zfs SUSPENDED with 4 data errors / 24 write errors. PBS up ~24h on current boot. agents hung task cleared. usb2-zfs ONLINE. +**Current state:** PBS rebooted May 14 17:41 CDT. After reboot: usb1-zfs MISSING (not imported — was previously SUSPENDED). usb2-zfs also MISSING (NEW — was ONLINE on May 5). Both pools failing to import. zfs-health-check.sh alerting to ntfy/zfs-health every 5 min. Drives still detected by kernel (sdb-sdf on USB paths). Pools not auto-imported after boot; manual import attempt needed. **Root cause (confirmed):** A 4-port USB hub at xHCI Bus 2 Port 6 intermittently loses power or signal, dropping all 4 ASM1153E (174c:55aa) USB-to-SATA bridges simultaneously. ZFS detects uncorrectable I/O on usb1-zfs and suspends the pool. The PBS `agents` process enters D-state waiting for ZFS TXG sync on the suspended pool. After 737+ seconds, the kernel logs a hung task warning. Eventually the system hangs and watchdog-reboots. @@ -179,11 +179,13 @@ The C3 issue (May 5 health check: i/o timeouts every 2 min under scanner load) i - Add Prometheus scrape target for Authelia metrics - Add alert: `AutheliaHighErrorRate` — `authelia_request_duration_seconds_count{status="408"}` rate elevated for >5m -### P5-05 — TLS Cert Renewal Verification (time-sensitive) +### P5-05 — TLS Cert Renewal Verification (RESOLVED 2026-05-16) 4 certs were in the 30–35d expiry window during the cert break-test (~May 5). Traefik should auto-renew via ACME when ≤30d remain. -**Action schedule:** +**RESOLVED:** All certs renewed to July 22 2026 (67 days remaining) — confirmed via live openssl check 2026-05-16. Traefik renewed automatically before expiry. + +**Original action schedule (for reference):** - **May 11:** SSH to active Traefik node (docker-node01), inspect acme.json, confirm the 4 certs have new `notAfter` dates. ```bash ssh tommy@192.168.99.186 "cat ~/traefik/acme.json | python3 -c \" @@ -229,6 +231,23 @@ Monitor: check `Unsafe Shutdowns` and `Power Cycles` in SMART at each health che --- +### P5-12 — Weekly Health Digest (DONE 2026-05-16) + +**Deliverable:** `monitoring/weekly-health-digest.sh` +**Deployed:** media-server `~/ansible/scripts/weekly-health-digest.sh` +**Cron:** `/etc/cron.d/weekly-health-digest` — Monday 07:00 CDT +**ntfy topic:** `health-digest` on 192.168.99.198:8095 (dedicated topic, not grafana-alerts) + +Lightweight weekly drift-catcher. Covers: PVE reachability, Prometheus target health, firing alerts, PBS backup status, PBS ZFS pool presence, filesystem >85%, ZFS pool state (das-mirror/compute3-zfs), Docker unhealthy containers, TLS cert expiry <21d. + +**Does NOT replace the quarterly full health check.** Full sweeps documented in `~/ansible/reports/health-check-*.md`. + +Break-tested 2026-05-16 08:54 CDT: correctly flagged PBS ZFS MISSING + Grafana firing alerts. Suppressed known-persistent issues (P5-10 .227, HAOS .100, immich_machine_learning). ntfy delivery confirmed. + +Subscriber reminder: subscribe ntfy client to topic `health-digest` on 192.168.99.198:8095. + +--- + ## Known Config Drift (not incidents — track for cleanup) | Item | Location | Drift |