monitoring: add weekly-health-digest.sh and Phase 5 incident log (P5-12)

Lightweight weekly drift-catcher that posts a concise digest to the dedicated ntfy/health-digest topic every Monday at 07:00. Covers: PVE reachability, Prometheus target health, firing alerts, PBS backup status, PBS ZFS pool presence, filesystem >85%, ZFS pool state, Docker unhealthy containers, and TLS cert expiry <21d. Does NOT replace the quarterly full health check — documented in PHASE5-incidents.md alongside open P5-07 and P5-10 items. Deployed: /etc/cron.d/weekly-health-digest on media-server. Break-tested 2026-05-16: correctly flagged PBS ZFS pools MISSING and Grafana firing alerts; suppressed known-persistent issues. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-16 08:56:30 -05:00
parent 640cd908df
commit a4726a4da6
2 changed files with 319 additions and 0 deletions
@@ -0,0 +1,57 @@
+# Phase 5 — Incident & Deliverable Log
+
+## P5-12 — Weekly Health Digest (2026-05-16)
+
+**Deliverable:** `monitoring/weekly-health-digest.sh`
+**Deployed to:** media-server (192.168.99.183) at `~/ansible/scripts/weekly-health-digest.sh`
+**Schedule:** `/etc/cron.d/weekly-health-digest` — Monday 07:00 CDT
+
+### What the digest covers
+
+Lightweight drift-catcher. Runs read-only checks each Monday and posts a single concise message to the **ntfy `health-digest` topic** (dedicated topic — not grafana-alerts, not prometheus-alerts).
+
+| Check | Method |
+|-------|--------|
+| PVE cluster reachable | Ping beast node-exporter |
+| Prometheus targets all up | `up{job="node-exporter"}` query — known-persistent down (.227/.100) suppressed |
+| Grafana/Prometheus firing alerts | `/api/v1/alerts` — PrometheusTargetDown/NodeDown suppressed (covered above) |
+| PBS backup freshness | `backup_status` textfile metric |
+| PBS ZFS pools imported | `zfs_pool_present` textfile metric (usb1-zfs, usb2-zfs) |
+| Filesystem >85% (all nodes) | Prometheus node_filesystem query |
+| ZFS pools ONLINE (cluster-wide) | `node_zfs_zpool_state` for degraded/faulted/suspended/unavail |
+| Docker containers unhealthy/restarting | `docker ps` on media-server (known-unhealthy `immich_machine_learning` suppressed) |
+| TLS cert expiry <21 days | Live openssl check on 6 representative domains |
+
+### What the digest does NOT do
+
+- Does NOT replace the quarterly full health check sweep.
+- Does NOT check HAOS internals, Frigate camera counts, Coral inference times, or Authelia error rates.
+- Does NOT check docker-node01/02 containers directly (no Docker API exposed).
+- Does NOT verify Keepalived MASTER/BACKUP state.
+- Does NOT check PBS datastore GC/verify schedules in detail.
+
+These areas require a human-run full sweep (see `~/ansible/reports/health-check-*.md`).
+
+### ntfy subscription
+
+Subscribe ntfy client to topic `health-digest` on `192.168.99.198:8095`.
+Same procedure as the `zfs-health` topic added in Phase 4D.
+
+### Break-test result (2026-05-16 08:54 CDT)
+
+Script ran successfully. Delivery confirmed on `health-digest` topic. Correctly flagged:
+- **PBS ZFS pools MISSING** (usb1-zfs + usb2-zfs — INC-001 + new usb2-zfs failure) ✓
+- **Grafana FIRING** (ContainerMemoryNearLimit, SwapHighWarning) ✓
+- All healthy checks reported OK ✓
+- Known-down nodes (.227 P5-10, .100 HAOS) correctly suppressed ✓
+
+---
+
+## Open Phase 5 Items
+
+| ID | Description | Status |
+|----|-------------|--------|
+| P5-07 | ansible-control disk at 79.2% (was 81.5%) | Ongoing — still approaching 85% |
+| P5-10 | Pi4 node-exporter wrong-arch binary (.227) | Open |
+| P5-12 | Weekly health digest | **DONE** (this entry) |
+