tommy/homelab-configs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add P5-12 to phase5-incident-log, update INC-001 and P5-05 states

Browse Source 
P5-12 (weekly-health-digest): documented scope, deployed location, cron
schedule, ntfy topic, break-test result. Removes duplicate
monitoring/PHASE5-incidents.md — P5-12 belongs in the canonical
runbooks/phase5-incident-log.md alongside all other phase items.

INC-001 state updated: PBS rebooted May 14; both usb1-zfs and usb2-zfs
now MISSING (usb2-zfs was previously ONLINE on May 5 — new degradation).
zfs-health-check alerting every 5 min to ntfy/zfs-health.

P5-05 marked RESOLVED: all certs renewed to Jul 22 2026, confirmed via
live openssl check on 2026-05-16.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
tommy
2026-05-16 08:58:16 -05:00
parent a4726a4da6
commit 2ee7c474f5
2 changed files with 23 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files
monitoring/PHASE5-incidents.md
-57
View File
@@ -1,57 +0,0 @@
# Phase 5 — Incident & Deliverable Log
## P5-12 — Weekly Health Digest (2026-05-16)
**Deliverable:** `monitoring/weekly-health-digest.sh`
**Deployed to:** media-server (192.168.99.183) at `~/ansible/scripts/weekly-health-digest.sh`
**Schedule:** `/etc/cron.d/weekly-health-digest` — Monday 07:00 CDT
### What the digest covers
Lightweight drift-catcher. Runs read-only checks each Monday and posts a single concise message to the **ntfy `health-digest` topic** (dedicated topic — not grafana-alerts, not prometheus-alerts).
| Check | Method |
|-------|--------|
| PVE cluster reachable | Ping beast node-exporter |
| Prometheus targets all up | `up{job="node-exporter"}` query — known-persistent down (.227/.100) suppressed |
| Grafana/Prometheus firing alerts | `/api/v1/alerts` — PrometheusTargetDown/NodeDown suppressed (covered above) |
| PBS backup freshness | `backup_status` textfile metric |
| PBS ZFS pools imported | `zfs_pool_present` textfile metric (usb1-zfs, usb2-zfs) |
| Filesystem >85% (all nodes) | Prometheus node_filesystem query |
| ZFS pools ONLINE (cluster-wide) | `node_zfs_zpool_state` for degraded/faulted/suspended/unavail |
| Docker containers unhealthy/restarting | `docker ps` on media-server (known-unhealthy `immich_machine_learning` suppressed) |
| TLS cert expiry <21 days | Live openssl check on 6 representative domains |
### What the digest does NOT do
- Does NOT replace the quarterly full health check sweep.
- Does NOT check HAOS internals, Frigate camera counts, Coral inference times, or Authelia error rates.
- Does NOT check docker-node01/02 containers directly (no Docker API exposed).
- Does NOT verify Keepalived MASTER/BACKUP state.
- Does NOT check PBS datastore GC/verify schedules in detail.
These areas require a human-run full sweep (see `~/ansible/reports/health-check-*.md`).
### ntfy subscription
Subscribe ntfy client to topic `health-digest` on `192.168.99.198:8095`.
Same procedure as the `zfs-health` topic added in Phase 4D.
### Break-test result (2026-05-16 08:54 CDT)
Script ran successfully. Delivery confirmed on `health-digest` topic. Correctly flagged:
- **PBS ZFS pools MISSING** (usb1-zfs + usb2-zfs — INC-001 + new usb2-zfs failure) ✓
- **Grafana FIRING** (ContainerMemoryNearLimit, SwapHighWarning) ✓
- All healthy checks reported OK ✓
- Known-down nodes (.227 P5-10, .100 HAOS) correctly suppressed ✓
---
## Open Phase 5 Items
| ID | Description | Status |
|----|-------------|--------|
| P5-07 | ansible-control disk at 79.2% (was 81.5%) | Ongoing — still approaching 85% |
| P5-10 | Pi4 node-exporter wrong-arch binary (.227) | Open |
| P5-12 | Weekly health digest | **DONE** (this entry) |