Commit Graph

3 Commits

Author SHA1 Message Date
tommy 7178f6ceea weekly-health-digest: add internet speed check (section 4c)
Reports the speedtest-tracker (on .183) newest result: flags STALE if
>14h old (2+ missed 6h cycles → scheduler stalled) or LOW if the best
of the last 4 tests falls below the 500/500 Mbps floor (best-of-4 avoids
flagging a transient ISP dip). Floors adjustable. Local SQLite read.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 18:13:16 -05:00
tommy a567f64508 weekly-health-digest: add LubeLogger + Vaultwarden PBS backup freshness
New section 4b flags either app backup if its newest PBS snapshot in
rust-usb is >26h old (catches a missed nightly run). Queries via the
lubelogger token (datastore-wide audit covers both groups) over keyed
SSH to docker-node01; read-only and failure-guarded.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 17:39:24 -05:00
tommy a4726a4da6 monitoring: add weekly-health-digest.sh and Phase 5 incident log (P5-12)
Lightweight weekly drift-catcher that posts a concise digest to the
dedicated ntfy/health-digest topic every Monday at 07:00. Covers:
PVE reachability, Prometheus target health, firing alerts, PBS backup
status, PBS ZFS pool presence, filesystem >85%, ZFS pool state, Docker
unhealthy containers, and TLS cert expiry <21d.

Does NOT replace the quarterly full health check — documented in
PHASE5-incidents.md alongside open P5-07 and P5-10 items.

Deployed: /etc/cron.d/weekly-health-digest on media-server.
Break-tested 2026-05-16: correctly flagged PBS ZFS pools MISSING and
Grafana firing alerts; suppressed known-persistent issues.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-16 08:56:46 -05:00