Commit Graph

2 Commits

Author SHA1 Message Date
tommy a567f64508 weekly-health-digest: add LubeLogger + Vaultwarden PBS backup freshness
New section 4b flags either app backup if its newest PBS snapshot in
rust-usb is >26h old (catches a missed nightly run). Queries via the
lubelogger token (datastore-wide audit covers both groups) over keyed
SSH to docker-node01; read-only and failure-guarded.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 17:39:24 -05:00
tommy a4726a4da6 monitoring: add weekly-health-digest.sh and Phase 5 incident log (P5-12)
Lightweight weekly drift-catcher that posts a concise digest to the
dedicated ntfy/health-digest topic every Monday at 07:00. Covers:
PVE reachability, Prometheus target health, firing alerts, PBS backup
status, PBS ZFS pool presence, filesystem >85%, ZFS pool state, Docker
unhealthy containers, and TLS cert expiry <21d.

Does NOT replace the quarterly full health check — documented in
PHASE5-incidents.md alongside open P5-07 and P5-10 items.

Deployed: /etc/cron.d/weekly-health-digest on media-server.
Break-tested 2026-05-16: correctly flagged PBS ZFS pools MISSING and
Grafana firing alerts; suppressed known-persistent issues.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-16 08:56:46 -05:00