Reports the speedtest-tracker (on .183) newest result: flags STALE if
>14h old (2+ missed 6h cycles → scheduler stalled) or LOW if the best
of the last 4 tests falls below the 500/500 Mbps floor (best-of-4 avoids
flagging a transient ISP dip). Floors adjustable. Local SQLite read.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New section 4b flags either app backup if its newest PBS snapshot in
rust-usb is >26h old (catches a missed nightly run). Queries via the
lubelogger token (datastore-wide audit covers both groups) over keyed
SSH to docker-node01; read-only and failure-guarded.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Deployed 2026-06-30 to close compute6's image-update blind spot; was running on
compute6 but not tracked. Includes the required security_opt apparmor=unconfined
(compute6 host norm) without which Diun can't read the docker socket.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Independent-hardware systemd timer on raspberrypi (.227) probing media-server
Prometheus /-/healthy every 60s; pages ntfy prometheus-critical after 3 fails
and on recovery. Catches media-server-dark, which the same-host Uptime-Kuma
monitor structurally cannot. Deployed + live-outage tested 2026-06-30.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
P5-12 (weekly-health-digest): documented scope, deployed location, cron
schedule, ntfy topic, break-test result. Removes duplicate
monitoring/PHASE5-incidents.md — P5-12 belongs in the canonical
runbooks/phase5-incident-log.md alongside all other phase items.
INC-001 state updated: PBS rebooted May 14; both usb1-zfs and usb2-zfs
now MISSING (usb2-zfs was previously ONLINE on May 5 — new degradation).
zfs-health-check alerting every 5 min to ntfy/zfs-health.
P5-05 marked RESOLVED: all certs renewed to Jul 22 2026, confirmed via
live openssl check on 2026-05-16.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Lightweight weekly drift-catcher that posts a concise digest to the
dedicated ntfy/health-digest topic every Monday at 07:00. Covers:
PVE reachability, Prometheus target health, firing alerts, PBS backup
status, PBS ZFS pool presence, filesystem >85%, ZFS pool state, Docker
unhealthy containers, and TLS cert expiry <21d.
Does NOT replace the quarterly full health check — documented in
PHASE5-incidents.md alongside open P5-07 and P5-10 items.
Deployed: /etc/cron.d/weekly-health-digest on media-server.
Break-tested 2026-05-16: correctly flagged PBS ZFS pools MISSING and
Grafana firing alerts; suppressed known-persistent issues.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
tommy