The tracked copy was a stale pre-refactor version (no sequential-test
safety, no textfile metric). Replace with the current script plus fixes
for the UPSBatteryTestFailed false alarm:
- Poll ups.test.result until terminal ("passed"/"failed") instead of a
single racy read — CyberPower updates the string slower than the OL
transition, which recorded 0 despite a passing test.
- to_gauge matches *passed* substring, not exact "Done and passed".
- Fix cron OR-logic: setting both DOM (1-7) and DOW (0) made cron run on
EITHER (daily Jul 1-7 + Sundays). Script now self-guards to day<=7;
cron fires every Sunday of the quarter months.
- Externalize NUT_PASS to /etc/ups-quarterly-test.env (root:600) so the
password is no longer embedded in the repo. Deploy via ups-fix.sh.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reports the speedtest-tracker (on .183) newest result: flags STALE if
>14h old (2+ missed 6h cycles → scheduler stalled) or LOW if the best
of the last 4 tests falls below the 500/500 Mbps floor (best-of-4 avoids
flagging a transient ISP dip). Floors adjustable. Local SQLite read.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New section 4b flags either app backup if its newest PBS snapshot in
rust-usb is >26h old (catches a missed nightly run). Queries via the
lubelogger token (datastore-wide audit covers both groups) over keyed
SSH to docker-node01; read-only and failure-guarded.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Deployed 2026-06-30 to close compute6's image-update blind spot; was running on
compute6 but not tracked. Includes the required security_opt apparmor=unconfined
(compute6 host norm) without which Diun can't read the docker socket.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Independent-hardware systemd timer on raspberrypi (.227) probing media-server
Prometheus /-/healthy every 60s; pages ntfy prometheus-critical after 3 fails
and on recovery. Catches media-server-dark, which the same-host Uptime-Kuma
monitor structurally cannot. Deployed + live-outage tested 2026-06-30.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
P5-12 (weekly-health-digest): documented scope, deployed location, cron
schedule, ntfy topic, break-test result. Removes duplicate
monitoring/PHASE5-incidents.md — P5-12 belongs in the canonical
runbooks/phase5-incident-log.md alongside all other phase items.
INC-001 state updated: PBS rebooted May 14; both usb1-zfs and usb2-zfs
now MISSING (usb2-zfs was previously ONLINE on May 5 — new degradation).
zfs-health-check alerting every 5 min to ntfy/zfs-health.
P5-05 marked RESOLVED: all certs renewed to Jul 22 2026, confirmed via
live openssl check on 2026-05-16.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Lightweight weekly drift-catcher that posts a concise digest to the
dedicated ntfy/health-digest topic every Monday at 07:00. Covers:
PVE reachability, Prometheus target health, firing alerts, PBS backup
status, PBS ZFS pool presence, filesystem >85%, ZFS pool state, Docker
unhealthy containers, and TLS cert expiry <21d.
Does NOT replace the quarterly full health check — documented in
PHASE5-incidents.md alongside open P5-07 and P5-10 items.
Deployed: /etc/cron.d/weekly-health-digest on media-server.
Break-tested 2026-05-16: correctly flagged PBS ZFS pools MISSING and
Grafana firing alerts; suppressed known-persistent issues.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
tommy