monitoring: off-host Prometheus dead-man's-switch watchdog (Pi .227)

Independent-hardware systemd timer on raspberrypi (.227) probing media-server Prometheus /-/healthy every 60s; pages ntfy prometheus-critical after 3 fails and on recovery. Catches media-server-dark, which the same-host Uptime-Kuma monitor structurally cannot. Deployed + live-outage tested 2026-06-30. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-30 21:57:56 -05:00
parent 0be9ae426b
commit 6af09ef9c6
5 changed files with 88 additions and 0 deletions
@@ -0,0 +1,36 @@
+# Off-host Prometheus watchdog (dead-man's-switch)
+
+Independent-hardware watchdog that pages if the media-server (.183) Prometheus
+becomes unreachable — the one failure the same-host Uptime-Kuma monitor cannot
+detect (media-server itself going dark).
+
+**Runs on:** Pi / raspberrypi (192.168.99.227), as root via systemd timer.
+
+## Behaviour
+- Timer fires every 60s (`OnBootSec=2min`, `OnUnitActiveSec=60s`).
+- Probes `http://192.168.99.183:9090/-/healthy` for the keyword `Healthy`.
+- After 3 consecutive failures (~3 min) publishes a **priority-5** "Prometheus DOWN"
+  page to ntfy topic `prometheus-critical` (compute6, `http://192.168.99.198:8095`);
+  re-pages every 30 checks while still down.
+- On recovery sends a **priority-4** "Prometheus recovered".
+- State: `/var/lib/prom-watchdog/state` ("FAILS ALERTED").
+
+Reaches the phone because the ntfy server has `upstream-base-url: https://ntfy.sh`
+(FCM/APNS relay).
+
+## Install (on the Pi)
+```
+sudo install -m 755 prometheus-watchdog.sh          /usr/local/bin/prometheus-watchdog.sh
+cp prom-watchdog.env.example prom-watchdog.env      # set NTFY_TOKEN only if ntfy auth is enabled
+sudo install -m 600 prom-watchdog.env               /etc/prom-watchdog.env
+sudo install -m 644 prometheus-watchdog.service     /etc/systemd/system/prometheus-watchdog.service
+sudo install -m 644 prometheus-watchdog.timer       /etc/systemd/system/prometheus-watchdog.timer
+sudo systemctl daemon-reload
+sudo systemctl enable --now prometheus-watchdog.timer
+```
+
+## Notes
+- `prom-watchdog.env` has an **empty** `NTFY_TOKEN` — the `prometheus-critical`
+  topic is unauthenticated on the LAN ntfy. If auth is ever enabled, put the real
+  token in the deployed `/etc/prom-watchdog.env` (mode 600) and do **not** commit it.
+- Deployed + live-outage tested 2026-06-30 (DOWN + recovered pages confirmed).