tommy
6af09ef9c6
Independent-hardware systemd timer on raspberrypi (.227) probing media-server Prometheus /-/healthy every 60s; pages ntfy prometheus-critical after 3 fails and on recovery. Catches media-server-dark, which the same-host Uptime-Kuma monitor structurally cannot. Deployed + live-outage tested 2026-06-30. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
37 lines
1.8 KiB
Markdown
37 lines
1.8 KiB
Markdown
# Off-host Prometheus watchdog (dead-man's-switch)
|
|
|
|
Independent-hardware watchdog that pages if the media-server (.183) Prometheus
|
|
becomes unreachable — the one failure the same-host Uptime-Kuma monitor cannot
|
|
detect (media-server itself going dark).
|
|
|
|
**Runs on:** Pi / raspberrypi (192.168.99.227), as root via systemd timer.
|
|
|
|
## Behaviour
|
|
- Timer fires every 60s (`OnBootSec=2min`, `OnUnitActiveSec=60s`).
|
|
- Probes `http://192.168.99.183:9090/-/healthy` for the keyword `Healthy`.
|
|
- After 3 consecutive failures (~3 min) publishes a **priority-5** "Prometheus DOWN"
|
|
page to ntfy topic `prometheus-critical` (compute6, `http://192.168.99.198:8095`);
|
|
re-pages every 30 checks while still down.
|
|
- On recovery sends a **priority-4** "Prometheus recovered".
|
|
- State: `/var/lib/prom-watchdog/state` ("FAILS ALERTED").
|
|
|
|
Reaches the phone because the ntfy server has `upstream-base-url: https://ntfy.sh`
|
|
(FCM/APNS relay).
|
|
|
|
## Install (on the Pi)
|
|
```
|
|
sudo install -m 755 prometheus-watchdog.sh /usr/local/bin/prometheus-watchdog.sh
|
|
cp prom-watchdog.env.example prom-watchdog.env # set NTFY_TOKEN only if ntfy auth is enabled
|
|
sudo install -m 600 prom-watchdog.env /etc/prom-watchdog.env
|
|
sudo install -m 644 prometheus-watchdog.service /etc/systemd/system/prometheus-watchdog.service
|
|
sudo install -m 644 prometheus-watchdog.timer /etc/systemd/system/prometheus-watchdog.timer
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl enable --now prometheus-watchdog.timer
|
|
```
|
|
|
|
## Notes
|
|
- `prom-watchdog.env` has an **empty** `NTFY_TOKEN` — the `prometheus-critical`
|
|
topic is unauthenticated on the LAN ntfy. If auth is ever enabled, put the real
|
|
token in the deployed `/etc/prom-watchdog.env` (mode 600) and do **not** commit it.
|
|
- Deployed + live-outage tested 2026-06-30 (DOWN + recovered pages confirmed).
|