# Off-host Prometheus watchdog (dead-man's-switch)

Independent-hardware watchdog that pages if the media-server (.183) Prometheus
becomes unreachable — the one failure the same-host Uptime-Kuma monitor cannot
detect (media-server itself going dark).

**Runs on:** Pi / raspberrypi (192.168.99.227), as root via systemd timer.

## Behaviour
- Timer fires every 60s (`OnBootSec=2min`, `OnUnitActiveSec=60s`).
- Probes `http://192.168.99.183:9090/-/healthy` for the keyword `Healthy`.
- After 3 consecutive failures (~3 min) publishes a **priority-5** "Prometheus DOWN"
  page to ntfy topic `prometheus-critical` (compute6, `http://192.168.99.198:8095`);
  re-pages every 30 checks while still down.
- On recovery sends a **priority-4** "Prometheus recovered".
- State: `/var/lib/prom-watchdog/state` ("FAILS ALERTED").

Reaches the phone because the ntfy server has `upstream-base-url: https://ntfy.sh`
(FCM/APNS relay).

## Install (on the Pi)
```
sudo install -m 755 prometheus-watchdog.sh          /usr/local/bin/prometheus-watchdog.sh
cp prom-watchdog.env.example prom-watchdog.env      # set NTFY_TOKEN only if ntfy auth is enabled
sudo install -m 600 prom-watchdog.env               /etc/prom-watchdog.env
sudo install -m 644 prometheus-watchdog.service     /etc/systemd/system/prometheus-watchdog.service
sudo install -m 644 prometheus-watchdog.timer       /etc/systemd/system/prometheus-watchdog.timer
sudo systemctl daemon-reload
sudo systemctl enable --now prometheus-watchdog.timer
```

## Notes
- `prom-watchdog.env` has an **empty** `NTFY_TOKEN` — the `prometheus-critical`
  topic is unauthenticated on the LAN ntfy. If auth is ever enabled, put the real
  token in the deployed `/etc/prom-watchdog.env` (mode 600) and do **not** commit it.
- Deployed + live-outage tested 2026-06-30 (DOWN + recovered pages confirmed).