homelab-configs/monitoring/prometheus-offhost-watchdog/README.md

Off-host Prometheus watchdog (dead-man's-switch)

Independent-hardware watchdog that pages if the media-server (.183) Prometheus becomes unreachable — the one failure the same-host Uptime-Kuma monitor cannot detect (media-server itself going dark).

Runs on: Pi / raspberrypi (192.168.99.227), as root via systemd timer.

Behaviour

• Timer fires every 60s (OnBootSec=2min, OnUnitActiveSec=60s).
• Probes http://192.168.99.183:9090/-/healthy for the keyword Healthy.
• After 3 consecutive failures (~3 min) publishes a priority-5 "Prometheus DOWN" page to ntfy topic prometheus-critical (compute6, http://192.168.99.198:8095); re-pages every 30 checks while still down.
• On recovery sends a priority-4 "Prometheus recovered".
• State: /var/lib/prom-watchdog/state ("FAILS ALERTED").

Reaches the phone because the ntfy server has upstream-base-url: https://ntfy.sh (FCM/APNS relay).

Install (on the Pi)

sudo install -m 755 prometheus-watchdog.sh          /usr/local/bin/prometheus-watchdog.sh
cp prom-watchdog.env.example prom-watchdog.env      # set NTFY_TOKEN only if ntfy auth is enabled
sudo install -m 600 prom-watchdog.env               /etc/prom-watchdog.env
sudo install -m 644 prometheus-watchdog.service     /etc/systemd/system/prometheus-watchdog.service
sudo install -m 644 prometheus-watchdog.timer       /etc/systemd/system/prometheus-watchdog.timer
sudo systemctl daemon-reload
sudo systemctl enable --now prometheus-watchdog.timer

Notes

• prom-watchdog.env has an empty NTFY_TOKEN — the prometheus-critical topic is unauthenticated on the LAN ntfy. If auth is ever enabled, put the real token in the deployed /etc/prom-watchdog.env (mode 600) and do not commit it.
• Deployed + live-outage tested 2026-06-30 (DOWN + recovered pages confirmed).