monitoring: add weekly-health-digest.sh and Phase 5 incident log (P5-12)
Lightweight weekly drift-catcher that posts a concise digest to the dedicated ntfy/health-digest topic every Monday at 07:00. Covers: PVE reachability, Prometheus target health, firing alerts, PBS backup status, PBS ZFS pool presence, filesystem >85%, ZFS pool state, Docker unhealthy containers, and TLS cert expiry <21d. Does NOT replace the quarterly full health check — documented in PHASE5-incidents.md alongside open P5-07 and P5-10 items. Deployed: /etc/cron.d/weekly-health-digest on media-server. Break-tested 2026-05-16: correctly flagged PBS ZFS pools MISSING and Grafana firing alerts; suppressed known-persistent issues. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
tommy
@@ -0,0 +1,57 @@
|
||||
# Phase 5 — Incident & Deliverable Log
|
||||
|
||||
## P5-12 — Weekly Health Digest (2026-05-16)
|
||||
|
||||
**Deliverable:** `monitoring/weekly-health-digest.sh`
|
||||
**Deployed to:** media-server (192.168.99.183) at `~/ansible/scripts/weekly-health-digest.sh`
|
||||
**Schedule:** `/etc/cron.d/weekly-health-digest` — Monday 07:00 CDT
|
||||
|
||||
### What the digest covers
|
||||
|
||||
Lightweight drift-catcher. Runs read-only checks each Monday and posts a single concise message to the **ntfy `health-digest` topic** (dedicated topic — not grafana-alerts, not prometheus-alerts).
|
||||
|
||||
| Check | Method |
|
||||
|-------|--------|
|
||||
| PVE cluster reachable | Ping beast node-exporter |
|
||||
| Prometheus targets all up | `up{job="node-exporter"}` query — known-persistent down (.227/.100) suppressed |
|
||||
| Grafana/Prometheus firing alerts | `/api/v1/alerts` — PrometheusTargetDown/NodeDown suppressed (covered above) |
|
||||
| PBS backup freshness | `backup_status` textfile metric |
|
||||
| PBS ZFS pools imported | `zfs_pool_present` textfile metric (usb1-zfs, usb2-zfs) |
|
||||
| Filesystem >85% (all nodes) | Prometheus node_filesystem query |
|
||||
| ZFS pools ONLINE (cluster-wide) | `node_zfs_zpool_state` for degraded/faulted/suspended/unavail |
|
||||
| Docker containers unhealthy/restarting | `docker ps` on media-server (known-unhealthy `immich_machine_learning` suppressed) |
|
||||
| TLS cert expiry <21 days | Live openssl check on 6 representative domains |
|
||||
|
||||
### What the digest does NOT do
|
||||
|
||||
- Does NOT replace the quarterly full health check sweep.
|
||||
- Does NOT check HAOS internals, Frigate camera counts, Coral inference times, or Authelia error rates.
|
||||
- Does NOT check docker-node01/02 containers directly (no Docker API exposed).
|
||||
- Does NOT verify Keepalived MASTER/BACKUP state.
|
||||
- Does NOT check PBS datastore GC/verify schedules in detail.
|
||||
|
||||
These areas require a human-run full sweep (see `~/ansible/reports/health-check-*.md`).
|
||||
|
||||
### ntfy subscription
|
||||
|
||||
Subscribe ntfy client to topic `health-digest` on `192.168.99.198:8095`.
|
||||
Same procedure as the `zfs-health` topic added in Phase 4D.
|
||||
|
||||
### Break-test result (2026-05-16 08:54 CDT)
|
||||
|
||||
Script ran successfully. Delivery confirmed on `health-digest` topic. Correctly flagged:
|
||||
- **PBS ZFS pools MISSING** (usb1-zfs + usb2-zfs — INC-001 + new usb2-zfs failure) ✓
|
||||
- **Grafana FIRING** (ContainerMemoryNearLimit, SwapHighWarning) ✓
|
||||
- All healthy checks reported OK ✓
|
||||
- Known-down nodes (.227 P5-10, .100 HAOS) correctly suppressed ✓
|
||||
|
||||
---
|
||||
|
||||
## Open Phase 5 Items
|
||||
|
||||
| ID | Description | Status |
|
||||
|----|-------------|--------|
|
||||
| P5-07 | ansible-control disk at 79.2% (was 81.5%) | Ongoing — still approaching 85% |
|
||||
| P5-10 | Pi4 node-exporter wrong-arch binary (.227) | Open |
|
||||
| P5-12 | Weekly health digest | **DONE** (this entry) |
|
||||
|
||||
Executable
+262
@@ -0,0 +1,262 @@
|
||||
#!/bin/bash
|
||||
# weekly-health-digest.sh — P5-12
|
||||
# Lightweight weekly drift-catcher. Read-only. Posts a concise digest to ntfy/health-digest.
|
||||
# Does NOT replace the quarterly full health check.
|
||||
# Schedule: Monday 07:00 via /etc/cron.d/weekly-health-digest
|
||||
# Runs on: media-server (192.168.99.183)
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
NTFY_URL="http://192.168.99.198:8095/health-digest"
|
||||
PROM_URL="http://localhost:9090"
|
||||
GRAFANA_URL="http://localhost:3005"
|
||||
TIMEOUT=10 # seconds per HTTP request before giving up on that check
|
||||
|
||||
# Node-exporter targets (from prometheus.yml)
|
||||
NODE_EXPORTER_TARGETS=(
|
||||
"192.168.99.200:9100" # beast
|
||||
"192.168.99.192:9100" # compute2
|
||||
"192.168.99.193:9100" # compute3
|
||||
"192.168.99.194:9100" # compute4
|
||||
"192.168.99.153:9100" # pbs
|
||||
"192.168.99.183:9100" # media-server
|
||||
"192.168.99.186:9100" # docker-node01
|
||||
"192.168.99.187:9100" # docker-node02
|
||||
"192.168.99.190:9100" # ansible-control
|
||||
"192.168.99.227:9100" # pi4 (P5-10 — expected down)
|
||||
"192.168.99.196:9100" # compute5
|
||||
"192.168.99.198:9100" # compute6
|
||||
"192.168.99.100:9100" # haos (expected 500)
|
||||
)
|
||||
|
||||
logger -t weekly-health-digest "starting weekly health digest"
|
||||
|
||||
# ── helpers ─────────────────────────────────────────────────────────────────
|
||||
|
||||
prom_query() {
|
||||
# $1 = PromQL query; returns JSON result array or empty string on failure
|
||||
curl -sf --max-time "$TIMEOUT" \
|
||||
"${PROM_URL}/api/v1/query?query=$(python3 -c "import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1]))" "$1")" \
|
||||
2>/dev/null | python3 -c "import json,sys; d=json.load(sys.stdin); print(json.dumps(d['data']['result']))" 2>/dev/null || echo "[]"
|
||||
}
|
||||
|
||||
prom_scalar() {
|
||||
# Returns first value from a query as a float, or -1 on failure
|
||||
local result
|
||||
result=$(prom_query "$1")
|
||||
python3 -c "
|
||||
import json,sys
|
||||
try:
|
||||
d=json.loads('$result')
|
||||
if d: print(float(d[0]['value'][1]))
|
||||
else: print(-1)
|
||||
except: print(-1)
|
||||
" 2>/dev/null || echo "-1"
|
||||
}
|
||||
|
||||
OK="OK"
|
||||
FLAG=""
|
||||
LINES=()
|
||||
ISSUES=0
|
||||
|
||||
add_line() {
|
||||
local label="$1" status="$2" detail="${3:-}"
|
||||
if [ "$status" = "OK" ]; then
|
||||
LINES+=(" ${label}: OK${detail:+ — $detail}")
|
||||
else
|
||||
LINES+=(" ${label}: *** ${status}${detail:+ — $detail} ***")
|
||||
ISSUES=$((ISSUES + 1))
|
||||
fi
|
||||
}
|
||||
|
||||
# ── 1. Proxmox quorum ────────────────────────────────────────────────────────
|
||||
# We can't SSH to PVE nodes from here directly, so we infer quorum from node
|
||||
# reachability. If beast (the PVE master) is up and answering node-exporter,
|
||||
# the cluster is likely healthy. Flag if beast is unreachable.
|
||||
beast_up=$(curl -sf --max-time 5 "http://192.168.99.200:9100/metrics" 2>/dev/null | grep -c "node_boot_time_seconds" || true)
|
||||
if [ "${beast_up}" -ge 1 ]; then
|
||||
add_line "PVE/quorum" "OK" "beast reachable"
|
||||
else
|
||||
add_line "PVE/quorum" "beast unreachable — quorum check not possible"
|
||||
fi
|
||||
|
||||
# ── 2. Prometheus node-exporter targets ─────────────────────────────────────
|
||||
targets_result=$(prom_query 'up{job="node-exporter"}')
|
||||
targets_down=$(echo "$targets_result" | python3 -c "
|
||||
import json,sys
|
||||
try:
|
||||
d=json.loads(sys.stdin.read())
|
||||
down=[r['metric'].get('instance','?') for r in d if float(r['value'][1])==0]
|
||||
# Filter known-persistent issues that don't need weekly noise
|
||||
known_down={'192.168.99.227:9100','192.168.99.100:9100'}
|
||||
new_down=[h for h in down if h not in known_down]
|
||||
print(','.join(new_down) if new_down else '')
|
||||
# Always report known-down count for awareness
|
||||
known_count=len([h for h in down if h in known_down])
|
||||
if known_count: print(f'(+{known_count} known-down: P5-10/HAOS)', end='', file=__import__('sys').stderr)
|
||||
except: print('')
|
||||
" 2>/tmp/weekly-health-digest-known.txt)
|
||||
known_note=$(cat /tmp/weekly-health-digest-known.txt 2>/dev/null || true)
|
||||
rm -f /tmp/weekly-health-digest-known.txt
|
||||
|
||||
if [ -z "$targets_down" ]; then
|
||||
add_line "node-exporter targets" "OK" "${known_note:-all up (excl. 2 known-down)}"
|
||||
else
|
||||
add_line "node-exporter targets" "NEW targets down: ${targets_down}" "${known_note}"
|
||||
fi
|
||||
|
||||
# ── 3. Grafana/Prometheus firing alerts ─────────────────────────────────────
|
||||
firing_result=$(curl -sf --max-time "$TIMEOUT" \
|
||||
"${PROM_URL}/api/v1/alerts" 2>/dev/null | python3 -c "
|
||||
import json,sys
|
||||
try:
|
||||
d=json.load(sys.stdin)
|
||||
firing=[a['labels'].get('alertname','?') for a in d['data']['alerts'] if a['state']=='firing']
|
||||
# Suppress known-persistent alerts that are tracked elsewhere
|
||||
known_ok={'PrometheusTargetDown','NodeDown'} # covered in target check above
|
||||
new_firing=[a for a in firing if a not in known_ok]
|
||||
if new_firing:
|
||||
print(','.join(sorted(set(new_firing))))
|
||||
else:
|
||||
suppressed=len([a for a in firing if a in known_ok])
|
||||
print(f'none (suppressed {suppressed} known)')
|
||||
except: print('query_failed')
|
||||
" 2>/dev/null || echo "query_failed")
|
||||
|
||||
if echo "$firing_result" | grep -qE "none|^$"; then
|
||||
add_line "Grafana alerts" "OK" "$firing_result"
|
||||
elif [ "$firing_result" = "query_failed" ]; then
|
||||
add_line "Grafana alerts" "Prometheus unreachable"
|
||||
else
|
||||
add_line "Grafana alerts" "FIRING: ${firing_result}"
|
||||
fi
|
||||
|
||||
# ── 4. PBS datastores — backup freshness >48h ────────────────────────────────
|
||||
# Check local backup metric: backup_status textfile on media-server node-exporter
|
||||
backup_status=$(prom_scalar 'backup_status{instance="192.168.99.183:9100"}')
|
||||
if python3 -c "import sys; sys.exit(0 if float('$backup_status') >= 1 else 1)" 2>/dev/null; then
|
||||
add_line "PBS/backups" "OK" "backup_status metric healthy"
|
||||
else
|
||||
add_line "PBS/backups" "backup_status=0 or unavailable — check PBS and local backup jobs"
|
||||
fi
|
||||
|
||||
# Also check PBS ZFS pool presence (covers the INC-001/usb1-zfs and usb2-zfs situation)
|
||||
pbs_zfs=$(prom_query 'zfs_pool_present')
|
||||
pbs_zfs_missing=$(echo "$pbs_zfs" | python3 -c "
|
||||
import json,sys
|
||||
try:
|
||||
d=json.loads(sys.stdin.read())
|
||||
missing=[r['metric'].get('pool','?') for r in d if float(r['value'][1])==0]
|
||||
print(','.join(missing) if missing else '')
|
||||
except: print('')
|
||||
" 2>/dev/null || echo "")
|
||||
if [ -z "$pbs_zfs_missing" ]; then
|
||||
add_line "PBS ZFS pools" "OK" "usb1-zfs + usb2-zfs imported"
|
||||
else
|
||||
add_line "PBS ZFS pools" "MISSING: ${pbs_zfs_missing}" "pools not imported on PBS — check enclosure/cables"
|
||||
fi
|
||||
|
||||
# ── 5. Filesystem >85% ───────────────────────────────────────────────────────
|
||||
fs_over=$(prom_query '(1 - node_filesystem_avail_bytes{fstype!~"tmpfs|fuse.lxcfs|overlay|squashfs"} / node_filesystem_size_bytes{fstype!~"tmpfs|fuse.lxcfs|overlay|squashfs"}) * 100 > 85' | python3 -c "
|
||||
import json,sys
|
||||
try:
|
||||
d=json.loads(sys.stdin.read())
|
||||
items=[f'{r[\"metric\"].get(\"instance\",\"?\").split(\":\")[0]}:{r[\"metric\"].get(\"mountpoint\",\"?\")} {float(r[\"value\"][1]):.0f}%' for r in d]
|
||||
print(', '.join(items) if items else '')
|
||||
except: print('')
|
||||
" 2>/dev/null || echo "")
|
||||
if [ -z "$fs_over" ]; then
|
||||
add_line "Disk usage >85%" "OK" "no filesystem over threshold"
|
||||
else
|
||||
add_line "Disk usage >85%" "OVER THRESHOLD: ${fs_over}"
|
||||
fi
|
||||
|
||||
# ── 6. ZFS pools ONLINE (cluster-wide) ──────────────────────────────────────
|
||||
zfs_bad=$(prom_query 'node_zfs_zpool_state{state=~"degraded|faulted|suspended|unavail|offline|removed"} == 1' | python3 -c "
|
||||
import json,sys
|
||||
try:
|
||||
d=json.loads(sys.stdin.read())
|
||||
items=[f'{r[\"metric\"].get(\"instance\",\"?\").split(\":\")[0]}:{r[\"metric\"].get(\"zpool\",\"?\")}({r[\"metric\"].get(\"state\",\"?\")})' for r in d]
|
||||
print(', '.join(items) if items else '')
|
||||
except: print('')
|
||||
" 2>/dev/null || echo "")
|
||||
if [ -z "$zfs_bad" ]; then
|
||||
add_line "ZFS pools" "OK" "das-mirror, compute3-zfs all ONLINE"
|
||||
else
|
||||
add_line "ZFS pools" "DEGRADED/FAULTED: ${zfs_bad}"
|
||||
fi
|
||||
|
||||
# ── 7. Docker unhealthy/restarting ──────────────────────────────────────────
|
||||
unhealthy=$(docker ps --format "{{.Names}}\t{{.Status}}" 2>/dev/null | \
|
||||
awk '/unhealthy|restarting/{print $1}' | \
|
||||
grep -v "immich_machine_learning" | \
|
||||
tr '\n' ',' | sed 's/,$//' || true)
|
||||
# immich_machine_learning is known-unhealthy — suppress
|
||||
if [ -z "$unhealthy" ]; then
|
||||
add_line "Docker (local)" "OK" "no new unhealthy/restarting containers"
|
||||
else
|
||||
add_line "Docker (local)" "UNHEALTHY/RESTARTING: ${unhealthy}"
|
||||
fi
|
||||
|
||||
# ── 8. TLS cert expiry <21 days ─────────────────────────────────────────────
|
||||
# Check live cert expiry on a representative set of domains
|
||||
cert_warn=""
|
||||
for domain in traefik.goattw.net auth.goattw.net grafana.goattw.net ha.goattw.net nvr.goattw.net home.goattw.net; do
|
||||
expiry=$(echo | timeout 8 openssl s_client -connect "${domain}:443" -servername "${domain}" 2>/dev/null \
|
||||
| openssl x509 -noout -enddate 2>/dev/null \
|
||||
| cut -d= -f2)
|
||||
if [ -n "$expiry" ]; then
|
||||
days=$(python3 -c "
|
||||
from datetime import datetime
|
||||
try:
|
||||
exp=datetime.strptime('$expiry','%b %d %H:%M:%S %Y %Z')
|
||||
now=datetime.utcnow()
|
||||
print(int((exp-now).days))
|
||||
except: print(999)
|
||||
" 2>/dev/null || echo "999")
|
||||
if [ "$days" -lt 21 ] 2>/dev/null; then
|
||||
cert_warn="${cert_warn}${domain}(${days}d) "
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
if [ -z "$cert_warn" ]; then
|
||||
add_line "TLS cert expiry" "OK" "all checked certs >21 days"
|
||||
else
|
||||
add_line "TLS cert expiry" "EXPIRING SOON: ${cert_warn}"
|
||||
fi
|
||||
|
||||
# ── Compose and send digest ──────────────────────────────────────────────────
|
||||
NOW=$(date '+%Y-%m-%d %H:%M %Z')
|
||||
|
||||
if [ "$ISSUES" -eq 0 ]; then
|
||||
PRIORITY="low"
|
||||
SUBJECT="Weekly Health Digest ${NOW} — ALL OK"
|
||||
TAGS="white_check_mark"
|
||||
else
|
||||
PRIORITY="high"
|
||||
SUBJECT="Weekly Health Digest ${NOW} — ${ISSUES} ISSUE(S) FLAGGED"
|
||||
TAGS="rotating_light"
|
||||
fi
|
||||
|
||||
BODY=$(printf '%s\n' "${LINES[@]}")
|
||||
|
||||
logger -t weekly-health-digest "digest complete: issues=${ISSUES} priority=${PRIORITY}"
|
||||
|
||||
# Post to ntfy health-digest topic
|
||||
HTTP_STATUS=$(curl -sf --max-time 15 \
|
||||
-H "Title: ${SUBJECT}" \
|
||||
-H "Priority: ${PRIORITY}" \
|
||||
-H "Tags: ${TAGS}" \
|
||||
-d "${BODY}" \
|
||||
"${NTFY_URL}" \
|
||||
-o /dev/null -w "%{http_code}" 2>/dev/null || echo "000")
|
||||
|
||||
if [ "$HTTP_STATUS" = "200" ] || [ "$HTTP_STATUS" = "201" ]; then
|
||||
logger -t weekly-health-digest "ntfy delivery OK (HTTP ${HTTP_STATUS})"
|
||||
else
|
||||
logger -t weekly-health-digest "WARNING: ntfy delivery failed (HTTP ${HTTP_STATUS})"
|
||||
fi
|
||||
|
||||
# Always exit 0 — cron should never see a failure from a read-only digest
|
||||
exit 0
|
||||
Reference in New Issue
Block a user