Wazuh VM (vmid 109, .199) ran node-exporter v1.8.2 all along but was never in
Prometheus's scrape list -> its 2-month outage was invisible. Add 192.168.99.199:9100
to the node-exporter job; the existing NodeDown alert (up{job=node-exporter}==0, 2m,
critical -> ntfy) now covers it. Also mirror prometheus.yml into git for the first
time (only alert_rules.yml was tracked) to end live-only drift.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ContainerMemoryNearLimit and ContainerMemoryAtLimit divided
container_memory_usage_bytes (which includes reclaimable inactive_file
page cache) by the container limit. On prometheus this read ~99% and
fired AtLimit critical ("OOM kill expected shortly") despite anon
working set being only ~365 MiB and memory.events oom_kill = 0.
Both rules now use container_memory_working_set_bytes, the OOM-relevant
figure (usage minus reclaimable cache). Validated with promtool and
reloaded via SIGHUP; both rules read inactive (~33%).
This versions a fix that was previously live-only at
/home/tommy/observability/prometheus/ and would silently revert on the
next repo-based redeploy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tommy