ContainerMemoryNearLimit and ContainerMemoryAtLimit divided
container_memory_usage_bytes (which includes reclaimable inactive_file
page cache) by the container limit. On prometheus this read ~99% and
fired AtLimit critical ("OOM kill expected shortly") despite anon
working set being only ~365 MiB and memory.events oom_kill = 0.
Both rules now use container_memory_working_set_bytes, the OOM-relevant
figure (usage minus reclaimable cache). Validated with promtool and
reloaded via SIGHUP; both rules read inactive (~33%).
This versions a fix that was previously live-only at
/home/tommy/observability/prometheus/ and would silently revert on the
next repo-based redeploy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tommy
16489cabe5