Rebooted 2026-07-01; post-boot checks green (nvidia/DKMS, 54 containers,
18 Prom targets, 0 failed units, /mnt/media). Documents the gluetun
caveat: a stale PIA WG peer does not survive a reboot -> run
pia-wg/regen-and-up.sh as a required post-reboot step.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Deployed 2026-06-30 to close compute6's image-update blind spot; was running on
compute6 but not tracked. Includes the required security_opt apparmor=unconfined
(compute6 host norm) without which Diun can't read the docker socket.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Old bouncer key was committed in cleartext (hist 7fac4fc) and is now rotated
(new traefik-bouncer-v2 + old bouncer deleted, verified 200 on both nodes).
Config now references /plugins-storage/crowdsec_lapi_key (mode-600, gitignored,
lives only on the nodes). Old key in history is inert post-rotation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replaces the Go pia-wg-config (Go 1.23 rejects PIA legacy-CN no-SAN certs) with
PIA's official CA-pinned curl flow. Adds CA-aware region auto-pick so ExpressVPN-CA
migrated regions (e.g. us_chicago) auto-skip; region moved to us_denver. Tunnel
restored + egress verified 2026-06-30.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Independent-hardware systemd timer on raspberrypi (.227) probing media-server
Prometheus /-/healthy every 60s; pages ntfy prometheus-critical after 3 fails
and on recovery. Catches media-server-dark, which the same-host Uptime-Kuma
monitor structurally cannot. Deployed + live-outage tested 2026-06-30.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
services.yaml now references key: "{{HOMEPAGE_VAR_HASS_TOKEN}}" instead of
the inline long-lived HA JWT (which was exposed in this public-fronted repo
and has been rotated + revoked). Secret now lives only in observability/.env
(untracked, mode 600). Add .gitignore for .env/secrets so this can't recur.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a command_line sensor (HAOS Media Partition Free, df of /media =
sda8) and an automation that stops the Frigate add-on when free space
drops below 50 GB for 2 min. Protects HAOS Core from the VAULT1 CIFS
fallthrough fill scenario (RB-001). Fires regardless of CIFS mount
state — the mount sensor can be stale during the crisis. Stop-only +
max-priority ntfy alert; no auto-remount (needs admin token).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ContainerMemoryNearLimit and ContainerMemoryAtLimit divided
container_memory_usage_bytes (which includes reclaimable inactive_file
page cache) by the container limit. On prometheus this read ~99% and
fired AtLimit critical ("OOM kill expected shortly") despite anon
working set being only ~365 MiB and memory.events oom_kill = 0.
Both rules now use container_memory_working_set_bytes, the OOM-relevant
figure (usage minus reclaimable cache). Validated with promtool and
reloaded via SIGHUP; both rules read inactive (~33%).
This versions a fix that was previously live-only at
/home/tommy/observability/prometheus/ and would silently revert on the
next repo-based redeploy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
HA: add binary_sensor.vault1_cifs_mounted (grep /proc/mounts every 5 min),
rest_command.vault1_push_heartbeat, and two automations — heartbeat push to
Uptime Kuma push monitor (id=45, 10-min timeout) and direct ntfy alert if
CIFS drops for >3 min. Catches the sda8-fallthrough scenario in <8 min.
Runbooks: root cause of credential file loss is NOT unclean shutdown but
the HAOS 17.3 supervisor (2026.05.x) deleting the cred file during mount
reload cycles. Exact moment: 2026-05-22 00:53:31 UTC, 13 min after the
HAOS 17.3 first-boot successful mount. NUT gap documented (HAOS not a NUT
slave); recommended to add Community NUT addon targeting cyberpower1@.200.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
VAULT1 supervisor mount had missing credential file since at least
2026-05-01 (lost after unclean shutdown). 22 days of Frigate recordings
accumulated on sda8 (299 GB) instead of the 4 TB USB drive. Fixed via
PUT /mounts/VAULT1 with hassio credentials; mount now shows 3.6T/915G.
Orphan dirs documented; RB-001 updated with credential recovery steps.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Symptom→Diagnosis→Recovery for the 2026-05-22 7.5h Frigate blackout.
Key findings: JMicron JMS56x enclosure (2-5) is self-powered and directly
connected; VAULT1 CIFS credential-file errors are a separate pre-existing
config drift, not the root cause. Also adds VAULT1 CIFS drift entry.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add rest_command.ntfy_grafana_alerts in configuration.yaml
(POST to https://ntfy.goattw.net/grafana-alerts)
- Add ha_core_version_update_alert automation in automations.yaml
Triggers on update.home_assistant_core_update installed_version change
Sends high-priority ntfy alert with old→new version and browser
hard-refresh reminder (Ctrl+Shift+R / Cmd+Shift+R)
INC-006: PBS upsmon DEADTIME=15s caused autonomous shutdown when Beast
NUT server was briefly unreachable during reboot. Fixed 2026-05-16 by
setting DEADTIME 300 on PBS.
INC-001: usb1-zfs and usb2-zfs manually imported 2026-05-16 09:14 CDT
after May 14 reboot left both MISSING. Both ONLINE, no errors. Scrubs
initiated. Cachefile updated at /etc/zfs/zpool.cache.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
P5-12 (weekly-health-digest): documented scope, deployed location, cron
schedule, ntfy topic, break-test result. Removes duplicate
monitoring/PHASE5-incidents.md — P5-12 belongs in the canonical
runbooks/phase5-incident-log.md alongside all other phase items.
INC-001 state updated: PBS rebooted May 14; both usb1-zfs and usb2-zfs
now MISSING (usb2-zfs was previously ONLINE on May 5 — new degradation).
zfs-health-check alerting every 5 min to ntfy/zfs-health.
P5-05 marked RESOLVED: all certs renewed to Jul 22 2026, confirmed via
live openssl check on 2026-05-16.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Lightweight weekly drift-catcher that posts a concise digest to the
dedicated ntfy/health-digest topic every Monday at 07:00. Covers:
PVE reachability, Prometheus target health, firing alerts, PBS backup
status, PBS ZFS pool presence, filesystem >85%, ZFS pool state, Docker
unhealthy containers, and TLS cert expiry <21d.
Does NOT replace the quarterly full health check — documented in
PHASE5-incidents.md alongside open P5-07 and P5-10 items.
Deployed: /etc/cron.d/weekly-health-digest on media-server.
Break-tested 2026-05-16: correctly flagged PBS ZFS pools MISSING and
Grafana firing alerts; suppressed known-persistent issues.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
platform quirk 'simple suspend' is applied by PVE kernel automatically
for i7-13700T platform (both nvme0 and nvme1). Not a cmdline parameter;
/etc/kernel/cmdline absent. Persists across kernel updates by default.
Verified: dmesg confirms quirk active on both drives at current boot.
P5-11 status: monitor only, no user action required.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
W11 (compute6 pve-firewall xtables conflicts): race condition between
pve-firewall's iptables_restore (no -w flag) and Docker's iptables
backend. Self-resolves within 10-120s; no firewall gap; no action taken.
Conflicts align to :58 minute mark = Docker daemon start time at last
compute6 boot (Apr 19 12:58).
W12 (compute2 corosync-qdevice boot failures): qnetd on PBS (.153);
PBS has crash-rebooted 34 times since Apr 21, each correlated with USB
ZFS import failure (same as C1, May 5 health check). Each PBS reboot
drops qnetd for ~60s; cluster quorum unaffected (6 node votes, threshold
3). No action this phase. Phase 5 items: fix PBS USB ZFS instability;
optionally migrate qnetd from PBS to Pi4 (.227).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
node02 was missing two blocks from node01 (canonical):
- strip-trailing-dot-speedtest middleware (regex redirect for speedtest.goattw.net. URLs)
- speedtest-trailing-dot router (catches trailing-dot Host header variant)
crowdsecLapiHost intentionally differs: node01 uses Docker service name
(crowdsec:8080, container on same host); node02 points to node01 IP
(192.168.99.186:8081, node02 has no local CrowdSec instance).
Added traefik-drift-check.sh — runs daily at 06:00 on ansible-control,
diffs both configs (excluding known crowdsecLapiHost difference),
posts to ntfy homelab-alerts on unexpected divergence.
Traefik hot-reloaded on node02 via SIGHUP — no restart required.
- Fix silent failure: script now posts to dedicated zfs-health ntfy topic
instead of grafana-alerts catch-all (pools were offline 12+ hours
undetected because alerts were buried in Grafana noise)
- Three explicit states: ONLINE (silent), DEGRADED (high priority),
MISSING (urgent priority) — empty zpool list output is now a MISSING
alert, not silently ignored
- Textfile metrics written atomically after loop completes only:
zfs_pool_present{pool=X} 0|1 and zfs_health_last_run_seconds
- Added trap cleanup so mid-script crash leaves previous .prom intact
- Logs each pool state to syslog via logger -t zfs-health-check
- Remove duplicate cron entry running as tommy (was firing twice per tick)
- Enable node_exporter textfile collector for Prometheus scraping
Incident: usb1-zfs and usb2-zfs offline since PBS boot (missing cachefile).
Imported and cachefile regenerated in this session. No data errors.
Refs: 2026-05-05 health check CRITICAL C1/C4
tommy
tommy