diff --git a/ha/automations.yaml b/ha/automations.yaml index b019b79..9dee5e2 100644 --- a/ha/automations.yaml +++ b/ha/automations.yaml @@ -46,3 +46,35 @@ Hard-refresh your browser: Ctrl+Shift+R (Win/Linux) or Cmd+Shift+R (Mac) to clear frontend cache. priority: high tags: warning,house + +- id: vault1_mount_heartbeat + alias: VAULT1 Mount – Push Heartbeat + description: "Every 5 min: if VAULT1 is CIFS-mounted, push heartbeat to Uptime Kuma. Missed heartbeats alert via ntfy." + triggers: + - platform: time_pattern + minutes: "/5" + conditions: + - condition: state + entity_id: binary_sensor.vault1_cifs_mounted + state: "on" + actions: + - action: rest_command.vault1_push_heartbeat + +- id: vault1_mount_lost_alert + alias: VAULT1 Mount – Alert on CIFS Lost + description: "Alert via ntfy if VAULT1 stops being CIFS-mounted (fallthrough to sda8 scenario)." + triggers: + - platform: state + entity_id: binary_sensor.vault1_cifs_mounted + to: "off" + for: "00:03:00" + actions: + - action: rest_command.ntfy_grafana_alerts + data: + title: "VAULT1 CIFS Lost – Frigate writing to sda8" + message: >- + /media/VAULT1 is no longer CIFS-mounted (//172.30.32.1/VAULT1 dropped). + Frigate recordings are going to sda8 (system partition) until fixed. + See RB-001 in runbooks/phase5-incident-log.md for recovery steps. + priority: high + tags: warning,house diff --git a/ha/configuration.yaml b/ha/configuration.yaml index 7e06acc..8844658 100644 --- a/ha/configuration.yaml +++ b/ha/configuration.yaml @@ -15,6 +15,16 @@ http: - 172.16.0.0/12 # <--- Trusts all Docker containers - 127.0.0.1 # <--- Localhost +command_line: + - binary_sensor: + name: "VAULT1 CIFS Mounted" + unique_id: vault1_cifs_mounted + command: "grep -c '172.30.32.1/VAULT1' /proc/mounts" + device_class: connectivity + scan_interval: 300 + payload_on: "1" + payload_off: "0" + rest_command: ntfy_grafana_alerts: url: https://ntfy.goattw.net/grafana-alerts @@ -25,3 +35,6 @@ rest_command: Title: "{{ title | default('Home Assistant') }}" Priority: "{{ priority | default('default') }}" Tags: "{{ tags | default('house') }}" + vault1_push_heartbeat: + url: "http://192.168.99.183:3001/api/push/yxhtvRPdixD4IY3Z4tVb?status=up&msg=VAULT1+CIFS+OK&ping=" + method: GET diff --git a/runbooks/phase5-incident-log.md b/runbooks/phase5-incident-log.md index 6165d18..2dc0dde 100644 --- a/runbooks/phase5-incident-log.md +++ b/runbooks/phase5-incident-log.md @@ -1,6 +1,6 @@ # Phase 5 — Incident Log / State Drift -Last updated: 2026-05-23 (INC-007 added; VAULT1 CIFS cred fixed; RB-001 credential recovery added) +Last updated: 2026-05-23 (RB-001: root cause confirmed — HAOS 17.3 supervisor mount-reload deletes cred file; VAULT1 monitor added) Purpose: Consolidates real findings from the Phase 4 worksession. Inputs to Phase 5 planning and repair. --- @@ -377,7 +377,19 @@ curl -s -H "Authorization: Bearer $TOKEN" 'http://supervisor/host/logs?lines=500 #### VAULT1 CIFS credential recovery (when credential file is lost) -The supervisor credential file at `/mnt/data/supervisor/.mounts_credentials/VAULT1` can be lost after an unclean HAOS shutdown (filesystem dirty bit). Symptoms: supervisor log shows `error 2 (No such file or directory) opening credential file` every 15 min; `/media/VAULT1` falls through to sda8 and Frigate writes recordings there instead of the USB drive. +**Root cause (identified 2026-05-23, investigated against HAOS supervisor journal):** + +The credential file loss is **NOT caused by unclean shutdown**. Confirmed: credential file survived May 6, May 21, and May 22 power-event boots (VAULT1 CIFS mounted successfully on all three). The file was deleted at **2026-05-22 00:53:31 UTC**, 13 minutes after a successful mount on the first HAOS 17.3 boot (rauc slot B). + +**What happened:** The new HAOS 17.3 supervisor (2026.05.x) performed a mount reload cycle (stop-start) on `mnt-data-supervisor-mounts-VAULT1`. The STOP step at 00:53:31 completed (`Deactivated successfully`) and deleted the credential file as part of cleanup. The immediately-following START step then failed with `error 2: No such file or directory opening credential file` — because the supervisor deleted the file but does not regenerate it from stored state on remount. + +**Trigger for the reload:** Unknown — likely the new supervisor version's startup validation of existing mounts, or a brief Samba connectivity blip that caused the supervisor to cycle the mount. VAULT2 was also reloaded at 00:54:12 but succeeded because VAULT2's credential file was not deleted (race condition or ordering difference). + +**Pattern:** This will recur every time the supervisor decides to cycle the VAULT1 CIFS mount. HAOS update to a newer supervisor version may fix or change this behavior. + +**NUT/UPS protection gap (separate issue):** HAOS has no NUT client. Unclean shutdowns from power events (cyberpower UPS→Beast→CT104 NUT chain, but HAOS is not a slave) can corrupt the journal but do NOT cause credential file loss per above analysis. Still recommended to add the HAOS NUT addon (Community store, monitor cyberpower1@192.168.99.200 as slave) to prevent hard power cuts to HAOS. + +The supervisor credential file at `/mnt/data/supervisor/.mounts_credentials/VAULT1` can be lost after the HAOS supervisor cycles the CIFS mount. Symptoms: supervisor log shows `error 2 (No such file or directory) opening credential file` every 15 min; `/media/VAULT1` falls through to sda8 and Frigate writes recordings there instead of the USB drive. To recover (requires ~2 min Frigate downtime): ```bash @@ -400,7 +412,9 @@ curl -s -X POST -H "Authorization: Bearer $TOKEN" http://supervisor/addons/ccab4 Orphan dirs on sda8 are outside Frigate's media path — handle manually (move or delete). #### Prevention / monitoring +- **Uptime Kuma "VAULT1 CIFS Mount"** push monitor (id=45, token=yxhtvRPdixD4IY3Z4tVb, 10-min heartbeat timeout): HA automation pushes heartbeat every 5 min when VAULT1 is CIFS-mounted. Missed heartbeat → Kuma → ntfy `grafana-alerts`. Added 2026-05-23. +- **HA binary_sensor `vault1_cifs_mounted`**: `grep -c '172.30.32.1/VAULT1' /proc/mounts` every 5 min. State off for >3 min → direct ntfy alert (`VAULT1 CIFS Lost – Frigate writing to sda8`). Added 2026-05-23. - Uptime Kuma "Frigate Stats" monitor (http://192.168.99.100:5000/api/stats, 60s, 2 retries) → ntfy `grafana-alerts` (added 2026-05-23) - Uptime Kuma "Home Assistant" monitor (http://192.168.99.100:8123, 60s) → ntfy `grafana-alerts` (added 2026-05-23) -- VAULT1 CIFS credential file issue is separate — supervisor retries every 15 min with `error 2 (No such file or directory)`. Does not affect Frigate (Frigate uses direct ext4 mount, not CIFS). Fix by recreating credential file if CIFS mount is intentional. - USB enclosure is self-powered (own PSU) and directly connected — no hub to replace. If repeated disconnects occur, suspect: (a) PSU/power cable on enclosure, (b) USB cable quality, (c) motherboard USB port. +- **Pending**: Add HAOS NUT addon (Community store) monitoring `cyberpower1@192.168.99.200` as slave to prevent hard power cuts from UPS events. NUT chain currently: Beast (master) → CT104 (slave). HAOS is not in the chain.