ha: VAULT1 CIFS mount monitor; runbooks: supervisor root-cause

HA: add binary_sensor.vault1_cifs_mounted (grep /proc/mounts every 5 min),
rest_command.vault1_push_heartbeat, and two automations — heartbeat push to
Uptime Kuma push monitor (id=45, 10-min timeout) and direct ntfy alert if
CIFS drops for >3 min. Catches the sda8-fallthrough scenario in <8 min.

Runbooks: root cause of credential file loss is NOT unclean shutdown but
the HAOS 17.3 supervisor (2026.05.x) deleting the cred file during mount
reload cycles. Exact moment: 2026-05-22 00:53:31 UTC, 13 min after the
HAOS 17.3 first-boot successful mount. NUT gap documented (HAOS not a NUT
slave); recommended to add Community NUT addon targeting cyberpower1@.200.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

ha/automations.yaml

@@ -46,3 +46,35 @@
         Hard-refresh your browser: Ctrl+Shift+R (Win/Linux) or Cmd+Shift+R (Mac) to clear frontend cache.
       priority: high
       tags: warning,house
+
+- id: vault1_mount_heartbeat
+  alias: VAULT1 Mount – Push Heartbeat
+  description: "Every 5 min: if VAULT1 is CIFS-mounted, push heartbeat to Uptime Kuma. Missed heartbeats alert via ntfy."
+  triggers:
+  - platform: time_pattern
+    minutes: "/5"
+  conditions:
+  - condition: state
+    entity_id: binary_sensor.vault1_cifs_mounted
+    state: "on"
+  actions:
+  - action: rest_command.vault1_push_heartbeat
+
+- id: vault1_mount_lost_alert
+  alias: VAULT1 Mount – Alert on CIFS Lost
+  description: "Alert via ntfy if VAULT1 stops being CIFS-mounted (fallthrough to sda8 scenario)."
+  triggers:
+  - platform: state
+    entity_id: binary_sensor.vault1_cifs_mounted
+    to: "off"
+    for: "00:03:00"
+  actions:
+  - action: rest_command.ntfy_grafana_alerts
+    data:
+      title: "VAULT1 CIFS Lost – Frigate writing to sda8"
+      message: >-
+        /media/VAULT1 is no longer CIFS-mounted (//172.30.32.1/VAULT1 dropped).
+        Frigate recordings are going to sda8 (system partition) until fixed.
+        See RB-001 in runbooks/phase5-incident-log.md for recovery steps.
+      priority: high
+      tags: warning,house

ha/configuration.yaml

@@ -15,6 +15,16 @@ http:
     - 172.16.0.0/12 # <--- Trusts all Docker containers
     - 127.0.0.1 # <--- Localhost
 
+command_line:
+  - binary_sensor:
+      name: "VAULT1 CIFS Mounted"
+      unique_id: vault1_cifs_mounted
+      command: "grep -c '172.30.32.1/VAULT1' /proc/mounts"
+      device_class: connectivity
+      scan_interval: 300
+      payload_on: "1"
+      payload_off: "0"
+
 rest_command:
   ntfy_grafana_alerts:
     url: https://ntfy.goattw.net/grafana-alerts
@@ -25,3 +35,6 @@ rest_command:
       Title: "{{ title | default('Home Assistant') }}"
       Priority: "{{ priority | default('default') }}"
       Tags: "{{ tags | default('house') }}"
+  vault1_push_heartbeat:
+    url: "http://192.168.99.183:3001/api/push/yxhtvRPdixD4IY3Z4tVb?status=up&msg=VAULT1+CIFS+OK&ping="
+    method: GET

runbooks/phase5-incident-log.md

@@ -1,6 +1,6 @@
 # Phase 5 — Incident Log / State Drift
 
-Last updated: 2026-05-23 (INC-007 added; VAULT1 CIFS cred fixed; RB-001 credential recovery added)
+Last updated: 2026-05-23 (RB-001: root cause confirmed — HAOS 17.3 supervisor mount-reload deletes cred file; VAULT1 monitor added)
 Purpose: Consolidates real findings from the Phase 4 worksession. Inputs to Phase 5 planning and repair.
 
 ---
@@ -377,7 +377,19 @@ curl -s -H "Authorization: Bearer $TOKEN" 'http://supervisor/host/logs?lines=500
 
 #### VAULT1 CIFS credential recovery (when credential file is lost)
 
-The supervisor credential file at `/mnt/data/supervisor/.mounts_credentials/VAULT1` can be lost after an unclean HAOS shutdown (filesystem dirty bit). Symptoms: supervisor log shows `error 2 (No such file or directory) opening credential file` every 15 min; `/media/VAULT1` falls through to sda8 and Frigate writes recordings there instead of the USB drive.
+**Root cause (identified 2026-05-23, investigated against HAOS supervisor journal):**
+
+The credential file loss is **NOT caused by unclean shutdown**. Confirmed: credential file survived May 6, May 21, and May 22 power-event boots (VAULT1 CIFS mounted successfully on all three). The file was deleted at **2026-05-22 00:53:31 UTC**, 13 minutes after a successful mount on the first HAOS 17.3 boot (rauc slot B).
+
+**What happened:** The new HAOS 17.3 supervisor (2026.05.x) performed a mount reload cycle (stop-start) on `mnt-data-supervisor-mounts-VAULT1`. The STOP step at 00:53:31 completed (`Deactivated successfully`) and deleted the credential file as part of cleanup. The immediately-following START step then failed with `error 2: No such file or directory opening credential file` — because the supervisor deleted the file but does not regenerate it from stored state on remount.
+
+**Trigger for the reload:** Unknown — likely the new supervisor version's startup validation of existing mounts, or a brief Samba connectivity blip that caused the supervisor to cycle the mount. VAULT2 was also reloaded at 00:54:12 but succeeded because VAULT2's credential file was not deleted (race condition or ordering difference).
+
+**Pattern:** This will recur every time the supervisor decides to cycle the VAULT1 CIFS mount. HAOS update to a newer supervisor version may fix or change this behavior.
+
+**NUT/UPS protection gap (separate issue):** HAOS has no NUT client. Unclean shutdowns from power events (cyberpower UPS→Beast→CT104 NUT chain, but HAOS is not a slave) can corrupt the journal but do NOT cause credential file loss per above analysis. Still recommended to add the HAOS NUT addon (Community store, monitor cyberpower1@192.168.99.200 as slave) to prevent hard power cuts to HAOS.
+
+The supervisor credential file at `/mnt/data/supervisor/.mounts_credentials/VAULT1` can be lost after the HAOS supervisor cycles the CIFS mount. Symptoms: supervisor log shows `error 2 (No such file or directory) opening credential file` every 15 min; `/media/VAULT1` falls through to sda8 and Frigate writes recordings there instead of the USB drive.
 
 To recover (requires ~2 min Frigate downtime):
 ```bash
@@ -400,7 +412,9 @@ curl -s -X POST -H "Authorization: Bearer $TOKEN" http://supervisor/addons/ccab4
 Orphan dirs on sda8 are outside Frigate's media path — handle manually (move or delete).
 
 #### Prevention / monitoring
+- **Uptime Kuma "VAULT1 CIFS Mount"** push monitor (id=45, token=yxhtvRPdixD4IY3Z4tVb, 10-min heartbeat timeout): HA automation pushes heartbeat every 5 min when VAULT1 is CIFS-mounted. Missed heartbeat → Kuma → ntfy `grafana-alerts`. Added 2026-05-23.
+- **HA binary_sensor `vault1_cifs_mounted`**: `grep -c '172.30.32.1/VAULT1' /proc/mounts` every 5 min. State off for >3 min → direct ntfy alert (`VAULT1 CIFS Lost – Frigate writing to sda8`). Added 2026-05-23.
 - Uptime Kuma "Frigate Stats" monitor (http://192.168.99.100:5000/api/stats, 60s, 2 retries) → ntfy `grafana-alerts` (added 2026-05-23)
 - Uptime Kuma "Home Assistant" monitor (http://192.168.99.100:8123, 60s) → ntfy `grafana-alerts` (added 2026-05-23)
-- VAULT1 CIFS credential file issue is separate — supervisor retries every 15 min with `error 2 (No such file or directory)`. Does not affect Frigate (Frigate uses direct ext4 mount, not CIFS). Fix by recreating credential file if CIFS mount is intentional.
 - USB enclosure is self-powered (own PSU) and directly connected — no hub to replace. If repeated disconnects occur, suspect: (a) PSU/power cable on enclosure, (b) USB cable quality, (c) motherboard USB port.
+- **Pending**: Add HAOS NUT addon (Community store) monitoring `cyberpower1@192.168.99.200` as slave to prevent hard power cuts from UPS events. NUT chain currently: Beast (master) → CT104 (slave). HAOS is not in the chain.