tommy/homelab-configs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ha: VAULT1 CIFS mount monitor; runbooks: supervisor root-cause

Browse Source 
HA: add binary_sensor.vault1_cifs_mounted (grep /proc/mounts every 5 min),
rest_command.vault1_push_heartbeat, and two automations — heartbeat push to
Uptime Kuma push monitor (id=45, 10-min timeout) and direct ntfy alert if
CIFS drops for >3 min. Catches the sda8-fallthrough scenario in <8 min.

Runbooks: root cause of credential file loss is NOT unclean shutdown but
the HAOS 17.3 supervisor (2026.05.x) deleting the cred file during mount
reload cycles. Exact moment: 2026-05-22 00:53:31 UTC, 13 min after the
HAOS 17.3 first-boot successful mount. NUT gap documented (HAOS not a NUT
slave); recommended to add Community NUT addon targeting cyberpower1@.200.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Tommy
2026-05-23 21:46:44 -05:00
parent 7e8ea78cee
commit af09358149
3 changed files with 62 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ha/automations.yaml
+32
View File
@@ -46,3 +46,35 @@
Hard-refresh your browser: Ctrl+Shift+R (Win/Linux) or Cmd+Shift+R (Mac) to clear frontend cache.
priority: high
tags: warning,house
- id: vault1_mount_heartbeat
alias: VAULT1 Mount – Push Heartbeat
description: "Every 5 min: if VAULT1 is CIFS-mounted, push heartbeat to Uptime Kuma. Missed heartbeats alert via ntfy."
triggers:
- platform: time_pattern
minutes: "/5"
conditions:
- condition: state
entity_id: binary_sensor.vault1_cifs_mounted
state: "on"
actions:
- action: rest_command.vault1_push_heartbeat
- id: vault1_mount_lost_alert
alias: VAULT1 Mount – Alert on CIFS Lost
description: "Alert via ntfy if VAULT1 stops being CIFS-mounted (fallthrough to sda8 scenario)."
triggers:
- platform: state
entity_id: binary_sensor.vault1_cifs_mounted
to: "off"
for: "00:03:00"
actions:
- action: rest_command.ntfy_grafana_alerts
data:
title: "VAULT1 CIFS Lost – Frigate writing to sda8"
message: >-
/media/VAULT1 is no longer CIFS-mounted (//172.30.32.1/VAULT1 dropped).
Frigate recordings are going to sda8 (system partition) until fixed.
See RB-001 in runbooks/phase5-incident-log.md for recovery steps.
priority: high
tags: warning,house
ha/configuration.yaml
+13
View File
@@ -15,6 +15,16 @@ http:
- 172.16.0.0/12 # <--- Trusts all Docker containers
- 127.0.0.1 # <--- Localhost
command_line:
- binary_sensor:
name: "VAULT1 CIFS Mounted"
unique_id: vault1_cifs_mounted
command: "grep -c '172.30.32.1/VAULT1' /proc/mounts"
device_class: connectivity
scan_interval: 300
payload_on: "1"
payload_off: "0"
rest_command:
ntfy_grafana_alerts:
url: https://ntfy.goattw.net/grafana-alerts
@@ -25,3 +35,6 @@ rest_command:
Title: "{{ title | default('Home Assistant') }}"
Priority: "{{ priority | default('default') }}"
Tags: "{{ tags | default('house') }}"
vault1_push_heartbeat:
url: "http://192.168.99.183:3001/api/push/yxhtvRPdixD4IY3Z4tVb?status=up&msg=VAULT1+CIFS+OK&ping="
method: GET
runbooks/phase5-incident-log.md
+17 -3
View File
@@ -1,6 +1,6 @@
# Phase 5 — Incident Log / State Drift
Last updated: 2026-05-23 (INC-007 added; VAULT1 CIFS cred fixed; RB-001 credential recovery added)
Last updated: 2026-05-23 (RB-001: root cause confirmed — HAOS 17.3 supervisor mount-reload deletes cred file; VAULT1 monitor added)
Purpose: Consolidates real findings from the Phase 4 worksession. Inputs to Phase 5 planning and repair.
---
@@ -377,7 +377,19 @@ curl -s -H "Authorization: Bearer $TOKEN" 'http://supervisor/host/logs?lines=500
#### VAULT1 CIFS credential recovery (when credential file is lost)
The supervisor credential file at `/mnt/data/supervisor/.mounts_credentials/VAULT1` can be lost after an unclean HAOS shutdown (filesystem dirty bit). Symptoms: supervisor log shows `error 2 (No such file or directory) opening credential file` every 15 min; `/media/VAULT1` falls through to sda8 and Frigate writes recordings there instead of the USB drive.
**Root cause (identified 2026-05-23, investigated against HAOS supervisor journal):**
The credential file loss is **NOT caused by unclean shutdown**. Confirmed: credential file survived May 6, May 21, and May 22 power-event boots (VAULT1 CIFS mounted successfully on all three). The file was deleted at **2026-05-22 00:53:31 UTC**, 13 minutes after a successful mount on the first HAOS 17.3 boot (rauc slot B).
**What happened:** The new HAOS 17.3 supervisor (2026.05.x) performed a mount reload cycle (stop-start) on `mnt-data-supervisor-mounts-VAULT1`. The STOP step at 00:53:31 completed (`Deactivated successfully`) and deleted the credential file as part of cleanup. The immediately-following START step then failed with `error 2: No such file or directory opening credential file` — because the supervisor deleted the file but does not regenerate it from stored state on remount.
**Trigger for the reload:** Unknown — likely the new supervisor version's startup validation of existing mounts, or a brief Samba connectivity blip that caused the supervisor to cycle the mount. VAULT2 was also reloaded at 00:54:12 but succeeded because VAULT2's credential file was not deleted (race condition or ordering difference).
**Pattern:** This will recur every time the supervisor decides to cycle the VAULT1 CIFS mount. HAOS update to a newer supervisor version may fix or change this behavior.
**NUT/UPS protection gap (separate issue):** HAOS has no NUT client. Unclean shutdowns from power events (cyberpower UPS→Beast→CT104 NUT chain, but HAOS is not a slave) can corrupt the journal but do NOT cause credential file loss per above analysis. Still recommended to add the HAOS NUT addon (Community store, monitor cyberpower1@192.168.99.200 as slave) to prevent hard power cuts to HAOS.
The supervisor credential file at `/mnt/data/supervisor/.mounts_credentials/VAULT1` can be lost after the HAOS supervisor cycles the CIFS mount. Symptoms: supervisor log shows `error 2 (No such file or directory) opening credential file` every 15 min; `/media/VAULT1` falls through to sda8 and Frigate writes recordings there instead of the USB drive.
To recover (requires ~2 min Frigate downtime):
```bash
@@ -400,7 +412,9 @@ curl -s -X POST -H "Authorization: Bearer $TOKEN" http://supervisor/addons/ccab4
Orphan dirs on sda8 are outside Frigate's media path — handle manually (move or delete).
#### Prevention / monitoring
- **Uptime Kuma "VAULT1 CIFS Mount"** push monitor (id=45, token=yxhtvRPdixD4IY3Z4tVb, 10-min heartbeat timeout): HA automation pushes heartbeat every 5 min when VAULT1 is CIFS-mounted. Missed heartbeat → Kuma → ntfy `grafana-alerts`. Added 2026-05-23.
- **HA binary_sensor `vault1_cifs_mounted`**: `grep -c '172.30.32.1/VAULT1' /proc/mounts` every 5 min. State off for >3 min → direct ntfy alert (`VAULT1 CIFS Lost – Frigate writing to sda8`). Added 2026-05-23.
- Uptime Kuma "Frigate Stats" monitor (http://192.168.99.100:5000/api/stats, 60s, 2 retries) → ntfy `grafana-alerts` (added 2026-05-23)
- Uptime Kuma "Home Assistant" monitor (http://192.168.99.100:8123, 60s) → ntfy `grafana-alerts` (added 2026-05-23)
- VAULT1 CIFS credential file issue is separate — supervisor retries every 15 min with `error 2 (No such file or directory)`. Does not affect Frigate (Frigate uses direct ext4 mount, not CIFS). Fix by recreating credential file if CIFS mount is intentional.
- USB enclosure is self-powered (own PSU) and directly connected — no hub to replace. If repeated disconnects occur, suspect: (a) PSU/power cable on enclosure, (b) USB cable quality, (c) motherboard USB port.
- **Pending**: Add HAOS NUT addon (Community store) monitoring `cyberpower1@192.168.99.200` as slave to prevent hard power cuts from UPS events. NUT chain currently: Beast (master) → CT104 (slave). HAOS is not in the chain.