diff --git a/runbooks/phase5-incident-log.md b/runbooks/phase5-incident-log.md index 316e858..6165d18 100644 --- a/runbooks/phase5-incident-log.md +++ b/runbooks/phase5-incident-log.md @@ -1,6 +1,6 @@ # Phase 5 — Incident Log / State Drift -Last updated: 2026-05-23 (INC-007 added — Frigate/VAULT1 USB enclosure outage) +Last updated: 2026-05-23 (INC-007 added; VAULT1 CIFS cred fixed; RB-001 credential recovery added) Purpose: Consolidates real findings from the Phase 4 worksession. Inputs to Phase 5 planning and repair. --- @@ -318,7 +318,7 @@ Subscriber reminder: subscribe ntfy client to topic `health-digest` on 192.168.9 | PBS ZFS pool feature upgrade | usb1-zfs, usb2-zfs | `zpool upgrade` deferred until pools are stable post-hardware fix | | grub cmdline not Ansible-enforced | PBS, Beast | `usb-storage.quirks=174c:55aa:u` applied manually; archived by backup playbook but not templated. Risk: silent regression if grub is rewritten by kernel upgrade or manual edit. Convert to Ansible-managed template for USB-quirk nodes. Not urgent. | | Alertmanager permission errors | media-server | Resolved (C2 May-05) — no action needed | -| VAULT1 CIFS credentials file | HAOS supervisor | `/mnt/data/supervisor/.mounts_credentials/VAULT1` does not exist; supervisor retries every 15 min but fails. Separate from USB health — Frigate accesses ext4 mounts directly, not via CIFS. Credential file needed if VAULT1 CIFS mount is intentional. | +| VAULT1 orphan dirs on sda8 | HAOS /media | `VAULT1.local-orphan-20260501-2200` (982 MB) and `VAULT1.local-orphan-20260523-0224` (299 GB) on sda8. ~22 days of recordings accumulated on HAOS system partition while CIFS was down. VAULT1 CIFS is now fixed (2026-05-23). Orphan dirs are outside Frigate's media path — Frigate will NOT clean them up. Move useful recordings to `/media/VAULT1` manually, or delete once retention window passes. sda8 has 1.4 TB free so no immediate risk. | --- @@ -375,6 +375,30 @@ curl -s -H "Authorization: Bearer $TOKEN" 'http://supervisor/host/logs?lines=500 ~3 minutes from enclosure reseat to Frigate healthy (HAOS reboot path). ~10–15 minutes if Frigate add-on restart is needed instead of full reboot. +#### VAULT1 CIFS credential recovery (when credential file is lost) + +The supervisor credential file at `/mnt/data/supervisor/.mounts_credentials/VAULT1` can be lost after an unclean HAOS shutdown (filesystem dirty bit). Symptoms: supervisor log shows `error 2 (No such file or directory) opening credential file` every 15 min; `/media/VAULT1` falls through to sda8 and Frigate writes recordings there instead of the USB drive. + +To recover (requires ~2 min Frigate downtime): +```bash +# 1. Stop Frigate +TOKEN=$(sudo cat /proc/*/environ 2>/dev/null | tr '\0' '\n' | grep '^SUPERVISOR_TOKEN=' | head -1 | cut -d= -f2-) +curl -s -X POST -H "Authorization: Bearer $TOKEN" http://supervisor/addons/ccab4aaf_frigate-fa/stop + +# 2. Move any accumulated orphan data off the path (Samba creds: hassio / check bootconfig.json) +sudo mv /media/VAULT1 /media/VAULT1.local-orphan-$(date -u +%Y%m%d-%H%M) + +# 3. Re-PUT the mount with credentials (this recreates the credential file) +curl -s -X PUT -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \ + -d '{"name":"VAULT1","type":"cifs","usage":"media","server":"172.30.32.1","share":"VAULT1","username":"hassio","password":""}' \ + http://supervisor/mounts/VAULT1 + +# 4. Verify: df -h /media/VAULT1 should show //172.30.32.1/VAULT1 at 3.6T +# 5. Start Frigate +curl -s -X POST -H "Authorization: Bearer $TOKEN" http://supervisor/addons/ccab4aaf_frigate-fa/start +``` +Orphan dirs on sda8 are outside Frigate's media path — handle manually (move or delete). + #### Prevention / monitoring - Uptime Kuma "Frigate Stats" monitor (http://192.168.99.100:5000/api/stats, 60s, 2 retries) → ntfy `grafana-alerts` (added 2026-05-23) - Uptime Kuma "Home Assistant" monitor (http://192.168.99.100:8123, 60s) → ntfy `grafana-alerts` (added 2026-05-23)