tommy/homelab-configs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

runbooks: add INC-007 Frigate/VAULT1 USB outage runbook (RB-001)

Browse Source 
Symptom→Diagnosis→Recovery for the 2026-05-22 7.5h Frigate blackout.
Key findings: JMicron JMS56x enclosure (2-5) is self-powered and directly
connected; VAULT1 CIFS credential-file errors are a separate pre-existing
config drift, not the root cause. Also adds VAULT1 CIFS drift entry.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Tommy
2026-05-23 21:09:54 -05:00
parent ce2c169e3a
commit 50e8dfccc7
1 changed files with 93 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
runbooks/phase5-incident-log.md
+93 -1
View File
@@ -1,6 +1,6 @@
# Phase 5 — Incident Log / State Drift
Last updated: 2026-05-16 (INC-006 added, INC-001 pool state updated)
Last updated: 2026-05-23 (INC-007 added — Frigate/VAULT1 USB enclosure outage)
Purpose: Consolidates real findings from the Phase 4 worksession. Inputs to Phase 5 planning and repair.
---
@@ -170,6 +170,35 @@ zpool status -v
---
### INC-006 — ASM1153E UAS Reset Loop: Class-Wide Issue (Beast + any future Cenmate host)
**Date discovered:** 2026-05-07
**Status:** PARTIALLY MITIGATED — Beast grub updated; pending reboot to activate
**Affected hardware:** All Cenmate USB enclosures using ASMedia ASM1153E bridges (USB ID `174c:55aa`). Confirmed on:
- PBS 4-bay (4× ASM1153E on xHCI Bus 2 Port 6) — quirk already applied, root cause of INC-001
- Beast 2-bay `das-mirror` (2× ASM1153E, `/dev/sde` + `/dev/sdf`) — quirk applied 2026-05-07, **Beast reboot pending**
- Unit 2 (2-bay, location TBD via physical inspection) — quirk status unknown until host identified
**Root cause:** The Linux UAS driver has a known interaction with ASM1153E firmware that causes periodic `uas_eh_abort_handler` / `uas_eh_device_reset_handler` cycles (~every 60s) under load. This is a chip-family pattern on Linux, not specific to any one enclosure or host. The fix is to force `usb-storage` mode (disabling UAS): kernel parameter `usb-storage.quirks=174c:55aa:u`.
**This is a class issue, not a one-off.** Every host with an ASM1153E-bridged enclosure needs this kernel parameter.
**Impact on Beast `das-mirror`:** Pool has been ONLINE and scrub-clean (last scrub 2026-04-12, 0 errors), but UAS resets were occurring in the background. Scrub result may have been from a window without active resets, or the pool survived despite them. Do NOT run `zpool scrub das-mirror` until Beast has rebooted with the quirk active — scrubbing under the buggy UAS driver generates noise and cannot be trusted as a clean baseline.
**Actions taken:**
- `/etc/default/grub` on Beast updated: `GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on usb-storage.quirks=174c:55aa:u"`
- `update-grub` run successfully on Beast; quirk present in all kernel entries in `/boot/grub/grub.cfg`
- Grub is archived (not templated) in Ansible; next run of `backup-node-configs.yml` will capture the new config
**Remaining actions:**
1. Reboot Beast at next maintenance window → verify `dmesg | grep uas` shows no reset loops
2. Run `zpool scrub das-mirror` post-reboot as first clean baseline
3. When Unit 2 is located: identify its host and apply same quirk before relying on it for PBS pool migration
4. If any additional Cenmate enclosures are added to the cluster, apply quirk at provisioning time
---
## Phase 4 — Completed Work
### Phase 4B — UPS Self-Tests + Alert (completed 2026-05-06)
@@ -287,4 +316,67 @@ Subscriber reminder: subscribe ntfy client to topic `health-digest` on 192.168.9
| Technitium DNS | Pi4 | No internal overrides for goattw.net (relies on Pi-hole as primary) |
| UPS self-test never run | cyberpower1 | Resolved by Phase 4B; now automated |
| PBS ZFS pool feature upgrade | usb1-zfs, usb2-zfs | `zpool upgrade` deferred until pools are stable post-hardware fix |
| grub cmdline not Ansible-enforced | PBS, Beast | `usb-storage.quirks=174c:55aa:u` applied manually; archived by backup playbook but not templated. Risk: silent regression if grub is rewritten by kernel upgrade or manual edit. Convert to Ansible-managed template for USB-quirk nodes. Not urgent. |
| Alertmanager permission errors | media-server | Resolved (C2 May-05) — no action needed |
| VAULT1 CIFS credentials file | HAOS supervisor | `/mnt/data/supervisor/.mounts_credentials/VAULT1` does not exist; supervisor retries every 15 min but fails. Separate from USB health — Frigate accesses ext4 mounts directly, not via CIFS. Credential file needed if VAULT1 CIFS mount is intentional. |
---
## Runbooks
### RB-001 — Frigate Cameras Down: VAULT1 USB Storage Outage
**Last triggered:** 2026-05-22 ~06:39 CDT (14:09 UTC) — 7.5 h blackout, resolved ~16:32 CDT (21:32 UTC)
#### Symptom
- Frigate `/api/stats` returns HTTP 500 (instead of normal 200 + JSON)
- All cameras show offline in Frigate UI (no live stream or thumbnails)
- Uptime Kuma "Frigate Stats" monitor fires → ntfy `grafana-alerts` alert
#### Hardware context (HAOS at 192.168.99.100)
- VAULT1 USB enclosure: JMicron JMS56x (VID:PID 152d:a580), USB 3.0 SuperSpeed, port 2-5
- **Self-powered** (bmAttributes=0xc0, bMaxPower=8mA) — has external PSU; bus-power issues are NOT the cause
- **Directly connected** to xHCI host controller (no intermediate hub — port `2-5`, not `2-5.x`)
- Two Coral USB TPUs also present: 1-2 (USB 2.0) and 2-7 (USB 3.0); these are independent
#### Diagnosis steps
```bash
# 1. Confirm Frigate stats API is down
curl -s -o /dev/null -w "%{http_code}" http://192.168.99.100:5000/api/stats
# 2. SSH into HAOS, check block devices
ssh -p 22222 hassio@192.168.99.100
sudo lsblk # VAULT1 shows as sdb; if absent, USB enclosure is disconnected
# 3. Check for D-state (uninterruptible sleep) processes blocking on lost device
sudo cat /proc/*/status 2>/dev/null | grep -B5 "State:.*D"
# 4. Verify USB enclosure is still enumerated
sudo ls /sys/bus/usb/devices/ | grep '2-5' # absent = USB disconnect
# 5. Check supervisor journal for USB and VAULT mount events
TOKEN=$(sudo cat /proc/*/environ 2>/dev/null | tr '\0' '\n' | grep '^SUPERVISOR_TOKEN=' | head -1 | cut -d= -f2-)
curl -s -H "Authorization: Bearer $TOKEN" 'http://supervisor/host/logs?lines=500' \
| grep -iE 'usb|vault|sd[a-z]|disconnect|error' | tail -40
```
#### Recovery
1. **Physically reseat the VAULT1 USB enclosure** — unplug USB cable, wait 5 s, replug. The enclosure has its own PSU so check the PSU/power cable if USB reseat alone doesn't restore the device.
2. Verify block device reappears: `sudo lsblk` should show sdb/sdb1 again.
3. Trigger HAOS host reboot via supervisor API (fastest path — Frigate add-on reinitializes cleanly):
```bash
TOKEN=$(sudo cat /proc/*/environ 2>/dev/null | tr '\0' '\n' | grep '^SUPERVISOR_TOKEN=' | head -1 | cut -d= -f2-)
curl -s -X POST -H "Authorization: Bearer $TOKEN" http://supervisor/host/reboot
```
4. Wait ~3 minutes for HAOS to come back. Confirm Frigate stats: `curl http://192.168.99.100:5000/api/stats` returns 200.
5. If cameras still offline after reboot, restart Frigate add-on: `curl -X POST -H "Authorization: Bearer $TOKEN" http://supervisor/addons/ccab4aaf_frigate-fa/restart`
#### Recovery time
~3 minutes from enclosure reseat to Frigate healthy (HAOS reboot path).
~10–15 minutes if Frigate add-on restart is needed instead of full reboot.
#### Prevention / monitoring
- Uptime Kuma "Frigate Stats" monitor (http://192.168.99.100:5000/api/stats, 60s, 2 retries) → ntfy `grafana-alerts` (added 2026-05-23)
- Uptime Kuma "Home Assistant" monitor (http://192.168.99.100:8123, 60s) → ntfy `grafana-alerts` (added 2026-05-23)
- VAULT1 CIFS credential file issue is separate — supervisor retries every 15 min with `error 2 (No such file or directory)`. Does not affect Frigate (Frigate uses direct ext4 mount, not CIFS). Fix by recreating credential file if CIFS mount is intentional.
- USB enclosure is self-powered (own PSU) and directly connected — no hub to replace. If repeated disconnects occur, suspect: (a) PSU/power cable on enclosure, (b) USB cable quality, (c) motherboard USB port.