diff --git a/runbooks/phase5-incident-log.md b/runbooks/phase5-incident-log.md index cd00c1e..316e858 100644 --- a/runbooks/phase5-incident-log.md +++ b/runbooks/phase5-incident-log.md @@ -1,6 +1,6 @@ # Phase 5 — Incident Log / State Drift -Last updated: 2026-05-16 (INC-006 added, INC-001 pool state updated) +Last updated: 2026-05-23 (INC-007 added — Frigate/VAULT1 USB enclosure outage) Purpose: Consolidates real findings from the Phase 4 worksession. Inputs to Phase 5 planning and repair. --- @@ -170,6 +170,35 @@ zpool status -v --- +### INC-006 — ASM1153E UAS Reset Loop: Class-Wide Issue (Beast + any future Cenmate host) + +**Date discovered:** 2026-05-07 +**Status:** PARTIALLY MITIGATED — Beast grub updated; pending reboot to activate + +**Affected hardware:** All Cenmate USB enclosures using ASMedia ASM1153E bridges (USB ID `174c:55aa`). Confirmed on: +- PBS 4-bay (4× ASM1153E on xHCI Bus 2 Port 6) — quirk already applied, root cause of INC-001 +- Beast 2-bay `das-mirror` (2× ASM1153E, `/dev/sde` + `/dev/sdf`) — quirk applied 2026-05-07, **Beast reboot pending** +- Unit 2 (2-bay, location TBD via physical inspection) — quirk status unknown until host identified + +**Root cause:** The Linux UAS driver has a known interaction with ASM1153E firmware that causes periodic `uas_eh_abort_handler` / `uas_eh_device_reset_handler` cycles (~every 60s) under load. This is a chip-family pattern on Linux, not specific to any one enclosure or host. The fix is to force `usb-storage` mode (disabling UAS): kernel parameter `usb-storage.quirks=174c:55aa:u`. + +**This is a class issue, not a one-off.** Every host with an ASM1153E-bridged enclosure needs this kernel parameter. + +**Impact on Beast `das-mirror`:** Pool has been ONLINE and scrub-clean (last scrub 2026-04-12, 0 errors), but UAS resets were occurring in the background. Scrub result may have been from a window without active resets, or the pool survived despite them. Do NOT run `zpool scrub das-mirror` until Beast has rebooted with the quirk active — scrubbing under the buggy UAS driver generates noise and cannot be trusted as a clean baseline. + +**Actions taken:** +- `/etc/default/grub` on Beast updated: `GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on usb-storage.quirks=174c:55aa:u"` +- `update-grub` run successfully on Beast; quirk present in all kernel entries in `/boot/grub/grub.cfg` +- Grub is archived (not templated) in Ansible; next run of `backup-node-configs.yml` will capture the new config + +**Remaining actions:** +1. Reboot Beast at next maintenance window → verify `dmesg | grep uas` shows no reset loops +2. Run `zpool scrub das-mirror` post-reboot as first clean baseline +3. When Unit 2 is located: identify its host and apply same quirk before relying on it for PBS pool migration +4. If any additional Cenmate enclosures are added to the cluster, apply quirk at provisioning time + +--- + ## Phase 4 — Completed Work ### Phase 4B — UPS Self-Tests + Alert (completed 2026-05-06) @@ -287,4 +316,67 @@ Subscriber reminder: subscribe ntfy client to topic `health-digest` on 192.168.9 | Technitium DNS | Pi4 | No internal overrides for goattw.net (relies on Pi-hole as primary) | | UPS self-test never run | cyberpower1 | Resolved by Phase 4B; now automated | | PBS ZFS pool feature upgrade | usb1-zfs, usb2-zfs | `zpool upgrade` deferred until pools are stable post-hardware fix | +| grub cmdline not Ansible-enforced | PBS, Beast | `usb-storage.quirks=174c:55aa:u` applied manually; archived by backup playbook but not templated. Risk: silent regression if grub is rewritten by kernel upgrade or manual edit. Convert to Ansible-managed template for USB-quirk nodes. Not urgent. | | Alertmanager permission errors | media-server | Resolved (C2 May-05) — no action needed | +| VAULT1 CIFS credentials file | HAOS supervisor | `/mnt/data/supervisor/.mounts_credentials/VAULT1` does not exist; supervisor retries every 15 min but fails. Separate from USB health — Frigate accesses ext4 mounts directly, not via CIFS. Credential file needed if VAULT1 CIFS mount is intentional. | + +--- + +## Runbooks + +### RB-001 — Frigate Cameras Down: VAULT1 USB Storage Outage + +**Last triggered:** 2026-05-22 ~06:39 CDT (14:09 UTC) — 7.5 h blackout, resolved ~16:32 CDT (21:32 UTC) + +#### Symptom +- Frigate `/api/stats` returns HTTP 500 (instead of normal 200 + JSON) +- All cameras show offline in Frigate UI (no live stream or thumbnails) +- Uptime Kuma "Frigate Stats" monitor fires → ntfy `grafana-alerts` alert + +#### Hardware context (HAOS at 192.168.99.100) +- VAULT1 USB enclosure: JMicron JMS56x (VID:PID 152d:a580), USB 3.0 SuperSpeed, port 2-5 +- **Self-powered** (bmAttributes=0xc0, bMaxPower=8mA) — has external PSU; bus-power issues are NOT the cause +- **Directly connected** to xHCI host controller (no intermediate hub — port `2-5`, not `2-5.x`) +- Two Coral USB TPUs also present: 1-2 (USB 2.0) and 2-7 (USB 3.0); these are independent + +#### Diagnosis steps +```bash +# 1. Confirm Frigate stats API is down +curl -s -o /dev/null -w "%{http_code}" http://192.168.99.100:5000/api/stats + +# 2. SSH into HAOS, check block devices +ssh -p 22222 hassio@192.168.99.100 +sudo lsblk # VAULT1 shows as sdb; if absent, USB enclosure is disconnected + +# 3. Check for D-state (uninterruptible sleep) processes blocking on lost device +sudo cat /proc/*/status 2>/dev/null | grep -B5 "State:.*D" + +# 4. Verify USB enclosure is still enumerated +sudo ls /sys/bus/usb/devices/ | grep '2-5' # absent = USB disconnect + +# 5. Check supervisor journal for USB and VAULT mount events +TOKEN=$(sudo cat /proc/*/environ 2>/dev/null | tr '\0' '\n' | grep '^SUPERVISOR_TOKEN=' | head -1 | cut -d= -f2-) +curl -s -H "Authorization: Bearer $TOKEN" 'http://supervisor/host/logs?lines=500' \ + | grep -iE 'usb|vault|sd[a-z]|disconnect|error' | tail -40 +``` + +#### Recovery +1. **Physically reseat the VAULT1 USB enclosure** — unplug USB cable, wait 5 s, replug. The enclosure has its own PSU so check the PSU/power cable if USB reseat alone doesn't restore the device. +2. Verify block device reappears: `sudo lsblk` should show sdb/sdb1 again. +3. Trigger HAOS host reboot via supervisor API (fastest path — Frigate add-on reinitializes cleanly): + ```bash + TOKEN=$(sudo cat /proc/*/environ 2>/dev/null | tr '\0' '\n' | grep '^SUPERVISOR_TOKEN=' | head -1 | cut -d= -f2-) + curl -s -X POST -H "Authorization: Bearer $TOKEN" http://supervisor/host/reboot + ``` +4. Wait ~3 minutes for HAOS to come back. Confirm Frigate stats: `curl http://192.168.99.100:5000/api/stats` returns 200. +5. If cameras still offline after reboot, restart Frigate add-on: `curl -X POST -H "Authorization: Bearer $TOKEN" http://supervisor/addons/ccab4aaf_frigate-fa/restart` + +#### Recovery time +~3 minutes from enclosure reseat to Frigate healthy (HAOS reboot path). +~10–15 minutes if Frigate add-on restart is needed instead of full reboot. + +#### Prevention / monitoring +- Uptime Kuma "Frigate Stats" monitor (http://192.168.99.100:5000/api/stats, 60s, 2 retries) → ntfy `grafana-alerts` (added 2026-05-23) +- Uptime Kuma "Home Assistant" monitor (http://192.168.99.100:8123, 60s) → ntfy `grafana-alerts` (added 2026-05-23) +- VAULT1 CIFS credential file issue is separate — supervisor retries every 15 min with `error 2 (No such file or directory)`. Does not affect Frigate (Frigate uses direct ext4 mount, not CIFS). Fix by recreating credential file if CIFS mount is intentional. +- USB enclosure is self-powered (own PSU) and directly connected — no hub to replace. If repeated disconnects occur, suspect: (a) PSU/power cable on enclosure, (b) USB cable quality, (c) motherboard USB port.