# Phase 5 — Incident Log / State Drift Last updated: 2026-05-23 (INC-007 added; VAULT1 CIFS cred fixed; RB-001 credential recovery added) Purpose: Consolidates real findings from the Phase 4 worksession. Inputs to Phase 5 planning and repair. --- ## Active Incidents ### INC-005 — NUT upssched earlyshutdown Bug + May 6 Cluster Shutdown (FIXED) **Discovered:** 2026-05-06 during Phase 5 health-check investigation **Fixed:** 2026-05-06 **Class:** Same as HAOS auto-update (Phase 1.1) — autonomous mechanism causing user-impacting outage without warning **What happened:** At 10:53:42 CDT on May 6, a brief real power disturbance caused both UPS units to switch to battery (OB) for 14–16 seconds. Line power was restored within ~25 seconds. Despite power returning cleanly, Beast's upssched fired a forced shutdown (FSD) 30 seconds after the initial OB event, initiating a clean cluster-wide shutdown. All 5 nodes (beast, compute3, compute5, media-server, docker-node02) went offline and required manual power-on ~7.5 hours later. **Root cause — upssched.conf bug (pre-existing since NUT install):** ``` # OLD — broken: AT ONBATT * START-TIMER earlyshutdown 30 # starts 30s FSD timer on any OB event AT ONLINE * CANCEL-TIMER onbatt online # cancels "onbatt" timer — NOT earlyshutdown # No AT ONLINE cancel for "earlyshutdown" existed → timer runs to completion regardless of power restoration ``` After 30 seconds, upssched-cmd called `upsmon -c fsd`, propagating FSD to all NUT slave nodes. The earlyshutdown timer was never cancelled because the cancel rule referenced the wrong timer name ("onbatt" vs "earlyshutdown"). Any OB event lasting more than 0 seconds would fire FSD 30 seconds later unconditionally. **Important: battery quick tests do NOT trigger this bug.** CyberPower CP1500 quick battery tests produce `OL DISCHRG` (on-line, discharging) status, not `OB` (on battery). The upssched `AT ONBATT` rule does not fire on `OL DISCHRG`. The May 6 10:53 event was a real power disturbance, not a test artifact. **Why latent:** The bug required an actual OB event. Power has been clean for at least the prior 17-day boot window on compute5 (no historical FSD events found in journal). The first observed trigger was the May 6 power disturbance. **Fix applied (Beast, 2026-05-06):** ``` # NEW — fixed: AT ONBATT * START-TIMER earlyshutdown 300 # extended: 5 min before FSD (batteries hold 22–66 min) AT ONLINE * CANCEL-TIMER earlyshutdown online # NEW: cancels timer when power returns ``` Config backed up as `/etc/nut/upssched.conf.bak`. Files committed to `homelab-configs/beast/nut/`. **Also fixed (same session):** `ups-quarterly-test.sh` rewritten to use sequential tests (not simultaneous), poll for OL restoration with 120s timeout, 35s OL buffer before second test, ntfy alert on timeout, abort instead of testing second UPS if first doesn't return OL. This was correct practice even though battery tests don't produce OB status — the sequential design is safer against any firmware variation and future UPS model changes. **Break-test result:** Battery quick test on cyberpower2 produced `OL DISCHRG` — no ONBATT event, no earlyshutdown timer, no FSD. Cluster remained up. An OB-producing break-test cannot be safely replicated without a real power event. The cancel logic is verified by code review: `AT ONLINE * CANCEL-TIMER earlyshutdown online` is unambiguous NUT upssched syntax. **Compute5 unsafe shutdown cross-reference:** 56 unsafe shutdowns on nvme0 (WD PC SN740) over 1,407h do NOT primarily reflect NUT FSD history. Only 1 FSD event found in journal history (today's). The elevated counts reflect: (a) pre-journal install period reboots, (b) PCIe power management on an i7-13700T platform. The SK Hynix PC711 (nvme1) has 2,362 power cycles — a separate PCIe runtime PM compatibility issue; `platform quirk: simple suspend` now applied by kernel. **P5-11 added:** Compute5 SK Hynix PC711 PCIe power management — verify `simple suspend` quirk persists across reboots. See below. --- ### INC-006 — PBS upsmon DEADTIME Autonomous Shutdown (FIXED 2026-05-16) **Discovered:** 2026-05-16 during health-check sweep root-cause analysis **Fixed:** 2026-05-16 09:05 CDT **Class:** Same as INC-005 — autonomous mechanism triggering unexpected node shutdown **What happened:** PBS (192.168.99.153) rebooted unexpectedly at 17:41 CDT on May 14, 2026 — 10 days after INC-005 was fixed on Beast. Journal analysis showed nut-monitor crash-looping from 16:22, then reconnecting at 17:08 and reporting "UPS unavailable / connection timed out" for the cyberpower1 UPS served by Beast. After 15 seconds of no response from Beast's upsd (during a period when Beast was rebooting or its NUT daemon was unavailable), PBS upsmon triggered an autonomous "power down" and initiated a clean shutdown at 17:41. **Root cause:** `/etc/nut/upsmon.conf` on PBS had `DEADTIME 15` — meaning upsmon would declare the UPS communication lost and shut the system down after only 15 seconds of no response from the master NUT server (Beast). During any Beast reboot, NUT config reload, or transient network blip, this 15-second window is trivially exceeded. PBS was configured as a NUT slave: `MONITOR cyberpower1@beast.goattw.net:3493 1 ... slave`. The default NUT DEADTIME of 15 seconds is appropriate for a standalone UPS monitor where a lost connection means the UPS itself is gone. For a slave node monitoring a remote master, it should be set to a value that survives normal master reboots (typically 120–300s). **Fix applied (PBS, 2026-05-16):** ```bash # Backup: /etc/nut/upsmon.conf.bak.20260516 # Change: DEADTIME 15 → DEADTIME 300 ``` `nut-monitor` restarted and confirmed `active`. No force flags needed. Config committed to `homelab-configs/monitoring/` (see P5-12 commit scope — add standalone commit if needed). **Downstream effects:** - Both PBS ZFS pools (usb1-zfs, usb2-zfs) were not auto-imported after the May 14 reboot (stale cachefile). Manually imported 2026-05-16; scrubs initiated. - INC-001 zfs-health-check.sh was alerting to ntfy/zfs-health on every 5-min poll since the reboot. **Check Beast's slave nodes:** If any other NUT slave (compute5, etc.) has `DEADTIME 15`, apply the same fix. Beast itself does not need this change (it is the NUT master / upsd host). --- ### INC-001 — PBS USB Hub Failure (ACTIVE — hardware fix in progress) **First event:** March 12, 2026 (earliest confirmed crash in journal) **Crash count:** 34+ unclean reboots since April 21; ~50+ total since March 12 **Current state (2026-05-16):** PBS rebooted May 14 17:41 CDT (INC-006 — upsmon DEADTIME triggered). After reboot both usb1-zfs and usb2-zfs were MISSING (stale cachefile + not auto-imported). Manually imported 2026-05-16 09:14 CDT — both ONLINE, no data errors. Scrubs started: usb1-zfs scrubbing (last scrub Apr 12, 613G), usb2-zfs completed immediately (last scrub May 10). Cachefile updated at `/etc/zfs/zpool.cache`. Cenmate enclosure dropped again at ~07:16 on current boot (DID_NO_CONNECT at 138697s uptime) — hub failure is ongoing. Hardware replacement remains the only durable fix. **Root cause (confirmed):** A 4-port USB hub at xHCI Bus 2 Port 6 intermittently loses power or signal, dropping all 4 ASM1153E (174c:55aa) USB-to-SATA bridges simultaneously. ZFS detects uncorrectable I/O on usb1-zfs and suspends the pool. The PBS `agents` process enters D-state waiting for ZFS TXG sync on the suspended pool. After 737+ seconds, the kernel logs a hung task warning. Eventually the system hangs and watchdog-reboots. The hub reconnects ~21 seconds after dropping, but ZFS remains suspended until manually cleared. **USB topology at failure:** ``` xHCI Bus 2, Port 6 — [FAILING HUB] ├─ Port 1: ASM1153E ACAAEBBB2E5D → sdj → usb1-zfs mirror leg 2 ├─ Port 2: ASM1153E ACAAEBBB2E5F → sdg → usb1-zfs mirror leg 1 ├─ Port 3: ASM1153E ACAAEBBB2E60 → sdh → usb2-zfs mirror leg 1 └─ Port 4: ASM1153E ACAAEBBB2E5E → sdi → usb2-zfs mirror leg 2 ``` Both pools are behind the same hub. One hub drop affects both pools simultaneously. **UAS quirk already applied:** `usb-storage.quirks=174c:55aa:u` in kernel cmdline disables UAS on the ASM1153E bridges. This was a pre-existing workaround for protocol-level issues but does not address hub power/connection failures. **Recovery procedure (manual steps, after hardware fix):** ```bash # 1. Confirm drives reconnected after hardware fix lsblk | grep sd # 2. Clear the suspended pool zpool clear usb1-zfs # 3. Run scrubs on both pools immediately zpool scrub usb1-zfs zpool scrub usb2-zfs # 4. Monitor scrub progress (run ~every 10 min until complete) zpool status -v # 5. After scrub completes with 0 errors, run PBS backup verify # In PBS web UI → Datastore → usb1-zfs → Verify All # Or via CLI: proxmox-backup-client verify --all # 6. If scrub finds unrepaired errors, see INC-002 ``` **Hardware fix options (ordered by preference):** 1. **Best:** Connect each ASMT bridge directly to a motherboard USB 3.0 port (bypass hub entirely). xHCI has 10 downstream ports; only 2 are currently used (Port 2: Toshiba, Port 6: hub). Count available physical USB 3.0 jacks; plug each bridge directly. 2. **Acceptable:** Replace hub with an **externally powered** USB hub (dedicated power brick, not bus-powered). 4× spinning HDDs exceed typical bus-power budget (900mA), likely causing brownouts. 3. **Do not:** Return drives to any bus-powered hub. **Kernel command line note:** The `usb-storage.delay_use=10` param is a 10-second delay before using USB storage. This was also a pre-existing workaround and should be left in place post-fix. --- ### INC-002 — PBS Backup Data Integrity Unknown (ACTIVE — verify post-recovery) **Context:** usb1-zfs has been suspended on every one of the 34+ crash reboots since April 21. Each suspension may have left partially-written ZFS TXGs. The pool has 4 confirmed data errors in the current boot's suspension event. **Risk:** Backup chunks written to usb1-zfs during or after a suspension event may be incomplete or corrupt. The PBS backup catalog may reference chunks that don't exist or are truncated. **Action required post-hardware-fix:** - Run `zpool scrub usb1-zfs` and verify 0 unrepaired errors. - Run PBS verify on the usb1-zfs datastore (proxmox-backup-client verify --all, or via web UI). - Any backup with a verify failure should be noted. If critical VMs have no verified backup, a fresh backup job should be run against a known-good target (Synology-Remote or rust-usb) immediately. **Status:** Blocked on hardware fix (INC-001). --- ### INC-003 — PBS Phase 1 C1 Re-Diagnosis **Original diagnosis (Phase 1):** usb1-zfs and usb2-zfs fail to import at boot due to stale cachefile. Fix applied: cleared cachefile, added `zpool import -f` to boot sequence. **Revised diagnosis:** The import failures were a downstream symptom. The actual root cause is INC-001 (USB hub drop). The pool is suspended at the time of crash-reboot; importing a suspended pool fails regardless of cachefile state. The cachefile fix helped on some boots by switching to scan-based import, but the underlying hardware problem continued causing the failures. **Lesson:** When troubleshooting recurring import failures, always check `dmesg` for USB disconnect events before the failure. A suspended pool at boot is usually caused by the prior boot's crash, not a cachefile issue. --- ### INC-004 — sdj Drive End-of-Life (PENDING) **Drive:** Hitachi HUA723030ALA640, Serial MK0371YVGRZ2AA, /dev/sdj **Pool:** usb1-zfs mirror leg 2 **Power-on hours:** 93,267 (≈10.6 years) **SMART:** Passes all checks. No reallocated or pending sectors. 1 UDMA CRC error (USB interface, consistent with hub issue, not drive failure). **Action:** Order a replacement 3TB+ drive (HGST Ultrastar or WD Gold recommended for NAS duty). Plan resilver after INC-001 hardware fix and INC-002 scrub pass are both confirmed clean. Do not start a resilver on a hub that may still be marginal. --- ### INC-006 — ASM1153E UAS Reset Loop: Class-Wide Issue (Beast + any future Cenmate host) **Date discovered:** 2026-05-07 **Status:** PARTIALLY MITIGATED — Beast grub updated; pending reboot to activate **Affected hardware:** All Cenmate USB enclosures using ASMedia ASM1153E bridges (USB ID `174c:55aa`). Confirmed on: - PBS 4-bay (4× ASM1153E on xHCI Bus 2 Port 6) — quirk already applied, root cause of INC-001 - Beast 2-bay `das-mirror` (2× ASM1153E, `/dev/sde` + `/dev/sdf`) — quirk applied 2026-05-07, **Beast reboot pending** - Unit 2 (2-bay, location TBD via physical inspection) — quirk status unknown until host identified **Root cause:** The Linux UAS driver has a known interaction with ASM1153E firmware that causes periodic `uas_eh_abort_handler` / `uas_eh_device_reset_handler` cycles (~every 60s) under load. This is a chip-family pattern on Linux, not specific to any one enclosure or host. The fix is to force `usb-storage` mode (disabling UAS): kernel parameter `usb-storage.quirks=174c:55aa:u`. **This is a class issue, not a one-off.** Every host with an ASM1153E-bridged enclosure needs this kernel parameter. **Impact on Beast `das-mirror`:** Pool has been ONLINE and scrub-clean (last scrub 2026-04-12, 0 errors), but UAS resets were occurring in the background. Scrub result may have been from a window without active resets, or the pool survived despite them. Do NOT run `zpool scrub das-mirror` until Beast has rebooted with the quirk active — scrubbing under the buggy UAS driver generates noise and cannot be trusted as a clean baseline. **Actions taken:** - `/etc/default/grub` on Beast updated: `GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on usb-storage.quirks=174c:55aa:u"` - `update-grub` run successfully on Beast; quirk present in all kernel entries in `/boot/grub/grub.cfg` - Grub is archived (not templated) in Ansible; next run of `backup-node-configs.yml` will capture the new config **Remaining actions:** 1. Reboot Beast at next maintenance window → verify `dmesg | grep uas` shows no reset loops 2. Run `zpool scrub das-mirror` post-reboot as first clean baseline 3. When Unit 2 is located: identify its host and apply same quirk before relying on it for PBS pool migration 4. If any additional Cenmate enclosures are added to the cluster, apply quirk at provisioning time --- ## Phase 4 — Completed Work ### Phase 4B — UPS Self-Tests + Alert (completed 2026-05-06) - Manual tests run: cyberpower1 and cyberpower2 both passed. - Script `/usr/local/bin/ups-quarterly-test.sh` updated to write Prometheus textfile metrics (`ups_battery_test_result`, `ups_battery_test_timestamp_seconds`). - Cron installed: `/etc/cron.d/ups-quarterly-test` — first Sunday of Jan/Apr/Jul/Oct at 03:00 on Beast. - Alert group `ups_battery_test` added: `UPSBatteryTestFailed` (result==0, for:5m), `UPSBatteryTestStale` (>100 days, for:6h). - Break-tested and confirmed. - Committed: docker-configs and ansible repos. ### Phase 4C — Root-Cause Documentation (completed 2026-05-06) - **W11 (compute6 xtables):** Race between pve-firewall iptables_restore (no -w flag) and Docker's iptables backend. Self-resolves in 10-120s. No firewall gap. No action taken. - **W12 (compute2 qdevice):** qnetd on PBS; PBS crash-rebooting 34+ times since April 21 (same as INC-001). Each reboot drops qnetd for ~60s. Cluster quorum unaffected. - Documented in `runbooks/W11-W12-root-cause-2026-05.md`. ### Phase 4A — PVE Version Drift (PAUSED — pending PBS stability) - Scope: Roll beast (9.1.6) and compute5 (9.1.6) to match cluster majority (9.1.7+). - Procedure: One node per session, 24h soak. Migrate guests off → upgrade → reboot → verify Keepalived/quorum/storage → migrate back. - **Hold until:** PBS hardware fix confirmed + usb1-zfs scrub clean + backup verify complete. --- ## Phase 5 — Pending Work Queue ### P5-01 — PBS Recovery (blocked on hardware) Sequence: hardware fix → zpool clear → scrub → PBS verify → confirm stable 48h → resume Phase 4A ### P5-02 — sdj Replacement (after P5-01 completes) Order drive → resilver → verify → remove old sdj ### P5-03 — qnetd Migration: PBS → Pi4 (after P5-01 + PBS stability confirmed) Move `corosync-qnetd` from PBS (.153) to Pi4 (.227). Steps documented in `runbooks/W11-W12-root-cause-2026-05.md`. 30-minute operation, no VM downtime, no migration. ### P5-04 — Authelia Prometheus Metrics + Request-Rate Alert The C3 issue (May 5 health check: i/o timeouts every 2 min under scanner load) is not caught by the existing `ContainerRestartLoop` alert. Requires: - Enable Authelia's Prometheus metrics endpoint (`telemetry.metrics` in configuration.yml) - Add Prometheus scrape target for Authelia metrics - Add alert: `AutheliaHighErrorRate` — `authelia_request_duration_seconds_count{status="408"}` rate elevated for >5m ### P5-05 — TLS Cert Renewal Verification (RESOLVED 2026-05-16) 4 certs were in the 30–35d expiry window during the cert break-test (~May 5). Traefik should auto-renew via ACME when ≤30d remain. **RESOLVED:** All certs renewed to July 22 2026 (67 days remaining) — confirmed via live openssl check 2026-05-16. Traefik renewed automatically before expiry. **Original action schedule (for reference):** - **May 11:** SSH to active Traefik node (docker-node01), inspect acme.json, confirm the 4 certs have new `notAfter` dates. ```bash ssh tommy@192.168.99.186 "cat ~/traefik/acme.json | python3 -c \" import json,sys,datetime d=json.load(sys.stdin) for r in d.get('myresolver',{}).get('Certificates',[]): domain=r['domain']['main'] # notAfter is base64 DER; use openssl or traefik logs to verify print(domain) \" 2>/dev/null || docker logs traefik --since 168h 2>&1 | grep -i 'renewed\|certificate\|acme'" ``` - **May 18:** Repeat check. If any cert still has the original expiry date, force-renew manually. ### P5-06 — Traefik node02 CrowdSec LAPI (intentional asymmetry — no action) node02 has no local CrowdSec; it correctly points to node01's LAPI at 192.168.99.186:8081 per Phase 2A design. Not drift. Known limitation: if node01 is down during failover, node02 Traefik has no CrowdSec coverage. ### P5-07 — ansible-control Disk Cleanup / Expansion At 80%, DiskFillPredicted24h firing. Clean `/tmp` artifacts and old report files, or expand the LVM. ### P5-08 — Pi4 node-exporter Wrong Architecture Binary SSH works as `tommy`. DNS (Technitium) and fail2ban both active. node-exporter fails with `Exec format error` — binary at `/usr/local/bin/node_exporter` is wrong architecture (x86_64 installed on ARM64). Fix: deploy correct arm64 binary. ```bash VERSION=1.8.2 curl -sL https://github.com/prometheus/node_exporter/releases/download/v${VERSION}/node_exporter-${VERSION}.linux-arm64.tar.gz \ | tar -xz --strip-components=1 -C /tmp node_exporter-${VERSION}.linux-arm64/node_exporter scp /tmp/node_exporter tommy@192.168.99.227:/tmp/node_exporter ssh tommy@192.168.99.227 "sudo mv /tmp/node_exporter /usr/local/bin/ && sudo chmod 755 /usr/local/bin/node_exporter && sudo systemctl restart node_exporter" ``` Pi4 hostname is also `raspberrypi` (clone drift — set to pi4 or similar). ### P5-09 — Phase 4A Resume (PVE Version Drift) After P5-01 completes and PBS is confirmed stable for 48h. ### P5-10 — Pi4 node-exporter ARM64 Deploy See P5-08. ### P5-11 — Compute5 SK Hynix PC711 PCIe Power Management (monitor only) nvme1n1 on compute5 (SK Hynix PC711 1TB, `0000:03:00.0`) has 2,362 power cycles and 84 unsafe shutdowns — indicative of PCIe runtime PM aggressively power-cycling the drive. **Quirk status verified 2026-05-06:** The kernel applies `platform quirk: setting simple suspend` automatically to both nvme0 and nvme1 — it is a built-in driver quirk for this CPU/chipset (i7-13700T platform), not a cmdline parameter. `/etc/kernel/cmdline` does not exist; `/proc/cmdline` has no nvme_core flags. The quirk persists across kernel updates by default. No user action required. Monitor: check `Unsafe Shutdowns` and `Power Cycles` in SMART at each health check. If counts continue accumulating after the quirk is active, escalate to drive replacement or PCIe slot investigation. WD PC SN740 (nvme0) on same node: 56 unsafe shutdowns in 1,407h — attributed to pre-journal setup period and PCIe PS interaction; no action unless accumulating. --- ### P5-12 — Weekly Health Digest (DONE 2026-05-16) **Deliverable:** `monitoring/weekly-health-digest.sh` **Deployed:** media-server `~/ansible/scripts/weekly-health-digest.sh` **Cron:** `/etc/cron.d/weekly-health-digest` — Monday 07:00 CDT **ntfy topic:** `health-digest` on 192.168.99.198:8095 (dedicated topic, not grafana-alerts) Lightweight weekly drift-catcher. Covers: PVE reachability, Prometheus target health, firing alerts, PBS backup status, PBS ZFS pool presence, filesystem >85%, ZFS pool state (das-mirror/compute3-zfs), Docker unhealthy containers, TLS cert expiry <21d. **Does NOT replace the quarterly full health check.** Full sweeps documented in `~/ansible/reports/health-check-*.md`. Break-tested 2026-05-16 08:54 CDT: correctly flagged PBS ZFS MISSING + Grafana firing alerts. Suppressed known-persistent issues (P5-10 .227, HAOS .100, immich_machine_learning). ntfy delivery confirmed. Subscriber reminder: subscribe ntfy client to topic `health-digest` on 192.168.99.198:8095. --- ## Known Config Drift (not incidents — track for cleanup) | Item | Location | Drift | |---|---|---| | Hostname mismatch | docker-node01 (.186) | `/etc/hostname` = ubuntu-template; should = docker-node01 | | crowdsecLapiHost | node02 dynamic_conf.yml | Points to 192.168.99.186:8081 (node01 IP); should point to local LAPI | | Technitium DNS | Pi4 | No internal overrides for goattw.net (relies on Pi-hole as primary) | | UPS self-test never run | cyberpower1 | Resolved by Phase 4B; now automated | | PBS ZFS pool feature upgrade | usb1-zfs, usb2-zfs | `zpool upgrade` deferred until pools are stable post-hardware fix | | grub cmdline not Ansible-enforced | PBS, Beast | `usb-storage.quirks=174c:55aa:u` applied manually; archived by backup playbook but not templated. Risk: silent regression if grub is rewritten by kernel upgrade or manual edit. Convert to Ansible-managed template for USB-quirk nodes. Not urgent. | | Alertmanager permission errors | media-server | Resolved (C2 May-05) — no action needed | | VAULT1 orphan dirs on sda8 | HAOS /media | `VAULT1.local-orphan-20260501-2200` (982 MB) and `VAULT1.local-orphan-20260523-0224` (299 GB) on sda8. ~22 days of recordings accumulated on HAOS system partition while CIFS was down. VAULT1 CIFS is now fixed (2026-05-23). Orphan dirs are outside Frigate's media path — Frigate will NOT clean them up. Move useful recordings to `/media/VAULT1` manually, or delete once retention window passes. sda8 has 1.4 TB free so no immediate risk. | --- ## Runbooks ### RB-001 — Frigate Cameras Down: VAULT1 USB Storage Outage **Last triggered:** 2026-05-22 ~06:39 CDT (14:09 UTC) — 7.5 h blackout, resolved ~16:32 CDT (21:32 UTC) #### Symptom - Frigate `/api/stats` returns HTTP 500 (instead of normal 200 + JSON) - All cameras show offline in Frigate UI (no live stream or thumbnails) - Uptime Kuma "Frigate Stats" monitor fires → ntfy `grafana-alerts` alert #### Hardware context (HAOS at 192.168.99.100) - VAULT1 USB enclosure: JMicron JMS56x (VID:PID 152d:a580), USB 3.0 SuperSpeed, port 2-5 - **Self-powered** (bmAttributes=0xc0, bMaxPower=8mA) — has external PSU; bus-power issues are NOT the cause - **Directly connected** to xHCI host controller (no intermediate hub — port `2-5`, not `2-5.x`) - Two Coral USB TPUs also present: 1-2 (USB 2.0) and 2-7 (USB 3.0); these are independent #### Diagnosis steps ```bash # 1. Confirm Frigate stats API is down curl -s -o /dev/null -w "%{http_code}" http://192.168.99.100:5000/api/stats # 2. SSH into HAOS, check block devices ssh -p 22222 hassio@192.168.99.100 sudo lsblk # VAULT1 shows as sdb; if absent, USB enclosure is disconnected # 3. Check for D-state (uninterruptible sleep) processes blocking on lost device sudo cat /proc/*/status 2>/dev/null | grep -B5 "State:.*D" # 4. Verify USB enclosure is still enumerated sudo ls /sys/bus/usb/devices/ | grep '2-5' # absent = USB disconnect # 5. Check supervisor journal for USB and VAULT mount events TOKEN=$(sudo cat /proc/*/environ 2>/dev/null | tr '\0' '\n' | grep '^SUPERVISOR_TOKEN=' | head -1 | cut -d= -f2-) curl -s -H "Authorization: Bearer $TOKEN" 'http://supervisor/host/logs?lines=500' \ | grep -iE 'usb|vault|sd[a-z]|disconnect|error' | tail -40 ``` #### Recovery 1. **Physically reseat the VAULT1 USB enclosure** — unplug USB cable, wait 5 s, replug. The enclosure has its own PSU so check the PSU/power cable if USB reseat alone doesn't restore the device. 2. Verify block device reappears: `sudo lsblk` should show sdb/sdb1 again. 3. Trigger HAOS host reboot via supervisor API (fastest path — Frigate add-on reinitializes cleanly): ```bash TOKEN=$(sudo cat /proc/*/environ 2>/dev/null | tr '\0' '\n' | grep '^SUPERVISOR_TOKEN=' | head -1 | cut -d= -f2-) curl -s -X POST -H "Authorization: Bearer $TOKEN" http://supervisor/host/reboot ``` 4. Wait ~3 minutes for HAOS to come back. Confirm Frigate stats: `curl http://192.168.99.100:5000/api/stats` returns 200. 5. If cameras still offline after reboot, restart Frigate add-on: `curl -X POST -H "Authorization: Bearer $TOKEN" http://supervisor/addons/ccab4aaf_frigate-fa/restart` #### Recovery time ~3 minutes from enclosure reseat to Frigate healthy (HAOS reboot path). ~10–15 minutes if Frigate add-on restart is needed instead of full reboot. #### VAULT1 CIFS credential recovery (when credential file is lost) The supervisor credential file at `/mnt/data/supervisor/.mounts_credentials/VAULT1` can be lost after an unclean HAOS shutdown (filesystem dirty bit). Symptoms: supervisor log shows `error 2 (No such file or directory) opening credential file` every 15 min; `/media/VAULT1` falls through to sda8 and Frigate writes recordings there instead of the USB drive. To recover (requires ~2 min Frigate downtime): ```bash # 1. Stop Frigate TOKEN=$(sudo cat /proc/*/environ 2>/dev/null | tr '\0' '\n' | grep '^SUPERVISOR_TOKEN=' | head -1 | cut -d= -f2-) curl -s -X POST -H "Authorization: Bearer $TOKEN" http://supervisor/addons/ccab4aaf_frigate-fa/stop # 2. Move any accumulated orphan data off the path (Samba creds: hassio / check bootconfig.json) sudo mv /media/VAULT1 /media/VAULT1.local-orphan-$(date -u +%Y%m%d-%H%M) # 3. Re-PUT the mount with credentials (this recreates the credential file) curl -s -X PUT -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \ -d '{"name":"VAULT1","type":"cifs","usage":"media","server":"172.30.32.1","share":"VAULT1","username":"hassio","password":""}' \ http://supervisor/mounts/VAULT1 # 4. Verify: df -h /media/VAULT1 should show //172.30.32.1/VAULT1 at 3.6T # 5. Start Frigate curl -s -X POST -H "Authorization: Bearer $TOKEN" http://supervisor/addons/ccab4aaf_frigate-fa/start ``` Orphan dirs on sda8 are outside Frigate's media path — handle manually (move or delete). #### Prevention / monitoring - Uptime Kuma "Frigate Stats" monitor (http://192.168.99.100:5000/api/stats, 60s, 2 retries) → ntfy `grafana-alerts` (added 2026-05-23) - Uptime Kuma "Home Assistant" monitor (http://192.168.99.100:8123, 60s) → ntfy `grafana-alerts` (added 2026-05-23) - VAULT1 CIFS credential file issue is separate — supervisor retries every 15 min with `error 2 (No such file or directory)`. Does not affect Frigate (Frigate uses direct ext4 mount, not CIFS). Fix by recreating credential file if CIFS mount is intentional. - USB enclosure is self-powered (own PSU) and directly connected — no hub to replace. If repeated disconnects occur, suspect: (a) PSU/power cable on enclosure, (b) USB cable quality, (c) motherboard USB port.