homelab-configs/runbooks/phase5-incident-log.md

Phase 5 — Incident Log / State Drift

Last updated: 2026-05-23 (RB-001: VAULT1 root cause + monitor added; HAOS NUT addon installed and verified) Purpose: Consolidates real findings from the Phase 4 worksession. Inputs to Phase 5 planning and repair.

---

Active Incidents

INC-005 — NUT upssched earlyshutdown Bug + May 6 Cluster Shutdown (FIXED)

Discovered: 2026-05-06 during Phase 5 health-check investigation
Fixed: 2026-05-06
Class: Same as HAOS auto-update (Phase 1.1) — autonomous mechanism causing user-impacting outage without warning

What happened:
At 10:53:42 CDT on May 6, a brief real power disturbance caused both UPS units to switch to battery (OB) for 14–16 seconds. Line power was restored within ~25 seconds. Despite power returning cleanly, Beast's upssched fired a forced shutdown (FSD) 30 seconds after the initial OB event, initiating a clean cluster-wide shutdown. All 5 nodes (beast, compute3, compute5, media-server, docker-node02) went offline and required manual power-on ~7.5 hours later.

Root cause — upssched.conf bug (pre-existing since NUT install):

# OLD — broken:
AT ONBATT * START-TIMER earlyshutdown 30   # starts 30s FSD timer on any OB event
AT ONLINE * CANCEL-TIMER onbatt online     # cancels "onbatt" timer — NOT earlyshutdown
# No AT ONLINE cancel for "earlyshutdown" existed → timer runs to completion regardless of power restoration

After 30 seconds, upssched-cmd called upsmon -c fsd, propagating FSD to all NUT slave nodes. The earlyshutdown timer was never cancelled because the cancel rule referenced the wrong timer name ("onbatt" vs "earlyshutdown"). Any OB event lasting more than 0 seconds would fire FSD 30 seconds later unconditionally.

Important: battery quick tests do NOT trigger this bug. CyberPower CP1500 quick battery tests produce OL DISCHRG (on-line, discharging) status, not OB (on battery). The upssched AT ONBATT rule does not fire on OL DISCHRG. The May 6 10:53 event was a real power disturbance, not a test artifact.

Why latent: The bug required an actual OB event. Power has been clean for at least the prior 17-day boot window on compute5 (no historical FSD events found in journal). The first observed trigger was the May 6 power disturbance.

Fix applied (Beast, 2026-05-06):

# NEW — fixed:
AT ONBATT * START-TIMER earlyshutdown 300  # extended: 5 min before FSD (batteries hold 22–66 min)
AT ONLINE * CANCEL-TIMER earlyshutdown online  # NEW: cancels timer when power returns

Config backed up as /etc/nut/upssched.conf.bak. Files committed to homelab-configs/beast/nut/.

Also fixed (same session): ups-quarterly-test.sh rewritten to use sequential tests (not simultaneous), poll for OL restoration with 120s timeout, 35s OL buffer before second test, ntfy alert on timeout, abort instead of testing second UPS if first doesn't return OL. This was correct practice even though battery tests don't produce OB status — the sequential design is safer against any firmware variation and future UPS model changes.

Break-test result:
Battery quick test on cyberpower2 produced OL DISCHRG — no ONBATT event, no earlyshutdown timer, no FSD. Cluster remained up. An OB-producing break-test cannot be safely replicated without a real power event. The cancel logic is verified by code review: AT ONLINE * CANCEL-TIMER earlyshutdown online is unambiguous NUT upssched syntax.

Compute5 unsafe shutdown cross-reference:
56 unsafe shutdowns on nvme0 (WD PC SN740) over 1,407h do NOT primarily reflect NUT FSD history. Only 1 FSD event found in journal history (today's). The elevated counts reflect: (a) pre-journal install period reboots, (b) PCIe power management on an i7-13700T platform. The SK Hynix PC711 (nvme1) has 2,362 power cycles — a separate PCIe runtime PM compatibility issue; platform quirk: simple suspend now applied by kernel.

P5-11 added: Compute5 SK Hynix PC711 PCIe power management — verify simple suspend quirk persists across reboots. See below.

---

INC-006 — PBS upsmon DEADTIME Autonomous Shutdown (FIXED 2026-05-16)

Discovered: 2026-05-16 during health-check sweep root-cause analysis Fixed: 2026-05-16 09:05 CDT Class: Same as INC-005 — autonomous mechanism triggering unexpected node shutdown

What happened: PBS (192.168.99.153) rebooted unexpectedly at 17:41 CDT on May 14, 2026 — 10 days after INC-005 was fixed on Beast. Journal analysis showed nut-monitor crash-looping from 16:22, then reconnecting at 17:08 and reporting "UPS unavailable / connection timed out" for the cyberpower1 UPS served by Beast. After 15 seconds of no response from Beast's upsd (during a period when Beast was rebooting or its NUT daemon was unavailable), PBS upsmon triggered an autonomous "power down" and initiated a clean shutdown at 17:41.

Root cause: /etc/nut/upsmon.conf on PBS had DEADTIME 15 — meaning upsmon would declare the UPS communication lost and shut the system down after only 15 seconds of no response from the master NUT server (Beast). During any Beast reboot, NUT config reload, or transient network blip, this 15-second window is trivially exceeded. PBS was configured as a NUT slave: MONITOR cyberpower1@beast.goattw.net:3493 1 ... slave.

The default NUT DEADTIME of 15 seconds is appropriate for a standalone UPS monitor where a lost connection means the UPS itself is gone. For a slave node monitoring a remote master, it should be set to a value that survives normal master reboots (typically 120–300s).

Fix applied (PBS, 2026-05-16):

# Backup: /etc/nut/upsmon.conf.bak.20260516
# Change:
DEADTIME 15   →   DEADTIME 300

nut-monitor restarted and confirmed active. No force flags needed. Config committed to homelab-configs/monitoring/ (see P5-12 commit scope — add standalone commit if needed).

Downstream effects:

• Both PBS ZFS pools (usb1-zfs, usb2-zfs) were not auto-imported after the May 14 reboot (stale cachefile). Manually imported 2026-05-16; scrubs initiated.
• INC-001 zfs-health-check.sh was alerting to ntfy/zfs-health on every 5-min poll since the reboot.

Check Beast's slave nodes: If any other NUT slave (compute5, etc.) has DEADTIME 15, apply the same fix. Beast itself does not need this change (it is the NUT master / upsd host).

---

INC-001 — PBS USB Hub Failure (ACTIVE — hardware fix in progress)

First event: March 12, 2026 (earliest confirmed crash in journal) Crash count: 34+ unclean reboots since April 21; ~50+ total since March 12 Current state (2026-05-16): PBS rebooted May 14 17:41 CDT (INC-006 — upsmon DEADTIME triggered). After reboot both usb1-zfs and usb2-zfs were MISSING (stale cachefile + not auto-imported). Manually imported 2026-05-16 09:14 CDT — both ONLINE, no data errors. Scrubs started: usb1-zfs scrubbing (last scrub Apr 12, 613G), usb2-zfs completed immediately (last scrub May 10). Cachefile updated at /etc/zfs/zpool.cache. Cenmate enclosure dropped again at ~07:16 on current boot (DID_NO_CONNECT at 138697s uptime) — hub failure is ongoing. Hardware replacement remains the only durable fix.

Root cause (confirmed):
A 4-port USB hub at xHCI Bus 2 Port 6 intermittently loses power or signal, dropping all 4 ASM1153E (174c:55aa) USB-to-SATA bridges simultaneously. ZFS detects uncorrectable I/O on usb1-zfs and suspends the pool. The PBS agents process enters D-state waiting for ZFS TXG sync on the suspended pool. After 737+ seconds, the kernel logs a hung task warning. Eventually the system hangs and watchdog-reboots.

The hub reconnects ~21 seconds after dropping, but ZFS remains suspended until manually cleared.

USB topology at failure:

xHCI Bus 2, Port 6 — [FAILING HUB]
  ├─ Port 1: ASM1153E ACAAEBBB2E5D → sdj → usb1-zfs mirror leg 2
  ├─ Port 2: ASM1153E ACAAEBBB2E5F → sdg → usb1-zfs mirror leg 1
  ├─ Port 3: ASM1153E ACAAEBBB2E60 → sdh → usb2-zfs mirror leg 1
  └─ Port 4: ASM1153E ACAAEBBB2E5E → sdi → usb2-zfs mirror leg 2

Both pools are behind the same hub. One hub drop affects both pools simultaneously.

UAS quirk already applied: usb-storage.quirks=174c:55aa:u in kernel cmdline disables UAS on the ASM1153E bridges. This was a pre-existing workaround for protocol-level issues but does not address hub power/connection failures.

Recovery procedure (manual steps, after hardware fix):

# 1. Confirm drives reconnected after hardware fix
lsblk | grep sd

# 2. Clear the suspended pool
zpool clear usb1-zfs

# 3. Run scrubs on both pools immediately
zpool scrub usb1-zfs
zpool scrub usb2-zfs

# 4. Monitor scrub progress (run ~every 10 min until complete)
zpool status -v

# 5. After scrub completes with 0 errors, run PBS backup verify
#    In PBS web UI → Datastore → usb1-zfs → Verify All
#    Or via CLI: proxmox-backup-client verify --all

# 6. If scrub finds unrepaired errors, see INC-002

Hardware fix options (ordered by preference):

1. Best: Connect each ASMT bridge directly to a motherboard USB 3.0 port (bypass hub entirely). xHCI has 10 downstream ports; only 2 are currently used (Port 2: Toshiba, Port 6: hub). Count available physical USB 3.0 jacks; plug each bridge directly.
2. Acceptable: Replace hub with an externally powered USB hub (dedicated power brick, not bus-powered). 4× spinning HDDs exceed typical bus-power budget (900mA), likely causing brownouts.
3. Do not: Return drives to any bus-powered hub.

Kernel command line note: The usb-storage.delay_use=10 param is a 10-second delay before using USB storage. This was also a pre-existing workaround and should be left in place post-fix.

---

INC-002 — PBS Backup Data Integrity Unknown (ACTIVE — verify post-recovery)

Context: usb1-zfs has been suspended on every one of the 34+ crash reboots since April 21. Each suspension may have left partially-written ZFS TXGs. The pool has 4 confirmed data errors in the current boot's suspension event.

Risk: Backup chunks written to usb1-zfs during or after a suspension event may be incomplete or corrupt. The PBS backup catalog may reference chunks that don't exist or are truncated.

Action required post-hardware-fix:

• Run zpool scrub usb1-zfs and verify 0 unrepaired errors.
• Run PBS verify on the usb1-zfs datastore (proxmox-backup-client verify --all, or via web UI).
• Any backup with a verify failure should be noted. If critical VMs have no verified backup, a fresh backup job should be run against a known-good target (Synology-Remote or rust-usb) immediately.

Status: Blocked on hardware fix (INC-001).

---

INC-003 — PBS Phase 1 C1 Re-Diagnosis

Original diagnosis (Phase 1): usb1-zfs and usb2-zfs fail to import at boot due to stale cachefile. Fix applied: cleared cachefile, added zpool import -f to boot sequence.

Revised diagnosis: The import failures were a downstream symptom. The actual root cause is INC-001 (USB hub drop). The pool is suspended at the time of crash-reboot; importing a suspended pool fails regardless of cachefile state. The cachefile fix helped on some boots by switching to scan-based import, but the underlying hardware problem continued causing the failures.

Lesson: When troubleshooting recurring import failures, always check dmesg for USB disconnect events before the failure. A suspended pool at boot is usually caused by the prior boot's crash, not a cachefile issue.

---

INC-004 — sdj Drive End-of-Life (PENDING)

Drive: Hitachi HUA723030ALA640, Serial MK0371YVGRZ2AA, /dev/sdj
Pool: usb1-zfs mirror leg 2
Power-on hours: 93,267 (≈10.6 years)
SMART: Passes all checks. No reallocated or pending sectors. 1 UDMA CRC error (USB interface, consistent with hub issue, not drive failure).

Action: Order a replacement 3TB+ drive (HGST Ultrastar or WD Gold recommended for NAS duty). Plan resilver after INC-001 hardware fix and INC-002 scrub pass are both confirmed clean. Do not start a resilver on a hub that may still be marginal.

---

INC-006 — ASM1153E UAS Reset Loop: Class-Wide Issue (Beast + any future Cenmate host)

Date discovered: 2026-05-07
Status: PARTIALLY MITIGATED — Beast grub updated; pending reboot to activate

Affected hardware: All Cenmate USB enclosures using ASMedia ASM1153E bridges (USB ID 174c:55aa). Confirmed on:

• PBS 4-bay (4× ASM1153E on xHCI Bus 2 Port 6) — quirk already applied, root cause of INC-001
• Beast 2-bay das-mirror (2× ASM1153E, /dev/sde + /dev/sdf) — quirk applied 2026-05-07, Beast reboot pending
• Unit 2 (2-bay, location TBD via physical inspection) — quirk status unknown until host identified

Root cause: The Linux UAS driver has a known interaction with ASM1153E firmware that causes periodic uas_eh_abort_handler / uas_eh_device_reset_handler cycles (~every 60s) under load. This is a chip-family pattern on Linux, not specific to any one enclosure or host. The fix is to force usb-storage mode (disabling UAS): kernel parameter usb-storage.quirks=174c:55aa:u.

This is a class issue, not a one-off. Every host with an ASM1153E-bridged enclosure needs this kernel parameter.

Impact on Beast das-mirror: Pool has been ONLINE and scrub-clean (last scrub 2026-04-12, 0 errors), but UAS resets were occurring in the background. Scrub result may have been from a window without active resets, or the pool survived despite them. Do NOT run zpool scrub das-mirror until Beast has rebooted with the quirk active — scrubbing under the buggy UAS driver generates noise and cannot be trusted as a clean baseline.

Actions taken:

• /etc/default/grub on Beast updated: GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on usb-storage.quirks=174c:55aa:u"
• update-grub run successfully on Beast; quirk present in all kernel entries in /boot/grub/grub.cfg
• Grub is archived (not templated) in Ansible; next run of backup-node-configs.yml will capture the new config

Remaining actions:

1. Reboot Beast at next maintenance window → verify dmesg | grep uas shows no reset loops
2. Run zpool scrub das-mirror post-reboot as first clean baseline
3. When Unit 2 is located: identify its host and apply same quirk before relying on it for PBS pool migration
4. If any additional Cenmate enclosures are added to the cluster, apply quirk at provisioning time

---

Phase 4 — Completed Work

Phase 4B — UPS Self-Tests + Alert (completed 2026-05-06)

• Manual tests run: cyberpower1 and cyberpower2 both passed.
• Script /usr/local/bin/ups-quarterly-test.sh updated to write Prometheus textfile metrics (ups_battery_test_result, ups_battery_test_timestamp_seconds).
• Cron installed: /etc/cron.d/ups-quarterly-test — first Sunday of Jan/Apr/Jul/Oct at 03:00 on Beast.
• Alert group ups_battery_test added: UPSBatteryTestFailed (result==0, for:5m), UPSBatteryTestStale (>100 days, for:6h).
• Break-tested and confirmed.
• Committed: docker-configs and ansible repos.

Phase 4C — Root-Cause Documentation (completed 2026-05-06)

• W11 (compute6 xtables): Race between pve-firewall iptables_restore (no -w flag) and Docker's iptables backend. Self-resolves in 10-120s. No firewall gap. No action taken.
• W12 (compute2 qdevice): qnetd on PBS; PBS crash-rebooting 34+ times since April 21 (same as INC-001). Each reboot drops qnetd for ~60s. Cluster quorum unaffected.
• Documented in runbooks/W11-W12-root-cause-2026-05.md.

Phase 4A — PVE Version Drift (PAUSED — pending PBS stability)

• Scope: Roll beast (9.1.6) and compute5 (9.1.6) to match cluster majority (9.1.7+).
• Procedure: One node per session, 24h soak. Migrate guests off → upgrade → reboot → verify Keepalived/quorum/storage → migrate back.
• Hold until: PBS hardware fix confirmed + usb1-zfs scrub clean + backup verify complete.

---

Phase 5 — Pending Work Queue

P5-01 — PBS Recovery (blocked on hardware)

Sequence: hardware fix → zpool clear → scrub → PBS verify → confirm stable 48h → resume Phase 4A

P5-02 — sdj Replacement (after P5-01 completes)

Order drive → resilver → verify → remove old sdj

P5-03 — qnetd Migration: PBS → Pi4 (after P5-01 + PBS stability confirmed)

Move corosync-qnetd from PBS (.153) to Pi4 (.227). Steps documented in runbooks/W11-W12-root-cause-2026-05.md. 30-minute operation, no VM downtime, no migration.

P5-04 — Authelia Prometheus Metrics + Request-Rate Alert

The C3 issue (May 5 health check: i/o timeouts every 2 min under scanner load) is not caught by the existing ContainerRestartLoop alert. Requires:

• Enable Authelia's Prometheus metrics endpoint (telemetry.metrics in configuration.yml)
• Add Prometheus scrape target for Authelia metrics
• Add alert: AutheliaHighErrorRate — authelia_request_duration_seconds_count{status="408"} rate elevated for >5m

P5-05 — TLS Cert Renewal Verification (RESOLVED 2026-05-16)

4 certs were in the 30–35d expiry window during the cert break-test (~May 5).
Traefik should auto-renew via ACME when ≤30d remain.

RESOLVED: All certs renewed to July 22 2026 (67 days remaining) — confirmed via live openssl check 2026-05-16. Traefik renewed automatically before expiry.

Original action schedule (for reference):

• May 11: SSH to active Traefik node (docker-node01), inspect acme.json, confirm the 4 certs have new notAfter dates.

  ssh tommy@192.168.99.186 "cat ~/traefik/acme.json | python3 -c \"
  import json,sys,datetime
  d=json.load(sys.stdin)
  for r in d.get('myresolver',{}).get('Certificates',[]):
      domain=r['domain']['main']
      # notAfter is base64 DER; use openssl or traefik logs to verify
      print(domain)
  \" 2>/dev/null || docker logs traefik --since 168h 2>&1 | grep -i 'renewed\|certificate\|acme'"
  

• May 18: Repeat check. If any cert still has the original expiry date, force-renew manually.

P5-06 — Traefik node02 CrowdSec LAPI (intentional asymmetry — no action)

node02 has no local CrowdSec; it correctly points to node01's LAPI at 192.168.99.186:8081 per Phase 2A design. Not drift. Known limitation: if node01 is down during failover, node02 Traefik has no CrowdSec coverage.

P5-07 — ansible-control Disk Cleanup / Expansion

At 80%, DiskFillPredicted24h firing. Clean /tmp artifacts and old report files, or expand the LVM.

P5-08 — Pi4 node-exporter Wrong Architecture Binary

SSH works as tommy. DNS (Technitium) and fail2ban both active. node-exporter fails with Exec format error — binary at /usr/local/bin/node_exporter is wrong architecture (x86_64 installed on ARM64). Fix: deploy correct arm64 binary.

VERSION=1.8.2
curl -sL https://github.com/prometheus/node_exporter/releases/download/v${VERSION}/node_exporter-${VERSION}.linux-arm64.tar.gz \
  | tar -xz --strip-components=1 -C /tmp node_exporter-${VERSION}.linux-arm64/node_exporter
scp /tmp/node_exporter tommy@192.168.99.227:/tmp/node_exporter
ssh tommy@192.168.99.227 "sudo mv /tmp/node_exporter /usr/local/bin/ && sudo chmod 755 /usr/local/bin/node_exporter && sudo systemctl restart node_exporter"

Pi4 hostname is also raspberrypi (clone drift — set to pi4 or similar).

P5-09 — Phase 4A Resume (PVE Version Drift)

After P5-01 completes and PBS is confirmed stable for 48h.

P5-10 — Pi4 node-exporter ARM64 Deploy

See P5-08.

P5-11 — Compute5 SK Hynix PC711 PCIe Power Management (monitor only)

nvme1n1 on compute5 (SK Hynix PC711 1TB, 0000:03:00.0) has 2,362 power cycles and 84 unsafe shutdowns — indicative of PCIe runtime PM aggressively power-cycling the drive.

Quirk status verified 2026-05-06: The kernel applies platform quirk: setting simple suspend automatically to both nvme0 and nvme1 — it is a built-in driver quirk for this CPU/chipset (i7-13700T platform), not a cmdline parameter. /etc/kernel/cmdline does not exist; /proc/cmdline has no nvme_core flags. The quirk persists across kernel updates by default. No user action required.

Monitor: check Unsafe Shutdowns and Power Cycles in SMART at each health check. If counts continue accumulating after the quirk is active, escalate to drive replacement or PCIe slot investigation. WD PC SN740 (nvme0) on same node: 56 unsafe shutdowns in 1,407h — attributed to pre-journal setup period and PCIe PS interaction; no action unless accumulating.

---

P5-12 — Weekly Health Digest (DONE 2026-05-16)

Deliverable: monitoring/weekly-health-digest.sh
Deployed: media-server ~/ansible/scripts/weekly-health-digest.sh
Cron: /etc/cron.d/weekly-health-digest — Monday 07:00 CDT
ntfy topic: health-digest on 192.168.99.198:8095 (dedicated topic, not grafana-alerts)

Lightweight weekly drift-catcher. Covers: PVE reachability, Prometheus target health, firing alerts, PBS backup status, PBS ZFS pool presence, filesystem >85%, ZFS pool state (das-mirror/compute3-zfs), Docker unhealthy containers, TLS cert expiry <21d.

Does NOT replace the quarterly full health check. Full sweeps documented in ~/ansible/reports/health-check-*.md.

Break-tested 2026-05-16 08:54 CDT: correctly flagged PBS ZFS MISSING + Grafana firing alerts. Suppressed known-persistent issues (P5-10 .227, HAOS .100, immich_machine_learning). ntfy delivery confirmed.

Subscriber reminder: subscribe ntfy client to topic health-digest on 192.168.99.198:8095.

---

Known Config Drift (not incidents — track for cleanup)

Item	Location	Drift
Hostname mismatch	docker-node01 (.186)	/etc/hostname = ubuntu-template; should = docker-node01
crowdsecLapiHost	node02 dynamic_conf.yml	Points to 192.168.99.186:8081 (node01 IP); should point to local LAPI
Technitium DNS	Pi4	No internal overrides for goattw.net (relies on Pi-hole as primary)
UPS self-test never run	cyberpower1	Resolved by Phase 4B; now automated
PBS ZFS pool feature upgrade	usb1-zfs, usb2-zfs	zpool upgrade deferred until pools are stable post-hardware fix
grub cmdline not Ansible-enforced	PBS, Beast	usb-storage.quirks=174c:55aa:u applied manually; archived by backup playbook but not templated. Risk: silent regression if grub is rewritten by kernel upgrade or manual edit. Convert to Ansible-managed template for USB-quirk nodes. Not urgent.
Alertmanager permission errors	media-server	Resolved (C2 May-05) — no action needed
VAULT1 orphan dirs on sda8	HAOS /media	VAULT1.local-orphan-20260501-2200 (982 MB) and VAULT1.local-orphan-20260523-0224 (299 GB) on sda8. ~22 days of recordings accumulated on HAOS system partition while CIFS was down. VAULT1 CIFS is now fixed (2026-05-23). Orphan dirs are outside Frigate's media path — Frigate will NOT clean them up. Move useful recordings to /media/VAULT1 manually, or delete once retention window passes. sda8 has 1.4 TB free so no immediate risk.

---

Runbooks

RB-001 — Frigate Cameras Down: VAULT1 USB Storage Outage

Last triggered: 2026-05-22 ~06:39 CDT (14:09 UTC) — 7.5 h blackout, resolved ~16:32 CDT (21:32 UTC)

Symptom

• Frigate /api/stats returns HTTP 500 (instead of normal 200 + JSON)
• All cameras show offline in Frigate UI (no live stream or thumbnails)
• Uptime Kuma "Frigate Stats" monitor fires → ntfy grafana-alerts alert

Hardware context (HAOS at 192.168.99.100)

• VAULT1 USB enclosure: JMicron JMS56x (VID:PID 152d:a580), USB 3.0 SuperSpeed, port 2-5
• Self-powered (bmAttributes=0xc0, bMaxPower=8mA) — has external PSU; bus-power issues are NOT the cause
• Directly connected to xHCI host controller (no intermediate hub — port 2-5, not 2-5.x)
• Two Coral USB TPUs also present: 1-2 (USB 2.0) and 2-7 (USB 3.0); these are independent

Diagnosis steps

# 1. Confirm Frigate stats API is down
curl -s -o /dev/null -w "%{http_code}" http://192.168.99.100:5000/api/stats

# 2. SSH into HAOS, check block devices
ssh -p 22222 hassio@192.168.99.100
sudo lsblk   # VAULT1 shows as sdb; if absent, USB enclosure is disconnected

# 3. Check for D-state (uninterruptible sleep) processes blocking on lost device
sudo cat /proc/*/status 2>/dev/null | grep -B5 "State:.*D"

# 4. Verify USB enclosure is still enumerated
sudo ls /sys/bus/usb/devices/ | grep '2-5'   # absent = USB disconnect

# 5. Check supervisor journal for USB and VAULT mount events
TOKEN=$(sudo cat /proc/*/environ 2>/dev/null | tr '\0' '\n' | grep '^SUPERVISOR_TOKEN=' | head -1 | cut -d= -f2-)
curl -s -H "Authorization: Bearer $TOKEN" 'http://supervisor/host/logs?lines=500' \
  | grep -iE 'usb|vault|sd[a-z]|disconnect|error' | tail -40

Recovery

1. Physically reseat the VAULT1 USB enclosure — unplug USB cable, wait 5 s, replug. The enclosure has its own PSU so check the PSU/power cable if USB reseat alone doesn't restore the device.
2. Verify block device reappears: sudo lsblk should show sdb/sdb1 again.
3. Trigger HAOS host reboot via supervisor API (fastest path — Frigate add-on reinitializes cleanly):

   TOKEN=$(sudo cat /proc/*/environ 2>/dev/null | tr '\0' '\n' | grep '^SUPERVISOR_TOKEN=' | head -1 | cut -d= -f2-)
   curl -s -X POST -H "Authorization: Bearer $TOKEN" http://supervisor/host/reboot
   

4. Wait ~3 minutes for HAOS to come back. Confirm Frigate stats: curl http://192.168.99.100:5000/api/stats returns 200.
5. If cameras still offline after reboot, restart Frigate add-on: curl -X POST -H "Authorization: Bearer $TOKEN" http://supervisor/addons/ccab4aaf_frigate-fa/restart

Recovery time

~3 minutes from enclosure reseat to Frigate healthy (HAOS reboot path). ~10–15 minutes if Frigate add-on restart is needed instead of full reboot.

VAULT1 CIFS credential recovery (when credential file is lost)

Root cause (identified 2026-05-23, investigated against HAOS supervisor journal):

The credential file loss is NOT caused by unclean shutdown. Confirmed: credential file survived May 6, May 21, and May 22 power-event boots (VAULT1 CIFS mounted successfully on all three). The file was deleted at 2026-05-22 00:53:31 UTC, 13 minutes after a successful mount on the first HAOS 17.3 boot (rauc slot B).

What happened: The new HAOS 17.3 supervisor (2026.05.x) performed a mount reload cycle (stop-start) on mnt-data-supervisor-mounts-VAULT1. The STOP step at 00:53:31 completed (Deactivated successfully) and deleted the credential file as part of cleanup. The immediately-following START step then failed with error 2: No such file or directory opening credential file — because the supervisor deleted the file but does not regenerate it from stored state on remount.

Trigger for the reload: Unknown — likely the new supervisor version's startup validation of existing mounts, or a brief Samba connectivity blip that caused the supervisor to cycle the mount. VAULT2 was also reloaded at 00:54:12 but succeeded because VAULT2's credential file was not deleted (race condition or ordering difference).

Pattern: This will recur every time the supervisor decides to cycle the VAULT1 CIFS mount. HAOS update to a newer supervisor version may fix or change this behavior.

NUT/UPS protection gap (resolved 2026-05-23): HAOS previously had no NUT client. Unclean shutdowns from power events (cyberpower UPS→Beast→CT104 NUT chain, but HAOS was not a slave) could corrupt the journal but do NOT cause credential file loss per above analysis. The HAOS Community NUT addon (a0d7b954_nut) has been installed and configured — see Prevention/monitoring section for details.

The supervisor credential file at /mnt/data/supervisor/.mounts_credentials/VAULT1 can be lost after the HAOS supervisor cycles the CIFS mount. Symptoms: supervisor log shows error 2 (No such file or directory) opening credential file every 15 min; /media/VAULT1 falls through to sda8 and Frigate writes recordings there instead of the USB drive.

To recover (requires ~2 min Frigate downtime):

# 1. Stop Frigate
TOKEN=$(sudo cat /proc/*/environ 2>/dev/null | tr '\0' '\n' | grep '^SUPERVISOR_TOKEN=' | head -1 | cut -d= -f2-)
curl -s -X POST -H "Authorization: Bearer $TOKEN" http://supervisor/addons/ccab4aaf_frigate-fa/stop

# 2. Move any accumulated orphan data off the path (Samba creds: hassio / check bootconfig.json)
sudo mv /media/VAULT1 /media/VAULT1.local-orphan-$(date -u +%Y%m%d-%H%M)

# 3. Re-PUT the mount with credentials (this recreates the credential file)
curl -s -X PUT -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"name":"VAULT1","type":"cifs","usage":"media","server":"172.30.32.1","share":"VAULT1","username":"hassio","password":"<from /addon_configs/1a32f091_sambanas/bootconfig.json>"}' \
  http://supervisor/mounts/VAULT1

# 4. Verify: df -h /media/VAULT1 should show //172.30.32.1/VAULT1 at 3.6T
# 5. Start Frigate
curl -s -X POST -H "Authorization: Bearer $TOKEN" http://supervisor/addons/ccab4aaf_frigate-fa/start

Orphan dirs on sda8 are outside Frigate's media path — handle manually (move or delete).

Prevention / monitoring

• Uptime Kuma "VAULT1 CIFS Mount" push monitor (id=45, token=yxhtvRPdixD4IY3Z4tVb, 10-min heartbeat timeout): HA automation pushes heartbeat every 5 min when VAULT1 is CIFS-mounted. Missed heartbeat → Kuma → ntfy grafana-alerts. Added 2026-05-23.
• HA binary_sensor vault1_cifs_mounted: grep -c '172.30.32.1/VAULT1' /proc/mounts every 5 min. State off for >3 min → direct ntfy alert (VAULT1 CIFS Lost – Frigate writing to sda8). Added 2026-05-23.
• Uptime Kuma "Frigate Stats" monitor (http://192.168.99.100:5000/api/stats, 60s, 2 retries) → ntfy grafana-alerts (added 2026-05-23)
• Uptime Kuma "Home Assistant" monitor (http://192.168.99.100:8123, 60s) → ntfy grafana-alerts (added 2026-05-23)
• USB enclosure is self-powered (own PSU) and directly connected — no hub to replace. If repeated disconnects occur, suspect: (a) PSU/power cable on enclosure, (b) USB cable quality, (c) motherboard USB port.
• HAOS NUT addon (a0d7b954_nut, installed 2026-05-23): netclient mode, monitors cyberpower1@192.168.99.200:3493 as secondary. Credentials: upsuser / Sparky$100 (same as CT104 slave). shutdown_host: true → runs /usr/bin/shutdownhost on FSD. NUT chain is now: Beast (master) → CT104 (secondary) + HAOS (secondary). Verified: upsc cyberpower1@192.168.99.200 returns full UPS data (OL, 36% load, battery 100%); Beast upsc -c cyberpower1 lists 192.168.99.100 as a connected client. No test.shutdown available on CP1500PFCLCDa firmware — live FSD test skipped (would cut UPS load).