homelab-configs/runbooks/phase5-incident-log.md

Phase 5 — Incident Log / State Drift

Last updated: 2026-05-24 (RB-001: VAULT1 root cause + monitor added; RB-002: Total Traefik 403 outage — CrowdSec SQLite journal lock + WAL mode fix) Purpose: Consolidates real findings from the Phase 4 worksession. Inputs to Phase 5 planning and repair.

---

Active Incidents

INC-005 — NUT upssched earlyshutdown Bug + May 6 Cluster Shutdown (FIXED)

Discovered: 2026-05-06 during Phase 5 health-check investigation
Fixed: 2026-05-06
Class: Same as HAOS auto-update (Phase 1.1) — autonomous mechanism causing user-impacting outage without warning

What happened:
At 10:53:42 CDT on May 6, a brief real power disturbance caused both UPS units to switch to battery (OB) for 14–16 seconds. Line power was restored within ~25 seconds. Despite power returning cleanly, Beast's upssched fired a forced shutdown (FSD) 30 seconds after the initial OB event, initiating a clean cluster-wide shutdown. All 5 nodes (beast, compute3, compute5, media-server, docker-node02) went offline and required manual power-on ~7.5 hours later.

Root cause — upssched.conf bug (pre-existing since NUT install):

# OLD — broken:
AT ONBATT * START-TIMER earlyshutdown 30   # starts 30s FSD timer on any OB event
AT ONLINE * CANCEL-TIMER onbatt online     # cancels "onbatt" timer — NOT earlyshutdown
# No AT ONLINE cancel for "earlyshutdown" existed → timer runs to completion regardless of power restoration

After 30 seconds, upssched-cmd called upsmon -c fsd, propagating FSD to all NUT slave nodes. The earlyshutdown timer was never cancelled because the cancel rule referenced the wrong timer name ("onbatt" vs "earlyshutdown"). Any OB event lasting more than 0 seconds would fire FSD 30 seconds later unconditionally.

Important: battery quick tests do NOT trigger this bug. CyberPower CP1500 quick battery tests produce OL DISCHRG (on-line, discharging) status, not OB (on battery). The upssched AT ONBATT rule does not fire on OL DISCHRG. The May 6 10:53 event was a real power disturbance, not a test artifact.

Why latent: The bug required an actual OB event. Power has been clean for at least the prior 17-day boot window on compute5 (no historical FSD events found in journal). The first observed trigger was the May 6 power disturbance.

Fix applied (Beast, 2026-05-06):

# NEW — fixed:
AT ONBATT * START-TIMER earlyshutdown 300  # extended: 5 min before FSD (batteries hold 22–66 min)
AT ONLINE * CANCEL-TIMER earlyshutdown online  # NEW: cancels timer when power returns

Config backed up as /etc/nut/upssched.conf.bak. Files committed to homelab-configs/beast/nut/.

Also fixed (same session): ups-quarterly-test.sh rewritten to use sequential tests (not simultaneous), poll for OL restoration with 120s timeout, 35s OL buffer before second test, ntfy alert on timeout, abort instead of testing second UPS if first doesn't return OL. This was correct practice even though battery tests don't produce OB status — the sequential design is safer against any firmware variation and future UPS model changes.

Break-test result:
Battery quick test on cyberpower2 produced OL DISCHRG — no ONBATT event, no earlyshutdown timer, no FSD. Cluster remained up. An OB-producing break-test cannot be safely replicated without a real power event. The cancel logic is verified by code review: AT ONLINE * CANCEL-TIMER earlyshutdown online is unambiguous NUT upssched syntax.

Compute5 unsafe shutdown cross-reference:
56 unsafe shutdowns on nvme0 (WD PC SN740) over 1,407h do NOT primarily reflect NUT FSD history. Only 1 FSD event found in journal history (today's). The elevated counts reflect: (a) pre-journal install period reboots, (b) PCIe power management on an i7-13700T platform. The SK Hynix PC711 (nvme1) has 2,362 power cycles — a separate PCIe runtime PM compatibility issue; platform quirk: simple suspend now applied by kernel.

P5-11 added: Compute5 SK Hynix PC711 PCIe power management — verify simple suspend quirk persists across reboots. See below.

---

INC-006 — PBS upsmon DEADTIME Autonomous Shutdown (FIXED 2026-05-16)

Discovered: 2026-05-16 during health-check sweep root-cause analysis Fixed: 2026-05-16 09:05 CDT Class: Same as INC-005 — autonomous mechanism triggering unexpected node shutdown

What happened: PBS (192.168.99.153) rebooted unexpectedly at 17:41 CDT on May 14, 2026 — 10 days after INC-005 was fixed on Beast. Journal analysis showed nut-monitor crash-looping from 16:22, then reconnecting at 17:08 and reporting "UPS unavailable / connection timed out" for the cyberpower1 UPS served by Beast. After 15 seconds of no response from Beast's upsd (during a period when Beast was rebooting or its NUT daemon was unavailable), PBS upsmon triggered an autonomous "power down" and initiated a clean shutdown at 17:41.

Root cause: /etc/nut/upsmon.conf on PBS had DEADTIME 15 — meaning upsmon would declare the UPS communication lost and shut the system down after only 15 seconds of no response from the master NUT server (Beast). During any Beast reboot, NUT config reload, or transient network blip, this 15-second window is trivially exceeded. PBS was configured as a NUT slave: MONITOR cyberpower1@beast.goattw.net:3493 1 ... slave.

The default NUT DEADTIME of 15 seconds is appropriate for a standalone UPS monitor where a lost connection means the UPS itself is gone. For a slave node monitoring a remote master, it should be set to a value that survives normal master reboots (typically 120–300s).

Fix applied (PBS, 2026-05-16):

# Backup: /etc/nut/upsmon.conf.bak.20260516
# Change:
DEADTIME 15   →   DEADTIME 300

nut-monitor restarted and confirmed active. No force flags needed. Config committed to homelab-configs/monitoring/ (see P5-12 commit scope — add standalone commit if needed).

Downstream effects:

• Both PBS ZFS pools (usb1-zfs, usb2-zfs) were not auto-imported after the May 14 reboot (stale cachefile). Manually imported 2026-05-16; scrubs initiated.
• INC-001 zfs-health-check.sh was alerting to ntfy/zfs-health on every 5-min poll since the reboot.

Check Beast's slave nodes: If any other NUT slave (compute5, etc.) has DEADTIME 15, apply the same fix. Beast itself does not need this change (it is the NUT master / upsd host).

---

INC-001 — PBS USB Hub Failure (ACTIVE — hardware fix in progress)

First event: March 12, 2026 (earliest confirmed crash in journal) Crash count: 34+ unclean reboots since April 21; ~50+ total since March 12 Current state (2026-05-16): PBS rebooted May 14 17:41 CDT (INC-006 — upsmon DEADTIME triggered). After reboot both usb1-zfs and usb2-zfs were MISSING (stale cachefile + not auto-imported). Manually imported 2026-05-16 09:14 CDT — both ONLINE, no data errors. Scrubs started: usb1-zfs scrubbing (last scrub Apr 12, 613G), usb2-zfs completed immediately (last scrub May 10). Cachefile updated at /etc/zfs/zpool.cache. Cenmate enclosure dropped again at ~07:16 on current boot (DID_NO_CONNECT at 138697s uptime) — hub failure is ongoing. Hardware replacement remains the only durable fix.

Root cause (confirmed):
A 4-port USB hub at xHCI Bus 2 Port 6 intermittently loses power or signal, dropping all 4 ASM1153E (174c:55aa) USB-to-SATA bridges simultaneously. ZFS detects uncorrectable I/O on usb1-zfs and suspends the pool. The PBS agents process enters D-state waiting for ZFS TXG sync on the suspended pool. After 737+ seconds, the kernel logs a hung task warning. Eventually the system hangs and watchdog-reboots.

The hub reconnects ~21 seconds after dropping, but ZFS remains suspended until manually cleared.

USB topology at failure:

xHCI Bus 2, Port 6 — [FAILING HUB]
  ├─ Port 1: ASM1153E ACAAEBBB2E5D → sdj → usb1-zfs mirror leg 2
  ├─ Port 2: ASM1153E ACAAEBBB2E5F → sdg → usb1-zfs mirror leg 1
  ├─ Port 3: ASM1153E ACAAEBBB2E60 → sdh → usb2-zfs mirror leg 1
  └─ Port 4: ASM1153E ACAAEBBB2E5E → sdi → usb2-zfs mirror leg 2

Both pools are behind the same hub. One hub drop affects both pools simultaneously.

UAS quirk already applied: usb-storage.quirks=174c:55aa:u in kernel cmdline disables UAS on the ASM1153E bridges. This was a pre-existing workaround for protocol-level issues but does not address hub power/connection failures.

Recovery procedure (manual steps, after hardware fix):

# 1. Confirm drives reconnected after hardware fix
lsblk | grep sd

# 2. Clear the suspended pool
zpool clear usb1-zfs

# 3. Run scrubs on both pools immediately
zpool scrub usb1-zfs
zpool scrub usb2-zfs

# 4. Monitor scrub progress (run ~every 10 min until complete)
zpool status -v

# 5. After scrub completes with 0 errors, run PBS backup verify
#    In PBS web UI → Datastore → usb1-zfs → Verify All
#    Or via CLI: proxmox-backup-client verify --all

# 6. If scrub finds unrepaired errors, see INC-002

Hardware fix options (ordered by preference):

1. Best: Connect each ASMT bridge directly to a motherboard USB 3.0 port (bypass hub entirely). xHCI has 10 downstream ports; only 2 are currently used (Port 2: Toshiba, Port 6: hub). Count available physical USB 3.0 jacks; plug each bridge directly.
2. Acceptable: Replace hub with an externally powered USB hub (dedicated power brick, not bus-powered). 4× spinning HDDs exceed typical bus-power budget (900mA), likely causing brownouts.
3. Do not: Return drives to any bus-powered hub.

Kernel command line note: The usb-storage.delay_use=10 param is a 10-second delay before using USB storage. This was also a pre-existing workaround and should be left in place post-fix.

---

INC-002 — PBS Backup Data Integrity Unknown (ACTIVE — verify post-recovery)

Context: usb1-zfs has been suspended on every one of the 34+ crash reboots since April 21. Each suspension may have left partially-written ZFS TXGs. The pool has 4 confirmed data errors in the current boot's suspension event.

Risk: Backup chunks written to usb1-zfs during or after a suspension event may be incomplete or corrupt. The PBS backup catalog may reference chunks that don't exist or are truncated.

Action required post-hardware-fix:

• Run zpool scrub usb1-zfs and verify 0 unrepaired errors.
• Run PBS verify on the usb1-zfs datastore (proxmox-backup-client verify --all, or via web UI).
• Any backup with a verify failure should be noted. If critical VMs have no verified backup, a fresh backup job should be run against a known-good target (Synology-Remote or rust-usb) immediately.

Status: Blocked on hardware fix (INC-001).

---

INC-003 — PBS Phase 1 C1 Re-Diagnosis

Original diagnosis (Phase 1): usb1-zfs and usb2-zfs fail to import at boot due to stale cachefile. Fix applied: cleared cachefile, added zpool import -f to boot sequence.

Revised diagnosis: The import failures were a downstream symptom. The actual root cause is INC-001 (USB hub drop). The pool is suspended at the time of crash-reboot; importing a suspended pool fails regardless of cachefile state. The cachefile fix helped on some boots by switching to scan-based import, but the underlying hardware problem continued causing the failures.

Lesson: When troubleshooting recurring import failures, always check dmesg for USB disconnect events before the failure. A suspended pool at boot is usually caused by the prior boot's crash, not a cachefile issue.

---

INC-004 — sdj Drive End-of-Life (PENDING)

Drive: Hitachi HUA723030ALA640, Serial MK0371YVGRZ2AA, /dev/sdj
Pool: usb1-zfs mirror leg 2
Power-on hours: 93,267 (≈10.6 years)
SMART: Passes all checks. No reallocated or pending sectors. 1 UDMA CRC error (USB interface, consistent with hub issue, not drive failure).

Action: Order a replacement 3TB+ drive (HGST Ultrastar or WD Gold recommended for NAS duty). Plan resilver after INC-001 hardware fix and INC-002 scrub pass are both confirmed clean. Do not start a resilver on a hub that may still be marginal.

---

INC-006 — ASM1153E UAS Reset Loop: Class-Wide Issue (Beast + any future Cenmate host)

Date discovered: 2026-05-07
Status: PARTIALLY MITIGATED — Beast grub updated; pending reboot to activate

Affected hardware: All Cenmate USB enclosures using ASMedia ASM1153E bridges (USB ID 174c:55aa). Confirmed on:

• PBS 4-bay (4× ASM1153E on xHCI Bus 2 Port 6) — quirk already applied, root cause of INC-001
• Beast 2-bay das-mirror (2× ASM1153E, /dev/sde + /dev/sdf) — quirk applied 2026-05-07, Beast reboot pending
• Unit 2 (2-bay, location TBD via physical inspection) — quirk status unknown until host identified

Root cause: The Linux UAS driver has a known interaction with ASM1153E firmware that causes periodic uas_eh_abort_handler / uas_eh_device_reset_handler cycles (~every 60s) under load. This is a chip-family pattern on Linux, not specific to any one enclosure or host. The fix is to force usb-storage mode (disabling UAS): kernel parameter usb-storage.quirks=174c:55aa:u.

This is a class issue, not a one-off. Every host with an ASM1153E-bridged enclosure needs this kernel parameter.

Impact on Beast das-mirror: Pool has been ONLINE and scrub-clean (last scrub 2026-04-12, 0 errors), but UAS resets were occurring in the background. Scrub result may have been from a window without active resets, or the pool survived despite them. Do NOT run zpool scrub das-mirror until Beast has rebooted with the quirk active — scrubbing under the buggy UAS driver generates noise and cannot be trusted as a clean baseline.

Actions taken:

• /etc/default/grub on Beast updated: GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on usb-storage.quirks=174c:55aa:u"
• update-grub run successfully on Beast; quirk present in all kernel entries in /boot/grub/grub.cfg
• Grub is archived (not templated) in Ansible; next run of backup-node-configs.yml will capture the new config

Remaining actions:

1. Reboot Beast at next maintenance window → verify dmesg | grep uas shows no reset loops
2. Run zpool scrub das-mirror post-reboot as first clean baseline
3. When Unit 2 is located: identify its host and apply same quirk before relying on it for PBS pool migration
4. If any additional Cenmate enclosures are added to the cluster, apply quirk at provisioning time

---

Phase 4 — Completed Work

Phase 4B — UPS Self-Tests + Alert (completed 2026-05-06)

• Manual tests run: cyberpower1 and cyberpower2 both passed.
• Script /usr/local/bin/ups-quarterly-test.sh updated to write Prometheus textfile metrics (ups_battery_test_result, ups_battery_test_timestamp_seconds).
• Cron installed: /etc/cron.d/ups-quarterly-test — first Sunday of Jan/Apr/Jul/Oct at 03:00 on Beast.
• Alert group ups_battery_test added: UPSBatteryTestFailed (result==0, for:5m), UPSBatteryTestStale (>100 days, for:6h).
• Break-tested and confirmed.
• Committed: docker-configs and ansible repos.

Phase 4C — Root-Cause Documentation (completed 2026-05-06)

• W11 (compute6 xtables): Race between pve-firewall iptables_restore (no -w flag) and Docker's iptables backend. Self-resolves in 10-120s. No firewall gap. No action taken.
• W12 (compute2 qdevice): qnetd on PBS; PBS crash-rebooting 34+ times since April 21 (same as INC-001). Each reboot drops qnetd for ~60s. Cluster quorum unaffected.
• Documented in runbooks/W11-W12-root-cause-2026-05.md.

Phase 4A — PVE Version Drift (PAUSED — pending PBS stability)

• Scope: Roll beast (9.1.6) and compute5 (9.1.6) to match cluster majority (9.1.7+).
• Procedure: One node per session, 24h soak. Migrate guests off → upgrade → reboot → verify Keepalived/quorum/storage → migrate back.
• Hold until: PBS hardware fix confirmed + usb1-zfs scrub clean + backup verify complete.

---

Phase 5 — Pending Work Queue

P5-01 — PBS Recovery (blocked on hardware)

Sequence: hardware fix → zpool clear → scrub → PBS verify → confirm stable 48h → resume Phase 4A

P5-02 — sdj Replacement (after P5-01 completes)

Order drive → resilver → verify → remove old sdj

P5-03 — qnetd Migration: PBS → Pi4 (after P5-01 + PBS stability confirmed)

Move corosync-qnetd from PBS (.153) to Pi4 (.227). Steps documented in runbooks/W11-W12-root-cause-2026-05.md. 30-minute operation, no VM downtime, no migration.

P5-04 — Authelia Prometheus Metrics + Request-Rate Alert

The C3 issue (May 5 health check: i/o timeouts every 2 min under scanner load) is not caught by the existing ContainerRestartLoop alert. Requires:

• Enable Authelia's Prometheus metrics endpoint (telemetry.metrics in configuration.yml)
• Add Prometheus scrape target for Authelia metrics
• Add alert: AutheliaHighErrorRate — authelia_request_duration_seconds_count{status="408"} rate elevated for >5m

P5-05 — TLS Cert Renewal Verification (RESOLVED 2026-05-16)

4 certs were in the 30–35d expiry window during the cert break-test (~May 5).
Traefik should auto-renew via ACME when ≤30d remain.

RESOLVED: All certs renewed to July 22 2026 (67 days remaining) — confirmed via live openssl check 2026-05-16. Traefik renewed automatically before expiry.

Original action schedule (for reference):

• May 11: SSH to active Traefik node (docker-node01), inspect acme.json, confirm the 4 certs have new notAfter dates.

  ssh tommy@192.168.99.186 "cat ~/traefik/acme.json | python3 -c \"
  import json,sys,datetime
  d=json.load(sys.stdin)
  for r in d.get('myresolver',{}).get('Certificates',[]):
      domain=r['domain']['main']
      # notAfter is base64 DER; use openssl or traefik logs to verify
      print(domain)
  \" 2>/dev/null || docker logs traefik --since 168h 2>&1 | grep -i 'renewed\|certificate\|acme'"
  

• May 18: Repeat check. If any cert still has the original expiry date, force-renew manually.

P5-06 — Traefik node02 CrowdSec LAPI (intentional asymmetry — no action)

node02 has no local CrowdSec; it correctly points to node01's LAPI at 192.168.99.186:8081 per Phase 2A design. Not drift. Known limitation: if node01 is down during failover, node02 Traefik has no CrowdSec coverage.

P5-07 — ansible-control Disk Cleanup / Expansion

At 80%, DiskFillPredicted24h firing. Clean /tmp artifacts and old report files, or expand the LVM.

P5-08 — Pi4 node-exporter Wrong Architecture Binary

SSH works as tommy. DNS (Technitium) and fail2ban both active. node-exporter fails with Exec format error — binary at /usr/local/bin/node_exporter is wrong architecture (x86_64 installed on ARM64). Fix: deploy correct arm64 binary.

VERSION=1.8.2
curl -sL https://github.com/prometheus/node_exporter/releases/download/v${VERSION}/node_exporter-${VERSION}.linux-arm64.tar.gz \
  | tar -xz --strip-components=1 -C /tmp node_exporter-${VERSION}.linux-arm64/node_exporter
scp /tmp/node_exporter tommy@192.168.99.227:/tmp/node_exporter
ssh tommy@192.168.99.227 "sudo mv /tmp/node_exporter /usr/local/bin/ && sudo chmod 755 /usr/local/bin/node_exporter && sudo systemctl restart node_exporter"

Pi4 hostname is also raspberrypi (clone drift — set to pi4 or similar).

P5-09 — Phase 4A Resume (PVE Version Drift)

After P5-01 completes and PBS is confirmed stable for 48h.

P5-10 — Pi4 node-exporter ARM64 Deploy

See P5-08.

P5-11 — Compute5 SK Hynix PC711 PCIe Power Management (monitor only)

nvme1n1 on compute5 (SK Hynix PC711 1TB, 0000:03:00.0) has 2,362 power cycles and 84 unsafe shutdowns — indicative of PCIe runtime PM aggressively power-cycling the drive.

Quirk status verified 2026-05-06: The kernel applies platform quirk: setting simple suspend automatically to both nvme0 and nvme1 — it is a built-in driver quirk for this CPU/chipset (i7-13700T platform), not a cmdline parameter. /etc/kernel/cmdline does not exist; /proc/cmdline has no nvme_core flags. The quirk persists across kernel updates by default. No user action required.

Monitor: check Unsafe Shutdowns and Power Cycles in SMART at each health check. If counts continue accumulating after the quirk is active, escalate to drive replacement or PCIe slot investigation. WD PC SN740 (nvme0) on same node: 56 unsafe shutdowns in 1,407h — attributed to pre-journal setup period and PCIe PS interaction; no action unless accumulating.

---

P5-12 — Weekly Health Digest (DONE 2026-05-16)

Deliverable: monitoring/weekly-health-digest.sh
Deployed: media-server ~/ansible/scripts/weekly-health-digest.sh
Cron: /etc/cron.d/weekly-health-digest — Monday 07:00 CDT
ntfy topic: health-digest on 192.168.99.198:8095 (dedicated topic, not grafana-alerts)

Lightweight weekly drift-catcher. Covers: PVE reachability, Prometheus target health, firing alerts, PBS backup status, PBS ZFS pool presence, filesystem >85%, ZFS pool state (das-mirror/compute3-zfs), Docker unhealthy containers, TLS cert expiry <21d.

Does NOT replace the quarterly full health check. Full sweeps documented in ~/ansible/reports/health-check-*.md.

Break-tested 2026-05-16 08:54 CDT: correctly flagged PBS ZFS MISSING + Grafana firing alerts. Suppressed known-persistent issues (P5-10 .227, HAOS .100, immich_machine_learning). ntfy delivery confirmed.

Subscriber reminder: subscribe ntfy client to topic health-digest on 192.168.99.198:8095.

---

P5-13 — media-server Kernel Upgrade 6.8.0-124 → 6.8.0-134 (DONE 2026-07-01)

Host: media-server (192.168.99.183 / VM 101) Rebooted: 2026-07-01 23:54 CDT into 6.8.0-134-generic (was running -124).

Kernel was staged earlier that day (6.8.0-134 on disk; nvidia-srv/580.159.03 DKMS module built + signed for -134). GRUB_DEFAULT=0 → reboot booted -134; -124 and -117 remain DKMS-built in the GRUB Advanced menu as fallbacks. Rollback strategy = PBS live backup of VM 101 (nvmebak storage can't snapshot without shutdown).

Post-boot checks — all green:

• uname -r = 6.8.0-134-generic
• nvidia-smi — both Quadro P2200s present, driver 580.159.03 loaded; dkms status shows nvidia-srv built for -134
• 54/54 containers running, none restarting/unhealthy
• 18/18 Prometheus targets up
• 0 failed systemd units
• /mnt/media (TrueNAS .29 NFS) mounted

⚠️ Gluetun caveat — a stale PIA WireGuard peer does NOT survive a host reboot. After the reboot gluetun came back on the last-written wg0.conf (Jun-30 peer, endpoint 181.41.206.145) which no longer passed traffic: tight VPN restart loop, healthchecks failing with dial tcp4: lookup github.com/cloudflare.com: i/o timeout, public_ip stuck at "". Even though that peer had been freshly registered Jun 30, it did not survive the cold start. Fixed by running the documented self-heal:

~/downloads/pia-wg/regen-and-up.sh    # regen fresh peer → recreate gluetun → reconnect sab/transmission

Result: fresh us_denver peer (endpoint 181.41.206.159:1337), gluetun Healthy, egress verified as PIA Denver, Colorado. Treat regen-and-up.sh as a required post-reboot step for the downloads stack — don't wait for the VpnTunnelDown alert.

---

Known Config Drift (not incidents — track for cleanup)

Item	Location	Drift
Hostname mismatch	docker-node01 (.186)	/etc/hostname = ubuntu-template; should = docker-node01
crowdsecLapiHost	node02 dynamic_conf.yml	Points to 192.168.99.186:8081 (node01 IP); should point to local LAPI
Technitium DNS	Pi4	No internal overrides for goattw.net (relies on Pi-hole as primary)
UPS self-test never run	cyberpower1	Resolved by Phase 4B; now automated
PBS ZFS pool feature upgrade	usb1-zfs, usb2-zfs	zpool upgrade deferred until pools are stable post-hardware fix
grub cmdline not Ansible-enforced	PBS, Beast	usb-storage.quirks=174c:55aa:u applied manually; archived by backup playbook but not templated. Risk: silent regression if grub is rewritten by kernel upgrade or manual edit. Convert to Ansible-managed template for USB-quirk nodes. Not urgent.
Alertmanager permission errors	media-server	Resolved (C2 May-05) — no action needed
VAULT1 orphan dirs on sda8	HAOS /media	VAULT1.local-orphan-20260501-2200 (982 MB) and VAULT1.local-orphan-20260523-0224 (299 GB) on sda8. ~22 days of recordings accumulated on HAOS system partition while CIFS was down. VAULT1 CIFS is now fixed (2026-05-23). Orphan dirs are outside Frigate's media path — Frigate will NOT clean them up. Move useful recordings to /media/VAULT1 manually, or delete once retention window passes. sda8 has 1.4 TB free so no immediate risk.

---

Runbooks

RB-001 — Frigate Cameras Down: VAULT1 USB Storage Outage

Last triggered: 2026-05-22 ~06:39 CDT (14:09 UTC) — 7.5 h blackout, resolved ~16:32 CDT (21:32 UTC)

Symptom

• Frigate /api/stats returns HTTP 500 (instead of normal 200 + JSON)
• All cameras show offline in Frigate UI (no live stream or thumbnails)
• Uptime Kuma "Frigate Stats" monitor fires → ntfy grafana-alerts alert

Hardware context (HAOS at 192.168.99.100)

• VAULT1 USB enclosure: JMicron JMS56x (VID:PID 152d:a580), USB 3.0 SuperSpeed, port 2-5
• Self-powered (bmAttributes=0xc0, bMaxPower=8mA) — has external PSU; bus-power issues are NOT the cause
• Directly connected to xHCI host controller (no intermediate hub — port 2-5, not 2-5.x)
• Two Coral USB TPUs also present: 1-2 (USB 2.0) and 2-7 (USB 3.0); these are independent

Diagnosis steps

# 1. Confirm Frigate stats API is down
curl -s -o /dev/null -w "%{http_code}" http://192.168.99.100:5000/api/stats

# 2. SSH into HAOS, check block devices
ssh -p 22222 hassio@192.168.99.100
sudo lsblk   # VAULT1 shows as sdb; if absent, USB enclosure is disconnected

# 3. Check for D-state (uninterruptible sleep) processes blocking on lost device
sudo cat /proc/*/status 2>/dev/null | grep -B5 "State:.*D"

# 4. Verify USB enclosure is still enumerated
sudo ls /sys/bus/usb/devices/ | grep '2-5'   # absent = USB disconnect

# 5. Check supervisor journal for USB and VAULT mount events
TOKEN=$(sudo cat /proc/*/environ 2>/dev/null | tr '\0' '\n' | grep '^SUPERVISOR_TOKEN=' | head -1 | cut -d= -f2-)
curl -s -H "Authorization: Bearer $TOKEN" 'http://supervisor/host/logs?lines=500' \
  | grep -iE 'usb|vault|sd[a-z]|disconnect|error' | tail -40

Recovery

1. Physically reseat the VAULT1 USB enclosure — unplug USB cable, wait 5 s, replug. The enclosure has its own PSU so check the PSU/power cable if USB reseat alone doesn't restore the device.
2. Verify block device reappears: sudo lsblk should show sdb/sdb1 again.
3. Trigger HAOS host reboot via supervisor API (fastest path — Frigate add-on reinitializes cleanly):

   TOKEN=$(sudo cat /proc/*/environ 2>/dev/null | tr '\0' '\n' | grep '^SUPERVISOR_TOKEN=' | head -1 | cut -d= -f2-)
   curl -s -X POST -H "Authorization: Bearer $TOKEN" http://supervisor/host/reboot
   

4. Wait ~3 minutes for HAOS to come back. Confirm Frigate stats: curl http://192.168.99.100:5000/api/stats returns 200.
5. If cameras still offline after reboot, restart Frigate add-on: curl -X POST -H "Authorization: Bearer $TOKEN" http://supervisor/addons/ccab4aaf_frigate-fa/restart

Recovery time

~3 minutes from enclosure reseat to Frigate healthy (HAOS reboot path). ~10–15 minutes if Frigate add-on restart is needed instead of full reboot.

VAULT1 CIFS credential recovery (when credential file is lost)

Root cause (identified 2026-05-23, investigated against HAOS supervisor journal):

The credential file loss is NOT caused by unclean shutdown. Confirmed: credential file survived May 6, May 21, and May 22 power-event boots (VAULT1 CIFS mounted successfully on all three). The file was deleted at 2026-05-22 00:53:31 UTC, 13 minutes after a successful mount on the first HAOS 17.3 boot (rauc slot B).

What happened: The new HAOS 17.3 supervisor (2026.05.x) performed a mount reload cycle (stop-start) on mnt-data-supervisor-mounts-VAULT1. The STOP step at 00:53:31 completed (Deactivated successfully) and deleted the credential file as part of cleanup. The immediately-following START step then failed with error 2: No such file or directory opening credential file — because the supervisor deleted the file but does not regenerate it from stored state on remount.

Trigger for the reload: Unknown — likely the new supervisor version's startup validation of existing mounts, or a brief Samba connectivity blip that caused the supervisor to cycle the mount. VAULT2 was also reloaded at 00:54:12 but succeeded because VAULT2's credential file was not deleted (race condition or ordering difference).

Pattern: This will recur every time the supervisor decides to cycle the VAULT1 CIFS mount. HAOS update to a newer supervisor version may fix or change this behavior.

NUT/UPS protection gap (resolved 2026-05-23): HAOS previously had no NUT client. Unclean shutdowns from power events (cyberpower UPS→Beast→CT104 NUT chain, but HAOS was not a slave) could corrupt the journal but do NOT cause credential file loss per above analysis. The HAOS Community NUT addon (a0d7b954_nut) has been installed and configured — see Prevention/monitoring section for details.

The supervisor credential file at /mnt/data/supervisor/.mounts_credentials/VAULT1 can be lost after the HAOS supervisor cycles the CIFS mount. Symptoms: supervisor log shows error 2 (No such file or directory) opening credential file every 15 min; /media/VAULT1 falls through to sda8 and Frigate writes recordings there instead of the USB drive.

To recover (requires ~2 min Frigate downtime):

# 1. Stop Frigate
TOKEN=$(sudo cat /proc/*/environ 2>/dev/null | tr '\0' '\n' | grep '^SUPERVISOR_TOKEN=' | head -1 | cut -d= -f2-)
curl -s -X POST -H "Authorization: Bearer $TOKEN" http://supervisor/addons/ccab4aaf_frigate-fa/stop

# 2. Move any accumulated orphan data off the path (Samba creds: hassio / check bootconfig.json)
sudo mv /media/VAULT1 /media/VAULT1.local-orphan-$(date -u +%Y%m%d-%H%M)

# 3. Re-PUT the mount with credentials (this recreates the credential file)
curl -s -X PUT -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"name":"VAULT1","type":"cifs","usage":"media","server":"172.30.32.1","share":"VAULT1","username":"hassio","password":"<from /addon_configs/1a32f091_sambanas/bootconfig.json>"}' \
  http://supervisor/mounts/VAULT1

# 4. Verify: df -h /media/VAULT1 should show //172.30.32.1/VAULT1 at 3.6T
# 5. Start Frigate
curl -s -X POST -H "Authorization: Bearer $TOKEN" http://supervisor/addons/ccab4aaf_frigate-fa/start

Orphan dirs on sda8 are outside Frigate's media path — handle manually (move or delete).

Prevention / monitoring

• Uptime Kuma "VAULT1 CIFS Mount" push monitor (id=45, token=yxhtvRPdixD4IY3Z4tVb, 10-min heartbeat timeout): HA automation pushes heartbeat every 5 min when VAULT1 is CIFS-mounted. Missed heartbeat → Kuma → ntfy grafana-alerts. Added 2026-05-23.
• HA binary_sensor vault1_cifs_mounted: grep -c '172.30.32.1/VAULT1' /proc/mounts every 5 min. State off for >3 min → direct ntfy alert (VAULT1 CIFS Lost – Frigate writing to sda8). Added 2026-05-23.
• Uptime Kuma "Frigate Stats" monitor (http://192.168.99.100:5000/api/stats, 60s, 2 retries) → ntfy grafana-alerts (added 2026-05-23)
• Uptime Kuma "Home Assistant" monitor (http://192.168.99.100:8123, 60s) → ntfy grafana-alerts (added 2026-05-23)
• USB enclosure is self-powered (own PSU) and directly connected — no hub to replace. If repeated disconnects occur, suspect: (a) PSU/power cable on enclosure, (b) USB cable quality, (c) motherboard USB port.
• HAOS NUT addon (a0d7b954_nut, installed 2026-05-23): netclient mode, monitors cyberpower1@192.168.99.200:3493 as secondary. Credentials: upsuser / Sparky$100 (same as CT104 slave). shutdown_host: true → runs /usr/bin/shutdownhost on FSD. NUT chain is now: Beast (master) → CT104 (secondary) + HAOS (secondary). Verified: upsc cyberpower1@192.168.99.200 returns full UPS data (OL, 36% load, battery 100%); Beast upsc -c cyberpower1 lists 192.168.99.100 as a connected client. No test.shutdown available on CP1500PFCLCDa firmware — live FSD test skipped (would cut UPS load).

---

RB-002 — Total Traefik Proxy 403 Outage: CrowdSec SQLite Journal Lock

Last triggered: 2026-05-24 ~03:02 UTC (22:02 CDT) — ~7 min outage, all services behind Traefik on docker-node01 (.186) returning 403. Resolved 03:11 UTC.

Symptom

• All services proxied by Traefik on .186 return HTTP 403 with empty body and ~10-second response time
• Browser shows "Access Denied"; curl -I https://home.goattw.net returns HTTP/2 403 with server: cloudflare (misleading — Cloudflare is passing the 403 through from the origin, not generating it)
• Traefik access log shows 403 0 … 10000ms or 10001ms on every router (homepage, grafana, immich, sabnzbd, smokeping, etc.)
• CrowdSec container shows Up and healthy; no obvious container crash

Why it looks like a Cloudflare block

The server: cloudflare header appears on all responses because Cloudflare proxies the domain. When the origin returns 403, Cloudflare passes the status through while adding its own headers. A bare content-length: 0 body (no CF HTML error page) is the tell that this is an upstream 403, not a CF WAF/Access block.

Diagnosis steps

# 1. Confirm 403 is from origin, not Cloudflare (bypass CF proxy)
curl -sI --resolve "home.goattw.net:443:192.168.99.186" https://home.goattw.net
# Expect: same 403 → origin is the source. Different result → CF-layer issue.

# 2. Check Traefik access log for the CrowdSec timeout signature (403 + ~10s response)
sudo tail -20 /home/tommy/traefik/logs/access.log
# Pattern to look for: "403 0" entries with "10000ms" or "10001ms" duration on EVERY router

# 3. Check Traefik container logs for middleware errors
docker logs traefik --tail 20 2>&1
# Look for: middleware "crowdsec-bouncer@file" does not exist  (rare startup race, not the lock issue)
# Look for: Error calling http://authelia:9091 (Authelia connectivity — separate issue)

# 4. Test CrowdSec LAPI directly — is it responding?
curl -s -o /dev/null -w "HTTP %{http_code}\n" \
  -H "X-Api-Key: VMCnws/j+9pmsT4YT+t3HzrvX8OhBCwoquwo4NqWJPs" \
  "http://localhost:8081/v1/decisions?ip=1.2.3.4"
# Healthy: HTTP 200 in <50ms.  Locked: hangs for seconds then HTTP 403.

# 5. Check CrowdSec logs for database lock errors
docker logs crowdsec --tail 30 2>&1 | grep -E "locked|error|warn"
# Signature: "database is locked" on fetching bouncer info, flushing metrics,
#            deleting orphan events, flushing allowlists

# 6. Inspect the data directory for a stale journal file
ls -lh /home/tommy/crowdsec/data/crowdsec.db*
# Stale lock: crowdsec.db-journal exists alongside crowdsec.db
# (WAL mode post-fix: crowdsec.db-wal and .db-shm may appear briefly during writes — normal)

# 7. Verify current journal mode
sudo sqlite3 /home/tommy/crowdsec/data/crowdsec.db "PRAGMA journal_mode;"
# Post-fix: should return "wal"
# Pre-fix (vulnerable state): returns "delete"

Root cause

CrowdSec's SQLite database in delete (rollback) journal mode holds an exclusive write lock for the entire duration of any write transaction. After ~27 hours of continuous operation, the database had grown to 8.2MB with significant fragmentation. CrowdSec's periodic maintenance tasks — flushing metrics, deleting orphan events, and flushing allowlists — all fired simultaneously at the top of the hour (03:00 UTC / 22:00 CDT), collectively acquiring and holding the exclusive write lock long enough to time out all concurrent bouncer reads. The Traefik CrowdSec bouncer plugin fails-closed on non-200 LAPI responses, returning HTTP 403 to Traefik, which passed 403s to every client.

Timeline (2026-05-24):

• 18:36 CDT May 22 — Docker bulk update completed; CrowdSec container started fresh on .186
• 22:00 CDT May 23 — CrowdSec's hourly maintenance tasks fired; cert-expiry-check.sh cron also ran (reads acme.json locally, makes no HTTP requests — coincidence, not a contributor)
• 22:02 CDT May 23 — First database is locked errors in CrowdSec LAPI log (27.5 h after container start)
• 22:09 CDT May 23 — crowdsec.db-journal last modified; DB had stopped accepting successful writes at 22:00
• 22:10 CDT May 23 — Diagnosed and CrowdSec stopped for repair
• 22:11 CDT May 23 — CrowdSec restarted; LAPI responding at 6–8 ms; all services restored

Trigger investigation: Checked journalctl -u docker for the 22:00–22:03 CDT window — no container restarts, no OOM events, no unexpected docker events. Scanner traffic (172.184.210.34) had zero hits in the two hours preceding the lock. No other cron jobs made HTTP requests at that time. The cause was entirely internal to CrowdSec: delete-mode SQLite contention under its own scheduled maintenance workload on a 27-hour-old, 8.2MB fragmented database.

Recovery

# Step 1: Back up the database before touching anything
sudo cp /home/tommy/crowdsec/data/crowdsec.db \
  /home/tommy/crowdsec/data/crowdsec.db.bak.$(date +%Y%m%d_%H%M%S)

# Step 2: Stop CrowdSec to release all DB locks
docker stop crowdsec

# Step 3: Verify integrity (must return "ok" before proceeding)
sudo sqlite3 /home/tommy/crowdsec/data/crowdsec.db "PRAGMA integrity_check;"

# Step 4: Switch to WAL mode and compact
# WAL mode: allows concurrent readers while writes are in-flight — eliminates the lock contention
# VACUUM: compacts fragmentation (DB was 8.2 MB → 3.5 MB after vacuum on 2026-05-24)
sudo sqlite3 /home/tommy/crowdsec/data/crowdsec.db "
PRAGMA journal_mode=WAL;
VACUUM;
PRAGMA wal_checkpoint(TRUNCATE);
PRAGMA journal_mode;
"
# Expected output: wal / 0|0|0 / wal
# The stale .db-journal file will be consumed and removed automatically

# Step 5: Start CrowdSec
docker start crowdsec
sleep 8   # wait for LAPI init

# Step 6: Verify LAPI is healthy
curl -s -o /dev/null -w "HTTP %{http_code}\n" \
  -H "X-Api-Key: VMCnws/j+9pmsT4YT+t3HzrvX8OhBCwoquwo4NqWJPs" \
  "http://localhost:8081/v1/decisions?ip=1.2.3.4"
# Expect: HTTP 200 in <50 ms. If still 403 or slow, repeat from Step 2.

# Step 7: Confirm end-to-end fix
curl -sI https://home.goattw.net | head -3
# Expect: HTTP/2 200

If PRAGMA integrity_check returns anything other than ok: The database is corrupted. Restore from backup or let CrowdSec reinitialize (decisions and alerts will be lost):

docker stop crowdsec
sudo rm /home/tommy/crowdsec/data/crowdsec.db /home/tommy/crowdsec/data/crowdsec.db-journal
docker start crowdsec
# CrowdSec will create a fresh empty DB on startup
# Re-register bouncer key after restart if needed

Recovery time

~3 minutes (stop → repair → start → verified).

Prevention / monitoring

• WAL mode is now persistent — the journal mode is stored in the DB file header and survives CrowdSec container restarts. After any future CrowdSec container recreation (docker compose up -d with image update), verify:

  sudo sqlite3 /home/tommy/crowdsec/data/crowdsec.db "PRAGMA journal_mode;"
  # Must return: wal
  

  If it returns delete, re-run Step 4 above before the next high-traffic period.
• Backup cleanup: Remove crowdsec.db.bak.* files after 48 h of stable operation.
• The crowdsec.db-wal and crowdsec.db-shm files that appear with WAL mode are normal — do not delete them while CrowdSec is running. They will be cleaned up on next checkpoint.
• Root-cause prevention: WAL mode eliminates the delete-journal write-lock contention. No additional rate limiting or CrowdSec configuration change is needed. The --entrypoints.websecure.http.middlewares=crowdsec-bouncer@file global setting in Traefik's compose is intentional (all-entrypoint protection) but means a CrowdSec LAPI failure is a total outage — consider adding crowdsecLapiTimeout to the bouncer config or setting a shorter timeout if reduced blast radius is desired in future.