Commit Graph

8 Commits

Author SHA1 Message Date
Tommy e61fa8fe60 docs(runbook): HAOS NUT addon installed and verified
- NUT addon a0d7b954_nut installed as netclient secondary on HAOS
- Monitors cyberpower1@192.168.99.200:3493 (upsuser/Sparky$100)
- HAOS confirmed in Beast's connected client list (upsc -c)
- Replaces "Pending" NUT gap entry in RB-001 Prevention section
- Updated root cause paragraph: NUT gap resolved 2026-05-23

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-23 22:00:46 -05:00
Tommy af09358149 ha: VAULT1 CIFS mount monitor; runbooks: supervisor root-cause
HA: add binary_sensor.vault1_cifs_mounted (grep /proc/mounts every 5 min),
rest_command.vault1_push_heartbeat, and two automations — heartbeat push to
Uptime Kuma push monitor (id=45, 10-min timeout) and direct ntfy alert if
CIFS drops for >3 min. Catches the sda8-fallthrough scenario in <8 min.

Runbooks: root cause of credential file loss is NOT unclean shutdown but
the HAOS 17.3 supervisor (2026.05.x) deleting the cred file during mount
reload cycles. Exact moment: 2026-05-22 00:53:31 UTC, 13 min after the
HAOS 17.3 first-boot successful mount. NUT gap documented (HAOS not a NUT
slave); recommended to add Community NUT addon targeting cyberpower1@.200.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-23 21:46:44 -05:00
Tommy 7e8ea78cee runbooks: fix VAULT1 CIFS creds + add credential recovery procedure
VAULT1 supervisor mount had missing credential file since at least
2026-05-01 (lost after unclean shutdown). 22 days of Frigate recordings
accumulated on sda8 (299 GB) instead of the 4 TB USB drive. Fixed via
PUT /mounts/VAULT1 with hassio credentials; mount now shows 3.6T/915G.
Orphan dirs documented; RB-001 updated with credential recovery steps.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-23 21:27:14 -05:00
Tommy 50e8dfccc7 runbooks: add INC-007 Frigate/VAULT1 USB outage runbook (RB-001)
Symptom→Diagnosis→Recovery for the 2026-05-22 7.5h Frigate blackout.
Key findings: JMicron JMS56x enclosure (2-5) is self-powered and directly
connected; VAULT1 CIFS credential-file errors are a separate pre-existing
config drift, not the root cause. Also adds VAULT1 CIFS drift entry.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-23 21:09:54 -05:00
tommy 4372080a3b docs: add INC-006 (PBS upsmon DEADTIME), update INC-001 pool state
INC-006: PBS upsmon DEADTIME=15s caused autonomous shutdown when Beast
NUT server was briefly unreachable during reboot. Fixed 2026-05-16 by
setting DEADTIME 300 on PBS.

INC-001: usb1-zfs and usb2-zfs manually imported 2026-05-16 09:14 CDT
after May 14 reboot left both MISSING. Both ONLINE, no errors. Scrubs
initiated. Cachefile updated at /etc/zfs/zpool.cache.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-16 16:49:30 -05:00
tommy 2ee7c474f5 docs: add P5-12 to phase5-incident-log, update INC-001 and P5-05 states
P5-12 (weekly-health-digest): documented scope, deployed location, cron
schedule, ntfy topic, break-test result. Removes duplicate
monitoring/PHASE5-incidents.md — P5-12 belongs in the canonical
runbooks/phase5-incident-log.md alongside all other phase items.

INC-001 state updated: PBS rebooted May 14; both usb1-zfs and usb2-zfs
now MISSING (usb2-zfs was previously ONLINE on May 5 — new degradation).
zfs-health-check alerting every 5 min to ntfy/zfs-health.

P5-05 marked RESOLVED: all certs renewed to Jul 22 2026, confirmed via
live openssl check on 2026-05-16.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-16 08:58:16 -05:00
tommy 640cd908df docs: P5-11 — compute5 nvme1 PCIe quirk verified, no action needed
platform quirk 'simple suspend' is applied by PVE kernel automatically
for i7-13700T platform (both nvme0 and nvme1). Not a cmdline parameter;
/etc/kernel/cmdline absent. Persists across kernel updates by default.
Verified: dmesg confirms quirk active on both drives at current boot.
P5-11 status: monitor only, no user action required.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-06 21:29:16 -05:00
tommy 909fe3dc12 docs: Phase 5 — incident log, maintenance runbook, NUT config archive
beast/nut/upssched.conf: fixed earlyshutdown timer (30s→300s) and added
  cancel rule. Archived post-fix from Beast.
beast/nut/upssched-cmd: archived for reference.

runbooks/phase5-incident-log.md:
  INC-005: NUT upssched earlyshutdown bug — cause, fix, break-test findings.
  INC-001 through INC-004: PBS USB hub/data/re-diagnosis/sdj end-of-life.
  P5-01 through P5-11: pending work queue including Pi4 node-exporter,
  compute5 PCIe PM, PBS recovery gate, qnetd migration, Authelia metrics,
  cert renewal verification, ansible-control disk, Phase 4A resume.

runbooks/maintenance-runbook.md: PBS recovery steps, UPS test procedure,
  cert renewal schedule (May 11/18), Phase 4A per-node procedure, qnetd
  migration, Authelia metrics setup, recurring checklist.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-06 21:25:49 -05:00