Full diagnosis + fix for "Unknown error during upload" on large files:
root cause is browser DNS-over-HTTPS bypassing LAN DNS -> Cloudflare mangles
the parallel chunk stream. Server/DNS already correct (Technitium+UniFi return
internal .185). Fix = disable browser DoH (.reg/PowerShell/GPO included).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Captures the beast.goattw.net -> 192.168.99.200 intent (LIVE state still wrong at
.185, correction never took), the NUT-must-not-depend-on-DNS design rule (ansible
3e5bfc4), and the open Technitium .184 config-backup verification.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sync pbs/zfs-health-check.sh to the deployed version and log the full
INC-001 resolution arc:
- recovery by importing pools on stable /dev/disk/by-id paths
- by-id cachefile pin so the fix survives USB re-enumeration
- wait-for-zfs-usb.sh + zfs-import-cache drop-in for the boot race
- alert split: devices-present (import) vs devices-absent (hardware),
new zfs_pool_members_present metric
- cron ownership fix: root-managed single entry, tommy evicted, unmarked
root line removed (double-fire root cause)
Marks the acute import/alert half of INC-001 resolved; USB-hub reliability
history retained as a separate hardware concern.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rebooted 2026-07-01; post-boot checks green (nvidia/DKMS, 54 containers,
18 Prom targets, 0 failed units, /mnt/media). Documents the gluetun
caveat: a stale PIA WG peer does not survive a reboot -> run
pia-wg/regen-and-up.sh as a required post-reboot step.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
HA: add binary_sensor.vault1_cifs_mounted (grep /proc/mounts every 5 min),
rest_command.vault1_push_heartbeat, and two automations — heartbeat push to
Uptime Kuma push monitor (id=45, 10-min timeout) and direct ntfy alert if
CIFS drops for >3 min. Catches the sda8-fallthrough scenario in <8 min.
Runbooks: root cause of credential file loss is NOT unclean shutdown but
the HAOS 17.3 supervisor (2026.05.x) deleting the cred file during mount
reload cycles. Exact moment: 2026-05-22 00:53:31 UTC, 13 min after the
HAOS 17.3 first-boot successful mount. NUT gap documented (HAOS not a NUT
slave); recommended to add Community NUT addon targeting cyberpower1@.200.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
VAULT1 supervisor mount had missing credential file since at least
2026-05-01 (lost after unclean shutdown). 22 days of Frigate recordings
accumulated on sda8 (299 GB) instead of the 4 TB USB drive. Fixed via
PUT /mounts/VAULT1 with hassio credentials; mount now shows 3.6T/915G.
Orphan dirs documented; RB-001 updated with credential recovery steps.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Symptom→Diagnosis→Recovery for the 2026-05-22 7.5h Frigate blackout.
Key findings: JMicron JMS56x enclosure (2-5) is self-powered and directly
connected; VAULT1 CIFS credential-file errors are a separate pre-existing
config drift, not the root cause. Also adds VAULT1 CIFS drift entry.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
INC-006: PBS upsmon DEADTIME=15s caused autonomous shutdown when Beast
NUT server was briefly unreachable during reboot. Fixed 2026-05-16 by
setting DEADTIME 300 on PBS.
INC-001: usb1-zfs and usb2-zfs manually imported 2026-05-16 09:14 CDT
after May 14 reboot left both MISSING. Both ONLINE, no errors. Scrubs
initiated. Cachefile updated at /etc/zfs/zpool.cache.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
P5-12 (weekly-health-digest): documented scope, deployed location, cron
schedule, ntfy topic, break-test result. Removes duplicate
monitoring/PHASE5-incidents.md — P5-12 belongs in the canonical
runbooks/phase5-incident-log.md alongside all other phase items.
INC-001 state updated: PBS rebooted May 14; both usb1-zfs and usb2-zfs
now MISSING (usb2-zfs was previously ONLINE on May 5 — new degradation).
zfs-health-check alerting every 5 min to ntfy/zfs-health.
P5-05 marked RESOLVED: all certs renewed to Jul 22 2026, confirmed via
live openssl check on 2026-05-16.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
platform quirk 'simple suspend' is applied by PVE kernel automatically
for i7-13700T platform (both nvme0 and nvme1). Not a cmdline parameter;
/etc/kernel/cmdline absent. Persists across kernel updates by default.
Verified: dmesg confirms quirk active on both drives at current boot.
P5-11 status: monitor only, no user action required.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
W11 (compute6 pve-firewall xtables conflicts): race condition between
pve-firewall's iptables_restore (no -w flag) and Docker's iptables
backend. Self-resolves within 10-120s; no firewall gap; no action taken.
Conflicts align to :58 minute mark = Docker daemon start time at last
compute6 boot (Apr 19 12:58).
W12 (compute2 corosync-qdevice boot failures): qnetd on PBS (.153);
PBS has crash-rebooted 34 times since Apr 21, each correlated with USB
ZFS import failure (same as C1, May 5 health check). Each PBS reboot
drops qnetd for ~60s; cluster quorum unaffected (6 node votes, threshold
3). No action this phase. Phase 5 items: fix PBS USB ZFS instability;
optionally migrate qnetd from PBS to Pi4 (.227).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
tommy
tommy