Sync pbs/zfs-health-check.sh to the deployed version and log the full
INC-001 resolution arc:
- recovery by importing pools on stable /dev/disk/by-id paths
- by-id cachefile pin so the fix survives USB re-enumeration
- wait-for-zfs-usb.sh + zfs-import-cache drop-in for the boot race
- alert split: devices-present (import) vs devices-absent (hardware),
new zfs_pool_members_present metric
- cron ownership fix: root-managed single entry, tommy evicted, unmarked
root line removed (double-fire root cause)
Marks the acute import/alert half of INC-001 resolved; USB-hub reliability
history retained as a separate hardware concern.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Fix silent failure: script now posts to dedicated zfs-health ntfy topic
instead of grafana-alerts catch-all (pools were offline 12+ hours
undetected because alerts were buried in Grafana noise)
- Three explicit states: ONLINE (silent), DEGRADED (high priority),
MISSING (urgent priority) — empty zpool list output is now a MISSING
alert, not silently ignored
- Textfile metrics written atomically after loop completes only:
zfs_pool_present{pool=X} 0|1 and zfs_health_last_run_seconds
- Added trap cleanup so mid-script crash leaves previous .prom intact
- Logs each pool state to syslog via logger -t zfs-health-check
- Remove duplicate cron entry running as tommy (was firing twice per tick)
- Enable node_exporter textfile collector for Prometheus scraping
Incident: usb1-zfs and usb2-zfs offline since PBS boot (missing cachefile).
Imported and cachefile regenerated in this session. No data errors.
Refs: 2026-05-05 health check CRITICAL C1/C4
tommy