ups-quarterly-test: sync tracked copy to live + fix false-alarm bugs

The tracked copy was a stale pre-refactor version (no sequential-test safety, no textfile metric). Replace with the current script plus fixes for the UPSBatteryTestFailed false alarm: - Poll ups.test.result until terminal ("passed"/"failed") instead of a single racy read — CyberPower updates the string slower than the OL transition, which recorded 0 despite a passing test. - to_gauge matches *passed* substring, not exact "Done and passed". - Fix cron OR-logic: setting both DOM (1-7) and DOW (0) made cron run on EITHER (daily Jul 1-7 + Sundays). Script now self-guards to day<=7; cron fires every Sunday of the quarter months. - Externalize NUT_PASS to /etc/ups-quarterly-test.env (root:600) so the password is no longer embedded in the repo. Deploy via ups-fix.sh. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
weekly-health-digest: add internet speed check (section 4c)
2026-07-07 18:31:46 -05:00 · 2026-07-07 18:13:16 -05:00 · 2026-07-07 17:39:24 -05:00 · 2026-07-06 22:31:40 -05:00 · 2026-07-06 22:15:45 -05:00 · 2026-07-05 19:07:04 -05:00
35 changed files with 1941 additions and 57 deletions
@@ -11,3 +11,4 @@ secrets.*.yaml
 credentials
 *credentials*
 *.secret
+crowdsec_lapi_key
@@ -0,0 +1,39 @@
+# 4K Quality-First arr stack — radarr-4k + sonarr-4k
+
+SEPARATE instances from the untouched 1080p stack (for Overseerr native-4K routing +
+isolation). Host: media-server (.183). Deployed by `deploy-arrs-4k.yml` (inline-compose,
+idempotent, changed=0).
+
+## Instances
+| | port (host:cont) | URL | root folder |
+|---|---|---|---|
+| radarr-4k | 7879:7878 | radarr-4k.goattw.net | /data/media/movies-4k |
+| sonarr-4k | 8990:8989 | sonarr-4k.goattw.net | /data/media/tv-4k |
+
+- Image `lscr.io/linuxserver/{radarr,sonarr}:latest`, PUID/PGID 1000, net `traefik_public`.
+- Mount `/mnt/media/data -> /data` — SAME single hardlink tree as the 1080p stack + download
+  clients (st_dev verified identical), so imports are instant hardlinks. NFS from
+  TrueNAS `Tank/Media`. Roots created 1000:3000 775 (match 1080p).
+
+## Wiring (Step 1, done)
+- **Prowlarr**: `Radarr (4K)`/`Sonarr (4K)` applications (fullSync), indexers synced.
+- **Overseerr**: radarr-4k/sonarr-4k as `is4k=true` servers (native 4K routing). The existing
+  1080p radarr was mislabeled `is4k=true` and was flipped to `is4k=false` (non-4k default).
+- **Plex**: SEPARATE libraries `Movies - 4K` (/data/media/movies-4k) + `TV - 4K` (/data/media/tv-4k),
+  not merged into the 1080p libraries.
+- **Download clients** (added post-Step-1, packaged in the playbook via `ensure-arrs-4k-dl.py`):
+  SABnzbd + Transmission on both instances, DISTINCT categories to avoid cross-import with 1080p —
+  sabnzbd `movies-4k`/`tv-4k`, transmission `radarr-uhd`/`sonarr-uhd` (Transmission category forbids digits).
+  Completed paths under /data (hardlinks intact).
+- **Homepage**: separate `Radarr 4K`/`Sonarr 4K` tiles in Media Automation (keys via HOMEPAGE_VAR, siteMonitor).
+- **Traefik**: file-provider routers (see traefik-router snippet), NOT docker labels.
+- **Tdarr**: does NOT watch movies-4k/tv-4k (its libraries are the specific 1080p /media/movies,
+  /media/tv) — quality-first keeps the grabbed file.
+
+## Step 2 — Recyclarr 4K (DONE)
+`deploy-recyclarr-4k.yml` (recyclarr image :8, config uses !env_var, .env from ansible-vault).
+Sync runs via **systemd timer** `recyclarr-4k-sync.timer` (daily 04:00, docker-run oneshot — NOT
+container cron). Profiles applied: radarr-4k = **Remux + WEB 2160p** (fd161a61), sonarr-4k =
+**WEB-2160p** (d1498e7d), assigned to the Overseerr 4K servers (replaced Ultra-HD placeholder).
+**DTS family explicitly pinned score 0** (Sonos-Arc down-weight; re-forced every sync so it can't
+revert to guide 2500/4500) — Dolby/TrueHD/Atmos left at guide default (positive).
@@ -0,0 +1,180 @@
+---
+- name: Deploy 4K arr instances (radarr-4k, sonarr-4k) on media-server
+  hosts: media-server
+  become: true
+  vars:
+    app_dir: /home/tommy/arrs-4k
+
+  tasks:
+    - name: Create arrs-4k directories
+      ansible.builtin.file:
+        path: "{{ item }}"
+        state: directory
+        owner: tommy
+        group: tommy
+        mode: "0755"
+      loop:
+        - "{{ app_dir }}"
+        - "{{ app_dir }}/radarr-4k"
+        - "{{ app_dir }}/sonarr-4k"
+
+    - name: Write docker-compose.yml
+      ansible.builtin.copy:
+        dest: "{{ app_dir }}/docker-compose.yml"
+        owner: tommy
+        group: tommy
+        mode: "0644"
+        content: |
+          # 4K Quality-First arr instances — SEPARATE from the untouched 1080p stack.
+          # Host: media-server (.183). Shares /mnt/media/data -> /data (single hardlink tree).
+          # Roots: radarr-4k -> /data/media/movies-4k ; sonarr-4k -> /data/media/tv-4k
+          # Routing: Traefik FILE provider on node01 dynamic_conf.yml (NO docker labels):
+          #   radarr-4k.goattw.net -> 192.168.99.183:7879 ; sonarr-4k.goattw.net -> 192.168.99.183:8990
+          #   middlewares: secure-headers@file, authelia-auth@file, crowdsec-bouncer@file
+          # Tdarr does NOT watch movies-4k/tv-4k (quality-first keeps the grabbed file).
+
+          services:
+            radarr-4k:
+              image: lscr.io/linuxserver/radarr:latest
+              container_name: radarr-4k
+              mem_limit: 1g
+              cpus: 1.0
+              environment:
+                - PUID=1000
+                - PGID=1000
+                - TZ=America/Chicago
+              networks:
+                - traefik_public
+              volumes:
+                - ./radarr-4k:/config
+                - /mnt/media/data:/data
+              ports:
+                - "7879:7878/tcp"
+              restart: unless-stopped
+
+            sonarr-4k:
+              image: lscr.io/linuxserver/sonarr:latest
+              container_name: sonarr-4k
+              mem_limit: 1g
+              cpus: 1.0
+              environment:
+                - PUID=1000
+                - PGID=1000
+                - TZ=America/Chicago
+              networks:
+                - traefik_public
+              volumes:
+                - ./sonarr-4k:/config
+                - /mnt/media/data:/data
+              ports:
+                - "8990:8989/tcp"
+              restart: unless-stopped
+
+          networks:
+            traefik_public:
+              external: true
+
+    - name: Deploy 4K arrs with docker compose
+      community.docker.docker_compose_v2:
+        project_src: "{{ app_dir }}"
+        state: present
+        pull: missing
+      register: r
+
+    - name: Result
+      ansible.builtin.debug:
+        msg: "arrs-4k {{ 'running (changed)' if r.changed else 'already up (no-op)' }}"
+
+    - name: Wait for 4K arr APIs (ports)
+      ansible.builtin.wait_for:
+        host: 192.168.99.183
+        port: "{{ item }}"
+        timeout: 90
+      loop: [7879, 8990]
+
+    # Pre-create the download-client COMPLETED category dirs so a fresh rebuild
+    # doesn't surface "download client places downloads in X but it does not
+    # appear to exist" health errors until the first grab lands in each category.
+    # Owner 1000:3000 (abc:media) 0775 matches the rest of the /data tree — the
+    # download-client app processes create category dirs this way on first use;
+    # a manual `docker exec <client> mkdir` (runs as ROOT in linuxserver images)
+    # produces root:0770 instead, which is the drift this task prevents.
+    # become=root chown is honored by the TrueNAS NFS export (no_root_squash).
+    - name: Ensure 4K download-client completed category dirs (abc:media 0775)
+      ansible.builtin.file:
+        path: "{{ item }}"
+        state: directory
+        owner: "1000"
+        group: "3000"
+        mode: "0775"
+      loop:
+        - /mnt/media/data/usenet/complete/movies-4k   # SABnzbd  -> radarr-4k
+        - /mnt/media/data/usenet/complete/tv-4k        # SABnzbd  -> sonarr-4k
+        - /mnt/media/data/torrents/radarr-uhd          # Transmission -> radarr-4k
+        - /mnt/media/data/torrents/sonarr-uhd          # Transmission -> sonarr-4k
+
+    - name: Install 4K download-client config script
+      ansible.builtin.copy:
+        dest: "{{ app_dir }}/ensure-arrs-4k-dl.py"
+        owner: tommy
+        group: tommy
+        mode: "0755"
+        content: |
+          #!/usr/bin/env python3
+          # Idempotent: ensure the 4K arr download clients + sabnzbd categories exist.
+          # Reads RADARR4K_KEY / SONARR4K_KEY / SAB_KEY from env. Prints CHANGED or NOCHANGE.
+          import os, json, urllib.request
+          H = "http://192.168.99.183"
+          SAB = os.environ["SAB_KEY"]
+          changed = False
+          
+          def sab_cats():
+              return json.load(urllib.request.urlopen(f"{H}:8081/api?mode=get_cats&output=json&apikey={SAB}"))["categories"]
+          def sab_add(cat):
+              urllib.request.urlopen(f"{H}:8081/api?mode=set_config&section=categories&keyword={cat}&dir={cat}&apikey={SAB}").read()
+          def dc_get(port, key):
+              return json.load(urllib.request.urlopen(urllib.request.Request(f"{H}:{port}/api/v3/downloadclient", headers={"X-Api-Key": key})))
+          def dc_post(port, key, body):
+              urllib.request.urlopen(urllib.request.Request(f"{H}:{port}/api/v3/downloadclient",
+                  data=json.dumps(body).encode(), method="POST", headers={"X-Api-Key": key, "Content-Type": "application/json"}))
+          
+          def sab_body(catf, catv, recent, older):
+              return {"enable": True, "protocol": "usenet", "priority": 1, "name": "SABnzbd",
+                      "implementation": "Sabnzbd", "configContract": "SabnzbdSettings",
+                      "fields": [{"name": "host", "value": "gluetun"}, {"name": "port", "value": 8080},
+                                 {"name": "apiKey", "value": SAB}, {"name": catf, "value": catv},
+                                 {"name": recent, "value": -100}, {"name": older, "value": -100}]}
+          def tr_body(catf, catv):
+              return {"enable": True, "protocol": "torrent", "priority": 1, "name": "Transmission",
+                      "implementation": "Transmission", "configContract": "TransmissionSettings",
+                      "fields": [{"name": "host", "value": "gluetun"}, {"name": "port", "value": 9091},
+                                 {"name": "urlBase", "value": "/transmission/"}, {"name": catf, "value": catv}]}
+          
+          cats = sab_cats()
+          for c in ("movies-4k", "tv-4k"):
+              if c not in cats:
+                  sab_add(c); changed = True
+          
+          plan = [
+              (7879, os.environ["RADARR4K_KEY"], [sab_body("movieCategory", "movies-4k", "recentMoviePriority", "olderMoviePriority"),
+                                                  tr_body("movieCategory", "radarr-uhd")]),
+              (8990, os.environ["SONARR4K_KEY"], [sab_body("tvCategory", "tv-4k", "recentTvPriority", "olderTvPriority"),
+                                                  tr_body("tvCategory", "sonarr-uhd")]),
+          ]
+          for port, key, bodies in plan:
+              have = {c["name"] for c in dc_get(port, key)}
+              for b in bodies:
+                  if b["name"] not in have:
+                      dc_post(port, key, b); changed = True
+          
+          print("CHANGED" if changed else "NOCHANGE")
+
+    - name: Ensure 4K download clients + sabnzbd categories (idempotent, cats movies-4k/tv-4k)
+      ansible.builtin.command: "python3 {{ app_dir }}/ensure-arrs-4k-dl.py"
+      environment:
+        RADARR4K_KEY: "{{ vault_radarr4k_apikey }}"
+        SONARR4K_KEY: "{{ vault_sonarr4k_apikey }}"
+        SAB_KEY: "{{ vault_sabnzbd_apikey }}"
+      register: dl_result
+      changed_when: "'CHANGED' in dl_result.stdout"
+      no_log: true
@@ -0,0 +1,175 @@
+---
+- name: Deploy Recyclarr 4K (config + systemd-timer sync) on media-server
+  hosts: media-server
+  become: true
+  vars:
+    app_dir: /home/tommy/recyclarr-4k
+
+  handlers:
+    - name: reload systemd
+      ansible.builtin.systemd_service:
+        daemon_reload: true
+
+  tasks:
+    - name: Create recyclarr-4k config dir
+      ansible.builtin.file:
+        path: "{{ app_dir }}/config"
+        state: directory
+        owner: tommy
+        group: tommy
+        mode: "0755"
+
+    - name: Write recyclarr.yml (no secrets; URLs/keys via !env_var)
+      ansible.builtin.copy:
+        dest: "{{ app_dir }}/config/recyclarr.yml"
+        owner: tommy
+        group: tommy
+        mode: "0644"
+        content: |
+          # yaml-language-server: $schema=https://schemas.recyclarr.dev/v8/config-schema.json
+          # 4K quality-first: radarr-4k (Remux+WEB 2160p) + sonarr-4k (WEB-2160p)
+          # TRaSH guide profiles + recommended naming + DTS neutral (Sonos Arc down-weight).
+          
+          radarr:
+            radarr-4k:
+              base_url: !env_var RADARR_4K_URL
+              api_key: !env_var RADARR_4K_APIKEY
+          
+              quality_definition:
+                type: movie
+          
+              quality_profiles:
+                - trash_id: fd161a61e3ab826d3a22d53f935696dd  # Remux + WEB 2160p
+                  reset_unmatched_scores:
+                    enabled: true
+          
+              custom_format_groups:
+                add:
+                  - trash_id: ff204bbcecdd487d1cefcefdbf0c278d  # [Optional] Golden Rule UHD
+                    select:
+                      - 839bea857ed2c0a8e084f3cbdbd65ecb  # x265 (no HDR/DV)
+                  - trash_id: a3ac6af01d78e4f21fcb75f601ac96df  # [Unwanted] Unwanted Formats
+                    select:
+                      - b8cd450cbfa689c0259a01d9e29ba3d6  # 3D
+                      - cae4ca30163749b891686f95532519bd  # AV1
+                      - b6832f586342ef70d9c128d40c07b872  # Bad Dual Groups
+                      - cc444569854e9de0b084ab2b8b1532b2  # Black and White Editions
+                      - ed38b889b31be83fda192888e2286d83  # BR-DISK
+                      - 0a3f082873eb454bde444150b70253cc  # Extras
+                      - e6886871085226c3da1830830146846c  # Generated Dynamic HDR
+                      - 90a6f9a284dff5103f6346090e6280c8  # LQ
+                      - e204b80c87be9497a8a6eaff48f72905  # LQ (Release Title)
+                      - 712d74cd88bceb883ee32f773656b1f5  # Sing-Along Versions
+                      - bfd8eb01832d646a0a89c4deb46f8564  # Upscaled
+          
+              media_naming:
+                folder: default
+                movie:
+                  rename: true
+                  standard: standard
+          
+              # DTS down-weight (Sonos Arc): neutral score 0 (NOT negative).
+              # Dolby/TrueHD/Atmos left at guide default (unset here).
+              custom_formats:
+                - trash_ids:
+                    - dcf3ec6938fa32445f590a4da84256cd  # DTS-HD MA
+                    - 2f22d89048b01681dde8afe203bf2e95  # DTS:X
+                    - 8e109e50e0a0b83a5098b056e13bf6db  # DTS-HD HRA
+                    - 1c1a4c5e823891c75bc50380a6866f73  # DTS (core)
+                    - f9f847ac70a0af62ea4a08280b859636  # DTS-ES
+                  assign_scores_to:
+                    - name: Remux + WEB 2160p
+                      score: 0
+          
+          sonarr:
+            sonarr-4k:
+              base_url: !env_var SONARR_4K_URL
+              api_key: !env_var SONARR_4K_APIKEY
+          
+              quality_definition:
+                type: series
+          
+              quality_profiles:
+                - trash_id: d1498e7d189fbe6c7110ceaabb7473e6  # WEB-2160p
+                  reset_unmatched_scores:
+                    enabled: true
+          
+              custom_format_groups:
+                add:
+                  - trash_id: e3f37512790f00d0e89e54fe5e790d1c  # [Optional] Golden Rule UHD
+                    select:
+                      - 9b64dff695c2115facf1b6ea59c9bd07  # x265 (no HDR/DV)
+                  - trash_id: 85fae4a2294965b75710ef2989c850eb  # [Streaming Services] HD/UHD boost
+                  - trash_id: 59c3af66780d08332fdc64e68297098f  # [Unwanted] Unwanted Formats
+                    select:
+                      - 15a05bc7c1a36e2b57fd628f8977e2fc  # AV1
+                      - 32b367365729d530ca1c124a0b180c64  # Bad Dual Groups
+                      - 85c61753df5da1fb2aab6f2a47426b09  # BR-DISK
+                      - 6f808933a71bd9666531610cb8c059cc  # BR-DISK (BTN)
+                      - fbcb31d8dabd2a319072b84fc0b7249c  # Extras
+                      - 9c11cd3f07101cdba90a2d81cf0e56b4  # LQ
+                      - e2315f990da2e2cbfc9fa5b7a6fcfe48  # LQ (Release Title)
+                      - 23297a736ca77c0fc8e70f8edd7ee56c  # Upscaled
+          
+              media_naming:
+                series: default
+                season: default
+                episodes:
+                  rename: true
+                  standard: default
+                  daily: default
+                  anime: default
+
+    - name: Write .env (4K instance URLs + keys from ansible-vault)
+      ansible.builtin.copy:
+        dest: "{{ app_dir }}/.env"
+        owner: tommy
+        group: tommy
+        mode: "0600"
+        content: |
+          RADARR_4K_URL=http://192.168.99.183:7879
+          RADARR_4K_APIKEY={{ vault_radarr4k_apikey }}
+          SONARR_4K_URL=http://192.168.99.183:8990
+          SONARR_4K_APIKEY={{ vault_sonarr4k_apikey }}
+      no_log: true
+
+    - name: Install recyclarr-4k sync service (docker-run oneshot, NOT container cron)
+      ansible.builtin.copy:
+        dest: /etc/systemd/system/recyclarr-4k-sync.service
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          [Unit]
+          Description=Recyclarr 4K sync (radarr-4k + sonarr-4k)
+          After=docker.service
+          Requires=docker.service
+
+          [Service]
+          Type=oneshot
+          ExecStart=/usr/bin/docker run --rm --env-file {{ app_dir }}/.env -v {{ app_dir }}/config:/config ghcr.io/recyclarr/recyclarr:8 sync
+      notify: reload systemd
+
+    - name: Install recyclarr-4k sync timer (daily, VPN-monitor pattern)
+      ansible.builtin.copy:
+        dest: /etc/systemd/system/recyclarr-4k-sync.timer
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          [Unit]
+          Description=Daily Recyclarr 4K sync
+
+          [Timer]
+          OnCalendar=*-*-* 04:00:00
+          Persistent=true
+
+          [Install]
+          WantedBy=timers.target
+      notify: reload systemd
+
+    - name: Enable recyclarr-4k timer
+      ansible.builtin.systemd_service:
+        name: recyclarr-4k-sync.timer
+        enabled: true
+        state: started
@@ -0,0 +1,27 @@
+# File-provider routers for the 4K arr UIs — merged into node01
+# /home/tommy/traefik/dynamic_conf.yml (live) under http.routers and http.services.
+# NOTE: watcher was flaky on restart-era; `docker restart traefik` after edits to be safe.
+
+# --- http.routers: ---
+    radarr-4k:
+      rule: "Host(`radarr-4k.goattw.net`)"
+      entryPoints: ["websecure"]
+      tls: { certResolver: letsencrypt }
+      service: radarr-4k-service
+      middlewares: ["secure-headers@file", "authelia-auth@file", "crowdsec-bouncer@file"]
+    sonarr-4k:
+      rule: "Host(`sonarr-4k.goattw.net`)"
+      entryPoints: ["websecure"]
+      tls: { certResolver: letsencrypt }
+      service: sonarr-4k-service
+      middlewares: ["secure-headers@file", "authelia-auth@file", "crowdsec-bouncer@file"]
+
+# --- http.services: ---
+    radarr-4k-service:
+      loadBalancer:
+        servers: [{ url: "http://192.168.99.183:7879" }]
+        serversTransport: default-transport
+    sonarr-4k-service:
+      loadBalancer:
+        servers: [{ url: "http://192.168.99.183:8990" }]
+        serversTransport: default-transport
@@ -0,0 +1,10 @@
+# PIA WireGuard peer regenerator for gluetun custom mode.
+# Uses PIA's official CA-pinned curl flow (token + addKey) instead of the Go
+# pia-wg-config binary, which modern-Go TLS breaks on PIA's legacy-CommonName
+# (no-SAN) server certs. See regen-entrypoint.sh / memory pia-wg-regen-broken-go-legacy-cn.
+FROM alpine:3.20
+RUN apk add --no-cache bash curl jq wireguard-tools ca-certificates
+COPY ca.rsa.4096.crt /etc/pia/ca.rsa.4096.crt
+COPY regen-entrypoint.sh /usr/local/bin/regen-entrypoint.sh
+RUN chmod +x /usr/local/bin/regen-entrypoint.sh
+ENTRYPOINT ["/usr/local/bin/regen-entrypoint.sh"]
@@ -0,0 +1,49 @@
+# gluetun PIA WireGuard peer regenerator
+
+Generates `wg0.conf` for gluetun **custom** mode (gluetun has no native PIA
+WireGuard). sabnzbd + transmission ride gluetun's netns. Live stack runs from
+`~/downloads/` on media-server (.183) — that compose is part of the docker stack,
+not this repo; the regenerator source is captured here.
+
+## Why this exists (2026-06-30 rewrite)
+The previous approach built `kylegrantlucas/pia-wg-config` (Go 1.23), which
+TLS-rejects PIA's legacy-CommonName (no-SAN) server certs:
+`x509: certificate relies on legacy Common Name field`. Compounding it, the old
+region `us_chicago` was migrated onto Kape/ExpressVPN-CA infra whose certs don't
+chain to PIA's `ca.rsa.4096.crt`.
+
+**Fix:** `regen-entrypoint.sh` now uses PIA's official **CA-pinned curl flow**
+(`authv3/generateToken` + `addKey`, `--connect-to <CN>::<ip>: --cacert`). OpenSSL
+honors CN-fallback on no-SAN certs where Go won't. It also does **CA-aware region
+auto-pick**: tries `$PIA_REGION` then a US-centric fallback list; the CA-pinned
+token step is the gate, so ExpressVPN-migrated regions auto-skip (self-heals future
+Kape migrations).
+
+## Files
+- `regen-entrypoint.sh` — the regenerator (container ENTRYPOINT)
+- `Dockerfile` — alpine + curl/jq/wireguard-tools; bakes PIA CA at `/etc/pia/ca.rsa.4096.crt`
+- `ca.rsa.4096.crt` — PIA's public CA (from pia-foss/manual-connections)
+- `regen-and-up.sh` — **recovery command**: regen peer → recreate gluetun → reconnect sab/transmission (run this on a `VpnTunnelDown` alert)
+- `regen-wg.sh` — regen config only (no gluetun restart)
+
+## Required docker-compose wiring (in the ~/downloads stack)
+```yaml
+  pia-wg-regen:
+    build: ./pia-wg
+    image: pia-wg-regen:v1.1.1
+    container_name: pia-wg-regen
+    environment:
+      - PIA_REGION=us_denver        # was us_chicago (moved to ExpressVPN CA)
+    volumes:
+      - ./.env:/pia/.env:ro         # OPENVPN_USER / OPENVPN_PASSWORD (gitignored — never commit)
+      - ./pia-wg/config:/config     # wg0.conf output (contains the private key — never commit)
+    restart: "no"
+```
+
+## Recovery
+```
+cd ~/downloads/pia-wg && ./regen-and-up.sh
+```
+Verify: `docker exec gluetun wget -qO- http://localhost:8000/v1/publicip/ip`
+should show a PIA IP (not the home WAN). NB: gluetun `/v1/openvpn/status` reads
+`stopped` under WireGuard — normal; use `/v1/publicip/ip`.
@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+set -euo pipefail
+# Self-heal recovery for a stale/dead PIA WireGuard tunnel — the one-line action to run
+# when the VpnTunnelDown alert fires. Registers a fresh peer, then recreates gluetun so
+# it loads the new wg0.conf, and reconnects SAB/Transmission through the restored netns.
+cd "$(dirname "$0")/.."   # -> ~/downloads
+echo "[regen-and-up] 1/3 registering a fresh PIA WG peer..."
+docker compose run --rm pia-wg-regen
+echo "[regen-and-up] 2/3 recreating gluetun with the new wg0.conf..."
+docker compose up -d --force-recreate --no-deps gluetun
+echo "[regen-and-up] 3/3 reconnecting dependents through the restored netns..."
+docker compose up -d --force-recreate sabnzbd transmission
+echo "[regen-and-up] done."
@@ -0,0 +1,116 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Regenerate a FRESH PIA WireGuard peer and write wg0.conf for gluetun (custom mode),
+# using PIA's official CA-pinned curl flow (token via authv3/generateToken + addKey).
+#
+# WHY NOT pia-wg-config: the pinned Go binary (Go 1.23) TLS-rejects PIA's legacy
+# CommonName (no-SAN) server certs. OpenSSL/curl still honor CN fallback, so the same
+# endpoints validate fine here. See memory: pia-wg-regen-broken-go-legacy-cn.
+#
+# CA-AWARE region auto-pick: PIA has migrated some regions (e.g. us_chicago) onto
+# Kape/ExpressVPN-CA infra whose certs do NOT chain to PIA's ca.rsa.4096.crt. Because
+# every request below is --cacert-pinned to the PIA CA, such a region fails the token
+# step (curl exit 60) and is transparently skipped. We try $PIA_REGION first, then a
+# US-centric PIA-native fallback list; the first region that validates + registers wins.
+#
+# Creds read from a read-only-mounted .env so the password never lands in this
+# container's Config.Env, the compose file, or any log.
+
+ENV_FILE="${ENV_FILE:-/pia/.env}"
+OUT="${OUT:-/config/wg0.conf}"
+CA="${CA:-/etc/pia/ca.rsa.4096.crt}"
+SERVERLIST="${SERVERLIST:-https://serverlist.piaservers.net/vpninfo/servers/v6}"
+PREFERRED="${PIA_REGION:-us_denver}"
+# fallback order if the preferred region isn't PIA-CA / is unreachable
+FALLBACKS="us_denver us_atlanta us_california us_east us_seattle ca_toronto"
+
+log(){ echo "[regen] $*" >&2; }
+
+[ -f "$CA" ] || { echo "ERROR: PIA CA not found at $CA" >&2; exit 1; }
+user="$(grep -E '^OPENVPN_USER='     "$ENV_FILE" | head -1 | cut -d= -f2-)"
+pass="$(grep -E '^OPENVPN_PASSWORD=' "$ENV_FILE" | head -1 | cut -d= -f2-)"
+if [ -z "${user:-}" ] || [ -z "${pass:-}" ]; then
+  echo "ERROR: OPENVPN_USER/OPENVPN_PASSWORD not found in $ENV_FILE" >&2; exit 1
+fi
+
+log "fetching PIA server list"
+servers="$(curl -fsS --max-time 20 "$SERVERLIST" | sed -n '1p')"
+[ -n "$servers" ] || { echo "ERROR: empty server list" >&2; exit 1; }
+
+# build ordered, de-duplicated candidate list: preferred first, then fallbacks
+candidates=""
+for r in "$PREFERRED" $FALLBACKS; do
+  case " $candidates " in *" $r "*) : ;; *) candidates="$candidates $r" ;; esac
+done
+
+# try_region <region> -> writes $OUT and returns 0 on success; non-zero to try next
+try_region() {
+  local region="$1" WG_IP WG_CN META_IP META_CN
+  read -r WG_IP WG_CN META_IP META_CN < <(
+    printf '%s' "$servers" | jq -r --arg r "$region" '
+      .regions[] | select(.id==$r) as $reg
+      | ($reg.servers.wg[0]) as $wg | ($reg.servers.meta[0]) as $meta
+      | "\($wg.ip) \($wg.cn) \($meta.ip) \($meta.cn)"' 2>/dev/null)
+  [ -n "${WG_IP:-}" ] && [ -n "${META_CN:-}" ] || { log "  $region: no wg/meta server, skip"; return 1; }
+
+  # token: CA-pinned generateToken on the META server (this is also the PIA-CA gate)
+  local tok_json rc=0 token tok_status
+  tok_json="$(curl -fsS --max-time 20 -u "$user:$pass" \
+    --connect-to "${META_CN}::${META_IP}:" --cacert "$CA" \
+    "https://${META_CN}/authv3/generateToken" 2>/dev/null)" || rc=$?
+  if [ "$rc" -ne 0 ]; then
+    [ "$rc" -eq 60 ] && log "  $region: cert not PIA-CA (curl 60), skip" || log "  $region: token curl failed (rc=$rc), skip"
+    return 1
+  fi
+  token="$(printf '%s' "$tok_json" | jq -r '.token // empty')"
+  tok_status="$(printf '%s' "$tok_json" | jq -r '.status // empty')"
+  [ -n "$token" ] || { log "  $region: no token (status=$tok_status), skip"; return 1; }
+
+  # fresh keypair; private key never leaves this container
+  local priv pub add_json add_status server_key peer_ip server_ip server_port dns
+  priv="$(wg genkey)"; pub="$(printf '%s' "$priv" | wg pubkey)"
+
+  add_json="$(curl -fsS --max-time 20 -G \
+    --connect-to "${WG_CN}::${WG_IP}:" --cacert "$CA" \
+    --data-urlencode "pt=${token}" --data-urlencode "pubkey=${pub}" \
+    "https://${WG_CN}:1337/addKey" 2>/dev/null)" || { log "  $region: addKey curl failed, skip"; return 1; }
+  add_status="$(printf '%s' "$add_json" | jq -r '.status // empty')"
+  [ "$add_status" = "OK" ] || { log "  $region: addKey status=$add_status, skip"; return 1; }
+
+  server_key="$(printf '%s' "$add_json" | jq -r '.server_key')"
+  peer_ip="$(printf '%s'    "$add_json" | jq -r '.peer_ip')"
+  server_ip="$(printf '%s'  "$add_json" | jq -r '.server_ip // empty')"; server_ip="${server_ip:-$WG_IP}"
+  server_port="$(printf '%s' "$add_json" | jq -r '.server_port // 1337')"
+  dns="$(printf '%s'        "$add_json" | jq -r '.dns_servers[0] // empty')"
+  for v in server_key peer_ip dns; do
+    [ -n "${!v}" ] && [ "${!v}" != "null" ] || { log "  $region: addKey missing $v, skip"; return 1; }
+  done
+
+  local TMP; TMP="$(mktemp "${OUT}.XXXXXX")"
+  cat > "$TMP" <<EOF
+[Interface]
+PrivateKey = ${priv}
+Address = ${peer_ip}
+DNS = ${dns}
+[Peer]
+PublicKey = ${server_key}
+AllowedIPs = 0.0.0.0/0
+Endpoint = ${server_ip}:${server_port}
+PersistentKeepalive = 25
+EOF
+  local field
+  for field in PrivateKey PublicKey Endpoint Address; do
+    grep -qE "^${field}" "$TMP" || { echo "ERROR: generated wg0.conf missing '$field'" >&2; rm -f "$TMP"; return 1; }
+  done
+  chmod 600 "$TMP"; mv "$TMP" "$OUT"
+  log "regen OK: region=$region endpoint=${server_ip}:${server_port} address=${peer_ip} (PIA-CA verified)"
+  return 0
+}
+
+for region in $candidates; do
+  log "trying region: $region"
+  if try_region "$region"; then exit 0; fi
+done
+echo "ERROR: no PIA-CA region succeeded (tried:$candidates)" >&2
+exit 1
@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+# Register a FRESH PIA WireGuard peer and rewrite ./pia-wg/config/wg0.conf by running
+# the pinned pia-wg-regen oneshot container. Safe to re-run (always a brand-new peer).
+# Note: this only regenerates the config; gluetun must be (re)started to load it
+# (use regen-and-up.sh for the full recovery, or `docker compose up -d`).
+cd "$(dirname "$0")/.."   # -> ~/downloads (compose project dir)
+exec docker compose run --rm pia-wg-regen
@@ -3,74 +3,158 @@
        icon: plex.png
        href: https://plex.goattw.net
        description: Home Media Streaming Server
+        siteMonitor: http://192.168.99.183:32400
        widget:
          type: plex
          url: http://192.168.99.183:32400
-          key: g4i7y7cTWsS4EygL2vzN
+          key: {{HOMEPAGE_VAR_PLEX_KEY}}

    - Overseerr:
        icon: overseerr.png
        href: https://request.goattw.net
        description: Request Management & Discovery
+        siteMonitor: http://overseerr:5055
        widget:
          type: overseerr
          url: http://overseerr:5055
-          key: MTc3MDQwOTY5OTQ4OTMxNjk3ODM1LWNlMWQtNGRjNC1iOTMyLWZjYzE3YmYwZWQxYw==
+          key: {{HOMEPAGE_VAR_OVERSEERR_KEY}}

    - Tautulli:
        icon: tautulli.png
        href: https://stats.goattw.net
        description: Plex Monitoring & Analytics
+        siteMonitor: http://tautulli:8181
        widget:
          type: tautulli
          url: http://tautulli:8181
-          key: dwsULxbMHNHPOVXh0kvxK7ixATtljfje
+          key: {{HOMEPAGE_VAR_TAUTULLI_KEY}}
+
+    - Navidrome:
+        icon: navidrome.png
+        href: https://music.goattw.net
+        description: Self-hosted Music Streaming
+        siteMonitor: http://192.168.99.183:4533
+        widget:
+          type: navidrome
+          url: http://192.168.99.183:4533
+          user: tommy
+          token: 6290d0465252b6ae006aaa3bdfe090bc
+          salt: qqoees

 - Media Automation:
    - Sonarr:
        icon: sonarr.png
        href: https://sonarr.goattw.net
        description: TV Series Tracking & Organization
+        siteMonitor: http://sonarr:8989
        widget:
          type: sonarr
          url: http://sonarr:8989
-          key: 76d8732935b64c849d62042256e83487
+          key: {{HOMEPAGE_VAR_SONARR_KEY}}

    - Radarr:
        icon: radarr.png
        href: https://radarr.goattw.net
        description: Movie Collection & Tracking
+        siteMonitor: http://radarr:7878
        widget:
          type: radarr
          url: http://radarr:7878
-          key: 9cf656b907414b36ae898857e52f5903
+          key: {{HOMEPAGE_VAR_RADARR_KEY}}
+    - Radarr 4K:
+        icon: radarr.png
+        href: https://radarr-4k.goattw.net
+        description: 4K Movies (quality-first)
+        siteMonitor: https://radarr-4k.goattw.net
+        widget:
+          type: radarr
+          url: http://radarr-4k:7878
+          key: {{HOMEPAGE_VAR_RADARR4K_KEY}}
+
+    - Sonarr 4K:
+        icon: sonarr.png
+        href: https://sonarr-4k.goattw.net
+        description: 4K TV (quality-first)
+        siteMonitor: https://sonarr-4k.goattw.net
+        widget:
+          type: sonarr
+          url: http://sonarr-4k:8989
+          key: {{HOMEPAGE_VAR_SONARR4K_KEY}}

    - Prowlarr:
        icon: prowlarr.png
        href: https://prowlarr.goattw.net
        description: Indexer & Proxy Management
+        siteMonitor: http://prowlarr:9696
        widget:
          type: prowlarr
          url: http://prowlarr:9696
-          key: e21828a3bc1d4eba9d98807dcae52744
+          key: {{HOMEPAGE_VAR_PROWLARR_KEY}}

    - Sabnzbd:
        icon: sabnzbd.png
        href: https://sabnzbd.goattw.net
        description: Usenet Binary Downloader
+        siteMonitor: http://192.168.99.183:8081
        widget:
          type: sabnzbd
-          url: http://gluetun:8080
-          key: ba53241bedd24aae95ae7da76c407525
+          url: http://192.168.99.183:8081
+          key: {{HOMEPAGE_VAR_SABNZBD_KEY}}

    - Transmission:
        icon: transmission.png
        href: https://transmission.goattw.net
        description: BitTorrent Client (Gluetun VPN)
+        siteMonitor: http://gluetun:9091
        widget:
          type: transmission
          url: http://gluetun:9091

+    - Tdarr:
+        icon: tdarr.png
+        href: https://tdarr.goattw.net
+        description: Distributed Video Transcoding
+        siteMonitor: http://192.168.99.183:8265
+        widget:
+          type: tdarr
+          url: http://192.168.99.183:8265
+
+    - Bazarr:
+        icon: bazarr.png
+        href: https://bazarr.goattw.net
+        description: Subtitle Management for Movies & TV
+        siteMonitor: http://192.168.99.183:6767
+        widget:
+          type: bazarr
+          url: http://192.168.99.183:6767
+          key: {{HOMEPAGE_VAR_BAZARR_KEY}}
+
+    - Lidarr:
+        icon: lidarr.png
+        href: https://lidarr.goattw.net
+        description: Music Collection & Download Manager
+        siteMonitor: http://192.168.99.183:8686
+        widget:
+          type: lidarr
+          url: http://192.168.99.183:8686
+          key: {{HOMEPAGE_VAR_LIDARR_KEY}}
+
+    - Kometa:
+        icon: kometa.png
+        href: https://github.com/Kometa-Team/Kometa
+        description: Plex Metadata & Collection Manager
+    - Plex-Utills:
+        icon: plex.png
+        href: https://plex-utills.goattw.net
+        description: Plex Poster Badges & Library Enhancement
+        siteMonitor: https://plex-utills.goattw.net
+
+    - Smokeping:
+        icon: smokeping.png
+        href: https://smokeping.goattw.net
+        description: Network Latency & Packet Loss Monitoring
+        siteMonitor: https://smokeping.goattw.net
+
 - Infrastructure:
    - Proxmox Beast:
        icon: proxmox.png
@@ -80,7 +164,7 @@
          type: proxmox
          url: https://192.168.99.200:8006
          username: "root@pam!homepage"
-          password: "4c9dc58e-ed76-42cc-8629-5ccc5ba7d7a9"
+          password: {{HOMEPAGE_VAR_PROXMOX_BEAST_PASSWORD}}
          node: beast
          validate_certs: false

@@ -91,8 +175,8 @@
        widget:
          type: proxmoxbackupserver
          url: https://192.168.99.153:8007
-          username: "root@pam!homepage"
-          password: "750d3b49-faa6-4d5b-8098-e8cfa857e227"
+          username: "homepage@pbs!widget"
+          password: {{HOMEPAGE_VAR_PROXMOX_BACKUP_SERVER_PASSWORD}}
          verify_ssl: false
          datastore: Synology-Remote

@@ -100,16 +184,18 @@
        icon: truenas.png
        href: http://192.168.99.29
        description: Network Attached Storage (ZFS)
+        siteMonitor: http://192.168.99.29
        widget:
          type: truenas
          url: http://192.168.99.29
-          key: 1-GiDoQVCfxZlV6tebAnrAmNNs69LBAKYkZkr9pR5HfgCuGQMe6jkBXpXQjo0qEq37
+          key: {{HOMEPAGE_VAR_TRUENAS_KEY}}
          enablePools: true

    - Traefik:
        icon: traefik.png
        href: https://traefik.goattw.net
        description: Edge Router & Reverse Proxy
+        siteMonitor: http://192.168.99.186:8080
        widget:
          type: traefik
          url: http://192.168.99.186:8080
@@ -118,16 +204,29 @@
        icon: technitium.png
        href: http://192.168.99.184:5380
        description: DNS Resolver & Ad Blocking
+        siteMonitor: http://192.168.99.184:5380
        widget:
          type: technitium
          url: http://192.168.99.184:5380
-          key: f007daa4ec28714f864c9cebf3a731d93bf9cc8f07e3d499ad62a79c15ed6d37
+          key: {{HOMEPAGE_VAR_TECHNITIUM_DNS_KEY}}
+
+    - NetBox:
+        icon: netbox.png
+        href: https://netbox.goattw.net
+        description: IP & Network Documentation
+        siteMonitor: https://netbox.goattw.net
+    - Filebrowser:
+        icon: filebrowser.png
+        href: https://files.goattw.net
+        description: Web File Manager
+        siteMonitor: https://files.goattw.net

 - Monitoring:
    - Uptime Kuma:
        icon: uptime-kuma.png
        href: https://uptime.goattw.net
        description: Service Health & Uptime
+        siteMonitor: http://uptime-kuma:3001
        widget:
          type: uptimekuma
          url: http://uptime-kuma:3001
@@ -137,110 +236,250 @@
        icon: beszel.png
        href: https://beszel.goattw.net
        description: Lightweight Resource Statistics
+        siteMonitor: http://beszel-hub:8090
        widget:
          type: beszel
          url: http://beszel-hub:8090
          username: twalker1971@gmail.com
-          password: fap6GTM*fuq@hzc1btx
+          password: {{HOMEPAGE_VAR_BESZEL_PASSWORD}}
          version: 2

    - Grafana:
        icon: grafana.png
        href: https://grafana.goattw.net
        description: Metrics & Dashboards
+        siteMonitor: https://grafana.goattw.net
+
+    - Loki:
+        icon: loki.png
+        href: https://grafana.goattw.net/explore
+        description: Log Aggregation & Search
+        siteMonitor: http://loki:3100/ready

    - Dozzle:
        icon: dozzle.png
        href: https://dozzle.goattw.net
        description: Real-time Docker Log Viewer
+        siteMonitor: https://dozzle.goattw.net

    - Scrutiny:
        icon: scrutiny.png
        href: https://scrutiny.goattw.net/web/dashboard
        description: S.M.A.R.T Drive Health Monitoring
+        siteMonitor: http://192.168.99.186:8083
        widget:
          type: scrutiny
          url: http://192.168.99.186:8083
+
+    - Wazuh:
+        icon: wazuh.png
+        href: https://wazuh.goattw.net
+        description: SIEM & Security Monitoring
+        siteMonitor: https://wazuh.goattw.net
+
+    - CrowdSec:
+        icon: crowdsec.png
+        href: https://app.crowdsec.net
+        description: Threat Detection & IP Blocking
+        siteMonitor: http://192.168.99.186:8081/health
+        widget:
+          type: crowdsec
+          url: http://192.168.99.186:8081
+          username: homepage-widget
+          password: {{HOMEPAGE_VAR_CROWDSEC_PASSWORD}}
+          limit24h: true
+
    - Frigate:
        icon: frigate.png
        href: http://192.168.99.100:5000
        description: NVR & Object Detection
+        siteMonitor: http://192.168.99.100:5000
        widget:
          type: frigate
          url: http://192.168.99.100:5000

+    - Healthchecks:
+        href: https://healthchecks.goattw.net
+        description: Cron job and task monitoring
+        siteMonitor: https://healthchecks.goattw.net
+        icon: healthchecks.png
+    - Speedtest Tracker:
+        icon: speedtest-tracker.png
+        href: https://speedtest.goattw.net
+        description: Internet Speed Tests
+        siteMonitor: https://speedtest.goattw.net
+
+    - WatchYourLAN:
+        icon: watchyourlan.png
+        href: https://watchyourlan.goattw.net
+        description: Network Device Scanner
+        siteMonitor: https://watchyourlan.goattw.net
+
+    - Glances:
+        icon: glances.png
+        href: https://glances.goattw.net
+        description: System Monitor
+        siteMonitor: https://glances.goattw.net
+
+    - Netdata:
+        icon: netdata.png
+        href: https://netdata.goattw.net
+        description: Real-time Metrics
+        siteMonitor: https://netdata.goattw.net
+
 - Management:
    - Gitea:
        icon: gitea.png
        href: https://gitea.goattw.net
        description: Self-hosted Git Server
+        siteMonitor: http://192.168.99.186:3002
        widget:
          type: gitea
          url: http://192.168.99.186:3002
-          key: 5755160e51ca2d20f4c89982cbb5a5466c3af5e7
+          key: {{HOMEPAGE_VAR_GITEA_KEY}}
+
+    - Semaphore:
+        icon: semaphore.png
+        href: https://semaphore.goattw.net
+        description: Ansible Web UI
+        siteMonitor: https://semaphore.goattw.net
+        
+
    - Portainer:
        icon: portainer.png
        href: https://portainer.goattw.net/
        description: GUI Container Orchestration
+        siteMonitor: https://portainer.goattw.net
        widget:
          type: portainer
-          url: http://192.168.99.186:9000
-          key: ptr_klxU4jSsfFl1KES58Mw+v9gN9gCZnUoKHRKLuopgZZ0=
+          url: https://192.168.99.186:9443
          env: 3
+          key: {{HOMEPAGE_VAR_PORTAINER_KEY}}

    - Dockge:
        icon: dockge.png
        href: https://dockge.goattw.net/
        description: Docker Compose Stack Manager
+        siteMonitor: https://dockge.goattw.net

+    - IT-Tools:
+        icon: it-tools.png
+        href: https://it-tools.goattw.net
+        description: Developer Utilities & Toolbox
+        siteMonitor: https://it-tools.goattw.net
    - Home Assistant:
        icon: home-assistant.png
        href: http://homeassistant.local:8123
        description: Home Automation & Control
+        siteMonitor: http://192.168.99.100:8123
        widget:
          type: homeassistant
          url: http://192.168.99.100:8123
          key: "{{HOMEPAGE_VAR_HASS_TOKEN}}"
          fields: ["sensor.processor_use", "binary_sensor.update_available"]
-
+    - Dagu:
+        icon: sh-dagu
+        href: https://dagu.goattw.net
+        description: DAG Workflow Scheduler
+        siteMonitor: https://dagu.goattw.net
+    - n8n:
+        icon: sh-n8n
+        href: https://n8n.goattw.net
+        description: Workflow Automation Platform
+        siteMonitor: https://n8n.goattw.net
+    - Node-RED:
+        icon: node-red.png
+        href: https://nodered.goattw.net
+        description: Visual Flow Automation
+        siteMonitor: https://nodered.goattw.net
+    - Woodpecker CI:
+        icon: sh-woodpecker-ci
+        href: https://ci.goattw.net
+        description: Gitea-native CI/CD Pipelines
+        siteMonitor: https://ci.goattw.net
+    - ntfy:
+        icon: ntfy.png
+        href: https://ntfy.goattw.net
+        description: Push Notifications
+        siteMonitor: https://ntfy.goattw.net
 - Cloud & Storage:
    - Nextcloud:
        icon: nextcloud.png
        href: https://cloud.goattw.net
        description: Private File Sync & Collaboration
+        siteMonitor: https://cloud.goattw.net

    - Immich:
        icon: immich.png
        href: https://immich.goattw.net/
        description: High Performance Photo Management
+        siteMonitor: http://immich-server:2283
        widget:
          type: immich
          url: http://immich-server:2283
-          key: Wr6y9fGbJQXA5GTr6nsd1LSbAhvuy0o7xrsTAG0SxmM
+          key: {{HOMEPAGE_VAR_IMMICH_KEY}}
          version: 2

    - Paperless:
        icon: paperless-ngx.png
        href: https://paperless.goattw.net
        description: Smart Document Management
+        siteMonitor: http://paperless:8000
        widget:
          type: paperlessngx
          url: http://paperless:8000
-          key: 9bb14bf55ffd6908a4411d490395a7cea8588fa8
+          key: {{HOMEPAGE_VAR_PAPERLESS_KEY}}

    - Vaultwarden:
        icon: vaultwarden.png
        href: https://vault.goattw.net
        description: Self-hosted Password Manager
+        siteMonitor: https://vault.goattw.net

 - AI & Productivity:
    - Open WebUI:
        icon: open-webui.png
        href: https://ai.goattw.net
        description: "The Beast AI (Qwen 14B / Llama 3)"
+        siteMonitor: https://ai.goattw.net


    - Stirling PDF:
        icon: stirling-pdf.png
        href: https://pdf.goattw.net
        description: All-in-one PDF Manipulation Tool
+        siteMonitor: https://pdf.goattw.net/api/v1/info/status
+
+    - Linkding:
+        icon: linkding.png
+        href: https://linkding.goattw.net
+        description: Bookmark Manager
+        siteMonitor: https://linkding.goattw.net
+
+    - Vikunja:
+        icon: vikunja.png
+        href: https://vikunja.goattw.net
+        description: Task Management
+        siteMonitor: https://vikunja.goattw.net
+    - LubeLogger:
+        icon: lubelogger.png
+        href: https://lubelogger.goattw.net
+        description: Vehicle & small-engine maintenance
+        siteMonitor: https://lubelogger.goattw.net
+        widget:
+          type: lubelogger
+          url: http://192.168.99.186:8099
+          username: tommy
+          password: {{HOMEPAGE_VAR_LUBELOGGER_PASSWORD}}
+
+- Food:
+    - Tandoor:
+        icon: sh-tandoor-recipes
+        href: https://tandoor.goattw.net
+        description: Self-hosted Recipe Manager & Meal Planner
+        siteMonitor: https://tandoor.goattw.net
+    - Mealie:
+        icon: mealie.png
+        href: https://mealie.goattw.net
+        description: Self-hosted Recipe Manager
+        siteMonitor: https://mealie.goattw.net
@@ -0,0 +1,56 @@
+# LubeLogger — deployment source of truth
+
+Vehicle & small-engine maintenance tracker. **Host:** docker-node01 (.186).
+Postgres 18 backend (tables in the `app` schema). In-app auth ENABLED (root user).
+
+## Routing (file provider)
+Published on the host at `:8099`; Traefik on node01 routes `lubelogger.goattw.net`
+→ `http://192.168.99.186:8099` via `dynamic_conf.yml` (see `traefik-router.dynamic_conf.yml`).
+Edge chain: TLS (letsencrypt) → CrowdSec (global) → Authelia (`authelia-auth@file`,
+`two_factor` off-LAN / bypass on-LAN). LubeLogger's own auth gates the LAN/`:8099` path.
+
+## Deploy (Ansible, idempotent)
+Playbook `deploy-lubelogger.yml` (runs from `~/ansible` on ansible-control):
+```
+ansible-playbook playbooks/deploy-lubelogger.yml
+```
+Inline-compose house style: `copy:` the compose + `community.docker.docker_compose_v2`.
+DB password is sourced from ansible-vault (`vault_lubelogger_db_password`) — never inline.
+Second converge is `changed=0`.
+
+## Backup (PBS) — deployed BY the playbook
+The backup script + systemd units are installed and enabled by `deploy-lubelogger.yml`
+(inline, idempotent). node01, daily 03:30, one PBS snapshot (`rust-usb`, host/lubelogger)
+with **three archives** in a single `proxmox-backup-client` job, prune `--keep-daily 14`:
+- `lubelogger-db.pxar`   — `pg_dump` (app schema + data)
+- `lubelogger-data.pxar` — `/App/data` volume (uploads, config, cartracker.db)
+- `lubelogger-keys.pxar` — ASP.NET DataProtection keyring (the ROOT USER credential + keys)
+
+The `docker-volume-backup` sidecar can't target a PBS datastore natively, so the volumes
+ride the same pbc job as the SQL dump (one datastore, one retention). PBS token comes from
+ansible-vault (`vault_lubelogger_pbs_token`) → templated to `/etc/lubelogger-backup.env`
+(root:600, NOT in git). Restore VERIFIED: SQL round-trip into throwaway postgres:18
+(row-count parity) + data/keys restore (keyring XML + a sentinel doc, byte-identical sha256).
+The `stop-during-backup` labels remain in case a sidecar is later added for a second target.
+
+## Reminders → ntfy
+`lubelogger-reminders-probe.{sh}` + units on ansible-control (.190) — daily 07:30.
+Reads `/api/vehicle/info` (Basic auth, creds in `/etc/lubelogger-reminders.env`,
+root:600, NOT in git), posts a digest to ntfy topic **garage** only when something is
+due/overdue. Subscribe to the `garage` topic in your ntfy app to receive it.
+
+## Diun
+App carries `diun.watch_repo=true` + semver `include_tags` so a new release tag pings
+the existing Diun→HA-webhook pipeline. DB (`postgres:18`) is watched by default (digest).
+
+## Homepage
+Tile + `lubelogger` widget under "AI & Productivity" in
+`observability/homepage/services.yaml` (live) — widget password via
+`HOMEPAGE_VAR_LUBELOGGER_PASSWORD` (observability/.env, not a git repo). See
+`homepage-tile.yaml`.
+
+## Secrets (none in this repo)
+- DB password: ansible-vault (`~/ansible`), and Postgres volume (already initialized).
+- PBS token: `/etc/lubelogger-backup.env` (reuses vaultwarden token — a dedicated
+  `lubelogger-backup` PBS token would be cleaner; needs PBS root to create).
+- Admin/basic-auth password: `/etc/lubelogger-reminders.env` + `observability/.env`.
@@ -0,0 +1,210 @@
+---
+- name: Deploy LubeLogger on docker-node01
+  hosts: docker-node01
+  become: true
+  vars:
+    app_dir: /home/tommy/lubelogger
+    app_port: 8099
+
+  handlers:
+    - name: reload systemd
+      ansible.builtin.systemd_service:
+        daemon_reload: true
+
+  tasks:
+    - name: Create lubelogger directory
+      ansible.builtin.file:
+        path: "{{ app_dir }}"
+        state: directory
+        owner: tommy
+        group: tommy
+        mode: "0755"
+
+    - name: Write docker-compose.yml
+      ansible.builtin.copy:
+        dest: "{{ app_dir }}/docker-compose.yml"
+        owner: tommy
+        group: tommy
+        mode: "0644"
+        content: |
+          # lubelogger - Vehicle & small-engine maintenance tracker
+          # Host: docker-node01 (.186), Port: {{ app_port }}
+          # Traefik: lubelogger.goattw.net (Authelia protected, @file router -> :{{ app_port }})
+          # Backend: PostgreSQL 18 (tables live in the `app` schema).
+          # In-app auth is ENABLED (root user) - the published host port is safe.
+          # Do NOT lose the lubelogger-keys volume (ASP.NET DataProtection keyring).
+
+          services:
+            lubelogger:
+              image: ghcr.io/hargata/lubelogger:v1.6.8
+              container_name: lubelogger
+              restart: unless-stopped
+              depends_on:
+                - lubelogger-db
+              ports:
+                - "{{ app_port }}:8080"
+              environment:
+                TZ: America/Chicago
+                LUBELOGGER_LOCALE_OVERRIDE: "en_US"
+                POSTGRES_CONNECTION: "Host=lubelogger-db:5432;Username=lubelogger;Password={{ vault_lubelogger_db_password }};Database=lubelogger;"
+              volumes:
+                - lubelogger-data:/App/data
+                - lubelogger-keys:/root/.aspnet/DataProtection-Keys
+              networks:
+                - lubelogger-int
+              labels:
+                - "docker-volume-backup.stop-during-backup=true"
+                # Diun watches by default (digest); watch_repo + semver filter
+                # so a NEW release tag (e.g. v1.6.9) pings the existing Diun->HA pipeline.
+                - "diun.enable=true"
+                - "diun.watch_repo=true"
+                - 'diun.include_tags=^v[0-9]+\.[0-9]+\.[0-9]+$'
+
+            lubelogger-db:
+              image: postgres:18
+              container_name: lubelogger-db
+              restart: unless-stopped
+              environment:
+                POSTGRES_USER: lubelogger
+                POSTGRES_PASSWORD: "{{ vault_lubelogger_db_password }}"
+                POSTGRES_DB: lubelogger
+                TZ: America/Chicago
+              volumes:
+                - lubelogger-db:/var/lib/postgresql
+                - /etc/localtime:/etc/localtime:ro
+              networks:
+                - lubelogger-int
+              healthcheck:
+                test: ["CMD-SHELL", "pg_isready -U lubelogger -d lubelogger"]
+                interval: 10s
+                timeout: 5s
+                retries: 5
+                start_period: 30s
+              labels:
+                - "docker-volume-backup.stop-during-backup=true"
+
+          networks:
+            lubelogger-int:
+              driver: bridge
+
+          volumes:
+            lubelogger-data:
+            lubelogger-keys:
+            lubelogger-db:
+
+    - name: Remove stale .env (secret now sourced from ansible-vault)
+      ansible.builtin.file:
+        path: "{{ app_dir }}/.env"
+        state: absent
+
+    - name: Deploy lubelogger with docker compose
+      community.docker.docker_compose_v2:
+        project_src: "{{ app_dir }}"
+        state: present
+        pull: missing
+      register: app_result
+
+    - name: Show deployment result
+      ansible.builtin.debug:
+        msg: "lubelogger is {{ 'running (changed)' if app_result.changed else 'already up (no-op)' }} on port {{ app_port }}"
+
+    # ---- Backup: pg_dump + data/keys volumes -> PBS (host/lubelogger, keep-daily 14) ----
+    # docker-volume-backup sidecar cannot target a PBS datastore natively, so the two
+    # volumes ride the same proxmox-backup-client job as the SQL dump: one snapshot,
+    # three archives, one retention. Secrets come from ansible-vault, never committed.
+
+    - name: Install PBS backup credentials (from vault)
+      ansible.builtin.copy:
+        dest: /etc/lubelogger-backup.env
+        owner: root
+        group: root
+        mode: "0600"
+        content: |
+          # Managed by deploy-lubelogger.yml. PBS push creds — NOT in git.
+          PBS_PASSWORD={{ vault_lubelogger_pbs_token }}
+          PBS_FINGERPRINT=90:6B:3E:85:C4:E7:77:F0:24:25:1D:B0:2C:F9:6D:08:AA:CE:08:CE:7B:04:4C:C7:C9:E0:CF:DD:99:44:DB:21
+          PBS_REPOSITORY='root@pam!lubelogger-backup@192.168.99.153:rust-usb'
+      no_log: true
+
+    - name: Install lubelogger backup script
+      ansible.builtin.copy:
+        dest: /usr/local/bin/lubelogger-backup.sh
+        owner: root
+        group: root
+        mode: "0700"
+        content: |
+          #!/bin/bash
+          # LubeLogger backup -> Proxmox Backup Server.
+          # One snapshot (host/lubelogger), three archives: Postgres logical dump +
+          # the data and keys volumes (uploads, config, ASP.NET DataProtection keyring).
+          # Managed by deploy-lubelogger.yml; driven by lubelogger-backup.timer.
+          set -euo pipefail
+          source /etc/lubelogger-backup.env
+          export PBS_PASSWORD PBS_FINGERPRINT
+
+          DATA_VOL=/var/lib/docker/volumes/lubelogger_lubelogger-data/_data
+          KEYS_VOL=/var/lib/docker/volumes/lubelogger_lubelogger-keys/_data
+
+          TMPDIR=$(mktemp -d)
+          trap 'rm -rf "$TMPDIR"' EXIT
+
+          # Logical DB dump (app schema + data); app stays online.
+          docker exec lubelogger-db pg_dump -U lubelogger lubelogger > "$TMPDIR/lubelogger.sql"
+
+          proxmox-backup-client backup \
+            "lubelogger-db.pxar:$TMPDIR" \
+            "lubelogger-data.pxar:$DATA_VOL" \
+            "lubelogger-keys.pxar:$KEYS_VOL" \
+            --repository "$PBS_REPOSITORY" \
+            --backup-type host \
+            --backup-id lubelogger
+
+          # Retention: token holds Datastore.Backup only (Prune deferred to a PBS
+          # group-scoped prune-job). Keep this non-fatal so nightly backups still
+          # succeed; it starts pruning automatically if Datastore.Prune is granted.
+          proxmox-backup-client prune "host/lubelogger" \
+            --repository "$PBS_REPOSITORY" \
+            --keep-daily 14 \
+            || echo "NOTE: prune skipped (Datastore.Prune not granted; retention via deferred PBS prune-job)"
+      notify: reload systemd
+
+    - name: Install backup systemd service unit
+      ansible.builtin.copy:
+        dest: /etc/systemd/system/lubelogger-backup.service
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          [Unit]
+          Description=LubeLogger backup (pg_dump + volumes) -> PBS
+          After=docker.service
+          Requires=docker.service
+
+          [Service]
+          Type=oneshot
+          ExecStart=/usr/local/bin/lubelogger-backup.sh
+      notify: reload systemd
+
+    - name: Install backup systemd timer unit
+      ansible.builtin.copy:
+        dest: /etc/systemd/system/lubelogger-backup.timer
+        owner: root
+        group: root
+        mode: "0644"
+        content: |
+          [Unit]
+          Description=Daily LubeLogger backup to PBS
+
+          [Timer]
+          OnCalendar=*-*-* 03:30:00
+          Persistent=true
+
+          [Install]
+          WantedBy=timers.target
+      notify: reload systemd
+
+    - name: Enable lubelogger backup timer
+      ansible.builtin.systemd_service:
+        name: lubelogger-backup.timer
+        enabled: true
+        state: started
@@ -0,0 +1,10 @@
+    - LubeLogger:
+        icon: lubelogger.png
+        href: https://lubelogger.goattw.net
+        description: Vehicle & small-engine maintenance
+        siteMonitor: https://lubelogger.goattw.net
+        widget:
+          type: lubelogger
+          url: http://192.168.99.186:8099
+          username: tommy
+          password: {{HOMEPAGE_VAR_LUBELOGGER_PASSWORD}}
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+set -euo pipefail
+# Daily LubeLogger reminder digest -> ntfy 'garage' topic.
+# Reads per-vehicle reminder urgency counts from LubeLogger's API and posts a
+# summary ONLY when something is due/overdue (PastDue + VeryUrgent + Urgent).
+# Runs on ansible-control (.190) via lubelogger-reminders.timer.
+# Shape mirrors the gluetun vpn-tunnel-probe (oneshot + timer, set -euo pipefail).
+
+source /etc/lubelogger-reminders.env   # LUBELOGGER_URL LUBELOGGER_APIKEY NTFY_URL
+
+if ! resp=$(curl -sf --max-time 20 -H "x-api-key: ${LUBELOGGER_APIKEY}" "${LUBELOGGER_URL}/api/vehicle/info"); then
+  logger -t lubelogger-reminders "ERROR: LubeLogger API query failed"
+  curl -sf --max-time 15 -H "Priority: high" -H "Tags: warning" \
+       -H "Title: LubeLogger reminder probe failed" \
+       -d "Could not query ${LUBELOGGER_URL}/api/vehicle/info" "$NTFY_URL" || true
+  exit 1
+fi
+
+total=$(printf '%s' "$resp" | jq '[.[] | (.PastDueReminderCount + .VeryUrgentReminderCount + .UrgentReminderCount)] | add // 0')
+
+if [ "${total:-0}" -gt 0 ]; then
+  body=$(printf '%s' "$resp" | jq -r '
+    .[]
+    | (.PastDueReminderCount + .VeryUrgentReminderCount + .UrgentReminderCount) as $due
+    | select($due > 0)
+    | "• \(.VehicleData.Year) \(.VehicleData.Make) \(.VehicleData.Model): "
+      + "\(.PastDueReminderCount) past-due, \(.VeryUrgentReminderCount) very-urgent, \(.UrgentReminderCount) urgent"')
+  curl -sf --max-time 15 \
+       -H "Title: Vehicle maintenance due (${total})" \
+       -H "Tags: wrench" -H "Priority: default" \
+       -d "$body" "$NTFY_URL" || true
+  logger -t lubelogger-reminders "posted ${total} due reminder(s) to ntfy garage"
+else
+  logger -t lubelogger-reminders "no due/overdue reminders"
+fi
@@ -0,0 +1,8 @@
+[Unit]
+Description=LubeLogger reminder digest -> ntfy garage
+Wants=network-online.target
+After=network-online.target
+
+[Service]
+Type=oneshot
+ExecStart=/home/tommy/ansible/scripts/lubelogger-reminders-probe.sh
@@ -0,0 +1,9 @@
+[Unit]
+Description=Daily LubeLogger reminder digest
+
+[Timer]
+OnCalendar=*-*-* 07:30:00
+Persistent=true
+
+[Install]
+WantedBy=timers.target
@@ -0,0 +1,21 @@
+# File-provider router for LubeLogger — merged into node01
+# /home/tommy/traefik/dynamic_conf.yml (live) under http.routers and http.services.
+# Traefik watch=true hot-reloads on change. (Reference copy; the live file is authoritative.)
+
+# --- under http.routers: ---
+    lubelogger:
+      rule: "Host(`lubelogger.goattw.net`)"
+      entryPoints:
+        - websecure
+      middlewares:
+        - authelia-auth@file        # crowdsec-bouncer@file applies globally at the websecure entrypoint
+      service: lubelogger-service
+      tls:
+        certResolver: letsencrypt
+
+# --- under http.services: ---
+    lubelogger-service:
+      loadBalancer:
+        servers:
+          - url: "http://192.168.99.186:8099"
+        serversTransport: default-transport
@@ -0,0 +1,13 @@
+# Diun on compute6
+
+Closes the compute6 image-update-visibility gap found in the 2026-06-30 drift audit
+(node01/02/media-server already ran Diun; compute6 did not). Mirrors the node01/02
+crazymax/diun pattern; watches compute6's ~10 local containers via the docker socket,
+notifies the same HA webhook.
+
+Deploys to compute6 at `~/diun/docker-compose.yml`.
+
+**compute6 quirk:** `security_opt: apparmor=unconfined` is REQUIRED here — compute6's
+default AppArmor profile blocks the container's docker-socket access (every compute6
+container runs unconfined for this reason). Without it Diun logs "permission denied"
+on the socket and finds 0 images.
@@ -0,0 +1,33 @@
+services:
+  # ============================================================
+  # DIUN - Docker Image Update Notifier (compute6)
+  # ============================================================
+  # Mirrors the node01/02 pattern (crazymax/diun) to close the compute6
+  # image-update visibility gap. Watches compute6's local containers via the
+  # docker socket; notifies the same HA webhook -> mobile_app_tomcell.
+  # No traefik_public here (doesn't exist on compute6) -> default bridge;
+  # Diun needs only the socket + outbound to registries and .100:8123.
+  # ============================================================
+  diun:
+    image: crazymax/diun:latest
+    container_name: diun
+    restart: unless-stopped
+    mem_limit: 256m
+    cpus: 0.5
+    environment:
+      - TZ=America/Chicago
+      - LOG_LEVEL=info
+      - LOG_JSON=false
+      - DIUN_WATCH_WORKERS=20
+      - DIUN_WATCH_SCHEDULE=0 8 * * *
+      - DIUN_WATCH_FIRSTCHECKNOTIF=false
+      - DIUN_PROVIDERS_DOCKER=true
+      - DIUN_PROVIDERS_DOCKER_WATCHBYDEFAULT=true
+      - DIUN_NOTIF_WEBHOOK_ENDPOINT=http://192.168.99.100:8123/api/webhook/diun_image_update
+      - DIUN_NOTIF_WEBHOOK_METHOD=POST
+      - DIUN_NOTIF_WEBHOOK_HEADERS_CONTENT-TYPE=application/json
+    security_opt:
+      - apparmor=unconfined
+    volumes:
+      - ./data:/data
+      - /var/run/docker.sock:/var/run/docker.sock:ro
@@ -0,0 +1,36 @@
+# Off-host Prometheus watchdog (dead-man's-switch)
+
+Independent-hardware watchdog that pages if the media-server (.183) Prometheus
+becomes unreachable — the one failure the same-host Uptime-Kuma monitor cannot
+detect (media-server itself going dark).
+
+**Runs on:** Pi / raspberrypi (192.168.99.227), as root via systemd timer.
+
+## Behaviour
+- Timer fires every 60s (`OnBootSec=2min`, `OnUnitActiveSec=60s`).
+- Probes `http://192.168.99.183:9090/-/healthy` for the keyword `Healthy`.
+- After 3 consecutive failures (~3 min) publishes a **priority-5** "Prometheus DOWN"
+  page to ntfy topic `prometheus-critical` (compute6, `http://192.168.99.198:8095`);
+  re-pages every 30 checks while still down.
+- On recovery sends a **priority-4** "Prometheus recovered".
+- State: `/var/lib/prom-watchdog/state` ("FAILS ALERTED").
+
+Reaches the phone because the ntfy server has `upstream-base-url: https://ntfy.sh`
+(FCM/APNS relay).
+
+## Install (on the Pi)
+```
+sudo install -m 755 prometheus-watchdog.sh          /usr/local/bin/prometheus-watchdog.sh
+cp prom-watchdog.env.example prom-watchdog.env      # set NTFY_TOKEN only if ntfy auth is enabled
+sudo install -m 600 prom-watchdog.env               /etc/prom-watchdog.env
+sudo install -m 644 prometheus-watchdog.service     /etc/systemd/system/prometheus-watchdog.service
+sudo install -m 644 prometheus-watchdog.timer       /etc/systemd/system/prometheus-watchdog.timer
+sudo systemctl daemon-reload
+sudo systemctl enable --now prometheus-watchdog.timer
+```
+
+## Notes
+- `prom-watchdog.env` has an **empty** `NTFY_TOKEN` — the `prometheus-critical`
+  topic is unauthenticated on the LAN ntfy. If auth is ever enabled, put the real
+  token in the deployed `/etc/prom-watchdog.env` (mode 600) and do **not** commit it.
+- Deployed + live-outage tested 2026-06-30 (DOWN + recovered pages confirmed).
@@ -0,0 +1,5 @@
+# Copy to /etc/prom-watchdog.env (mode 600) on the Pi.
+# NTFY_TOKEN stays empty while the prometheus-critical topic is unauthenticated;
+# if ntfy auth is ever enabled, set it in the DEPLOYED file only (never commit — *.env is gitignored).
+NTFY_URL=http://192.168.99.198:8095/prometheus-critical
+NTFY_TOKEN=
@@ -0,0 +1,8 @@
+[Unit]
+Description=Off-host Prometheus dead-man's-switch (probes media-server .183)
+After=network-online.target
+Wants=network-online.target
+[Service]
+Type=oneshot
+EnvironmentFile=/etc/prom-watchdog.env
+ExecStart=/usr/local/bin/prometheus-watchdog.sh
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+set -euo pipefail
+TARGET_URL="http://192.168.99.183:9090/-/healthy"
+EXPECT="Healthy"
+NTFY_URL="${NTFY_URL:?}"          # from EnvironmentFile
+NTFY_TOKEN="${NTFY_TOKEN:-}"
+THRESHOLD=3                        # consecutive fails before first page (~3 min)
+RESEND_EVERY=30                    # re-page every N checks while still down
+TIMEOUT=10
+STATE_DIR="/var/lib/prom-watchdog"; STATE="$STATE_DIR/state"
+mkdir -p "$STATE_DIR"
+read -r FAILS ALERTED 2>/dev/null < "$STATE" || { FAILS=0; ALERTED=0; }
+notify() {  # $1=priority $2=tags $3=title $4=message
+  local a=(-s -H "Priority: $1" -H "Tags: $2" -H "Title: $3" -d "$4")
+  [ -n "$NTFY_TOKEN" ] && a+=(-H "Authorization: Bearer $NTFY_TOKEN")
+  curl "${a[@]}" "$NTFY_URL" >/dev/null 2>&1 || true
+}
+if curl -fsS --max-time "$TIMEOUT" "$TARGET_URL" 2>/dev/null | grep -q "$EXPECT"; then
+  [ "$ALERTED" -eq 1 ] && notify 4 white_check_mark "Prometheus recovered" \
+    "media-server Prometheus responding again (off-host watchdog)."
+  echo "0 0" > "$STATE"; exit 0
+fi
+FAILS=$((FAILS + 1))
+if [ "$ALERTED" -eq 0 ] && [ "$FAILS" -ge "$THRESHOLD" ]; then
+  notify 5 "rotating_light,skull" "Prometheus DOWN (off-host watchdog)" \
+    "Pi cannot reach Prometheus at $TARGET_URL after $FAILS checks. media-server may be down or prometheus stopped. ALERTING IS BLIND until resolved."
+  ALERTED=1
+elif [ "$ALERTED" -eq 1 ] && [ $((FAILS % RESEND_EVERY)) -eq 0 ]; then
+  notify 5 rotating_light "Prometheus STILL down" "Unreachable ~${FAILS} min."
+fi
+echo "$FAILS $ALERTED" > "$STATE"; exit 0
@@ -0,0 +1,8 @@
+[Unit]
+Description=Run Prometheus watchdog every minute
+[Timer]
+OnBootSec=2min
+OnUnitActiveSec=60s
+AccuracySec=5s
+[Install]
+WantedBy=timers.target
@@ -1,29 +1,162 @@
 #!/bin/bash
-# Quarterly UPS battery self-test — runs via cron first Sunday of Jan/Apr/Jul/Oct
-NUT_PASS='Sparky$100'
+# Quarterly UPS battery self-test — first Sunday of Jan/Apr/Jul/Oct at 03:00
+# Writes results to Prometheus textfile collector for alerting.
+#
+# Safety: tests are SEQUENTIAL — second UPS not tested until first is confirmed
+# back on line power. Abort if first UPS doesn't return OL within timeout.
+# This prevents both UPS being on battery simultaneously, which would trigger
+# the upssched earlyshutdown FSD timer.
+
+# NUT_PASS is sourced from a root-only file kept OUT of git (see CRED_FILE below).
 NUT_USER='upsadmin'
 NUT_HOST='localhost'
+TEXTFILE_DIR='/var/lib/node_exporter/textfile_collector'
+OUTFILE="${TEXTFILE_DIR}/ups_test.prom"
+OL_TIMEOUT=120    # seconds to wait for OL restoration before aborting
+OL_BUFFER=35      # seconds to wait after OL confirmed before starting next test
+                  # (>30s earlyshutdown timer to ensure no residual timer from prior state)
+RESULT_TIMEOUT=90 # seconds to wait for ups.test.result to reach a terminal state

-logger -t ups-quarterly-test 'starting quarterly battery self-test'
+log() { logger -t ups-quarterly-test "$*"; }
+ts()  { date '+%Y-%m-%dT%H:%M:%S'; }

-logger -t ups-quarterly-test 'testing cyberpower1 (Tower)'
-upscmd -u "$NUT_USER" -p "$NUT_PASS" cyberpower1@$NUT_HOST test.battery.start.quick 2>&1 |   logger -t ups-quarterly-test
-
-sleep 120
-
-result1=$(upsc cyberpower1@$NUT_HOST ups.test.result 2>/dev/null)
-logger -t ups-quarterly-test "cyberpower1 result: $result1"
-
-logger -t ups-quarterly-test 'testing cyberpower2 (Rackmount)'
-upscmd -u "$NUT_USER" -p "$NUT_PASS" cyberpower2@$NUT_HOST test.battery.start.quick 2>&1 |   logger -t ups-quarterly-test
-
-sleep 120
-
-result2=$(upsc cyberpower2@$NUT_HOST ups.test.result 2>/dev/null)
-logger -t ups-quarterly-test "cyberpower2 result: $result2"
-
-if [[ "$result1" == 'Done and passed' && "$result2" == 'Done and passed' ]]; then
-    logger -t ups-quarterly-test 'PASS: both UPS batteries healthy'
+# Load NUT credentials from a root-only file (chmod 600), kept OUT of git so the
+# password never lands in a repo. Created by ups-fix.sh.
+CRED_FILE='/etc/ups-quarterly-test.env'
+if [[ -r "$CRED_FILE" ]]; then
+    # shellcheck source=/dev/null
+    source "$CRED_FILE"
 else
-    logger -t ups-quarterly-test 'FAIL: one or more UPS batteries need attention'
+    log "$(ts) ERROR: missing ${CRED_FILE} (defines NUT_PASS) — aborting"
+    exit 1
+fi
+
+# FIX: cron cannot express "first Sunday" (setting both DOM and DOW makes cron
+# OR them). Cron now runs every Sunday of the quarter months; this guard limits
+# execution to the FIRST Sunday (day-of-month 1-7).
+if (( $(date +%-d) > 7 )); then
+    log "$(ts) not the first Sunday of the month (day $(date +%-d)) — skipping"
+    exit 0
+fi
+
+log "$(ts) starting quarterly battery self-test"
+
+# Wait for a given UPS to return to OL after a test.
+# Returns 0 if OL confirmed within timeout, 1 if timeout exceeded.
+wait_for_online() {
+    local ups="$1"
+    local deadline=$(( $(date +%s) + OL_TIMEOUT ))
+    while (( $(date +%s) < deadline )); do
+        local status
+        status=$(upsc "${ups}@${NUT_HOST}" ups.status 2>/dev/null)
+        log "$(ts) ${ups} status: ${status}"
+        if [[ "$status" == OL* ]]; then
+            log "$(ts) ${ups} confirmed OL"
+            return 0
+        fi
+        sleep 5
+    done
+    log "$(ts) ERROR: ${ups} did not return OL within ${OL_TIMEOUT}s — ABORTING"
+    return 1
+}
+
+# Run test on one UPS and return result string.
+# Returns "Done and passed", "Done and failed", or "TIMEOUT"
+run_test() {
+    local ups="$1"
+    local label="$2"
+    local result
+
+    log "$(ts) submitting test: ${ups} (${label})"
+    upscmd -u "$NUT_USER" -p "$NUT_PASS" "${ups}@${NUT_HOST}" test.battery.start.quick 2>&1 \
+        | logger -t ups-quarterly-test
+
+    log "$(ts) ${ups} test submitted — polling for OB then OL"
+
+    # Wait for test to complete (UPS goes OB then returns OL)
+    if ! wait_for_online "$ups"; then
+        echo "TIMEOUT"
+        return
+    fi
+
+    log "$(ts) ${ups} OL confirmed — waiting ${OL_BUFFER}s buffer before reading result"
+    sleep "$OL_BUFFER"
+
+    # FIX: CyberPower updates ups.test.result slower than the OL transition, so a
+    # single read races and can catch a non-terminal value (e.g. "In progress")
+    # → false 0 despite a passing test. Poll until the result is terminal.
+    local rdeadline=$(( $(date +%s) + RESULT_TIMEOUT ))
+    while (( $(date +%s) < rdeadline )); do
+        result=$(upsc "${ups}@${NUT_HOST}" ups.test.result 2>/dev/null)
+        if [[ "$result" == *[Pp]assed* || "$result" == *[Ff]ailed* ]]; then
+            break
+        fi
+        log "$(ts) ${ups} result not terminal yet: '${result}' — waiting"
+        sleep 5
+    done
+
+    log "$(ts) ${ups} test result: ${result}"
+    echo "$result"
+}
+
+# FIX: match on "passed" substring (robust to formatting) rather than an exact
+# "Done and passed" string.
+to_gauge() {
+    [[ "$1" == *[Pp]assed* ]] && echo 1 || echo 0
+}
+
+ntfy_alert() {
+    local msg="$1"
+    log "$(ts) ntfy: ${msg}"
+    curl -s -X POST \
+        -H "Title: UPS Test Warning" \
+        -H "Priority: high" \
+        -d "$msg" \
+        http://192.168.99.198:8081/homelab 2>/dev/null || true
+}
+
+# --- Test cyberpower1 ---
+result1=$(run_test cyberpower1 "Tower")
+
+if [[ "$result1" == "TIMEOUT" ]]; then
+    ntfy_alert "cyberpower1 quarterly test: did not return OL within ${OL_TIMEOUT}s — cyberpower2 NOT tested. Manual inspection required."
+    log "$(ts) ABORT: cyberpower1 timeout — cyberpower2 test skipped"
+    g1=0
+    result2="skipped"
+    g2=0
+else
+    log "$(ts) cyberpower1 complete. Waiting ${OL_BUFFER}s before starting cyberpower2 test"
+    sleep "$OL_BUFFER"
+
+    # --- Test cyberpower2 ---
+    result2=$(run_test cyberpower2 "Rackmount")
+
+    if [[ "$result2" == "TIMEOUT" ]]; then
+        ntfy_alert "cyberpower2 quarterly test: did not return OL within ${OL_TIMEOUT}s. Manual inspection required."
+        log "$(ts) ABORT: cyberpower2 timeout"
+        g2=0
+    else
+        g2=$(to_gauge "$result2")
+    fi
+    g1=$(to_gauge "$result1")
+fi
+
+now=$(date +%s)
+
+{
+    printf '# HELP ups_battery_test_result Last quarterly UPS battery self-test: 1=passed, 0=failed/unknown.\n'
+    printf '# TYPE ups_battery_test_result gauge\n'
+    printf 'ups_battery_test_result{ups="cyberpower1"} %s\n' "$g1"
+    printf 'ups_battery_test_result{ups="cyberpower2"} %s\n' "$g2"
+    printf '# HELP ups_battery_test_timestamp_seconds Unix timestamp of last UPS battery self-test run.\n'
+    printf '# TYPE ups_battery_test_timestamp_seconds gauge\n'
+    printf 'ups_battery_test_timestamp_seconds{ups="cyberpower1"} %s\n' "$now"
+    printf 'ups_battery_test_timestamp_seconds{ups="cyberpower2"} %s\n' "$now"
+} > "${OUTFILE}.tmp" && mv "${OUTFILE}.tmp" "$OUTFILE"
+
+if [[ "$g1" == "1" && "$g2" == "1" ]]; then
+    log "$(ts) PASS: both UPS batteries healthy"
+else
+    log "$(ts) FAIL: cyberpower1=${result1} cyberpower2=${result2}"
+    ntfy_alert "UPS quarterly test FAILED: cyberpower1=${result1} cyberpower2=${result2}"
 fi
@@ -156,6 +156,78 @@ else
    add_line "PBS ZFS pools" "MISSING: ${pbs_zfs_missing}" "pools not imported on PBS — check enclosure/cables"
 fi

+# ── 4b. Off-host app backups: LubeLogger + Vaultwarden freshness (<26h) ──────
+# Both push nightly from docker-node01 (.186) to PBS rust-usb: host/vaultwarden
+# (root cron 03:00) and host/lubelogger (systemd timer 03:30). We ask PBS for the
+# newest snapshot time of each group via the lubelogger token, which has
+# datastore-wide audit on rust-usb (covers both groups). Read-only; keyed SSH +
+# passwordless sudo on .186. Quoted heredoc → remote-side var expansion only.
+backup_ages=$(ssh -o ConnectTimeout=8 -o BatchMode=yes tommy@192.168.99.186 'sudo bash -s' 2>/dev/null <<'REMOTE' || true
+source /etc/lubelogger-backup.env 2>/dev/null
+export PBS_PASSWORD PBS_FINGERPRINT
+for grp in lubelogger vaultwarden; do
+  t=$(proxmox-backup-client snapshot list "host/$grp" --repository "$PBS_REPOSITORY" --output-format json 2>/dev/null \
+      | python3 -c 'import sys,json
+try:
+    d=json.load(sys.stdin); print(max(x["backup-time"] for x in d) if d else 0)
+except: print(0)' 2>/dev/null || echo 0)
+  echo "$grp $t"
+done
+REMOTE
+)
+
+now_epoch=$(date +%s)
+for svc in lubelogger vaultwarden; do
+    ts=$(echo "$backup_ages" | awk -v s="$svc" '$1==s{print $2}')
+    if [ -z "$ts" ] || [ "$ts" = "0" ]; then
+        add_line "Backup ${svc}→PBS" "no snapshot / query failed — check .186 backup job"
+    else
+        age_h=$(( (now_epoch - ts) / 3600 ))
+        if [ "$age_h" -lt 26 ]; then
+            add_line "Backup ${svc}→PBS" "OK" "newest snapshot ${age_h}h ago"
+        else
+            add_line "Backup ${svc}→PBS" "STALE — newest snapshot ${age_h}h ago (>26h)"
+        fi
+    fi
+done
+
+# ── 4c. Internet speed (speedtest-tracker on .183) ──────────────────────────
+# Tracker records Ookla tests every 6h (SPEEDTEST_SCHEDULE). Flag if the newest
+# completed result is stale (>14h = 2+ missed cycles → scheduler stalled) or if
+# the best of the last 4 tests is below the floor (using best-of-4, not the
+# single latest, avoids flagging a transient ISP dip). Floors are ADJUSTABLE —
+# set to catch a real regression (e.g. the old 100Mb-uplink cap ≈ 94 Mbps) or
+# sustained ISP degradation, not normal variance. DB is local + tommy-readable.
+ST_DB="/home/tommy/speedtest-tracker/data/database.sqlite"
+ST_FLOOR_DOWN=500   # Mbps
+ST_FLOOR_UP=500     # Mbps
+if [ -r "$ST_DB" ]; then
+    st_row=$(sqlite3 "$ST_DB" "
+      SELECT
+        CAST((julianday('now') - julianday(max(created_at)))*24 AS INT) || '|' ||
+        CAST((SELECT download FROM results WHERE status='completed' ORDER BY id DESC LIMIT 1)*8/1e6 AS INT) || '|' ||
+        CAST((SELECT upload   FROM results WHERE status='completed' ORDER BY id DESC LIMIT 1)*8/1e6 AS INT) || '|' ||
+        CAST((SELECT max(download) FROM (SELECT download FROM results WHERE status='completed' ORDER BY id DESC LIMIT 4))*8/1e6 AS INT) || '|' ||
+        CAST((SELECT max(upload)   FROM (SELECT upload   FROM results WHERE status='completed' ORDER BY id DESC LIMIT 4))*8/1e6 AS INT)
+      FROM results WHERE status='completed';" 2>/dev/null)
+    st_age=$(echo "$st_row"  | cut -d'|' -f1)
+    st_dl=$(echo "$st_row"   | cut -d'|' -f2)
+    st_ul=$(echo "$st_row"   | cut -d'|' -f3)
+    st_mdl=$(echo "$st_row"  | cut -d'|' -f4)
+    st_mul=$(echo "$st_row"  | cut -d'|' -f5)
+    if [ -z "$st_age" ]; then
+        add_line "Internet speed" "no completed speedtest results found"
+    elif [ "$st_age" -gt 14 ]; then
+        add_line "Internet speed" "STALE — newest test ${st_age}h ago (expected every 6h; scheduler stalled?)"
+    elif [ "$st_mdl" -lt "$ST_FLOOR_DOWN" ] || [ "$st_mul" -lt "$ST_FLOOR_UP" ]; then
+        add_line "Internet speed" "LOW — best of last 4: ${st_mdl}/${st_mul} Mbps down/up (floor ${ST_FLOOR_DOWN}/${ST_FLOOR_UP})"
+    else
+        add_line "Internet speed" "OK" "${st_dl}/${st_ul} Mbps down/up, ${st_age}h ago"
+    fi
+else
+    add_line "Internet speed" "speedtest DB unreadable ($ST_DB)"
+fi
+
 # ── 5. Filesystem >85% ───────────────────────────────────────────────────────
 fs_over=$(prom_query '(1 - node_filesystem_avail_bytes{fstype!~"tmpfs|fuse.lxcfs|overlay|squashfs"} / node_filesystem_size_bytes{fstype!~"tmpfs|fuse.lxcfs|overlay|squashfs"}) * 100 > 85' | python3 -c "
 import json,sys
@@ -0,0 +1,62 @@
+global:
+  scrape_interval: 15s
+  evaluation_interval: 15s
+
+rule_files:
+  - "alert_rules.yml"
+
+alerting:
+  alertmanagers:
+    - static_configs:
+        - targets:
+            - "alertmanager:9093"
+
+scrape_configs:
+  - job_name: prometheus
+    static_configs:
+      - targets: ["prometheus:9090"]
+
+  - job_name: alertmanager
+    static_configs:
+      - targets: ["alertmanager:9093"]
+
+  - job_name: node-exporter
+    static_configs:
+      - targets:
+          - "192.168.99.200:9100"
+          - "192.168.99.192:9100"
+          - "192.168.99.193:9100"
+          - "192.168.99.194:9100"
+          - "192.168.99.153:9100"
+          - "192.168.99.183:9100"
+          - "192.168.99.186:9100"
+          - "192.168.99.187:9100"
+          - "192.168.99.190:9100"
+          - "192.168.99.227:9100"
+          - "192.168.99.196:9100"
+          - "192.168.99.198:9100"
+          - "192.168.99.199:9100"
+          - "192.168.99.100:9100"
+    metric_relabel_configs:
+      - source_labels: [nodename]
+        regex: '0d869efa-prometheus-node-exporter'
+        target_label: nodename
+        replacement: haos
+
+  - job_name: nut
+    static_configs:
+      - targets: ['192.168.99.200']
+    metrics_path: /metrics
+    params:
+      target: ['192.168.99.200']
+    relabel_configs:
+      - target_label: __address__
+        replacement: '192.168.99.183:9995'
+
+  - job_name: cadvisor
+    static_configs:
+      - targets: ["cadvisor:8080"]
+
+  - job_name: crowdsec
+    static_configs:
+      - targets: ["192.168.99.186:6060"]
@@ -1,8 +1,11 @@
 #!/bin/bash
 # ZFS pool health check — /usr/local/bin/zfs-health-check.sh
 # Runs every 5 min via root crontab on PBS.
-# Three states: ONLINE (silent), DEGRADED/FAULTED (ntfy high), MISSING (ntfy urgent).
+# States: ONLINE (silent), DEGRADED/FAULTED (ntfy high), NOT-IMPORTED w/ members
+#   present (ntfy high — failed import, hardware fine), MISSING members (ntfy urgent).
 # Writes Prometheus textfile metrics after full loop completes only.
+# NOTE: deployed by Ansible role `pbs` (roles/pbs/files/zfs-health-check.sh).
+#       Edit the role file, not the live copy — the playbook overwrites it.

 NTFY_URL="https://ntfy.goattw.net/zfs-health"
 EXPECTED_POOLS=(usb1-zfs usb2-zfs)
@@ -18,6 +21,8 @@ logger -t zfs-health-check "starting pool health check"
 METRICS=""
 METRICS+="# HELP zfs_pool_present 1 if pool imported and healthy, 0 if missing\n"
 METRICS+="# TYPE zfs_pool_present gauge\n"
+METRICS+="# HELP zfs_pool_members_present count of expected by-id members visible when a pool is not imported\n"
+METRICS+="# TYPE zfs_pool_members_present gauge\n"
 METRICS+="# HELP zfs_health_last_run_seconds Unix timestamp of last successful check\n"
 METRICS+="# TYPE zfs_health_last_run_seconds gauge\n"

@@ -40,16 +45,45 @@ for pool in "${EXPECTED_POOLS[@]}"; do
            || logger -t zfs-health-check "WARNING: ntfy post failed for $pool"

    else
-        # Empty or error — pool not imported / missing
-        logger -t zfs-health-check "pool $pool: MISSING — alerting"
+        # Pool not imported. Distinguish two very different causes before alerting:
+        #   devices ABSENT  -> real hardware (cable/hub/enclosure/power)
+        #   devices PRESENT -> pool merely not imported (stale cache, boot race) -> import fixes it
+        case "$pool" in
+            usb1-zfs) members="/dev/disk/by-id/usb-ASMT_2115_ACAAEBBB2E5F-0:0-part1 /dev/disk/by-id/wwn-0x5000cca234ca7110-part1" ;;
+            usb2-zfs) members="/dev/disk/by-id/wwn-0x50014ee2075b3126-part1 /dev/disk/by-id/wwn-0x50014ee211ed75f6-part1" ;;
+            *)        members="" ;;
+        esac
+
+        present=0; total=0
+        for m in $members; do
+            total=$((total+1))
+            [ -e "$m" ] && present=$((present+1))
+        done
+
        METRICS+="zfs_pool_present{pool=\"$pool\"} 0\n"
-        curl -sf \
-            -H "Priority: urgent" \
-            -H "Tags: rotating_light" \
-            -H "Title: PBS ZFS pool MISSING: $pool" \
-            -d "Pool $pool not imported on PBS. Check cables. Fix: zpool import $pool" \
-            "$NTFY_URL" \
-            || logger -t zfs-health-check "WARNING: ntfy post failed for $pool"
+        METRICS+="zfs_pool_members_present{pool=\"$pool\"} $present\n"
+
+        if [ "$total" -gt 0 ] && [ "$present" -eq "$total" ]; then
+            # All members enumerated — this is a failed import, NOT a hardware fault.
+            logger -t zfs-health-check "pool $pool: NOT IMPORTED (all $total members present) — alerting"
+            curl -sf \
+                -H "Priority: high" \
+                -H "Tags: warning" \
+                -H "Title: PBS ZFS pool not imported: $pool" \
+                -d "Pool $pool not imported, but all $total members are present. Hardware is fine — likely failed auto-import. Fix: zpool import -d /dev/disk/by-id $pool" \
+                "$NTFY_URL" \
+                || logger -t zfs-health-check "WARNING: ntfy post failed for $pool"
+        else
+            # Members missing (or unknown pool) — genuine hardware suspicion.
+            logger -t zfs-health-check "pool $pool: MISSING ($present/$total members present) — alerting"
+            curl -sf \
+                -H "Priority: urgent" \
+                -H "Tags: rotating_light" \
+                -H "Title: PBS ZFS pool MISSING: $pool" \
+                -d "Pool $pool not imported; only $present of $total members visible. Check cables/enclosure/power. Then: zpool import -d /dev/disk/by-id $pool" \
+                "$NTFY_URL" \
+                || logger -t zfs-health-check "WARNING: ntfy post failed for $pool"
+        fi
    fi
 done

@@ -0,0 +1,42 @@
+# Internal DNS — source of truth
+
+Technitium is authoritative for the **internal** view of `goattw.net` (split-horizon;
+`goattw.net` is Cloudflare-fronted publicly). Technitium records are managed in the
+**Technitium UI — there is no IaC**, so this file is the human source-of-truth: if a
+record is wrong or a rebuild loses it, reconcile against this table.
+
+- Primary DNS: **Technitium @ 192.168.99.184**
+- Secondary DNS: **router @ 192.168.99.1** (serves the same answers; also went to .185 below)
+
+## Records that must be correct
+
+| Name | Must resolve to | Why |
+|------|-----------------|-----|
+| `beast.goattw.net` | **192.168.99.200** | Beast = PVE master / NUT (upsd) master. |
+
+Note a broad `*.goattw.net → 192.168.99.185` internal-ingress pattern exists (e.g.
+`pbs.goattw.net` also answers `.185`). Specific host records like `beast` must be
+explicit A-records that override the wildcard.
+
+## ⚠️ OPEN (2026-07-03): the beast correction did NOT take
+
+`dig beast.goattw.net @192.168.99.184` returns **192.168.99.185** — authoritative
+(`aa` flag, static TTL 3600), not a cache. The intended edit to `.200` was never
+applied (or reverted). `.185` runs no `upsd`, which is what broke NUT (below).
+**ACTION (supervised): re-add/fix `beast.goattw.net A → 192.168.99.200` in Technitium,
+then re-verify `dig beast.goattw.net @192.168.99.184` → .200.**
+
+## Design rule: NUT must not depend on DNS
+
+Because `beast.goattw.net` regressed to `.185`, every node whose `upsmon.conf` monitored
+`cyberpower*@beast.goattw.net` lost UPS communication (no graceful shutdown on power
+events). Fixed by pinning the MONITOR target to the **IP `192.168.99.200`**, captured in
+Ansible `deploy_nut_client.yml` (commit `3e5bfc4`). Affected nodes: pbs (hand-fixed
+2026-07-03), compute2/3/4, docker-node01/02 (repaired on next supervised redeploy).
+
+## OPEN: Technitium config backup
+
+Confirm `.184`'s Technitium **config volume** is captured in a backup (so the internal
+zone survives a rebuild) — either as a PBS VM/host backup of `.184`, or Technitium's own
+**Settings → scheduled backup**. Status: **TO CONFIRM** (not verifiable without PVE/PBS
+or `.184` access from the current host).
@@ -0,0 +1,74 @@
+# Homelab Hardware Inventory
+
+_Gathered 2026-07-06 via SSH probe (all hosts) + Proxmox read-only API. Serials/DIMM/SMART captured only on nodes with passwordless sudo (compute5, compute6); the rest are pending a sudo pass._
+
+## PVE cluster — physical nodes (6)
+
+| Node | IP | Model | CPU | RAM | Storage | iGPU | NIC |
+|---|---|---|---|---|---|---|---|
+| **beast** | .200 | Gigabyte X570 AORUS MASTER (BIOS F39) | Ryzen 9 5950X (16c/32t) | 128 GB | 2× Samsung 870 EVO 1TB SATA, 2× Crucial BX500 2TB SATA, 2× HGST 8TB USB HDD (USB-ZFS), Sabrent Rocket 4TB NVMe, Samsung 990 PRO 2TB NVMe | — (2× P2200 discrete, passed to VM 101) | 1 GbE |
+| **compute2** | .192 | Dell OptiPlex 3090 | i5-10500T (6c/12t) | 32 GB | Crucial MX500 1TB SATA, Samsung 970 EVO+ 1TB NVMe | UHD 630 | 1 GbE |
+| **compute3** | .193 | Dell OptiPlex 3080 | i5-10500T (6c/12t) | 32 GB | Crucial BX500 1TB SATA, Samsung 990 PRO 2TB NVMe | UHD 630 | 1 GbE |
+| **compute4** | .194 | Dell OptiPlex 7050 | i7-7700T (4c/8t) | 16 GB | Bliksem 120GB SATA, SK hynix PC601 1TB NVMe | HD 630 | 1 GbE |
+| **compute5** | .196 | Dell OptiPlex Micro Plus 7010 (SN 1JPM7Y3) | i7-13700T (16c/24t) | 32 GB DDR5-4800 | WD SN740 256GB NVMe (18% wear), SK hynix PC711 1TB NVMe | UHD 770 | 1 GbE |
+| **compute6** | .198 | GMKtec NucBox K10 | i9-13900HK (14c/20t) | 64 GB DDR5-5600 (2×32, A-DATA) | EDILOCA EN206 256GB SATA, Crucial P5 1TB + P3 Plus 1TB NVMe | Iris Xe | **100 Mb ⚠️** |
+
+All PVE nodes on kernel 6.17.13-pve. **beast = PVE master + NUT master**, holds the GPUs and most critical guests.
+
+## Standalone physical hosts
+- **pbs** (.153) — HP EliteDesk 800 G4 DM 35W, i5-8500T (6c), 32 GB, Samsung 870 EVO 1TB SATA + Samsung 990 EVO 1TB NVMe + USB drives backing the USB-ZFS datastores (1TB Toshiba MD1000, 2× Hitachi 3TB, Toshiba 2TB, WD10EZEX 1TB), 1 GbE. Proxmox Backup Server 7.0.0-3 — **not** a cluster member.
+- **pi4** (.227) — Raspberry Pi 4 (Cortex-A72 4c, 4 GB, 32 GB microSD), 1 GbE, aarch64. Off-host Prometheus watchdog.
+
+## Storage appliances
+- **TrueNAS Scale = VM 100 on beast** (4 vCPU, 34 GB) — serves `tank` over NFS (`/mnt/Tank/Media` → media-server `/mnt/media`). Physical disk backing = beast's disks (exact passthrough map TBD).
+- **Synology** (.82) — external physical NAS (model TBD, no SSH). NFS `/volume1/PBS` → PBS datastore.
+
+## GPUs
+- **2× NVIDIA Quadro P2200** (5 GB GDDR5 each) — physically in beast, **PCI-passthrough to Media-Server VM 101** (driver 580.159.03). Used for Plex / Immich / Tdarr transcode.
+- iGPUs for QuickSync: compute4 (HD 630) + compute6 (Iris Xe) run Tdarr nodes.
+
+## UPS (NUT master = beast)
+- **cyberpower1** — CyberPower CP1500PFCLCDa (1500 VA / 1000 W tower), ~31% load.
+- **cyberpower2** — CyberPower CP1500PFCRM2U (1500 VA / 1000 W rack), ~12% load.
+
+## Guest / workload allocation
+
+**PVE VMs & LXCs**
+- **beast**: 100 TrueNAS (4c/34G), 101 Media-Server (8c/17G/230G + 2 GPU), 102 dns-server LXC (Technitium), 103 nextcloud LXC (8c/9G), 104 nut LXC, 108 docker-node02 (8c/4G), 900 ubuntu-template (stopped).
+- **compute2**: 106 docker-node01 (4c/6G), 107 ansible-control (2c/2G).
+- **compute3**: 109 wazuh (4c/9G).
+
+**Host-level Docker on PVE nodes** (NOT LXC-wrapped — `pct list` empty on these):
+- **compute4**: tdarr-node.
+- **compute5**: node-exporter.
+- **compute6**: tdarr-node, diun, vikunja, speedtest-tracker, ntfy, node-exporter, netdata, glances, watchyourlan, **linkding**.
+
+## Disk health & serials (SMART via Proxmox API, 2026-07-06 — all PASSED)
+Wear = % life remaining (100 = new).
+
+| Node | Device | Model | Serial | Size | Wear |
+|---|---|---|---|---|---|
+| beast | nvme0n1 | Samsung 990 PRO 2TB | S7L9NJ0Y428635T | 2 TB | 98% |
+| beast | nvme1n1 | Sabrent SB-ROCKET-4TB | R24093062000007 | 4 TB | 100% |
+| beast | sda / sdb | Samsung 870 EVO 1TB | S75BNL0XC20021M / …26272M | 1 TB | 99% |
+| beast | sdc / sdd | Crucial BX500 2TB | 2306E6A93ADA / …3C38 | 2 TB | 98% |
+| beast | sde / sdf | HGST HUS728T8TALE6L4 8TB (USB) | VGGANA5G / VGK1WUDG | 8 TB | HDD |
+| compute2 | nvme0n1 | Samsung 970 EVO Plus 1TB | S6S1NS0TB13410F | 1 TB | 99% |
+| compute2 | sda | Crucial MX500 1TB | 2310E6B97787 | 1 TB | 91% |
+| compute3 | nvme0n1 | Samsung 990 PRO 2TB | S73WNU0XA01509Y | 2 TB | 99% |
+| compute3 | sda | Crucial BX500 1TB | 2306E6A8BFB7 | 1 TB | 97% |
+| compute4 | nvme0n1 | SK hynix PC601 1TB | NS03N5816101Y3M12 | 1 TB | 98% |
+| compute4 | sda | Bliksem 128GB | YS202010005501AA | 128 GB | 100% |
+| compute5 | nvme0n1 | WD SN740 256GB | 22340J800493 | 256 GB | 82% |
+| compute5 | nvme1n1 | SK hynix PC711 1TB | AJ0BNA4391070C757 | 1 TB | 100% |
+| compute6 | nvme0n1 | Crucial P5 1TB | 20512C08236A | 1 TB | 96% |
+| compute6 | nvme1n1 | Crucial P3 Plus 1TB | 25144F6C8F15 | 1 TB | 100% |
+| compute6 | sda | EDILOCA EN206 256GB | MX_00000000000060007 | 256 GB | HDD/N-A |
+
+Most-worn SSD = compute5 WD SN740 (82% remaining) — still healthy. pbs disk serials/SMART not yet captured (PBS API token mismatch, sudo pw rejected).
+
+## Notable findings / TODOs
+- ⚠️ **compute6 NIC runs at 100 Mb** (ethtool-confirmed; the K10 has 2.5 GbE) — bad cable/switch port. `speedtest-tracker` runs on this node, so its WAN results are capped/misleading. Fix the cable/port.
+- **beast concentration / SPOF** — PVE master + NUT master + GPUs + TrueNAS + Media-Server + Nextcloud + DNS all on one machine.
+- **Kernel skew** — docker-node01/02 + ansible-control on 6.8.0-111 (pending update); media-server 6.8.0-134; PVE nodes 6.17.13-pve.
+- **Disk SMART/serials: DONE for all 6 PVE nodes** (via Proxmox API, no sudo). Still deferred: **DIMM layout + system serial on beast/compute2/3/4** (need `sudo`; the password tried was rejected), **pbs disk SMART/serials** (PBS API token id unknown, sudo pw rejected), TrueNAS `tank` disk-passthrough map, Synology model.
@@ -0,0 +1,46 @@
+# Nextcloud large-file upload fix (CT103) — "Unknown error during upload"
+
+**Date:** 2026-07-05
+**Symptom:** Uploading large movie files via the Nextcloud web UI fails with *"Unknown error during upload"* (small files fine).
+**ROOT CAUSE:** **Traefik 30 s `responseHeaderTimeout`.** Large files upload in chunks, then a final `MOVE` **assembles** them server-side — for a multi-GB movie that takes **>30 s** (read+write the whole file over NFS). Traefik's `default-transport` had `responseHeaderTimeout: 30s`, so it 504'd the MOVE at 30 s → browser "Unknown error." Nextcloud's log shows *"Could not open file: …/<chunk>, doesn't seem to exist"* — that's the *symptom* (assembly stream cut off mid-read), not the cause.
+**FIX:** dedicated `nextcloud-transport` serversTransport with `responseHeaderTimeout: 3600s`, and point `nextcloud-service` at it. Applied to both Traefik nodes.
+
+> **NOTE / red herring:** I initially chased **Cloudflare / browser DoH** (Cloudflare *is* in front at `cloud.goattw.net`, CF caps bodies at ~100 MB). That was WRONG — the failing client was on the LAN direct (`ClientHost 192.168.99.238` in Traefik's access log), and the kill was a clean `504 ... 30000ms`. DoH-disable `.reg` (in git history of this file) is harmless but was not the fix. Lesson: **read the proxy access log first** — the `504 @ 30000ms` was the whole answer.
+
+---
+
+## Setup
+- Nextcloud = **AIO**, Docker inside **LXC CT103** on Beast (.200). Access: `ssh tommy@192.168.99.200` → **`sudo pct exec 103 -- ...`** (`sudo pct` NOPASSWD on Beast). occ = `docker exec -u www-data nextcloud-aio-nextcloud php /var/www/html/occ ...`.
+- Public `cloud.goattw.net` → Cloudflare → Traefik. LAN `cloud.goattw.net` → Technitium/UniFi return **192.168.99.185** (internal Traefik, LE cert) → Traefik (.186 active / .187 backup) → backend **http://192.168.99.31:11000** (aio-apache). Data dir `/mnt/ncdata` = NFS TrueNAS (.29). NC v32.
+- Size limits were never the issue — already 16 G; AIO manages php.ini via env, don't hand-edit.
+
+## The fix (in `traefik/dynamic_conf.yml` on .186 AND .187)
+Add a transport next to `default-transport`:
+```yaml
+    nextcloud-transport:
+      forwardingTimeouts:
+        dialTimeout: 30s
+        responseHeaderTimeout: 3600s   # was 30s on default-transport → killed assembly
+        idleConnTimeout: 90s
+```
+Point the service at it:
+```yaml
+    nextcloud-service:
+      loadBalancer:
+        servers: [{ url: "http://192.168.99.31:11000" }]
+        serversTransport: nextcloud-transport   # was: default-transport
+```
+Mirrored in this repo: `traefik/dynamic_conf_node01.yml` (.186) and `dynamic_conf_node02.yml` (.187).
+
+## ⚠️ How to edit Traefik dynamic_conf WITHOUT breaking it (hard-won)
+`dynamic_conf.yml` is a **single-file bind mount** into the Traefik container. Therefore:
+- **Edit IN-PLACE only** (`cp tmp dynamic_conf.yml`, `sed -i`) — keeps the same inode, Traefik hot-reloads, no restart. This is how .187 was done.
+- **NEVER `mv` a new file over it.** `mv` swaps the inode → the container stays pinned to the OLD inode (serves stale config, silently) → only a **container restart** re-links it. This is how .186 got stuck (had to `docker restart traefik`).
+- **Don't stream-write it** (`base64 -d > file`, slow redirects) — Traefik's watcher can read a half-written file → `yaml: could not find expected ':'` parse error → **it drops the WHOLE file config** (all `@file` middlewares: authelia, crowdsec, secure-headers vanish). Write to a temp, validate (`python3 -c 'import yaml;yaml.safe_load(open("f"))'`), then `cp` in-place.
+- After any edit, verify: `curl -s http://<node>:8080/api/rawdata` → `services.nextcloud-service@file.loadBalancer.serversTransport` == `nextcloud-transport@file`, and `middlewares` count unchanged (8).
+
+## Verification (done)
+8 GB incompressible (`/dev/urandom`) chunked upload straight through Traefik: **MOVE = 201, assembly_time = 31.7 s**, exact size. Old config 504'd the identical test at 30.0 s. ✅
+
+## If large files STILL fail after this
+Next timeouts to check, in order: aio-apache `APACHE_MAX_TIME` (already 3600), PHP `max_execution_time` (3600), Cloudflare (only relevant for *external* clients — its ~100 s proxy timeout + 100 MB body cap; use the internal `.185` path or the desktop client from outside).
@@ -80,7 +80,7 @@ DEADTIME 15   →   DEADTIME 300

 ---

-### INC-001 — PBS USB Hub Failure (ACTIVE — hardware fix in progress)
+### INC-001 — PBS USB Hub Failure (import/alerting RESOLVED + hardened 2026-07-02 — see P5-14; hardware-hub history retained below)

 **First event:** March 12, 2026 (earliest confirmed crash in journal)
 **Crash count:** 34+ unclean reboots since April 21; ~50+ total since March 12
@@ -307,6 +307,77 @@ Subscriber reminder: subscribe ntfy client to topic `health-digest` on 192.168.9

 ---

+### P5-13 — media-server Kernel Upgrade 6.8.0-124 → 6.8.0-134 (DONE 2026-07-01)
+
+**Host:** media-server (192.168.99.183 / VM 101)
+**Rebooted:** 2026-07-01 23:54 CDT into `6.8.0-134-generic` (was running `-124`).
+
+Kernel was staged earlier that day (6.8.0-134 on disk; `nvidia-srv/580.159.03` DKMS module built + signed for -134). `GRUB_DEFAULT=0` → reboot booted -134; `-124` and `-117` remain DKMS-built in the GRUB Advanced menu as fallbacks. Rollback strategy = PBS live backup of VM 101 (nvmebak storage can't snapshot without shutdown).
+
+**Post-boot checks — all green:**
+- `uname -r` = `6.8.0-134-generic`
+- `nvidia-smi` — both Quadro P2200s present, driver 580.159.03 loaded; `dkms status` shows nvidia-srv built for -134
+- 54/54 containers running, none restarting/unhealthy
+- 18/18 Prometheus targets up
+- 0 failed systemd units
+- `/mnt/media` (TrueNAS .29 NFS) mounted
+
+**⚠️ Gluetun caveat — a stale PIA WireGuard peer does NOT survive a host reboot.** After the reboot gluetun came back on the last-written `wg0.conf` (Jun-30 peer, endpoint `181.41.206.145`) which no longer passed traffic: tight VPN restart loop, healthchecks failing with `dial tcp4: lookup github.com/cloudflare.com: i/o timeout`, `public_ip` stuck at `""`. Even though that peer had been freshly registered Jun 30, it did not survive the cold start. Fixed by running the documented self-heal:
+```bash
+~/downloads/pia-wg/regen-and-up.sh    # regen fresh peer → recreate gluetun → reconnect sab/transmission
+```
+Result: fresh `us_denver` peer (endpoint `181.41.206.159:1337`), gluetun Healthy, egress verified as PIA Denver, Colorado. **Treat `regen-and-up.sh` as a required post-reboot step for the downloads stack — don't wait for the VpnTunnelDown alert.**
+
+---
+
+### P5-14 — INC-001 PBS USB-ZFS Recovery + Import/Alert Hardening (DONE 2026-07-02)
+
+**Host:** PBS (192.168.99.153). Closes the acute import/alert half of INC-001 (see full section above).
+
+**Context — this instance was NOT a hardware drop.** Both `usb1-zfs` and `usb2-zfs` again showed MISSING (`zpool list` = "no pools available"; `zfs-import-cache.service` + `zfs-import@usb{1,2}-zfs` units failed). But all four ASM1153E bridges were enumerated on the hub and all four mirror members had intact `zfs_member` labels. Root cause was a **failed auto-import**: stale cache-import + `zfs-import-scan` disabled (no fallback) + non-deterministic `/dev/sdX` USB enumeration order through the hub, so cached device paths no longer matched.
+
+**1. Resolution — import by stable by-id paths** (no `-f` needed):
+```bash
+zpool import -d /dev/disk/by-id usb1-zfs
+zpool import -d /dev/disk/by-id usb2-zfs
+zpool status -P     # confirmed paths now read /dev/disk/by-id/... not /dev/sdX
+```
+Both ONLINE, 0 errors. `usb1-zfs` 613G data intact (mounted `/mnt/datastore/usb1-zfs`); `usb2-zfs` ~empty. Scrubs clean.
+
+**2. by-id cachefile pin** (persists the fix across USB re-enumeration):
+```bash
+zpool set cachefile=/etc/zfs/zpool.cache usb1-zfs
+zpool set cachefile=/etc/zfs/zpool.cache usb2-zfs
+```
+Also codified as a task in the `pbs` Ansible role.
+
+**3. Boot-race guard — `wait-for-zfs-usb.sh` + systemd drop-in.** `systemd-udev-settle` drains the current udev queue but does not wait for slow USB hub/enclosure enumeration, so `zfs-import-cache.service` can fire before the by-id member symlinks exist. Added:
+- `/usr/local/sbin/wait-for-zfs-usb.sh` — polls for the 4 by-id members, max 60s, **always exits 0** (never wedges boot)
+- `/etc/systemd/system/zfs-import-cache.service.d/wait-for-usb.conf` — `ExecStartPre=/usr/local/sbin/wait-for-zfs-usb.sh`
+
+Real validation is a PBS reboot (deferred to a maintenance window).
+
+**4. Alert split — devices-present vs devices-absent** (fixes the "check cables" misfire). The `zfs-health-check.sh` MISSING branch previously always cried *"Check cables. Fix: zpool import"* — it could not tell a real hardware drop from a pool merely unimported. Now it tests member presence:
+- all by-id members present → ntfy **high**: "not imported, hardware fine — Fix: `zpool import -d /dev/disk/by-id`"
+- members missing → ntfy **urgent**: "only N of M visible — check cables/enclosure/power"
+
+New metric `zfs_pool_members_present` gives Prometheus a scrapable signal (a fully-unimported pool emits no `node_zfs_zpool_state` series, so Prometheus was previously blind to this exact case).
+
+**5. Cron ownership fix (double-fire root cause).** The alert script was running **twice** every 5 min — once from an unmarked root crontab entry (the original) and once from the Ansible-managed `user: tommy` entry (which also carried a duplicated `#Ansible:` marker). This is what produced the **paired ~9:55/10:00 ntfy notifications**: two independent runs firing seconds apart. Worse, the tommy run **silently failed to write** the node_exporter textfile (dir is `root:root`), so `zfs_pool_members_present` would never have landed under it. Converged to ONE root-managed entry:
+- Ansible cron task repointed `user: tommy` → `user: root`, `state: present` (mirrors the working `docker-image-prune` pattern)
+- added eviction task (`name: zfs-health-check`, `user: tommy`, `state: absent`) to drop the tommy job + duplicate marker
+- one-time **manual** delete of the unmarked original root line (Ansible can't dedupe an entry lacking its marker — done first, before the converge, to avoid a momentary double-root window)
+
+**Deployment.** Script + cron ownership + cachefile now live in the `pbs` Ansible role (`tommy/ansible` commit `c9a028b`); deploy via `ansible-playbook pbs.yml --tags zfs --ask-become-pass`. The role had been shipping an inline **stub** that drifted far behind the live script — running it would have clobbered the good 3-state/metrics script; now reconciled to `roles/pbs/files/zfs-health-check.sh` (this repo's `pbs/zfs-health-check.sh` is the synced copy).
+
+**Scope note.** The acute import failure and the alert/cron bugs are resolved and hardened. The long-running USB-**hub reliability** history above (34+ crash-reboots) is a separate hardware concern — these changes harden import + alerting, they do not replace the hub. Current state: all members enumerated, both pools ONLINE and stable.
+
+**Deferred / passive follow-ups (nothing to actively revisit):**
+- **Reboot verification** — `wait-for-zfs-usb.sh` (ExecStartPre) is only exercised on a cold boot. On the next PBS reboot, confirm it ran: `journalctl -b -u zfs-import-cache.service | grep wait-for-zfs-usb` (expect "all N members present after Ns", then a clean cache import with both pools auto-imported).
+- **Orphaned cron marker** — root's crontab still carries a stray `#Ansible: Prune PBS task logs` marker with **no matching task** in the `pbs` role (unmanaged leftover from a removed task). Harmless — no job is re-created by any playbook. Delete via `crontab -e` (as root) whenever convenient.
+
+---
+
 ## Known Config Drift (not incidents — track for cleanup)

 | Item | Location | Drift |
@@ -8,6 +8,11 @@ http:
        idleConnTimeout: 90s
    insecure-transport:
      insecureSkipVerify: true
+    nextcloud-transport:
+      forwardingTimeouts:
+        dialTimeout: 30s
+        responseHeaderTimeout: 3600s
+        idleConnTimeout: 90s

  middlewares:

@@ -79,7 +84,7 @@ http:
      plugin:
        crowdsec-bouncer:
          enabled: true
-          crowdsecLapiKey: VMCnws/j+9pmsT4YT+t3HzrvX8OhBCwoquwo4NqWJPs
+          crowdsecLapiKeyFile: /plugins-storage/crowdsec_lapi_key
          crowdsecLapiHost: crowdsec:8080
          crowdsecMode: live

@@ -463,7 +468,7 @@ http:
    nextcloud-service:
      loadBalancer:
        servers: [{ url: "http://192.168.99.31:11000" }]
-        serversTransport: default-transport
+        serversTransport: nextcloud-transport

    homeassistant-service:
      loadBalancer:
@@ -8,6 +8,11 @@ http:
        idleConnTimeout: 90s
    insecure-transport:
      insecureSkipVerify: true
+    nextcloud-transport:
+      forwardingTimeouts:
+        dialTimeout: 30s
+        responseHeaderTimeout: 3600s
+        idleConnTimeout: 90s

  middlewares:

@@ -79,7 +84,7 @@ http:
      plugin:
        crowdsec-bouncer:
          enabled: true
-          crowdsecLapiKey: VMCnws/j+9pmsT4YT+t3HzrvX8OhBCwoquwo4NqWJPs
+          crowdsecLapiKeyFile: /plugins-storage/crowdsec_lapi_key
          crowdsecLapiHost: 192.168.99.186:8081
          crowdsecMode: live

@@ -463,7 +468,7 @@ http:
    nextcloud-service:
      loadBalancer:
        servers: [{ url: "http://192.168.99.31:11000" }]
-        serversTransport: default-transport
+        serversTransport: nextcloud-transport

    homeassistant-service:
      loadBalancer:
Author	SHA1	Message	Date
tommy	250fea8679	ups-quarterly-test: sync tracked copy to live + fix false-alarm bugs The tracked copy was a stale pre-refactor version (no sequential-test safety, no textfile metric). Replace with the current script plus fixes for the UPSBatteryTestFailed false alarm: - Poll ups.test.result until terminal ("passed"/"failed") instead of a single racy read — CyberPower updates the string slower than the OL transition, which recorded 0 despite a passing test. - to_gauge matches passed substring, not exact "Done and passed". - Fix cron OR-logic: setting both DOM (1-7) and DOW (0) made cron run on EITHER (daily Jul 1-7 + Sundays). Script now self-guards to day<=7; cron fires every Sunday of the quarter months. - Externalize NUT_PASS to /etc/ups-quarterly-test.env (root:600) so the password is no longer embedded in the repo. Deploy via ups-fix.sh. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-07 18:31:46 -05:00
tommy	7178f6ceea	weekly-health-digest: add internet speed check (section 4c) Reports the speedtest-tracker (on .183) newest result: flags STALE if >14h old (2+ missed 6h cycles → scheduler stalled) or LOW if the best of the last 4 tests falls below the 500/500 Mbps floor (best-of-4 avoids flagging a transient ISP dip). Floors adjustable. Local SQLite read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-07 18:13:16 -05:00
tommy	a567f64508	weekly-health-digest: add LubeLogger + Vaultwarden PBS backup freshness New section 4b flags either app backup if its newest PBS snapshot in rust-usb is >26h old (catches a missed nightly run). Queries via the lubelogger token (datastore-wide audit covers both groups) over keyed SSH to docker-node01; read-only and failure-guarded. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-07 17:39:24 -05:00
tommy	b43007c9a9	hardware-inventory: add disk SMART/serials for all PVE nodes (via Proxmox API) All 19 PVE-node disks health=PASSED. Serials + wear captured without sudo. Still pending: DIMM/system-serial on 4 nodes + PBS disk SMART. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-06 22:31:40 -05:00
tommy	a3aed327ec	Add hardware inventory (PVE nodes, standalone hosts, GPUs, UPS, allocation map) Gathered 2026-07-06 via SSH probe + Proxmox read-only API. Flags: compute6 NIC @100Mb (bad cable/port), beast SPOF concentration. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-06 22:15:45 -05:00
tommy	5c66bbacdd	traefik: nextcloud-transport (3600s responseHeaderTimeout) — fix large-upload 504 Real root cause of Nextcloud "Unknown error during upload" on large files: Traefik's default-transport responseHeaderTimeout=30s killed the chunk-assembly MOVE (which takes >30s for multi-GB files) -> 504. Added a dedicated nextcloud-transport (3600s) and pointed nextcloud-service at it, on both nodes. Proven: 8GB upload MOVE now 201 @ 31.7s (was 504 @ 30.0s). Rewrote the runbook: corrected cause (was mis-attributed to Cloudflare/DoH), plus the hard-won gotcha on editing single-file-bind-mounted dynamic_conf (in-place cp only; never mv; validate before write). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-05 19:07:04 -05:00
tommy	c6861c71f0	runbooks: Nextcloud large-upload fix (CT103) — browser DoH, not a size limit Full diagnosis + fix for "Unknown error during upload" on large files: root cause is browser DNS-over-HTTPS bypassing LAN DNS -> Cloudflare mangles the parallel chunk stream. Server/DNS already correct (Technitium+UniFi return internal .185). Fix = disable browser DoH (.reg/PowerShell/GPO included). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-05 18:17:48 -05:00
tommy	cfe95a3aef	monitoring: scrape Wazuh VM node-exporter (.199) + track prometheus.yml Wazuh VM (vmid 109, .199) ran node-exporter v1.8.2 all along but was never in Prometheus's scrape list -> its 2-month outage was invisible. Add 192.168.99.199:9100 to the node-exporter job; the existing NodeDown alert (up{job=node-exporter}==0, 2m, critical -> ntfy) now covers it. Also mirror prometheus.yml into git for the first time (only alert_rules.yml was tracked) to end live-only drift. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-03 16:54:46 -05:00
tommy	9653af3d20	homepage: add siteMonitor status dots to 54 service tiles Every tile lacking a dot now has one, validated from the Homepage container's own network vantage (status <400 = green). Targets chosen to avoid false-red: - internal http:// url where a widget exposes one (bypasses edge/redirects) - public href for href-only services (all resolve to own login/200 — none Authelia-gated) - unauth health endpoints where the base URL false-reds: CrowdSec /health, Loki :3100/ready, Stirling /api/v1/info/status OMITTED (would be permanent false-red, flagged not added): - Proxmox Beast, PBS: self-signed TLS -> SSL error (widgets already report health) - Kometa: a CLI tool, no web service (href is a GitHub link) Wazuh got a dot but currently renders RED — its backend returns 502 (genuinely unreachable via Traefik); that's true state, not a false-red. Investigate separately. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-03 16:20:25 -05:00
tommy	6002af049f	runbooks: add dns-records.md source-of-truth Captures the beast.goattw.net -> 192.168.99.200 intent (LIVE state still wrong at .185, correction never took), the NUT-must-not-depend-on-DNS design rule (ansible 3e5bfc4), and the open Technitium .184 config-backup verification. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-03 11:29:43 -05:00
tommy	5ba7246b45	4K: pre-create download-client completed category dirs in playbook radarr-4k/sonarr-4k surfaced 'download client places downloads in X but it does not appear to exist' health errors: the SABnzbd (movies-4k/tv-4k) and Transmission (radarr-uhd/sonarr-uhd) completed-category dirs dont exist until the first grab lands in each category. On a rebuild those errors recur until downloads happen. Add an idempotent file: task creating all four at 1000:3000 (abc:media) 0775 to match the /data tree. Root cause of the original root:0770 drift on movies-4k: a manual docker exec <client> mkdir runs as ROOT in linuxserver images (umask 007), not as the app user abc. Live dirs normalized via host chown (NFS honors no_root_squash); both instances now report 0 download-client-dir errors. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-03 10:11:18 -05:00
tommy	b25c3a4d5c	4K: download clients (packaged, idempotent) + Homepage tiles Add SABnzbd+Transmission to radarr-4k/sonarr-4k with distinct categories (sabnzbd movies-4k/tv-4k, transmission radarr-uhd/sonarr-uhd; Transmission forbids digits) so they dont cross-import the 1080p instances. Packaged into deploy-arrs-4k.yml via an idempotent ensure-arrs-4k-dl.py (changed=0). Homepage Radarr 4K/Sonarr 4K tiles (HOMEPAGE_VAR keys, siteMonitor). 4K path proven: Obsession routed to radarr-4k, grabbed WEBDL-2160p, 1080p rejected 'not wanted in profile'. Keys via vault, no secrets. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-03 09:43:18 -05:00
tommy	40da21016b	4K Step 2: Recyclarr config + systemd-timer sync (radarr-4k/sonarr-4k) Remux+WEB 2160p (radarr-4k) + WEB-2160p (sonarr-4k) profiles applied and assigned to the Overseerr 4K servers. DTS family explicitly pinned score 0 (Sonos-Arc), Dolby/Atmos at guide default. recyclarr:8, sync via systemd timer (daily, docker-run oneshot). Idempotent (changed=0); keys via ansible-vault, .env not committed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-03 01:01:53 -05:00
tommy	72d540fb85	4K quality-first arr stack: radarr-4k + sonarr-4k (Step 1 standup+wiring) Separate instances (ports 7879/8990) on the shared /mnt/media/data hardlink tree. Ansible inline-compose (idempotent). Wired: Prowlarr apps+sync, Overseerr native-4K servers (existing radarr is4k mislabel corrected), separate Plex 4K libraries, Traefik file-provider routers. Tdarr excludes the 4K roots. Recyclarr = deferred Step 2. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-03 00:12:07 -05:00
tommy	4b69f74f40	lubelogger+homepage: dedicated PBS tokens - backup job -> dedicated root@pam!lubelogger-backup token (DatastoreBackup); prune made non-fatal pending a group-scoped PBS prune-job (Datastore.Prune deferred). - Homepage PBS widget -> homepage@pbs!widget (Datastore.Audit); Proxmox Beast (root@pam!homepage) left intact, deferred. Secrets in vault/.env only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-02 23:40:32 -05:00
tommy	9559b219bf	lubelogger probe: use read-only x-api-key instead of admin basic-auth Non-admin service credential (View-scoped LubeLogger API key 'homepage-svc') for the reminder probe; admin password removed from /etc/lubelogger-reminders.env. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-02 22:35:59 -05:00
tommy	739db1ddee	homepage: crowdsec widget -> dedicated homepage-widget machine Repoint the CrowdSec widget from the engine's own 'localhost' machine cred to a dedicated 'homepage-widget' machine (cscli machines add, engine localhost cred untouched). Username change only; password in observability/.env. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-02 21:59:15 -05:00
tommy	b22697b2b9	homepage: canonize services.yaml mirror to live + redact all widget secrets Mirror was stale (missing ~28 services incl. LubeLogger). Sync live->mirror and redact every inline widget secret (19: plex/overseerr/tautulli/arr/sab keys + proxmox/pbs/beszel/crowdsec passwords) to {{HOMEPAGE_VAR_*}} refs. Values now live only in observability/.env (root:600, not a git repo), injected via env_file. Verified: 21 referenced vars == 21 defined in container env (no referenced-but- undefined); homepage healthy, widgets authenticate. Full-file + diff scan: clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-02 21:17:15 -05:00
tommy	6de37c8712	LubeLogger: back up data+keys volumes to PBS via the playbook Close the volume-backup gap: the pg_dump covered Postgres only; the ASP.NET DataProtection keyring (root-user credential) and uploaded docs live in the lubelogger-{data,keys} volumes. The docker-volume-backup sidecar can't target PBS natively, so both volumes now ride the same proxmox-backup-client job as the SQL dump — one host/lubelogger snapshot, three .pxar archives, keep-daily 14. Backup script+units are now installed by the Ansible playbook (idempotent, changed=0); PBS token from ansible-vault. Restore verified: keyring XML + sentinel doc byte-identical (sha256). Removes the now-playbook-managed standalone script/units. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-02 15:28:31 -05:00
tommy	c9aac20e73	Add LubeLogger deployment (docker-node01) Ansible-managed (inline-compose, file-provider router, host :8099), idempotent (changed=0). PBS-backed pg_dump (daily, keep 14) with verified round-trip restore; ntfy 'garage' reminder probe (.190); Diun repo-watch for new tags; Homepage tile+widget. Secrets via ansible-vault / /etc env files / HOMEPAGE_VAR — none in repo. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-02 15:14:20 -05:00
tommy	9e6afe27c5	runbooks: INC-001 PBS USB-ZFS recovery + import/alert hardening (P5-14) Sync pbs/zfs-health-check.sh to the deployed version and log the full INC-001 resolution arc: - recovery by importing pools on stable /dev/disk/by-id paths - by-id cachefile pin so the fix survives USB re-enumeration - wait-for-zfs-usb.sh + zfs-import-cache drop-in for the boot race - alert split: devices-present (import) vs devices-absent (hardware), new zfs_pool_members_present metric - cron ownership fix: root-managed single entry, tommy evicted, unmarked root line removed (double-fire root cause) Marks the acute import/alert half of INC-001 resolved; USB-hub reliability history retained as a separate hardware concern. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-02 11:24:00 -05:00
tommy	09a466b2c6	runbooks: log P5-13 media-server kernel upgrade to 6.8.0-134 Rebooted 2026-07-01; post-boot checks green (nvidia/DKMS, 54 containers, 18 Prom targets, 0 failed units, /mnt/media). Documents the gluetun caveat: a stale PIA WG peer does not survive a reboot -> run pia-wg/regen-and-up.sh as a required post-reboot step. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-02 00:01:48 -05:00
tommy	3746f3fe7c	monitoring: capture compute6 Diun compose (closes audit visibility gap) Deployed 2026-06-30 to close compute6's image-update blind spot; was running on compute6 but not tracked. Includes the required security_opt apparmor=unconfined (compute6 host norm) without which Diun can't read the docker socket. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-30 22:44:42 -05:00
tommy	0dea324551	traefik: de-secret CrowdSec LAPI key -> crowdsecLapiKeyFile (rotated) Old bouncer key was committed in cleartext (hist `7fac4fc`) and is now rotated (new traefik-bouncer-v2 + old bouncer deleted, verified 200 on both nodes). Config now references /plugins-storage/crowdsec_lapi_key (mode-600, gitignored, lives only on the nodes). Old key in history is inert post-rotation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-30 22:41:35 -05:00
tommy	0f3973023a	gluetun-pia: CA-pinned regen + CA-aware region auto-pick (fix legacy-CN break) Replaces the Go pia-wg-config (Go 1.23 rejects PIA legacy-CN no-SAN certs) with PIA's official CA-pinned curl flow. Adds CA-aware region auto-pick so ExpressVPN-CA migrated regions (e.g. us_chicago) auto-skip; region moved to us_denver. Tunnel restored + egress verified 2026-06-30. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-30 22:01:16 -05:00
tommy	6af09ef9c6	monitoring: off-host Prometheus dead-man's-switch watchdog (Pi .227) Independent-hardware systemd timer on raspberrypi (.227) probing media-server Prometheus /-/healthy every 60s; pages ntfy prometheus-critical after 3 fails and on recovery. Catches media-server-dark, which the same-host Uptime-Kuma monitor structurally cannot. Deployed + live-outage tested 2026-06-30. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-30 21:58:57 -05:00