runbooks: RB-002 total Traefik 403 — CrowdSec SQLite journal lock + WAL fix

All services behind Traefik on docker-node01 (.186) returned HTTP 403 for ~7 minutes (22:02–22:11 CDT 2026-05-23) due to CrowdSec SQLite database lock. Root cause: SQLite delete-journal mode serializes readers under write lock. After 27.5 h of operation (8.2 MB fragmented DB), CrowdSec hourly maintenance tasks (metrics flush, orphan event purge, allowlist flush) fired simultaneously, holding the exclusive write lock long enough to time out all concurrent bouncer reads. Traefik bouncer plugin fails-closed → HTTP 403 on every proxied service. Fix: stopped CrowdSec, switched DB to WAL mode (PRAGMA journal_mode=WAL) + VACUUM (8.2 MB → 3.5 MB), restarted. WAL mode persists in DB file header. Runbook covers: Cloudflare passthrough mislead, --resolve bypass diagnosis, LAPI health check, journal file detection, step-by-step recovery, trigger investigation methodology, and blast-radius note on entrypoint-level middleware. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-23 22:23:16 -05:00
parent e61fa8fe60
commit 8a734c638a
1 changed files with 130 additions and 1 deletions
@@ -1,6 +1,6 @@
 # Phase 5 — Incident Log / State Drift

-Last updated: 2026-05-23 (RB-001: VAULT1 root cause + monitor added; HAOS NUT addon installed and verified)
+Last updated: 2026-05-24 (RB-001: VAULT1 root cause + monitor added; RB-002: Total Traefik 403 outage — CrowdSec SQLite journal lock + WAL mode fix)
 Purpose: Consolidates real findings from the Phase 4 worksession. Inputs to Phase 5 planning and repair.

 ---
@@ -418,3 +418,132 @@ Orphan dirs on sda8 are outside Frigate's media path — handle manually (move o
 - Uptime Kuma "Home Assistant" monitor (http://192.168.99.100:8123, 60s) → ntfy `grafana-alerts` (added 2026-05-23)
 - USB enclosure is self-powered (own PSU) and directly connected — no hub to replace. If repeated disconnects occur, suspect: (a) PSU/power cable on enclosure, (b) USB cable quality, (c) motherboard USB port.
 - **HAOS NUT addon** (`a0d7b954_nut`, installed 2026-05-23): netclient mode, monitors `cyberpower1@192.168.99.200:3493` as secondary. Credentials: `upsuser / Sparky$100` (same as CT104 slave). `shutdown_host: true` → runs `/usr/bin/shutdownhost` on FSD. NUT chain is now: Beast (master) → CT104 (secondary) + HAOS (secondary). Verified: `upsc cyberpower1@192.168.99.200` returns full UPS data (OL, 36% load, battery 100%); Beast `upsc -c cyberpower1` lists 192.168.99.100 as a connected client. No `test.shutdown` available on CP1500PFCLCDa firmware — live FSD test skipped (would cut UPS load).
+
+
+---
+
+### RB-002 — Total Traefik Proxy 403 Outage: CrowdSec SQLite Journal Lock
+
+**Last triggered:** 2026-05-24 ~03:02 UTC (22:02 CDT) — ~7 min outage, all services behind Traefik on docker-node01 (.186) returning 403. Resolved 03:11 UTC.
+
+#### Symptom
+- **All** services proxied by Traefik on `.186` return HTTP 403 with empty body and ~10-second response time
+- Browser shows "Access Denied"; `curl -I https://home.goattw.net` returns `HTTP/2 403` with `server: cloudflare` (misleading — Cloudflare is passing the 403 through from the origin, not generating it)
+- Traefik access log shows `403 0 … 10000ms` or `10001ms` on **every** router (homepage, grafana, immich, sabnzbd, smokeping, etc.)
+- CrowdSec container shows `Up` and healthy; no obvious container crash
+
+#### Why it looks like a Cloudflare block
+The `server: cloudflare` header appears on all responses because Cloudflare proxies the domain. When the origin returns 403, Cloudflare passes the status through while adding its own headers. A bare `content-length: 0` body (no CF HTML error page) is the tell that this is an upstream 403, not a CF WAF/Access block.
+
+#### Diagnosis steps
+```bash
+# 1. Confirm 403 is from origin, not Cloudflare (bypass CF proxy)
+curl -sI --resolve "home.goattw.net:443:192.168.99.186" https://home.goattw.net
+# Expect: same 403 → origin is the source. Different result → CF-layer issue.
+
+# 2. Check Traefik access log for the CrowdSec timeout signature (403 + ~10s response)
+sudo tail -20 /home/tommy/traefik/logs/access.log
+# Pattern to look for: "403 0" entries with "10000ms" or "10001ms" duration on EVERY router
+
+# 3. Check Traefik container logs for middleware errors
+docker logs traefik --tail 20 2>&1
+# Look for: middleware "crowdsec-bouncer@file" does not exist  (rare startup race, not the lock issue)
+# Look for: Error calling http://authelia:9091 (Authelia connectivity — separate issue)
+
+# 4. Test CrowdSec LAPI directly — is it responding?
+curl -s -o /dev/null -w "HTTP %{http_code}\n" \
+  -H "X-Api-Key: VMCnws/j+9pmsT4YT+t3HzrvX8OhBCwoquwo4NqWJPs" \
+  "http://localhost:8081/v1/decisions?ip=1.2.3.4"
+# Healthy: HTTP 200 in <50ms.  Locked: hangs for seconds then HTTP 403.
+
+# 5. Check CrowdSec logs for database lock errors
+docker logs crowdsec --tail 30 2>&1 | grep -E "locked|error|warn"
+# Signature: "database is locked" on fetching bouncer info, flushing metrics,
+#            deleting orphan events, flushing allowlists
+
+# 6. Inspect the data directory for a stale journal file
+ls -lh /home/tommy/crowdsec/data/crowdsec.db*
+# Stale lock: crowdsec.db-journal exists alongside crowdsec.db
+# (WAL mode post-fix: crowdsec.db-wal and .db-shm may appear briefly during writes — normal)
+
+# 7. Verify current journal mode
+sudo sqlite3 /home/tommy/crowdsec/data/crowdsec.db "PRAGMA journal_mode;"
+# Post-fix: should return "wal"
+# Pre-fix (vulnerable state): returns "delete"
+```
+
+#### Root cause
+CrowdSec's SQLite database in **`delete` (rollback) journal mode** holds an exclusive write lock for the entire duration of any write transaction. After ~27 hours of continuous operation, the database had grown to 8.2MB with significant fragmentation. CrowdSec's periodic maintenance tasks — `flushing metrics`, `deleting orphan events`, and `flushing allowlists` — all fired simultaneously at the top of the hour (03:00 UTC / 22:00 CDT), collectively acquiring and holding the exclusive write lock long enough to time out all concurrent bouncer reads. The Traefik CrowdSec bouncer plugin fails-closed on non-200 LAPI responses, returning HTTP 403 to Traefik, which passed 403s to every client.
+
+**Timeline (2026-05-24):**
+- `18:36 CDT May 22` — Docker bulk update completed; CrowdSec container started fresh on `.186`
+- `22:00 CDT May 23` — CrowdSec's hourly maintenance tasks fired; `cert-expiry-check.sh` cron also ran (reads `acme.json` locally, **makes no HTTP requests** — coincidence, not a contributor)
+- `22:02 CDT May 23` — First `database is locked` errors in CrowdSec LAPI log (27.5 h after container start)
+- `22:09 CDT May 23` — `crowdsec.db-journal` last modified; DB had stopped accepting successful writes at 22:00
+- `22:10 CDT May 23` — Diagnosed and CrowdSec stopped for repair
+- `22:11 CDT May 23` — CrowdSec restarted; LAPI responding at 6–8 ms; all services restored
+
+**Trigger investigation:** Checked `journalctl -u docker` for the 22:00–22:03 CDT window — no container restarts, no OOM events, no unexpected docker events. Scanner traffic (172.184.210.34) had zero hits in the two hours preceding the lock. No other cron jobs made HTTP requests at that time. The cause was entirely internal to CrowdSec: `delete`-mode SQLite contention under its own scheduled maintenance workload on a 27-hour-old, 8.2MB fragmented database.
+
+#### Recovery
+```bash
+# Step 1: Back up the database before touching anything
+sudo cp /home/tommy/crowdsec/data/crowdsec.db \
+  /home/tommy/crowdsec/data/crowdsec.db.bak.$(date +%Y%m%d_%H%M%S)
+
+# Step 2: Stop CrowdSec to release all DB locks
+docker stop crowdsec
+
+# Step 3: Verify integrity (must return "ok" before proceeding)
+sudo sqlite3 /home/tommy/crowdsec/data/crowdsec.db "PRAGMA integrity_check;"
+
+# Step 4: Switch to WAL mode and compact
+# WAL mode: allows concurrent readers while writes are in-flight — eliminates the lock contention
+# VACUUM: compacts fragmentation (DB was 8.2 MB → 3.5 MB after vacuum on 2026-05-24)
+sudo sqlite3 /home/tommy/crowdsec/data/crowdsec.db "
+PRAGMA journal_mode=WAL;
+VACUUM;
+PRAGMA wal_checkpoint(TRUNCATE);
+PRAGMA journal_mode;
+"
+# Expected output: wal / 0|0|0 / wal
+# The stale .db-journal file will be consumed and removed automatically
+
+# Step 5: Start CrowdSec
+docker start crowdsec
+sleep 8   # wait for LAPI init
+
+# Step 6: Verify LAPI is healthy
+curl -s -o /dev/null -w "HTTP %{http_code}\n" \
+  -H "X-Api-Key: VMCnws/j+9pmsT4YT+t3HzrvX8OhBCwoquwo4NqWJPs" \
+  "http://localhost:8081/v1/decisions?ip=1.2.3.4"
+# Expect: HTTP 200 in <50 ms. If still 403 or slow, repeat from Step 2.
+
+# Step 7: Confirm end-to-end fix
+curl -sI https://home.goattw.net | head -3
+# Expect: HTTP/2 200
+```
+
+**If `PRAGMA integrity_check` returns anything other than `ok`:**
+The database is corrupted. Restore from backup or let CrowdSec reinitialize (decisions and alerts will be lost):
+```bash
+docker stop crowdsec
+sudo rm /home/tommy/crowdsec/data/crowdsec.db /home/tommy/crowdsec/data/crowdsec.db-journal
+docker start crowdsec
+# CrowdSec will create a fresh empty DB on startup
+# Re-register bouncer key after restart if needed
+```
+
+#### Recovery time
+~3 minutes (stop → repair → start → verified).
+
+#### Prevention / monitoring
+- **WAL mode is now persistent** — the journal mode is stored in the DB file header and survives CrowdSec container restarts. After any future CrowdSec container recreation (`docker compose up -d` with image update), verify:
+  ```bash
+  sudo sqlite3 /home/tommy/crowdsec/data/crowdsec.db "PRAGMA journal_mode;"
+  # Must return: wal
+  ```
+  If it returns `delete`, re-run Step 4 above before the next high-traffic period.
+- **Backup cleanup:** Remove `crowdsec.db.bak.*` files after 48 h of stable operation.
+- The `crowdsec.db-wal` and `crowdsec.db-shm` files that appear with WAL mode are normal — do **not** delete them while CrowdSec is running. They will be cleaned up on next checkpoint.
+- **Root-cause prevention:** WAL mode eliminates the delete-journal write-lock contention. No additional rate limiting or CrowdSec configuration change is needed. The `--entrypoints.websecure.http.middlewares=crowdsec-bouncer@file` global setting in Traefik's compose is intentional (all-entrypoint protection) but means a CrowdSec LAPI failure is a **total outage** — consider adding `crowdsecLapiTimeout` to the bouncer config or setting a shorter timeout if reduced blast radius is desired in future.