runbooks: RB-002 total Traefik 403 — CrowdSec SQLite journal lock + WAL fix
All services behind Traefik on docker-node01 (.186) returned HTTP 403 for ~7
minutes (22:02–22:11 CDT 2026-05-23) due to CrowdSec SQLite database lock.
Root cause: SQLite delete-journal mode serializes readers under write lock.
After 27.5 h of operation (8.2 MB fragmented DB), CrowdSec hourly maintenance
tasks (metrics flush, orphan event purge, allowlist flush) fired simultaneously,
holding the exclusive write lock long enough to time out all concurrent bouncer
reads. Traefik bouncer plugin fails-closed → HTTP 403 on every proxied service.
Fix: stopped CrowdSec, switched DB to WAL mode (PRAGMA journal_mode=WAL) +
VACUUM (8.2 MB → 3.5 MB), restarted. WAL mode persists in DB file header.
Runbook covers: Cloudflare passthrough mislead, --resolve bypass diagnosis,
LAPI health check, journal file detection, step-by-step recovery, trigger
investigation methodology, and blast-radius note on entrypoint-level middleware.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
tommy
8a734c638a