From 5c66bbacdd6d299f068a5019633d346a9ae9c2a5 Mon Sep 17 00:00:00 2001 From: tommy Date: Sun, 5 Jul 2026 19:07:04 -0500 Subject: [PATCH] =?UTF-8?q?traefik:=20nextcloud-transport=20(3600s=20respo?= =?UTF-8?q?nseHeaderTimeout)=20=E2=80=94=20fix=20large-upload=20504?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Real root cause of Nextcloud "Unknown error during upload" on large files: Traefik's default-transport responseHeaderTimeout=30s killed the chunk-assembly MOVE (which takes >30s for multi-GB files) -> 504. Added a dedicated nextcloud-transport (3600s) and pointed nextcloud-service at it, on both nodes. Proven: 8GB upload MOVE now 201 @ 31.7s (was 504 @ 30.0s). Rewrote the runbook: corrected cause (was mis-attributed to Cloudflare/DoH), plus the hard-won gotcha on editing single-file-bind-mounted dynamic_conf (in-place cp only; never mv; validate before write). Co-Authored-By: Claude Opus 4.8 --- runbooks/nextcloud-large-upload-fix.md | 85 +++++++++++--------------- traefik/dynamic_conf_node01.yml | 7 ++- traefik/dynamic_conf_node02.yml | 7 ++- 3 files changed, 46 insertions(+), 53 deletions(-) diff --git a/runbooks/nextcloud-large-upload-fix.md b/runbooks/nextcloud-large-upload-fix.md index d6ddd69..ecdedb8 100644 --- a/runbooks/nextcloud-large-upload-fix.md +++ b/runbooks/nextcloud-large-upload-fix.md @@ -1,63 +1,46 @@ # Nextcloud large-file upload fix (CT103) — "Unknown error during upload" **Date:** 2026-07-05 -**Symptom:** Uploading large movie files via the Nextcloud **web UI** fails with *"Unknown error during upload"* (small files fine). -**TL;DR root cause:** **Browser DNS-over-HTTPS (Secure DNS).** The browser bypasses LAN DNS, resolves `cloud.goattw.net` via Cloudflare, and Cloudflare mangles the parallel chunk stream. **Not** a size limit, **not** the server, **not** DNS zone config. -**TL;DR fix:** Disable browser DoH on the upload machines (see `.reg` below). Nothing server-side to change. +**Symptom:** Uploading large movie files via the Nextcloud web UI fails with *"Unknown error during upload"* (small files fine). +**ROOT CAUSE:** **Traefik 30 s `responseHeaderTimeout`.** Large files upload in chunks, then a final `MOVE` **assembles** them server-side — for a multi-GB movie that takes **>30 s** (read+write the whole file over NFS). Traefik's `default-transport` had `responseHeaderTimeout: 30s`, so it 504'd the MOVE at 30 s → browser "Unknown error." Nextcloud's log shows *"Could not open file: …/, doesn't seem to exist"* — that's the *symptom* (assembly stream cut off mid-read), not the cause. +**FIX:** dedicated `nextcloud-transport` serversTransport with `responseHeaderTimeout: 3600s`, and point `nextcloud-service` at it. Applied to both Traefik nodes. + +> **NOTE / red herring:** I initially chased **Cloudflare / browser DoH** (Cloudflare *is* in front at `cloud.goattw.net`, CF caps bodies at ~100 MB). That was WRONG — the failing client was on the LAN direct (`ClientHost 192.168.99.238` in Traefik's access log), and the kill was a clean `504 ... 30000ms`. DoH-disable `.reg` (in git history of this file) is harmless but was not the fix. Lesson: **read the proxy access log first** — the `504 @ 30000ms` was the whole answer. --- -## Setup (so future-you knows the layout) -- Nextcloud = **AIO (All-In-One)**, running **Docker inside LXC CT103** on Beast (.200). App container `nextcloud-aio-nextcloud` (PHP-FPM), front `nextcloud-aio-apache` (:11000), managed by `nextcloud-aio-mastercontainer` (:8080). Compose: `/root/nextcloud-aio/docker-compose.yml`. NC v32. -- **Access CT103:** `.103` is NOT LAN-reachable / no direct SSH. Use: `ssh tommy@192.168.99.200` then **`sudo pct exec 103 -- ...`** (`sudo pct` is NOPASSWD on Beast). occ = `docker exec -u www-data nextcloud-aio-nextcloud php /var/www/html/occ ...`. -- Data dir `/mnt/ncdata` = NFS from TrueNAS (.29), 17T. Fronted at `cloud.goattw.net` via **Cloudflare → Traefik (.186) → NC**. +## Setup +- Nextcloud = **AIO**, Docker inside **LXC CT103** on Beast (.200). Access: `ssh tommy@192.168.99.200` → **`sudo pct exec 103 -- ...`** (`sudo pct` NOPASSWD on Beast). occ = `docker exec -u www-data nextcloud-aio-nextcloud php /var/www/html/occ ...`. +- Public `cloud.goattw.net` → Cloudflare → Traefik. LAN `cloud.goattw.net` → Technitium/UniFi return **192.168.99.185** (internal Traefik, LE cert) → Traefik (.186 active / .187 backup) → backend **http://192.168.99.31:11000** (aio-apache). Data dir `/mnt/ncdata` = NFS TrueNAS (.29). NC v32. +- Size limits were never the issue — already 16 G; AIO manages php.ini via env, don't hand-edit. -## Why it fails (the evidence chain) -1. **Size limits were never the issue** — already 16G (PHP upload_max_filesize/post_max_size=16G, APACHE_MAX_SIZE=16G, max_execution_time=3600). In AIO you don't edit php.ini; it's env-managed by the mastercontainer. -2. **Real error** (nextcloud.log): chunked-upload **assembly** fails — `MOVE …/uploads/…/.file "Could not open file …/, file doesn't seem to exist"` (Sabre ServiceUnavailable / AssemblyStream.php). A chunk is missing at assembly. -3. **Backend + Traefik are fine** — proven by a browser-like parallel(5) 2GB/205×10MiB chunked upload: succeeds straight to the backend (`localhost:11000`) AND through Traefik (`.185` and `.186`, Cloudflare bypassed via `curl --resolve`). 0 failures, exact assembly, every time. -4. **Cloudflare is the culprit.** `cloud.goattw.net` public record → Cloudflare (2606:4700::/32). CF body-limit probe: 100MB→passes, 105MB→413. With 10MiB chunks the *size* isn't the trip — CF drops/errors a parallel chunk PUT in transit → browser doesn't retry → assembly finds it missing. -5. **DNS is already split-horizon-correct.** Technitium (.184) AND the UniFi gateway (.1) both answer `cloud.goattw.net` A → **192.168.99.185** (internal Traefik, valid Let's Encrypt cert, >100MB passes). So any client using LAN DNS gets the good internal path. -6. **So why does it still fail?** The browser uses **DoH (Secure DNS)** → ignores LAN DNS/UniFi/Technitium → asks Cloudflare directly → takes the broken CF path. `nslookup cloud.goattw.net` shows `.185` (OS is fine) while Chrome still hits Cloudflare = DoH red-handed. - -## THE FIX — disable browser DoH (machine-wide, locks it off) -Save as `disable-doh.reg`, run as admin on each upload machine (or push via GPO/RMM): - -```reg -Windows Registry Editor Version 5.00 - -[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Google\Chrome] -"DnsOverHttpsMode"="off" - -[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Edge] -"DnsOverHttpsMode"="off" - -[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\BraveSoftware\Brave] -"DnsOverHttpsMode"="off" - -[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Mozilla\Firefox\DNSOverHTTPS] -"Enabled"=dword:00000000 -"Locked"=dword:00000001 +## The fix (in `traefik/dynamic_conf.yml` on .186 AND .187) +Add a transport next to `default-transport`: +```yaml + nextcloud-transport: + forwardingTimeouts: + dialTimeout: 30s + responseHeaderTimeout: 3600s # was 30s on default-transport → killed assembly + idleConnTimeout: 90s ``` - -PowerShell (as admin): -```powershell -'Google\Chrome','Microsoft\Edge','BraveSoftware\Brave' | ForEach-Object { - $k = "HKLM:\SOFTWARE\Policies\$_"; New-Item $k -Force | Out-Null - Set-ItemProperty $k -Name DnsOverHttpsMode -Value 'off' -Type String } -$ff='HKLM:\SOFTWARE\Policies\Mozilla\Firefox\DNSOverHTTPS'; New-Item $ff -Force | Out-Null -Set-ItemProperty $ff -Name Enabled -Value 0 -Type DWord -Set-ItemProperty $ff -Name Locked -Value 1 -Type DWord +Point the service at it: +```yaml + nextcloud-service: + loadBalancer: + servers: [{ url: "http://192.168.99.31:11000" }] + serversTransport: nextcloud-transport # was: default-transport ``` +Mirrored in this repo: `traefik/dynamic_conf_node01.yml` (.186) and `dynamic_conf_node02.yml` (.187). -GPO equivalents (domain-wide): Chrome *DNS-over-HTTPS mode = off*; Edge *Configure DNS-over-HTTPS = Disabled*; Firefox *DNS Over HTTPS Enabled=false, Locked=true*. +## ⚠️ How to edit Traefik dynamic_conf WITHOUT breaking it (hard-won) +`dynamic_conf.yml` is a **single-file bind mount** into the Traefik container. Therefore: +- **Edit IN-PLACE only** (`cp tmp dynamic_conf.yml`, `sed -i`) — keeps the same inode, Traefik hot-reloads, no restart. This is how .187 was done. +- **NEVER `mv` a new file over it.** `mv` swaps the inode → the container stays pinned to the OLD inode (serves stale config, silently) → only a **container restart** re-links it. This is how .186 got stuck (had to `docker restart traefik`). +- **Don't stream-write it** (`base64 -d > file`, slow redirects) — Traefik's watcher can read a half-written file → `yaml: could not find expected ':'` parse error → **it drops the WHOLE file config** (all `@file` middlewares: authelia, crowdsec, secure-headers vanish). Write to a temp, validate (`python3 -c 'import yaml;yaml.safe_load(open("f"))'`), then `cp` in-place. +- After any edit, verify: `curl -s http://:8080/api/rawdata` → `services.nextcloud-service@file.loadBalancer.serversTransport` == `nextcloud-transport@file`, and `middlewares` count unchanged (8). -**After:** fully restart the browser → verify at `chrome://policy` / `edge://policy` / `about:policies` (Firefox) → retry the big upload. It now rides the internal `.185` path, no Cloudflare. +## Verification (done) +8 GB incompressible (`/dev/urandom`) chunked upload straight through Traefik: **MOVE = 201, assembly_time = 31.7 s**, exact size. Old config 504'd the identical test at 30.0 s. ✅ -## Alternatives (not chosen) -- **Nextcloud desktop/WebDAV client** — retries dropped chunks, works through CF as-is (good interim, no config). -- **Grey-cloud `cloud.goattw.net`** (DNS-only in Cloudflare) — fixes every client incl. DoH, but exposes origin + drops CF WAF/DDoS. Also may break if CF ingress is a Tunnel. Not done. -- Raising `max_chunk_size` — MISFIRE (100MiB sits on CF's edge). Reverted; NC left at stock 10MiB default. - -## Gotcha -UniFi DHCP was already handing out a resolver that returns `.185` — the DNS side needed **no change**. The only lever is the browser. DoH is the recurring trap: it silently overrides all network DNS. +## If large files STILL fail after this +Next timeouts to check, in order: aio-apache `APACHE_MAX_TIME` (already 3600), PHP `max_execution_time` (3600), Cloudflare (only relevant for *external* clients — its ~100 s proxy timeout + 100 MB body cap; use the internal `.185` path or the desktop client from outside). diff --git a/traefik/dynamic_conf_node01.yml b/traefik/dynamic_conf_node01.yml index 7d9873e..0150942 100644 --- a/traefik/dynamic_conf_node01.yml +++ b/traefik/dynamic_conf_node01.yml @@ -8,6 +8,11 @@ http: idleConnTimeout: 90s insecure-transport: insecureSkipVerify: true + nextcloud-transport: + forwardingTimeouts: + dialTimeout: 30s + responseHeaderTimeout: 3600s + idleConnTimeout: 90s middlewares: @@ -463,7 +468,7 @@ http: nextcloud-service: loadBalancer: servers: [{ url: "http://192.168.99.31:11000" }] - serversTransport: default-transport + serversTransport: nextcloud-transport homeassistant-service: loadBalancer: diff --git a/traefik/dynamic_conf_node02.yml b/traefik/dynamic_conf_node02.yml index 3d0a22a..b3664ec 100644 --- a/traefik/dynamic_conf_node02.yml +++ b/traefik/dynamic_conf_node02.yml @@ -8,6 +8,11 @@ http: idleConnTimeout: 90s insecure-transport: insecureSkipVerify: true + nextcloud-transport: + forwardingTimeouts: + dialTimeout: 30s + responseHeaderTimeout: 3600s + idleConnTimeout: 90s middlewares: @@ -463,7 +468,7 @@ http: nextcloud-service: loadBalancer: servers: [{ url: "http://192.168.99.31:11000" }] - serversTransport: default-transport + serversTransport: nextcloud-transport homeassistant-service: loadBalancer: