traefik: nextcloud-transport (3600s responseHeaderTimeout) — fix large-upload 504
Real root cause of Nextcloud "Unknown error during upload" on large files: Traefik's default-transport responseHeaderTimeout=30s killed the chunk-assembly MOVE (which takes >30s for multi-GB files) -> 504. Added a dedicated nextcloud-transport (3600s) and pointed nextcloud-service at it, on both nodes. Proven: 8GB upload MOVE now 201 @ 31.7s (was 504 @ 30.0s). Rewrote the runbook: corrected cause (was mis-attributed to Cloudflare/DoH), plus the hard-won gotcha on editing single-file-bind-mounted dynamic_conf (in-place cp only; never mv; validate before write). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
tommy
@@ -1,63 +1,46 @@
|
||||
# Nextcloud large-file upload fix (CT103) — "Unknown error during upload"
|
||||
|
||||
**Date:** 2026-07-05
|
||||
**Symptom:** Uploading large movie files via the Nextcloud **web UI** fails with *"Unknown error during upload"* (small files fine).
|
||||
**TL;DR root cause:** **Browser DNS-over-HTTPS (Secure DNS).** The browser bypasses LAN DNS, resolves `cloud.goattw.net` via Cloudflare, and Cloudflare mangles the parallel chunk stream. **Not** a size limit, **not** the server, **not** DNS zone config.
|
||||
**TL;DR fix:** Disable browser DoH on the upload machines (see `.reg` below). Nothing server-side to change.
|
||||
**Symptom:** Uploading large movie files via the Nextcloud web UI fails with *"Unknown error during upload"* (small files fine).
|
||||
**ROOT CAUSE:** **Traefik 30 s `responseHeaderTimeout`.** Large files upload in chunks, then a final `MOVE` **assembles** them server-side — for a multi-GB movie that takes **>30 s** (read+write the whole file over NFS). Traefik's `default-transport` had `responseHeaderTimeout: 30s`, so it 504'd the MOVE at 30 s → browser "Unknown error." Nextcloud's log shows *"Could not open file: …/<chunk>, doesn't seem to exist"* — that's the *symptom* (assembly stream cut off mid-read), not the cause.
|
||||
**FIX:** dedicated `nextcloud-transport` serversTransport with `responseHeaderTimeout: 3600s`, and point `nextcloud-service` at it. Applied to both Traefik nodes.
|
||||
|
||||
> **NOTE / red herring:** I initially chased **Cloudflare / browser DoH** (Cloudflare *is* in front at `cloud.goattw.net`, CF caps bodies at ~100 MB). That was WRONG — the failing client was on the LAN direct (`ClientHost 192.168.99.238` in Traefik's access log), and the kill was a clean `504 ... 30000ms`. DoH-disable `.reg` (in git history of this file) is harmless but was not the fix. Lesson: **read the proxy access log first** — the `504 @ 30000ms` was the whole answer.
|
||||
|
||||
---
|
||||
|
||||
## Setup (so future-you knows the layout)
|
||||
- Nextcloud = **AIO (All-In-One)**, running **Docker inside LXC CT103** on Beast (.200). App container `nextcloud-aio-nextcloud` (PHP-FPM), front `nextcloud-aio-apache` (:11000), managed by `nextcloud-aio-mastercontainer` (:8080). Compose: `/root/nextcloud-aio/docker-compose.yml`. NC v32.
|
||||
- **Access CT103:** `.103` is NOT LAN-reachable / no direct SSH. Use: `ssh tommy@192.168.99.200` then **`sudo pct exec 103 -- ...`** (`sudo pct` is NOPASSWD on Beast). occ = `docker exec -u www-data nextcloud-aio-nextcloud php /var/www/html/occ ...`.
|
||||
- Data dir `/mnt/ncdata` = NFS from TrueNAS (.29), 17T. Fronted at `cloud.goattw.net` via **Cloudflare → Traefik (.186) → NC**.
|
||||
## Setup
|
||||
- Nextcloud = **AIO**, Docker inside **LXC CT103** on Beast (.200). Access: `ssh tommy@192.168.99.200` → **`sudo pct exec 103 -- ...`** (`sudo pct` NOPASSWD on Beast). occ = `docker exec -u www-data nextcloud-aio-nextcloud php /var/www/html/occ ...`.
|
||||
- Public `cloud.goattw.net` → Cloudflare → Traefik. LAN `cloud.goattw.net` → Technitium/UniFi return **192.168.99.185** (internal Traefik, LE cert) → Traefik (.186 active / .187 backup) → backend **http://192.168.99.31:11000** (aio-apache). Data dir `/mnt/ncdata` = NFS TrueNAS (.29). NC v32.
|
||||
- Size limits were never the issue — already 16 G; AIO manages php.ini via env, don't hand-edit.
|
||||
|
||||
## Why it fails (the evidence chain)
|
||||
1. **Size limits were never the issue** — already 16G (PHP upload_max_filesize/post_max_size=16G, APACHE_MAX_SIZE=16G, max_execution_time=3600). In AIO you don't edit php.ini; it's env-managed by the mastercontainer.
|
||||
2. **Real error** (nextcloud.log): chunked-upload **assembly** fails — `MOVE …/uploads/…/.file "Could not open file …/<N>, file doesn't seem to exist"` (Sabre ServiceUnavailable / AssemblyStream.php). A chunk is missing at assembly.
|
||||
3. **Backend + Traefik are fine** — proven by a browser-like parallel(5) 2GB/205×10MiB chunked upload: succeeds straight to the backend (`localhost:11000`) AND through Traefik (`.185` and `.186`, Cloudflare bypassed via `curl --resolve`). 0 failures, exact assembly, every time.
|
||||
4. **Cloudflare is the culprit.** `cloud.goattw.net` public record → Cloudflare (2606:4700::/32). CF body-limit probe: 100MB→passes, 105MB→413. With 10MiB chunks the *size* isn't the trip — CF drops/errors a parallel chunk PUT in transit → browser doesn't retry → assembly finds it missing.
|
||||
5. **DNS is already split-horizon-correct.** Technitium (.184) AND the UniFi gateway (.1) both answer `cloud.goattw.net` A → **192.168.99.185** (internal Traefik, valid Let's Encrypt cert, >100MB passes). So any client using LAN DNS gets the good internal path.
|
||||
6. **So why does it still fail?** The browser uses **DoH (Secure DNS)** → ignores LAN DNS/UniFi/Technitium → asks Cloudflare directly → takes the broken CF path. `nslookup cloud.goattw.net` shows `.185` (OS is fine) while Chrome still hits Cloudflare = DoH red-handed.
|
||||
|
||||
## THE FIX — disable browser DoH (machine-wide, locks it off)
|
||||
Save as `disable-doh.reg`, run as admin on each upload machine (or push via GPO/RMM):
|
||||
|
||||
```reg
|
||||
Windows Registry Editor Version 5.00
|
||||
|
||||
[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Google\Chrome]
|
||||
"DnsOverHttpsMode"="off"
|
||||
|
||||
[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Edge]
|
||||
"DnsOverHttpsMode"="off"
|
||||
|
||||
[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\BraveSoftware\Brave]
|
||||
"DnsOverHttpsMode"="off"
|
||||
|
||||
[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Mozilla\Firefox\DNSOverHTTPS]
|
||||
"Enabled"=dword:00000000
|
||||
"Locked"=dword:00000001
|
||||
## The fix (in `traefik/dynamic_conf.yml` on .186 AND .187)
|
||||
Add a transport next to `default-transport`:
|
||||
```yaml
|
||||
nextcloud-transport:
|
||||
forwardingTimeouts:
|
||||
dialTimeout: 30s
|
||||
responseHeaderTimeout: 3600s # was 30s on default-transport → killed assembly
|
||||
idleConnTimeout: 90s
|
||||
```
|
||||
|
||||
PowerShell (as admin):
|
||||
```powershell
|
||||
'Google\Chrome','Microsoft\Edge','BraveSoftware\Brave' | ForEach-Object {
|
||||
$k = "HKLM:\SOFTWARE\Policies\$_"; New-Item $k -Force | Out-Null
|
||||
Set-ItemProperty $k -Name DnsOverHttpsMode -Value 'off' -Type String }
|
||||
$ff='HKLM:\SOFTWARE\Policies\Mozilla\Firefox\DNSOverHTTPS'; New-Item $ff -Force | Out-Null
|
||||
Set-ItemProperty $ff -Name Enabled -Value 0 -Type DWord
|
||||
Set-ItemProperty $ff -Name Locked -Value 1 -Type DWord
|
||||
Point the service at it:
|
||||
```yaml
|
||||
nextcloud-service:
|
||||
loadBalancer:
|
||||
servers: [{ url: "http://192.168.99.31:11000" }]
|
||||
serversTransport: nextcloud-transport # was: default-transport
|
||||
```
|
||||
Mirrored in this repo: `traefik/dynamic_conf_node01.yml` (.186) and `dynamic_conf_node02.yml` (.187).
|
||||
|
||||
GPO equivalents (domain-wide): Chrome *DNS-over-HTTPS mode = off*; Edge *Configure DNS-over-HTTPS = Disabled*; Firefox *DNS Over HTTPS Enabled=false, Locked=true*.
|
||||
## ⚠️ How to edit Traefik dynamic_conf WITHOUT breaking it (hard-won)
|
||||
`dynamic_conf.yml` is a **single-file bind mount** into the Traefik container. Therefore:
|
||||
- **Edit IN-PLACE only** (`cp tmp dynamic_conf.yml`, `sed -i`) — keeps the same inode, Traefik hot-reloads, no restart. This is how .187 was done.
|
||||
- **NEVER `mv` a new file over it.** `mv` swaps the inode → the container stays pinned to the OLD inode (serves stale config, silently) → only a **container restart** re-links it. This is how .186 got stuck (had to `docker restart traefik`).
|
||||
- **Don't stream-write it** (`base64 -d > file`, slow redirects) — Traefik's watcher can read a half-written file → `yaml: could not find expected ':'` parse error → **it drops the WHOLE file config** (all `@file` middlewares: authelia, crowdsec, secure-headers vanish). Write to a temp, validate (`python3 -c 'import yaml;yaml.safe_load(open("f"))'`), then `cp` in-place.
|
||||
- After any edit, verify: `curl -s http://<node>:8080/api/rawdata` → `services.nextcloud-service@file.loadBalancer.serversTransport` == `nextcloud-transport@file`, and `middlewares` count unchanged (8).
|
||||
|
||||
**After:** fully restart the browser → verify at `chrome://policy` / `edge://policy` / `about:policies` (Firefox) → retry the big upload. It now rides the internal `.185` path, no Cloudflare.
|
||||
## Verification (done)
|
||||
8 GB incompressible (`/dev/urandom`) chunked upload straight through Traefik: **MOVE = 201, assembly_time = 31.7 s**, exact size. Old config 504'd the identical test at 30.0 s. ✅
|
||||
|
||||
## Alternatives (not chosen)
|
||||
- **Nextcloud desktop/WebDAV client** — retries dropped chunks, works through CF as-is (good interim, no config).
|
||||
- **Grey-cloud `cloud.goattw.net`** (DNS-only in Cloudflare) — fixes every client incl. DoH, but exposes origin + drops CF WAF/DDoS. Also may break if CF ingress is a Tunnel. Not done.
|
||||
- Raising `max_chunk_size` — MISFIRE (100MiB sits on CF's edge). Reverted; NC left at stock 10MiB default.
|
||||
|
||||
## Gotcha
|
||||
UniFi DHCP was already handing out a resolver that returns `.185` — the DNS side needed **no change**. The only lever is the browser. DoH is the recurring trap: it silently overrides all network DNS.
|
||||
## If large files STILL fail after this
|
||||
Next timeouts to check, in order: aio-apache `APACHE_MAX_TIME` (already 3600), PHP `max_execution_time` (3600), Cloudflare (only relevant for *external* clients — its ~100 s proxy timeout + 100 MB body cap; use the internal `.185` path or the desktop client from outside).
|
||||
|
||||
@@ -8,6 +8,11 @@ http:
|
||||
idleConnTimeout: 90s
|
||||
insecure-transport:
|
||||
insecureSkipVerify: true
|
||||
nextcloud-transport:
|
||||
forwardingTimeouts:
|
||||
dialTimeout: 30s
|
||||
responseHeaderTimeout: 3600s
|
||||
idleConnTimeout: 90s
|
||||
|
||||
middlewares:
|
||||
|
||||
@@ -463,7 +468,7 @@ http:
|
||||
nextcloud-service:
|
||||
loadBalancer:
|
||||
servers: [{ url: "http://192.168.99.31:11000" }]
|
||||
serversTransport: default-transport
|
||||
serversTransport: nextcloud-transport
|
||||
|
||||
homeassistant-service:
|
||||
loadBalancer:
|
||||
|
||||
@@ -8,6 +8,11 @@ http:
|
||||
idleConnTimeout: 90s
|
||||
insecure-transport:
|
||||
insecureSkipVerify: true
|
||||
nextcloud-transport:
|
||||
forwardingTimeouts:
|
||||
dialTimeout: 30s
|
||||
responseHeaderTimeout: 3600s
|
||||
idleConnTimeout: 90s
|
||||
|
||||
middlewares:
|
||||
|
||||
@@ -463,7 +468,7 @@ http:
|
||||
nextcloud-service:
|
||||
loadBalancer:
|
||||
servers: [{ url: "http://192.168.99.31:11000" }]
|
||||
serversTransport: default-transport
|
||||
serversTransport: nextcloud-transport
|
||||
|
||||
homeassistant-service:
|
||||
loadBalancer:
|
||||
|
||||
Reference in New Issue
Block a user