homelab-configs/runbooks/nextcloud-large-upload-fix.md

Nextcloud large-file upload fix (CT103) — "Unknown error during upload"

Date: 2026-07-05 Symptom: Uploading large movie files via the Nextcloud web UI fails with "Unknown error during upload" (small files fine). ROOT CAUSE: Traefik 30 s responseHeaderTimeout. Large files upload in chunks, then a final MOVE assembles them server-side — for a multi-GB movie that takes >30 s (read+write the whole file over NFS). Traefik's default-transport had responseHeaderTimeout: 30s, so it 504'd the MOVE at 30 s → browser "Unknown error." Nextcloud's log shows "Could not open file: …/, doesn't seem to exist" — that's the symptom (assembly stream cut off mid-read), not the cause. FIX: dedicated nextcloud-transport serversTransport with responseHeaderTimeout: 3600s, and point nextcloud-service at it. Applied to both Traefik nodes.

NOTE / red herring: I initially chased Cloudflare / browser DoH (Cloudflare is in front at cloud.goattw.net, CF caps bodies at ~100 MB). That was WRONG — the failing client was on the LAN direct (ClientHost 192.168.99.238 in Traefik's access log), and the kill was a clean 504 ... 30000ms. DoH-disable .reg (in git history of this file) is harmless but was not the fix. Lesson: read the proxy access log first — the 504 @ 30000ms was the whole answer.

---

Setup

• Nextcloud = AIO, Docker inside LXC CT103 on Beast (.200). Access: ssh tommy@192.168.99.200 → sudo pct exec 103 -- ... (sudo pct NOPASSWD on Beast). occ = docker exec -u www-data nextcloud-aio-nextcloud php /var/www/html/occ ....
• Public cloud.goattw.net → Cloudflare → Traefik. LAN cloud.goattw.net → Technitium/UniFi return 192.168.99.185 (internal Traefik, LE cert) → Traefik (.186 active / .187 backup) → backend http://192.168.99.31:11000 (aio-apache). Data dir /mnt/ncdata = NFS TrueNAS (.29). NC v32.
• Size limits were never the issue — already 16 G; AIO manages php.ini via env, don't hand-edit.

The fix (in traefik/dynamic_conf.yml on .186 AND .187)

Add a transport next to default-transport:

    nextcloud-transport:
      forwardingTimeouts:
        dialTimeout: 30s
        responseHeaderTimeout: 3600s   # was 30s on default-transport → killed assembly
        idleConnTimeout: 90s

Point the service at it:

    nextcloud-service:
      loadBalancer:
        servers: [{ url: "http://192.168.99.31:11000" }]
        serversTransport: nextcloud-transport   # was: default-transport

Mirrored in this repo: traefik/dynamic_conf_node01.yml (.186) and dynamic_conf_node02.yml (.187).

⚠️ How to edit Traefik dynamic_conf WITHOUT breaking it (hard-won)

dynamic_conf.yml is a single-file bind mount into the Traefik container. Therefore:

• Edit IN-PLACE only (cp tmp dynamic_conf.yml, sed -i) — keeps the same inode, Traefik hot-reloads, no restart. This is how .187 was done.
• NEVER mv a new file over it. mv swaps the inode → the container stays pinned to the OLD inode (serves stale config, silently) → only a container restart re-links it. This is how .186 got stuck (had to docker restart traefik).
• Don't stream-write it (base64 -d > file, slow redirects) — Traefik's watcher can read a half-written file → yaml: could not find expected ':' parse error → it drops the WHOLE file config (all @file middlewares: authelia, crowdsec, secure-headers vanish). Write to a temp, validate (python3 -c 'import yaml;yaml.safe_load(open("f"))'), then cp in-place.
• After any edit, verify: curl -s http://<node>:8080/api/rawdata → services.nextcloud-service@file.loadBalancer.serversTransport == nextcloud-transport@file, and middlewares count unchanged (8).

Verification (done)

8 GB incompressible (/dev/urandom) chunked upload straight through Traefik: MOVE = 201, assembly_time = 31.7 s, exact size. Old config 504'd the identical test at 30.0 s. ✅

If large files STILL fail after this

Next timeouts to check, in order: aio-apache APACHE_MAX_TIME (already 3600), PHP max_execution_time (3600), Cloudflare (only relevant for external clients — its ~100 s proxy timeout + 100 MB body cap; use the internal .185 path or the desktop client from outside).