tommy
5c66bbacdd
Real root cause of Nextcloud "Unknown error during upload" on large files: Traefik's default-transport responseHeaderTimeout=30s killed the chunk-assembly MOVE (which takes >30s for multi-GB files) -> 504. Added a dedicated nextcloud-transport (3600s) and pointed nextcloud-service at it, on both nodes. Proven: 8GB upload MOVE now 201 @ 31.7s (was 504 @ 30.0s). Rewrote the runbook: corrected cause (was mis-attributed to Cloudflare/DoH), plus the hard-won gotcha on editing single-file-bind-mounted dynamic_conf (in-place cp only; never mv; validate before write). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
47 lines
4.2 KiB
Markdown
47 lines
4.2 KiB
Markdown
# Nextcloud large-file upload fix (CT103) — "Unknown error during upload"
|
|
|
|
**Date:** 2026-07-05
|
|
**Symptom:** Uploading large movie files via the Nextcloud web UI fails with *"Unknown error during upload"* (small files fine).
|
|
**ROOT CAUSE:** **Traefik 30 s `responseHeaderTimeout`.** Large files upload in chunks, then a final `MOVE` **assembles** them server-side — for a multi-GB movie that takes **>30 s** (read+write the whole file over NFS). Traefik's `default-transport` had `responseHeaderTimeout: 30s`, so it 504'd the MOVE at 30 s → browser "Unknown error." Nextcloud's log shows *"Could not open file: …/<chunk>, doesn't seem to exist"* — that's the *symptom* (assembly stream cut off mid-read), not the cause.
|
|
**FIX:** dedicated `nextcloud-transport` serversTransport with `responseHeaderTimeout: 3600s`, and point `nextcloud-service` at it. Applied to both Traefik nodes.
|
|
|
|
> **NOTE / red herring:** I initially chased **Cloudflare / browser DoH** (Cloudflare *is* in front at `cloud.goattw.net`, CF caps bodies at ~100 MB). That was WRONG — the failing client was on the LAN direct (`ClientHost 192.168.99.238` in Traefik's access log), and the kill was a clean `504 ... 30000ms`. DoH-disable `.reg` (in git history of this file) is harmless but was not the fix. Lesson: **read the proxy access log first** — the `504 @ 30000ms` was the whole answer.
|
|
|
|
---
|
|
|
|
## Setup
|
|
- Nextcloud = **AIO**, Docker inside **LXC CT103** on Beast (.200). Access: `ssh tommy@192.168.99.200` → **`sudo pct exec 103 -- ...`** (`sudo pct` NOPASSWD on Beast). occ = `docker exec -u www-data nextcloud-aio-nextcloud php /var/www/html/occ ...`.
|
|
- Public `cloud.goattw.net` → Cloudflare → Traefik. LAN `cloud.goattw.net` → Technitium/UniFi return **192.168.99.185** (internal Traefik, LE cert) → Traefik (.186 active / .187 backup) → backend **http://192.168.99.31:11000** (aio-apache). Data dir `/mnt/ncdata` = NFS TrueNAS (.29). NC v32.
|
|
- Size limits were never the issue — already 16 G; AIO manages php.ini via env, don't hand-edit.
|
|
|
|
## The fix (in `traefik/dynamic_conf.yml` on .186 AND .187)
|
|
Add a transport next to `default-transport`:
|
|
```yaml
|
|
nextcloud-transport:
|
|
forwardingTimeouts:
|
|
dialTimeout: 30s
|
|
responseHeaderTimeout: 3600s # was 30s on default-transport → killed assembly
|
|
idleConnTimeout: 90s
|
|
```
|
|
Point the service at it:
|
|
```yaml
|
|
nextcloud-service:
|
|
loadBalancer:
|
|
servers: [{ url: "http://192.168.99.31:11000" }]
|
|
serversTransport: nextcloud-transport # was: default-transport
|
|
```
|
|
Mirrored in this repo: `traefik/dynamic_conf_node01.yml` (.186) and `dynamic_conf_node02.yml` (.187).
|
|
|
|
## ⚠️ How to edit Traefik dynamic_conf WITHOUT breaking it (hard-won)
|
|
`dynamic_conf.yml` is a **single-file bind mount** into the Traefik container. Therefore:
|
|
- **Edit IN-PLACE only** (`cp tmp dynamic_conf.yml`, `sed -i`) — keeps the same inode, Traefik hot-reloads, no restart. This is how .187 was done.
|
|
- **NEVER `mv` a new file over it.** `mv` swaps the inode → the container stays pinned to the OLD inode (serves stale config, silently) → only a **container restart** re-links it. This is how .186 got stuck (had to `docker restart traefik`).
|
|
- **Don't stream-write it** (`base64 -d > file`, slow redirects) — Traefik's watcher can read a half-written file → `yaml: could not find expected ':'` parse error → **it drops the WHOLE file config** (all `@file` middlewares: authelia, crowdsec, secure-headers vanish). Write to a temp, validate (`python3 -c 'import yaml;yaml.safe_load(open("f"))'`), then `cp` in-place.
|
|
- After any edit, verify: `curl -s http://<node>:8080/api/rawdata` → `services.nextcloud-service@file.loadBalancer.serversTransport` == `nextcloud-transport@file`, and `middlewares` count unchanged (8).
|
|
|
|
## Verification (done)
|
|
8 GB incompressible (`/dev/urandom`) chunked upload straight through Traefik: **MOVE = 201, assembly_time = 31.7 s**, exact size. Old config 504'd the identical test at 30.0 s. ✅
|
|
|
|
## If large files STILL fail after this
|
|
Next timeouts to check, in order: aio-apache `APACHE_MAX_TIME` (already 3600), PHP `max_execution_time` (3600), Cloudflare (only relevant for *external* clients — its ~100 s proxy timeout + 100 MB body cap; use the internal `.185` path or the desktop client from outside).
|