diff --git a/runbooks/nextcloud-large-upload-fix.md b/runbooks/nextcloud-large-upload-fix.md new file mode 100644 index 0000000..d6ddd69 --- /dev/null +++ b/runbooks/nextcloud-large-upload-fix.md @@ -0,0 +1,63 @@ +# Nextcloud large-file upload fix (CT103) — "Unknown error during upload" + +**Date:** 2026-07-05 +**Symptom:** Uploading large movie files via the Nextcloud **web UI** fails with *"Unknown error during upload"* (small files fine). +**TL;DR root cause:** **Browser DNS-over-HTTPS (Secure DNS).** The browser bypasses LAN DNS, resolves `cloud.goattw.net` via Cloudflare, and Cloudflare mangles the parallel chunk stream. **Not** a size limit, **not** the server, **not** DNS zone config. +**TL;DR fix:** Disable browser DoH on the upload machines (see `.reg` below). Nothing server-side to change. + +--- + +## Setup (so future-you knows the layout) +- Nextcloud = **AIO (All-In-One)**, running **Docker inside LXC CT103** on Beast (.200). App container `nextcloud-aio-nextcloud` (PHP-FPM), front `nextcloud-aio-apache` (:11000), managed by `nextcloud-aio-mastercontainer` (:8080). Compose: `/root/nextcloud-aio/docker-compose.yml`. NC v32. +- **Access CT103:** `.103` is NOT LAN-reachable / no direct SSH. Use: `ssh tommy@192.168.99.200` then **`sudo pct exec 103 -- ...`** (`sudo pct` is NOPASSWD on Beast). occ = `docker exec -u www-data nextcloud-aio-nextcloud php /var/www/html/occ ...`. +- Data dir `/mnt/ncdata` = NFS from TrueNAS (.29), 17T. Fronted at `cloud.goattw.net` via **Cloudflare → Traefik (.186) → NC**. + +## Why it fails (the evidence chain) +1. **Size limits were never the issue** — already 16G (PHP upload_max_filesize/post_max_size=16G, APACHE_MAX_SIZE=16G, max_execution_time=3600). In AIO you don't edit php.ini; it's env-managed by the mastercontainer. +2. **Real error** (nextcloud.log): chunked-upload **assembly** fails — `MOVE …/uploads/…/.file "Could not open file …/, file doesn't seem to exist"` (Sabre ServiceUnavailable / AssemblyStream.php). A chunk is missing at assembly. +3. **Backend + Traefik are fine** — proven by a browser-like parallel(5) 2GB/205×10MiB chunked upload: succeeds straight to the backend (`localhost:11000`) AND through Traefik (`.185` and `.186`, Cloudflare bypassed via `curl --resolve`). 0 failures, exact assembly, every time. +4. **Cloudflare is the culprit.** `cloud.goattw.net` public record → Cloudflare (2606:4700::/32). CF body-limit probe: 100MB→passes, 105MB→413. With 10MiB chunks the *size* isn't the trip — CF drops/errors a parallel chunk PUT in transit → browser doesn't retry → assembly finds it missing. +5. **DNS is already split-horizon-correct.** Technitium (.184) AND the UniFi gateway (.1) both answer `cloud.goattw.net` A → **192.168.99.185** (internal Traefik, valid Let's Encrypt cert, >100MB passes). So any client using LAN DNS gets the good internal path. +6. **So why does it still fail?** The browser uses **DoH (Secure DNS)** → ignores LAN DNS/UniFi/Technitium → asks Cloudflare directly → takes the broken CF path. `nslookup cloud.goattw.net` shows `.185` (OS is fine) while Chrome still hits Cloudflare = DoH red-handed. + +## THE FIX — disable browser DoH (machine-wide, locks it off) +Save as `disable-doh.reg`, run as admin on each upload machine (or push via GPO/RMM): + +```reg +Windows Registry Editor Version 5.00 + +[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Google\Chrome] +"DnsOverHttpsMode"="off" + +[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Edge] +"DnsOverHttpsMode"="off" + +[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\BraveSoftware\Brave] +"DnsOverHttpsMode"="off" + +[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Mozilla\Firefox\DNSOverHTTPS] +"Enabled"=dword:00000000 +"Locked"=dword:00000001 +``` + +PowerShell (as admin): +```powershell +'Google\Chrome','Microsoft\Edge','BraveSoftware\Brave' | ForEach-Object { + $k = "HKLM:\SOFTWARE\Policies\$_"; New-Item $k -Force | Out-Null + Set-ItemProperty $k -Name DnsOverHttpsMode -Value 'off' -Type String } +$ff='HKLM:\SOFTWARE\Policies\Mozilla\Firefox\DNSOverHTTPS'; New-Item $ff -Force | Out-Null +Set-ItemProperty $ff -Name Enabled -Value 0 -Type DWord +Set-ItemProperty $ff -Name Locked -Value 1 -Type DWord +``` + +GPO equivalents (domain-wide): Chrome *DNS-over-HTTPS mode = off*; Edge *Configure DNS-over-HTTPS = Disabled*; Firefox *DNS Over HTTPS Enabled=false, Locked=true*. + +**After:** fully restart the browser → verify at `chrome://policy` / `edge://policy` / `about:policies` (Firefox) → retry the big upload. It now rides the internal `.185` path, no Cloudflare. + +## Alternatives (not chosen) +- **Nextcloud desktop/WebDAV client** — retries dropped chunks, works through CF as-is (good interim, no config). +- **Grey-cloud `cloud.goattw.net`** (DNS-only in Cloudflare) — fixes every client incl. DoH, but exposes origin + drops CF WAF/DDoS. Also may break if CF ingress is a Tunnel. Not done. +- Raising `max_chunk_size` — MISFIRE (100MiB sits on CF's edge). Reverted; NC left at stock 10MiB default. + +## Gotcha +UniFi DHCP was already handing out a resolver that returns `.185` — the DNS side needed **no change**. The only lever is the browser. DoH is the recurring trap: it silently overrides all network DNS.