Fix upload concurrency, bound the request body, make backups consistent

Four audit findings on the photo feature.

The upload route was async, so its blocking SQLite work ran on the event loop:
under contention it stalled every other request for SQLite's busy timeout, not
just its own. It also read the photo count without the write lock, so
overlapping uploads all observed the same total and stored past the ceiling
together. It is a synchronous endpoint now, running in the threadpool, taking
BEGIN IMMEDIATE before re-checking the part, the ceiling and the position, and
committing before it returns. Reverting either half makes the new test die with
the same TimeoutError the audit reported.

The 8MB cap protected nothing: Starlette parses and spools an entire multipart
body before a route's dependencies run — before the login check — so the bytes
were already on disk by the time anything rejected them, and an anonymous
caller could make us write them. A plain ASGI middleware outside routing now
refuses an over-large body first, and Caddy enforces the same ceiling at the
edge.

The documented backup captured the database and the photos at two different
moments while the app stayed writable, so a photo deleted in between left the
saved database pointing at a file the archive did not contain. tools/backup.sh
stops the app for the few seconds the copy takes and verifies afterwards that
every referenced photo is in the archive.

Cleanup could destroy data rather than merely litter: prune-images could delete
a file between an upload writing it and inserting its row, and deletions
unlinked before their transaction committed. Pruning now ignores anything under
an hour old unless forced, and deletes commit before unlinking — an orphaned
file is recoverable, a row without its photo is not. check-images reports drift
in both directions and fails only on the direction that loses data.

Checks go from 271 to 291.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jay
2026-08-25 09:57:18 -04:00
parent 96d2a1a087
commit a56cea6e2a
6 changed files with 362 additions and 53 deletions
+38 -22
View File
@@ -39,6 +39,13 @@ and complicate the backup. Uploads are sniffed by content rather than trusted by
their declared type, so nothing that claims to be a JPEG can come back out as
something a browser will execute; SVG is refused for the same reason.
Request size is capped in **two** places, because one is not enough. Starlette
parses and spools an entire multipart body before a route's dependencies run —
which means before the login check — so a route-level cap would only fire after
the bytes had already been written to disk by someone who doesn't know the
password. An ASGI middleware outside routing rejects an over-large body first,
and Caddy enforces the same ceiling at the edge.
Search is SQLite FTS5 over name, description, manufacturer, MPN, spec values,
tags, category and location, with prefix matching so results narrow as you type.
Typing `1.75mm`, `prusament`, `0603` or `Bin A3` all find the right things.
@@ -55,8 +62,8 @@ Then open http://127.0.0.1:8123.
## Tests
```sh
.venv/bin/python -m tests.test_api # 203 checks, in-process
.venv/bin/python -m tests.test_concurrency # 25 checks, against a real uvicorn
.venv/bin/python -m tests.test_api # 214 checks, in-process
.venv/bin/python -m tests.test_concurrency # 33 checks, against a real uvicorn
```
`test_api` exercises the API end to end against a throwaway database — the auth
@@ -85,7 +92,7 @@ is silent enough to look like the app is broken.
`test_concurrency` needs a real server process, because a lost update only shows
up when two requests genuinely overlap inside SQLite. It fires overlapping
adjustments, patches and creates at one part and asserts the stock log always
adjustments, patches, uploads and creates at one part and asserts the stock log always
sums to the stored quantity; races taxonomy renames against reads to check the
search index never describes a name the tree no longer has; and races session
revocations to check no epoch increment is lost.
@@ -158,28 +165,37 @@ repository.
The database is in the `parts_parts_data` docker volume, which survives
rebuilds.
A backup needs **two** things: the database and the photo files. The database
must be captured with SQLite's backup API rather than `cp` — it runs in WAL
mode, so recently committed rows may still live in `parts.db-wal` and copying
`parts.db` alone can silently lose them. `VACUUM INTO` snapshots a live
database consistently:
```sh
docker exec parts python -c \
"import sqlite3; sqlite3.connect('/data/parts.db').execute(\"VACUUM INTO '/data/backup.db'\")"
docker cp parts:/data/backup.db ./parts-backup-$(date +%F).db
docker exec parts rm /data/backup.db
# the photos, which the database only holds pointers to
docker exec parts tar -cf - -C /data images > ./parts-images-$(date +%F).tar
./tools/backup.sh /path/to/backups
```
Restore by stopping the container, copying the database back over
`/data/parts.db`, deleting any leftover `-wal`/`-shm` alongside it, and
unpacking the image tar into `/data`. If the two ever drift apart,
`docker exec parts python -m app.admin prune-images` deletes files nothing
points at; rows whose file is missing surface as a broken thumbnail rather than
an error.
A backup is **two** resources that reference each other — the database and the
photo files — so capturing them at different moments is not a backup. A photo
deleted between the two steps leaves the saved database pointing at a file the
archive doesn't contain; one added leaves the reverse. Neither is repairable
afterwards, and no amount of pruning fixes the direction that lost data.
So the script stops the app for the few seconds the copy takes. With no process
attached, `parts.db` and its `-wal`/`-shm` sidecars are a consistent set (which
is also why `cp parts.db` alone is wrong on a running database — recent commits
may still be sitting in the WAL) and the images directory cannot move
underneath. It then verifies that every photo the database references is
actually present in the archive, and fails loudly if not.
Restore by stopping the container and unpacking the archive into the volume.
Two integrity commands, neither of which is a routine step:
```sh
docker exec parts python -m app.admin check-images # drift, in both directions
docker exec parts python -m app.admin prune-images # delete files nothing references
```
`check-images` exits non-zero only for a referenced photo whose file is missing
— that is data loss, where a stray file is just clutter. `prune-images` ignores
anything less than an hour old, because an upload writes its file before
inserting its row and a young orphan is indistinguishable from an upload still
in flight; `--all` overrides that and is only safe with the app stopped.
## API