Fix upload concurrency, bound the request body, make backups consistent
Four audit findings on the photo feature. The upload route was async, so its blocking SQLite work ran on the event loop: under contention it stalled every other request for SQLite's busy timeout, not just its own. It also read the photo count without the write lock, so overlapping uploads all observed the same total and stored past the ceiling together. It is a synchronous endpoint now, running in the threadpool, taking BEGIN IMMEDIATE before re-checking the part, the ceiling and the position, and committing before it returns. Reverting either half makes the new test die with the same TimeoutError the audit reported. The 8MB cap protected nothing: Starlette parses and spools an entire multipart body before a route's dependencies run — before the login check — so the bytes were already on disk by the time anything rejected them, and an anonymous caller could make us write them. A plain ASGI middleware outside routing now refuses an over-large body first, and Caddy enforces the same ceiling at the edge. The documented backup captured the database and the photos at two different moments while the app stayed writable, so a photo deleted in between left the saved database pointing at a file the archive did not contain. tools/backup.sh stops the app for the few seconds the copy takes and verifies afterwards that every referenced photo is in the archive. Cleanup could destroy data rather than merely litter: prune-images could delete a file between an upload writing it and inserting its row, and deletions unlinked before their transaction committed. Pruning now ignores anything under an hour old unless forced, and deletes commit before unlinking — an orphaned file is recoverable, a row without its photo is not. check-images reports drift in both directions and fails only on the direction that loses data. Checks go from 271 to 291. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -39,6 +39,13 @@ and complicate the backup. Uploads are sniffed by content rather than trusted by
|
||||
their declared type, so nothing that claims to be a JPEG can come back out as
|
||||
something a browser will execute; SVG is refused for the same reason.
|
||||
|
||||
Request size is capped in **two** places, because one is not enough. Starlette
|
||||
parses and spools an entire multipart body before a route's dependencies run —
|
||||
which means before the login check — so a route-level cap would only fire after
|
||||
the bytes had already been written to disk by someone who doesn't know the
|
||||
password. An ASGI middleware outside routing rejects an over-large body first,
|
||||
and Caddy enforces the same ceiling at the edge.
|
||||
|
||||
Search is SQLite FTS5 over name, description, manufacturer, MPN, spec values,
|
||||
tags, category and location, with prefix matching so results narrow as you type.
|
||||
Typing `1.75mm`, `prusament`, `0603` or `Bin A3` all find the right things.
|
||||
@@ -55,8 +62,8 @@ Then open http://127.0.0.1:8123.
|
||||
## Tests
|
||||
|
||||
```sh
|
||||
.venv/bin/python -m tests.test_api # 203 checks, in-process
|
||||
.venv/bin/python -m tests.test_concurrency # 25 checks, against a real uvicorn
|
||||
.venv/bin/python -m tests.test_api # 214 checks, in-process
|
||||
.venv/bin/python -m tests.test_concurrency # 33 checks, against a real uvicorn
|
||||
```
|
||||
|
||||
`test_api` exercises the API end to end against a throwaway database — the auth
|
||||
@@ -85,7 +92,7 @@ is silent enough to look like the app is broken.
|
||||
|
||||
`test_concurrency` needs a real server process, because a lost update only shows
|
||||
up when two requests genuinely overlap inside SQLite. It fires overlapping
|
||||
adjustments, patches and creates at one part and asserts the stock log always
|
||||
adjustments, patches, uploads and creates at one part and asserts the stock log always
|
||||
sums to the stored quantity; races taxonomy renames against reads to check the
|
||||
search index never describes a name the tree no longer has; and races session
|
||||
revocations to check no epoch increment is lost.
|
||||
@@ -158,28 +165,37 @@ repository.
|
||||
The database is in the `parts_parts_data` docker volume, which survives
|
||||
rebuilds.
|
||||
|
||||
A backup needs **two** things: the database and the photo files. The database
|
||||
must be captured with SQLite's backup API rather than `cp` — it runs in WAL
|
||||
mode, so recently committed rows may still live in `parts.db-wal` and copying
|
||||
`parts.db` alone can silently lose them. `VACUUM INTO` snapshots a live
|
||||
database consistently:
|
||||
|
||||
```sh
|
||||
docker exec parts python -c \
|
||||
"import sqlite3; sqlite3.connect('/data/parts.db').execute(\"VACUUM INTO '/data/backup.db'\")"
|
||||
docker cp parts:/data/backup.db ./parts-backup-$(date +%F).db
|
||||
docker exec parts rm /data/backup.db
|
||||
|
||||
# the photos, which the database only holds pointers to
|
||||
docker exec parts tar -cf - -C /data images > ./parts-images-$(date +%F).tar
|
||||
./tools/backup.sh /path/to/backups
|
||||
```
|
||||
|
||||
Restore by stopping the container, copying the database back over
|
||||
`/data/parts.db`, deleting any leftover `-wal`/`-shm` alongside it, and
|
||||
unpacking the image tar into `/data`. If the two ever drift apart,
|
||||
`docker exec parts python -m app.admin prune-images` deletes files nothing
|
||||
points at; rows whose file is missing surface as a broken thumbnail rather than
|
||||
an error.
|
||||
A backup is **two** resources that reference each other — the database and the
|
||||
photo files — so capturing them at different moments is not a backup. A photo
|
||||
deleted between the two steps leaves the saved database pointing at a file the
|
||||
archive doesn't contain; one added leaves the reverse. Neither is repairable
|
||||
afterwards, and no amount of pruning fixes the direction that lost data.
|
||||
|
||||
So the script stops the app for the few seconds the copy takes. With no process
|
||||
attached, `parts.db` and its `-wal`/`-shm` sidecars are a consistent set (which
|
||||
is also why `cp parts.db` alone is wrong on a running database — recent commits
|
||||
may still be sitting in the WAL) and the images directory cannot move
|
||||
underneath. It then verifies that every photo the database references is
|
||||
actually present in the archive, and fails loudly if not.
|
||||
|
||||
Restore by stopping the container and unpacking the archive into the volume.
|
||||
|
||||
Two integrity commands, neither of which is a routine step:
|
||||
|
||||
```sh
|
||||
docker exec parts python -m app.admin check-images # drift, in both directions
|
||||
docker exec parts python -m app.admin prune-images # delete files nothing references
|
||||
```
|
||||
|
||||
`check-images` exits non-zero only for a referenced photo whose file is missing
|
||||
— that is data loss, where a stray file is just clutter. `prune-images` ignores
|
||||
anything less than an hour old, because an upload writes its file before
|
||||
inserting its row and a young orphan is indistinguishable from an upload still
|
||||
in flight; `--all` overrides that and is only safe with the app stopped.
|
||||
|
||||
## API
|
||||
|
||||
|
||||
Reference in New Issue
Block a user