Fix upload concurrency, bound the request body, make backups consistent
Four audit findings on the photo feature. The upload route was async, so its blocking SQLite work ran on the event loop: under contention it stalled every other request for SQLite's busy timeout, not just its own. It also read the photo count without the write lock, so overlapping uploads all observed the same total and stored past the ceiling together. It is a synchronous endpoint now, running in the threadpool, taking BEGIN IMMEDIATE before re-checking the part, the ceiling and the position, and committing before it returns. Reverting either half makes the new test die with the same TimeoutError the audit reported. The 8MB cap protected nothing: Starlette parses and spools an entire multipart body before a route's dependencies run — before the login check — so the bytes were already on disk by the time anything rejected them, and an anonymous caller could make us write them. A plain ASGI middleware outside routing now refuses an over-large body first, and Caddy enforces the same ceiling at the edge. The documented backup captured the database and the photos at two different moments while the app stayed writable, so a photo deleted in between left the saved database pointing at a file the archive did not contain. tools/backup.sh stops the app for the few seconds the copy takes and verifies afterwards that every referenced photo is in the archive. Cleanup could destroy data rather than merely litter: prune-images could delete a file between an upload writing it and inserting its row, and deletions unlinked before their transaction committed. Pruning now ignores anything under an hour old unless forced, and deletes commit before unlinking — an orphaned file is recoverable, a row without its photo is not. check-images reports drift in both directions and fails only on the direction that loses data. Checks go from 271 to 291. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -39,6 +39,13 @@ and complicate the backup. Uploads are sniffed by content rather than trusted by
|
||||
their declared type, so nothing that claims to be a JPEG can come back out as
|
||||
something a browser will execute; SVG is refused for the same reason.
|
||||
|
||||
Request size is capped in **two** places, because one is not enough. Starlette
|
||||
parses and spools an entire multipart body before a route's dependencies run —
|
||||
which means before the login check — so a route-level cap would only fire after
|
||||
the bytes had already been written to disk by someone who doesn't know the
|
||||
password. An ASGI middleware outside routing rejects an over-large body first,
|
||||
and Caddy enforces the same ceiling at the edge.
|
||||
|
||||
Search is SQLite FTS5 over name, description, manufacturer, MPN, spec values,
|
||||
tags, category and location, with prefix matching so results narrow as you type.
|
||||
Typing `1.75mm`, `prusament`, `0603` or `Bin A3` all find the right things.
|
||||
@@ -55,8 +62,8 @@ Then open http://127.0.0.1:8123.
|
||||
## Tests
|
||||
|
||||
```sh
|
||||
.venv/bin/python -m tests.test_api # 203 checks, in-process
|
||||
.venv/bin/python -m tests.test_concurrency # 25 checks, against a real uvicorn
|
||||
.venv/bin/python -m tests.test_api # 214 checks, in-process
|
||||
.venv/bin/python -m tests.test_concurrency # 33 checks, against a real uvicorn
|
||||
```
|
||||
|
||||
`test_api` exercises the API end to end against a throwaway database — the auth
|
||||
@@ -85,7 +92,7 @@ is silent enough to look like the app is broken.
|
||||
|
||||
`test_concurrency` needs a real server process, because a lost update only shows
|
||||
up when two requests genuinely overlap inside SQLite. It fires overlapping
|
||||
adjustments, patches and creates at one part and asserts the stock log always
|
||||
adjustments, patches, uploads and creates at one part and asserts the stock log always
|
||||
sums to the stored quantity; races taxonomy renames against reads to check the
|
||||
search index never describes a name the tree no longer has; and races session
|
||||
revocations to check no epoch increment is lost.
|
||||
@@ -158,28 +165,37 @@ repository.
|
||||
The database is in the `parts_parts_data` docker volume, which survives
|
||||
rebuilds.
|
||||
|
||||
A backup needs **two** things: the database and the photo files. The database
|
||||
must be captured with SQLite's backup API rather than `cp` — it runs in WAL
|
||||
mode, so recently committed rows may still live in `parts.db-wal` and copying
|
||||
`parts.db` alone can silently lose them. `VACUUM INTO` snapshots a live
|
||||
database consistently:
|
||||
|
||||
```sh
|
||||
docker exec parts python -c \
|
||||
"import sqlite3; sqlite3.connect('/data/parts.db').execute(\"VACUUM INTO '/data/backup.db'\")"
|
||||
docker cp parts:/data/backup.db ./parts-backup-$(date +%F).db
|
||||
docker exec parts rm /data/backup.db
|
||||
|
||||
# the photos, which the database only holds pointers to
|
||||
docker exec parts tar -cf - -C /data images > ./parts-images-$(date +%F).tar
|
||||
./tools/backup.sh /path/to/backups
|
||||
```
|
||||
|
||||
Restore by stopping the container, copying the database back over
|
||||
`/data/parts.db`, deleting any leftover `-wal`/`-shm` alongside it, and
|
||||
unpacking the image tar into `/data`. If the two ever drift apart,
|
||||
`docker exec parts python -m app.admin prune-images` deletes files nothing
|
||||
points at; rows whose file is missing surface as a broken thumbnail rather than
|
||||
an error.
|
||||
A backup is **two** resources that reference each other — the database and the
|
||||
photo files — so capturing them at different moments is not a backup. A photo
|
||||
deleted between the two steps leaves the saved database pointing at a file the
|
||||
archive doesn't contain; one added leaves the reverse. Neither is repairable
|
||||
afterwards, and no amount of pruning fixes the direction that lost data.
|
||||
|
||||
So the script stops the app for the few seconds the copy takes. With no process
|
||||
attached, `parts.db` and its `-wal`/`-shm` sidecars are a consistent set (which
|
||||
is also why `cp parts.db` alone is wrong on a running database — recent commits
|
||||
may still be sitting in the WAL) and the images directory cannot move
|
||||
underneath. It then verifies that every photo the database references is
|
||||
actually present in the archive, and fails loudly if not.
|
||||
|
||||
Restore by stopping the container and unpacking the archive into the volume.
|
||||
|
||||
Two integrity commands, neither of which is a routine step:
|
||||
|
||||
```sh
|
||||
docker exec parts python -m app.admin check-images # drift, in both directions
|
||||
docker exec parts python -m app.admin prune-images # delete files nothing references
|
||||
```
|
||||
|
||||
`check-images` exits non-zero only for a referenced photo whose file is missing
|
||||
— that is data loss, where a stray file is just clutter. `prune-images` ignores
|
||||
anything less than an hour old, because an upload writes its file before
|
||||
inserting its row and a young orphan is indistinguishable from an upload still
|
||||
in flight; `--all` overrides that and is only safe with the app stopped.
|
||||
|
||||
## API
|
||||
|
||||
|
||||
+57
-8
@@ -5,6 +5,7 @@ cannot help with — a forgotten password — and for the initial handover.
|
||||
|
||||
docker exec -it parts python -m app.admin set-password
|
||||
docker exec parts python -m app.admin show-status
|
||||
docker exec parts python -m app.admin check-images
|
||||
docker exec parts python -m app.admin prune-images
|
||||
"""
|
||||
|
||||
@@ -41,25 +42,72 @@ def clear_password(_argv):
|
||||
return 0
|
||||
|
||||
|
||||
def prune_images(_argv):
|
||||
# An upload writes its file before inserting the row, so for a moment a live
|
||||
# file legitimately has no row. Pruning anything younger than this would delete
|
||||
# a photo out from under a request that is still in flight.
|
||||
PRUNE_MIN_AGE_SECONDS = 3600
|
||||
|
||||
|
||||
def prune_images(argv):
|
||||
"""Delete image files with no row pointing at them.
|
||||
|
||||
Uploads write the file before the row, so an ill-timed crash can strand
|
||||
one. Nothing else creates orphans; this is a sweeper, not a routine step.
|
||||
Only files older than an hour, unless --all is given — and --all is only
|
||||
safe with the app stopped, because a file younger than its row is exactly
|
||||
what an in-flight upload looks like.
|
||||
"""
|
||||
import os
|
||||
import time
|
||||
|
||||
ignore_age = "--all" in argv
|
||||
with db.session() as conn:
|
||||
known = {
|
||||
os.path.basename(db.image_path(r["token"], r["mime"]))
|
||||
for r in conn.execute("SELECT token, mime FROM part_images").fetchall()
|
||||
}
|
||||
removed = 0
|
||||
removed = skipped = 0
|
||||
now = time.time()
|
||||
for name in os.listdir(db.IMAGE_DIR):
|
||||
if name not in known:
|
||||
os.remove(os.path.join(db.IMAGE_DIR, name))
|
||||
if name in known:
|
||||
continue
|
||||
path = os.path.join(db.IMAGE_DIR, name)
|
||||
if not ignore_age and now - os.path.getmtime(path) < PRUNE_MIN_AGE_SECONDS:
|
||||
skipped += 1
|
||||
continue
|
||||
os.remove(path)
|
||||
removed += 1
|
||||
print(f"{removed} orphaned image file(s) removed; {len(known)} kept.")
|
||||
print(f"{removed} orphaned image file(s) removed; {len(known)} referenced file(s) kept.")
|
||||
if skipped:
|
||||
print(f"{skipped} skipped as too recent to be certain they are orphans "
|
||||
f"(stop the app and re-run with --all to include them).")
|
||||
return 0
|
||||
|
||||
|
||||
def check_images(_argv):
|
||||
"""Report drift in both directions between the database and the files."""
|
||||
import os
|
||||
|
||||
with db.session() as conn:
|
||||
rows = conn.execute("SELECT token, mime, part_id FROM part_images").fetchall()
|
||||
expected = {os.path.basename(db.image_path(r["token"], r["mime"])): r for r in rows}
|
||||
present = set(os.listdir(db.IMAGE_DIR))
|
||||
missing = sorted(set(expected) - present)
|
||||
orphans = sorted(present - set(expected))
|
||||
print(f"{len(expected)} referenced photo(s), {len(present)} file(s) on disk")
|
||||
for name in missing:
|
||||
print(f" MISSING FILE part {expected[name]['part_id']} {name}")
|
||||
for name in orphans:
|
||||
print(f" orphan file {name}")
|
||||
# A referenced photo with no file is data loss; a stray file is only clutter.
|
||||
return 1 if missing else 0
|
||||
|
||||
|
||||
def list_image_files(_argv):
|
||||
"""Every filename the database expects to exist. Used by the backup script."""
|
||||
import os
|
||||
|
||||
with db.session() as conn:
|
||||
for r in conn.execute("SELECT token, mime FROM part_images ORDER BY id").fetchall():
|
||||
print(os.path.basename(db.image_path(r["token"], r["mime"])))
|
||||
return 0
|
||||
|
||||
|
||||
@@ -79,7 +127,8 @@ def show_status(_argv):
|
||||
|
||||
|
||||
COMMANDS = {"set-password": set_password, "clear-password": clear_password,
|
||||
"show-status": show_status, "prune-images": prune_images}
|
||||
"show-status": show_status, "prune-images": prune_images,
|
||||
"check-images": check_images, "list-image-files": list_image_files}
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
|
||||
+97
-18
@@ -32,6 +32,9 @@ STATIC_DIR = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file
|
||||
# that nothing can fill the volume by accident.
|
||||
MAX_IMAGE_BYTES = 8 * 1024 * 1024
|
||||
MAX_IMAGES_PER_PART = 12
|
||||
# Multipart framing adds boundaries and headers around the file, so the whole
|
||||
# request is allowed slightly more than one image.
|
||||
MAX_BODY_BYTES = MAX_IMAGE_BYTES + 1024 * 1024
|
||||
|
||||
CSP = (
|
||||
"default-src 'self'; "
|
||||
@@ -45,6 +48,62 @@ CSP = (
|
||||
)
|
||||
|
||||
|
||||
class BodySizeLimit:
|
||||
"""Reject an over-large request before anything parses it.
|
||||
|
||||
Starlette parses and spools an entire multipart body *before* the route's
|
||||
dependencies run, so a route-level cap — and the login check — both happen
|
||||
after the bytes have already landed in the container's temporary storage.
|
||||
A caller who doesn't know the password could still make us write them.
|
||||
|
||||
This is plain ASGI so it sits outside routing. Caddy enforces the same
|
||||
ceiling at the edge; this is the backstop for anything that reaches the app
|
||||
directly.
|
||||
"""
|
||||
|
||||
def __init__(self, app, max_bytes: int):
|
||||
self.app = app
|
||||
self.max_bytes = max_bytes
|
||||
|
||||
async def __call__(self, scope, receive, send):
|
||||
if scope["type"] != "http":
|
||||
return await self.app(scope, receive, send)
|
||||
|
||||
for name, value in scope.get("headers") or []:
|
||||
if name == b"content-length":
|
||||
try:
|
||||
declared = int(value)
|
||||
except ValueError:
|
||||
break
|
||||
if declared > self.max_bytes:
|
||||
return await self._too_large(send)
|
||||
break
|
||||
|
||||
seen = 0
|
||||
|
||||
async def limited_receive():
|
||||
nonlocal seen
|
||||
message = await receive()
|
||||
if message.get("type") == "http.request":
|
||||
seen += len(message.get("body", b""))
|
||||
if seen > self.max_bytes:
|
||||
# Chunked upload with no declared length: cut the stream so
|
||||
# the parser fails rather than letting it run unbounded.
|
||||
return {"type": "http.request", "body": b"", "more_body": False}
|
||||
return message
|
||||
|
||||
await self.app(scope, limited_receive, send)
|
||||
|
||||
async def _too_large(self, send):
|
||||
body = json.dumps(
|
||||
{"detail": f"Request body must be {self.max_bytes // (1024 * 1024)}MB or smaller"}
|
||||
).encode()
|
||||
await send({"type": "http.response.start", "status": 413,
|
||||
"headers": [(b"content-type", b"application/json"),
|
||||
(b"content-length", str(len(body)).encode())]})
|
||||
await send({"type": "http.response.body", "body": body})
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(_app: FastAPI):
|
||||
db.init()
|
||||
@@ -52,6 +111,8 @@ async def lifespan(_app: FastAPI):
|
||||
|
||||
|
||||
app = FastAPI(title="Parts Inventory", docs_url=None, redoc_url=None, lifespan=lifespan)
|
||||
# Added last, so it wraps everything else and runs before routing or body parsing.
|
||||
app.add_middleware(BodySizeLimit, max_bytes=MAX_BODY_BYTES)
|
||||
|
||||
|
||||
# --- validated field types --------------------------------------------------
|
||||
@@ -609,6 +670,10 @@ def delete_part(part_id: int, conn=Depends(db.get_db)):
|
||||
if cur.rowcount == 0:
|
||||
raise HTTPException(404, "Part not found")
|
||||
conn.execute("DELETE FROM parts_fts WHERE rowid = ?", (part_id,))
|
||||
# Commit before unlinking. If the order were reversed and the commit failed,
|
||||
# the rows would come back pointing at files that no longer exist — and a
|
||||
# missing photo is unrecoverable, where a stranded file is just clutter.
|
||||
conn.commit()
|
||||
db.delete_image_files(images)
|
||||
return {"deleted": part_id}
|
||||
|
||||
@@ -672,26 +737,24 @@ def list_images(part_id: int, conn=Depends(db.get_db)):
|
||||
|
||||
@app.post("/api/parts/{part_id}/images", dependencies=[Depends(auth.require_auth)],
|
||||
status_code=201)
|
||||
async def upload_image(
|
||||
def upload_image(
|
||||
part_id: int,
|
||||
file: UploadFile = File(...),
|
||||
caption: str = Form(default=""),
|
||||
conn=Depends(db.get_db),
|
||||
):
|
||||
if conn.execute("SELECT 1 FROM parts WHERE id = ?", (part_id,)).fetchone() is None:
|
||||
raise HTTPException(404, "Part not found")
|
||||
"""Store one photo against a part.
|
||||
|
||||
existing = conn.execute(
|
||||
"SELECT COUNT(*) AS n FROM part_images WHERE part_id = ?", (part_id,)
|
||||
).fetchone()["n"]
|
||||
if existing >= MAX_IMAGES_PER_PART:
|
||||
raise HTTPException(409, f"A part can hold at most {MAX_IMAGES_PER_PART} images")
|
||||
|
||||
# Read with a ceiling rather than trusting Content-Length, which a client
|
||||
# controls. One byte over the cap is enough to know it is too big.
|
||||
Deliberately a synchronous endpoint. FastAPI runs these in the threadpool,
|
||||
whereas an async one runs on the event loop — and this does blocking SQLite
|
||||
work, so under contention it would stall every other request for the length
|
||||
of SQLite's busy timeout rather than just queueing this one.
|
||||
"""
|
||||
# Read and sniff before taking any lock: the bytes are the slow part, and
|
||||
# holding SQLite's write lock across them would serialise unrelated writes.
|
||||
data = b""
|
||||
while len(data) <= MAX_IMAGE_BYTES:
|
||||
chunk = await file.read(256 * 1024)
|
||||
chunk = file.file.read(256 * 1024)
|
||||
if not chunk:
|
||||
break
|
||||
data += chunk
|
||||
@@ -710,25 +773,39 @@ async def upload_image(
|
||||
path = db.image_path(token, mime)
|
||||
with open(path, "wb") as fh:
|
||||
fh.write(data)
|
||||
|
||||
try:
|
||||
nxt = conn.execute(
|
||||
# Everything that reads-then-writes happens under the write lock: the
|
||||
# existence check, the ceiling, and the position. Without it, overlapping
|
||||
# uploads all observe the same count and sail past the limit together.
|
||||
db.begin_immediate(conn)
|
||||
if conn.execute("SELECT 1 FROM parts WHERE id = ?", (part_id,)).fetchone() is None:
|
||||
raise HTTPException(404, "Part not found")
|
||||
existing = conn.execute(
|
||||
"SELECT COUNT(*) AS n FROM part_images WHERE part_id = ?", (part_id,)
|
||||
).fetchone()["n"]
|
||||
if existing >= MAX_IMAGES_PER_PART:
|
||||
raise HTTPException(409, f"A part can hold at most {MAX_IMAGES_PER_PART} images")
|
||||
position = conn.execute(
|
||||
"SELECT COALESCE(MAX(position), -1) + 1 AS n FROM part_images WHERE part_id = ?",
|
||||
(part_id,),
|
||||
).fetchone()["n"]
|
||||
conn.execute(
|
||||
"INSERT INTO part_images(part_id, token, mime, bytes, caption, position) "
|
||||
"VALUES (?,?,?,?,?,?)",
|
||||
(part_id, token, mime, len(data), caption.strip()[:300], nxt),
|
||||
)
|
||||
conn.execute(
|
||||
"UPDATE parts SET updated_at = datetime('now') WHERE id = ?", (part_id,)
|
||||
(part_id, token, mime, len(data), caption.strip()[:300], position),
|
||||
)
|
||||
conn.execute("UPDATE parts SET updated_at = datetime('now') WHERE id = ?", (part_id,))
|
||||
# Commit here rather than leaving it to dependency cleanup, so the write
|
||||
# lock is released before the response is built rather than after.
|
||||
conn.commit()
|
||||
except Exception:
|
||||
# Don't leave a file behind for a row that never landed.
|
||||
db.delete_image_files([{"token": token, "mime": mime}])
|
||||
raise
|
||||
|
||||
return {"token": token, "mime": mime, "bytes": len(data),
|
||||
"caption": caption.strip()[:300], "position": nxt}
|
||||
"caption": caption.strip()[:300], "position": position}
|
||||
|
||||
|
||||
@app.get("/api/parts/{part_id}/images/{token}", dependencies=[Depends(auth.require_auth)])
|
||||
@@ -770,6 +847,8 @@ def delete_image(part_id: int, token: str, conn=Depends(db.get_db)):
|
||||
row = _image_row(conn, part_id, token)
|
||||
conn.execute("DELETE FROM part_images WHERE part_id = ? AND token = ?", (part_id, token))
|
||||
conn.execute("UPDATE parts SET updated_at = datetime('now') WHERE id = ?", (part_id,))
|
||||
# Commit first: an orphaned file is recoverable, a row without its file is not.
|
||||
conn.commit()
|
||||
db.delete_image_files([row])
|
||||
return {"deleted": token}
|
||||
|
||||
|
||||
+28
-1
@@ -451,6 +451,24 @@ with TestClient(app) as client:
|
||||
check("a rejected upload leaves no file behind",
|
||||
len(os.listdir(_imgdb.IMAGE_DIR)) == 1, str(os.listdir(_imgdb.IMAGE_DIR)))
|
||||
|
||||
# The body limit sits outside routing, so it answers before the multipart
|
||||
# parser spools anything and before the login check runs.
|
||||
import app.main as _mainmod
|
||||
huge = b"x" * (_mainmod.MAX_BODY_BYTES + 4096)
|
||||
r = client.post(f"/api/parts/{photo_part}/images",
|
||||
content=huge, headers={"Content-Type": "application/octet-stream"})
|
||||
check("an over-large body is refused outright", r.status_code == 413, str(r.status_code))
|
||||
check("and it says so as JSON", "smaller" in r.json()["detail"])
|
||||
|
||||
anon = TestClient(app)
|
||||
r = anon.post(f"/api/parts/{photo_part}/images",
|
||||
content=huge, headers={"Content-Type": "application/octet-stream"})
|
||||
check("an unauthenticated over-large body is refused before the login check",
|
||||
r.status_code == 413, f"{r.status_code} — 401 would mean it was parsed first")
|
||||
check("a normal unauthenticated upload still gets 401",
|
||||
anon.post(f"/api/parts/{photo_part}/images",
|
||||
files={"file": ("x.png", make_png(), "image/png")}).status_code == 401)
|
||||
|
||||
check("photos on a missing part 404",
|
||||
post_image(999999, make_png()).status_code == 404)
|
||||
check("fetching an unknown token 404s",
|
||||
@@ -498,10 +516,19 @@ with TestClient(app) as client:
|
||||
orphan = os.path.join(_imgdb.IMAGE_DIR, "stranded.jpg")
|
||||
with open(orphan, "wb") as fh:
|
||||
fh.write(b"\xff\xd8\xffleftover")
|
||||
check("prune-images runs", _admin_img.main(["prune-images"]) == 0)
|
||||
check("prune-images spares a file too recent to be sure about",
|
||||
_admin_img.main(["prune-images"]) == 0 and os.path.exists(orphan))
|
||||
check("--all prunes it", _admin_img.main(["prune-images", "--all"]) == 0)
|
||||
check("the orphan is gone", not os.path.exists(orphan))
|
||||
check("the referenced file is kept",
|
||||
client.get(f"/api/parts/{sweep_part}/images/{kept}").status_code == 200)
|
||||
check("check-images is clean when nothing has drifted",
|
||||
_admin_img.main(["check-images"]) == 0)
|
||||
# A referenced photo whose file vanished is the direction that matters.
|
||||
os.remove(os.path.join(_imgdb.IMAGE_DIR, os.listdir(_imgdb.IMAGE_DIR)[0]))
|
||||
check("check-images reports a missing file as a failure",
|
||||
_admin_img.main(["check-images"]) == 1)
|
||||
check("list-image-files works", _admin_img.main(["list-image-files"]) == 0)
|
||||
client.delete(f"/api/parts/{sweep_part}")
|
||||
|
||||
# --- password management through the API ---
|
||||
|
||||
@@ -6,6 +6,9 @@ genuinely overlapping inside SQLite, which means a real server and real threads.
|
||||
|
||||
import http.cookiejar
|
||||
import json
|
||||
import secrets
|
||||
import struct
|
||||
import zlib
|
||||
import os
|
||||
import shutil
|
||||
import signal
|
||||
@@ -42,6 +45,44 @@ def call(opener, base, path, method="GET", body=None):
|
||||
return json.load(opener(r, timeout=60))
|
||||
|
||||
|
||||
def make_png(w=40, h=40, rgb=(120, 60, 200)):
|
||||
raw = b"".join(b"\x00" + bytes(rgb) * w for _ in range(h))
|
||||
|
||||
def chunk(tag, data):
|
||||
body = tag + data
|
||||
return struct.pack(">I", len(data)) + body + struct.pack(">I", zlib.crc32(body))
|
||||
|
||||
return (b"\x89PNG\r\n\x1a\n"
|
||||
+ chunk(b"IHDR", struct.pack(">IIBBBBB", w, h, 8, 2, 0, 0, 0))
|
||||
+ chunk(b"IDAT", zlib.compress(raw))
|
||||
+ chunk(b"IEND", b""))
|
||||
|
||||
|
||||
def multipart(content, filename="photo.png", ctype="image/png", fields=None):
|
||||
"""Build a multipart body by hand — no third-party HTTP client in the tests."""
|
||||
boundary = "----parts" + secrets.token_hex(8)
|
||||
out = []
|
||||
for key, value in (fields or {}).items():
|
||||
out.append(f"--{boundary}\r\nContent-Disposition: form-data; "
|
||||
f'name="{key}"\r\n\r\n{value}\r\n'.encode())
|
||||
out.append(f"--{boundary}\r\nContent-Disposition: form-data; "
|
||||
f'name="file"; filename="{filename}"\r\n'
|
||||
f"Content-Type: {ctype}\r\n\r\n".encode() + content + b"\r\n")
|
||||
out.append(f"--{boundary}--\r\n".encode())
|
||||
return b"".join(out), f"multipart/form-data; boundary={boundary}"
|
||||
|
||||
|
||||
def post_image(base, part_id, content, timeout=30):
|
||||
body, ctype = multipart(content)
|
||||
r = urllib.request.Request(f"{base}/api/parts/{part_id}/images", data=body,
|
||||
method="POST", headers={"Content-Type": ctype})
|
||||
try:
|
||||
with urllib.request.urlopen(r, timeout=timeout) as resp:
|
||||
return resp.status
|
||||
except urllib.error.HTTPError as e:
|
||||
return e.code
|
||||
|
||||
|
||||
@contextmanager
|
||||
def server(port, **env_extra):
|
||||
"""Run a real uvicorn against a throwaway database."""
|
||||
@@ -62,7 +103,7 @@ def server(port, **env_extra):
|
||||
time.sleep(0.25)
|
||||
else:
|
||||
raise RuntimeError("server never started")
|
||||
yield base
|
||||
yield base, tmp
|
||||
finally:
|
||||
proc.send_signal(signal.SIGINT)
|
||||
try:
|
||||
@@ -95,7 +136,7 @@ def session_phase():
|
||||
password = "concurrency-test-password"
|
||||
with server(PORT + 1, PARTS_AUTH="on", PARTS_PASSWORD=password,
|
||||
PARTS_SECRET="concurrency-test-secret",
|
||||
PARTS_LOGIN_MAX_FAILURES="500") as base:
|
||||
PARTS_LOGIN_MAX_FAILURES="500") as (base, _tmp):
|
||||
# --- the plain case first ---
|
||||
keeper = session_for(base, password)
|
||||
others = [session_for(base, password) for _ in range(5)]
|
||||
@@ -142,7 +183,7 @@ def session_phase():
|
||||
|
||||
|
||||
def main():
|
||||
with server(PORT, PARTS_AUTH="off"):
|
||||
with server(PORT, PARTS_AUTH="off") as (_base, tmp):
|
||||
# --- concurrent decrements must not lose updates ---
|
||||
pid = req("/api/parts", "POST", {"name": "race target", "quantity": 100})["id"]
|
||||
with ThreadPoolExecutor(max_workers=50) as ex:
|
||||
@@ -244,6 +285,55 @@ def main():
|
||||
check("all concurrent creates are searchable",
|
||||
req("/api/parts?q=bulk&limit=100")["total"] == 25,
|
||||
str(req("/api/parts?q=bulk&limit=100")["total"]))
|
||||
# --- concurrent photo uploads ---
|
||||
# An async endpoint doing blocking SQLite work stalls the whole event
|
||||
# loop under contention; an unlocked count lets every overlapping upload
|
||||
# observe the same total and sail past the ceiling together. This checks
|
||||
# both: the ceiling holds exactly, and the server stays responsive while
|
||||
# it is being hammered.
|
||||
shot_part = req("/api/parts", "POST", {"name": "photo race", "quantity": 1})["id"]
|
||||
attempts = 20
|
||||
latency = []
|
||||
|
||||
def ping():
|
||||
start = time.time()
|
||||
try:
|
||||
call(urllib.request.urlopen, _base, "/healthz")
|
||||
except Exception:
|
||||
pass
|
||||
latency.append(time.time() - start)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=attempts + 4) as ex:
|
||||
futures = [ex.submit(post_image, _base, shot_part, make_png(60, 60, (i * 9, 40, 90)))
|
||||
for i in range(attempts)]
|
||||
for _ in range(4):
|
||||
ex.submit(ping)
|
||||
codes = [f.result() for f in futures]
|
||||
|
||||
created = codes.count(201)
|
||||
refused = codes.count(409)
|
||||
check("no upload returned a server error",
|
||||
not [c for c in codes if c >= 500], str(sorted(set(codes))))
|
||||
check("every upload was answered", len(codes) == attempts and None not in codes)
|
||||
check("exactly the ceiling was stored", created == 12, f"{created} created, {refused} refused")
|
||||
check("the rest were refused as over the limit", created + refused == attempts,
|
||||
str(sorted(set(codes))))
|
||||
stored = req(f"/api/parts/{shot_part}/images")["items"]
|
||||
check("the database agrees with the ceiling", len(stored) == 12, str(len(stored)))
|
||||
check("positions are unique", len({i["position"] for i in stored}) == 12,
|
||||
str(sorted(i["position"] for i in stored)))
|
||||
|
||||
image_dir = os.path.join(tmp, "images")
|
||||
on_disk = len(os.listdir(image_dir))
|
||||
check("refused uploads left no files behind", on_disk == 12, f"{on_disk} files")
|
||||
|
||||
worst = max(latency) if latency else 0
|
||||
# A blocked event loop parks unrelated requests for SQLite's busy
|
||||
# timeout, which is 15s. Anything in that neighbourhood means the
|
||||
# upload is running on the loop again.
|
||||
check("the server stayed responsive during the upload storm",
|
||||
worst < 5.0, f"slowest /healthz was {worst:.1f}s")
|
||||
|
||||
session_phase()
|
||||
|
||||
print()
|
||||
|
||||
Executable
+48
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env bash
|
||||
# Consistent backup of the parts inventory.
|
||||
#
|
||||
# The database and the photos are two resources that reference each other, so
|
||||
# capturing them at different moments is not a backup: a photo deleted between
|
||||
# the two steps leaves the saved database pointing at a file that isn't in the
|
||||
# archive, and one added leaves the reverse. Neither is repairable afterwards.
|
||||
#
|
||||
# So the app is stopped for the few seconds it takes to copy both. With no
|
||||
# process attached, parts.db and its -wal/-shm sidecars are a consistent set and
|
||||
# the images directory cannot move underneath us.
|
||||
#
|
||||
# ./tools/backup.sh [output-directory]
|
||||
set -euo pipefail
|
||||
|
||||
OUT_DIR="$(cd "${1:-$PWD}" && pwd)"
|
||||
cd "$(dirname "$0")/.."
|
||||
STAMP="$(date +%F-%H%M%S)"
|
||||
NAME="parts-backup-$STAMP.tar.gz"
|
||||
VOLUME="$(docker compose config --format json 2>/dev/null \
|
||||
| python3 -c 'import json,sys; print(list(json.load(sys.stdin)["volumes"])[0])' 2>/dev/null || echo parts_data)"
|
||||
VOLUME="$(docker volume ls --format '{{.Name}}' | grep -E "parts.*data" | head -1)"
|
||||
|
||||
echo "Stopping the app so both halves are captured at one moment..."
|
||||
docker compose stop parts >/dev/null
|
||||
|
||||
restart() { docker compose start parts >/dev/null && echo "App restarted."; }
|
||||
trap restart EXIT
|
||||
|
||||
# What the database expects to exist, read while nothing can be writing.
|
||||
EXPECTED="$(docker run --rm -v "$VOLUME":/data -e PARTS_DB=/data/parts.db \
|
||||
--entrypoint python parts -m app.admin list-image-files | sort)"
|
||||
|
||||
docker run --rm -v "$VOLUME":/data -v "$OUT_DIR":/out alpine \
|
||||
tar -czf "/out/$NAME" -C /data .
|
||||
|
||||
# Verify every referenced photo actually made it into the archive.
|
||||
IN_TAR="$(tar -tzf "$OUT_DIR/$NAME" | sed -n 's#^\./images/##p' | sort)"
|
||||
MISSING="$(comm -23 <(printf '%s\n' "$EXPECTED" | grep -v '^$' || true) \
|
||||
<(printf '%s\n' "$IN_TAR" | grep -v '^$' || true) || true)"
|
||||
if [ -n "$MISSING" ]; then
|
||||
echo "BACKUP INCOMPLETE — referenced photos missing from the archive:" >&2
|
||||
printf ' %s\n' $MISSING >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
COUNT="$(printf '%s\n' "$EXPECTED" | grep -c . || true)"
|
||||
echo "Wrote $OUT_DIR/$NAME ($(du -h "$OUT_DIR/$NAME" | cut -f1)), $COUNT photo(s) verified present."
|
||||
Reference in New Issue
Block a user