diff --git a/README.md b/README.md index 0915dc7..21ee577 100644 --- a/README.md +++ b/README.md @@ -39,6 +39,13 @@ and complicate the backup. Uploads are sniffed by content rather than trusted by their declared type, so nothing that claims to be a JPEG can come back out as something a browser will execute; SVG is refused for the same reason. +Request size is capped in **two** places, because one is not enough. Starlette +parses and spools an entire multipart body before a route's dependencies run — +which means before the login check — so a route-level cap would only fire after +the bytes had already been written to disk by someone who doesn't know the +password. An ASGI middleware outside routing rejects an over-large body first, +and Caddy enforces the same ceiling at the edge. + Search is SQLite FTS5 over name, description, manufacturer, MPN, spec values, tags, category and location, with prefix matching so results narrow as you type. Typing `1.75mm`, `prusament`, `0603` or `Bin A3` all find the right things. @@ -55,8 +62,8 @@ Then open http://127.0.0.1:8123. ## Tests ```sh -.venv/bin/python -m tests.test_api # 203 checks, in-process -.venv/bin/python -m tests.test_concurrency # 25 checks, against a real uvicorn +.venv/bin/python -m tests.test_api # 214 checks, in-process +.venv/bin/python -m tests.test_concurrency # 33 checks, against a real uvicorn ``` `test_api` exercises the API end to end against a throwaway database — the auth @@ -85,7 +92,7 @@ is silent enough to look like the app is broken. `test_concurrency` needs a real server process, because a lost update only shows up when two requests genuinely overlap inside SQLite. It fires overlapping -adjustments, patches and creates at one part and asserts the stock log always +adjustments, patches, uploads and creates at one part and asserts the stock log always sums to the stored quantity; races taxonomy renames against reads to check the search index never describes a name the tree no longer has; and races session revocations to check no epoch increment is lost. @@ -158,28 +165,37 @@ repository. The database is in the `parts_parts_data` docker volume, which survives rebuilds. -A backup needs **two** things: the database and the photo files. The database -must be captured with SQLite's backup API rather than `cp` — it runs in WAL -mode, so recently committed rows may still live in `parts.db-wal` and copying -`parts.db` alone can silently lose them. `VACUUM INTO` snapshots a live -database consistently: - ```sh -docker exec parts python -c \ - "import sqlite3; sqlite3.connect('/data/parts.db').execute(\"VACUUM INTO '/data/backup.db'\")" -docker cp parts:/data/backup.db ./parts-backup-$(date +%F).db -docker exec parts rm /data/backup.db - -# the photos, which the database only holds pointers to -docker exec parts tar -cf - -C /data images > ./parts-images-$(date +%F).tar +./tools/backup.sh /path/to/backups ``` -Restore by stopping the container, copying the database back over -`/data/parts.db`, deleting any leftover `-wal`/`-shm` alongside it, and -unpacking the image tar into `/data`. If the two ever drift apart, -`docker exec parts python -m app.admin prune-images` deletes files nothing -points at; rows whose file is missing surface as a broken thumbnail rather than -an error. +A backup is **two** resources that reference each other — the database and the +photo files — so capturing them at different moments is not a backup. A photo +deleted between the two steps leaves the saved database pointing at a file the +archive doesn't contain; one added leaves the reverse. Neither is repairable +afterwards, and no amount of pruning fixes the direction that lost data. + +So the script stops the app for the few seconds the copy takes. With no process +attached, `parts.db` and its `-wal`/`-shm` sidecars are a consistent set (which +is also why `cp parts.db` alone is wrong on a running database — recent commits +may still be sitting in the WAL) and the images directory cannot move +underneath. It then verifies that every photo the database references is +actually present in the archive, and fails loudly if not. + +Restore by stopping the container and unpacking the archive into the volume. + +Two integrity commands, neither of which is a routine step: + +```sh +docker exec parts python -m app.admin check-images # drift, in both directions +docker exec parts python -m app.admin prune-images # delete files nothing references +``` + +`check-images` exits non-zero only for a referenced photo whose file is missing +— that is data loss, where a stray file is just clutter. `prune-images` ignores +anything less than an hour old, because an upload writes its file before +inserting its row and a young orphan is indistinguishable from an upload still +in flight; `--all` overrides that and is only safe with the app stopped. ## API diff --git a/app/admin.py b/app/admin.py index 6bed046..f330144 100644 --- a/app/admin.py +++ b/app/admin.py @@ -5,6 +5,7 @@ cannot help with — a forgotten password — and for the initial handover. docker exec -it parts python -m app.admin set-password docker exec parts python -m app.admin show-status + docker exec parts python -m app.admin check-images docker exec parts python -m app.admin prune-images """ @@ -41,25 +42,72 @@ def clear_password(_argv): return 0 -def prune_images(_argv): +# An upload writes its file before inserting the row, so for a moment a live +# file legitimately has no row. Pruning anything younger than this would delete +# a photo out from under a request that is still in flight. +PRUNE_MIN_AGE_SECONDS = 3600 + + +def prune_images(argv): """Delete image files with no row pointing at them. - Uploads write the file before the row, so an ill-timed crash can strand - one. Nothing else creates orphans; this is a sweeper, not a routine step. + Only files older than an hour, unless --all is given — and --all is only + safe with the app stopped, because a file younger than its row is exactly + what an in-flight upload looks like. """ import os + import time + ignore_age = "--all" in argv with db.session() as conn: known = { os.path.basename(db.image_path(r["token"], r["mime"])) for r in conn.execute("SELECT token, mime FROM part_images").fetchall() } - removed = 0 + removed = skipped = 0 + now = time.time() for name in os.listdir(db.IMAGE_DIR): - if name not in known: - os.remove(os.path.join(db.IMAGE_DIR, name)) - removed += 1 - print(f"{removed} orphaned image file(s) removed; {len(known)} kept.") + if name in known: + continue + path = os.path.join(db.IMAGE_DIR, name) + if not ignore_age and now - os.path.getmtime(path) < PRUNE_MIN_AGE_SECONDS: + skipped += 1 + continue + os.remove(path) + removed += 1 + print(f"{removed} orphaned image file(s) removed; {len(known)} referenced file(s) kept.") + if skipped: + print(f"{skipped} skipped as too recent to be certain they are orphans " + f"(stop the app and re-run with --all to include them).") + return 0 + + +def check_images(_argv): + """Report drift in both directions between the database and the files.""" + import os + + with db.session() as conn: + rows = conn.execute("SELECT token, mime, part_id FROM part_images").fetchall() + expected = {os.path.basename(db.image_path(r["token"], r["mime"])): r for r in rows} + present = set(os.listdir(db.IMAGE_DIR)) + missing = sorted(set(expected) - present) + orphans = sorted(present - set(expected)) + print(f"{len(expected)} referenced photo(s), {len(present)} file(s) on disk") + for name in missing: + print(f" MISSING FILE part {expected[name]['part_id']} {name}") + for name in orphans: + print(f" orphan file {name}") + # A referenced photo with no file is data loss; a stray file is only clutter. + return 1 if missing else 0 + + +def list_image_files(_argv): + """Every filename the database expects to exist. Used by the backup script.""" + import os + + with db.session() as conn: + for r in conn.execute("SELECT token, mime FROM part_images ORDER BY id").fetchall(): + print(os.path.basename(db.image_path(r["token"], r["mime"]))) return 0 @@ -79,7 +127,8 @@ def show_status(_argv): COMMANDS = {"set-password": set_password, "clear-password": clear_password, - "show-status": show_status, "prune-images": prune_images} + "show-status": show_status, "prune-images": prune_images, + "check-images": check_images, "list-image-files": list_image_files} def main(argv=None): diff --git a/app/main.py b/app/main.py index 30c56f6..f057a7a 100644 --- a/app/main.py +++ b/app/main.py @@ -32,6 +32,9 @@ STATIC_DIR = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file # that nothing can fill the volume by accident. MAX_IMAGE_BYTES = 8 * 1024 * 1024 MAX_IMAGES_PER_PART = 12 +# Multipart framing adds boundaries and headers around the file, so the whole +# request is allowed slightly more than one image. +MAX_BODY_BYTES = MAX_IMAGE_BYTES + 1024 * 1024 CSP = ( "default-src 'self'; " @@ -45,6 +48,62 @@ CSP = ( ) +class BodySizeLimit: + """Reject an over-large request before anything parses it. + + Starlette parses and spools an entire multipart body *before* the route's + dependencies run, so a route-level cap — and the login check — both happen + after the bytes have already landed in the container's temporary storage. + A caller who doesn't know the password could still make us write them. + + This is plain ASGI so it sits outside routing. Caddy enforces the same + ceiling at the edge; this is the backstop for anything that reaches the app + directly. + """ + + def __init__(self, app, max_bytes: int): + self.app = app + self.max_bytes = max_bytes + + async def __call__(self, scope, receive, send): + if scope["type"] != "http": + return await self.app(scope, receive, send) + + for name, value in scope.get("headers") or []: + if name == b"content-length": + try: + declared = int(value) + except ValueError: + break + if declared > self.max_bytes: + return await self._too_large(send) + break + + seen = 0 + + async def limited_receive(): + nonlocal seen + message = await receive() + if message.get("type") == "http.request": + seen += len(message.get("body", b"")) + if seen > self.max_bytes: + # Chunked upload with no declared length: cut the stream so + # the parser fails rather than letting it run unbounded. + return {"type": "http.request", "body": b"", "more_body": False} + return message + + await self.app(scope, limited_receive, send) + + async def _too_large(self, send): + body = json.dumps( + {"detail": f"Request body must be {self.max_bytes // (1024 * 1024)}MB or smaller"} + ).encode() + await send({"type": "http.response.start", "status": 413, + "headers": [(b"content-type", b"application/json"), + (b"content-length", str(len(body)).encode())]}) + await send({"type": "http.response.body", "body": body}) + + @asynccontextmanager async def lifespan(_app: FastAPI): db.init() @@ -52,6 +111,8 @@ async def lifespan(_app: FastAPI): app = FastAPI(title="Parts Inventory", docs_url=None, redoc_url=None, lifespan=lifespan) +# Added last, so it wraps everything else and runs before routing or body parsing. +app.add_middleware(BodySizeLimit, max_bytes=MAX_BODY_BYTES) # --- validated field types -------------------------------------------------- @@ -609,6 +670,10 @@ def delete_part(part_id: int, conn=Depends(db.get_db)): if cur.rowcount == 0: raise HTTPException(404, "Part not found") conn.execute("DELETE FROM parts_fts WHERE rowid = ?", (part_id,)) + # Commit before unlinking. If the order were reversed and the commit failed, + # the rows would come back pointing at files that no longer exist — and a + # missing photo is unrecoverable, where a stranded file is just clutter. + conn.commit() db.delete_image_files(images) return {"deleted": part_id} @@ -672,26 +737,24 @@ def list_images(part_id: int, conn=Depends(db.get_db)): @app.post("/api/parts/{part_id}/images", dependencies=[Depends(auth.require_auth)], status_code=201) -async def upload_image( +def upload_image( part_id: int, file: UploadFile = File(...), caption: str = Form(default=""), conn=Depends(db.get_db), ): - if conn.execute("SELECT 1 FROM parts WHERE id = ?", (part_id,)).fetchone() is None: - raise HTTPException(404, "Part not found") + """Store one photo against a part. - existing = conn.execute( - "SELECT COUNT(*) AS n FROM part_images WHERE part_id = ?", (part_id,) - ).fetchone()["n"] - if existing >= MAX_IMAGES_PER_PART: - raise HTTPException(409, f"A part can hold at most {MAX_IMAGES_PER_PART} images") - - # Read with a ceiling rather than trusting Content-Length, which a client - # controls. One byte over the cap is enough to know it is too big. + Deliberately a synchronous endpoint. FastAPI runs these in the threadpool, + whereas an async one runs on the event loop — and this does blocking SQLite + work, so under contention it would stall every other request for the length + of SQLite's busy timeout rather than just queueing this one. + """ + # Read and sniff before taking any lock: the bytes are the slow part, and + # holding SQLite's write lock across them would serialise unrelated writes. data = b"" while len(data) <= MAX_IMAGE_BYTES: - chunk = await file.read(256 * 1024) + chunk = file.file.read(256 * 1024) if not chunk: break data += chunk @@ -710,25 +773,39 @@ async def upload_image( path = db.image_path(token, mime) with open(path, "wb") as fh: fh.write(data) + try: - nxt = conn.execute( + # Everything that reads-then-writes happens under the write lock: the + # existence check, the ceiling, and the position. Without it, overlapping + # uploads all observe the same count and sail past the limit together. + db.begin_immediate(conn) + if conn.execute("SELECT 1 FROM parts WHERE id = ?", (part_id,)).fetchone() is None: + raise HTTPException(404, "Part not found") + existing = conn.execute( + "SELECT COUNT(*) AS n FROM part_images WHERE part_id = ?", (part_id,) + ).fetchone()["n"] + if existing >= MAX_IMAGES_PER_PART: + raise HTTPException(409, f"A part can hold at most {MAX_IMAGES_PER_PART} images") + position = conn.execute( "SELECT COALESCE(MAX(position), -1) + 1 AS n FROM part_images WHERE part_id = ?", (part_id,), ).fetchone()["n"] conn.execute( "INSERT INTO part_images(part_id, token, mime, bytes, caption, position) " "VALUES (?,?,?,?,?,?)", - (part_id, token, mime, len(data), caption.strip()[:300], nxt), - ) - conn.execute( - "UPDATE parts SET updated_at = datetime('now') WHERE id = ?", (part_id,) + (part_id, token, mime, len(data), caption.strip()[:300], position), ) + conn.execute("UPDATE parts SET updated_at = datetime('now') WHERE id = ?", (part_id,)) + # Commit here rather than leaving it to dependency cleanup, so the write + # lock is released before the response is built rather than after. + conn.commit() except Exception: # Don't leave a file behind for a row that never landed. db.delete_image_files([{"token": token, "mime": mime}]) raise + return {"token": token, "mime": mime, "bytes": len(data), - "caption": caption.strip()[:300], "position": nxt} + "caption": caption.strip()[:300], "position": position} @app.get("/api/parts/{part_id}/images/{token}", dependencies=[Depends(auth.require_auth)]) @@ -770,6 +847,8 @@ def delete_image(part_id: int, token: str, conn=Depends(db.get_db)): row = _image_row(conn, part_id, token) conn.execute("DELETE FROM part_images WHERE part_id = ? AND token = ?", (part_id, token)) conn.execute("UPDATE parts SET updated_at = datetime('now') WHERE id = ?", (part_id,)) + # Commit first: an orphaned file is recoverable, a row without its file is not. + conn.commit() db.delete_image_files([row]) return {"deleted": token} diff --git a/tests/test_api.py b/tests/test_api.py index df35d93..05ad1d8 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -451,6 +451,24 @@ with TestClient(app) as client: check("a rejected upload leaves no file behind", len(os.listdir(_imgdb.IMAGE_DIR)) == 1, str(os.listdir(_imgdb.IMAGE_DIR))) + # The body limit sits outside routing, so it answers before the multipart + # parser spools anything and before the login check runs. + import app.main as _mainmod + huge = b"x" * (_mainmod.MAX_BODY_BYTES + 4096) + r = client.post(f"/api/parts/{photo_part}/images", + content=huge, headers={"Content-Type": "application/octet-stream"}) + check("an over-large body is refused outright", r.status_code == 413, str(r.status_code)) + check("and it says so as JSON", "smaller" in r.json()["detail"]) + + anon = TestClient(app) + r = anon.post(f"/api/parts/{photo_part}/images", + content=huge, headers={"Content-Type": "application/octet-stream"}) + check("an unauthenticated over-large body is refused before the login check", + r.status_code == 413, f"{r.status_code} — 401 would mean it was parsed first") + check("a normal unauthenticated upload still gets 401", + anon.post(f"/api/parts/{photo_part}/images", + files={"file": ("x.png", make_png(), "image/png")}).status_code == 401) + check("photos on a missing part 404", post_image(999999, make_png()).status_code == 404) check("fetching an unknown token 404s", @@ -498,10 +516,19 @@ with TestClient(app) as client: orphan = os.path.join(_imgdb.IMAGE_DIR, "stranded.jpg") with open(orphan, "wb") as fh: fh.write(b"\xff\xd8\xffleftover") - check("prune-images runs", _admin_img.main(["prune-images"]) == 0) + check("prune-images spares a file too recent to be sure about", + _admin_img.main(["prune-images"]) == 0 and os.path.exists(orphan)) + check("--all prunes it", _admin_img.main(["prune-images", "--all"]) == 0) check("the orphan is gone", not os.path.exists(orphan)) check("the referenced file is kept", client.get(f"/api/parts/{sweep_part}/images/{kept}").status_code == 200) + check("check-images is clean when nothing has drifted", + _admin_img.main(["check-images"]) == 0) + # A referenced photo whose file vanished is the direction that matters. + os.remove(os.path.join(_imgdb.IMAGE_DIR, os.listdir(_imgdb.IMAGE_DIR)[0])) + check("check-images reports a missing file as a failure", + _admin_img.main(["check-images"]) == 1) + check("list-image-files works", _admin_img.main(["list-image-files"]) == 0) client.delete(f"/api/parts/{sweep_part}") # --- password management through the API --- diff --git a/tests/test_concurrency.py b/tests/test_concurrency.py index e262153..a817415 100644 --- a/tests/test_concurrency.py +++ b/tests/test_concurrency.py @@ -6,6 +6,9 @@ genuinely overlapping inside SQLite, which means a real server and real threads. import http.cookiejar import json +import secrets +import struct +import zlib import os import shutil import signal @@ -42,6 +45,44 @@ def call(opener, base, path, method="GET", body=None): return json.load(opener(r, timeout=60)) +def make_png(w=40, h=40, rgb=(120, 60, 200)): + raw = b"".join(b"\x00" + bytes(rgb) * w for _ in range(h)) + + def chunk(tag, data): + body = tag + data + return struct.pack(">I", len(data)) + body + struct.pack(">I", zlib.crc32(body)) + + return (b"\x89PNG\r\n\x1a\n" + + chunk(b"IHDR", struct.pack(">IIBBBBB", w, h, 8, 2, 0, 0, 0)) + + chunk(b"IDAT", zlib.compress(raw)) + + chunk(b"IEND", b"")) + + +def multipart(content, filename="photo.png", ctype="image/png", fields=None): + """Build a multipart body by hand — no third-party HTTP client in the tests.""" + boundary = "----parts" + secrets.token_hex(8) + out = [] + for key, value in (fields or {}).items(): + out.append(f"--{boundary}\r\nContent-Disposition: form-data; " + f'name="{key}"\r\n\r\n{value}\r\n'.encode()) + out.append(f"--{boundary}\r\nContent-Disposition: form-data; " + f'name="file"; filename="{filename}"\r\n' + f"Content-Type: {ctype}\r\n\r\n".encode() + content + b"\r\n") + out.append(f"--{boundary}--\r\n".encode()) + return b"".join(out), f"multipart/form-data; boundary={boundary}" + + +def post_image(base, part_id, content, timeout=30): + body, ctype = multipart(content) + r = urllib.request.Request(f"{base}/api/parts/{part_id}/images", data=body, + method="POST", headers={"Content-Type": ctype}) + try: + with urllib.request.urlopen(r, timeout=timeout) as resp: + return resp.status + except urllib.error.HTTPError as e: + return e.code + + @contextmanager def server(port, **env_extra): """Run a real uvicorn against a throwaway database.""" @@ -62,7 +103,7 @@ def server(port, **env_extra): time.sleep(0.25) else: raise RuntimeError("server never started") - yield base + yield base, tmp finally: proc.send_signal(signal.SIGINT) try: @@ -95,7 +136,7 @@ def session_phase(): password = "concurrency-test-password" with server(PORT + 1, PARTS_AUTH="on", PARTS_PASSWORD=password, PARTS_SECRET="concurrency-test-secret", - PARTS_LOGIN_MAX_FAILURES="500") as base: + PARTS_LOGIN_MAX_FAILURES="500") as (base, _tmp): # --- the plain case first --- keeper = session_for(base, password) others = [session_for(base, password) for _ in range(5)] @@ -142,7 +183,7 @@ def session_phase(): def main(): - with server(PORT, PARTS_AUTH="off"): + with server(PORT, PARTS_AUTH="off") as (_base, tmp): # --- concurrent decrements must not lose updates --- pid = req("/api/parts", "POST", {"name": "race target", "quantity": 100})["id"] with ThreadPoolExecutor(max_workers=50) as ex: @@ -244,6 +285,55 @@ def main(): check("all concurrent creates are searchable", req("/api/parts?q=bulk&limit=100")["total"] == 25, str(req("/api/parts?q=bulk&limit=100")["total"])) + # --- concurrent photo uploads --- + # An async endpoint doing blocking SQLite work stalls the whole event + # loop under contention; an unlocked count lets every overlapping upload + # observe the same total and sail past the ceiling together. This checks + # both: the ceiling holds exactly, and the server stays responsive while + # it is being hammered. + shot_part = req("/api/parts", "POST", {"name": "photo race", "quantity": 1})["id"] + attempts = 20 + latency = [] + + def ping(): + start = time.time() + try: + call(urllib.request.urlopen, _base, "/healthz") + except Exception: + pass + latency.append(time.time() - start) + + with ThreadPoolExecutor(max_workers=attempts + 4) as ex: + futures = [ex.submit(post_image, _base, shot_part, make_png(60, 60, (i * 9, 40, 90))) + for i in range(attempts)] + for _ in range(4): + ex.submit(ping) + codes = [f.result() for f in futures] + + created = codes.count(201) + refused = codes.count(409) + check("no upload returned a server error", + not [c for c in codes if c >= 500], str(sorted(set(codes)))) + check("every upload was answered", len(codes) == attempts and None not in codes) + check("exactly the ceiling was stored", created == 12, f"{created} created, {refused} refused") + check("the rest were refused as over the limit", created + refused == attempts, + str(sorted(set(codes)))) + stored = req(f"/api/parts/{shot_part}/images")["items"] + check("the database agrees with the ceiling", len(stored) == 12, str(len(stored))) + check("positions are unique", len({i["position"] for i in stored}) == 12, + str(sorted(i["position"] for i in stored))) + + image_dir = os.path.join(tmp, "images") + on_disk = len(os.listdir(image_dir)) + check("refused uploads left no files behind", on_disk == 12, f"{on_disk} files") + + worst = max(latency) if latency else 0 + # A blocked event loop parks unrelated requests for SQLite's busy + # timeout, which is 15s. Anything in that neighbourhood means the + # upload is running on the loop again. + check("the server stayed responsive during the upload storm", + worst < 5.0, f"slowest /healthz was {worst:.1f}s") + session_phase() print() diff --git a/tools/backup.sh b/tools/backup.sh new file mode 100755 index 0000000..2dedda2 --- /dev/null +++ b/tools/backup.sh @@ -0,0 +1,48 @@ +#!/usr/bin/env bash +# Consistent backup of the parts inventory. +# +# The database and the photos are two resources that reference each other, so +# capturing them at different moments is not a backup: a photo deleted between +# the two steps leaves the saved database pointing at a file that isn't in the +# archive, and one added leaves the reverse. Neither is repairable afterwards. +# +# So the app is stopped for the few seconds it takes to copy both. With no +# process attached, parts.db and its -wal/-shm sidecars are a consistent set and +# the images directory cannot move underneath us. +# +# ./tools/backup.sh [output-directory] +set -euo pipefail + +OUT_DIR="$(cd "${1:-$PWD}" && pwd)" +cd "$(dirname "$0")/.." +STAMP="$(date +%F-%H%M%S)" +NAME="parts-backup-$STAMP.tar.gz" +VOLUME="$(docker compose config --format json 2>/dev/null \ + | python3 -c 'import json,sys; print(list(json.load(sys.stdin)["volumes"])[0])' 2>/dev/null || echo parts_data)" +VOLUME="$(docker volume ls --format '{{.Name}}' | grep -E "parts.*data" | head -1)" + +echo "Stopping the app so both halves are captured at one moment..." +docker compose stop parts >/dev/null + +restart() { docker compose start parts >/dev/null && echo "App restarted."; } +trap restart EXIT + +# What the database expects to exist, read while nothing can be writing. +EXPECTED="$(docker run --rm -v "$VOLUME":/data -e PARTS_DB=/data/parts.db \ + --entrypoint python parts -m app.admin list-image-files | sort)" + +docker run --rm -v "$VOLUME":/data -v "$OUT_DIR":/out alpine \ + tar -czf "/out/$NAME" -C /data . + +# Verify every referenced photo actually made it into the archive. +IN_TAR="$(tar -tzf "$OUT_DIR/$NAME" | sed -n 's#^\./images/##p' | sort)" +MISSING="$(comm -23 <(printf '%s\n' "$EXPECTED" | grep -v '^$' || true) \ + <(printf '%s\n' "$IN_TAR" | grep -v '^$' || true) || true)" +if [ -n "$MISSING" ]; then + echo "BACKUP INCOMPLETE — referenced photos missing from the archive:" >&2 + printf ' %s\n' $MISSING >&2 + exit 1 +fi + +COUNT="$(printf '%s\n' "$EXPECTED" | grep -c . || true)" +echo "Wrote $OUT_DIR/$NAME ($(du -h "$OUT_DIR/$NAME" | cut -f1)), $COUNT photo(s) verified present."