Perf: skip needless dedup re-cluster + interlock word-search grids
Two things found while chasing the recurring ~15min slowness: - dedup.py: cluster_duplicates re-ran an O(n²) cosine pass over ALL ~3.7k articles and rewrote duplicate_of for every one of them EVERY cycle — even when nothing new arrived (embedded=0) — ~53s CPU + a large WAL commit that starved live API reads (/api/brief 2-7s). Now skip the re-cluster entirely when nothing new was embedded (clusters can't have changed). Verified: cycle drops from ~53s to ~1s and /api/brief stays at 20ms through a cycle, vs 2-7s before. (A real new article still triggers a full re-cluster.) - games.py _build_grid: word placement took the first random valid spot, so words rarely crossed. Now gather valid placements and PREFER ones that cross an already-placed word (shared matching letter), falling back to any valid spot — so the grid interlocks like a real word search. Every word still placed (tests green). NOTE: changes today's grid layouts, so an in-progress word search resets once. Also added a systemd drop-in (Nice=19/CPUWeight=20/IOWeight=10/ionice-idle) to deprioritize the batch cycle — minor, the dedup skip is the real fix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -166,6 +166,16 @@ def dedup(
|
||||
embed_limit: int | None = None,
|
||||
) -> dict:
|
||||
embedded = ensure_embeddings(conn, client, limit=embed_limit)
|
||||
if embedded == 0:
|
||||
# Nothing new entered the corpus → the clusters and duplicate_of links are
|
||||
# unchanged, so skip the full re-cluster. It was re-running an O(n²) cosine
|
||||
# pass over EVERY article and rewriting duplicate_of for all ~3.7k of them
|
||||
# every cycle (~53s + a large WAL commit), which starved live API reads
|
||||
# (/api/brief 2-7s). Most cycles find no new articles, so this makes the
|
||||
# cycle near-instant and keeps reads fast. A real new article re-runs it.
|
||||
dups = conn.execute("SELECT COUNT(*) FROM articles WHERE duplicate_of IS NOT NULL").fetchone()[0]
|
||||
return {"embedded": 0, "articles": 0, "clusters": 0, "duplicate_clusters": 0,
|
||||
"duplicates": dups, "skipped": True}
|
||||
stats = cluster_duplicates(conn, threshold=threshold, window_days=window_days)
|
||||
stats["embedded"] = embedded
|
||||
return stats
|
||||
|
||||
Reference in New Issue
Block a user