Sources: LLM deep-preview, source search, duplicate-add guard

Three admin Sources upgrades:
- Deep preview: a per-candidate "🔬 Deep preview" button runs the REAL
  classifier on an 8-item sample (the same model that judges live articles),
  versus the fast keyword heuristic the add/Re-preview path uses. Preview now
  carries `classified`, surfaced as a "model-checked" vs "quick estimate"
  badge — so the acceptance % is no longer ambiguously heuristic. conn is
  released during the ~30-60s model pass; postJSON has no client timeout.
- Search: free-text box over the sources table (name / category / feed URL /
  homepage), folded into the existing status filter, with a live match count
  and empty state. Makes "is this already added?" a glance.
- Duplicate-add guard: sources.find_existing_feed() + feed_key() normalize
  scheme/www/trailing-slash/case, so re-adding a feed that's already a live
  source or a queued candidate is refused with a 409 naming where it lives
  (DB already enforced exact-URL uniqueness; this catches the near-miss
  variants and overwrite-on-promote footgun).

Tests: test_candidate_deep_preview_and_dedup (deep flag wires the model +
uses the small sample; exact/www/slash/case variants all 409). 224 pytest +
11 vitest green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
jay
2026-06-11 21:19:15 -04:00
parent ba1a29d12a
commit e1ac19351e
4 changed files with 126 additions and 11 deletions
+31
View File
@@ -67,6 +67,37 @@ def upsert_sources(conn: sqlite3.Connection, source_defs: list[dict]) -> int:
return count
# --- Duplicate detection (catch the same feed added twice) --------------------
def feed_key(url: str) -> str:
"""A loose comparison key for spotting the same feed added twice despite
trivial differences (scheme, www, trailing slash, case). Compare-only — the
feed_url is always STORED exactly as entered; this just powers dup warnings."""
try:
p = urlsplit((url or "").strip().lower())
host = p.netloc.removeprefix("www.")
path = p.path.rstrip("/")
return host + path + (("?" + p.query) if p.query else "")
except Exception: # noqa: BLE001 — never let a weird URL break add
return (url or "").strip().lower()
def find_existing_feed(conn: sqlite3.Connection, url: str) -> dict | None:
"""Is this feed already a live source or a pending candidate? Matches on the
loose key, so http/https + www + trailing-slash variants are all caught."""
key = feed_key(url)
for r in conn.execute("SELECT id, name, feed_url, status FROM sources"):
if feed_key(r["feed_url"]) == key:
return {"kind": "source", "id": r["id"], "name": r["name"], "status": r["status"]}
for r in conn.execute(
"SELECT id, name, feed_url, status FROM source_candidates WHERE status NOT IN ('rejected','promoted')"
):
if feed_key(r["feed_url"]) == key:
return {"kind": "candidate", "id": r["id"], "name": r["name"] or r["feed_url"], "status": r["status"]}
return None
# --- Supervised source candidates (staging before the real sources table) ----