Sources: LLM deep-preview, source search, duplicate-add guard
Three admin Sources upgrades: - Deep preview: a per-candidate "🔬 Deep preview" button runs the REAL classifier on an 8-item sample (the same model that judges live articles), versus the fast keyword heuristic the add/Re-preview path uses. Preview now carries `classified`, surfaced as a "model-checked" vs "quick estimate" badge — so the acceptance % is no longer ambiguously heuristic. conn is released during the ~30-60s model pass; postJSON has no client timeout. - Search: free-text box over the sources table (name / category / feed URL / homepage), folded into the existing status filter, with a live match count and empty state. Makes "is this already added?" a glance. - Duplicate-add guard: sources.find_existing_feed() + feed_key() normalize scheme/www/trailing-slash/case, so re-adding a feed that's already a live source or a queued candidate is refused with a 409 naming where it lives (DB already enforced exact-URL uniqueness; this catches the near-miss variants and overwrite-on-promote footgun). Tests: test_candidate_deep_preview_and_dedup (deep flag wires the model + uses the small sample; exact/www/slash/case variants all 409). 224 pytest + 11 vitest green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -67,6 +67,37 @@ def upsert_sources(conn: sqlite3.Connection, source_defs: list[dict]) -> int:
|
||||
return count
|
||||
|
||||
|
||||
# --- Duplicate detection (catch the same feed added twice) --------------------
|
||||
|
||||
|
||||
def feed_key(url: str) -> str:
|
||||
"""A loose comparison key for spotting the same feed added twice despite
|
||||
trivial differences (scheme, www, trailing slash, case). Compare-only — the
|
||||
feed_url is always STORED exactly as entered; this just powers dup warnings."""
|
||||
try:
|
||||
p = urlsplit((url or "").strip().lower())
|
||||
host = p.netloc.removeprefix("www.")
|
||||
path = p.path.rstrip("/")
|
||||
return host + path + (("?" + p.query) if p.query else "")
|
||||
except Exception: # noqa: BLE001 — never let a weird URL break add
|
||||
return (url or "").strip().lower()
|
||||
|
||||
|
||||
def find_existing_feed(conn: sqlite3.Connection, url: str) -> dict | None:
|
||||
"""Is this feed already a live source or a pending candidate? Matches on the
|
||||
loose key, so http/https + www + trailing-slash variants are all caught."""
|
||||
key = feed_key(url)
|
||||
for r in conn.execute("SELECT id, name, feed_url, status FROM sources"):
|
||||
if feed_key(r["feed_url"]) == key:
|
||||
return {"kind": "source", "id": r["id"], "name": r["name"], "status": r["status"]}
|
||||
for r in conn.execute(
|
||||
"SELECT id, name, feed_url, status FROM source_candidates WHERE status NOT IN ('rejected','promoted')"
|
||||
):
|
||||
if feed_key(r["feed_url"]) == key:
|
||||
return {"kind": "candidate", "id": r["id"], "name": r["name"] or r["feed_url"], "status": r["status"]}
|
||||
return None
|
||||
|
||||
|
||||
# --- Supervised source candidates (staging before the real sources table) ----
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user