Add semantic cross-source dedup via local embeddings
- LocalModelClient.embed() calls the OpenAI-compatible /embeddings endpoint (local nomic model); base_url shared with chat, model via GOODNEWS_EMBED_MODEL. - New article_embeddings table and articles.duplicate_of column (+ migration). - dedup module: embeds missing articles, clusters near-identical stories within a date window by cosine similarity (pure-stdlib, vectors normalised once), and marks all but the highest-ranked member of each cluster as a duplicate. - 'dedup' CLI command; cycle now runs poll -> classify -> dedup -> brief. - Feed and brief queries hide duplicates, so a story carried by multiple outlets shows once. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
+1
-1
@@ -49,7 +49,7 @@ def feed(
|
||||
offset: int = 0,
|
||||
) -> list[dict]:
|
||||
"""Return ranked articles, optionally filtered by topic and/or flavor."""
|
||||
clauses = []
|
||||
clauses = ["a.duplicate_of IS NULL"]
|
||||
params: list = []
|
||||
if accepted_only:
|
||||
clauses.append("s.accepted = 1")
|
||||
|
||||
Reference in New Issue
Block a user