Add semantic cross-source dedup via local embeddings
- LocalModelClient.embed() calls the OpenAI-compatible /embeddings endpoint (local nomic model); base_url shared with chat, model via GOODNEWS_EMBED_MODEL. - New article_embeddings table and articles.duplicate_of column (+ migration). - dedup module: embeds missing articles, clusters near-identical stories within a date window by cosine similarity (pure-stdlib, vectors normalised once), and marks all but the highest-ranked member of each cluster as a duplicate. - 'dedup' CLI command; cycle now runs poll -> classify -> dedup -> brief. - Feed and brief queries hide duplicates, so a story carried by multiple outlets shows once. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -118,6 +118,7 @@ def _candidate_articles(
|
||||
JOIN sources src ON src.id = a.source_id
|
||||
JOIN article_scores s ON s.article_id = a.id
|
||||
WHERE s.accepted = 1
|
||||
AND a.duplicate_of IS NULL
|
||||
AND date(COALESCE(a.published_at, a.discovered_at)) <= date(?)
|
||||
AND date(COALESCE(a.published_at, a.discovered_at)) > date(?, '-' || ? || ' days')
|
||||
AND a.id NOT IN (
|
||||
|
||||
Reference in New Issue
Block a user