Sources: per-source paywall override (3-state) — fix domain-rule mis-flags
The Articles inspector revealed paywall is domain-coarse: nytimes.com is flagged, so NY Times Learning's free Word-of-the-Day inherits 🔒 — and that flag isn't cosmetic, it deprioritizes the content in feed sort + lead selection. Add a per-source override so admins can correct it after inspecting. - sources.paywall_override: NULL (domain rule) | 'free' | 'paywalled'. - paywall.py: keep low-level is_paywalled(url) (domain); add is_paywalled_for_source (url, override) for the EFFECTIVE decision — never patched the domain helper globally (per Codex), so "domain says X" stays distinguishable from "overridden". - Threaded everywhere ranking/UI touches paywall, via src.paywall_override on the shared _ARTICLE_COLUMNS + the source-aware helper: feed sort, /api/since, replace, lead selection, Article badge, brief composition (briefs.py), digest, source_health (table 🔒), the Articles inspector, and the review/attention check — so ranking and UI always agree. - Endpoint POST /api/admin/sources/{id}/paywall {override}; admin UI: a select in the inspector header (Use domain rule / Treat as free / Treat as paywalled) + the basis ("ON (domain)" / "OFF (override)"), optimistic so the panel stays open. Test: domain rule → paywalled in table+inspector+feed badge; 'free' → off in all three; validation 422 + 404. 242 pytest + 11 vitest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
+3
-3
@@ -7,7 +7,7 @@ from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from .paywall import is_paywalled
|
||||
from .paywall import is_paywalled, is_paywalled_for_source
|
||||
|
||||
|
||||
def load_sources(path: Path | str) -> list[dict]:
|
||||
@@ -244,7 +244,7 @@ def review_sources(
|
||||
now = datetime.now(timezone.utc)
|
||||
flagged = []
|
||||
sources = conn.execute(
|
||||
"SELECT id, name, consecutive_failures FROM sources WHERE active = 1"
|
||||
"SELECT id, name, consecutive_failures, paywall_override FROM sources WHERE active = 1"
|
||||
).fetchall()
|
||||
|
||||
for s in sources:
|
||||
@@ -292,7 +292,7 @@ def review_sources(
|
||||
avg_rage = sum(r["ragebait_score"] or 0 for r in recent) / n
|
||||
if avg_rage > 3:
|
||||
reasons.append(f"high ragebait (avg {avg_rage:.1f})")
|
||||
paywalled = sum(1 for r in recent if is_paywalled(r["canonical_url"])) / n
|
||||
paywalled = sum(1 for r in recent if is_paywalled_for_source(r["canonical_url"], s["paywall_override"])) / n
|
||||
if paywalled > 0.5:
|
||||
reasons.append(f"paywall-heavy ({paywalled * 100:.0f}%)")
|
||||
|
||||
|
||||
Reference in New Issue
Block a user