thejayman77
2dbe73430c
The Articles inspector revealed paywall is domain-coarse: nytimes.com is flagged, so NY Times Learning's free Word-of-the-Day inherits 🔒 — and that flag isn't cosmetic, it deprioritizes the content in feed sort + lead selection. Add a per-source override so admins can correct it after inspecting. - sources.paywall_override: NULL (domain rule) | 'free' | 'paywalled'. - paywall.py: keep low-level is_paywalled(url) (domain); add is_paywalled_for_source (url, override) for the EFFECTIVE decision — never patched the domain helper globally (per Codex), so "domain says X" stays distinguishable from "overridden". - Threaded everywhere ranking/UI touches paywall, via src.paywall_override on the shared _ARTICLE_COLUMNS + the source-aware helper: feed sort, /api/since, replace, lead selection, Article badge, brief composition (briefs.py), digest, source_health (table 🔒), the Articles inspector, and the review/attention check — so ranking and UI always agree. - Endpoint POST /api/admin/sources/{id}/paywall {override}; admin UI: a select in the inspector header (Use domain rule / Treat as free / Treat as paywalled) + the basis ("ON (domain)" / "OFF (override)"), optimistic so the panel stays open. Test: domain rule → paywalled in table+inspector+feed badge; 'free' → off in all three; validation 422 + 404. 242 pytest + 11 vitest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
56 lines
1.8 KiB
Python
56 lines
1.8 KiB
Python
"""Domain-level paywall hints.
|
|
|
|
We never fetch article pages, so a paywall can only be inferred from the host.
|
|
This is a curated, conservative list of hard/soft paywalls — enough to label a
|
|
card "subscription may be required" and to prefer readable stories for the lead
|
|
and for replacements. It will never be perfect; it's an honest hint, not a gate.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from urllib.parse import urlsplit
|
|
|
|
# Host suffixes considered paywalled. Subdomains match (news.nature.com → nature.com).
|
|
PAYWALL_DOMAINS = {
|
|
"newscientist.com",
|
|
"nature.com",
|
|
"nytimes.com",
|
|
"wsj.com",
|
|
"ft.com",
|
|
"economist.com",
|
|
"wired.com",
|
|
"theatlantic.com",
|
|
"washingtonpost.com",
|
|
"bloomberg.com",
|
|
"technologyreview.com",
|
|
"newyorker.com",
|
|
"scientificamerican.com",
|
|
"nationalgeographic.com",
|
|
"thetimes.co.uk",
|
|
"telegraph.co.uk",
|
|
"foreignpolicy.com",
|
|
"hbr.org",
|
|
"harpers.org",
|
|
}
|
|
|
|
|
|
def is_paywalled(url: str | None) -> bool:
|
|
"""Low-level DOMAIN rule. Keep this distinct from the source-aware decision so
|
|
callers can tell 'domain says paywalled' from 'this source is overridden'."""
|
|
host = urlsplit(url or "").netloc.lower()
|
|
if host.startswith("www."):
|
|
host = host[4:]
|
|
return any(host == d or host.endswith("." + d) for d in PAYWALL_DOMAINS)
|
|
|
|
|
|
def is_paywalled_for_source(url: str | None, override: str | None = None) -> bool:
|
|
"""The EFFECTIVE paywall decision used for ranking/lead/badges: a per-source
|
|
override (set in admin after inspecting the articles) wins over the domain
|
|
rule — 'free' clears a false positive (e.g. NY Times Learning), 'paywalled'
|
|
flags a false negative. NULL falls back to the domain rule."""
|
|
if override == "free":
|
|
return False
|
|
if override == "paywalled":
|
|
return True
|
|
return is_paywalled(url)
|