Hardening pass: scheduler backoff, FK cascade, a11y, test safety net
Pre-traffic cleanup from an audit: * Scheduler: poll_due_sources now keys on the last *attempt* (success or failure), not the last success, and scales the wait by the consecutive- failure streak (capped at a day). A failing feed (e.g. Phys.org's HTTP 429s) used to be retried every cycle because it had no successful run; it now backs off and recovers on its own. Extracted due_source_rows() + tests. * FK hygiene: deleting a daily_brief is supposed to cascade to its items, but SQLite enforces foreign keys per-connection — connect() already sets the pragma, so the cascade is correct going forward; added a regression test. (Orphaned items + Phys.org settings were cleaned directly on the live DB.) * a11y: modal/drawer dialogs are now focusable (tabindex), close on Escape (window) and on backdrop click via a target check (dropping the inner stopPropagation handlers). Build is warning-free. * tests: conftest points any un-mocked LLM client at a closed port with a 1s timeout, so an accidental real call fails fast instead of hanging the suite. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
+32
-11
@@ -35,14 +35,32 @@ def poll_all_sources(conn: sqlite3.Connection, limit: int | None = None) -> dict
|
||||
).fetchall(), limit)
|
||||
|
||||
|
||||
def poll_due_sources(conn: sqlite3.Connection, limit: int | None = None) -> dict:
|
||||
"""Poll only active sources whose last successful poll is older than their
|
||||
poll_interval_minutes (or that have never been polled successfully).
|
||||
# A failing source backs off so it isn't re-hit every scheduler cycle: the
|
||||
# effective wait grows with the failure streak, capped at a day. This keeps a
|
||||
# rate-limited feed (HTTP 429) resting instead of hammered, and lets it recover
|
||||
# on its own once the limit lifts.
|
||||
MAX_BACKOFF_MINUTES = 1440
|
||||
|
||||
This is what makes poll_interval_minutes meaningful and lets a scheduler run
|
||||
frequently without re-hitting feeds that are not yet due.
|
||||
|
||||
def poll_due_sources(conn: sqlite3.Connection, limit: int | None = None) -> dict:
|
||||
"""Poll only active sources whose last *attempt* (success OR failure) is
|
||||
older than their effective interval, or that have never been polled.
|
||||
|
||||
Keying on the last attempt — not the last success — is what stops a
|
||||
perpetually-failing feed from being retried every cycle. The effective
|
||||
interval is poll_interval_minutes scaled up by the consecutive-failure
|
||||
streak (capped at MAX_BACKOFF_MINUTES), so healthy feeds keep their cadence
|
||||
while broken ones step down to occasional retries.
|
||||
"""
|
||||
rows = conn.execute(
|
||||
return _poll_rows(conn, due_source_rows(conn), limit)
|
||||
|
||||
|
||||
def due_source_rows(conn: sqlite3.Connection) -> list[sqlite3.Row]:
|
||||
"""Active sources currently due to poll (see poll_due_sources for the rule).
|
||||
|
||||
Split out so the due/backoff decision can be tested without the network.
|
||||
"""
|
||||
return conn.execute(
|
||||
"""
|
||||
SELECT s.*
|
||||
FROM sources s
|
||||
@@ -50,17 +68,20 @@ def poll_due_sources(conn: sqlite3.Connection, limit: int | None = None) -> dict
|
||||
AND (
|
||||
NOT EXISTS (
|
||||
SELECT 1 FROM ingest_runs r
|
||||
WHERE r.source_id = s.id AND r.status = 'ok'
|
||||
WHERE r.source_id = s.id AND r.finished_at IS NOT NULL
|
||||
)
|
||||
OR (
|
||||
SELECT MAX(r.finished_at) FROM ingest_runs r
|
||||
WHERE r.source_id = s.id AND r.status = 'ok'
|
||||
) <= datetime('now', '-' || s.poll_interval_minutes || ' minutes')
|
||||
WHERE r.source_id = s.id AND r.finished_at IS NOT NULL
|
||||
) <= datetime(
|
||||
'now',
|
||||
'-' || MIN(?, s.poll_interval_minutes * (1 + s.consecutive_failures)) || ' minutes'
|
||||
)
|
||||
)
|
||||
ORDER BY s.id
|
||||
"""
|
||||
""",
|
||||
(MAX_BACKOFF_MINUTES,),
|
||||
).fetchall()
|
||||
return _poll_rows(conn, rows, limit)
|
||||
|
||||
|
||||
def _poll_rows(conn: sqlite3.Connection, rows: list[sqlite3.Row], limit: int | None) -> dict:
|
||||
|
||||
Reference in New Issue
Block a user