Reliability: slow ≠ failed — SW nav timeout, slow-boot telemetry, de-bot stats

Root cause of the intermittent white screen: the shell HTML is no-cache
(cf-cache-status: DYNAMIC), so every page-open does a synchronous round-trip
to the residential origin before any pixel renders — and the SW's network-first
navigation only fell back to the cached shell on REJECTION, never on slowness.
A stalled fetch meant staring at white with a perfectly good shell in cache.
The boot seatbelt couldn't see it either: it lives inside the HTML that hadn't
arrived yet, so slow boots left no telemetry.

- service-worker: race navigation fetch vs 2.5s grace timer. Network wins →
  fresh HTML as before; timer/5xx/failure → cached shell instantly, network
  response still refreshes the cache in the background. Safe due to the 14-day
  immutable-chunk grace window. Caps the white screen at ~2.5s for repeat
  visitors on any network.
- app.html: beacon `boot-slow: Nms (html Nms) on 4g` when mount takes >4s —
  the "white screen, then it loaded" glitches finally leave a trace, with
  HTML-arrival timing to separate slow-origin from slow-JS.
- admin: bot UAs (HeadlessChrome/bot/spider/crawl/…) excluded from the
  headline "Load errors today" count — throttled crawlers trip the 10s boot
  check routinely (the one recorded error was HeadlessChrome on X11, not a
  phone). Bots stay visible in the list, tagged + dimmed.

Tests: telemetry test extended for bot flag + filtered counts. 223 pytest +
11 vitest green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

goodnews/queries.py

@@ -13,6 +13,18 @@ from datetime import UTC, datetime, timedelta
 from .feeds import MAX_BACKOFF_MINUTES
 from .paywall import is_paywalled
 
+# UA substrings that mark automated clients. Crawlers run JS on a throttled
+# budget and trip the boot-failure beacon routinely — without this filter they
+# read as real users seeing blank screens.
+BOT_UA_MARKS = ("headlesschrome", "bot", "spider", "crawl", "python", "curl", "wget", "phantomjs")
+_NOT_BOT_SQL = " AND ".join(f"instr(lower(user_agent), '{m}')=0" for m in BOT_UA_MARKS)
+
+
+def is_bot_ua(ua: str | None) -> bool:
+    low = (ua or "").lower()
+    return any(m in low for m in BOT_UA_MARKS)
+
+
 # Composite ranking used everywhere a "best first" order is needed. Kept as one
 # expression so brief, category feeds, and the API all rank identically.
 RANK_SCORE_SQL = (
@@ -565,10 +577,17 @@ def admin_stats(conn: sqlite3.Connection, days: int = 30) -> dict:
         "top_topics": top_topics,
         "shares": shares,
         "daily": daily,
-        # Boot-failure seatbelt signal — blank-screen risk surfacing.
+        # Boot-failure seatbelt signal — blank-screen risk surfacing. Bots are
+        # excluded from the headline counts: throttled crawlers fail the boot
+        # check routinely and would read as real users seeing blank screens.
         "client_errors": {
-            "today": scalar("SELECT COUNT(*) FROM client_errors WHERE date(created_at)=date('now')"),
-            "window": scalar("SELECT COUNT(*) FROM client_errors WHERE created_at>=date('now',?)", (since,)),
+            "today": scalar(
+                f"SELECT COUNT(*) FROM client_errors WHERE date(created_at)=date('now') AND {_NOT_BOT_SQL}"
+            ),
+            "window": scalar(
+                f"SELECT COUNT(*) FROM client_errors WHERE created_at>=date('now',?) AND {_NOT_BOT_SQL}",
+                (since,),
+            ),
         },
     }