upbeatBytes/README.md

# goodNews

Local-first constructive news ingestion prototype.

The first milestone is intentionally small: collect public RSS/Atom metadata, dedupe it, store short source-provided snippets, and attach early reason-coded heuristic scores. It does not store full article bodies.

## Commands

From this directory:

```bash
python3 -m goodnews init-db
python3 -m goodnews import-sources
python3 -m goodnews poll --limit 3
python3 -m goodnews rescore
python3 -m goodnews check-llm --base-url http://127.0.0.1:1234/v1 --model gpt-oss
python3 -m goodnews classify --limit 10 --base-url http://127.0.0.1:1234/v1 --model gpt-oss
python3 -m goodnews dedup --base-url http://127.0.0.1:1234/v1
python3 -m goodnews check-feeds
python3 -m goodnews build-brief --date 2026-05-27 --replace
python3 -m goodnews show-brief
python3 -m goodnews list-recent --limit 10
python3 -m goodnews list-recent --accepted-only --limit 10
python3 -m goodnews list-category --topic animals --flavor discovery
python3 -m goodnews list-category --topic environment --flavor solution
python3 -m goodnews source-report
python3 -m goodnews list-runs
```

The SQLite database lives at:

```txt
data/goodnews.sqlite3
```

Sources live at:

```txt
config/sources.toml
```

## Categories

When classified by the local model, each article is tagged with one **topic**
and one **flavor**, allowing browsable category feeds (e.g. "feel-good animals",
"environment solutions") via `list-category`:

- **Topics:** science, environment, health, community, culture, animals
- **Flavors:** breakthrough, discovery, solution, feelgood, perspective

The allowed values live in `goodnews/taxonomy.py`. The accept/reject gate is kept
deliberately broad ("not dreary"); ranking and category filters do the curation.

## Deduplication

Two layers:

- **Exact**: a URL hash UNIQUE constraint drops the literal same link at ingest.
- **Semantic**: `dedup` embeds each article's title+snippet with the local
  embedding model, clusters near-identical stories within a few-day window
  (cosine similarity), and marks all but the highest-ranked in each cluster as
  `duplicate_of` the representative. Feed and brief queries hide duplicates, so
  the same story carried by several outlets appears once. This runs as part of
  `cycle`, so the scheduler keeps the corpus deduped automatically.

## Stored Article Data

For each article, the database stores:

- source
- canonical URL
- title
- short RSS/Atom description or summary
- author, if present
- published timestamp, if present
- image URL, if present
- language, if present
- hashes used for dedupe
- heuristic scores and reason codes

## Web / API

The optional `web` extra adds a FastAPI service and a small static site that
consumes it. The same JSON API backs both the website and any future companion
app; its auto-generated OpenAPI docs at `/docs` are the shared contract.

```bash
pip install -e '.[web]'          # or: .venv/bin/pip install -e '.[web]'
python3 -m goodnews serve                  # http://127.0.0.1:8000
python3 -m goodnews serve --host 0.0.0.0   # expose on the network
```

Endpoints:

- `GET /` — the static site (daily five + topic/flavor browsing)
- `GET /healthz` — liveness + scored-article count
- `GET /api/categories` — the topic/flavor taxonomy
- `GET /api/category-counts` — article counts per topic/flavor
- `GET /api/feed?topic=&flavor=&limit=&offset=` — ranked, filtered articles
- `GET /api/brief?date=&limit=` — a daily brief (latest if no date)
- `GET /api/brief-dates` — available brief dates
- `GET /docs` — interactive OpenAPI documentation

The ingestion CLI stays pure-stdlib; only the `web` extra pulls in FastAPI/uvicorn,
so the two halves can be deployed and upgraded independently.

## Deployment

The database is never baked into the image — the API and the ingestion CLI share
one SQLite file via a mounted volume. Run ingestion (`poll`, `classify`,
`build-brief`) on a schedule against the same file.

```bash
docker build -t goodnews .
docker run -p 8000:8000 -v /srv/goodnews/data:/data goodnews
```

`GOODNEWS_DB` controls the database path (defaults to `data/goodnews.sqlite3`).
Put a reverse proxy (Caddy/nginx) in front for TLS once a domain is attached.

## Scheduling

A single idempotent command runs the whole pipeline and is safe to invoke as
often as you like — it only polls sources that are *due* (per each source's
`poll_interval_minutes`), only classifies articles the model hasn't seen, and
rebuilds the current day's brief:

```bash
python3 -m goodnews cycle                 # poll due -> classify new -> dedup -> rebuild today's brief
python3 -m goodnews cycle --force         # poll every active source regardless of interval
python3 -m goodnews cycle --no-classify   # skip the LLM step (e.g. model box offline)
```

A systemd timer runs it every 15 minutes. Unit files live in `deploy/`:

```bash
sudo install -d /etc/goodnews
sudo install -m 644 deploy/goodnews.env.example /etc/goodnews/goodnews.env  # then edit
sudo install -m 644 deploy/goodnews.service deploy/goodnews.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now goodnews.timer

systemctl list-timers goodnews.timer          # when it next runs
journalctl -u goodnews.service -f             # watch cycle output
```

`/etc/goodnews/goodnews.env` supplies `GOODNEWS_LLM_BASE_URL`, `GOODNEWS_LLM_MODEL`,
and `GOODNEWS_DB` to the scheduled run. The timer uses `Persistent=true`, so a
run missed while the machine was off is caught up on the next boot.

## Next Steps

1. Run the poller for a few days and inspect which sources produce useful candidates.
2. Add source-level quality notes and deactivate noisy feeds.
3. Replace or supplement `heuristic-v0` with a local model classifier.
4. Add a daily brief builder that selects 5 items using scores and source diversity.
5. Add a small web/API layer once the ingest data looks trustworthy.

## Local Model Configuration

The `classify` command expects an OpenAI-compatible local chat-completions server.

You can pass settings directly:

```bash
python3 -m goodnews classify --base-url http://127.0.0.1:1234/v1 --model gpt-oss --limit 10
```

Or use environment variables:

```bash
export GOODNEWS_LLM_BASE_URL=http://127.0.0.1:1234/v1
export GOODNEWS_LLM_MODEL=gpt-oss
python3 -m goodnews classify --limit 10
```

`classify` rewrites the current score/reason row for selected candidates. `rescore` can restore the fast heuristic scores.