retroDE_ps2/docs/decisions/0011-gs-dump-ingestion.md

0011 — GS dump ingestion (Ch340): parse, census, translate a supported subset

Status: ACCEPTED — Ch340 CLOSED (2026-06-21) as a parser + census / fail-closed victory. Brick 5 (authentic on-silicon render) is explicitly waived because the authentic dump contains zero supported segments — that is the census doing its job, not a failure.

Closeout (honest framing, per Codex)

• Authentic cubes_demo GS dump (MIT homebrew, content-clean) parsed deterministically, 0 malformed.
• Container format pinned from PCSX2 source and validated byte-exact; byte-exact synthetic parser test passes (tools/test_gs.sh).
• Primitive reconstruction (GS vertex-kick model) works: 648 triangles + 540 sprites.
• Support census classified every primitive; histograms + reasons emitted to captures/gs/reports/cubes.census.txt (aggregate only).
• Translator failed closed with no scene: every authentic triangle is textured (TME=1, no real- texture path) and sprites are unsupported. Nothing was approximated.
• Core trust-boundary goal achieved: authentic GS traffic enters the pipeline and unsupported content is reported, not faked. The translator→ps2_feeder→staging path is proven on the supported synthetic fixture; authentic silicon render is deferred to the census-derived blocker.
• Mechanical top blocker → Ch341: textured-triangle ingestion (real texture state/upload/bind). Do NOT hunt another dump for a convenient flat segment; do NOT substitute synthetic silicon for authentic Brick 5.

Original design follows (the boundary it set still holds).

Goal & trust boundary

Authentic GS traffic enters the proven host pipeline, is decoded deterministically, and a strictly-supported subset reaches pixels with no hidden approximation. Ch340's honest victory is that property — NOT a full game frame rendering. Real captures will expose unsupported textures, transfers, blend/state, and primitive modes; those are reported, never approximated.

Pipeline

.gs[.xz]  ──(container parser)──►  raw packets  ──(GIF/GS decoder)──►  normalized event stream
   │                                                                          │
   │ (local, gitignored)                                                      ▼
   └────────────────────────────────────────────────►  support census + histograms (reports/)
                                                                              │
                                              supported subset ──(translator)─┴─►  ps2_feeder scene file
                                                                                    (Ch339 encoder streams it)

The translator emits the Ch339 text scene grammar (tri/trig/tritile/rect/go); it does NOT re-implement the staging format. ps2_feeder -f scene.txt renders it on the existing bitstream.

Normalized event stream (schema v1, versioned)

A flat, ordered list; every event carries byte_off, frame_idx, event_idx. Event kinds:

• FRAME_BOUNDARY {field} — VSync packet (frame delimiter).
• GIFTAG {path, nloop, eop, pre, prim, flg, nreg, regs} — decoded GIF tag header.
• GSREG {addr, name, value} — a GS register write (from A+D, REGLIST, or PACKED expansion).
• IMAGE {qwc, dst_fmt} — an IMAGE-mode (HWREG/texture/FB upload) transfer; payload bytes summarized, NOT inlined into committable output.
• READFIFO {qwc} — local→host transfer.
• MALFORMED {reason} — structural decode failure at this offset.

GSREG.name covers the register set we already encode in bake.py/ps2_feeder: PRIM, RGBAQ, ST, UV, XYZ2/XYZ3, XYZF2/XYZF3, TEX0_1/2, CLAMP, FOG, TEX1/2, FRAME_1/2, ZBUF_1/2, TEST_1/2, ALPHA_1/2, SCISSOR, PRMODE/PRMODECONT, BITBLTBUF, TRXPOS, TRXREG, TRXDIR, etc. Unknown addrs → GSREG with name="UNKNOWN_0xNN" (reported, not dropped).

Support census (every event classified)

• translated — emitted into the ps2_feeder scene (the supported subset, below).
• ignored (justified) — safely skippable with a stated reason (e.g. FOG with FGE off; a redundant state write; a NOP). The justification is explicit per category.
• unsupported — a real effect we cannot faithfully reproduce yet (textured prim, sprite with a real texture, blend mode ≠ the proven source-over, PSM we don't render, Z format, dest-alpha test, scissor we don't honor, TRIANGLE_FAN/STRIP we haven't reduced, lines/points, IMAGE texture upload). Recorded with frame/event/byte offset + the exact reason. Never approximated.
• malformed — structural failure (bad GIF tag, truncated payload, length mismatch).

Reports (committable, no game content): per-dump JSON/text with frame count, a GS-register-write histogram, a primitive-mode histogram, and the full unsupported/malformed list with offsets+reasons.

Supported subset that reaches pixels (Ch340 v1)

Matches what the feeder + ps2_feeder already render faithfully on silicon:

• PRIM = TRIANGLE (prim type 3), with IIP flat or gouraud (per-vertex RGBAQ).
• Vertices via XYZ2/XYZ3 (Z honored — Ch338 cross-batch Z is correct).
• Color via RGBAQ (MODULATE through the unity texel — matches the proven path).
• FRAME_1/ZBUF_1/TEST_1(GEQUAL)/ALPHA_1(the proven source-over) within the supported envelope. A draw segment qualifies only if EVERY primitive in it is in this subset and the state matches the proven envelope. Sprites→rect and triangle-strip→triangle reduction are candidate Ch341 work, decided from the census, not pre-built.

Acceptance (Codex)

1. Byte-exact parser tests on a small synthetic .gs fixture (captures/gs/synthetic/, authored once the real container format is confirmed).
2. One authentic dump parses deterministically (same events every run).
3. Frame / register / primitive histograms emitted.
4. Unsupported events carry frame/event/byte offset + reason.
5. ≥1 carefully chosen supported frame/segment translates to a ps2_feeder scene and renders on silicon.
6. Translation failures stop before board access.

Bricks (gated on the dump)

1. Inspect the real dump bytes; confirm the container framing (header, compression, packet types). Build the container parser + a matching synthetic fixture; byte-exact parser test.
2. GIF/GS decoder → normalized events (GIF tag + A+D/REGLIST/PACKED register expansion). Unit-tested against hand-built GIF packets (the encode side already exists in bake.py).
3. Census + histograms over the normalized stream; emit reports.
4. Translator: supported-subset events → ps2_feeder scene file; everything else → census. Fail closed.
5. Pick a supported segment from the census; render via ps2_feeder -f; confirm on silicon.

Ch341 is then chosen from the census's highest-impact blocker, not guessed now.