trikkeSensors/docs/binary-transport-validation-2026-08-17.md

Binary Transport Validation — 2026-08-17

This milestone replaced high-volume device-side CSV with the versioned TRK1 binary stream shared by future USB, BLE, and storage paths. Mapped raw sensor counts remain authoritative. Calibration metadata travels as float32 values, and the host reconstructs the prior 23-column diagnostic CSV without discarding raw data.

Verification layers

• Host compilation of the production C encoder with -Wall -Wextra -Werror
• Fragmented C-encoder-to-Python-parser contract test
• Deliberately corrupted CRC test with stream resynchronization
• Full eight-record, timestamp-saturation, invalid-size/count, and truncated-tail encoder/parser contract cases
• ESP-IDF firmware build and flash on the assembled ESP32-C3 prototype
• Live USB capture followed by independent offline re-decoding
• Tracked three- and seven-second hardware-outage captures with regression assertions for integrity, retention order, and overflow accounting

USB text-conversion finding

The first live capture exposed that the USB console's default CRLF mode inserted a carriage return whenever a binary byte equaled LF (0x0A). CRC rejected every affected frame and the parser resynchronized at the next TRK1 magic. No invalid sample entered decoded CSV.

Before binary output begins, firmware now changes the USB Serial/JTAG VFS transmit mode to ESP_LINE_ENDINGS_LF, which means no byte modification. Startup logs and readable metadata are flushed first.

USB VFS resilience qualification

The 5.12-second queue guarantee applies when the transport reports failure or backpressure accurately. The current ESP-IDF USB Serial/JTAG VFS/stdio path surfaces a physical disconnect, but it has a weaker connected-stall case: when a host remains connected and stops draining, the lower-level timeout can discard bytes even though stdio reports a successful write. Firmware therefore cannot retain that particular frame or increment its drop counter. The receiver still detects the loss through CRC resynchronization and packet/sample sequence gaps.

ESP-IDF's direct USB Serial/JTAG driver provides bounded writes and an explicit transmit-drain wait, allowing a connected stall to become observable to the transport policy. That is a useful improvement for the common transport layer. It is not proof of receiver delivery; application acknowledgements and replay are needed for that stronger guarantee and are planned with BLE integration.

Final hardware capture

captures/binary_v1_smoke2.trk and its decoded CSV contain:

• 7,184 samples over 71.830 seconds
• 912 total frames, including 14 repeated metadata frames
• Packet gaps and resets: 0
• Sample gaps and resets: 0
• Timestamp anomalies: 0
• CRC and header failures: 0
• Queue/read/output drops: 0
• Acquisition-loop overruns: 0
• ADXL345 overruns: 0
• L3G4200D data-ready clear: 0
• L3G4200D overruns: 192

Offline decoding of the saved .trk file produced CSV byte-for-byte identical to the CSV rendered during live capture.

The validated stream occupied 177,184 bytes, or 2,466.7 bytes/s including frame headers and repeated metadata. That is 8.47 MiB/hour and about 19.7 kbit/s before BLE link overhead, far below the previous CSV stream.

Task separation and buffer

The 100 Hz I2C acquisition runs at FreeRTOS priority 10. USB encoding/output runs at priority 5 and receives samples through a 512-entry queue (about 5.12 seconds at 100 Hz). The hardware capture's zero timing anomalies and zero loop overruns confirm that packet encoding, CRC, float metadata, and USB output did not disturb the acquisition cadence.

When output failure is reported, handling is transactional: firmware retains and retries the same encoded packet with a scheduler delay instead of discarding it or dequeuing more samples. The queue therefore accumulates the transport backlog. If an outage outlasts the queue, acquisition drops and counts new samples while preserving the oldest queued data for ordered delivery after reconnection.

Forced transport-outage validation

A temporary validation build made the packet writer report transport failure for a fixed interval while acquisition continued normally. The failure injection was removed before the production build.

With a three-second forced outage, all 2,144 observed samples arrived contiguously from sequence 0 through 2,143. Packet gaps, sample gaps, drops, loop overruns, and timestamp-saturation flags were all zero.

With a seven-second forced outage, the queue preserved samples 0 through 511 and then dropped 138 new samples after reaching capacity. Delivery resumed at sample 650. The cumulative drop count and observed sequence gap both equaled 138. The timestamp difference from sample 511 to 650 was exactly 1,390,000 us, matching 139 sample intervals, and no saturation flag was emitted. This verifies both the oldest-data retention policy and the new exact timestamp re-anchor after overflow.

The exact validated byte streams are tracked as executable regression fixtures:

• tests/fixtures/forced_outage_3s.trk, SHA-256 01482816cdaa668e4681c33c8baa1df331d733b9bbcbc4f448ece25e88185ad6
• tests/fixtures/forced_outage_7s.trk, SHA-256 2ea8a5742944bdebc13bec2ccdbceba75f0bb71e48c856b0f86285878e190cd3

The protocol tests verify both hashes, fragmented parsing with no CRC/header or trailing-byte errors, the complete sequence ranges, the sole 138-sample overflow gap, cumulative counters, and the exact 1,390,000 us timestamp re-anchor.