112 lines
5.4 KiB
Markdown
112 lines
5.4 KiB
Markdown
# Binary Transport Validation — 2026-08-17
|
|
|
|
This milestone replaced high-volume device-side CSV with the versioned `TRK1`
|
|
binary stream shared by future USB, BLE, and storage paths. Mapped raw sensor
|
|
counts remain authoritative. Calibration metadata travels as float32 values, and
|
|
the host reconstructs the prior 23-column diagnostic CSV without discarding raw
|
|
data.
|
|
|
|
## Verification layers
|
|
|
|
- Host compilation of the production C encoder with `-Wall -Wextra -Werror`
|
|
- Fragmented C-encoder-to-Python-parser contract test
|
|
- Deliberately corrupted CRC test with stream resynchronization
|
|
- Full eight-record, timestamp-saturation, invalid-size/count, and truncated-tail
|
|
encoder/parser contract cases
|
|
- ESP-IDF firmware build and flash on the assembled ESP32-C3 prototype
|
|
- Live USB capture followed by independent offline re-decoding
|
|
- Tracked three- and seven-second hardware-outage captures with regression
|
|
assertions for integrity, retention order, and overflow accounting
|
|
|
|
## USB text-conversion finding
|
|
|
|
The first live capture exposed that the USB console's default CRLF mode inserted
|
|
a carriage return whenever a binary byte equaled LF (`0x0A`). CRC rejected every
|
|
affected frame and the parser resynchronized at the next `TRK1` magic. No invalid
|
|
sample entered decoded CSV.
|
|
|
|
Before binary output begins, firmware now changes the USB Serial/JTAG VFS transmit
|
|
mode to `ESP_LINE_ENDINGS_LF`, which means no byte modification. Startup logs and
|
|
readable metadata are flushed first.
|
|
|
|
## USB VFS resilience qualification
|
|
|
|
The 5.12-second queue guarantee applies when the transport reports failure or
|
|
backpressure accurately. The current ESP-IDF USB Serial/JTAG VFS/stdio path
|
|
surfaces a physical disconnect, but it has a weaker connected-stall case: when a
|
|
host remains connected and stops draining, the lower-level timeout can discard
|
|
bytes even though stdio reports a successful write. Firmware therefore cannot
|
|
retain that particular frame or increment its drop counter. The receiver still
|
|
detects the loss through CRC resynchronization and packet/sample sequence gaps.
|
|
|
|
ESP-IDF's direct USB Serial/JTAG driver provides bounded writes and an explicit
|
|
transmit-drain wait, allowing a connected stall to become observable to the
|
|
transport policy. That is a useful improvement for the common transport layer.
|
|
It is not proof of receiver delivery; application acknowledgements and replay
|
|
are needed for that stronger guarantee and are planned with BLE integration.
|
|
|
|
## Final hardware capture
|
|
|
|
`captures/binary_v1_smoke2.trk` and its decoded CSV contain:
|
|
|
|
- 7,184 samples over 71.830 seconds
|
|
- 912 total frames, including 14 repeated metadata frames
|
|
- Packet gaps and resets: 0
|
|
- Sample gaps and resets: 0
|
|
- Timestamp anomalies: 0
|
|
- CRC and header failures: 0
|
|
- Queue/read/output drops: 0
|
|
- Acquisition-loop overruns: 0
|
|
- ADXL345 overruns: 0
|
|
- L3G4200D data-ready clear: 0
|
|
- L3G4200D overruns: 192
|
|
|
|
Offline decoding of the saved `.trk` file produced CSV byte-for-byte identical to
|
|
the CSV rendered during live capture.
|
|
|
|
The validated stream occupied 177,184 bytes, or 2,466.7 bytes/s including frame
|
|
headers and repeated metadata. That is 8.47 MiB/hour and about 19.7 kbit/s before
|
|
BLE link overhead, far below the previous CSV stream.
|
|
|
|
## Task separation and buffer
|
|
|
|
The 100 Hz I2C acquisition runs at FreeRTOS priority 10. USB encoding/output runs
|
|
at priority 5 and receives samples through a 512-entry queue (about 5.12 seconds
|
|
at 100 Hz). The hardware capture's zero timing anomalies and zero loop overruns
|
|
confirm that packet encoding, CRC, float metadata, and USB output did not disturb
|
|
the acquisition cadence.
|
|
|
|
When output failure is reported, handling is transactional: firmware retains and
|
|
retries the same encoded packet with a scheduler delay instead of discarding it
|
|
or dequeuing more samples. The queue therefore accumulates the transport backlog.
|
|
If an outage outlasts the queue, acquisition drops and counts new samples while
|
|
preserving the oldest queued data for ordered delivery after reconnection.
|
|
|
|
## Forced transport-outage validation
|
|
|
|
A temporary validation build made the packet writer report transport failure for
|
|
a fixed interval while acquisition continued normally. The failure injection was
|
|
removed before the production build.
|
|
|
|
With a three-second forced outage, all 2,144 observed samples arrived contiguously
|
|
from sequence 0 through 2,143. Packet gaps, sample gaps, drops, loop overruns, and
|
|
timestamp-saturation flags were all zero.
|
|
|
|
With a seven-second forced outage, the queue preserved samples 0 through 511 and
|
|
then dropped 138 new samples after reaching capacity. Delivery resumed at sample
|
|
650. The cumulative drop count and observed sequence gap both equaled 138. The
|
|
timestamp difference from sample 511 to 650 was exactly 1,390,000 us, matching
|
|
139 sample intervals, and no saturation flag was emitted. This verifies both the
|
|
oldest-data retention policy and the new exact timestamp re-anchor after overflow.
|
|
|
|
The exact validated byte streams are tracked as executable regression fixtures:
|
|
|
|
- `tests/fixtures/forced_outage_3s.trk`, SHA-256
|
|
`01482816cdaa668e4681c33c8baa1df331d733b9bbcbc4f448ece25e88185ad6`
|
|
- `tests/fixtures/forced_outage_7s.trk`, SHA-256
|
|
`2ea8a5742944bdebc13bec2ccdbceba75f0bb71e48c856b0f86285878e190cd3`
|
|
|
|
The protocol tests verify both hashes, fragmented parsing with no CRC/header or
|
|
trailing-byte errors, the complete sequence ranges, the sole 138-sample overflow
|
|
gap, cumulative counters, and the exact 1,390,000 us timestamp re-anchor.
|