trikkeSensors/docs/binary-record-v1.md

# TRK1 Binary Telemetry — Version 1

The binary stream is the shared transport and storage representation for USB,
BLE, and any later nonvolatile buffer. All multibyte integers and IEEE-754
float32 values are little-endian. Frames are self-identifying and may be split or
combined by an underlying byte transport.

## Frame header (36 bytes)

| Offset | Size | Field |
| ---: | ---: | --- |
| 0 | 4 | ASCII magic `TRK1` |
| 4 | 1 | Wire version (`1`) |
| 5 | 1 | Packet type: metadata `1`, samples `2`, status `3` |
| 6 | 1 | Header size (`36`) |
| 7 | 1 | Record size (`0` or `20`) |
| 8 | 1 | Record count (`0` or 1–8) |
| 9 | 1 | Packet flags |
| 10 | 2 | Payload size |
| 12 | 4 | Monotonic packet sequence |
| 16 | 8 | Base ESP timer timestamp in microseconds |
| 24 | 4 | Cumulative samples lost to sensor read failure or queue overflow |
| 28 | 4 | Cumulative acquisition-loop overruns |
| 32 | 4 | IEEE CRC-32 |

CRC uses polynomial `0xEDB88320`, initial value `0xFFFFFFFF`, and final XOR
`0xFFFFFFFF`. It covers header bytes 4–31 followed by the complete payload. The
magic and stored CRC field are excluded.

Packet flag bit 0 means at least one sample timestamp delta saturated. Both host
tools report any frame carrying this flag instead of silently treating its
reconstructed timestamps as exact.

## Sample record (20 bytes)

| Offset | Size | Field |
| ---: | ---: | --- |
| 0 | 4 | Sample sequence |
| 4 | 2 | Timestamp delta from the preceding record, in 10 us units |
| 6 | 2 | Enclosure accel X raw (`int16`) |
| 8 | 2 | Enclosure accel Y raw (`int16`) |
| 10 | 2 | Enclosure accel Z raw (`int16`) |
| 12 | 2 | Enclosure gyro X raw (`int16`) |
| 14 | 2 | Enclosure gyro Y raw (`int16`) |
| 16 | 2 | Enclosure gyro Z raw (`int16`) |
| 18 | 1 | Raw ADXL345 `INT_SOURCE` |
| 19 | 1 | Raw L3G4200D `STATUS_REG` |

The first record has delta zero and uses the frame's base timestamp. Each later
timestamp is reconstructed by cumulatively adding its delta. Firmware ends the
current packet before a delta exceeds the representable 655.35 ms range, making
the next sample the exact base timestamp of a new packet. As a defensive encoder
fallback, an unrepresentable delta is stored as `0xFFFF` and sets packet flag bit
0. Because intra-packet deltas are rounded to 10 us while each packet base keeps
the exact ESP timer value, an integrity check for an exact 10,000 us interval can
report up to +/-5 us at packet boundaries; changing the transport's packet size
changes how often that harmless quantization boundary appears. Sample sequence
gaps remain detectable independently.

Mapped raw counts are authoritative. The original sensor-native axes can be
reconstructed because the mappings are lossless:

```text
accel native = (-enclosure_y, enclosure_x, enclosure_z)
gyro native  = ( enclosure_x, enclosure_y, enclosure_z)
```

## Metadata payload (48 bytes)

Metadata frames repeat approximately every five seconds so a receiver may attach
midstream. The payload contains:

- Sample rate, accelerometer range, gyroscope range, and mapping/calibration flags
- Three accel offset float32 values
- Three accel counts/g float32 values
- Three gyro bias float32 values
- Nominal gyro mdps/LSB float32 value

A byte-stream receiver may begin inside an incomplete frame. It discards bytes
until a magic/header/CRC combination validates. Host tools report any rejection
before that first valid frame separately from CRC failures after synchronization.

## Status payload (32 bytes)

Status frames use packet type `3`, record size/count zero, and payload version
`1`. They are emitted at startup and approximately every five seconds. Offsets
0 and 2 are uint16 payload version and payload size; the remaining fields are
cumulative uint32 counters:

| Offset | Field |
| ---: | --- |
| 4 | Sensor read failures |
| 8 | Sample-queue overflows |
| 12 | Initial transport submissions that accepted zero bytes |
| 16 | BLE disconnects after a connection was established |
| 20 | BLE notification enqueue/send failures |
| 24 | BLE frame replays after disconnect, subscription change, or ACK timeout |
| 28 | Malformed, stale, premature, or wrong-connection ACK writes |

The header's cumulative dropped-sample count remains the sum of sensor read
failures and queue overflows, preserving version-1 receiver compatibility while
the status payload makes the causes independently observable. USB-specific BLE
counters remain zero.

## Buffering

Acquisition runs in a dedicated higher-priority task and writes complete samples
to a 1024-entry RAM queue. The lower-priority output task batches up to eight
records per frame. At 100 Hz this queue represents about 10.24 seconds of
decoupling when the transport reports backpressure or failure accurately. A
failed write retains and retries the same encoded packet rather than dequeuing
more samples, so the queue accumulates the outage backlog. After reconnection,
the oldest retained data is sent first. If the queue fills, acquisition drops
new samples rather than overwriting older ones; sequence gaps and the cumulative
lost-sample counter expose that permanent loss.

The shared transport state machine distinguishes three nonfatal states. `RETRY`
is valid only from initial submission: it means zero bytes were accepted and the
complete frame may be submitted again. `PENDING` means the backend owns an
in-flight frame, so firmware may only poll that transfer. A `RETRY` returned by
polling fails closed as `FATAL`, because generic code cannot prove whole-frame
resubmission is duplicate-safe. `COMPLETE` permits the output task to reuse its
packet buffer and consume more samples. This prevents a timeout after partial
progress from causing an ambiguous whole-frame duplicate.

The direct USB Serial/JTAG backend atomically copies a complete frame into its TX
ring, then polls a bounded transmit-drain wait. A timeout remains `PENDING`; it
does not trigger resubmission. This closes the VFS/stdio path's silent-discard
case for a connected host that stops draining.

USB drain is not end-to-end application delivery confirmation. A host process
may attach after earlier frames have already left the endpoint, or fail after
the endpoint accepts them. CRC and sequence checks make resulting loss visible,
but an application acknowledgement and replay window are still required to
guarantee receipt. Accordingly, USB `COMPLETE` means endpoint drain, while
reliable BLE reserves `COMPLETE` for an application ACK of the exact frame. See
`ble-transport-v1.md` for fragmentation, replay, and UUIDs.

Receivers report bytes left in an incomplete trailing frame when capture ends.
Those bytes cannot pass CRC validation and are not silently admitted as samples.