add framed binary telemetry transport
This commit is contained in:
@@ -0,0 +1,81 @@
|
||||
# TRK1 Binary Telemetry — Version 1
|
||||
|
||||
The binary stream is the shared transport and storage representation for USB,
|
||||
BLE, and any later nonvolatile buffer. All multibyte integers and IEEE-754
|
||||
float32 values are little-endian. Frames are self-identifying and may be split or
|
||||
combined by an underlying byte transport.
|
||||
|
||||
## Frame header (36 bytes)
|
||||
|
||||
| Offset | Size | Field |
|
||||
| ---: | ---: | --- |
|
||||
| 0 | 4 | ASCII magic `TRK1` |
|
||||
| 4 | 1 | Wire version (`1`) |
|
||||
| 5 | 1 | Packet type: metadata `1`, samples `2` |
|
||||
| 6 | 1 | Header size (`36`) |
|
||||
| 7 | 1 | Record size (`0` or `20`) |
|
||||
| 8 | 1 | Record count (`0` or 1–8) |
|
||||
| 9 | 1 | Packet flags |
|
||||
| 10 | 2 | Payload size |
|
||||
| 12 | 4 | Monotonic packet sequence |
|
||||
| 16 | 8 | Base ESP timer timestamp in microseconds |
|
||||
| 24 | 4 | Cumulative samples lost to read failure, queue overflow, or output failure |
|
||||
| 28 | 4 | Cumulative acquisition-loop overruns |
|
||||
| 32 | 4 | IEEE CRC-32 |
|
||||
|
||||
CRC uses polynomial `0xEDB88320`, initial value `0xFFFFFFFF`, and final XOR
|
||||
`0xFFFFFFFF`. It covers header bytes 4–31 followed by the complete payload. The
|
||||
magic and stored CRC field are excluded.
|
||||
|
||||
Packet flag bit 0 means at least one sample timestamp delta saturated.
|
||||
|
||||
## Sample record (20 bytes)
|
||||
|
||||
| Offset | Size | Field |
|
||||
| ---: | ---: | --- |
|
||||
| 0 | 4 | Sample sequence |
|
||||
| 4 | 2 | Timestamp delta from the preceding record, in 10 us units |
|
||||
| 6 | 2 | Enclosure accel X raw (`int16`) |
|
||||
| 8 | 2 | Enclosure accel Y raw (`int16`) |
|
||||
| 10 | 2 | Enclosure accel Z raw (`int16`) |
|
||||
| 12 | 2 | Enclosure gyro X raw (`int16`) |
|
||||
| 14 | 2 | Enclosure gyro Y raw (`int16`) |
|
||||
| 16 | 2 | Enclosure gyro Z raw (`int16`) |
|
||||
| 18 | 1 | Raw ADXL345 `INT_SOURCE` |
|
||||
| 19 | 1 | Raw L3G4200D `STATUS_REG` |
|
||||
|
||||
The first record has delta zero and uses the frame's base timestamp. Each later
|
||||
timestamp is reconstructed by cumulatively adding its delta. A delta that cannot
|
||||
fit is stored as `0xFFFF` and sets packet flag bit 0. Sample sequence gaps remain
|
||||
detectable independently.
|
||||
|
||||
Mapped raw counts are authoritative. The original sensor-native axes can be
|
||||
reconstructed because the mappings are lossless:
|
||||
|
||||
```text
|
||||
accel native = (-enclosure_y, enclosure_x, enclosure_z)
|
||||
gyro native = ( enclosure_x, enclosure_y, enclosure_z)
|
||||
```
|
||||
|
||||
## Metadata payload (48 bytes)
|
||||
|
||||
Metadata frames repeat approximately every five seconds so a receiver may attach
|
||||
midstream. The payload contains:
|
||||
|
||||
- Sample rate, accelerometer range, gyroscope range, and mapping/calibration flags
|
||||
- Three accel offset float32 values
|
||||
- Three accel counts/g float32 values
|
||||
- Three gyro bias float32 values
|
||||
- Nominal gyro mdps/LSB float32 value
|
||||
|
||||
A byte-stream receiver may begin inside an incomplete frame. It discards bytes
|
||||
until a magic/header/CRC combination validates. Host tools report any rejection
|
||||
before that first valid frame separately from CRC failures after synchronization.
|
||||
|
||||
## Buffering
|
||||
|
||||
Acquisition runs in a dedicated higher-priority task and writes complete samples
|
||||
to a 128-entry RAM queue. The lower-priority output task batches up to eight
|
||||
records per frame. At 100 Hz this queue represents about 1.28 seconds of
|
||||
decoupling from a blocked transport. Queue overflow never overwrites an older
|
||||
sample silently: sequence gaps and the cumulative lost-sample counter expose it.
|
||||
@@ -0,0 +1,57 @@
|
||||
# Binary Transport Validation — 2026-08-17
|
||||
|
||||
This milestone replaced high-volume device-side CSV with the versioned `TRK1`
|
||||
binary stream shared by future USB, BLE, and storage paths. Mapped raw sensor
|
||||
counts remain authoritative. Calibration metadata travels as float32 values, and
|
||||
the host reconstructs the prior 23-column diagnostic CSV without discarding raw
|
||||
data.
|
||||
|
||||
## Verification layers
|
||||
|
||||
- Host compilation of the production C encoder with `-Wall -Wextra -Werror`
|
||||
- Fragmented C-encoder-to-Python-parser contract test
|
||||
- Deliberately corrupted CRC test with stream resynchronization
|
||||
- ESP-IDF firmware build and flash on the assembled ESP32-C3 prototype
|
||||
- Live USB capture followed by independent offline re-decoding
|
||||
|
||||
## USB text-conversion finding
|
||||
|
||||
The first live capture exposed that the USB console's default CRLF mode inserted
|
||||
a carriage return whenever a binary byte equaled LF (`0x0A`). CRC rejected every
|
||||
affected frame and the parser resynchronized at the next `TRK1` magic. No invalid
|
||||
sample entered decoded CSV.
|
||||
|
||||
Before binary output begins, firmware now changes the USB Serial/JTAG VFS transmit
|
||||
mode to `ESP_LINE_ENDINGS_LF`, which means no byte modification. Startup logs and
|
||||
readable metadata are flushed first.
|
||||
|
||||
## Final hardware capture
|
||||
|
||||
`captures/binary_v1_smoke2.trk` and its decoded CSV contain:
|
||||
|
||||
- 7,184 samples over 71.830 seconds
|
||||
- 912 total frames, including 14 repeated metadata frames
|
||||
- Packet gaps and resets: 0
|
||||
- Sample gaps and resets: 0
|
||||
- Timestamp anomalies: 0
|
||||
- CRC and header failures: 0
|
||||
- Queue/read/output drops: 0
|
||||
- Acquisition-loop overruns: 0
|
||||
- ADXL345 overruns: 0
|
||||
- L3G4200D data-ready clear: 0
|
||||
- L3G4200D overruns: 192
|
||||
|
||||
Offline decoding of the saved `.trk` file produced CSV byte-for-byte identical to
|
||||
the CSV rendered during live capture.
|
||||
|
||||
The validated stream occupied 177,184 bytes, or 2,466.7 bytes/s including frame
|
||||
headers and repeated metadata. That is 8.47 MiB/hour and about 19.7 kbit/s before
|
||||
BLE link overhead, far below the previous CSV stream.
|
||||
|
||||
## Task separation and buffer
|
||||
|
||||
The 100 Hz I2C acquisition runs at FreeRTOS priority 10. USB encoding/output runs
|
||||
at priority 5 and receives samples through a 128-entry queue (about 1.28 seconds
|
||||
at 100 Hz). The hardware capture's zero timing anomalies and zero loop overruns
|
||||
confirm that packet encoding, CRC, float metadata, and USB output did not disturb
|
||||
the acquisition cadence.
|
||||
@@ -31,6 +31,11 @@ coefficients:
|
||||
| Y | -4.400892 | 259.825135 | 3.848742 |
|
||||
| Z | +11.776132 | 245.755573 | 4.069084 |
|
||||
|
||||
The retained digits make the firmware calculation reproducible; they are not a
|
||||
claim of sub-count measurement accuracy. Independent face selection changed the
|
||||
derived values by up to about 0.03 counts, and repositioning changed a face mean
|
||||
by about one count.
|
||||
|
||||
Firmware converts a mapped raw value to integer milligravity with:
|
||||
|
||||
```text
|
||||
@@ -52,14 +57,25 @@ The central five thousand samples of the flat stationary interval produced:
|
||||
| Z | -7.0238 | 10.4259 |
|
||||
|
||||
The instructed positive motion was top/USB-edge lift for +X, left-edge lift for
|
||||
+Y, and counterclockwise rotation viewed from the cover for +Z. Every outward
|
||||
stroke was positive on its intended gyro channel, and every return stroke was
|
||||
negative. Axis assignment and polarity are therefore confirmed.
|
||||
+Y, and counterclockwise rotation viewed from the cover for +Z. Accelerometer
|
||||
tilt kinematics independently confirmed X and Y polarity. Rotation around Z while
|
||||
flat is rotation around gravity, so the accelerometer cannot independently check
|
||||
that sign. Z polarity follows from the right-handed enclosure frame, the
|
||||
right-handed L3G4200D frame, the identity gyro mapping, and the verified X/Y
|
||||
polarities. Axis assignment and polarity are therefore confirmed.
|
||||
|
||||
Firmware subtracts the measured zero-rate bias and converts with the nominal
|
||||
L3G4200D +/-500 dps scale of 17.5 mdps/LSB. Gyro scale itself was not measured
|
||||
because no controlled angular-rate reference was available.
|
||||
|
||||
The compile-time bias is only an initial correction. Twelve quiet stretches in
|
||||
this session showed real zero-rate wander of roughly 50 mdps on X/Z and 72 mdps
|
||||
on Y; the separate smoke capture differed by up to 63 mdps. That is enough to
|
||||
accumulate several degrees per minute if integrated. Runtime re-zeroing during a
|
||||
verified stationary interval is required before fusion, heading integration, or
|
||||
turn counting. Raw gyro counts therefore remain authoritative in the binary
|
||||
record.
|
||||
|
||||
## Freshness under motion
|
||||
|
||||
Using gyro magnitude greater than 500 counts from stationary bias as a
|
||||
@@ -86,4 +102,10 @@ and checked with a 3,814-record flat smoke capture:
|
||||
- Mean bias-corrected gyro: (+25.4, -62.9, -54.8) mdps
|
||||
|
||||
The small residual horizontal acceleration is consistent with the enclosure not
|
||||
being perfectly level. The largest residual gyro mean is 0.063 dps.
|
||||
being perfectly level. The largest residual gyro mean is 0.063 dps, which is
|
||||
evidence of the fixed-bias limitation described above rather than a long-term
|
||||
heading guarantee.
|
||||
|
||||
The two floating-point startup metadata lines were subsequently observed after a
|
||||
live device reset. They rendered the intended float32 calibration values before
|
||||
the binary stream began.
|
||||
|
||||
Reference in New Issue
Block a user