DEV-1023/DEV-1024: one timestamp-unwrap rule, and a Shimmer3 SD file that parses

[marknolan]

Two tickets, because they turned out to be entangled — see Why both below. Depends on
log-and-stream-common#136,
which specifies the rule and ships the conformance vectors this runs. Mirrors
Shimmer-Java-Android-API#300,
which this API has to agree with — the two read the same recordings.

DEV-1023 — one rule for every way the counter can read backwards

Firmware stamps a packet when its sample tick starts it and does not publish a packet it
never stamped, so 0x000000 means the record is invalid, not that the counter reached its
origin. Read as a roll-over, one such record adds 2²⁴ ticks = 512.002 s to every later
sample, permanently.

Rejecting an exact zero is right for that fault and wrong for everything else that makes
the counter read backwards: a corrupted non-zero value, a duplicate and a reordered packet
all still added a modulo. Each sample is now classified by its modular forward distance
from the last — duplicate, reordered, invalid zero, or forward — with forward as the
default everything else falls through to.

  if forward == 0:            hold                          // duplicate
  elif backwards <= W:        lastUnwrapped - backwards     // reordered packet
  elif modulo == 2^24 and raw == 0 and lastRaw < modulo - 32768:
                              reject, state untouched       // never stamped
  else:                       lastUnwrapped + forward       // forward; a wrap iff raw < lastRaw

  W = 0 when the rate is unknown; else min(8 * 32768 / rateHz, modulo / 8)

Three things in it are load-bearing, all verified by mutation:

	why
Modular distance, not unwrapped values	Asking "is the candidate below the last one?" misses a packet arriving late from before a wrap boundary — its candidate sits nearly a modulo ahead, so it is accepted and the next real sample reads as a second wrap
Forward motion is the default	A roll-over preceded by a long dropout is still a roll-over
An unknown rate gives zero, never infinity	32768 / 0 is PositiveInfinity in C#; an infinite window makes every backward step a reorder and loses every wrap. SamplingRate defaults to 0, so that is the normal state until an inquiry or an SD header has run

The window is eight sample periods, not modulo / 8: a reorder swaps adjacent packets, a
dropout spanning the wrap point is most of a modulo, and modulo / 8 confuses them — on
the 2-byte counter every dropout between 1.75 s and 2.0 s would read as a reorder and lose
the wrap, which is an ordinary Bluetooth gap.

Shimmer2/2R get a window of zero. Their tick domain is unsettled — this API divides
their 16-bit counter by 1024 while the Java driver divides by 32768 — so a rate-derived
window would be wrong in one of the two. Follow-up ticket to settle it.

All three copies in this repo are updated: ShimmerDevice, TestSerialPort (own
hard-coded rate), and ShimmerCaptureXamarin, which is a vendored fork not built
alongside ShimmerAPI. It cannot reference TimestampUnwrap, so it carries the rule
inline, flagged as uncompiled and untested here — keeping the two in step is manual.

DEV-1024 — ShimmerSDLog could not parse a Shimmer3 file at all

The timestamp modulo was left at the 2-byte default. TimeStampPacketRawMaxValue is
only ever set by UpdateBasedOnCompatibilityCode(), which runs on the Bluetooth
connect path; ShimmerSDLog never called it. A 3-byte counter unwrapped against a 2-byte
modulo adds 65536 per roll-over instead of 2²⁴, and the recording reads as a negative
duration. Now derived from the firmware version for every hardware version.

Sync-when-logging blocks were not handled. The firmware heads each 512-byte write
buffer with a 9-byte offset record; it was decoded as sample data, so the record stream
lost alignment at the first block. The flag was not even read — both header lines that
would set it sit inside comment blocks. Now read from trial config 0 bit 2, with the offset
record consumed at each block boundary and exposed raw via LastSyncOffset().

And the header rate was integer division. 32768 / rawSamplingRate had two integral
operands, so a divider of 65 gave 504 Hz rather than 504.123 and 640 gave 51 rather than
51.2 — and a divider of zero threw DivideByZeroException out of the constructor, taking
the whole import with it. That fed nothing before; it feeds the reorder window now.

Why both

The zero-timestamp rule is scoped to the 3-byte counter. While the modulo defect stood it
could never fire on the SD path — the drop loop was dead code there. Shipping it alone
would have claimed a fix it did not deliver. The Bluetooth path was unaffected throughout.

Tests

47 passed, 0 failed.

TimestampUnwrapVectorsTest runs the 26 shared vectors. This project has no way to load a
data file, so they are transcribed into a static array by
crosscheck_timestamp_unwrap.py --emit csharp rather than by hand, with the source
revision and commit named in the file. It also covers the window derivations and the values
JSON cannot spell — infinity being the one that matters, since it is what a division by an
unset rate produces.

ShimmerSDLogParseTest is the end-to-end half: the window comes from the header rate, so
these prove that rate reaches the unwrapper. Its file builder now takes per-row timestamp
overrides, so a test can write out-of-order or duplicated records — the firmware does not
produce those, but a corrupt card or a future writer could.

Five mutations checked, each caught:

mutation	caught by
compare unwrapped values instead of modular distances	TestAllSharedVectorsAgree
size the window as modulo / 8	3 tests, including the SD drop tests — an oversized window reclassifies the zero-stamped records as reorders and stops dropping them
let an unknown rate become an infinite window	TestWindowIsZeroForEveryShapeOfUnknownRate
drop the first-sample sentinel	TestAllSharedVectorsAgree
restore the integer division in the header rate	TestHeaderRateIsNotTruncatedToAnInteger

ShimmerTest.TestMethodDeviceName is excluded from the run: it fails on a
System.IO.Ports assembly-binding problem unrelated to this change, confirmed
pre-existing by stashing the changes and re-running it on the untouched branch.

Still outstanding

ShimmerBLE/ShimmerBLEAPI/Sensors/Sensor.cs has a fourth naive unwrap for Verisense —
different device family and counter semantics, so it is recorded as a follow-up rather
than changed here.

DEV-1023, DEV-1024.

🤖 Generated with Claude Code

[marknolan]

Follow-up commit: an adversarial review of this branch found that "no sample
yet" was not distinguishable from a real sample.

This API keeps an unwrapped value and a cycle count rather than the previous raw
value, so the reset state had to be encoded somehow, and (0, 0) was the
encoding. That state is also reachable: a reordered packet landing exactly on the
counter's origin leaves both at zero in the middle of a stream. The next packet
is then read as a first sample and passed through, so one arriving from just
before the origin is placed a whole modulo late rather than sixteen ticks behind.

Found by running the two formulations of the rule against each other rather than
by reading them:

	[520, 0, 16777200]
keeps lastRaw — web SDK, pyshimmer	[520, 0, -16]
(0, 0) as the reset state — here, before	[520, 0, 16777200]

512 seconds apart on the same input, between APIs that are meant to be
identical, and no vector covered it.

The unwrap is now told outright. The existing overload stays, still inferring it
from (0, 0), for callers written before the distinction existed — with a test
stating what it does with this sequence, so the compatibility boundary is
deliberate rather than discovered.

The sequence is now the shared conformance vector
reorder-onto-origin-then-earlier-packet-24bit
(ShimmerResearch/log-and-stream-common#136), so all five host APIs are held to
the same answer. It is the first vector whose final cycle is negative — a real
state here, since the raw value is derived back out of it.

Probability in the field is negligible: it needs a reorder to land exactly on the
origin, within one window of it. The reason to fix it rather than note it is that
the class of problem is the one this whole workstream exists to stop — two
families of implementation that cannot represent the same state, with nothing
testing the difference.