The trace exists. The logs exist. The join says they have never met.
This interactive lab demonstrates a cross-signal correlation failure: a valid 128-bit trace ID is parsed as an integer by a log pipeline and serialized back to hexadecimal without fixed-width padding. Leading zeroes disappear. The trace store retains the canonical ID, the log index stores a shorter string, and an exact join silently orphans every log in that cohort.
The executable model is written in Hy, a Lisp dialect embedded in Python. A dependency-free Python HTTP adapter runs the model and serves a responsive investigation workbench with OTLP-shaped traces, metrics, and logs.
The model balances 16,000 trace IDs across all sixteen possible first hexadecimal nibbles:
| Observation | Result |
|---|---|
Valid traces beginning with 0 |
1,000 |
| Affected share | 6.25% |
| Logs per trace | 4 |
| Orphaned logs | 4,000 |
| Exact correlation coverage | 93.75% |
| Hidden incident traces at a 25% incident rate | 250 |
The fleet-wide 93.75% can look healthy. The 0… cohort has 0% exact
log-to-trace correlation.
The representative identity makes the transformation explicit:
canonical trace: 00a123456789abcdef0123456789abcd (32 nibbles)
numeric log ID: a123456789abcdef0123456789abcd (30 nibbles)
fixed-width log: 00a123456789abcdef0123456789abcd (32 nibbles)
The number denotes the same magnitude, but identifiers are not magnitudes. They are byte sequences with a canonical wire representation.
W3C Trace Context defines trace-id as exactly 16 bytes represented by 32
lowercase hexadecimal characters. OpenTelemetry's Logs Data Model stores
TraceId as a byte sequence, and OTLP JSON represents trace and span ID byte
arrays as hexadecimal strings.
- W3C Trace Context:
trace-id - OpenTelemetry Logs Data Model
- OTLP JSON Protobuf encoding
- Trace context in non-OTLP log formats
The malformed log record in this lab is intentionally visible in the payload. It represents the bad string produced by a legacy log ETL boundary; it is not presented as a valid OTLP trace ID.
Request rates, exporter success, and trace counts can all remain normal. Diagnosing the failure requires correlating representation evidence:
- canonical and indexed trace ID widths;
- unmatched log counts grouped by ID length;
- exact join coverage in basis points;
- the first-nibble cohort boundary;
- warning logs that carry both the expected and indexed values;
- a trace event describing the missed correlation;
- recovered joins after fixed-width encoding.
The lab emits:
lab.correlation.orphaned_logs
lab.correlation.join_coverage
lab.trace_id.numeric_width
lab.correlation.repair_recovered_logs
Telemetry.sh can keep those metrics beside the example trace and both log representations, turning “logs are missing” into a precise encoding diagnosis.
Requirements:
- Python 3.9 or newer
- Hy 1.3
- Node.js (syntax check only)
- GNU Make
python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
make check
make runOpen http://127.0.0.1:3000.
Useful endpoints:
GET /healthz
GET /api/simulate
GET /api/telemetry
Every control is also a query parameter:
/api/simulate?traceCount=32000&logsPerTrace=6&coercionPercent=50&incidentPercent=40&leadingZeroNibbles=4
docker compose up --buildThe image runs the complete model and test suite during its build, then starts the service as an unprivileged user.
Do not move trace IDs through numeric columns or numeric JSON values. Preserve the original 16-byte value when possible. At text boundaries:
- encode all 16 bytes;
- emit exactly 32 hexadecimal characters;
- normalize to lowercase where required by the target format;
- validate width and content before indexing;
- monitor correlation misses grouped by encoded width.
Padding a value after a known, lossless integer conversion can recover leading zeroes:
canonical = format(trace_id_integer, "032x")Preserving the byte sequence end to end is safer because it makes lossy numeric interpretation impossible.
model/canonicalize.hy executable Hy model
service/lab.py model adapter and correlated telemetry
service/server.py dependency-free HTTP and static service
public/ investigation workbench
tests/ model, telemetry, and static checks
MIT