Resource Identity Collapse Lab

An executable CUE lab that demonstrates how a seemingly helpful OpenTelemetry Collector resource transform can erase service identity, merge unrelated SLO populations, and hide a real incident.

The default scenario sends equal traffic through eight logical services. Seven have a 1% error rate; returns-api has a 40% error rate. The real maximum is 40%, well above a 10% alert threshold. A blanket UPSERT service.name = commerce turns all eight services into one resource population, and the visible rate falls to 5.875%. The alert stops firing.

The browser UI compares four strategies:

Strategy	Indexed service IDs	Recoverable IDs	Visible max	Alert
Source truth	8	8	40%	fires
Blind upsert	1	1	5.875%	hidden
Preserve, then upsert	1	8	40%	fires
insert if missing	8	8	40%	fires

This is intentionally different from a generic cardinality or attribute truncation demo. Every attribute value is valid and low-cardinality. The failure is many legitimate identities becoming one.

Why this happens

OpenTelemetry defines service.name as the logical name of a service, and expects it to be unique within a service.namespace. The Collector resource processor's actions have materially different behavior:

• insert writes a key only when it does not exist.
• update changes a key only when it already exists.
• upsert performs either operation, so it replaces every existing service.name in this lab.

The destructive configuration is only three lines:

processors:
  resource/blind:
    attributes:
      - key: service.name
        value: commerce
        action: upsert

If a normalized name is unavoidable, preserve the source key first:

processors:
  resource/preserve:
    attributes:
      - key: source.service.name
        from_attribute: service.name
        action: insert
      - key: service.name
        value: commerce
        action: upsert

If the actual goal is only to backfill missing names, use insert.

Run it

Requirements: Node.js 20+ and CUE 0.17.1.

npm ci
npm start

Open http://localhost:8080.

Or use the published container setup:

docker compose up --build

The official cuelang/cue:0.17.1 image supplies the exact CUE runtime inside the final Node container.

Use the model from the command line

npm run simulate
npm run simulate -- --service-count 12 --hot-error-percent 60
npm run telemetry

You can also evaluate the source directly:

cue export model/scenario.cue -e output \
  -t serviceCount=8 \
  -t requestsPerService=5000 \
  -t hotErrorPercent=40 \
  -t baselinePercent=1 \
  -t alertThreshold=10

The API accepts the same controls:

curl 'http://localhost:8080/api/simulate?serviceCount=8&hotErrorPercent=40'
curl 'http://localhost:8080/api/telemetry?serviceCount=8&hotErrorPercent=40'

/api/telemetry returns correlated OTLP-shaped JSON:

• Metrics: visible/recoverable service counts, collisions, maximum error percentage, alert state, plus per-service request/error populations carrying each strategy's actual processed resource attributes.
• Logs: one diagnostic decision per transform strategy.
• Traces: one span per resource-processing path with success/error status.

The payload is designed to make the investigation natural in telemetry.sh: compare lab.resource.identity_collisions with lab.slo.max_error_percent, then group by the preserved source.service.name.

Runtime proof

Every simulation response includes:

• The exact CUE runtime version.
• A SHA-256 digest of model/scenario.cue.
• The source path evaluated to produce the response.

The Node server does not reimplement the simulation. It invokes cue export for every scenario, then adapts that result into the web and OTLP responses.

Verify

npm run check

The checks validate the CUE model, exercise alternate scenarios, inspect the runtime proof, verify OTLP metrics/logs/traces, and call every HTTP endpoint. CI also builds the clean Linux container image.

References

• OpenTelemetry service resource semantic conventions
• OpenTelemetry Collector resource processor
• CUE installation

License

MIT