OTLP Partial Success Lab

An executable Jsonnet lab showing why an OTLP export can return HTTP 200 while permanently rejecting part of its telemetry payload.

The default export window sends:

• 30 trace-export batches.
• 10,000 spans per batch.
• 300,000 spans in total.
• An 8% receiver-side rejection rate.

Every request returns HTTP 200 OK. A monitor that only checks transport status reports 100% success. The response body tells a different story:

{
  "partialSuccess": {
    "rejectedSpans": 800,
    "errorMessage": "span policy rejected records with an unsupported schema revision"
  }
}

Across the default window, the receiver accepts 276,000 spans and rejects 24,000. Actual delivery is 92%, leaving an eight-percentage-point gap behind a perfect request-success rate.

Why the loss is terminal

OTLP uses the normal export response message for both full and partial success. A receiver that only accepts part of a request populates partial_success, sets rejected_spans, and may include an explanatory message.

The client must not retry a request after receiving a populated partial-success response. That avoids replaying the accepted portion, but it also means the rejected spans are permanently dropped. Alerting only on connection errors or HTTP status codes cannot see this failure.

This lab models an OTLP/HTTP JSON response for readability. The same partial success semantics apply to OTLP/gRPC and protobuf responses.

Two monitoring realities

Monitor	HTTP success	Observed rejection	Unaccounted spans
Status only	100%	0	24,000
Partial-success aware	100%	24,000	0

The status-only monitor is intentionally naive. OpenTelemetry's exporter self-telemetry conventions say rejected spans must count as failed export outcomes, while only non-rejected spans count as successful.

What telemetry.sh reveals

The generated OTLP-shaped payload includes:

• lab.otlp.export.request.count with HTTP 200 transport outcomes.
• lab.otlp.partial_success.response.count.
• lab.otlp.span.sent, lab.otlp.span.accepted, and lab.otlp.span.rejected.
• lab.otlp.delivery.ratio and lab.otlp.visibility.gap.
• otel.sdk.exporter.span.exported, split between successful spans and error.type="rejected".
• A status-only exporter span with no response event.
• A response-aware exporter span with an otlp.partial_success event.
• INFO and WARN log records that preserve both interpretations of the same HTTP response.

That lets telemetry.sh distinguish transport availability from telemetry acceptance, quantify the missing data, and pivot to the receiver's rejection reason.

Run it

Requirements:

• Node.js 20+
• Go 1.24.5+ for building the Jsonnet CLI

npm ci
make start

Open http://localhost:8080.

Or run the clean Linux container:

docker compose up --build

Use the Jsonnet model directly

The model has no infrastructure side effects. Inputs are supplied as Jsonnet external variables:

make tools

.tools/jsonnet \
  --ext-str batches=30 \
  --ext-str spans_per_batch=10000 \
  --ext-str rejection_rate_basis_points=800 \
  model/main.jsonnet

Convenience commands:

make simulate
make telemetry

JSONNET_BIN="$PWD/.tools/jsonnet" \
  npm run simulate -- --batches 12 --rejection-rate-basis-points 1500

HTTP endpoints expose the same model:

curl 'http://localhost:8080/api/simulate?batches=30'
curl 'http://localhost:8080/api/telemetry?rejectionRateBasisPoints=800'

Runtime proof

Every model and telemetry response includes the go-jsonnet runtime version, runtime platform, SHA-256 digest of model/main.jsonnet, and model path. The Node server does not duplicate the acceptance or visibility-gap calculation.

Verify

npm ci
make check

Tests cover the default loss, response-aware monitoring, true full success, bounded inputs, exporter outcome metrics, trace and log evidence, runtime proof, and every HTTP endpoint. CI also builds the multi-stage Alpine image with go-jsonnet 0.22.0.

References

• OpenTelemetry Protocol: partial success
• OpenTelemetry SDK exporter metrics
• Jsonnet standard library

License

MIT