Right-Censoring Lab

An interactive Futhark lab showing how completed-only latency telemetry can report a healthier p95 precisely because the slowest requests timed out and disappeared.

Default scenario: 10,000 requests, a 500 ms client deadline, and a 30% cohort that completes at 900 ms. The completed-only histogram reports a 90 ms p95 from 70% of request starts. Ground truth is 900 ms.

telemetry.sh makes this failure mode easy to expose by putting request starts, terminal timeout outcomes, duration coverage, and correlated client/server spans in one investigation.

The blind spot

NIST defines right censoring as observing an item only up to a time threshold, with its later failure time unknown. A client timeout creates the same statistical boundary for request latency: a timeout at 500 ms tells us the eventual duration is at least 500 ms. It does not tell us that the request completed in 500 ms.

A completed-only histogram silently changes its population:

request starts                 10,000
├─ completed by 500 ms          7,000  → duration histogram
└─ still running at 500 ms      3,000  → missing from histogram

latency observation coverage = duration observations / request starts
                             = 7,000 / 10,000
                             = 70%

This is distinct from coordinated omission. The requests were admitted and started; their terminal durations were then excluded by the observation window.

Run it

Requirements:

• Futhark 0.26.4
• Python 3.8 or later
• Node.js (for the JavaScript syntax check)

make check
make run

Open http://localhost:8080. No package manager or application dependency is required.

Or run the pinned container build:

docker compose up --build

Three instrumentation contracts

Mode	Recorded latency population	Default p95	Coverage	Meaning
Completed only	Requests finishing before the deadline	90 ms	70%	Biased sample; timeout durations vanish
Deadline events	Completions plus terminal timeout lower bounds	≥500 ms	100%	Tail is visible, but right-censored
Server completion	Eventual server-side completion spans	900 ms	100%	Full synthetic ground truth

The pure model is in model/censoring.fut. It creates a request-duration array, classifies every request in parallel with map, and reconciles the cohorts with reduce. The dependency-free Python service invokes the compiled model for every scenario and exposes:

• GET /api/simulate — chart model and instrumentation comparison
• GET /api/telemetry — correlated metrics, logs, and trace specimen
• GET /healthz — readiness check

Example:

curl 'http://localhost:8080/api/simulate?requestCount=10000&slowPercent=30&timeoutMs=500&slowLatencyMs=900&mode=completed_only'

What to record

1. Count every admitted request with lab.request.started.
2. Count every crossed observation boundary with lab.request.censored{reason="client_timeout"}.
3. Reconcile duration histogram count against starts.
4. Record a terminal timeout event at the deadline when eventual completion is unavailable.
5. Preserve trace correlation so a client timeout can be connected to server work that continues after the caller stops waiting.

OpenTelemetry's Trace SDK sends finished spans to span processors after End, so a client span ending at its deadline and a later server span are separate, correlatable observations. The HTTP semantic conventions define request duration metrics and use error.type=timeout for timeout failures. See telemetry/signals.md for the concrete signal contract.

Investigate in telemetry.sh

The UI includes copyable investigation recipes:

metrics
| let coverage = sum("http.client.request.duration.count") / sum("lab.request.started")
| chart coverage

metrics
| project p95("http.client.request.duration"), rate("lab.request.censored")
| timeseries

traces
| where server.duration_ms > client.deadline_ms
| project trace_id, client.duration_ms, server.duration_ms, error.type

The first query is the guardrail: never interpret a latency percentile without knowing what fraction of admitted work produced a duration observation.

Verification

make check

The check compiles the Futhark model, exercises all three instrumentation contracts, runs model/API/static security tests, and syntax-checks the browser code. CI repeats those checks and performs a container smoke test.

References

• NIST Engineering Statistics Handbook: censoring
• OpenTelemetry Trace SDK: span processors
• OpenTelemetry HTTP metrics semantic conventions

License

MIT