An interactive telemetry lab about the part of a cumulative counter reset that reset-aware math cannot recover.
A process handles requests continuously, but its cumulative counter resets between two scrapes. Four reconstructions tell four different stories:
| Strategy | Default result | What happened |
|---|---|---|
| Endpoint subtraction | −1,949,200 | The stream identity changed, so the arithmetic is meaningless |
| Clamp negative to zero | 0 | A tidy chart erases all 120,000 requests |
| Reset-aware reconstruction | 110,800 | The reset is handled, but 9,200 pre-exit increments were never scraped |
| Shutdown boundary flush | 120,000 | A final old-process point closes the gap |
The default model uses a 300-second window, 400 requests/second, a 30-second scrape interval, and a restart at second 173. The final old-process scrape is at second 150. The 23 seconds between that scrape and process exit contain 9,200 real requests that no downstream query can infer from the sampled counter alone.
The interface makes the missing interval tangible, while the API emits the same evidence as correlated OTLP-shaped signals:
- Metrics include the original cumulative monotonic Sum, with
StartTimeUnixNanochanging at restart, plus observed/hidden/coverage gauges for all four strategies. - Logs mark process restart, reset detection, and the hypothetical shutdown boundary flush.
- Traces evaluate each reconstruction strategy and mark incomplete results as errors.
- Every response proves that the model was executed by Gnuplot and includes a SHA-256 of the model source.
Inspect the evidence:
curl 'http://localhost:8080/api/simulate'
curl 'http://localhost:8080/api/telemetry'
curl 'http://localhost:8080/healthz'Requirements:
- Node.js 24+
- Gnuplot 5.4+
npm ci
npm startOpen http://localhost:8080.
Or use Docker:
docker compose up --buildmodel/main.gp is the source of truth. It calculates the
counter trajectory, scrape points, reset boundary, and reconstruction results,
then writes JSON:
gnuplot \
-e 'WINDOW=300; SCRAPE=30; RATE=400; RESTART=173; START=2000000' \
model/main.gpFor a constant request rate r, the actual window population is:
actual = r × window
The reset-aware reconstruction can add the non-negative deltas on both sides of the reset, but the final portion of the old process was never sampled:
hidden = r × (restart_time − previous_scrape_time)
observed = actual − hidden
Changing scrape cadence changes the maximum blind interval. It does not make an unobserved boundary measurable.
GET /api/simulate and GET /api/telemetry accept:
| Query parameter | Range | Default | Meaning |
|---|---|---|---|
windowSeconds |
60–900 | 300 | Observation window |
scrapeIntervalSeconds |
5–120 | 30 | Time between samples |
requestsPerSecond |
1–10,000 | 400 | Constant workload |
restartSecond |
1–window−1 | 173 | Counter reset time |
startingCounter |
0–1,000,000,000 | 2,000,000 | First cumulative value |
Example:
curl 'http://localhost:8080/api/simulate?windowSeconds=600&scrapeIntervalSeconds=20&requestsPerSecond=250&restartSecond=207'The OpenTelemetry metrics data model defines cumulative monotonic Sums as non-decreasing within one stream and uses start timestamps to describe their time windows. Its reset and gap guidance explains how receivers recognize new cumulative streams and why frequent restarts relative to collection intervals reduce rate accuracy:
Prometheus rate() and increase() automatically adjust for breaks in
monotonic counters, which avoids naive negative subtraction:
Reset detection solves stream reconstruction. It cannot reconstruct events that occurred after the last sample and before the old writer disappeared. Closing that interval requires another observation: a shutdown flush, a durable delta handoff, or application-level evidence.
npm run checkThe test suite executes the real Gnuplot model, checks boundary arithmetic and input clamping, validates OTLP reset timestamps, inspects all three signal types, and exercises the live HTTP endpoints. CI repeats these checks and builds the container image.
MIT