Skip to content

feat(samples): measurable, production-realistic sample with a control group - #7

Merged
FranRuiz98 merged 5 commits into
masterfrom
feat/samples-production-realistic
Jul 25, 2026
Merged

feat(samples): measurable, production-realistic sample with a control group#7
FranRuiz98 merged 5 commits into
masterfrom
feat/samples-production-realistic

Conversation

@FranRuiz98

Copy link
Copy Markdown
Owner

Turns the sample from something that demonstrates the library into something that measures it, and makes it impossible to rot.

Why

The sample worked, but it had three problems:

  • It could not measure anything. There was no control group, so "caching helps" was an assertion, not a number.
  • Several features were invisible. /greetings existed but nothing ever called it, so v2.2.0's headline Vary support was never exercised. Per-request policies, CoalesceKeyHeaders, only-if-cached, immutable and the size ceiling had no demonstration at all.
  • Nothing built it. The sample projects were not in Stampede.Http.slnx and no CI job touched them, so they could stop compiling — or stop working — silently.

What changed

Measurement

  • client-baseline: the same image, workload and Polly pipeline with Sample:Pipeline:Enabled=false, which removes the two Stampede.Http handlers and nothing else.
  • The origin now exports Prometheus metrics, counting every request that actually reached it, tagged by endpoint and by the X-Client header the clients send. The with/without comparison is one PromQL query.
  • The Grafana dashboard is rebuilt around that: origin req/s per client, origin load avoided as a percentage, and a per-endpoint breakdown — plus caching and coalescing sections keyed on a client label rather than the scrape instance.

Feature coverage

A narrated feature tour: 12 scenarios, each verified against the origin's own request counters rather than the client's opinion of what happened. Covers Vary variants, CoalesceKeyHeaders, conditional pass-through, ForceRevalidate / BypassCache / NoStore / BypassCoalescing, only-if-cached, immutable, MaxBodySizeBytes, NormalizeQueryParameters and Last-Modified revalidation. The origin gains the endpoints those need (/tenants/data, /assets/{id}, /ledger, /docs/{id}, /bulk, /search, /health, POST /stats/reset).

Realism

  • The client goes from a console script to an ASP.NET Core service: typed client, IOptions with validation, ILogger, endpoints under /api so load can be driven by real inbound HTTP, IHostedService with graceful shutdown.
  • /metrics is served on the application port, which removes the HttpListener host-binding workaround the old side-channel exporter needed (METRICS__HOST and its long explanatory comment are gone).
  • k6 profile driving identical arrival patterns at both pipelines.
  • OTLP traces + Jaeger: a coalesced burst is one outgoing span, which no counter can show.
  • config/ bind-mounted with reloadOnChange to demonstrate IOptionsMonitor, plus GET /api/config to read the effective options.

Hygiene

  • Both sample projects added to Stampede.Http.slnx, so CI compiles them.
  • scripts/smoke-test.sh and a CI job that brings the whole stack up and asserts what the README claims.
  • Real healthchecks and depends_on conditions instead of a 30-attempt retry loop.
  • *.sh / Dockerfile / *.yml pinned to LF in .gitattributes — a CRLF shell script fails in a Linux container.
  • The root README links the sample, which it never mentioned.

Things found by actually running it

Three of these were only visible once the stack was up, and each is fixed in place:

  1. The coalescer's MaxResponseBodyBytes fires before the cache's MaxBodySizeBytes, and throws. The /bulk scenario killed the process. The two ceilings fail differently — the cache declines silently, the coalescer throws for every waiter — so the sample now sets the coalescer's limit above the cache's, with a comment explaining why.
  2. Grafana silently ignored dashboard edits. The JSON pinned "version": 2, and the file provisioner skips a provisioned dashboard whose version has not increased. Removed.
  3. The "deduplicated requests" panel sat at zero forever. The metric was correct; the workload was not. Phase 3 issued its probes sequentially, so no two requests overlapped and the coalescer correctly had nothing to do. Added a burst of 8 concurrent GET /slow every 5 iterations. It targets /slow, which is excluded from the comparison selector, so the headline percentage stays a floor rather than a figure inflated by choosing favourable traffic.

For reviewers

The headline "origin load avoided" reads about 40%, and the README now spends a section explaining why that number is honest but pessimistic rather than just printing it:

  • /flaky accounts for most of what is not avoided, because a failure cannot be cached ahead of timestale-if-error rescues the caller after the origin has failed.
  • The sample runs 5–10 s TTLs against a ~3 s poll so expiry is visible in a 90-second demo. The ceiling is 1 − interval / max-age; /catalog and /feed land within a few points of it.
  • The per-client normalisation hides the best result: because client-a and client-b share one Redis cache they share the refresh work, so together they cost the origin less than the single uncached client on /catalog and /feed. On /flaky the opposite holds — uncacheable failures scale with replica count.
  • On /slow, where the burst lands, the saving is ~95% (0.30 req/s at the origin for the control client vs 0.00–0.02 with the library).

No library code changed — this is entirely samples/, plus the solution file, CI, .gitattributes and a link from the root README.

Verification

  • 343 tests pass; solution builds warning-free.
  • scripts/smoke-test.sh green from a clean stack: feature tour 12/12, a 20-caller burst costs 0 origin calls against the control client's 28.
  • k6 run completes: p95 1.37 ms against the Stampede.Http client.
  • Prometheus scraping all four targets; Jaeger receiving from sample-api and sample-client; hot reload confirmed end to end.

The sample demonstrated the library but could not measure it, left several
features invisible, and was not built by anything — so it could rot silently.

Measurement
- Add client-baseline: the same image and workload with Sample:Pipeline:Enabled
  =false, which removes the two Stampede.Http handlers and nothing else.
- The origin now exports Prometheus metrics, counting every request that
  actually reached it, tagged by endpoint and by the X-Client header. The
  with/without comparison is one PromQL query.
- Rebuild the Grafana dashboard around that: origin req/s per client, origin
  load avoided as a percentage, plus caching and coalescing sections keyed on a
  `client` label rather than the scrape `instance`.

Feature coverage
- Add a narrated feature tour: 12 scenarios, each verified against the origin's
  own counters. Covers Vary variants, CoalesceKeyHeaders, conditional
  pass-through, ForceRevalidate/BypassCache/NoStore/BypassCoalescing,
  only-if-cached, immutable, MaxBodySizeBytes, NormalizeQueryParameters and
  Last-Modified revalidation. /greetings existed but nothing ever called it.
- Add the origin endpoints those need: /tenants/data, /assets/{id}, /ledger,
  /docs/{id}, /bulk, /search, /health, POST /stats/reset.

Realism
- Convert the client from a console script to an ASP.NET Core service: typed
  client, IOptions + validation, ILogger, endpoints under /api so load can be
  driven by real inbound HTTP, IHostedService with graceful shutdown.
- Serve /metrics on the application port, removing the HttpListener host-binding
  workaround the old side-channel exporter needed.
- Add a k6 profile driving identical arrival patterns at both pipelines.
- Add OTLP traces + Jaeger: a coalesced burst is one outgoing span, which no
  counter can show.
- Bind-mount config/ with reloadOnChange to demonstrate IOptionsMonitor, and
  expose /api/config to read the effective options.

Hygiene
- Add both sample projects to Stampede.Http.slnx so CI compiles them.
- Add scripts/smoke-test.sh and a CI job that runs the whole stack and asserts
  what the README claims.
- Real healthchecks and depends_on conditions instead of a 30-attempt retry loop.
- Pin *.sh/Dockerfile/*.yml to LF in .gitattributes.
- Link the sample from the root README, which never mentioned it.

The coalescer's MaxResponseBodyBytes is raised above the cache's
MaxBodySizeBytes: the two ceilings fail differently, and the /bulk scenario
needs the cache to decline silently rather than the coalescer to throw.

Verified end to end: 343 tests pass, solution builds warning-free, smoke test
green from a clean stack, k6 run completes with p95 1.37 ms against the
Stampede.Http client.
…nting it

The dashboard reported a single "origin load avoided" number with no way to
tell whether it was good, and it is misleading in both directions.

- Add an "Origin load avoided by endpoint" panel. It is the one that explains
  the headline: /catalog and /feed sit near their theoretical ceiling while
  /flaky sits far below, because its 503s reach the origin no matter what.
- Document the breakdown in the README with measured numbers, the
  1 - interval/max-age ceiling, and why the sample's 5-10 s TTLs are
  deliberately hostile.
- Record what the per-client normalisation hides: because client-a and
  client-b share one Redis cache they share the refresh work, so together they
  cost the origin less on /catalog and /feed than the single uncached client.
  On /flaky the opposite holds — uncacheable failures scale with replica count.
- Note that the figures move by several points between windows, since /flaky's
  60 s failure cycle never divides evenly into a 5 m rate window.
- Drop the pinned "version" from the dashboard JSON: Grafana's file provisioner
  skips a provisioned dashboard whose version has not increased, so edits were
  silently ignored until the container was restarted.
The "deduplicated requests" panel sat at zero forever. The metric was fine —
the counter had already reached 21 during the opening stampede and the feature
tour — but Phase 3 issued its three probes sequentially, so no two requests
were ever in flight and the coalescer correctly had nothing to do. A steady
state with zero concurrency models no real service and made the library's
headline feature look dead.

- Add a burst of 8 concurrent GET /slow every 5 iterations (BurstEvery /
  BurstSize). It alternates visibly: 0 ms while the 30 s entry is warm, 2000 ms
  the moment it expires, when one origin call serves all eight.
- Target /slow, which is excluded from the origin-load comparison selector, so
  the headline percentage is unaffected and stays a floor rather than a figure
  inflated by the traffic shape chosen to demo coalescing.
- Chart deduplication as increase over 2 min rather than a 1 min rate: the
  signal is inherently bursty and a rate flattens it to nothing.
- Note on the in-flight panel that it is a gauge sampled every 5 s, so short
  bursts can fall between scrapes and read zero.
- Document the /slow saving that the headline deliberately leaves out:
  0.30 req/s at the origin for the control client, 0.00-0.02 with the library.
- Widen the smoke test's burst tolerance to 2, since the workload's own /slow
  burst can now land inside the measurement window.
… endpoint

A single "origin load avoided" figure invited the wrong conclusion. /flaky
returns 503 for a third of every minute, and uncacheable failures are the
largest single source of origin traffic in the sample — the cached clients
actually send MORE of it than the control client, because every replica must
ask before stale-if-error can rescue it and Polly then retries twice. Failures
scale with replica count; cache hits are shared.

- Add a second headline stat, "Origin load avoided (healthy traffic)", whose
  selector differs from the first only by excluding /flaky, so the pair
  isolates one variable rather than confounding two. Reads ~57% against the
  all-traffic ~37%.
- Move "Cache hit ratio" out of the origin-load row into the Caching section,
  where it belongs — it is a client-side measurement, and its description now
  says why it is not the same thing as origin load avoided.
- Consolidate the README's three overlapping tables into one rate table plus a
  scenario table, and drop the trailing PromQL snippet now that a panel does
  the same job.
- Record the aggregate result the per-client normalisation hides: excluding
  /flaky, two Stampede.Http replicas generate 45% less origin load than a
  single uncached client — twice the application capacity for half the traffic.
The samples-smoke job failed on one check: the origin's /metrics did not yet
contain sample_api_origin_requests. Eleven of twelve checks passed, including
the 12/12 feature tour, so the stack was fine — the test was not.

An OpenTelemetry counter emits nothing at all until it has recorded its first
measurement, and the Prometheus exporter caches scrape responses on top of
that. Reproduced locally: 1 s after the origin starts serving, /metrics has no
matching lines; 2 s later it has eleven. A container reporting healthy does not
mean its instruments are exported yet, and asserting on that with a single
request is a race that a loaded CI runner loses.

- Add retry_until, and poll the metric assertions for up to 60 s rather than
  sampling them once.
- On failure, print what the endpoint actually returned, so the next failure is
  diagnosable from the CI log instead of needing a local reproduction.
- Replace "Prometheus answers queries" with "the sample-api target is up",
  which is what the dashboards actually depend on; the previous check passed
  even while the origin was not being scraped at all.
- Give the cache-hit check the same treatment.
@FranRuiz98
FranRuiz98 merged commit 523fe64 into master Jul 25, 2026
3 checks passed
@FranRuiz98
FranRuiz98 deleted the feat/samples-production-realistic branch July 25, 2026 11:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant