feat(samples): measurable, production-realistic sample with a control group - #7
Merged
Merged
Conversation
The sample demonstrated the library but could not measure it, left several
features invisible, and was not built by anything — so it could rot silently.
Measurement
- Add client-baseline: the same image and workload with Sample:Pipeline:Enabled
=false, which removes the two Stampede.Http handlers and nothing else.
- The origin now exports Prometheus metrics, counting every request that
actually reached it, tagged by endpoint and by the X-Client header. The
with/without comparison is one PromQL query.
- Rebuild the Grafana dashboard around that: origin req/s per client, origin
load avoided as a percentage, plus caching and coalescing sections keyed on a
`client` label rather than the scrape `instance`.
Feature coverage
- Add a narrated feature tour: 12 scenarios, each verified against the origin's
own counters. Covers Vary variants, CoalesceKeyHeaders, conditional
pass-through, ForceRevalidate/BypassCache/NoStore/BypassCoalescing,
only-if-cached, immutable, MaxBodySizeBytes, NormalizeQueryParameters and
Last-Modified revalidation. /greetings existed but nothing ever called it.
- Add the origin endpoints those need: /tenants/data, /assets/{id}, /ledger,
/docs/{id}, /bulk, /search, /health, POST /stats/reset.
Realism
- Convert the client from a console script to an ASP.NET Core service: typed
client, IOptions + validation, ILogger, endpoints under /api so load can be
driven by real inbound HTTP, IHostedService with graceful shutdown.
- Serve /metrics on the application port, removing the HttpListener host-binding
workaround the old side-channel exporter needed.
- Add a k6 profile driving identical arrival patterns at both pipelines.
- Add OTLP traces + Jaeger: a coalesced burst is one outgoing span, which no
counter can show.
- Bind-mount config/ with reloadOnChange to demonstrate IOptionsMonitor, and
expose /api/config to read the effective options.
Hygiene
- Add both sample projects to Stampede.Http.slnx so CI compiles them.
- Add scripts/smoke-test.sh and a CI job that runs the whole stack and asserts
what the README claims.
- Real healthchecks and depends_on conditions instead of a 30-attempt retry loop.
- Pin *.sh/Dockerfile/*.yml to LF in .gitattributes.
- Link the sample from the root README, which never mentioned it.
The coalescer's MaxResponseBodyBytes is raised above the cache's
MaxBodySizeBytes: the two ceilings fail differently, and the /bulk scenario
needs the cache to decline silently rather than the coalescer to throw.
Verified end to end: 343 tests pass, solution builds warning-free, smoke test
green from a clean stack, k6 run completes with p95 1.37 ms against the
Stampede.Http client.
…nting it The dashboard reported a single "origin load avoided" number with no way to tell whether it was good, and it is misleading in both directions. - Add an "Origin load avoided by endpoint" panel. It is the one that explains the headline: /catalog and /feed sit near their theoretical ceiling while /flaky sits far below, because its 503s reach the origin no matter what. - Document the breakdown in the README with measured numbers, the 1 - interval/max-age ceiling, and why the sample's 5-10 s TTLs are deliberately hostile. - Record what the per-client normalisation hides: because client-a and client-b share one Redis cache they share the refresh work, so together they cost the origin less on /catalog and /feed than the single uncached client. On /flaky the opposite holds — uncacheable failures scale with replica count. - Note that the figures move by several points between windows, since /flaky's 60 s failure cycle never divides evenly into a 5 m rate window. - Drop the pinned "version" from the dashboard JSON: Grafana's file provisioner skips a provisioned dashboard whose version has not increased, so edits were silently ignored until the container was restarted.
The "deduplicated requests" panel sat at zero forever. The metric was fine — the counter had already reached 21 during the opening stampede and the feature tour — but Phase 3 issued its three probes sequentially, so no two requests were ever in flight and the coalescer correctly had nothing to do. A steady state with zero concurrency models no real service and made the library's headline feature look dead. - Add a burst of 8 concurrent GET /slow every 5 iterations (BurstEvery / BurstSize). It alternates visibly: 0 ms while the 30 s entry is warm, 2000 ms the moment it expires, when one origin call serves all eight. - Target /slow, which is excluded from the origin-load comparison selector, so the headline percentage is unaffected and stays a floor rather than a figure inflated by the traffic shape chosen to demo coalescing. - Chart deduplication as increase over 2 min rather than a 1 min rate: the signal is inherently bursty and a rate flattens it to nothing. - Note on the in-flight panel that it is a gauge sampled every 5 s, so short bursts can fall between scrapes and read zero. - Document the /slow saving that the headline deliberately leaves out: 0.30 req/s at the origin for the control client, 0.00-0.02 with the library. - Widen the smoke test's burst tolerance to 2, since the workload's own /slow burst can now land inside the measurement window.
… endpoint A single "origin load avoided" figure invited the wrong conclusion. /flaky returns 503 for a third of every minute, and uncacheable failures are the largest single source of origin traffic in the sample — the cached clients actually send MORE of it than the control client, because every replica must ask before stale-if-error can rescue it and Polly then retries twice. Failures scale with replica count; cache hits are shared. - Add a second headline stat, "Origin load avoided (healthy traffic)", whose selector differs from the first only by excluding /flaky, so the pair isolates one variable rather than confounding two. Reads ~57% against the all-traffic ~37%. - Move "Cache hit ratio" out of the origin-load row into the Caching section, where it belongs — it is a client-side measurement, and its description now says why it is not the same thing as origin load avoided. - Consolidate the README's three overlapping tables into one rate table plus a scenario table, and drop the trailing PromQL snippet now that a panel does the same job. - Record the aggregate result the per-client normalisation hides: excluding /flaky, two Stampede.Http replicas generate 45% less origin load than a single uncached client — twice the application capacity for half the traffic.
The samples-smoke job failed on one check: the origin's /metrics did not yet contain sample_api_origin_requests. Eleven of twelve checks passed, including the 12/12 feature tour, so the stack was fine — the test was not. An OpenTelemetry counter emits nothing at all until it has recorded its first measurement, and the Prometheus exporter caches scrape responses on top of that. Reproduced locally: 1 s after the origin starts serving, /metrics has no matching lines; 2 s later it has eleven. A container reporting healthy does not mean its instruments are exported yet, and asserting on that with a single request is a race that a loaded CI runner loses. - Add retry_until, and poll the metric assertions for up to 60 s rather than sampling them once. - On failure, print what the endpoint actually returned, so the next failure is diagnosable from the CI log instead of needing a local reproduction. - Replace "Prometheus answers queries" with "the sample-api target is up", which is what the dashboards actually depend on; the previous check passed even while the origin was not being scraped at all. - Give the cache-hit check the same treatment.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Turns the sample from something that demonstrates the library into something that measures it, and makes it impossible to rot.
Why
The sample worked, but it had three problems:
/greetingsexisted but nothing ever called it, so v2.2.0's headlineVarysupport was never exercised. Per-request policies,CoalesceKeyHeaders,only-if-cached,immutableand the size ceiling had no demonstration at all.Stampede.Http.slnxand no CI job touched them, so they could stop compiling — or stop working — silently.What changed
Measurement
client-baseline: the same image, workload and Polly pipeline withSample:Pipeline:Enabled=false, which removes the two Stampede.Http handlers and nothing else.X-Clientheader the clients send. The with/without comparison is one PromQL query.clientlabel rather than the scrapeinstance.Feature coverage
A narrated feature tour: 12 scenarios, each verified against the origin's own request counters rather than the client's opinion of what happened. Covers
Varyvariants,CoalesceKeyHeaders, conditional pass-through,ForceRevalidate/BypassCache/NoStore/BypassCoalescing,only-if-cached,immutable,MaxBodySizeBytes,NormalizeQueryParametersandLast-Modifiedrevalidation. The origin gains the endpoints those need (/tenants/data,/assets/{id},/ledger,/docs/{id},/bulk,/search,/health,POST /stats/reset).Realism
IOptionswith validation,ILogger, endpoints under/apiso load can be driven by real inbound HTTP,IHostedServicewith graceful shutdown./metricsis served on the application port, which removes theHttpListenerhost-binding workaround the old side-channel exporter needed (METRICS__HOSTand its long explanatory comment are gone).config/bind-mounted withreloadOnChangeto demonstrateIOptionsMonitor, plusGET /api/configto read the effective options.Hygiene
Stampede.Http.slnx, so CI compiles them.scripts/smoke-test.shand a CI job that brings the whole stack up and asserts what the README claims.depends_onconditions instead of a 30-attempt retry loop.*.sh/Dockerfile/*.ymlpinned to LF in.gitattributes— a CRLF shell script fails in a Linux container.Things found by actually running it
Three of these were only visible once the stack was up, and each is fixed in place:
MaxResponseBodyBytesfires before the cache'sMaxBodySizeBytes, and throws. The/bulkscenario killed the process. The two ceilings fail differently — the cache declines silently, the coalescer throws for every waiter — so the sample now sets the coalescer's limit above the cache's, with a comment explaining why."version": 2, and the file provisioner skips a provisioned dashboard whose version has not increased. Removed.GET /slowevery 5 iterations. It targets/slow, which is excluded from the comparison selector, so the headline percentage stays a floor rather than a figure inflated by choosing favourable traffic.For reviewers
The headline "origin load avoided" reads about 40%, and the README now spends a section explaining why that number is honest but pessimistic rather than just printing it:
/flakyaccounts for most of what is not avoided, because a failure cannot be cached ahead of time —stale-if-errorrescues the caller after the origin has failed.1 − interval / max-age;/catalogand/feedland within a few points of it.client-aandclient-bshare one Redis cache they share the refresh work, so together they cost the origin less than the single uncached client on/catalogand/feed. On/flakythe opposite holds — uncacheable failures scale with replica count./slow, where the burst lands, the saving is ~95% (0.30 req/s at the origin for the control client vs 0.00–0.02 with the library).No library code changed — this is entirely
samples/, plus the solution file, CI,.gitattributesand a link from the root README.Verification
scripts/smoke-test.shgreen from a clean stack: feature tour 12/12, a 20-caller burst costs 0 origin calls against the control client's 28.sample-apiandsample-client; hot reload confirmed end to end.