Skip to content

feat(benchmarks): end every benchmark with a conclusion, not just a chart - #48

Merged
jonaas-dev merged 1 commit into
mainfrom
feat/benchmark-conclusions
Sep 6, 2026
Merged

feat(benchmarks): end every benchmark with a conclusion, not just a chart#48
jonaas-dev merged 1 commit into
mainfrom
feat/benchmark-conclusions

Conversation

@jonaas-dev

Copy link
Copy Markdown
Owner

The tool showed a chart, a table and an EXPLAIN plan, then left the reader to work out what any of it meant. For a project whose purpose is teaching basic SQL behaviour, the lesson was the missing part.

Every benchmark now returns a Takeaway: a verdict, the evidence behind it, and the advice that follows. Every number is computed from the run that just happened, so it cannot drift away from the chart above it — and the dataclass refuses to be constructed without evidence.

The organising idea: who actually pays

CostCentre is what makes the four benchmarks a curriculum rather than four charts, because it decides the fix.

Benchmark Cost centre Verdict it produced on a real run
select_star CLIENT "Asking for every column cost 5.3x more time for exactly the same rows"
index_usage TRANSFER "The index made PostgreSQL 16.0x faster, but the request only 2.5x"
pagination DATABASE "By page 100, OFFSET makes PostgreSQL do 56x more work to return the same 100 rows"
join_vs_subquery TRANSFER "JOIN shipped 2.4x more rows than IN and EXISTS to answer the same question"

select_star is the case that motivated this. PostgreSQL is barely involved — the time goes to the driver decoding 16 columns instead of 3, and in a real service to re-serializing them to JSON. The bill lands in your API process and on the frontend waiting for it, which is the actual reason not to return everything. Saying "SELECT * is slow" without saying where hides the fix.

pagination is the deliberate counterexample: there the database really is doing the extra work, and no client tuning helps.

Read together: "the query is slow" is not a diagnosis. The same symptom has a different cure depending on whether the time goes to the planner, the wire, or the driver. The README now opens with that arc.

Also

  • Conclusions render above the chart in both the live and the stored result views, and are persisted with the run so history keeps them.
  • Dropped a misleading growth ratio in the pagination takeaway that divided by 0.01 ms — the resolution floor of EXPLAIN output, so the figure was noise (it read "395x" next to an unrelated "474x").

Verification

  • ruff check . clean, 71 tests (was 60), 93.4% coverage.
  • New gates: every benchmark must produce a takeaway with evidence; select_star must blame the client and pagination must blame the database — if a future change makes SELECT * look like a database problem, CI fails.
  • Full compose stack: all four render the conclusion and the cost centre, and it survives a round trip through the history view.

🤖 Generated with Claude Code

https://claude.ai/code/session_014abw4B6YUf54giaEyPQpbo

…hart

The tool showed a chart, a table and an EXPLAIN plan, then left the reader to
work out what any of it meant. For a project whose purpose is teaching basic
SQL behaviour, the lesson was the missing part.

Every benchmark now returns a Takeaway: a verdict, the evidence behind it,
and the advice that follows. Every number in it is computed from the run that
just happened, so it cannot drift away from the chart above it — and the
dataclass refuses to be built without evidence.

The organising idea is CostCentre: who actually pays.

- select_star -> CLIENT. PostgreSQL is barely involved; the time goes to the
  driver decoding 16 columns instead of 3, and in a real service to
  re-serializing them to JSON. The bill lands in the API process and on the
  frontend waiting for it, which is the whole reason not to return
  everything.
- index_usage -> TRANSFER. The index is worth ~8x to the database but ~2x to
  the caller. An index speeds up finding rows, never sending them.
- pagination -> DATABASE. Unlike the above, this really is PostgreSQL doing
  the extra work, so only a query change fixes it.
- join_vs_subquery -> TRANSFER. The three patterns answer different
  questions; JOIN ships 2.4x more rows for the same users.

Read together they make one point: "the query is slow" is not a diagnosis.
The same symptom has a different cure depending on whether the time goes to
the planner, the wire or the driver.

Conclusions are rendered above the chart in both the live and the stored
result views, and persisted with the run so history keeps them.

Also drops a misleading growth ratio in the pagination takeaway that divided
by 0.01 ms — the resolution floor of EXPLAIN output, so the figure was noise.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014abw4B6YUf54giaEyPQpbo
@jonaas-dev
jonaas-dev merged commit 2f9f5e7 into main Sep 6, 2026
1 check passed
@jonaas-dev
jonaas-dev deleted the feat/benchmark-conclusions branch September 6, 2026 19:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant