feat(benchmarks): end every benchmark with a conclusion, not just a chart

README.md

@@ -29,6 +29,24 @@ vs `EXISTS`). This tool runs the queries, times them, and shows the query plan b
 
 ---
 
+## What you'll learn
+
+Every benchmark ends with a conclusion the tool derives from the run it just did — never a
+hand-written blurb, so it cannot drift away from the numbers above it. The point of each one is to
+show **where the cost is actually paid**, because that is what decides the fix:
+
+| Benchmark | The lesson | Who pays |
+|---|---|---|
+| **SELECT \* vs columns** | You are not straining the database, you are straining your serializer. Fetching 16 columns instead of 3 costs ~5x end to end while PostgreSQL's own time barely moves — the bill is decoding rows into objects and, in a real service, re-serializing them to JSON. | **The client** — your API process and the frontend waiting on it |
+| **Index usage** | An index speeds up *finding* rows, never *sending* them. The same index is worth ~8x to PostgreSQL but only ~2x to the caller, because `SELECT *` ships the same wide rows either way. | **The result set** |
+| **OFFSET vs keyset** | This one really *is* the database's problem. `OFFSET 100000` makes PostgreSQL walk and discard 100,000 rows to return 100. No client tuning helps; only changing the query does. | **PostgreSQL** |
+| **JOIN vs IN vs EXISTS** | They do not answer the same question. `JOIN` emits one row per child, `IN`/`EXISTS` one row per parent — 2.4x more data for the same users. Choose by the shape you need, not by a stopwatch. | **The result set** |
+
+Read together they make one point: *"the query is slow"* is not a diagnosis. The same symptom has a
+different cure depending on whether the time is going to the planner, the wire, or your driver.
+
+---
+
 ## How the measurements work
 
 The numbers are only worth something if the method is. This is what the tool does on every data point:
@@ -328,10 +346,16 @@ class CityFilterBenchmark(BenchmarkBase):
 | `required_tables` | `list[str]` | Enforced by `check_requirements()` |
 
 `BenchmarkResult` carries `queries` (a list of `QueryResult`), a `comparison_table` DataFrame, a
-`plot_buffer` PNG, and optionally `explain_plans`. When `explain_plans` is empty the runner collects
+`plot_buffer` PNG, a `takeaway`, and optionally `explain_plans`. When `explain_plans` is empty the runner collects
 them automatically — supply them yourself when the plan must be captured at a specific moment, as
 `index_usage` does.
 
+**End with a conclusion.** Set `takeaway=` on your `BenchmarkResult`: a `Takeaway` carries a
+`verdict`, a `CostCentre` (`DATABASE`, `CLIENT` or `TRANSFER`), the `points` of evidence behind it
+and the `advice` that follows. Build every number in it from the measurements you just took — a
+takeaway that repeats a number the chart does not show is how a benchmark starts lying. It refuses
+to be constructed without evidence, on purpose.
+
 **Helpers worth using:** `measure(cursor, query, params)` returns a `Measurement` with
 `rows_fetched`, `wall_ms`, `server_ms` and a `client_share` property, doing the warm-up, the median
 and the `EXPLAIN` for you. `table_row_count(conn)` lets you size your data points to the dataset

app/history.py

@@ -12,6 +12,20 @@
 TIMESTAMP_FORMAT = "%Y-%m-%dT%H-%M-%S"
 
 
+def _takeaway_to_dict(takeaway) -> dict | None:
+    if takeaway is None:
+        return None
+    return {
+        "verdict": takeaway.verdict,
+        "cost_centre": {
+            "label": takeaway.cost_centre.label,
+            "description": takeaway.cost_centre.description,
+        },
+        "points": takeaway.points,
+        "advice": takeaway.advice,
+    }
+
+
 def _unique_dir(dirname: str) -> Path:
     """Two runs within the same second would otherwise overwrite each other."""
     candidate = RESULTS_DIR / dirname
@@ -50,6 +64,7 @@ def save_result(result: BenchmarkResult, params: dict) -> str:
         "description": result.description,
         "timestamp": timestamp,
         "params": params,
+        "takeaway": _takeaway_to_dict(result.takeaway),
         "queries": [
             {"name": q.name, "query": q.query, "limits": q.limits, "times": q.times,
              "rows_fetched": q.rows_fetched}

app/routes.py

@@ -22,6 +22,7 @@ def _results_page(**kwargs):
         "selected_benchmark": None,
         "selected_size": "medium",
         "sizes": SIZES,
+        "takeaway": None,
     }
     return render_template("results.html", **{**defaults, **kwargs})
 
@@ -73,6 +74,7 @@ def generate():
         benchmark_title=result.title,
         benchmark_description=result.description,
         explain_plans=result.explain_plans or None,
+        takeaway=result.takeaway,
         result_id=result_id,
     )
 
@@ -108,6 +110,7 @@ def view_result(result_id):
     return render_template(
         "view_result.html",
         result=data,
+        takeaway=data.get("takeaway"),
         plot_data=data.get("plot_data"),
         comparison_table=data.get("comparison_table"),
         explain_plans=data.get("explain_plans"),

app/templates/results.html

@@ -90,6 +90,36 @@ <h5 class="mb-0"><i class="bi bi-bar-chart me-2"></i>{{ benchmark_title }}</h5>
                 </div>
                 {% endif %}
 
+                {% if takeaway %}
+                <div class="card shadow-sm mb-4 border-success">
+                    <div class="card-header bg-success text-white d-flex align-items-center">
+                        <i class="bi bi-lightbulb me-2"></i>
+                        <h5 class="mb-0">What this tells you</h5>
+                    </div>
+                    <div class="card-body">
+                        <p class="fs-5 fw-semibold mb-3">{{ takeaway.verdict }}.</p>
+
+                        <div class="mb-3">
+                            <span class="badge bg-dark me-2">Cost paid by</span>
+                            <strong>{{ takeaway.cost_centre.label }}</strong>
+                            <div class="text-muted small mt-1">{{ takeaway.cost_centre.description }}</div>
+                        </div>
+
+                        <ul class="mb-3">
+                            {% for point in takeaway.points %}
+                            <li class="mb-1">{{ point }}</li>
+                            {% endfor %}
+                        </ul>
+
+                        {% if takeaway.advice %}
+                        <div class="alert alert-light border mb-0">
+                            <i class="bi bi-arrow-right-circle me-1"></i>{{ takeaway.advice }}
+                        </div>
+                        {% endif %}
+                    </div>
+                </div>
+                {% endif %}
+
                 {% if plot_data %}
                 <div class="card shadow-sm mb-4">
                     <div class="card-header bg-white">

app/templates/view_result.html

@@ -41,6 +41,36 @@ <h5 class="mb-0"><i class="bi bi-bar-chart me-2"></i>{{ result.title or result.b
             </div>
         </div>
 
+        {% if takeaway %}
+        <div class="card shadow-sm mb-4 border-success">
+            <div class="card-header bg-success text-white d-flex align-items-center">
+                <i class="bi bi-lightbulb me-2"></i>
+                <h5 class="mb-0">What this tells you</h5>
+            </div>
+            <div class="card-body">
+                <p class="fs-5 fw-semibold mb-3">{{ takeaway.verdict }}.</p>
+
+                <div class="mb-3">
+                    <span class="badge bg-dark me-2">Cost paid by</span>
+                    <strong>{{ takeaway.cost_centre.label }}</strong>
+                    <div class="text-muted small mt-1">{{ takeaway.cost_centre.description }}</div>
+                </div>
+
+                <ul class="mb-3">
+                    {% for point in takeaway.points %}
+                    <li class="mb-1">{{ point }}</li>
+                    {% endfor %}
+                </ul>
+
+                {% if takeaway.advice %}
+                <div class="alert alert-light border mb-0">
+                    <i class="bi bi-arrow-right-circle me-1"></i>{{ takeaway.advice }}
+                </div>
+                {% endif %}
+            </div>
+        </div>
+        {% endif %}
+
         {% if plot_data %}
         <div class="card shadow-sm mb-4">
             <div class="card-header bg-white">

benchmarks/base.py

@@ -3,6 +3,7 @@
 import time
 from abc import ABC, abstractmethod
 from dataclasses import dataclass, field
+from enum import Enum
 from io import BytesIO
 
 import pandas as pd
@@ -41,6 +42,58 @@ def client_share(self) -> float | None:
         return max(0.0, (self.wall_ms - self.server_ms) / self.wall_ms)
 
 
+class CostCentre(Enum):
+    """Who actually pays for a slow query.
+
+    Keeping these apart is the single most useful thing this tool does: a cost
+    paid in the driver is fixed by asking for less data, while a cost paid in
+    PostgreSQL is fixed by changing the query or the indexes. Optimising the
+    wrong one wastes weeks.
+    """
+
+    DATABASE = (
+        "PostgreSQL",
+        "PostgreSQL itself does the extra work — more rows scanned, a worse plan. "
+        "Fix it with indexes or a different query shape.",
+    )
+    CLIENT = (
+        "the client",
+        "PostgreSQL is barely involved. The time goes to transferring rows and to the "
+        "driver turning bytes into objects, so the bill lands in your API process, not "
+        "your database. Fix it by asking for less data.",
+    )
+    TRANSFER = (
+        "the result set",
+        "Both sides are doing reasonable work; there is simply more data crossing the "
+        "wire than the question required. Fix it by returning fewer rows or columns.",
+    )
+
+    def __init__(self, label, description):
+        self.label = label
+        self.description = description
+
+
+@dataclass
+class Takeaway:
+    """The conclusion a benchmark reached, derived from the run it just did.
+
+    Built from measured values rather than written by hand, so it cannot drift
+    away from the chart above it.
+    """
+
+    verdict: str
+    cost_centre: CostCentre
+    points: list[str]
+    #: What to do about it.
+    advice: str = ""
+
+    def __post_init__(self):
+        if not self.verdict.strip():
+            raise ValueError("a takeaway needs a verdict")
+        if not self.points:
+            raise ValueError("a takeaway needs evidence; a conclusion without it is an opinion")
+
+
 @dataclass
 class QueryResult:
     name: str
@@ -63,6 +116,7 @@ class BenchmarkResult:
     comparison_table: pd.DataFrame
     plot_buffer: BytesIO
     explain_plans: dict[str, str] = field(default_factory=dict)
+    takeaway: "Takeaway | None" = None
 
 
 def server_execution_ms(cursor, query, params=None) -> float | None:
@@ -120,6 +174,16 @@ def speedup(slow: float | None, fast: float | None) -> str:
     return f"{slow / fast:.1f}x"
 
 
+def ratio(slow: float | None, fast: float | None) -> float | None:
+    if slow is None or fast is None or fast <= 0:
+        return None
+    return slow / fast
+
+
+def percent(value: float | None) -> str:
+    return "n/a" if value is None else f"{value * 100:.0f}%"
+
+
 def plot_server_series(ax, query, color):
     """Dashed companion line showing PostgreSQL's own time.
 

benchmarks/index_usage.py

@@ -11,10 +11,13 @@
 from benchmarks.base import (
     BenchmarkBase,
     BenchmarkResult,
+    CostCentre,
     QueryResult,
+    Takeaway,
     format_ms,
     measure,
     plot_server_series,
+    ratio,
     speedup,
 )
 from benchmarks.registry import register
@@ -79,6 +82,41 @@ def run(self, conn) -> BenchmarkResult:
             comparison_table=self._build_comparison(q1, q2),
             plot_buffer=self._build_plot(q1, q2),
             explain_plans={q1.name: no_index_plan, q2.name: with_index_plan},
+            takeaway=self._takeaway(q1, q2),
+        )
+
+    def _takeaway(self, q1: QueryResult, q2: QueryResult) -> Takeaway:
+        i = len(q1.limits) // 2
+        rows = q1.rows_fetched[i]
+        total = ratio(q1.times[i], q2.times[i])
+        server = ratio(q1.server_times[i], q2.server_times[i])
+
+        points = [
+            f"Both queries return the same {rows:,} rows; only the index differs.",
+            f"Inside PostgreSQL: {format_ms(q1.server_times[i])} -> "
+            f"{format_ms(q2.server_times[i])}" + (f" ({server:.1f}x faster)." if server else "."),
+            f"End to end: {format_ms(q1.times[i])} -> {format_ms(q2.times[i])}"
+            + (f" (only {total:.1f}x faster)." if total else "."),
+        ]
+        if server and total and server > total:
+            points.append(
+                f"The index is worth {server:.1f}x to the database but the request only sees "
+                f"{total:.1f}x, because shipping {rows:,} wide rows costs the same however "
+                "they were found."
+            )
+
+        return Takeaway(
+            verdict=(
+                f"The index made PostgreSQL {server:.1f}x faster, but the request only {total:.1f}x"
+                if server and total else "The index sped up the scan"
+            ),
+            cost_centre=CostCentre.TRANSFER,
+            points=points,
+            advice=(
+                "An index speeds up *finding* rows, never *sending* them. Indexing the filter is "
+                "the right move, but as long as you SELECT * the transfer cost caps what you can "
+                "win. Narrow the projection and the index gain reaches the caller."
+            ),
         )
 
     def teardown(self, conn) -> None:

benchmarks/join_vs_subquery.py

@@ -11,9 +11,12 @@
 from benchmarks.base import (
     BenchmarkBase,
     BenchmarkResult,
+    CostCentre,
     QueryResult,
+    Takeaway,
     format_ms,
     measure,
+    ratio,
 )
 from benchmarks.registry import register
 
@@ -112,6 +115,49 @@ def run(self, conn) -> BenchmarkResult:
                 q.name: run_explain(conn, q.query, (EXPLAIN_THRESHOLD,))
                 for q in (q1, q2, q3)
             },
+            takeaway=self._takeaway(q1, q2, q3),
+        )
+
+    def _takeaway(self, q1: QueryResult, q2: QueryResult, q3: QueryResult) -> Takeaway:
+        i = 0
+        join_rows, user_rows = q1.rows_fetched[i], q2.rows_fetched[i]
+        fanout = ratio(join_rows, user_rows)
+
+        points = [
+            f"JOIN returned {join_rows:,} rows; IN and EXISTS returned {user_rows:,}"
+            + (f" — {fanout:.1f}x more data for the same users." if fanout else "."),
+            "That is not a tie-break, it is a different answer: JOIN emits one row per "
+            "matching order, IN and EXISTS one row per matching user.",
+            f"Inside PostgreSQL, JOIN is the fastest of the three "
+            f"({format_ms(q1.server_times[i])} vs {format_ms(q2.server_times[i])} and "
+            f"{format_ms(q3.server_times[i])}).",
+            f"End to end it is the slowest ({format_ms(q1.times[i])} vs "
+            f"{format_ms(q2.times[i])} and {format_ms(q3.times[i])}), because those extra "
+            "rows still have to reach you.",
+        ]
+        if q2.server_times[i] and q3.server_times[i]:
+            close = abs(q2.server_times[i] - q3.server_times[i]) / max(q2.server_times[i], 0.001)
+            if close < 0.35:
+                points.append(
+                    "IN and EXISTS land within noise of each other: the planner rewrites both "
+                    "into the same semi-join. Check the EXPLAIN output below."
+                )
+
+        return Takeaway(
+            verdict=(
+                f"JOIN shipped {fanout:.1f}x more rows than IN and EXISTS to answer "
+                "the same question"
+                if fanout else "The three patterns do not return the same rows"
+            ),
+            cost_centre=CostCentre.TRANSFER,
+            points=points,
+            advice=(
+                "Pick the pattern by the shape of the answer you need, not by a benchmark. If you "
+                "need columns from the child table, JOIN is the only option that gives them. If "
+                "you only need to know the parent matched, IN or EXISTS say so without "
+                "duplicating the parent row per child — and you avoid a DISTINCT to undo the "
+                "damage later."
+            ),
         )
 
     def teardown(self, conn) -> None: