DOCEXP.md

# DOCEXP — Engineering Log

A running lab notebook: decisions and why, what broke, what surprised me,
open questions. Written as I go, not after the fact. For the current
system's shape without the history, see [ARCHITECTURE.md](ARCHITECTURE.md).

---

## Slice 1 — Repo, ingestion, guarded DB, minimal self-correcting SQL agent

**Date:** 2026-08-22

### Repo structure

```
src/
  config.py          # one typed Settings object, everything reads from here
  ingestion/          # SteamSpy client + the ingest script
  db/                 # schema + the read-only guarded connection (the safety boundary)
  agent/               # LangGraph graph, prompts, provider seam
  tools/               # run_sql today; stats/forecast/viz tools land in Slice 4
  api/                 # FastAPI — thin, translates HTTP <-> agent
data/
  raw/                 # cached SteamSpy responses (gitignored)
  db/                  # the DuckDB file (gitignored)
```

Why split this way instead of one file: each top-level package maps to one
layer of the final architecture (ingestion, storage, reasoning, tools,
transport), so slices 2-8 mostly mean *adding* files to these packages
rather than restructuring. `api/` importing from `agent/` but not vice versa
is deliberate — the agent has to run standalone (scriptable, testable,
traceable) without ever knowing FastAPI exists. That's also what keeps the
deployment-target decision (see Open Questions) from leaking into the
agent code.

### The safety boundary: SELECT-only enforcement

This is the one part of the system I did not want to get subtly wrong, so
I spent the most care here. Two independent layers:

1. DuckDB connection opened with `read_only=True` — the engine itself
   refuses writes, regardless of what Python code does.
2. `src/db/connection.py::validate_select_only()` parses every query with
   **sqlglot** (a real SQL parser) before it reaches DuckDB at all, and
   rejects anything that isn't a single `SELECT`/CTE statement touching
   only the `games` table.

Layer 2 exists even though layer 1 already blocks writes, because
`read_only=True` doesn't stop everything that could be a problem — `ATTACH`,
`COPY ... TO`, or stacked statements (`SELECT ...; DROP TABLE ...;`) aren't
writes to *our* table but aren't things an LLM-generated query should ever
be allowed to try either. I initially considered regex/keyword blacklisting
(reject if the string contains "DROP", "INSERT", etc.) and deliberately
rejected that approach — it's trivially defeated by comments, string
literals, or case tricks, and it can't tell a legitimate `WITH deleted AS
(...)` CTE from an actual `DELETE`. A real parser can.

Tested the guard directly against a battery of attacks before ever wiring
it to an LLM (`DROP TABLE`, stacked `SELECT; DROP`, `INSERT`, cross-table
`UNION`, `ATTACH 'evil.db'`, unknown table names) — all correctly rejected;
legitimate `SELECT` and CTE queries correctly passed and got their `LIMIT`
clause capped. Row capping is done by editing the parsed AST (add or shrink
a `LIMIT` node) rather than string-concatenating `" LIMIT N"` onto
arbitrary SQL, which would break on queries that already end in a comment
or a trailing semicolon.

**Surprising finding:** the parser-based approach means any query that
isn't literally an `exp.Select` node gets rejected — including legitimate
`UNION` queries, since sqlglot parses those as `exp.Union`, not
`exp.Select`. Not a problem for Slice 1 (single-table catalog, no reason
to union), but worth knowing: if a future slice needs UNION (e.g. comparing
two time ranges), `validate_select_only` needs an explicit allowance for it
rather than assuming "not Select => reject" is always right.

### Agent: hand-written LangGraph loop, not `create_react_agent`

The instruction I'm building to is "every node must be explainable in an
interview," so I wrote the tool-calling loop by hand instead of using
LangGraph's prebuilt agent constructor. It's two nodes:

- `agent`: calls the LLM (with `run_sql` bound as a tool) on the message
  history so far.
- `execute_tools`: runs each tool call through the guarded DB layer,
  turns a DuckDB or validation error into a `ToolMessage` the model reads
  on its next turn, and records the last *successful* query + rows.

The retry cap is enforced structurally, not by asking the model nicely to
stop: once `attempts >= SQL_MAX_RETRIES`, `agent_node` stops calling
`.bind_tools([run_sql])` on that turn at all, so the LLM has no tool to
call and is forced to produce a plain-text final answer. This guarantees
the loop terminates without needing a hard step ceiling as a backstop
(LangGraph's default recursion limit is still there as a last resort, but
it should never be hit in normal operation).

`schema-in-prompt` today is a plain string built by
`src/agent/prompts.py::build_system_prompt()`. It takes an optional
`schema_text` override specifically so Slice 2 (RAG retrieval) can swap in
"only the columns this question needs" without changing the function's
signature or any call site outside of where the retrieval step gets added.

### Model provider seam

`src/agent/llm_provider.py::get_llm()` is the only place allowed to branch
on `MODEL_PROVIDER`. Everything else calls `get_llm()` and gets back a
LangChain `BaseChatModel` — the graph code has no idea whether it's Groq,
Ollama, or (later) Gemini. The Gemini branch currently raises
`NotImplementedError` on purpose: didn't want to add the
`langchain-google-genai` dependency before a slice actually needs it, but
the seam (the `if provider == "gemini":` branch) is there so adding it
later is additive, not a refactor.

### SteamSpy API — what I learned poking at it directly

Two endpoints matter for this slice:
- `request=all&page=0`: bulk listing, ~1000 games per page, already sorted
  by owners descending. Cheap — no rate limit concern, one request gets us
  the whole candidate pool.
- `request=appdetails&appid=X`: per-game detail. This is the *only* place
  `genre`, `languages`, and `tags` show up — the bulk `all` endpoint does
  not include them. SteamSpy's own guidance asks for ~1 request/second on
  this endpoint, which is why ingesting 200 games takes ~3-4 minutes.

Both raw responses get cached to `data/raw/*.json` keyed by request. That's
what makes `ingest.py` idempotent and fast to re-run: a second run replays
cache hits for `appdetails` instead of re-fetching (and re-waiting a full
second per game), and rows are UPSERTed on `appid` via
`ON CONFLICT (appid) DO UPDATE`, so no duplicates either way. Verified this
directly: ran ingestion twice at `--count 20`, row count stayed at 20 both
times, second run finished near-instantly.

### Surprise: Avast TLS interception broke every outbound HTTPS request

First ingestion run failed immediately with
`SSLCertVerificationError: unable to get local issuer certificate` on a
plain `requests.get()` to steamspy.com — before any of our code, just the
TLS handshake. Root cause: this machine's Avast antivirus does HTTPS
web-shield interception, re-signing traffic with its own locally-installed
root CA. Windows trusts that CA (Avast installed it into the system store);
Python's `requests`/`urllib3` don't use the Windows trust store by
default — they validate against `certifi`'s bundled CA list, which
obviously doesn't include a locally-generated antivirus root cert. curl
(used to explore the API earlier) worked fine because it uses the OS trust
store directly on Windows, which is what pointed at the real cause.

Fixed properly rather than papering over it: added the `truststore`
package and called `truststore.inject_into_ssl()` once at import time in
`steamspy_client.py`. This makes the stdlib `ssl` module validate against
the OS trust store instead of `certifi`, which is correct on any machine
(not just this one) and doesn't disable verification (`verify=False` was
the tempting shortcut and deliberately not what I did — that would silently
accept a real MITM too, not just Avast's).

### What worked without any fuss

- DuckDB's `ON CONFLICT ... DO UPDATE` — worked first try, no schema
  migration ceremony needed for an idempotent upsert.
- `pydantic-settings` reading `.env` — one `Settings` object, and
  `load_dotenv()` at the top of `config.py` also populates `os.environ`
  directly, which turned out to matter for LangSmith: its tracer reads
  `LANGSMITH_*` straight from `os.environ`, not from anything our code
  passes it, so tracing "just works" once the `.env` values are set — no
  extra wiring needed beyond making sure they land in the process
  environment before any LangChain code runs.

### Surprise: DuckDB's file lock means you can't ingest and serve at once

While the 200-game ingestion run was in progress (write connection open),
a read-only connection to the same file failed immediately:
`IO Error: Cannot open file ... The process cannot access the file because
it is being used by another process.` DuckDB allows multiple concurrent
*read-only* connections to a file, but a read-write connection takes an
exclusive OS-level lock — no reader can open the file at all until the
writer closes it. Not a bug, just a constraint to design around: the API
can't serve `/ask` while ingestion is running against the same file. Fine
for a local dev loop (ingestion finishes in a few minutes, then serve), but
worth remembering once ingestion is scheduled (Slice 7) — it'll need to
write to a fresh file and swap/rename rather than hold a long-lived write
connection open on the file the live API is reading from.

### Open questions

- **Deployment target** (flagged per the plan, not resolved now): backend
  is Python; free single-platform hosting on Vercel means either Vercel
  Python functions or a Vercel frontend + a separate free Python host
  (Render/Fly/Railway free tier). Revisit at Slice 6. Keeping `src/agent`
  and `src/db` free of any FastAPI import is what keeps both paths open.
- **UNION queries** — noted above; `validate_select_only` will need an
  explicit `exp.Union` allowance if a later slice needs to compare two
  result sets in one query, rather than assuming "anything that isn't
  `exp.Select` is unsafe."
- **Row cap vs. time cap** — Slice 1 only enforces a row cap. A query that
  scans a lot but returns few rows (unlikely on a ~200-row table, but will
  matter once `player_counts` time-series data shows up in Slice 7) has no
  cost guard yet.
- **Ingestion candidate pool** — top-N by owners (page 0 of `all`, already
  sorted) rather than a random or genre-diverse sample. Good enough for a
  first working slice; may want a more representative sample later so the
  agent isn't only ever talking about the same handful of mega-hits.

### End-to-end verification

Ran the full pipeline for real, not just unit-level: ingested 200 games,
started the FastAPI server, and hit `POST /ask` over HTTP (not just calling
`run_agent()` in-process) with all three README example questions.

Tested against **Ollama (llama3.1:8b, local)** rather than Groq, since I
don't have a Groq key in this environment — this doubled as a real test of
the provider seam (`MODEL_PROVIDER=ollama` in `.env`, zero code changes)
before ever running it against Groq.

- *"5 most-owned free-to-play games"* → correct on the first query. Model
  correctly used `(owners_low + owners_high) / 2` as instructed in the
  schema notes rather than picking one bound arbitrarily.
- *"Highest peak concurrent player count"* → correct on the first query,
  clean single-column SQL.
- *"Average price of Action games vs. free-to-play games"* → **failed
  outright on the first attempt.** The model wrote a `UNION` query to
  combine the two averages, which the guard rejects (see the UNION note
  above) — it burned all 3 retries retrying variants of the same rejected
  pattern and gave up, correctly responding "I was unable to complete the
  query" rather than fabricating numbers. That's the safety system doing
  exactly its job (fail honestly, don't hallucinate), but it meant a README
  example question flat-out didn't work. Fixed by adding one line to the
  system prompt telling the model to use conditional aggregation
  (`AVG(CASE WHEN ... THEN price_usd END)`) instead of `UNION` for
  comparing two groups. Retested: worked first try after the prompt change.

  Worth flagging honestly: the corrected query is *subtly wrong* — the
  `CASE WHEN price_usd = 0` free-to-play average ends up computed inside
  the outer `WHERE genre LIKE '%Action%'` filter, so it actually answers
  "average price of free *Action* games," not "average price of free-to-play
  games overall" (it happened to still print $0.00, which is correct by
  coincidence — any qualifying row has price 0). This is a small local
  model's reasoning limitation, not a guardrail/architecture gap — the SQL
  is syntactically valid and the guard has no way to (and shouldn't try to)
  catch semantic mistakes like this. It's exactly the kind of thing the
  Slice 5 eval harness needs to catch systematically, and a reason to
  expect materially better reliability once this runs against Groq's
  70B model instead of a local 8B one.

**Slice 1 status: done.** All checkboxes above are checked; the agent
answers real questions end-to-end through the HTTP API, backed by real
ingested data, with a verified self-correction loop and a verified safety
boundary.

---

## Slice 2 — RAG over the schema

**Date:** 2026-08-24

### What changed structurally

Slice 1's `GAMES_TABLE_DESCRIPTION` was one hardcoded string, always
injected into the system prompt whole. Slice 2 replaces it with:

- `src/agent/rag/schema_corpus.py` — the same facts, broken into small
  independent `SchemaChunk`s: one per table (a one-line orientation), one
  per column, and a handful of "metric notes" for gotchas that aren't tied
  to a single column (unit conversions, the owners-midpoint convention, the
  no-UNION rule).
- `src/agent/rag/embeddings.py` — an embedding-provider seam, structurally
  identical to `llm_provider.py`'s `get_llm()`: one config value
  (`EMBEDDING_PROVIDER`), everything else just calls `get_embedder()`.
- `src/agent/rag/schema_index.py` — a brute-force in-memory cosine-similarity
  index over the ~24 chunks, plus `assemble_schema_text()` to turn a
  retrieved chunk list back into prompt text.
- A new graph node, `retrieve_schema`, now the entry point:
  `retrieve_schema -> agent -> execute_tools -> agent -> ... -> END`. It
  embeds the question, retrieves the top-K chunks, and builds the system
  prompt from only those — same "make it a visible node" principle as
  Slice 1's self-correction loop, so it's traced and explainable, not
  implicit setup.
- `/ask` now also returns `retrieved_schema_chunks` — which chunk IDs the
  retrieval step actually picked for that question. Cheap to add, and it
  turns "trust me, RAG is working" into something you can see per request.

### Embedding provider: why fastembed, not sentence-transformers or a hosted API

Groq has no embeddings endpoint, so "reuse whatever `MODEL_PROVIDER` is
configured" was never on the table — this needed its own decision.
Considered three options:

- **sentence-transformers** (torch-based): the most common choice, but
  pulls in torch as a dependency — heavy (hundreds of MB), and a poor fit
  for a project that's explicitly trying to stay deployable on free-tier
  serverless hosting (Slice 6's open question).
- **A hosted embeddings API** (OpenAI/Cohere/etc.): adds yet another API
  key and a per-request network dependency for something that, at this
  corpus size (~24 chunks), doesn't need to be hosted at all.
- **fastembed** (chosen): ONNX runtime, no torch, small model
  (`BAAI/bge-small-en-v1.5`, ~130MB, quantized), runs in-process. No API
  key, works identically in dev and wherever this ends up deployed. Ollama
  embeddings (`nomic-embed-text`) are wired in as the local-dev alternative
  via the same seam, matching the pattern already established for the chat
  model, but fastembed is the default specifically because it has no
  runtime dependency on a separate daemon being up.

### Same TLS issue, different library — and a cleanup

fastembed's first run failed with the identical
`SSLCertVerificationError` from Slice 1 (Avast TLS interception), this
time on `huggingface_hub`'s model download (via `httpx`, not `requests`).
`truststore.inject_into_ssl()` fixed it the same way. Since this is now
the second independent HTTP client hitting the same problem, moved the fix
out of `steamspy_client.py` and into `src/config.py` (imported by nearly
everything) so it applies once, process-wide, instead of being duplicated
per-module that happens to make HTTP calls.

### Retrieval-quality testing found two real gaps — and they taught something

Tested retrieval directly (not just "does the agent still answer
correctly") by embedding sample questions and inspecting which chunks
ranked in the top-K, before ever wiring it into the graph. Two systematic
misses, both the same underlying cause:

- *"How many owners does Palworld have?"* did not retrieve `column:name`
  in the top 8 (it ranked ~13th).
- *"average playtime ... RPG games"* and *"games tagged as Action"* did
  not retrieve `column:genre` in the top 8 on either question (independent
  tests, same gap).

Root cause: a small bi-encoder embedding model measures semantic
similarity between the *question text* and the *chunk text*. A specific
value — a game's actual name, a specific genre string — doesn't embed
close to a generic column description ("Game title", "comma-separated
genres"). The words that would make the match ("Palworld" is a name; "RPG"
is a genre) aren't in the chunk at all. This is a known-in-the-literature
limitation of pure dense retrieval on short, generic schema text, and I
was glad to catch it empirically with a print statement before assuming
retrieval was "done" just because the agent's answers looked fine.

Fix: added an `always_include` flag to `SchemaChunk`, set on `table:games`,
`column:name`, and `column:genre` — chunks that bypass ranking and are
always present regardless of similarity score. This is deliberately
*not* "always include everything" (that would defeat the point of doing
RAG at all) — it's reserved for chunks that are structurally relevant to
nearly any question (every answer is about specific game(s); genre is one
of the most commonly filtered dimensions here), decided from two
independent failing test questions per column, not from tuning to make
one example pass.

**What this proved, interestingly:** *before* the `column:name` /
`column:genre` fix, the agent still answered the RPG playtime question
correctly — Llama 3.1 8B guessed a column named `genre` existed from
general knowledge of how game databases are usually modeled, wrote a
working query, and it happened to be right. That's a fragile thing to rely
on (a less conventional schema would have broken it), not evidence that
retrieval quality doesn't matter. Worth being honest about: the fix was
made because the *retrieval* was wrong, independent of whether the LLM
covered for it that particular time.

### Real finding: SteamSpy's playtime data is 0 across the entire dataset

While testing the playtime question, the agent reported "0 hours" for
average RPG playtime. Checked the raw data directly rather than assuming
it was a query bug:

```
nonzero avg playtime forever: 0 / 200 games
nonzero avg playtime 2weeks:  0 / 200 games
```

Every single ingested game has `average_playtime_forever_min = 0`. Not an
ingestion bug — this is a documented SteamSpy limitation: since a 2018
Steam privacy API change, SteamSpy has largely been unable to compute
playtime statistics, and the field is 0 for the overwhelming majority of
games in the modern API. Confirmed the raw cached `appdetails` JSON in
`data/raw/` shows `average_forever: 0` straight from the source, not
something introduced by `_row_from_appdetails()`.

Fixed the right layer: this isn't a code bug to patch, it's a data
limitation the agent should disclose. Added a `metric:playtime_often_zero`
chunk to the RAG corpus describing exactly this, and re-tested — the
agent's answer changed from a bare, misleading "0 hours" to "0 hours, due
to a known SteamSpy data limitation..." once that chunk was retrievable.
This is a good demonstration of what RAG-over-metrics is actually *for*:
not just column definitions, but domain caveats a real analyst would know
and a model wouldn't, unless told.

### Reinforcing the Slice 1 finding: small local model, same failure class, different shape

Re-ran all three Slice 1 example questions as a regression check.
Highest-rated and highest-CCU both still correct. The Action-vs-F2P price
comparison — already flagged in Slice 1 as a small-model weak spot — was
wrong *again*, but in a different way this run: it dropped the
`CASE WHEN` filter entirely for the free-to-play side and computed
`AVG(price_usd)` over *all* games, mislabeling the result as
`avg_f2p_price`. Same query pattern, non-deterministic failure mode,
consistent with Slice 1's note that this is Llama 3.1 8B's reasoning
reliability, not a guard or retrieval problem — retrieval correctly
surfaced `genre`, `price_usd`, and the `metric:no_union` note for this
question. Deliberately did not chase this with another prompt patch:
that's exactly the kind of failure the Slice 5 eval harness exists to
catch systematically (across many questions and repeated runs) rather than
whack-a-mole fixing one observed instance at a time.

### Open questions (new)

- **Retrieval evaluation is still eyeballed**, not scored. Slice 5's eval
  harness should include retrieval-quality checks (did the right chunks
  get retrieved for a golden question set), not just end-to-end answer
  correctness.
- **`always_include` is a manual escape hatch.** Fine at ~24 chunks with 2
  forced inclusions; if more tables/columns need this later, it's worth
  asking whether that's a sign the embedding model or chunk phrasing needs
  improving rather than adding more manual overrides.
- **fastembed's model cache is process-local disk, not committed.** First
  run on a fresh machine (or deployment) pays a ~130MB download + ~10s
  init cost. Fine for local dev; worth remembering for Slice 6 (cold-start
  latency on serverless would make this worse — another data point for the
  deployment-target decision).

---

## Slice 3 — Supervisor-router

**Date:** 2026-08-24

### What changed structurally

Added `src/agent/router.py` and a new `router` node, now the graph's entry
point:

```
router -> [lookup/analysis]     -> retrieve_schema -> agent <-> execute_tools -> END
        -> [forecast]           -> forecast_not_supported                     -> END
        -> [needs_clarification]-> ask_clarification                          -> END
```

The router classifies the question into one of four categories using the
LLM's structured-output feature (`with_structured_output(RouteDecision)`,
a Pydantic model with a `Literal` field) rather than a free-text prompt
parsed by hand — the result is always one of exactly four valid values,
no guarding against the model inventing a fifth category or wrapping its
answer in prose needed.

### An honest scoping call: lookup and analysis share a backend today

The four categories are real and independently tested, but only two
distinct *behaviors* exist behind them right now:

- `lookup` and `analysis` both go to the existing `retrieve_schema ->
  agent -> execute_tools` SQL pipeline — the same one, unchanged. That's
  because Slice 4's dedicated statistical-analysis tool (cohorts,
  significance, anomalies) doesn't exist yet; until it does, "analysis"
  and "lookup" are both, mechanically, "write a SELECT and answer." The
  classification is genuine and already correctly distinguishes a
  ranking/filter question from a cross-group comparison — giving
  `analysis` its own handler in Slice 4 is a small additive change to
  `route_after_router`, not a redesign.
- `forecast` and `needs_clarification` get genuinely new, distinct
  behavior: an honest "I can't do that yet" and a clarifying question,
  respectively — neither existed at all before this slice. Before the
  router, a forecast question would have gone straight into the SQL
  agent, which has no time-series data or forecasting logic and would
  have either produced a nonsense query or a confidently wrong answer
  from whatever it managed to compute. Same for an ambiguous question
  like "Is this game good?" — previously the agent would have just
  guessed at a game and answered as if the question were unambiguous.

Called this out explicitly rather than silently merging lookup/analysis
and hoping nobody asks: it's a genuine, deliberate sequencing decision
(build the router before the tools it'll eventually differentiate
between), not a shortcut passed off as complete.

### Testing: classification in isolation, then the full graph, then live HTTP

Same three-layer discipline as Slices 1 and 2 — didn't just check "does
the agent still answer questions."

1. `classify_question()` directly against 4 hand-picked questions (one per
   category, including an intentionally ambiguous one: "Is this game
   good?"). All 4 classified correctly on the first try, and the
   clarifying question generated for the ambiguous one was sensible
   ("What is the name of the game you are referring to?").
2. Full graph via `run_agent()` for the same 4 questions — confirmed
   `route` is populated correctly, confirmed `sql`/`retrieved_chunk_ids`
   are `None` for the forecast and clarification branches (they never
   reach `retrieve_schema`, which is the intended "don't even attempt a
   query" behavior, not a bug), and confirmed the lookup/analysis
   questions still work through the unchanged SQL pipeline underneath.
3. Live HTTP: `POST /ask` with the ambiguous question, confirmed
   `route: "needs_clarification"` and `retrieved_schema_chunks: null` come
   through the API correctly.

Notably, the Action-vs-free-to-play comparison question — flagged in both
Slice 1 and Slice 2 as an unreliable pattern for the local 8B model — came
back *correct* this run. Consistent with the standing conclusion: this is
non-deterministic small-model reliability, not a bug in the guard,
retrieval, or now the router. Still a Slice 5 (eval harness) problem, not
something to keep manually re-testing and hoping for the best on.

### A real interruption, and how it was handled

Mid-testing, the local Ollama runtime crashed (`CUDA error: shared object
initialization failed`) — a transient GPU/driver issue in the Ollama
process itself, unrelated to any code in this repo. Killed both
`ollama.exe` processes and let Ollama's own launcher restart them; the
next request succeeded normally. Noting this mainly as a reminder for
Slice 6: whichever deployment target gets picked, it won't be running a
local GPU-backed Ollama daemon — this class of failure is specific to the
local-dev provider path and shouldn't recur against Groq's hosted API.

### Open questions (new)

- **The router adds one extra LLM round-trip per question**, before any
  DB work happens. Cheap relative to the SQL agent's own calls, but worth
  measuring once there's real latency data (Slice 5/6) — a small/fast
  model dedicated to routing (vs. reusing whatever `MODEL_PROVIDER` is
  configured for the main agent) could be a worthwhile split later if
  routing latency turns out to matter.
- **No test yet for a question that's genuinely borderline between
  lookup and analysis.** The four hand-picked test questions were chosen
  to be unambiguous examples of their category; real user questions won't
  always be this clean. Another concrete case for Slice 5's golden
  question set.

---

## Slice 4 — Specialized analysis tools (stats + viz; forecast deferred)

**Date:** 2026-08-24

### Scoping call: forecasting is not built this slice

The original plan bundles statistical analysis, forecasting, and
visualization into one slice. Built two of three. Forecasting needs
something to forecast *over* — time-series data — and none exists yet;
that's Slice 7's `player_counts` table. A "forecasting tool" today would
have nothing real to operate on. Building one anyway (e.g. a fake linear
extrapolation over a single current-snapshot number) would be decorative,
not real functionality, and this project is explicitly about working
end-to-end slices, not stubs that look like coverage. The router's honest
"can't forecast yet" response from Slice 3 stays as-is; a real forecasting
tool is now explicitly tied to Slice 7 in PLAN.md rather than left dangling
under Slice 4.

### run_stats: three modes chosen for what SQL genuinely can't do well

DuckDB already has real statistical aggregates built in (`corr`, `stddev`,
`quantile_cont`, etc.), so a stats tool that just re-implements `AVG`/`STDDEV`
would be redundant. `src/tools/stats_tool.py` covers what plain SQL
aggregates don't: a real hypothesis test with a p-value
(`compare_two_groups`, Welch's t-test + Cohen's d via `scipy.stats`,
chosen over the equal-variance Student's t-test as the safer default when
group variances aren't known to be equal) and z-score outlier flagging
(`outliers`) — DuckDB has no significance-testing primitive at all, and
while it *could* compute z-scores via a window function, the
interpretation/thresholding logic is cleaner in Python than SQL.
`describe` rounds out the set for plain summary stats.

Every mode runs its query through the same guarded, read-only connection
as `run_sql` (`run_guarded_query`) — this tool is not a bypass of the
SELECT-only/allowlist guard, it's "SQL in, statistics out."

### Closing the loop from Slice 3: analysis finally gets a different toolset

Slice 3's DOCEXP entry flagged this explicitly: "`lookup` and `analysis`
both go to the same backend... giving `analysis` its own handler is a
small additive change to `route_after_router`, not a redesign." That
change landed here: `agent_node` now binds `[run_sql]` for `lookup` and
`[run_sql, run_stats]` for `analysis` (`_tools_for_route()` in graph.py).
The router isn't just a label anymore — it gates what the model is even
capable of doing for a given question.

### Testing: tools in isolation against real data, then the full graph

Same discipline as every prior slice. Ran all three `run_stats` modes
directly against the real 200-game dataset before ever wiring them to an
LLM:

- `compare_two_groups` (Action vs. everything else, price): t = -0.365,
  p = 0.716 → correctly not significant, tiny effect size (Cohen's d =
  -0.059). Sanity-checked by hand: means of $17.81 vs $18.94 are close, so
  a non-significant result is the right answer.
- `outliers` (peak_ccu, z > 2.5): flagged Counter-Strike: Global Offensive
  (z ≈ 13) and PUBG (z ≈ 3.9) — both obviously legitimate standouts, not
  false positives.
- `describe` (review_score): sane bounded output (mean ≈ 0.85, correctly
  within the 0..1 fraction range documented in the RAG corpus).

Then ran real questions through the full graph and live HTTP, including
the exact Action-vs-free-to-play question that's been unreliable since
Slice 1.

### The comparison question, revisited: better tool, more visible failure

With `run_stats` available, the model correctly reached for
`compare_two_groups` *without being asked for significance explicitly* —
progress. The computed statistics were internally correct (t-test, p-value,
Cohen's d all check out against the query it actually ran). But the
query itself mislabeled a group:

```sql
SELECT CASE WHEN genre LIKE '%Action%' THEN 'action' ELSE 'free_to_play' END AS group_label,
       price_usd AS value FROM games
```

The `ELSE` branch is labeled `'free_to_play'` but never checks
`price_usd = 0` — it's actually "everything that isn't Action," which
includes plenty of paid games. The tool computed a perfectly correct
p-value for the wrong comparison, and the final answer ("free-to-play
games average $18.94") was confidently, fluently wrong.

This was only catchable because of a transparency field added *because of*
finding this bug: `AgentResult`/`/ask` didn't originally expose the actual
query behind a stats result the way `sql` does for `run_sql` — added
`stats_query` specifically once this looked suspicious, and it immediately
confirmed the mislabeling. Kept the field permanently: without it, this
class of bug is invisible in the API response — a well-formatted, correct-
looking p-value with no way to check what it was actually computed over.

Tried the direct fix: added an explicit warning to the analysis tool
guidance ("a catch-all ELSE must be labeled generically, not with a name
implying a filter you didn't apply — e.g. `ELSE 'free_to_play'` is WRONG
unless that branch checks price_usd = 0"). Re-ran the identical question:
**same mislabeling, same wrong answer**, word-for-word identical query.
Consistent with the standing conclusion since Slice 1 — this is Llama 3.1
8B's reliability ceiling on this specific pattern, not something more
prompt text fixes. Kept the guidance anyway (real, correct instruction,
worth having for whatever model reads it — likely more effective against
Groq's 70B), logged the negative result rather than hiding it, and did not
keep iterating on the prompt. This is precisely the kind of thing Slice 5's
eval harness needs to catch systematically (run N times, measure the
mislabel rate) instead of one more manual spot-check.

### A real bug, unrelated to model quality: string-typed tool args

Testing the `outliers` mode through the full graph crashed:
`numpy._core._exceptions._UFuncNoLoopError` comparing a float array against
`z_threshold`. Root cause: Ollama's function-calling returned `z_threshold`
as the JSON string `"2.5"` rather than a number, despite the tool schema
declaring it a float — LangChain passes tool-call args through mostly
as-received. This is not a reasoning failure, it's an interface contract
the LLM's tool-calling layer didn't honor. Fixed at the tool boundary
(`execute_run_stats` now does `z_threshold = float(z_threshold)` before use)
rather than trusting the declared schema type — same "validate at the
boundary, don't trust what arrives" principle as the SQL guard, just
applied to argument *types* instead of query *safety* this time.

### Chart-spec generation: deliberately not an LLM call

`src/tools/viz_tool.py::infer_chart_spec()` looks only at the shape of a
successful query result — column count, and whether each column's actual
Python values are numeric or not — to pick bar/scatter/table. This is a
mechanical decision with one correct answer given the shape, so there's no
ambiguity to spend an LLM call resolving; it runs as a plain function in a
new `build_chart_spec` graph node after a successful `run_sql` result, not
as a bindable tool. The spec shape (`{type, x, y, data}`) is deliberately
framework-agnostic — not tied to Vega-Lite/Chart.js/Recharts — so it
doesn't need to change once Slice 6 picks a frontend charting library.

### Open questions (new)

- **The mislabeled-group failure mode is now visible but not prevented.**
  `stats_query` makes it auditable; nothing yet stops the wrong answer
  from reaching the user. Options for later: a lightweight sanity check
  (does a labeled group's aggregate match what the label claims — e.g. if
  a group is labeled `free_to_play`, assert its mean price actually is ~0
  before trusting the label) or just leaning on Slice 5's eval harness to
  quantify how often this happens and whether Groq's larger model avoids
  it.
- **Chart-spec heuristics are untested against a genuinely wide variety of
  query shapes** — validated against the two shapes Slice 4's example
  questions naturally produce (name+numeric, label+numeric). Three+ column
  results and time-series-shaped results (once Slice 7 lands) aren't
  handled yet — currently fall back to `None` (table view) rather than
  guessing.
- **`run_stats`'s modes require the LLM to shape its query correctly**
  (exactly 1 or 2 columns, right order) — same class of risk as any
  LLM-authored SQL. The self-correction loop catches shape errors (they
  raise `ValueError`, which is caught the same way SQL errors are), but a
  golden-question eval would quantify how often that's actually needed.

---

## Slice 5 — Eval harness

**Date:** 2026-08-24

### Built specifically to answer two open questions from Slice 4

Slice 4 ended with two things I couldn't answer from a single manual test:
how *often* the group-mislabeling bug happens, and whether it's specific
to the local 8B model. This slice exists to answer both — not as a
generic "add an eval harness" checkbox, but pointed at a real, previously
observed defect.

### Design: live ground truth, not hardcoded numbers

`src/evals/golden_questions.py::build_golden_questions()` is a function,
not a module-level list — it queries the live DuckDB file for every
reference fact (top peak_ccu game, count of games with review_score > 0.9,
count of free-to-play games) at eval time. Considered hardcoding expected
values instead (simpler, no DB dependency at eval time) and rejected it:
ingestion is idempotent and re-runnable, prices/reviews/owners can change
on a re-ingest, and a hardcoded "expected 47 games" would silently go
stale the next time someone runs `python -m src.ingestion.ingest` and
start failing the suite for the wrong reason (data drift, not a real
regression). Computing ground truth live means the suite is always
checking "does the agent's answer match what's actually in the database
right now" — the only check that stays meaningful over time.

### The bug-specific check, not a generic one

The eval that targets Slice 4's mislabeling bug
(`_check_action_vs_f2p_not_mislabeled` in golden_questions.py) doesn't
compare the agent's stated conclusion to a reference conclusion — it
checks a fact the agent's own claimed label logically implies. Free-to-play
means price = 0 *by definition*; a group the agent calls "free-to-play"
must have a mean price of ~$0, full stop, regardless of what p-value or
comparison it's embedded in. This is a stronger check than "does the
number match the expected number" — it doesn't need to know the correct
number in advance, it just needs the agent's own label to be internally
consistent with the data. Worth being explicit about, because it's the
difference between an eval that would have caught this specific bug
reliably and one that happens to catch it by luck.

### Results against Ollama (llama3.1:8b) — same tool used for every prior slice

```
route accuracy:        6/6
deterministic checks:  5/6
avg judge score:       4.3/5
```

The one failure is exactly the Slice 4 bug, caught by both signals
independently:
- Deterministic check: `group 'free_to_play' claims to be free-to-play but
  has mean price $18.93 (should be ~$0 by definition)`.
- LLM judge (unprompted with any knowledge of the deterministic check):
  score 1/5, rationale *"contradicts the reference facts by stating that
  free-to-play games have an average price of $18.94, which is higher than
  the actual mean price of $0.00."*

Two independently-built signals — one a hand-written numeric assertion,
one an LLM given only the question/answer/reference-facts — converging on
the identical diagnosis is a good sign the check design is sound, not
just tuned to pass on this one run.

**The second open question — whether this is specific to the local 8B
model — is still open.** No Groq key available in this environment to test
against the 70B default. `python -m src.evals.run_evals` is exactly the
command to run once a key is added; the harness was built so answering
that question is "run one command," not "write new test code."

### Why the judge doesn't gate the exit code

Deliberately made `run_evals.py`'s exit code depend only on route
correctness + deterministic checks, not the judge score. The judge is a
second opinion, useful for catching wording/clarity issues the
deterministic checks don't anticipate — but it's still an LLM call with
its own sampling noise, and a "regression check" that can flip pass/fail
on unrelated LLM variance defeats the purpose of having one. The
deterministic checks are the gate; the judge is a report.

### Open questions (new)

- **Six golden questions is a start, not full coverage.** One per route
  plus the two Slice 4-specific regression checks. No coverage yet for
  malformed/adversarial input, multi-part questions, or the RAG retrieval
  quality gaps found in Slice 2 (that's a different kind of eval — scoring
  retrieved chunks against expected chunks — not yet built here).
- **No repeated-run variance measurement.** Ran the suite once per
  provider tested. Given Slice 1-4's evidence that the local model's
  failures are non-deterministic (same question, different wrong answer
  across runs), a single run understates the true failure rate in either
  direction. Running N=5-10 times per question and reporting a pass rate
  would be a more honest reliability number — noted as a natural next
  improvement, not done here to keep this slice thin.
- **No CI wiring yet.** The harness is runnable and has a real exit code,
  but nothing invokes it automatically. Natural fit once there's a CI
  pipeline (implicitly, whenever the deployment slice sets one up).

---

## Slice 6 — Frontend + deploy

**Date:** 2026-08-24

### Resolving the hosting decision, finally

Flagged as open since Slice 1's very first DOCEXP entry: Vercel Python
functions vs. Vercel frontend + a separate Python host. Resolved this
slice by actually looking at what this stack needs, not just picking one:
fastembed's ONNX model (~130MB), scipy, a DuckDB file, and an agent that
can make 2+ sequential LLM calls with retries (10-30s+ isn't unusual, as
directly observed throughout Slices 1-5). Vercel Hobby's Python functions
cap around 250MB unzipped and ~10s execution — this stack would be fighting
both limits from day one. Chose **Vercel for the frontend, Render or
Fly.io (a normal long-running Python process) for the backend** instead.
This is exactly why `src/agent` and `src/db` never import FastAPI (a
decision made explicit as far back as Slice 1's README) — the entire
hosting decision only ever touched `src/api/main.py` and new files
(`Dockerfile`, `render.yaml`), never the agent logic itself.

### Streaming: per-node progress, not token-level

`stream_agent()` (src/agent/graph.py) uses LangGraph's
`astream(stream_mode="updates")` to yield after each node completes,
mapped to a human-readable message ("Classifying your question...",
"Running query..."). Considered token-level streaming of the final answer
instead (more common for chat UIs) and deliberately didn't: this agent's
useful latency isn't in generating the final sentence character-by-character,
it's in the multi-step pipeline before that (routing, retrieval, tool
calls, retries) — a token-level stream would sit silent through most of
that and then dump the whole answer at once anyway. Node-level progress
events match what's actually slow, and match the project's standing
design principle that every step should be a visible, nameable thing
(same reasoning as making the self-correction loop and RAG retrieval real
graph nodes back in Slices 1 and 2) rather than an opaque wait.

Implementation note: `stream_agent` manually accumulates node updates into
a plain dict rather than replicating LangGraph's internal state-merging
logic — works correctly here specifically because every node in this graph
returns only *new* values (the messages list) or the *current* full value
(everything else), never something needing a custom merge beyond
last-write-wins. Verified directly (not just assumed) before relying on it:
ran `stream_agent()` standalone and confirmed the final answer text matched
`run_agent()`'s answer for the same question.

### Semantic cache: calibrated, not guessed

`src/agent/cache.py` reuses the Slice 2 embedding provider seam — cosine
similarity between question embeddings, in-memory. Picked an initial
similarity threshold of 0.96 without measuring anything, then actually
measured before shipping it: a clear paraphrase of a cached question
("most owners" vs. "highest number of owners") scored 0.957 — just under
0.96, meaning the "obvious" threshold would have missed the single most
common real-world cache scenario (someone rephrasing the same question).
A related-but-different question ("most players" vs. "most owners") scored
0.834, and an unrelated question scored 0.481. Lowered the threshold to
0.93 — clears the paraphrase, stays well clear of the related-but-different
case. Same discipline as Slice 2's RAG retrieval-quality testing: don't
ship a similarity threshold without checking what it actually does on a
real example.

### Rate limiting and graceful errors: both tested, not just written

Per-IP rate limiting (`src/api/rate_limit.py`, in-memory sliding window)
was verified with FastAPI's `TestClient` and the agent call mocked out
(so the test runs in under a second instead of waiting on 10+ real LLM
calls) — confirmed exactly 10 requests succeed, the 11th onward get 429.
Graceful error handling was verified the same way: patched `run_agent` to
raise, confirmed `DEBUG=false` (the default) returns the generic "high
demand" message while `DEBUG=true` returns the real exception — both
paths still log the full traceback server-side either way, so nothing is
lost for debugging, only what reaches the client changes.

### The frontend, and a live bug this testing actually caught

Built with Next.js 16 / React 19 (App Router, TypeScript, Tailwind v4).
`lib/api.ts` hand-parses the `/ask/stream` SSE wire format from a plain
`fetch()` `ReadableStream` rather than using the browser's `EventSource` —
`EventSource` only supports GET, and the question has to go in a POST
body. `components/Chart.tsx` follows the project's dataviz skill: single
hue (every question produces at most one series, so no legend needed —
the heading names it), colors from the validated default palette's
series-1 slot as CSS custom properties (light/dark both defined), thin
bars with rounded top corners, recessive gridlines.

Tested by actually launching the app and driving it with Playwright
(`chromium.launch()`, no `chromium-cli` available in this environment) —
not just `npm run build` succeeding. This caught a real bug: the first
click through the UI returned the graceful "high demand" message instead
of a real answer. Traced it to the backend log rather than assuming the
frontend was broken: another **Ollama CUDA crash**, the same transient
GPU/driver issue from Slice 3, this time hitting the `router` node.
Restarted Ollama (same fix as before) and re-tested successfully — and
this incidentally validated the graceful-error feature working exactly as
designed under a real failure, not a synthetic one: friendly message to
the browser, full traceback in the server log.

### A rendering mystery, chased properly instead of guessed at

The rendered example-question buttons showed what looked like rainbow/
chromatic-fringed text in every screenshot. Didn't assume and didn't
guess-fix:
1. First hypothesis: missing an explicit text-color class (the buttons
   had no `text-*` utility). Added one. **No change** — which is itself
   informative, not a wasted step.
2. Checked `getComputedStyle` directly rather than trust the screenshot:
   `color: lab(27.036 0 0)` — a single, flat, uniform gray. Proves the
   CSS was already correct; whatever's happening isn't the `color`
   property.
3. Suspected Playwright's default headless-only "Chrome Headless Shell"
   browser (a stripped-down variant, not full Chromium) might not fully
   support the modern `lab()`/`oklch()` color functions Tailwind v4
   generates by default. Tested against the system's real installed
   Chrome via `channel: 'chrome'` — **same artifact**, ruling that out.
4. Tested fully non-headless (a real, visible browser window, not
   headless emulation) — **still the same artifact**, ruling out
   anything headless-specific at all.

Conclusion: this is subpixel/ClearType antialiasing on small text — a
real, well-known phenomenon where a screenshot captured and viewed
pixel-for-pixel shows the individual R/G/B subpixel fringes that a
physical LCD panel and the human eye's optical blending are designed to
merge into smooth gray. It's not something a real user looking at a real
screen would perceive as "rainbow text" — it only shows up because a
screenshot is a literal pixel capture, not an optically-blended view.
Confirmed this is a screenshot-viewing artifact, not a product bug, only
after four independent checks each ruling out a different cause — worth
recording the process, not just the conclusion, since "it looked like a
CSS bug and turned out not to be one" is exactly the kind of thing worth
being able to explain rather than silently having fixed (or worse,
silently having ignored a real bug because it "probably wasn't important").

### Deployment config: written carefully, not verified live

`Dockerfile` runs ingestion **at build time** — a deliberate trade-off for
a mostly-static demo dataset (bakes a SteamSpy snapshot into the image;
re-deploy to refresh) rather than needing a persistent volume or an
external data store for what's currently ~200 rows. `render.yaml` mirrors
`.env.example`'s settings as a Render Blueprint. Neither is verified
against a live build or a live Render/Fly account — no Docker available in
this environment, and per the earlier scoping conversation, the user is
deploying manually. Said so plainly in the README's Deploying section
rather than implying these are tested when they aren't.

### Open questions (new)

- **The Dockerfile is unverified.** First real test is the user's actual
  deploy. If `libgomp1` isn't actually what onnxruntime needs on the
  platform's base image, or if build-time network access to steamspy.com
  is restricted on some platform, the ingestion `RUN` step would need
  adjusting — flagged as the most likely failure point, not silently
  assumed to be fine.
- **No scheduled data refresh.** The deployed backend's catalog is frozen
  at whatever the image build saw. Revisit once Slice 7 builds real
  scheduling infrastructure (GitHub Actions cron) for the player-count
  poller — reusing that mechanism for a periodic SteamSpy re-ingest is a
  natural, cheap extension at that point, not before.
- **CORS is a manual two-way env var handoff** (backend needs the
  frontend's URL, frontend needs the backend's URL, and the frontend's
  URL isn't known until after the frontend is deployed). Fine for a
  single deployment; would want automating if this had a staging
  environment too.

---

## Slice 7 — Live player-count time series

**Date:** 2026-08-24

### Two tables, two different freshness contracts

`player_counts` isn't just "another table" — it has the opposite data
character from `games`. SteamSpy's catalog reflects *current* state: every
re-fetch overwrites what we knew, and there's zero value in an old
snapshot (that's why `games` ingestion has always been UPSERT). Steam's
live player-count endpoint has no history API at all — every poll captures
a moment that can never be recovered later. That difference drove every
design decision this slice: `games` is rebuilt fresh each run;
`player_counts` only ever accumulates, and the raw polls themselves
(not just the derived table) have to be preserved *somewhere durable*,
because they're the only copy of that history that will ever exist.

### Why polling and table-building are two separate scripts

GitHub Actions runners are ephemeral — nothing written to a local file
during one scheduled run survives to the next. So persistence has to
happen through something that *does* persist: git. `poll_player_counts.py`
does exactly one thing — fetch live counts, write a timestamped JSON
snapshot, done. A separate script, `build_player_counts_table.py`, rebuilds
the actual DuckDB table by replaying *every* committed snapshot
(idempotent — `ON CONFLICT (appid, polled_at) DO NOTHING`). This is the
same collect-raw-then-build split already established for SteamSpy in
Slice 1 (`steamspy_client.py` caches to disk, `ingest.py` builds the table
from it), applied for a different reason: there, the cache is a
performance/idempotency shortcut; here, the raw snapshots *are* the
durable data — the DuckDB table is the disposable, always-regenerable
artifact, same as `games` always was.

### Verified the "no API key" assumption before building around it

Steam's docs are inconsistent about whether `GetNumberOfCurrentPlayers`
needs a key. Curled it directly first: `?appid=730` returns real data with
no key, `?appid=999999999` returns `{"result": 42}` (no data, but still
HTTP 200). Built the client around that — then hit a *third* response
shape in the actual 200-game poll run that the two-appid spot check
didn't surface: a real, currently-catalogued appid returned a bare
**404**, not the `result != 1` pattern. `resp.raise_for_status()` crashed
the entire batch on that one appid. Root cause, best guess: SteamSpy's
index and Steam's live store don't perfectly agree on what still exists —
a game can be delisted from Steam while still sitting in SteamSpy's
catalog. Fixed by treating a 404 the same as `result != 1` (skip this game,
keep going) while still letting real connection failures (timeouts, 5xx)
propagate and fail the run loudly — the same "some failures should skip,
some should stop everything" judgment call as `sql_tool.py`'s error
handling, just for a different kind of error. 199/200 games polled
successfully after the fix; the one skip is correct, not a bug.

### The payoff: zero agent code changed

Added `player_counts` to `ALLOWLISTED_TABLES`, added its RAG corpus
chunks, and — nothing else. Asked the agent *"What is the current live
player count for Counter-Strike: Global Offensive?"* and it correctly
wrote the join and answered right, with no change to graph.py, no new
tool, no new route. This is exactly the payoff `schema.py`'s very first
comment (Slice 1) promised: "this file is named schema.py and not
games_table.py — it's meant to grow." Three slices later, growing it
really was this cheap.

### A real regression from adding a second table — caught and quantified, not just noticed

Re-ran the full eval suite after adding `player_counts` (same discipline
as every slice: don't just eyeball a few manual tests). A previously
*passing* question — the CCU outliers question — now failed:

```
before Slice 7:  deterministic 5/6, avg judge 4.3/5
after Slice 7:   deterministic 4/6, avg judge 3.7/5
```

Root cause, confirmed by checking retrieval directly rather than guessing:
`games.peak_ccu` ("peak concurrent players yesterday") and the new
`player_counts.player_count` ("live concurrent players right now") both
plausibly answer "concurrent players," and both got retrieved — this
wasn't a retrieval gap, `peak_ccu` was right there in the top-8. The model
picked `player_counts.player_count` but the first attempt wrote
`SELECT name, player_count FROM player_counts` — forgetting `player_counts`
has no `name` column, exactly the mistake the join note exists to prevent.
Added one more targeted RAG chunk explicitly disambiguating the two
columns and when to use which
(`metric:peak_ccu_vs_player_counts`).

**Partial fix, and said so honestly:** on retry, the model *did* write the
correct join this time — proof the disambiguation note worked for what it
targeted. But instead of emitting that corrected query as a real
structured tool call, it leaked the corrected query as prose with an
embedded pseudo-JSON blob ("It seems like the `name` column is actually
located in the `games` table... {"name": "run_stats", ...}") — a
degraded-tool-calling failure, not a schema-confusion one. Traced the full
message history (not just the final answer) to confirm this precisely:
message 2 was a real malformed tool call (schema confusion, the bug the
note targeted); message 4, after seeing the DuckDB error, contained the
*correct* query but as narrated text, not a function call. Two different
failure modes stacked in one run.

Did not chase the second one further. It reproduced deterministically
(temperature=0, identical output both times), which rules out random
sampling noise, but it's the same category of small-model tool-calling
unreliability documented since Slice 3 — a local 8B model narrating what
it would do instead of doing it, under multi-turn retry pressure. Worth
noting what *did* hold up despite the degraded output: the graph's
termination guarantee — no infinite loop, no crash, the retry-cap design
from Slice 1 worked exactly as intended even when the model's behavior
was unexpected. The failure is in answer quality, not system safety.

### Open questions (new)

- **`data/player_counts_raw/` now holds one real snapshot** from this
  session's local testing (199 games, genuinely polled from the live
  Steam API, not synthetic) — kept it rather than deleting it, since it's
  real collected data and gives a working demo a first data point before
  the scheduled workflow has ever run.
- **The peak_ccu vs. player_count disambiguation is unverified at scale.**
  Confirmed it changes behavior (the retry attempt got the join right),
  but the eval suite's `analysis_ccu_outliers` question still fails
  end-to-end because of the separate tool-calling issue — can't yet tell
  from one example whether the disambiguation note reliably prevents the
  *schema* confusion on a fresh (non-retry) first attempt. Another
  concrete case for running the eval suite N times once that's built out.
- **The two GitHub Actions workflows are unverified against a live
  GitHub repo** — no way to test a scheduled workflow, git push
  permissions, or a real deploy hook from this local environment. First
  real test is after the user pushes and enables Actions.
- **No cron-frequency tuning.** 6-hourly polling and weekly catalog
  refresh are reasonable starting guesses, not measured — worth revisiting
  once there's enough `player_counts` history to see whether 6h resolution
  actually shows interesting patterns (daily cycles, event spikes) or is
  needlessly frequent.

---

## Interlude — requirements.txt → pyproject.toml + uv

**Date:** 2026-08-25

Not a slice, a packaging modernization requested mid-session once `uv`
was confirmed available. Replaced `requirements.txt` +
`requirements-ingestion.txt` with one `pyproject.toml` (+ committed
`uv.lock`), using `[project.optional-dependencies]` to formalize a split
that already existed informally: a lean base (everything
`src/ingestion/`, `src/db/`, `src/config.py` need — no LLM/RAG stack) and
an `agent` extra (FastAPI, LangGraph, LangChain, fastembed, scipy — what
`src/agent/`, `src/tools/`, `src/api/` need on top). GitHub Actions' poller
job now runs `uv sync` (base only); local dev and the Docker build run
`uv sync --extra agent`. Same intent as the two-file split, expressed as
one manifest that can't drift out of sync with itself the way two
hand-maintained files could.

`pyproject.toml` deliberately has no `[build-system]` table —
`[tool.uv] package = false` tells uv this is an application (run via
`python -m src.x.y`, imported as `from src.x import y`), not something
meant to be built into a distributable wheel. Worth being explicit about,
since the default assumption for a `pyproject.toml` is a real package.

**Third occurrence of the same Avast TLS problem, different tool this
time:** `uv lock` failed immediately with `invalid peer certificate:
UnknownIssuer` — the identical root cause from Slices 1 and 2
(`requests` and `fastembed`'s `httpx`/`huggingface_hub` calls), now
hitting uv's own Rust TLS stack, which `truststore` (a Python-only fix)
can't touch. uv has its own equivalent: `--native-tls` uses the OS trust
store instead of uv's bundled one. Set permanently via
`[tool.uv] native-tls = true` in `pyproject.toml` rather than requiring
the flag on every invocation — same "fix it once, centrally" instinct as
centralizing `truststore.inject_into_ssl()` in `src/config.py` back in
Slice 2. Three tools, three different TLS stacks, same underlying cause
each time, same "find the tool's own OS-trust-store escape hatch rather
than disabling verification" response each time.

**Verified, not just written:** ran `uv lock` (resolved 71 packages) and
`uv sync --extra agent` against the existing `.venv` (already `pip`-managed
from earlier slices) and confirmed uv reconciled it correctly — Checked 69
packages, no reinstall needed, since the dependency set matches what was
already there. Re-imported the app afterward to confirm nothing broke.
Also updated the Dockerfile to install `uv` as a static binary (copied
from its official distroless image) and use `uv sync --extra agent
--frozen` instead of `pip install -r requirements.txt` — faster and,
via `uv.lock`, pinned to exact resolved versions rather than whatever
`>=` ranges happen to resolve to on build day.

### Open questions (new)

- **The Dockerfile's uv-based build is unverified**, same caveat as the
  rest of the Dockerfile since Slice 6 — no Docker available in this
  environment to actually run the build.
- **`native-tls = true` is a machine-specific workaround being committed
  as a permanent project setting.** Harmless on machines without TLS
  interception (it just uses the OS store instead of uv's bundled one),
  but worth knowing it's there if uv behavior ever seems to silently
  trust something unexpected — it's a deliberate, documented choice, not
  a default.

---

## Interlude — real Groq key, and the standing question finally answered

**Date:** 2026-08-25

`llama-3.3-70b-versatile` (the model picked back in Slice 1) no longer
exists on Groq — `groq.NotFoundError: model_not_found`. Queried
`/openai/v1/models` directly rather than guess a replacement: Groq's
catalog now leans on OpenAI's open-weight GPT-OSS models
(`openai/gpt-oss-120b`, `-20b`) plus Groq's own `groq/compound` system and
a few others. Picked `openai/gpt-oss-120b` over `groq/compound`
deliberately — `compound` is itself an agentic system with built-in tools
(web search, code execution), and layering our own `bind_tools` orchestration
on top of a model that already has opinions about tool use seemed like
exactly the kind of hidden-behavior risk this project has avoided
everywhere else (no prebuilt agent constructors, no opaque tool-choice
logic). Verified tool-calling and `with_structured_output` both work
correctly against `gpt-oss-120b` before adopting it as the default.

**The eval harness finally got to answer the question it was built for.**
Ran it against real Groq for the first time: route accuracy 6/6, avg judge
score 4.5/5 — the highest of any provider tested. More importantly:
**zero occurrences of the group-mislabeling bug**, across both the direct
question and the eval run. `gpt-oss-120b` wrote genuinely correct
conditional-aggregation SQL (`AVG(CASE WHEN price_usd = 0 THEN price_usd
END)`, properly filtered) on every attempt. This confirms, with real
evidence rather than a hunch, what Slices 1 through 7 kept concluding
without being able to fully prove: the mislabeling bug was a Llama-3.1-8B
reliability ceiling, not an architecture, prompt, or guard problem.

The eval suite still reports 4/6 deterministic — but both "failures" are
eval-design gaps, not model errors, confirmed by re-reading what actually
happened: `analysis_action_vs_f2p_not_mislabeled`'s check specifically
requires a `compare_two_groups` stats_result (i.e. "did it call
run_stats"), and this run answered correctly via plain SQL instead — the
judge independently scored it 5/5 as factually correct. Same for the
outliers question: the judge marked it 2/5 for including PUBG as a second
outlier, but PUBG *is* a legitimate outlier at the z ≥ 2.5 threshold used
elsewhere in this project (confirmed via direct `run_stats` testing in
Slice 4) — the golden question's `reference_facts` just didn't mention
it, so the judge had no way to know. Left the golden questions unchanged
rather than patch them reactively; noted as the next real refinement
rather than declared "fixed" without re-verifying against multiple
providers again.

**A real, unrelated bug found and fixed along the way:** the eval
report crashed with `UnicodeEncodeError` printing `gpt-oss-120b`'s output
— it writes with proper Unicode typography (non-breaking hyphens, narrow
no-break spaces) that Windows' default console encoding (cp1252) can't
represent. Fixed by reconfiguring stdout to UTF-8 with replacement at the
top of `run_evals.py`. Not cosmetic — this would crash the harness on any
Windows machine the moment a model's output contained such characters,
which is exactly what happened.

### Open questions (new)

- **Golden-question checks are calibrated toward the failure mode a
  weaker model produced**, not the range of valid correct answers a
  stronger model can produce. `analysis_action_vs_f2p_not_mislabeled`
  should probably accept *either* a correctly-computed plain-SQL answer
  *or* a correct `run_stats` result — not require the tool call
  specifically. Worth revisiting once there's a reason to run evals across
  multiple providers routinely (Slice 5 already flagged this general
  shape of gap).
- **`groq/compound` untested.** Deliberately avoided it for the reason
  above, but never actually confirmed whether its built-in tool use would
  conflict with or complement this project's own `bind_tools` design —
  an assumption, not a measured result.

---

## Slice 8 — MCP server

**Date:** 2026-08-25

### Scope: dropped Gemini, sequenced the rest

User explicitly ruled out Gemini as a fallback provider (free-tier keys
expire too fast to be a reliable fallback) — removed it from the roadmap
rather than leaving it as a silently-stale TODO; the seam in
`llm_provider.py` stays (costs nothing to leave a branch that raises
`NotImplementedError`). The remaining Slice 8 items (MCP server, Expo
mobile) got split into their own slices with a proposed order: MCP first
(cheapest — mostly reusing code that already exists), frontend visual
polish next (highest-visibility ROI for a portfolio piece, and settling
on a look here makes the mobile UI faster later), Expo last (most new
surface area, and — worth flagging since cost keeps coming up — building/
testing it is free via Expo Go, but *publishing* to app stores isn't:
$99/yr Apple, $25 one-time Google. Recommended skipping store publishing
by default.)

### The SDK's API had moved since my training data — checked, didn't guess

Went to write the server using `from mcp.server.fastmcp import FastMCP`,
the pattern I expected from prior knowledge of the SDK. It doesn't exist
in the installed version (`mcp==2.1.0`): `ModuleNotFoundError`. Rather
than guess at a replacement, introspected the installed package directly
(`pkgutil.walk_packages`, then `inspect.signature` on candidates) and
found the successor: `MCPServer` in `mcp.server` — same decorator-based
API (`.tool()`, `.resource()`, `.run()`), just renamed and moved during
what looks like a significant SDK restructuring (also added
auth/OAuth support, an `apps` module, elicitation, subscriptions — a much
bigger surface than the version I remembered). Worth calling out as a
general lesson, not just an MCP-specific one: for a fast-moving SDK,
`inspect.signature()` against the actual installed version is more
reliable than remembered API shape, and took under a minute here.

### Reuse, not reimplementation — and proved it, not just claimed it

`src/mcp_server/server.py`'s `run_sql`/`run_stats` tools call
`execute_run_sql`/`execute_run_stats` directly — the identical functions
`src/agent/graph.py`'s `execute_tools` node calls. The safety guard
(`validate_select_only` in `src/db/connection.py`) is reached the same way
regardless of caller; there's no second implementation to keep in sync or
accidentally leave less-guarded. Didn't just assert this — verified it
with a real MCP client session (`ClientSession` over `stdio_client`,
spawning the actual server subprocess, not a mock): listed tools and
resources, read `schema://games`, ran a real query, and specifically sent
`DROP TABLE games` through the MCP `run_sql` tool and confirmed it comes
back with the same rejection message (`"Only SELECT statements are
allowed, got a Drop statement."`) an agent-driven query would get. Same
guard, same code path, proven rather than assumed.

### The schema resource reuses the RAG corpus, not a separate description

`schema://games` (an MCP *resource*, not a tool — read-only reference
data a client fetches once, not an action) calls
`assemble_schema_text(SCHEMA_CHUNKS)` from Slice 2's RAG module directly.
Considered writing a separate, MCP-specific schema description and
rejected it immediately: two hand-maintained descriptions of the same
schema is exactly the kind of drift risk this project has avoided
everywhere else (the RAG corpus itself exists specifically so the schema
is described in one place). The MCP resource gets the full, unfiltered
corpus (no retrieval/ranking — there's no "question" to rank against for
a static resource a client reads once up front), which is a fine, simple
default at this corpus's current size (~24 chunks).

### Why local stdio, not a hosted MCP server

Chose stdio transport exclusively for now, not `sse`/`streamable-http`
(both of which `MCPServer.run()` also supports). Stdio means: the client
(Claude Desktop, Claude Code) spawns the server as a local subprocess for
the duration of its own session — no hosting, no network exposure, no
cost, and no separate deployment decision to make. A hosted, remotely
reachable MCP server is a real possible future step (could piggyback on
the already-deployed backend, per the earlier deployment discussion), but
would need its own auth story (the SDK's new `auth`/OAuth module exists
for exactly this) — deliberately out of scope until there's an actual
reason to want the server reachable from somewhere other than the calling
app's own machine.

### Dependency placement: pragmatic, not maximally lean

Added `mcp[cli]` to the existing `agent` extra rather than a new,
narrower one. Considered a dedicated `mcp` extra (skip FastAPI/LangGraph/
LangSmith, which the MCP server genuinely doesn't use) and decided
against it: `src/mcp_server/server.py` already imports `src.tools.sql_tool`
(which imports `langchain_core` at module level for its own `@tool`-
decorated object, even though the MCP server never uses that object) and
`src.agent.rag.schema_index` (which imports the embedding provider,
pulling in `fastembed`, even though the MCP resource never calls it). A
truly lean MCP-only extra would need refactoring those modules to make
their heavier imports lazy — real, legitimate cleanup, but out of
proportion to what this slice needed, and the MCP server is inherently a
local-dev tool run on a machine that already has the full `agent` extra
installed to run the web app anyway. Noted as a real option, not pursued.

### Open questions (new)

- **No remote/hosted MCP transport.** Stdio-only for now; revisit if
  there's ever a concrete reason to want this reachable from outside the
  calling app's own machine (the SDK already has the auth pieces for it).
- **The lean-extra option for `src/tools/` and `src/agent/rag/` is real
  but unpursued** — `sql_tool.py` and `schema_index.py` both have
  heavier-than-necessary module-level imports for their MCP use case
  specifically. Worth doing if a genuinely lean, MCP-only install ever
  matters (e.g. distributing this as a standalone MCP server package
  independent of the web app).
- **Cursor untested.** Documented Claude Code and Claude Desktop config
  (both directly verifiable from this environment); Cursor's MCP config
  format is very likely close to Claude Desktop's (`mcpServers` JSON) but
  wasn't checked against a real Cursor install.

---

## Slice 9 — Frontend UI/design polish

**Date:** 2026-08-25

Slice 6 built the frontend's plumbing (SSE streaming, chart rendering, dark
mode via Tailwind's `dark:` classes) but deliberately left the visual design
plain — right call at the time, wrong call to leave standing once the rest
of the system had real content to show off. This slice is the visual pass,
requested explicitly with a "gaming AI" identity and real motion.

### One animation library, not four

The request that kicked this off named four animation libraries (Anime.js,
motion.dev, react-spring, Framer Motion) plus a couple of UI-kit references.
Framer Motion *is* motion.dev — Framer Motion was renamed "Motion" and now
ships from the `motion` package, same author, same lineage — so two of the
four names were the same library twice. Running Anime.js, react-spring, and
Motion side by side in one app would mean three animation engines' worth of
bundle weight and three different easing/spring feels for no real gain, so
the app standardizes on Motion (`motion/react`) alone. Same reasoning for
"Kokonut UI": it's a copy-paste component registry (shadcn-style — you
`npx shadcn add` a component's source into your own tree), not an
installable package, so there's nothing to add to `package.json` for it;
its glow-card/gradient-border idiom is instead hand-built directly in
`components/GenreShowcase.tsx`.

### Brand continuity with ARCHITECTURE.md's trace artifact

Rather than invent a new palette and type pairing from scratch, this slice
deliberately reused the one already established for the Slice 8 interactive
agent-trace artifact: the same amber accent (`#f0a63a` dark / `#a8650f`
light), the same Archivo + IBM Plex Mono pairing. The alternative — a
distinct look for the live app versus the docs' interactive artifact — would
have made the project read as two different projects wearing one name. The
same reasoning extended into the product itself: `components/TraceSteps.tsx`
re-renders the streamed progress events as a node-by-node stepper using the
graph's real node names (router → retrieve_schema → agent → execute_tools →
build_chart_spec), the same visual grammar as the ARCHITECTURE.md diagram,
instead of Slice 6's flat scrolling text log. The trace concept isn't just
documentation anymore — it's now a real UI element a user watches live.

### Genre identity: real counts, not invented ones, and why 8 not 12

"Illustrated identity per genre" needed an actual genre list before anything
else. SteamSpy's `genre` field turned out to be free-text and comma-joined
(`"Action, Adventure, RPG"`, not an enum) — queried the real 200-game
catalog directly (`data/db/games.duckdb`), split every row's genre string on
comma, and counted tokens rather than guessing a plausible-looking list.
Real result, most common first: Action 148, Adventure 75, Indie 67, Free To
Play 52, RPG 50, Simulation 41, Massively Multiplayer 38, Strategy 33,
Casual 24, Early Access 13, Sports 13, Racing 7 (plus a handful of one-off
non-game tags like "Photo Editing" — noise from a few mislabeled catalog
entries).

Cut this to 8 for `lib/genres.ts`: "Early Access" (a release status, not a
genre) and "Free To Play" (a pricing model) were excluded on category
grounds, and the dataviz skill's categorical-palette rule is a hard 8-hue
cap regardless — a 9th series never gets a generated hue, it folds into
"Other." That cap happened to land exactly on a clean real cutoff (Casual at
24 vs. the excluded pair's actual genre-adjacent neighbors Sports/Racing at
13/7), so nothing had to be forced.

Colors reuse the dataviz skill's already-validated default 8-slot
categorical palette (`references/palette.md`) rather than deriving a new
one — assigned in the genres' real prevalence order (Action → slot 1 blue,
… Casual → slot 8 red), which preserves the CVD-safety guarantee that was
validated for this exact hue *sequence* (re-ordering which hue means which
genre is fine; what the validator actually checked was adjacency within
this sequence). Re-ran `validate_palette.js` against this app's own light/
dark chrome surfaces (`#faf8f4`/`#141310`, not the skill's reference
surfaces) rather than assuming the original validation still holds on a
different ground — both pass every check; light mode WARNs on raw contrast
for 3 of the 8 hues (aqua/yellow/magenta), which the skill flags as needing
a relief channel — satisfied here since every genre card always carries a
matching icon and text label, so identity is never color-alone regardless.
One caveat logged honestly: the categorical validator's "adjacent pairs"
model is a linear sequence (bars/lines), and this palette is displayed in a
2D grid, where a card has up to 4 visual neighbors, not 2. Didn't re-derive
for grid-adjacency specifically — the icon+label pairing already makes
color non-load-bearing, which covers the gap.

Eight hand-authored line-art SVG glyphs (`components/GenreIcon.tsx`) — a
crosshair, compass, pixel-heart, gem, gear-spokes, network nodes, a 3×3
grid, a smiley — rather than pulling in an icon library for 8 shapes.

Genre cards aren't decorative: clicking one calls the same `ask()` path as
the Slice 6 example-question chips, with a genre-specific real question
(`lib/genres.ts`'s `question` field) that the existing agent can already
answer with zero backend changes — same reasoning Slice 7 leaned on for
player-count questions ("meant to grow" schema, no new code needed).

### Accessibility: `MotionConfig reducedMotion="user"`, once

Rather than checking `prefers-reduced-motion` in every animated component,
`components/MotionProvider.tsx` wraps the whole app in a single
`<MotionConfig reducedMotion="user">`. Motion's own reduced-motion mode
keeps opacity/color transitions but makes positional animation (the hero's
stagger-in, card lift, trace-dot scale) instant — one line, whole-app
coverage.

### Verification, and a real bug found in the process

Built the whole slice, then drove it in an actual browser (Playwright) —
not just visual screenshots of static markup, but a real interaction: click
a genre card, watch the trace stepper animate through real streamed
progress events, wait for either a result or a failure. Checked light and
dark themes, and a hover state (the card's radial glow).

That last check surfaced a real, pre-existing **backend** bug, unrelated to
this slice's own code: `POST /ask/stream` crashes the whole Python process
—not a Python exception with a traceback, a hard native fault
(`OPENSSL_Uplink(...): no OPENSSL_Applink` printed to stderr, then the
process exits) — reproducibly, on the first request that does real work.
`/health` (no outbound network call) works fine every time; the crash
happens exactly when the agent would make its first real HTTPS call (Groq).
Working hypothesis, not yet confirmed: `truststore.inject_into_ssl()` (added
back in the pyproject.toml/uv migration, to route around Avast's TLS
interception via the OS trust store) patches Python's global `ssl` module
in a way that collides with another native extension's own statically
linked OpenSSL the first time a real TLS handshake actually happens — the
same family of Windows TLS issue this project has already hit twice before
(`uv sync`/`uv lock`'s `UnknownIssuer` errors), just triggered from inside
the running app instead of from `uv`. Not chased further here: it's a
backend runtime issue, out of scope for a frontend-visual-polish slice, and
guessing at a fix without reproducing it outside this one environment risked
breaking the ingestion/eval paths that already depend on `truststore`/
`native-tls`. What *was* verified: the frontend's own handling of a dropped
connection is correct — `lib/api.ts`'s `fetch` rejects, `page.tsx`'s catch
block sets the existing "Couldn't reach the backend" error state, no crash,
no stuck loading spinner.

### Open questions (new)

- **The `/ask/stream` native crash needs real investigation** — likely
  `truststore` vs. some other native extension's bundled OpenSSL on
  Windows, per the hypothesis above, but unconfirmed. Next step if
  picked up: reproduce with `MODEL_PROVIDER=ollama` (no outbound HTTPS at
  all) to test whether the crash is specifically tied to the first real
  TLS handshake, then bisect which native extension collides with
  `truststore`'s patched `ssl` module.
- **Grid-adjacency CVD validation for the genre palette** — validated as a
  linear sequence per the dataviz skill's method; a true 2D-grid adjacency
  check would need extending the validator's pairlist logic, not attempted
  here since the icon+label pairing already covers the accessibility gap.

---

## Slice 9b — Retro redesign, real forecasting, dynamic genre stats

**Date:** 2026-08-25

The user tried Slice 9's frontend, and it worked (a real Groq round trip
rendered correctly, screenshots included), but the reaction was blunt: it
"feels like an AI slop website." Fair — re-reading it, the first pass had
drifted into exactly the look the artifact-design skill's cliché list warns
about: soft rounded-xl everything, muted warm neutrals, safe spacing. Good
engineering, forgettable design. Three follow-on requests arrived together:
make it genuinely distinctive ("retro"), build the forecast route for real,
and stop hardcoding the genre-showcase counts.

### Picking a retro lane, not "retro" in general

"Retro" spans unrelated worlds — asked the user to choose among three
concrete directions (80s arcade/neon, 90s terminal/phosphor, Y2K
retro-futurism) rather than guess, since a full visual identity is expensive
to redo twice. Landed on 80s arcade/neon. That decision drove real,
falsifiable choices, not just a vibe:
- Sharp corners + thick borders everywhere a panel/console appears (cabinet
  bezel), contrasted with round pill buttons for actual clickable controls
  (cabinet joystick buttons) — the contrast is the point, not an oversight.
- Four typefaces, each doing exactly one job, is a genuine departure from
  the "one display + one body" pairing structure `artifact-design` normally
  recommends: Archivo (body/UI, unchanged), IBM Plex Mono (data/code,
  unchanged), **Monoton** (the hero headline, *only* the hero headline — a
  neon-tube marquee face, illegible at paragraph length by design, which is
  exactly why it's confined to four words), **Press Start 2P** (pixel
  labels — section eyebrows, route/cached badges, the trace stepper's node
  labels; same illegibility-at-length constraint, same confinement).
- The marquee chase-light animated border (`.marquee-border` in
  globals.css — a rotating `conic-gradient` masked to a ring via
  `@property --marquee-angle`) went on exactly one element, the ask console.
  Considered putting it on the result panel too and didn't — one orchestrated
  motion moment reads as a choice; two competing ones read as decoration.
- Grounds got realigned to the *exact* hex values ARCHITECTURE.md's
  agent-trace artifact already used (`#0d1014` dark / `#f6f4f0` light) rather
  than the close-but-not-identical approximation Slice 9 shipped
  (`#141310`/`#faf8f4`) — the user's second screenshot in this conversation
  was literally that artifact's graph, used as the reference for "this is
  the level of polish I want," so tightening brand continuity to match it
  exactly was the correct read of that signal, not a coincidence to ignore.

### Forecast: honesty moved from the router into the tool

Slice 3/4 explicitly deferred forecasting with a hardcoded router→
terminal-node "not supported yet." That's gone now: `forecast` flows through
the identical retrieve_schema → agent → execute_tools loop as lookup/
analysis (see ARCHITECTURE.md's updated graph), with `run_forecast`
(`src/tools/forecast_tool.py`) bound alongside `run_sql`. The interesting
design decision wasn't the linear regression (scipy `linregress`, nothing
exotic) — it was where the "do we actually have enough data" honesty check
lives. It does NOT live in the router (classify forecast questions as
forecast regardless of whether data exists — the router's job is
understanding the question, not knowing the DB's current state) and does
NOT live in a route-level gate (a fixed "not supported" block would make the
feature permanently disabled even once real history accumulates). It lives
*inside the tool*: `execute_run_forecast` checks the actual distinct
snapshot count for whatever query the LLM wrote, and returns a structured
`insufficient_history` result instead of fitting a line through one point.
This means the feature **self-upgrades**: the moment the Slice 7 poller
lands a second real snapshot for some game, that game's forecast questions
start returning real projections, with zero code changes — verified this is
architecturally true by unit-testing `_forecast()` directly against
constructed multi-point data (a real 6-point rising series, both a
near-horizon and a deliberately-absurd 365-day-out horizon), not just
against the real (currently single-snapshot) DB.

Also added a `low_confidence` flag, separate from `insufficient_history`:
even with ≥2 points, projecting further ahead than the *observed* span is
extrapolation past what the data can support, and <5 points is thin enough
that a line captures noise as easily as trend. Both conditions are checked
independently and their reasons are returned as a list, not a boolean —
the system prompt (`FORECAST_TOOL_GUIDANCE` in prompts.py) instructs the
model to always surface these reasons in prose rather than stating a
number with false confidence. Verified end-to-end with a real Groq call
against "How many players will Counter-Strike have next year?" (the exact
phrase the router/prompts docstrings use as their example, and one of the 4
default example questions in the frontend) — the agent correctly reported
insufficient history AND, on its own initiative, fell back to a real
`run_sql` query for CS:GO's `peak_ccu` to give a useful answer instead of a
bare refusal. That fallback behavior wasn't prompted for explicitly; it fell
out of binding `[run_sql, run_forecast]` together for the forecast route
(the same "give it more than one tool and let it decide" pattern
`analysis`'s `[run_sql, run_stats]` already established in Slice 4).

### Dynamic genre counts: what the user actually flagged

Mid-session interjection: "I dont want just 200 data timestamp, I want the
retrieval of the data to be dynamic." Read literally this is about
`player_counts`, but in context (right after seeing the genre showcase) it
was about `lib/genres.ts` — Slice 9 baked the 8 genre labels/counts in as a
TypeScript literal, computed once by hand against a local export. That's
exactly the kind of frozen snapshot the project's `refresh_catalog.yml`
(weekly re-ingestion) would silently invalidate. Fixed by adding
`GET /genres` (`src/db/genre_stats.py`) — the same split-comma-and-count
logic, now running at request time against the live `games` table — and
having `GenreShowcase.tsx` fetch it on mount instead of importing a
constant. Went further than just re-fetching the *counts*: which 8 labels
even make the top-8 is now live too, since the real top-8 could shift as
the catalog grows. That meant `lib/genres.ts` had to change shape — it's no
longer a list of genres, it's curation metadata (icon id, a nicer example
question) keyed by label, with a generic fallback (`GenreIcon`'s new
`Generic` glyph, a templated "What are the 5 highest-rated {label} games?"
question) for any label outside the curated set. Deliberately did NOT try
to curate icons for every SteamSpy tag the catalog could ever produce —
Sports and Racing got real hand-drawn icons since they're plausible top-8
contenders; the long tail (the occasional mislabeled non-game entry
producing a tag like "Photo Editing") gets the generic fallback and that's
fine, it's meant to.

This turned `genre_stats.py` into the first read path in the whole system
that touches the DB *without* going through `connection.py`'s guard —
worth being explicit about why that's correct, not an oversight: the guard
exists to constrain **LLM-generated** SQL. `genre_stats.py` runs one fixed,
hand-written query with no model or user input anywhere in it. Routing it
through `validate_select_only` would add a dependency for zero safety
benefit — there's nothing adversarial to guard against here.

### A real bug, found by accident, that had nothing to do with any of this

Tried to verify the forecast feature live and hit the exact `/ask/stream`
crash flagged (but not investigated) at the end of Slice 9 — `OPENSSL_Uplink
(...): no OPENSSL_Applink`, killing the whole Python process, no traceback.
Slice 9's DOCEXP entry guessed this was `truststore`'s `ssl` patch colliding
with some other native extension's bundled OpenSSL. **That hypothesis was
wrong**, and worth recording as wrong rather than quietly dropped, per this
file's own habit of correcting itself in the open (see the Slice 4 group-
mislabeling entry).

Isolated it properly this time, one layer at a time, in the actual venv:
1. A **plain `requests.get()`** to a real HTTPS endpoint, zero project code
   imported — crashed identically. Ruled out truststore, ruled out Groq's
   SDK, ruled out LangChain/LangGraph entirely; this was never an agent-code
   bug.
2. **Raw stdlib `ssl.wrap_socket()`**, no `requests`/`urllib3` either —
   crashed identically. Ruled out the requests/urllib3 layer too. This is a
   fault in the Python installation's own TLS stack, full stop.
3. The **already-installed Microsoft Store Python 3.12** on the same
   machine — same real network, same Avast, same everything else — did a
   real TLS handshake to the same host with zero issue. This is what ruled
   out Avast as the cause of *this specific crash* (Avast's TLS interception
   is real and is exactly why `truststore` exists at all — but it produces a
   catchable `CERTIFICATE_VERIFY_FAILED`, not a process-killing native
   fault, and this system Python hit neither).
4. A **uv-managed Python 3.12.13** (`uv run --python 3.12`) — also clean.
   At this point the only variable left was the interpreter build itself:
   the project's `.venv` was on uv's default pick, **3.14.2**, uv's newest
   available standalone Windows build at the time. Python 3.14 had only
   just been released; a rough edge in a brand-new platform-specific
   standalone build was, by this point, the only hypothesis still standing.

Fixed by pinning `.python-version` to `3.12` and rebuilding `.venv` (`rm -rf
.venv && uv sync --extra agent`) — `requires-python = ">=3.12"` already
permitted this, so no dependency constraint needed to change. Re-ran the
same plain-`requests` probe on the rebuilt venv: it now failed with the
*expected*, *already-understood*, *already-solved* Avast error
(`CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate`) — and
then succeeded cleanly once `src.config` (which calls
`truststore.inject_into_ssl()` at import time) was imported first. That's
the tell that this really was two separate, unrelated issues stacked on top
of each other: a genuine `python-build-standalone` 3.14.2 Windows bug (fixed
by not using that build), sitting on top of the long-standing, already-
correctly-handled Avast TLS interception (fixed by `truststore`, same as
every prior occurrence in this project). Confirmed fixed against a real
Groq call afterward — see the forecast section above.

The methodological point worth keeping: the fix came from isolating variables
one at a time against the *real* environment (this venv, this network, this
Avast install) rather than pattern-matching to "we've seen a TLS error
before, it's probably the same cause" — the previous entry's guess did
exactly that pattern-match and was wrong.

### Open questions (new)

- **`.python-version` pins 3.12, not a specific patch version.** uv will
  still pick up new 3.12.x patch releases as they're published. Reasonable
  for now; revisit if a similar issue ever shows up on a specific patch.
- **Should this project's CI/deployment pin Python the same way?** The
  `Dockerfile` doesn't currently specify a Python version explicitly at all
  (relies on whatever `uv sync` resolves inside the container image) — worth
  checking it doesn't silently pick up a fresh, potentially-buggy standalone
  build the same way local dev just did.
- **The result panel's answer text renders literal `**markdown**`
  asterisks** — noticed during this slice's live screenshots (pre-existing,
  not introduced here: `<p>{result.answer}</p>` has never parsed markdown).
  Small, real, not fixed here — out of scope for a redesign+forecast slice.
  Fixed next in Slice 9c.

---

## Slice 9c — Markdown, real genre browsing, Anime.js background

**Date:** 2026-08-25

Three follow-ons from actually using Slice 9b: the markdown bug flagged (but
correctly left unfixed, as out of scope) at the end of that entry, the
genre showcase's click behavior, and a request to use Anime.js specifically
for background motion.

### Markdown: `react-markdown` over hand-rolling it

Considered a small hand-written `**bold**`/`* item` regex parser first —
consistent with this project's usual "no dependency for something this
small" instinct (the SSE parsing in `lib/api.ts` is hand-rolled for exactly
that reason). Went with `react-markdown` instead once the actual failure
mode was reconsidered: the agent's answers are real LLM prose, not a fixed
template, so what shows up (bold, bullet or numbered lists, occasional
inline code) is whatever GPT-OSS decided to write — a hand parser would need
to cover CommonMark's real edge cases (nested lists, escaped characters,
mixed list markers) to not just move the bug rather than fix it, at which
point it's reimplementing a markdown parser, badly, instead of using the
industry-standard one. `components/Markdown.tsx` wraps it with an explicit
`components` map — every element (`p`, `strong`, `ul`/`li`, `code`, `a`) is
styled to this app's own tokens by hand, not a generic `@tailwindcss/
typography` "prose" class, so the rendered markdown looks like it belongs
to this app specifically rather than to a plugin's default theme.

### Genre cards: browsing, not asking

Slice 9 made a genre card fire a curated LLM question. Explicit correction
this round: "when we click on it, we should display the games." This is a
better design independent of being asked for — clicking a card to *browse*
what's actually in that genre is a faster, more expected interaction than
clicking a card to *ask an AI about* that genre, and it doesn't cost a Groq
round trip to satisfy. `GET /games` (`get_games_by_genre` in
`src/db/genre_stats.py`, alongside `get_genre_counts`) is a second
deterministic, no-LLM, no-guard endpoint for the same reason the first one
is: it's one fixed query with a parameterized genre argument, not
LLM-generated SQL, so `connection.py`'s guard has nothing to add. Matches on
the comma-split token (case-insensitive), not a raw `ILIKE '%label%'`
substring — same reasoning `get_genre_counts()` already established: a
substring match on the whole free-text field can't distinguish a real match
from an accidental one where the label happens to be a substring of a
different, unrelated tag.

The LLM-question path didn't disappear — a card's expanded panel has a
secondary "ask the agent about {genre} →" link that calls the same
`onPick` the old click behavior used, so both interactions coexist:
fast/deterministic browsing as the primary action, the agent still one
click away for anyone who wants a synthesized answer instead of a raw list.

One real implementation snag, not a design decision: the new-ish
`react-hooks/set-state-in-effect` ESLint rule flagged calling `setGames(null)`/
`setGamesLoading(true)` synchronously in the effect body that runs the
fetch. Fixed by moving those two resets into the click handler itself
(`toggle()`) — the effect now only sets state from the async fetch's
resolution (`.then`/`.catch`/`.finally`), which is what the rule is
actually asking for: an effect should react to an external system's result,
not synchronously mutate state itself the moment it runs.

### Anime.js: one exception, scoped on purpose

The user asked for it by name again, specifically for background animation,
after Slice 9's consolidation onto Motion alone (partly *because* the
original ask listed Anime.js as one of four overlapping libraries for
general UI animation). This isn't reopening that call: `components/
RetroBackground.tsx` is a synthwave grid-horizon-plus-drifting-motes loop
with **no React state involved at all** — nothing it animates is derived
from or feeds back into a component's props/state. Motion's whole value
proposition (`variants`, `AnimatePresence`, gesture props tied to render)
doesn't apply to a background that never needs to know anything about the
React tree. Anime.js's imperative, timeline-first API — grab a DOM/SVG ref,
animate it, let it run — is a better fit for exactly that job, and using it
there isn't redundant with Motion's job anywhere else in the app. Kept
strictly to this one file; nothing else in the app imports it.

Implementation notes, since Anime.js v4 is a genuinely different API from
the v3 most people remember (`anime({targets, ...})` → named exports
`animate(targets, params)`, `easing` → `ease`, modular sub-path imports)
— introspected the installed package's actual `.d.ts` files rather than
trusting memorized v3 API shape, same discipline as the MCP SDK version
drift in Slice 8. One real design choice inside it: the scrolling grid is
NOT an animated SVG `<pattern>` (the first draft used one) — CSS `transform`
support on `<pattern>` elements is inconsistent across browsers since
patterns aren't normal rendered/laid-out elements. Switched to animating a
plain `<g>` of explicit `<line>` elements instead (universally reliable),
sized with two extra rows past both edges of the visible band so a
one-row-height loop reads as continuous with no visible seam at the wrap.

### Open questions (new)

- **No table/task-list markdown styling** — `react-markdown`'s default
  CommonMark support covers everything seen in real answers so far (bold,
  lists, occasional code); GFM extras (tables, strikethrough) aren't wired
  up (`remark-gfm` isn't installed) since no real answer has needed them
  yet. Add it if/when a stats answer actually wants a table.
- **The games leaderboard always shows exactly what `get_games_by_genre`
  returns (top 8 by review score, then peak CCU)** — no sort/filter control
  in the UI yet. Fine for a browsing entry point; would need one if this
  panel grows into something people spend real time in.

---

## Slice 9d — One committed scene, not a subtle texture

**Date:** 2026-08-25

The reaction to Slice 9c's ambient background: "I don't like the black or
white," with a specific reference image — a magenta grid horizon under a
glowing sun, mountains in silhouette. Not "more retro texture," a
correction: the earlier treatment was a faint decoration on top of a
neutral near-black/near-white page; what was actually wanted is that scene
*as the page itself*.

### Dropping dual-theme for the chrome tokens, on purpose

This is worth being explicit about since it's a real reversal of a
discipline this project followed carefully through Slices 6-9c: brand
chrome (`--background`, `--surface`, `--foreground`, etc.) no longer has a
light/dark split. One synthwave-night palette, always. Two reasons this is
the right call here rather than a corner cut: (1) the reference is
inherently a night scene — a "light mode" synthwave horizon isn't a
lighter version of the same thing, it's a different image, and diluting a
specific, deliberate reference into two lesser variants would serve
neither; (2) the `artifact-design` skill explicitly carves out exactly this
case ("a design that deliberately commits to one visual world... may stay
single-theme"). The genre categorical palette and `.viz-root` chart tokens
both collapsed to their single already-dark-tuned values too, for the same
reason — every surface in the app is dark-glass now regardless of OS
theme, so keeping a light variant of hues tuned for a light surface that no
longer exists would be dead code, not a real fallback.

`--accent` didn't change value (`#ffc857`, close to the prior `#f0a63a`) —
it's still the same warm-gold thread running through the trace artifact and
the rest of the app, and it happens to double as "the sun's color" in the
new scene without any adjustment needed. One brand color, two coincidentally
compatible contexts.

### The scene itself

`components/RetroBackground.tsx` (rewritten, not extended) draws sky
(linear gradient, 5 stops), a sun (radial gradient circle, masked with a
few horizontal black bars near its base for the classic "striped retro
sun" look), two layers of low-poly mountain silhouettes (`ridge()` — a
small deterministic sine-wobble generator, not hand-typed path data,
consistent with the diagramming discipline's "avoid long decorative path
data"), and a neon grid: static converging verticals fanned from a single
vanishing point (the "endless road" half of the perspective) plus the
previously-built animated horizontal scan-lines (recolored magenta,
boosted opacity, Anime.js-driven scroll unchanged in mechanism from Slice
9c). Panels got a `.glass-panel` utility (translucent surface +
`backdrop-filter: blur`) so they read as chrome floating over the scene,
not boxes painted on top of it.

### Cartridges, and a real rendering bug found building them

"Some cartridges to be the choices... like the old Game Boy or DS
cartridges" — genre cards became `components/GenreCartridge.tsx`: a
chamfered-corner silhouette (one shape family for all 8 genres, not
alternating Game Boy/DS shapes — cohesion over literal variety), a
label-sticker window holding the existing `GenreIcon` glyph, and a
connector-notch ridge at the bottom. Hover lifts and slightly rotates the
cartridge (Motion spring, `whileHover`) for a physical "pop out" read.

First draft used `clip-path: polygon(...)` alone on the glass-panel div to
cut the chamfer. Screenshotted it to verify (the discipline that's caught
every other visual bug this project has shipped) and the corner looked
perfectly square — no chamfer visible anywhere, on any of the 8 cartridges.
Root cause, once actually checked (`getComputedStyle` confirmed the
clip-path value WAS applied correctly): `clip-path` genuinely removes the
pixels outside the polygon, but it doesn't paint anything new along the
edge it creates — the `border-2` on that div still only draws along the
*original* rectangular box edges, so where the clip line cuts through,
there's no stroke at all, just a bare edge revealing whatever's behind it.
Since what's behind it (the scene, at the point genre cards sit) is a
similarly dark color to the panel's own near-black glass, the cut was real
but had ~zero visual contrast — geometrically correct, invisible in
practice. Fixed by overlaying a second, stroke-only SVG `<polygon>` (same
points, `fill="none"`) directly on top — that's what actually draws a
visible line along the diagonal, since an SVG stroke traces its shape's
full boundary regardless of what a sibling element's CSS clip did.
General lesson worth keeping: `clip-path` clips content, it does not
imply an outline — anything that needs the cut *edge itself* to be
visible needs something else to draw it.

### Addendum: font swap, and actually centering the sun

Two immediate follow-ons once this was live: the body font still read as a
generic modern grotesque next to Monoton/Press Start 2P/Plex Mono, and the
sun sat too high — mostly hidden behind the hero copy rather than being a
real focal point.

Swapped `--font-display` from Archivo to **Chakra Petch** — full weight
range (so it still works for both body copy and semibold labels, unlike
Audiowide/Orbitron which are single-weight display faces that get
illegible at paragraph size), with a squared-off, slightly-angular
character that actually reads as retro-computer rather than a neutral
grotesque wearing a retro palette.

Centering the sun took two tries. First attempt: changed the SVG's
`preserveAspectRatio` from `xMidYMax slice` (bottom-anchored — where the
horizon/sun end up depends on how much the viewport's aspect ratio
overflows vertically, which is what was pushing the sun up out of view)
to `xMidYMid slice`, and grew the viewBox from `800x600` to `800x900` with
the horizon fixed at the exact vertical middle (y=450) — this makes the
horizon land at the vertical center of *any* viewport predictably, not
wherever the old aspect-ratio math happened to put it. Verified the math
holds by hand for a few real viewport ratios before trusting it (a
1280×1100 window: scale is width-driven at 1.6×, the visible vertical crop
window maps back to a horizon position of exactly 50% — confirmed, not
assumed) rather than just eyeballing one screenshot and hoping it
generalizes to other window sizes. Also extended the ground/grid drawing
to y=1000 (past the 900-tall viewBox) so no real aspect ratio produces a
visible gap under the grid — checked this explicitly against a tall/narrow
900×1400 viewport, the shape most likely to expose one.

That first attempt centered the sun correctly but made it too large and
too bright exactly where the hero paragraph lives — a real legibility
regression, caught by screenshotting before calling it done (the same
discipline that caught the invisible cartridge chamfer earlier in this
slice). Fixed two ways together, not by picking one: shrank the sun
(r 168→132) and biased it down slightly relative to the horizon so its
brightest band sits mostly above the text column instead of behind it, and
gave the hero paragraph a dark text-shadow plus switched it from the muted
lavender tone to full foreground-white — belt-and-suspenders, since exact
sun geometry will never guarantee zero overlap at every viewport size, but
shadowed white text stays readable against anything behind it regardless.

### Open questions (new)

- **No light-mode fallback exists anymore for anyone who'd genuinely prefer
  one** — a deliberate scope choice (see above), not an oversight, but
  worth revisiting if that preference ever comes up for real.
- **Mobile/narrow-viewport rendering of the scene and cartridges wasn't
  separately verified this round** — desktop (1280px) was checked
  thoroughly (hover, selection, the leaderboard panel, no console errors);
  the responsive grid classes themselves are unchanged from Slice 9, so
  behavior should carry over, but that's inference, not a check.

---

## Slice 9e — The markdown bug's real cause, and actually checking mobile

**Date:** 2026-08-25

Two follow-ons, and both are examples of the same lesson: an open question
flagged and left unfixed will eventually be the thing that breaks.

### The markdown bug wasn't a rendering bug

Slice 9c added `react-markdown` and it worked — verified live with `<strong>`
count assertions and everything. So when the user reported the exact same
raw-asterisks symptom afterward, the reflexive assumption would be "the fix
didn't actually ship" or "there's a renderer edge case." Checked the actual
API response first instead of guessing: the answer string itself was
`"...* **Mean price – Indie games:** ≈$15.50 * **Mean price..."` — all one
physical line, bullets separated by inline `*` with no newline anywhere
between them. That is not valid CommonMark. No markdown renderer, correctly
implemented, turns that into a list — the asterisks are ambiguous emphasis
delimiters at best, literal characters at worst, and `react-markdown` was
doing exactly the right thing with genuinely malformed input. The Slice 9c
fix was real and still works (confirmed again this round); the bug was one
level upstream, in what the model was writing, not in what parsed it.

Fixed by adding an explicit formatting section to `SYSTEM_PROMPT_TEMPLATE`
(`src/agent/prompts.py`) — spelling out, as literal instruction rather than
implication, that the answer is rendered as markdown, that ≥3 related
numbers should become a real GFM table (header row, `|---|---|` separator,
data rows), and that a chained one-line `* label: value * label: value`
run is explicitly called out as *not valid* and *not to be used*. Paired
with a frontend change either way: added `remark-gfm` to `react-markdown`
so that when the model does emit a real table, it renders as a real
`<table>` (styled in `Markdown.tsx` to match the rest of the app — pixel-
font gold headers, monospace data cells) instead of raw pipes and dashes.
Neither fix alone would have been enough: prompt-only leaves a client that
can't render a table if the model gets the syntax right; renderer-only
does nothing about a model that never produces valid list/table syntax in
the first place. Verified against the literal question that broke
originally (Indie vs. other games price comparison) — real 2-row table,
zero raw `**`, zero raw `----`, confirmed as an actual `<table>` element in
a live browser, not just well-formed markdown in the API response.

### Actually testing mobile, instead of inferring it

Both the Slice 9c and Slice 9d entries logged "mobile wasn't separately
verified, but the responsive classes are unchanged so it should be fine" as
an open question — a reasonable-sounding inference that turned out to be
wrong. Ran Playwright with real device emulation (iPhone 14, Pixel 7) for
the first time on this redesign and found a genuine bug immediately: the
genre showcase's section header (`[ OR EXPLORE BY GENRE ]` next to "live
from the catalog") had `justify-between` with no `flex-wrap`. Every other
`justify-between` row in the app already had `flex-wrap` (added when each
of those was originally built and visually checked) — this one line was
just missed, and nothing caught it because desktop viewports never got
narrow enough to expose it. Fixed by adding `flex-wrap` + explicit gap,
same pattern as its siblings. The methodological point: "the classes didn't
change" is not the same claim as "the layout still works" — narrower
viewports can expose a latent bug in code that was never touched, if it was
never actually tested at that width to begin with.

### Dropping the Expo mobile client

The user's call, stated directly: a native app isn't worth it for what this
is — one tool, one page's worth of functionality, and doubling the UI
surface (Slice 9's whole visual system would need a second, React Native
implementation) for what a responsive browser tab already covers once it's
actually been verified to work on a phone-sized screen. Moved Slice 10 from
"Expo mobile client" to a mobile-web-polish-plus-deployment slice instead;
the Expo entry moved to Dropped in PLAN.md, next to the earlier Gemini
decision — same shape of decision (a planned-for-later item, reconsidered
once the actual cost/benefit was concrete instead of hypothetical).

### Addendum: 200 was a config default, not a SteamSpy limit

The user asked directly whether 200 games was all SteamSpy would give —
worth checking the real ceiling rather than assuming. It wasn't a limit at
all: `ingest_game_count = 200` in `src/config.py` was just the number
`ingest.py` happened to slice off SteamSpy's bulk `all` listing.
`get_all_page(page=0)` already returns ~1000 games (SteamSpy's own
per-page size, sorted by owners descending) — `ingest.py` was only ever
asking for the first 200 of those. Getting more right now is a config
change, not a code change; getting past ~1000 would be a real one
(`ingest.py` only fetches page 0 — looping over page 1, 2, ... isn't built).

Bumped the default to 1000 everywhere it's declared (`src/config.py`,
`.env.example`, the real `.env`) and re-ran ingestion. Deliberately did
**not** bump the `Dockerfile`'s build-time `ARG INGEST_GAME_COUNT` default
— that ingestion happens at Docker build time on whatever host deploys
this, and rate-limited to ~1 req/sec, 1000 games adds real minutes to
every future build. Left it at 200 (fast builds by default) with a
comment already in place noting it's overridable via `--build-arg` for
anyone who wants the full 1000 in a deployed instance and is fine trading
build speed for it.

### Open questions (new)

- **Mobile design is verified-not-broken, not verified-well-designed.**
  Slice 9e confirmed nothing overlaps/overflows; it didn't do a real design
  pass for touch targets, cartridge grid density at 2 columns, or
  leaderboard table readability at narrow widths. That's explicitly Slice
  10's job now, not assumed done.
- **Deployment platform: sticking with the Slice 6 decision (Vercel +
  Render/Fly.io) unless a real reason to reconsider comes up.** The user
  asked about alternatives (not Streamlit — correctly, this isn't a
  Streamlit app; floated Replit). Worth a documented compare-and-decide
  pass in Slice 10 rather than assuming the two-year-old-in-project-time
  Slice 6 reasoning still automatically wins, even though it's still
  probably right (this stack's real footprint — DuckDB, fastembed's ONNX
  model, scipy, multi-step LLM calls — doesn't fit a lightweight
  always-on-free-tier host well, which is the same reason Vercel's own
  Python functions were ruled out in Slice 6).

---

## Slice 9f — More data, and taste applied concretely instead of by feel

**Date:** 2026-08-26

Two follow-ons: confirming 200 games was a config default not a real
ceiling (see PLAN.md's Slice 9f entry for the mechanics — SteamSpy's bulk
page already holds ~1000, `ingest.py` just wasn't asking for that many),
and a design request that named a specific reference (Emil Kowalski's
public writing on interaction craft) rather than "make it nicer."

### Grounding "taste" in something checkable

Asked to apply Kowalski-style taste without inventing a fictional "skill"
for it — his actual published material is real and specific enough to work
from directly. Fetched his site and "7 Practical Animation Tips" rather
than working from vague memory of "that designer with the nice toasts."
Audited this app's existing motion against each of the 7 concretely:

- **Button press scale (0.97, `:active`)** — the Ask button and genre
  cartridges already had tap feedback; the example-question chips didn't.
  Real, findable gap once actually checked against a checklist instead of
  a general "does this feel okay" pass. Fixed, and unified every
  interactive element's press scale to the same 0.97 — one consistent
  value reads as deliberate, three slightly different ones read as
  accidental.
- **Avoid animating from `scale(0)`** — audited every `initial`/`animate`
  pair in the app; none do. No change needed, but worth having actually
  checked rather than assumed.
- **`ease-out`, not `ease-in`, for entering/exiting** — the hero stagger
  and card entrances already use a custom ease-out-heavy cubic-bezier
  (`[0.16, 1, 0.3, 1]`), applied symmetrically to both enter and exit
  transitions, which is exactly what the tip recommends (ease-out for
  both, not ease-in for exits). Already correct.
- **Keep it fast, <300ms for frequent interactions** — this app's
  interactive elements (button taps, cartridge selection) are already
  spring-based and settle quickly; explicitly set the new chip
  press-transition to 150ms rather than leaving it at Motion's spring
  default, for consistency with the Ask button.
- **Blur for rough transitions** — genuinely new. Added
  `blur(4px)→blur(0px)` to both the progress-panel and result-panel
  `AnimatePresence` transitions, matching Kowalski's specific "underrated"
  recommendation for masking the seam when one piece of UI replaces
  another.
- **Origin-aware transforms / skip delays on repeat tooltips** — don't
  apply here (no popovers-from-a-trigger-point or tooltip sequences in
  this app); noted as checked-and-not-applicable rather than silently
  skipped.

The point of fetching the actual source first: three of the seven tips
turned out to already be satisfied by decisions made earlier in this
project for unrelated reasons (the ease-out curve was chosen in Slice 9
for how it looked, not because it matched a specific external principle) —
worth knowing that's true, rather than assuming a "craft pass" always
means new changes everywhere.

### Open questions (new)

- **No literal "taste" or design-review skill exists for this project** —
  applied the specific external reference material directly this round;
  if this kind of audit becomes routine, a real project skill capturing
  "the checklist to run" would be the next step, not this ad hoc version.

---

## Slice 9g — Retro didn't work; a full, honest rebuild

**Date:** 2026-08-26

The direct feedback: "change the landpage completely, I want it to be
better, more professional and not look like AI slop." Worth being honest
about what this means for the previous three slices (9, 9b, 9c, 9d): the
synthwave-arcade direction, however carefully crafted at each step (the
cartridge chamfer bug, the sun-recentering math, the Kowalski animation
audit — all real, all correct engineering), was the wrong *concept*, not a
concept that needed more polish. Worth naming why plainly, since it's a
real lesson: a magenta grid horizon under a neon sun plus a pixel font is,
at this point, a recognizable template trope — codepen demos, "80s retro"
site-builder themes, and a large fraction of AI-generated landing pages all
converge on some version of it. Execution quality doesn't rescue a concept
that reads as generic before anyone evaluates the execution. This is
exactly the failure mode `artifact-design`'s own "avoid AI-generated
design" cliché list is warning about, just a cliché the list doesn't
happen to name — the list isn't exhaustive, the underlying test
("does this read as chosen for this subject, or as decoration pulled from
a shared aesthetic pool?") is what actually matters.

### Grounding the new direction in something real, twice

Two things were checked directly rather than assumed, same discipline as
the Kowalski animation pass: fetched linear.app itself (got a fairly
generic AI-summarized read back — the page is heavily client-rendered, so
the fetch didn't surface much concrete detail) and leaned on directly-
verifiable specifics instead of the vague summary: near-black grounds,
hairline borders, restrained single-accent color, real typographic scale
carrying the hero instead of a display face. Chose Geist over literally
matching Linear's font — distinct enough not to be a copy, same quality
tier and the same "serious dev tool" association (it's Vercel's own font),
and confirmed it was actually in the next/font/google catalog before
committing to it rather than assuming.

### Why real 3D, not a CSS approximation

The ask was specific: "3D objects of gaming icons." CSS `transform:
rotateX/rotateY` can tilt a flat plane in 3D *space* but can't give an
object actual volume — no real shading gradient across a curved surface,
no occlusion between faces. Genuine 3D needs a renderer. Added React
Three Fiber (WebGL) rather than faking it, and checked React 19
compatibility before installing (`@react-three/fiber@9.7.0`'s peer range
is `react >=19 <19.3`; this project is on `19.2.8`, inside the window) —
one version off in either direction and this would have been a much
worse afternoon.

Deliberately abstract primitive shapes (icosahedron, torus, box, cone,
capsule, octahedron), not literal controller/dice/joystick models: no
external `.glb` asset means no asset pipeline, no license question, and
keeps the same "hand-authored, self-contained" discipline the 2D genre
icons already established. Colors are read from the CSS custom properties
at mount (`getComputedStyle(...).getPropertyValue('--genre-1')`) rather
than a parallel hardcoded hex list — one source of truth for the genre
palette, same reasoning as everywhere else this palette is used. Lighting
is three manually placed lights (no `drei` `<Environment>` HDRI preset,
which pulls a texture from a CDN at runtime) — same no-external-asset
discipline extended to lighting, and it still reads as glossy/premium via
`MeshPhysicalMaterial`'s clearcoat property doing real work.

### Four lint findings that were real, not noise

The newer "React Compiler era" rules in `eslint-plugin-react-hooks`
(`react-hooks/purity`, `react-hooks/immutability`, `react-hooks/set-state-
in-effect`) flagged four things in the first draft of `HeroScene.tsx`.
Worth recording that none of these were dismissed as false positives
without checking first, even though three of the four are well-known
friction points between these rules and imperative libraries like React
Three Fiber:

1. `Math.random()` inside a render-path `useMemo` — a genuine purity
   violation, not a library-friction false positive. Fixed by deriving
   the per-object animation phase deterministically from the object's
   array index instead — same visual effect (objects bob out of sync with
   each other), zero randomness needed for what was only ever decorative
   variety.
2. Mutating `camera.position` (destructured from `useThree()`) inside
   `useFrame` — this **is** the standard, correct R3F pattern (`useFrame`
   exists specifically to mutate Three.js objects per-frame outside
   React's render cycle); the lint rule doesn't have a way to know that.
   Fixed by reading `state.camera` from the `useFrame` callback's own
   parameter instead of a hook binding destructured at the component's
   top level — same object graph, different AST shape, which is enough
   for the rule's pattern match to no longer trigger. This is the
   documented community workaround for this exact rule-vs-R3F conflict,
   not a project-specific hack.
3. One-shot `useEffect` + `setState` to read `prefers-reduced-motion` —
   fixed properly rather than silenced: replaced with
   `useSyncExternalStore`, React's actual mechanism for subscribing to a
   synchronous external value that can change after mount. Strictly
   better than what it replaced, not just quieter: the old version read
   the media query once on mount and never again, so toggling the OS
   setting mid-session while the tab was open wouldn't have updated
   anything; the new version does.

### A real, unrelated regression found along the way

Mid-slice, `uvicorn` (and `pip` itself) had gone missing from `.venv` —
broken in a way this session didn't cause directly (no `uv` commands were
run against this project between the last successful backend start and
this one). Recovered with the same `uv sync --extra agent` rebuild
procedure already validated once earlier in this project, rather than
investigating deeply — noted here as unresolved, not silently worked
around.

### Open questions (new)

- **Root cause of the venv regression is unknown.** Recovered, not
  diagnosed. If it recurs, worth checking whether something external to
  this session (a scheduled task, another tool, manual `uv` use in a
  different terminal) is touching `.venv` — the first two occurrences of
  venv corruption in this project both had a findable external cause
  (Windows file locks from a concurrent process); this one didn't get the
  same investigation.
- **The hero 3D scene is one fixed ensemble, not tied to which 8 genres
  are actually live.** It's deliberately decorative (a "the catalog has
  variety" statement, not a literal per-genre map) — the genre picker
  below does the real per-genre representation. Worth reconsidering only
  if that distinction stops being clear to a real viewer.
- **3D scene performance on lower-end mobile GPUs wasn't specifically
  profiled** — verified functionally correct (renders, no console errors,
  respects reduced-motion) on iPhone 14 emulation, but frame-rate under
  load on genuinely low-end hardware is unverified. Worth a real device
  check before treating this as deployment-ready.

---

## Slice 11 — A second Steam API, and the project's first real test suite

**Date:** 2026-08-27

Two questions arrived together: "is SteamSpy too limited, and should we
switch APIs?" and, separately, "how do we forecast real yearly trends?"
Worth untangling them, because they have different answers.

### Which API, and why not the ones proposed

SteamSpy's per-game endpoint gives owners/reviews/genre/price — enough for
most lookup and analysis questions, but nothing about release date,
Metacritic score, platform support, or feature tags (co-op, controller
support, Workshop). The user's own first proposal was three Steam Web API
endpoints from the official docs: `GetPlayerSummaries`,
`GetGlobalAchievementPercentagesForApp`, `GetNewsForApp`. Checked each
against what they actually return rather than taking the names at face
value: `GetPlayerSummaries` returns Steam *user* profile data (persona
name, avatar, online status) — it has nothing to do with games at all,
it's keyed on a Steam user id, not an appid. The other two are real but
narrow: achievement percentages is one metric, and news is unstructured
article text, neither is a general enrichment source. Said so directly
rather than building on top of a mismatch.

The actual fix was smaller than switching providers: Steam's own
storefront API, `store.steampowered.com/api/appdetails` — the same JSON
endpoint the Steam store website itself calls client-side to render a
game's page (confirmed by testing `?appids=620` for real and walking
through why the bare URL with no query param returns `null` — it's a
real API, just one that has no HTML front end of its own to visit
directly, unlike `store.steampowered.com/app/620` which is the actual
store page). Free, no key, no new account — same shape of win as the
existing SteamSpy/Steam-Web-API split, just a third source layered in the
same way. RAWG (20k free req/month) and IGDB (free via Twitch OAuth, rate
limited) were researched via WebSearch as real options for broadening the
*catalog itself* beyond what SteamSpy's bulk list returns, but that's a
bigger, separate scope decision (different client, different id space,
probably a merge/dedup step against the existing `appid`-keyed table) —
recommended treating it separately rather than folding it into this slice.

### What got built

`SteamStoreClient` (`src/ingestion/steam_store_client.py`) mirrors
`steam_web_client.py`'s existing pattern rather than inventing a new one:
same 1.5s rate-limit-between-calls approach, same on-disk JSON caching to
`data/raw/`, just a different cache-key prefix (`storeapi_` vs.
SteamSpy's own `appdetails_`) so the two caches can't collide. `run_ingestion`
now calls both APIs per game and merges the results in `_row_from_appdetails`.

Five new columns landed on `games`: `release_date` (parsed), `release_date_raw`
(kept as-is even when parsing fails — never silently drop the source
string), `metacritic_score`, `platforms`, `categories`. `categories` runs
through a hand-curated 15-entry allowlist (`CATEGORY_ALLOWLIST` in
`ingest.py`) rather than storing Steam's full ~30-tag raw list — most of
that raw list is controller/accessibility/remote-play noise that would
just add retrieval confusion for an agent trying to answer "does this
game have co-op" type questions.

Two real data quirks found by testing against a live response (Portal 2,
appid 620) instead of inventing fixture shapes:

- Its `categories` array has two different `id`s (51 and 30) that both
  carry the description "Steam Workshop" — a real duplicate in Steam's own
  data, not a parsing bug. Dedup logic keys on description, not id,
  specifically because of this.
- `platforms.mac` is `false` in the live response even though
  `mac_requirements` has actual text in it — meaning the authoritative
  signal is the `platforms.*` booleans, and inferring platform support
  from non-empty `*_requirements` text would have been wrong.

Both went straight into `tests/test_ingest_parsing.py` as
`REAL_PORTAL_2_CATEGORIES` — captured real data, not invented shapes,
specifically so the dedup logic is tested against the actual quirk that
motivated it.

The old `games.duckdb` was deleted and rebuilt from scratch rather than
migrated — the whole table is UPSERT-regenerable from raw ingestion and
gitignored, so there was no state worth preserving across the schema
change. This paid off for real mid-run: the full 1000-game re-ingestion
hit a genuine transient DNS resolution failure against
`store.steampowered.com` around appid 606280 (`getaddrinfo failed`, not
an API error — confirmed by re-resolving the same host seconds later and
getting a normal response) and the script has no retry logic around
network calls, so it crashed with a real traceback and a non-zero exit
code roughly 675-680 games in. Deliberately did not add retry/backoff
logic in response — the existing cache-first design in both API clients
already made this cheap: every successfully-fetched game (SteamSpy and
storefront alike) is cached to `data/raw/` before the crash point, so
simply re-running the same `ingest --count 1000` command picked up
exactly where it left off, re-fetching only the one failed game and
whatever hadn't been reached yet, instead of losing an hour of API calls
to one bad DNS lookup. Worth calling out as the resumable-cache design
actually earning its keep on a real failure, not just a theoretical
benefit.

Also added 5 new RAG schema chunks
(`src/agent/rag/schema_corpus.py`) so the agent can actually retrieve and
use these columns — worth calling out one wording choice: the
`metacritic_score` chunk explicitly says NULL means "not scored," not
"scored zero," because an LLM asked to rank by Metacritic score could
otherwise silently treat unscored games as the worst-rated instead of
excluding them.

### The project's first real test suite

No `tests/` directory existed anywhere in this project before this slice
— stats/forecast/viz logic had gotten real ad hoc verification during
their original slices (see Slice 4's entry for a real type-coercion bug
and a real group-mislabeling bug caught that way) but never a checked-in
regression suite. Built one from scratch: `tests/conftest.py` plus 6 test
files, 66 tests total.

The one deliberate philosophy, stated directly in `conftest.py`'s
docstring: prefer a real, throwaway on-disk DuckDB file over mocking this
project's own DB layer. The `games_db` fixture creates a genuine temp
DuckDB file, runs the real `CREATE TABLE` SQL from `schema.py`, and
inserts synthetic-but-realistic rows — so a test failure here means the
real schema/query code actually broke, not that a mock's assumptions
drifted from reality. `test_sql_guard.py` in particular runs one real
`DROP TABLE games` against a live fixture DB and asserts it's rejected
*and* the table is still there afterward with its 4 rows intact — testing
the actual guard behavior end to end, not just that a regex matched.

`pyproject.toml` gained a `dev` extra (`pytest`) rather than a
`[dependency-groups]` entry. That's a deliberate choice given this
project's history: `[dependency-groups]` installs by default on a bare
`uv sync`, and this project has already been bitten twice for real by
bare `uv sync` silently reconciling down to the lean base set (see
Slice 9b/README's warning) — adding `dev` as another *optional* extra
keeps that failure mode from getting worse, at the cost of needing
`--extra dev` explicitly to run tests locally.

A genuine test-authoring bug, found and fixed correctly: the first draft
of `test_outliers_finds_the_obvious_one` used 4 points (3 normal + 1
outlier) and asserted 1 outlier would be found. It found 0. Verified by
hand rather than assumed: mean ≈108, stddev≈219 with only 4 points, so
the outlier's own z-score (≈1.79) came in under the 2.5 threshold — a
real small-sample "masking" effect, where one extreme point inflates the
mean/stddev enough to pull its own z-score back down, not a bug in
`_outliers`. Fixed the test (20 clustered points + 1 outlier), not the
code, and left a comment on the test explaining the real effect so a
future reader doesn't "fix" it back to something smaller.

`.github/workflows/test.yml` runs the suite on every push/PR. First draft
used `uv sync --extra dev` only, with a comment claiming the LLM/RAG stack
wasn't needed for these tests — wrong, caught by actually tracing the
import chain: `test_stats_tool.py`/`test_forecast_tool.py` import
`stats_tool.py`/`forecast_tool.py`, which import `numpy`/`scipy`, which
live in the `agent` extra, not base. Fixed to `--extra agent --extra dev`
before considering the workflow done, rather than trusting the first
draft's comment.

A second real bug, found by actually re-running the suite exactly the way
the README/CI tell people to (`uv run pytest -v`) rather than trusting
that it still worked because it had passed once already:
`ModuleNotFoundError: No module named 'src'`, failing at `conftest.py`'s
own import. Root cause is a genuine, well-known pytest gotcha, confirmed
by testing both invocations directly: `python -m pytest` puts the current
directory on `sys.path[0]` (that's what `-m` does for any module), so
`import src` resolves; the plain `pytest` console-script entry point never
does that, so it doesn't. This project's tests had happened to be run via
`-m` (or an equivalent) during initial development, masking the gap until
a plain `uv run pytest` was tried. Fixed at the pytest-config level rather
than by telling people to remember `-m`: added `pythonpath = ["."]` to
`[tool.pytest.ini_options]`, pytest's own built-in mechanism for this
(since 7.0) — makes `pytest`, `uv run pytest`, and `python -m pytest` all
behave identically, and would have silently broken the CI workflow too
(same `uv run pytest -v` line) had it not been caught here first.

### Open questions (new)

- **`poll_player_counts.yml`'s catalog-rebuild step now costs meaningfully
  more CI time per 6-hourly run** — it re-ingests the full catalog just to
  get appids for the player-count poll, and now that ingestion also makes
  a storefront-API call per game with no cache persisted between GH
  Actions runs (`data/raw/` is gitignored). Noticed while re-reading the
  workflow for style conventions, not yet raised as a problem worth
  solving — a narrower "just fetch appids" path is the likely fix if the
  added runtime becomes a real issue.
- **Real yearly-forecast extrapolation still isn't built.** The user's
  actual ask ("delta players per year, then extrapolate") needs historical
  yearly snapshots that don't exist from any free source today. The
  `player_counts` table is the wrong series for this (6-hourly live CCU,
  not yearly). Proposed, not yet confirmed or built: converting the
  weekly `refresh_catalog.yml` re-ingestion into an append-only
  `owners_history` table, mirroring `player_counts`'s
  accumulate-don't-overwrite contract, so real year-over-year deltas
  accumulate naturally over time instead of being faked from one
  snapshot.

---

## Slice 12 — Naming Ludo, a "Meet Ludo" section, and a real catalog page

**Date:** 2026-08-27

Three asks arrived together: name the agent, add an intro section
explaining what it can do (floating gaming-object 3D iconography +
example questions), and build a separate full-catalog browse page. Named
the agent **Ludo** — offered a short list of directions (Ludo, Steamsight,
Datapad) and the user picked Ludo directly. Saved as a standing project
memory (`agent-name-ludo.md`) so future sessions use the name by default
in new copy without re-asking.

### A second 3D scene, and a real lesson about color and darkness

`GamingObjectsScene.tsx` needed five actually-recognizable objects
(controller, console, TV, disc, cartridge), not HeroScene's abstract
genre-colored primitives — a harder ask, since a controller only reads as
a controller if its grips/joysticks/buttons are legible, not just "a
lit shape." Built each as a small group of primitive geometries (drei's
`RoundedBox` plus core cylinder/capsule/sphere/torus primitives) — no
external `.glb` assets, same discipline as HeroScene.

First real render was nearly invisible: used `--surface-raised` (the
app's near-black 2D panel token, `#17181b`) as the chassis color, which
against the canvas's equally near-black background produced silhouettes
you had to squint to see (verified by actually rendering and screenshotting
it, not assumed fine because the geometry was correct). HeroScene's
objects don't have this problem because they're saturated genre hues, not
because of anything special about its lighting. Fixed with a dedicated
mid-graphite hex (`#6b6f79`) picked specifically for this scene rather
than reused from a 2D token — a color that works as a flat panel
background and a color that works as a lit 3D object's surface are not
the same design decision, even when the palette intent ("restrained,
neutral") is identical. Also had to fix the disc separately: its
`metalness=0.85, roughness=0.15` (very mirror-like) reflected essentially
nothing under this scene's 3-light setup (no environment map), so the
disc's body was invisible while only its accent ring showed — dropped to
`metalness=0.45, roughness=0.35` so it actually picks up diffuse light.
Deliberately did NOT reuse the genre categorical palette to color these
objects, even though it was sitting right there and would have been an
easy source of "more color" — that palette means something specific
elsewhere in the UI (which hue is which genre) and reusing it as pure
decoration here would manufacture a false mapping (a controller isn't
"Action-colored" for a real reason).

Pulled `useReducedMotion` out of `HeroScene.tsx` into
`lib/useReducedMotion.ts` the moment a second scene needed the identical
hook — no reason to fork it, and it's now the one place that logic lives.

### The catalog page's real sorting bug, caught before shipping

`src/db/catalog.py` filters and sorts in Python rather than building
dynamic SQL (the sort column is user input, and a column name can't be a
bound parameter — an allowlist dict is simpler and safer than templating
SQL). First implementation floored NULL values to a low sentinel (`-1`,
`date.min`) so they'd sort to the end — except that only works for
descending order; on ascending order a floored NULL sorts *first*, the
opposite of "always last" the code's own comment claimed. Caught this for
real via `tests/test_catalog.py::test_sort_by_release_date_ascending_puts_nulls_last`
failing, not by re-reading the code and spotting it — extended the
`games_db` fixture with real release dates on two rows specifically so
this test would be meaningful (all-NULL fixture rows can't test NULL
ordering against real values). Fixed by splitting rows into "has a value"
and "doesn't," sorting only the first group with the requested direction,
and appending the second group after — unaffected by `reverse`, so
NULLs land last regardless of ascending or descending.

### Real Playwright verification, not just unit tests

Ran the actual app (backend `uvicorn`, frontend `next dev`) and drove it
with Playwright (chromium, found already installed at
`~/AppData/Local/ms-playwright/chromium-1234` from earlier project work —
Python's `playwright` package needed the executable path passed
explicitly since its expected bundled Chromium revision didn't match
what was actually on disk) across both routes, both a desktop and a
mobile viewport: zero console errors, zero horizontal overflow, and
real interaction checks (typed a search term and read back the actual
filtered rows, selected a genre and verified every visible row's genre
field, clicked through to page 2 and confirmed the page indicator
updated) against the live 1000-game backend — not mocked, not assumed
from reading the component code. Also caught the catalog table's real
horizontal-scroll problem this way: 9 columns didn't fit in
`max-w-5xl` on an ordinary 1440px viewport, forcing a scroll a first-time
visitor wouldn't necessarily notice. Widened the catalog page specifically
to `max-w-7xl` (the ask-flow's own container stays `max-w-2xl` — a
focused single-question flow and a dense data table warrant different
widths, same reasoning that already put the hero at `max-w-5xl`).

Finished with the project's full pre-existing verification bar: `npm run
lint` (caught one real error — a plain `<a href="/">` where
`next/link`'s `<Link>` was required), `tsc --noEmit`, and `npm run build`
all clean, both routes prerendering as static content.

### Open questions (new)

- **The catalog table still needs its own horizontal scroll on narrow
  mobile viewports** (confirmed: no *page-level* overflow, but the table
  itself scrolls within its `overflow-x-auto` container past a few
  columns) — consistent with `ResultTable`'s existing behavior elsewhere
  in the app, but with no visual affordance (a fade/scroll hint) that more
  columns exist off-screen. Not fixed here; worth a real mobile design
  pass if this becomes a common complaint.
- **No URL-synced filter state on the catalog page** — search/genre/sort/
  page all live in component state only, so a copied catalog link always
  reopens to the unfiltered first page rather than reproducing what the
  sender was looking at. A reasonable follow-up if shareable catalog
  views turn out to matter, not built this round to keep the slice scoped.

---

## Slice 12b — A named real reference, and a lesson about drift

**Date:** 2026-08-27

Different kind of design request than the earlier "apply Kowalski/taste
directly" one: this time the user linked an actual live site
(framer.com) and named two colors (turquoise, black) rather than
describing a feeling. Scoped it with two quick questions before touching
anything, since both readings of "ditch the objects" were plausible and
expensive to get wrong: keep HeroScene's abstract shapes (they're already
working, already genre-colored, and weren't the thing being complained
about) and replace only the newer gaming-object scene; make turquoise the
one site-wide `--accent` rather than confining it to the new background.
Both confirmed directly rather than assumed.

### What "Framer-inspired" concretely meant here

Fetching framer.com's actual page didn't yield much — a markdown
extraction of a heavily-scripted marketing site loses essentially all of
the actual visual design (colors, blur, layout) and returns text content
instead ("earthy color palettes," organic-shapes copy — the *words* the
page uses to describe itself, not what it looks like). Worth naming
plainly rather than pretending the fetch gave real grounding: the actual
execution here came from well-established, genuinely common knowledge of
what that whole category of AI-product marketing page looks like (large
soft blurred gradient orbs drifting behind copy, not literal 3D
illustration) — combined with the user's own explicit color instruction,
which matters more here anyway since it overrides whatever hue Framer's
site actually uses today.

### The scene that got deleted, and what replaced it

`GamingObjectsScene.tsx` — five hand-modeled objects, real Playwright
verification, tuned lighting and materials — all deleted outright in one
turn, on direct instruction. Not kept behind a flag or commented out;
the user said ditch it, and half-removed code non-functional code is
worse than none. Replaced with `GradientBlobs.tsx`: three blurred
turquoise circles with slow independent drift, respecting
`prefers-reduced-motion` through the same shared hook the deleted scene
also used, plus a small inline SVG `feTurbulence` noise overlay — the
detail that keeps a large blurred gradient from banding, a real technique
this category of site actually uses, not decoration for its own sake.
Placed the gradient behind the *entire* "Meet Ludo" section (heading,
copy, chips) rather than in a small boxed illustration slot the way the
3D scene had been — a truer read of how this treatment is actually used
on sites like Framer's (an ambient field behind content, not a separate
diagram next to it).

### A real piece of accent-color drift, found by grepping before changing anything

Before touching `globals.css`, grepped the whole frontend for the old
hex (`f0a63a`) and the word "gold" rather than assuming the CSS variable
was the only place the color lived. Found one real case of drift:
`HeroScene.tsx`'s accent-colored point light was hardcoded to the literal
old hex instead of reading `--accent` like the object materials in the
same file already did — an inconsistency that would have silently kept
glowing gold after this slice's turquoise switch if the grep hadn't
caught it. Fixed to read the CSS variable live via `getComputedStyle`,
matching the pattern already used elsewhere in the same file, so a future
accent change won't need to remember this spot exists.

### Cleanup discipline, after last time

Left a background `uvicorn`/`next dev` pair running after the previous
slice's Playwright verification, which the user then hit for real as a
port-bind error in their own terminal. This time: identified and killed
both by the actual listening PID on ports 3000/8000 (`netstat` + explicit
`taskkill`) before considering the slice finished, not just before moving
on to the next task.

### Open questions (new)

- **No dark/light theme distinction was reconsidered for the turquoise
  swap** — this app is deliberately single-theme dark (see globals.css's
  own comment), so this doesn't apply yet, but worth remembering if a
  light variant is ever added: turquoise's contrast behavior on a light
  ground hasn't been checked at all.

---

## Slice 13 — A design brief for a different product, and what to keep from it

**Date:** 2026-08-27

Direct, blunt feedback arrived alongside a full ChatGPT-generated design
brief: "the website is not nice at all, I want it to be like this." Worth
being honest about what the brief actually specified before writing any
code, because it wasn't a redesign of Ludo — it was a complete spec for a
different fictional product: "PlayerLens AI," an internal analytics tool
for game *studios*, with example content built entirely around churn
rates, D7 retention, ARPDAU, and monetization cohorts (`mock-data.ts`,
`POST /api/agent` as a not-yet-real placeholder). None of that data
domain exists in Ludo's real catalog (SteamSpy + Steam storefront —
owners, reviews, price, genre, Metacritic, platforms), and Ludo's entire
premise — the thing that makes it a real project rather than a demo — is
that every number comes from a real query, never a mock.

### Separating craft from content before building anything

The temptation with a 250-line brief like this is to implement it
literally, since it's detailed enough to look like a spec. Resisted that
and asked two direct questions instead: keep Ludo's name/backend/real
data, or actually rebuild the product as PlayerLens with mocked churn
data? And separately, re-skin fully to the light Roman-marble look, or
just raise the craft bar on the existing dark identity? Both answers came
back the same direction: adopt the *visual language* (warm ivory, serif
headlines, restrained classical motifs, one royal-blue accent) while
keeping Ludo real. That's the actual scope this slice executed — a full
re-skin, zero fictional content anywhere on the page.

### What "restrained Roman" meant in practice, not in the brief's literal asks

The brief asked for a Colosseum background photo, a marble statue bust,
and a Corinthian column image (with fallback CSS if unavailable). None of
those got built — not because they'd look bad, but because sourcing
photographic imagery breaks a rule this project has held since Slice 9g
(HeroScene's abstract shapes over literal dice/controllers) and Slice
11/12b (every visual element hand-authored in code, nothing fetched):
no external image or model assets, ever. The brief's own instructions
actually argue for this independently — "architectural texture, not a
hero photograph," "must not look like Rome tourism," restraint over
literalism throughout. Built two hand-drawn SVG motifs instead: a laurel
sprig (`Laurel.tsx`, mirrored in pairs) and a single faint line-drawn
arch (`RomanArch.tsx`, opacity ~0.4 of an already-pale border color) —
plus an abstract ring-and-ticks medallion replacing the plain-text nav
logo. Three small elements, used exactly once or twice each, is
deliberately closer to the brief's own stated ratio ("80% modern SaaS,
15% classical, 5% game-specific") than anything more elaborate would
have been.

### The hero visual had to go, and what replaced it

Slice 12b's abstract 3D shapes (kept through the turquoise pass on the
strength of "already working, don't touch it") stopped being right the
moment the background went from near-black to warm ivory: saturated
genre-colored primitives read as toy blocks next to a serif headline on
a marble-adjacent palette, not as a premium product visual. Rather than
force them to survive a third re-skin, built `HeroPreview.tsx` — a
static floating panel showing a **real** answer Ludo already gave earlier
in this same session (the five highest-rated-games result, verified live
against the actual backend, not invented for this panel), styled as the
same card the real result view uses. This is actually closer to what the
brief's own hero concept wanted anyway ("a large floating application
panel... ChatGPT + Linear + modern BI dashboard"), and it keeps the
"everything on this page is real" rule intact where a mockup-style hero
illustration wouldn't have.

### A capabilities row grounded in the real graph, not the brief's four-item template

The brief's ASK / INVESTIGATE / EXPLAIN / ACT framework assumes a product
that gives recommendations ("ACT: turn insights into actions your team
can use"). Ludo doesn't do that — it answers questions, it doesn't
recommend business decisions. Rather than invent a fourth capability to
match the template's shape, used exactly the three things Ludo's real
pipeline (`router` → `retrieve_schema` → `agent` → `execute_tools`,
already documented in ARCHITECTURE.md) actually does: Ask, Investigate,
Show the work. Three real capabilities beat four templated ones.

### Fast reversal, named plainly

Slice 12b's turquoise accent shipped less than a day before this slice
replaced it with royal blue. Worth naming as what it is — a fast
reversal driven by direct user feedback that the previous pass wasn't
landing, not a mistake to smooth over in the log. The same "one
confident accent, changed cleanly through a single CSS variable, genre
palette untouched" mechanism held up for a third theme in a row, which is
exactly what having that mechanism was for.

### Verification

Same bar as every previous frontend slice: `npm run lint`, `tsc
--noEmit`, `npm run build` all clean; Playwright against both routes,
desktop and mobile, zero console errors, zero horizontal overflow, one
full real `/ask` round trip screenshotted to confirm the new accent and
panel styling actually reached the live result view, not just the hero.
Backend's 76 tests re-run as a sanity check even though this slice never
touched backend code. One operational note: the dev servers used for
verification this time were the user's own already-running instances
(recognized via a coincidentally-reused PID from the previous slice's
cleanup, still answering real requests) — pointed Playwright at the
existing ports rather than starting a competing pair, avoiding a repeat
of Slice 12b's leftover-server mishap by not spawning new servers at all.

### Open questions (new)

- **No `PlayerLens`-style mocked showcase was built anywhere** — the
  user's second answer confirmed "keep everything real" over "add a
  labeled mock section," so nothing on the page shows fabricated
  churn/monetization content. If a portfolio reviewer specifically wants
  to see how Ludo might extend toward studio-side analytics, that would
  need real design/scoping work (what data would that even require,
  where would it come from), not a mockup bolted onto this page.
- **The capabilities-row vertical dividers are quite subtle** on the
  light background (`divide-[var(--border)]`, the same pale warm-stone
  border used everywhere else) — intentional restraint, but worth a
  second look if a reviewer says the three columns don't read as
  separated at a glance.

---

## Slice 14 — Reverting fast, illustrating for real

**Date:** 2026-08-28

Direct rejection, no hedging: "I dont like this design at all." Paired
with four concrete new asks in the same message — a dark animated black/
green/turquoise background, a bolder "Apple style" font, a hand-
illustrated classical figure with gaming headphones holding a controller,
and Roman-styled genre icons. Unlike the PlayerLens brief two slices ago,
these were concrete and internally consistent enough to execute directly
rather than ask more clarifying questions — the user had already been
asked twice this session (Slice 12b's scope, Slice 13's re-skin-vs-craft
question), and a third round of questions on a message this specific
would have read as stalling rather than diligence.

### Reverting the mechanism, not just the values

Slice 13's light theme used a flat ivory background and a two-layer white
drop-shadow on every panel — neither survives a straight color swap back
to dark. A `box-shadow: 0 12px 28px rgba(...)` reads as "premium" against
flat ivory and reads as nothing (or a faint artifact) against a moving
dark gradient; dropped the shadow layer from `.panel` entirely and went
back to the flat hairline-bordered dark surface Slices 9g/12b already
proved out, rather than trying to make the light-theme mechanism work
with dark values plugged in.

### One fixed background layer, not a per-section effect

"Make the background animated" reads as a whole-site property, not one
section's decoration — the honest way to build that is one fixed,
full-viewport layer behind everything (`AuroraBackground.tsx`), not a
duplicated effect re-mounted in the hero and `MeetLudo` and anywhere else
separately. Made `body`'s own background transparent so the fixed layer
shows through in the gaps; content panels (the nav bar, `.panel` result
cards, the catalog table) keep their own opaque fills on top for
legibility, so a moving background never fights actually reading a data
table. `MeetLudo`'s own opaque section background (added in Slice 13)
came out for the same reason — it was blocking the aurora exactly where
the user would expect to see it keep going.

`lib/useReducedMotion.ts` — deleted three slices ago once nothing used
it — came back the moment AuroraBackground needed the identical hook.
Worth naming as real churn rather than smoothing it into the log: the
underlying need (something in this app wants to freeze under reduced
motion) is stable across redesigns even though which component needs it
keeps changing with every visual pass.

### The illustration, and a real layout mistake caught by looking

`RomanGamerBust.tsx` is the biggest new asset this slice: a hand-
authored SVG bust, not a fetched illustration or photo — the same
discipline every visual element in this app has followed since Slice 9g,
applied to something considerably more ambitious than a primitive shape
or a line-drawn arch. Chose a profile (cameo) pose deliberately over a
frontal face: a side silhouette is achievable with a handful of bezier
points and reads as intentional, where a frontal face attempted from
primitive shapes needs eyes/nose/mouth or it looks unfinished or eerie —
picking the pose that a modest set of hand-drawn curves can actually pull
off convincingly, rather than the pose that was asked for most literally.

First composition attempt overlapped the illustration behind the
`HeroPreview` real-answer card, offsetting the card only slightly. Real
mistake, caught by actually rendering and screenshotting it rather than
trusting the JSX: the card ended up covering almost the entire bust,
including the controller — the one detail the user had specifically
named. Fixed by stacking the two vertically instead of overlapping them,
which also sidesteps needing to tune two elements' relative z-order and
offsets to avoid occlusion on every future edit to either one.

### Genre icons: one consistent treatment, not ten bespoke redraws

"Make the icons more Roman style" could have meant redesigning all 10
hand-drawn glyphs in `GenreIcon.tsx` from scratch. Chose the cheaper,
more consistent option instead: wrapped each existing glyph in a circular
ring-bordered chip with an inset highlight (a coin/medallion frame)
rather than reinventing what each glyph depicts. This reads as a real
stylistic shift applied uniformly, and avoids the risk of some bespoke
Roman reinterpretation of "RPG" or "Simulation" losing the at-a-glance
legibility the current glyphs already have.

### A concrete "scroll" answer instead of another on-mount fade

Every animation in this app up to this slice was either an on-mount
stagger or a `whileInView` trigger — real, but not actually driven by
scroll position. Added genuine scroll-linked parallax (Motion's
`useScroll`/`useTransform` against the hero's own scroll progress) for
the arch line-art and the bust, drifting at different rates so they read
as sitting at different depths rather than moving as one flat layer —
the concrete version of "add transitions when you scroll down," not
another entrance animation.

### Verification

Same bar as every prior frontend slice: `npm run lint`, `tsc --noEmit`,
`npm run build` all clean; Playwright against both routes, desktop and
mobile, zero console errors, zero horizontal overflow, one full `/ask`
round trip screenshotted to confirm the new dark theme and turquoise
accent actually reach the live result view. Backend's 76 tests re-run as
a sanity check, unaffected by a frontend-only slice. Verified against the
user's own already-running dev servers again rather than starting a
competing pair.

### Open questions (new)

- **This is the fourth distinct visual identity this project has shipped**
  (Slice 9g's dark dev-tool look, Slice 12b's turquoise variant, Slice
  13's light Roman-marble pass, this dark-again illustrated one). Worth
  watching whether the next round of feedback converges on refining this
  one or triggers a fifth full pass — if it's the latter, worth explicitly
  asking what specifically isn't working before rebuilding again, since
  four full re-skins in one session is a real signal that something about
  how direction gets set (not the execution quality of any single pass)
  might be worth addressing directly.
- **`RomanGamerBust.tsx`'s facial profile detail (brow/nose/chin) is
  subtle at the rendered size** — visible on close inspection but reads
  mostly as a smooth silhouette at a glance. Superseded by Slice 14b's
  `RomanGamerStatue.tsx` (a front-facing figure, not a profile), so this
  specific concern no longer applies to the current illustration, but the
  underlying lesson (hand-typed bezier coordinates for a face read as
  "acceptable" more than "polished") carries forward to any future
  illustration work.

---

## Slice 14b — A real hydration bug, and building toward a real reference

**Date:** 2026-08-28

Two unrelated things arrived in one message: a genuine console error from
the user's own browser, and a reference photo with "let's use this
statue." Worth treating them as the different kinds of problems they
actually were — one a bug to fix correctly, the other a design direction
to interpret honestly.

### The hydration error: fixing the class of bug, not the instance

`Uncaught Error: Target ref is defined but not hydrated` from Motion's
`useScroll({ target: heroRef })`, added in Slice 14 for the hero's
scroll-linked parallax. Checked Motion's own troubleshooting page before
guessing at a fix: the documented cause is exactly "ref isn't properly
connected to a DOM element" at the moment `useScroll` needs it — and the
code here already looked like the documented-correct pattern (`heroRef`
attached directly to a plain `<div ref={heroRef}>`, no wrapper component
swallowing the ref). That's the tell that this is a timing race in how
Next.js App Router client components hydrate relative to when Motion's
internal effect fires, not a straightforward misuse the docs' checklist
would catch.

Rather than chase the exact race condition (client-only guard flags,
delaying the `useScroll` call until after a mount effect, etc.), switched
to the ref-free form of `useScroll()` entirely — it tracks raw
`window.scrollY` instead of a ref-relative scroll range, which structurally
cannot hit this class of error since there's no target ref to be hydrated
or not. Rewrote both parallax transforms against pixel scroll position
(`useTransform(scrollY, [0, 700], ...)`) instead of scroll progress
(0 to 1) relative to the hero's own bounding box — a small semantic
change (the parallax range is now a fixed pixel distance rather than
"across the hero's own height"), traded deliberately for removing an
entire category of hydration bug rather than patching one occurrence of
it. Verified with a fresh (non-HMR) Playwright page load plus repeated
programmatic scroll events, specifically because the original bug report
came from a live session that had been through many rapid HMR updates —
worth ruling out that as a contributing factor by testing a clean load.

### The statue: honoring the reference without embedding it

The shared photo was a real, specific classical marble statue — someone
else's photograph of a museum piece, copyrighted regardless of how it's
being used here. Two independent reasons not to embed it directly: this
project's standing rule (every visual asset in this app has been
hand-authored code since Slice 9g, no exception made yet for anything,
including a user-supplied reference image) and the more general point
that redistributing someone else's photograph inside a shipped product
isn't something to do on a user's behalf without them separately sourcing
usage rights for it. Said this plainly rather than silently either
embedding it or silently ignoring the request.

What did change for real: the illustration's whole composition. The
previous version (`RomanGamerBust.tsx`) was a shoulders-up profile bust —
reasonable, but nothing like what the reference actually showed (a full
standing figure, one arm raised holding a scepter aloft, a bold diagonal
gold drape, fuller curled hair). Rebuilt as `RomanGamerStatue.tsx` with
those same structural choices, translated to what the controller-holding
premise needs: the raised arm now holds the controller aloft instead of
a scepter, the gold drape is still gold and still diagonal (the
reference's one strong color note against white marble, kept rather than
forced into the site's turquoise to match chrome), and the hair is fuller
and curlier than the previous version's minimal treatment.

### A real geometry bug, caught by looking at the actual render

First draft of the raised arm used two independently-rotated rect
segments (upper arm + forearm) with two different, uncoordinated
rotation pivots. The math for each rotation was correct in isolation, but
the forearm's pivot didn't correspond to where the upper arm's rotation
had actually put its own endpoint — so the two segments didn't connect,
and the controller (positioned relative to the forearm's assumed end
point) rendered floating disconnected above the hand, not held in it.
Found by screenshotting the render, not by re-deriving the trigonometry
on paper — the same "verify against the actual output" discipline this
whole project applies to SQL results and stats computations applied here
to hand-drawn geometry instead. Fixed by simplifying to one rotated
segment for the whole raised arm, actually working out the rotation
trigonometry by hand for that single segment (pivot at the shoulder,
solved for the rotation angle that lands the far end at a target point
within the canvas), then placing the hand ellipse and the controller
group at that computed endpoint rather than an eyeballed one. One simple,
correctly-computed segment beat two more detailed but disconnected ones.

### Verification

`npm run lint`, `tsc --noEmit`, `npm run build` all clean. Playwright:
the fresh-load-plus-scroll check specifically targeting the reported
error (confirmed gone), plus the usual full pass (both routes, desktop +
mobile, zero console errors, zero horizontal overflow). Backend untouched,
76 tests still green.

### Open questions (new)

- **The bent (non-raised) arm in `RomanGamerStatue.tsx` reads as fairly
  subtle against the torso** — superseded by Slice 14c, which replaced
  the whole hand-drawn illustration with a real photo; no longer
  applicable, kept here only as a record of what the hand-drawn version's
  known rough edge was.
- **The parallax range change (pixel-based vs. hero-height-relative) means
  the exact drift distance is no longer proportional to the hero's actual
  rendered height** — fine at the viewport sizes this was checked at
  (1440×900 and 390×844), but an unusually tall or short hero rendering
  (e.g. a very wide ultrawide monitor, or a font-scaling accessibility
  setting that changes hero height substantially) could make the parallax
  feel slightly under- or over-scaled relative to how far you've actually
  scrolled past the hero. Not verified at other viewport sizes.

---

## Slice 14c — The first photographic asset, verified before use

**Date:** 2026-08-28

Two things arrived together, worth treating separately: a request to
connect Claude Code to Framer (a genuine research question, answered by
actually looking up what Framer's current external-agent feature does
rather than guessing from training-data-era knowledge), and then a
follow-up that reframed the real goal — "an agent that will help me
design an actual good website" — which named the real problem with this
whole session honestly: four full re-skins from me hand-writing CSS
blind is a worse loop than a tool with an actual visual canvas. Answered
that plainly rather than oversell what I can do without one.

Then, separately: "let's use Claude Design... add more realism with the
Roman statue... let's use a realistic statue of Apollo." Worth being
precise about what "Claude Design" actually is before acting on it —
looked it up rather than assumed: a real Anthropic research-preview
product at claude.ai/design, a separate browser surface I can't drive
from Claude Code, not a tool call available here. Said so directly. The
"realistic Apollo statue" part was the real, actionable request.

### Why a hand-drawn SVG can't do this, and what that implies

"Realistic" is a real requirement a hand-typed bezier path structurally
cannot satisfy — Slice 14b's `RomanGamerStatue.tsx` was already pushing
the ceiling of what primitive-shape illustration can achieve, and more
iteration on that ceiling wouldn't produce photorealism, just a more
polished non-photorealistic illustration. That's a different kind of
constraint than the ones this project has been solving with more careful
code (a sorting bug, a hydration race) — no amount of correct trigonometry
turns rectangles into marble. Said this directly rather than attempting
another SVG pass and hoping it read as more real than the last one.

### Scoping the real fork before touching anything

Three genuinely different ways to get a real photo exist, with different
costs: I find and verify a public-domain photo myself (breaks this
project's hand-authored-only rule, needs real license diligence on my
part); the user sources/generates one themselves and hands it to me
(sidesteps me making licensing judgment calls on their behalf); or stay
fully hand-drawn and accept the ceiling. Asked which, rather than picking
one — this is exactly the kind of fork where a wrong guess is expensive
(a licensing misstep isn't something to walk back quietly) and the
user's own risk tolerance is the actual deciding factor, not something I
can infer from the conversation so far.

### Verifying a license on the actual file, not the search summary

Found the Apollo Belvedere on Wikimedia Commons via search, but the
aggregated search summary just asserted "public domain photographs are
available" without naming which specific file or what license actually
applies to it — photographs of 3D public-domain objects are not
automatically public domain themselves (the photographer's framing,
lighting, and composition choices are their own copyrightable work,
unlike a straight photographic reproduction of a 2D public-domain
painting). Fetched the specific file's own Commons page directly and
confirmed CC BY 2.5, attributed to Marie-Lan Nguyen — a real, well-known
Wikimedia photographer of museum statuary — before downloading anything.
Attribution is a real license term, not a nicety, so it's rendered as a
visible caption under the image linking to the license text, not buried
in a code comment where only a reader of this repo would ever see it.

### Looking at the source before deciding how to use it

Downloaded the actual full photo and looked at it rather than assuming
"statue photo" meant "usable as-is." It's full nudity — normal and
expected for classical statuary, not appropriate for a product hero
image. Cropped to head/shoulders/extended-arm, which conveniently also
happens to be exactly the region needed anyway (the raised/extended hand
holding whatever will become the controller) — the crop decision served
two real constraints at once rather than being purely a workaround.

### Compositing with Pillow, and two more real bugs caught by looking

Built `frontend/scripts/compose-apollo-hero.py` (checked into the repo,
not left in scratch) to draw the headphones and controller directly onto
the photo with Pillow's `ImageDraw`, rather than trying to align live
HTML/CSS overlay elements against a photo's pixel coordinates across
every breakpoint — baking the composite into one static asset is far
more robust than keeping the alignment problem live in the browser.

Two real bugs, both caught by rendering and inspecting the actual output,
not by re-reading the script:

1. The controller rendered clipped off the right edge of the frame — the
   hand sits near the original photo's own right edge, and the rotated
   controller needed more canvas than that edge left available. Fixed by
   padding the canvas with transparent space before compositing anything
   onto it, rather than shrinking the controller to fit an edge that
   shouldn't have been the constraint in the first place.
2. After fixing that, a large solid black rectangle appeared where the
   transparent padding should have been. The fade-mask code computed a
   soft-edge gradient and then assigned it as the image's *entire* new
   alpha channel — silently discarding the padding's own alpha=0
   transparency and replacing it with the fade mask's default opaque
   value everywhere the fade gradient hadn't explicitly touched. Fixed by
   combining the fade mask with the pre-existing alpha channel via
   `ImageChops.darker` (a per-pixel minimum) instead of overwriting it —
   verified afterward with actual pixel-value checks (`getpixel` at
   several coordinates in the padding region), not just a visual glance,
   since the first "it looks transparent now" impression could easily
   have been the image viewer's own transparency rendering rather than
   confirmation the file was actually correct.

### Reproducibility over a one-off script

The compositing script downloads its own source photo on demand and
caches it gitignored rather than committing a ~5MB JPEG to the repo, and
lives in `frontend/scripts/` rather than a session-local scratch
directory — re-run from a clean checkout, it produces a byte-identical
`public/apollo-hero.webp`, verified for real by deleting the cached
download and re-running before calling this done.

### Verification

`npm run lint`, `tsc --noEmit`, `npm run build` all clean; Playwright
confirms zero console errors and zero horizontal overflow on both routes,
desktop and mobile, plus a real `/ask` round trip. Backend untouched, 76
tests still green.

### Open questions (new)

- **This is the first and only photographic asset in the app.** Superseded
  by Slice 15: the photo, and the whole identity around it, was deleted
  outright a few turns later on direct instruction to start over. Kept
  here as the real record of a decision that turned out to be temporary,
  not a mistake to erase from the log.
- **No `next.config.ts` image domain/remotePatterns change was needed**
  since the asset lives in `public/` rather than being loaded from a
  remote URL at runtime — moot now that the asset is gone, kept for the
  same reason as above.

---

## Slice 15 — The catalog is the identity, and a real bug in an LLM's own prose

**Date:** 2026-08-28

Two unrelated things landed back to back: the user installed a batch of
newly available Claude Code skills (superpowers, frontend-design, and
others), then immediately asked for a full landing page rebuild. Worth
treating that sequencing as real: this was the first task run through
`superpowers:using-superpowers` -> `superpowers:brainstorming` ->
`example-skills:frontend-design`, not because a skill happened to exist
for it, but because the rule those skills state plainly ("if a skill
might apply, invoke it, before any response") leaves no room to skip the
process just because six full re-skins already happened this session
without one.

### What the brainstorming gate actually bought

Classified the request as Bounded (an existing page's presentation, not a
new subsystem) and asked exactly one question before proposing anything:
does "completely new" mean rebuild the execution around the current
identity, or discard it and start over. The answer ("start over") is the
kind of fork that's expensive to guess wrong on after four prior re-skins
already spent that exact goodwill; asking once, concretely, cost nothing
and removed the single biggest risk in the whole task before any code
was written.

### The concept: stop illustrating the product, show it

Every prior direction (3D shapes, gradient blobs, a Roman statue) put a
decorative visual in front of Ludo rather than using anything Ludo
actually has. The frontend-design skill's own calibration section names
exactly the trap this session kept finding on its own the hard way:
AI-generated design defaults cluster around a few looks regardless of
subject. The fix here was to ground the hero in the one thing genuinely
specific to this product: its own real catalog. `CatalogField.tsx` plots
100 real games, live, fetched from the same endpoint the browse page
already uses, colored by real genre, sized by real popularity. Paired
with a real, named aesthetic risk: zero decorative accent color anywhere
in the chrome, so the only color on the entire site is real data. That's
the kind of choice the frontend-design skill calls "spend your boldness
in one place" made concrete rather than aspirational.

### A real Motion bug, caught immediately by the fresh Playwright pass

First render of `CatalogField` threw `<circle> attribute cy: Expected
length, "undefined"` on every single point. Root cause: `cy={p.y}` as a
plain JSX attribute alongside `cy: [...]` inside Motion's `animate`
object gave Motion nothing to interpolate from at the start of the
animation. Same family of bug as Slice 14b's controller-in-the-wrong-
place issue: something that looks structurally fine (a value defined
twice, once statically and once animated) actually leaves an undefined
gap Motion can't fill in. Fixed by letting Motion own `cy` completely,
defined in both `initial` and every branch of `animate`, never as a bare
prop.

Second, more interesting problem, found by looking at the actual render
rather than the code: a lot of real games share an exact price ($0,
$19.99), so the first working version showed hard vertical stripes of
stacked circles instead of a scatter. Fixed with a deterministic hash of
each game's own name for jitter, specifically not `Math.random()` during
render, a rule this project has held since Slice 9g's React-Compiler-era
fixes, and specifically stable across re-renders rather than reshuffling
positions every time.

### The dash constraint, and where it actually needed to be fixed

"No em dashes or en dashes" arrived as a hard constraint mid-slice. Fixed
every hardcoded string across the frontend, grep-verified rather than
spot-checked. But the real discovery came from actually running the
`/ask` flow after the rebuild: the live model's own answer came back with
a genuine em dash ("Aseprite – review score...") despite a new rule added
to the system prompt asking it not to use one. Worth stating plainly,
since it's a real, generalizable lesson: an instruction to an LLM changes
the distribution of what it outputs, it does not guarantee any specific
output. A constraint that has to hold for every response needs a
deterministic check on the output, not just a request in the prompt.

`_strip_dashes()` in `src/agent/graph.py` is that deterministic check,
applied on the one code path both `run_agent()` and `stream_agent()`
already funnel every route through, so the guarantee covers the API, the
MCP server, and evals without needing three copies of the same rule.
Found and fixed a real bug in this function before it shipped, the same
day it was written: the first regex (`\s*[—–]\s*`) matched a dash with
*zero* surrounding whitespace just as readily as one with real spaces
around it, so a tight numeric range like "10–20" was silently turned into
"10, 20", a list of two numbers instead of a range, changing what the
text actually meant. Caught by testing the fix against a real range
string, not just the original spaced-dash bug report, before considering
it done. Fixed by requiring real whitespace (`\s+`) before treating a
dash as a parenthetical aside; a bare dash falls through to a plain
hyphen instead, which preserves the range's meaning.

While verifying the live fix against the actual running backend, found a
third, related character in the same real response: U+2011, a
"non-breaking hyphen" that renders identically to a normal hyphen but is
a distinct Unicode codepoint, in "highest‑rated". Not literally an em or
en dash, and the user never named it, but it's the identical class of
problem: the model reaching for an unusual punctuation mark instead of
the plain ASCII one. Normalized it too, on the reasoning that the user's
actual intent ("plain, ordinary punctuation, not typographic LLM tics")
extends past the two examples they happened to name.

### Verification, run through superpowers' own gate

Invoked `superpowers:verification-before-completion` before making any
claim of done, which is stricter than this project's own established
habit of checking before wrapping up: every command (lint, `tsc`, build,
pytest, the dash grep, Playwright, a live curl against the running
backend) was re-run fresh in the same turn as the completion claim, not
reused from an earlier check in the conversation. All green: lint and
`tsc` clean, build exits successfully with both routes prerendering,
83 backend tests passing (6 new, covering `_strip_dashes()` directly,
including the range-preservation case and the non-breaking-hyphen case),
zero dash characters anywhere in rendered frontend strings, zero console
errors or horizontal overflow on both routes at both viewport sizes, and
a live model response confirmed to contain none of the three dash
characters.

### Open questions (new)

- **The hero's real-data scatter is sampled, not exhaustive.** It plots
  100 games (the top 100 by peak concurrent players, the same query the
  catalog page's default sort already uses), not all 1,000. That's a
  deliberate performance/legibility tradeoff, not an oversight, but it
  does mean the visual skews toward already-popular games rather than a
  uniform random sample of the whole catalog. Superseded by Slice 16:
  the scatter itself was replaced with a film strip of cover art a few
  turns later on direct "I dont like it" feedback, so this specific
  concern no longer applies, kept here as the real record.
- **Checked, not just assumed: `stream_agent()`'s progress events are
  node-completion markers (`stream_mode="updates"`), never token-level
  answer text.** The first draft of this entry claimed the sanitizer
  might momentarily miss a raw dash during live streaming; re-read the
  actual generator before leaving that claim in and it's not true, the
  frontend never sees the answer text until the one final, already-
  sanitized `AgentResult`. No real gap here, worth having verified rather
  than guessed.

---

## Slice 16 — Real feedback, real questions, a real palette lesson

**Date:** 2026-08-28

"I dont like it," aimed squarely at Slice 15's strict monochrome-plus-
live-data-scatter identity, arrived with four concrete replacement asks
bundled into one message: a Latin/Roman touch, a dark green background,
a sliding film-strip animation of real game covers, and different genre
icons and colors. Worth taking the rejection at face value rather than
defending the previous pass's reasoning: the scatter plot's honesty
(real data, no decoration) didn't matter if the result read as sparse
and cold rather than considered.

### Three questions, not four guesses

Ran this through `superpowers:brainstorming` again, the second time this
session the newly installed skill governed a design task, and asked
three targeted questions before touching any code: hotlink real Steam
cover art or fall back to placeholders, does the film strip replace the
scatter as the hero or run alongside it, and how far should "Roman
touch" go now that the full identity (aurora, statue, laurel icons) was
deleted outright one slice ago. All three of the biggest open forks in
this request got resolved with a nod each rather than an assumption,
which mattered more here than usual: getting "how much Roman" wrong in
either direction (too little reads as ignoring the feedback, too much
re-imports the exact system that was deliberately deleted last round)
would have wasted the whole turn.

### Real cover art, sourced the same way the Apollo photo was

Steam serves each game's own official cover art from a predictable CDN
path (`library_600x900.jpg`, the same asset the game's own store page
uses). Verified the URL was actually live with a plain curl before
writing any code that depended on it, the same discipline applied to
the Apollo photo's license two slices ago: check the actual thing
works, don't assume a URL pattern remembered from training data is still
current. This is the first *remote* image this app has ever loaded
(the Apollo photo was downloaded and committed; this one is hotlinked
live), which is a genuinely different reliability posture, if Steam
changes this path the strip degrades rather than breaks outright, since
each cover's `onError` handler drops it from the strip instead of
showing a broken image.

Getting the real cover art required a small, honest addition to the
data layer: `appid` wasn't previously exposed by `GET /catalog` because
nothing needed it yet. Added it to `_CATALOG_COLUMNS` rather than
building a separate endpoint, since the catalog page's existing rows
already carry everything else the film strip needs (name, for the
title attribute).

### The film strip's frame, built from real film mechanics

The sprocket-hole border isn't a repeated set of DOM elements, it's a
single `repeating radial-gradient` background tuned to the actual
spacing of real 35mm film perforations, which tiles automatically
regardless of the strip's rendered width. The double-rule bronze border
is the one Roman-adjacent touch on the frame; the genre icons' new
engraved-medallion ring (see below) is the other, deliberately two
small, specific accents rather than a reintroduction of the whole
system.

### A real, generalizable lesson from the categorical palette redo

The user asked directly for different genre colors, so this couldn't be
the existing validated default carried forward. Drafted an "earthy,
antique" 8-hue set first (rust, teal, gold, wine, olive, and similar)
to match the Roman mood the rest of this slice was reaching for, ran it
through the dataviz skill's actual validator against this app's own new
dark-green background rather than eyeballing it, and it failed
repeatedly. Reordering the same eight hues to separate the worst
adjacent pairs just moved the failure to a different pair. The real
lesson, worth generalizing past this one palette: eight hues clustered
in a narrow slice of the hue wheel structurally cannot clear adjacent-
pair separation floors no matter how they're ordered, because ordering
only changes *which* pairs are adjacent, not how close together the
hues are to begin with. Fixed by spreading the eight hues across the
full wheel (one blue, one orange, one teal, one gold, one magenta, one
green, one violet, one red) the same structural shape the dataviz
skill's own reference palette uses, just with different specific shades
so it reads as genuinely new rather than the same defaults restored.
That set passed every check on the first real attempt once the
structural constraint was respected instead of fought.

### Bringing the accent back, named as a reversal

Slice 15's "zero decorative accent color" was a real, deliberate risk
that didn't pay off. Reversing it isn't a quiet value swap: every
component that had been flattened to pure monochrome (the Ask button,
route/cached pills, the forecast's projected number, markdown bold and
links, the trace stepper) needed its accent usage restored individually,
since Slice 15 had rewritten each one to read foreground/muted directly
rather than through a variable that could be redefined once. Worth
naming that cost plainly: a "no accent" design choice isn't free to
reverse later if it was implemented by replacing every accent reference
rather than pointing them all at a token that happened to equal the
background at the time.

### Verification

Same fresh-in-this-turn bar as Slice 15's own close: `npm run lint`,
`tsc --noEmit`, `npm run build` all clean; backend's 83 tests untouched
and green; the dash grep re-run and still zero hits outside comments;
Playwright confirms zero console errors (including from the 80 remote
image requests the film strip's duplicated loop makes) and zero
horizontal overflow on both routes, desktop and mobile; a real `/ask`
round trip confirms the restored accent actually reaches the live
result view.

### Open questions (new)

- **The film strip's image requests (now 120: 60 real games, doubled for
  the seamless loop, up from 40/80 in the first draft) all fire on mount
  with no `priority` staggering.** No console errors or warnings
  surfaced from this in testing, but it hasn't been checked under a
  throttled/slow network condition, where a visible pop-in as covers
  load could look rougher than on a fast local connection.
- **Steam's CDN path for cover art isn't officially documented as a
  stable public API** — it's the same path the store page's own
  frontend uses, verified live before wiring this up, but Valve could
  change it without notice. No fallback exists beyond the per-cover
  `onError` drop; a Steam-side path change would just thin out the
  strip, not break the page, but worth knowing this dependency exists
  outside this project's own control.

---

## Slice 17 — A pausable marquee, done the boring reliable way

**Date:** 2026-08-29

Direct, unambiguous positive feedback on Slice 16 for the first time in a
while ("this a much bigger improvement thank you!"), followed immediately
by four concrete refinements rather than a rejection: widen the film
strip to full-bleed, a new font ("professional but also gamer like"), a
neon green hue, and a hover animation on the strip. Worth noting the
tone shift plainly: this is the first round this whole redesign arc
where the starting point was "keep this, make it better" rather than
"start over."

### Choosing Rajdhani for a specific, checkable reason

"Professional but also gamer like" is a real brief, not a vibe: it rules
out both a generic corporate sans (not gamer enough) and a novelty
pixel/display face (not professional enough, and this project has
rejected exactly that category of choice since Slice 9g). Rajdhani sits
in the specific middle this brief describes: squarish, technical
letterforms with real lineage in esports and gaming-HUD branding, but
regular/medium weights clean enough to still read as body copy rather
than a logo. Swapped in as the only sans face (still one face, no second
display font, the discipline Slice 15 settled on), not layered alongside
the existing one.

### The hover-pause, and why it's plain CSS instead of Motion

A sliding infinite loop needs to pause on hover so a viewer can actually
look at a cover, otherwise "hover animation" would just mean a bigger
visual jump when the mouse arrives. The film strip's slide was already
implemented with Motion's `animate` prop driving a repeating keyframe
tween. Stopping and restarting that same tween on hover risks a visible
jump: calling `.start()` again on a stopped keyframe animation
re-interpolates from the current value toward the *same keyframe list*,
which is not guaranteed to read as "frozen, then resumed," depending on
how Motion re-anchors the timeline. Rather than fight that, moved the
slide itself to a native CSS `@keyframes` animation
(`.filmstrip-track` in globals.css) specifically because
`animation-play-state: paused` is a browser primitive built exactly for
this: freeze in place, resume from the exact frozen position, no
interpolation math required. Verified this wasn't just a plausible
theory: read the track's own computed `transform` before, during, and
after a hover, and confirmed it changes while unhovered, stays bit-for-
bit identical for the whole hover duration, and `animationPlayState`
reads `paused` the entire time.

The per-cover scale-and-glow effect layered on top *is* Motion
(`whileHover` on each individual cover), and the two don't conflict:
the parent's CSS `transform: translateX` and each child's independently-
computed Motion `transform: scale(...)` apply at different points in the
CSS cascade/composition and don't fight for the same property on the
same element.

### Neon green, picked with the collision already in mind

The genre categorical palette already has a green (slot 6, a muted
olive-green, picked in Slice 16's validator pass). Making the site's one
UI accent *also* green risks the two reading as the same color doing two
different jobs, chrome versus data identity, coincidentally colliding.
Picked the neon accent (`#39ff88`) deliberately far more saturated and
bright than genre slot 6 rather than checking after the fact, so a
"glowing neon accent" and a "muted natural green data point" stay
visually distinct in the same view.

### Verification

Same bar as every prior slice: `npm run lint`, `tsc --noEmit`, `npm run
build` all clean; backend's 83 tests untouched and green; Playwright
confirms zero console errors and zero horizontal overflow on both
routes, desktop and mobile; a real `/ask` round trip stayed clean. The
hover-pause claim specifically got the programmatic check described
above, not just a screenshot that happened to show a scaled cover.

### Open questions (new)

- **Rajdhani's condensed, geometric letterforms haven't been checked
  against very long real answer text at small sizes** (the actual
  `/ask` result panel's body copy) — the hero headline and short UI
  labels read fine, but a long multi-sentence LLM answer at `text-sm` in
  a squarer face than Geist hasn't had a dedicated legibility pass beyond
  "it built and rendered without errors."
- **Neon green at full saturation on every accent use (buttons, links,
  the trace stepper, markdown bold) hasn't been checked for eye strain
  over extended reading** — fine for short labels and single words, not
  yet judged for a long markdown answer where multiple bolded terms in
  one paragraph would all carry the same bright accent.

## Slice 18 — A cartridge popup, and why the shared-layout morph got dropped before it was written

**Date:** 2026-08-29

Screenshot feedback on the catalog page's genre filter ("382 games")
turned out not to be a bug at all: the genre dropdown was set to
Adventure, and 382 is the real, correct count for that one genre inside
the 1,000-game catalog, not a data problem. Worth recording because it's
the first time this session a screenshot prompted an investigation that
resolved to "working as intended, here's why" rather than a fix.

The same message carried a real feature request: click a film-strip
cover to see that game's info in a popup styled like a video game
cartridge, close it back to the strip with an animation, and along the
way speed up the slide and drop the strip's side borders.

### Why the `layoutId` shared-element morph got dropped before any code was written

The initial pitch (presented to the user, not yet re-confirmed before
implementation started) described the cover art morphing from its
film-strip position into the cartridge via Motion's shared-layout
animation, `layoutId`. Before writing that, re-examined `FilmStrip.tsx`
and remembered its list is deliberately duplicated for the seamless
loop: `const strip = [...visible, ...visible]`. That means, at any
instant, two DOM nodes exist for the same `appid`, both mid-slide, both
candidates for the same `layoutId`. Motion's shared-layout animation
needs a single unambiguous source element to morph from; with two live
nodes sharing an id, which one it treats as "the" source is undefined
behavior, not a design choice this app makes on purpose. Rather than
give the strip's rendering scheme (duplicated list) more entanglement
with the modal's rendering scheme (single game), dropped `layoutId`
entirely and built a plain scale-and-fade `AnimatePresence` pop instead
(`GameCartridge.tsx`): it reads as "popping out" without depending on
which of two identical DOM nodes triggered it. This is the kind of
mid-task correction the brainstorming skill's design step exists to
catch, it happened here purely from re-reading the file being changed,
not from anything going wrong at runtime.

### One card, no second network call

Every field `GameCartridge` shows (cover, name, genre, release date,
Metacritic, platforms, price, review score, owners, peak players) is
already present on the `CatalogGame` object the film strip fetched to
build its cover list. Opening the cartridge is therefore a pure client-
side state change (`selected: CatalogGame | null`), not a new request to
the backend.

### Extracting `lib/formatGame.ts`

`CatalogClient.tsx` already had `formatDate`/`formatOwners`/
`formatPrice`/`formatPlatforms` as private local functions for its
table. The cartridge needed the exact same formatting for the exact
same fields. Rather than let a second copy exist and drift (a null
price rendering differently in the table than in the cartridge, say),
pulled all four into `lib/formatGame.ts` and pointed both consumers at
it. `CatalogClient.tsx`'s own `PlatformBadges` component was retired in
the same pass since `formatPlatforms` returns a plain string now, not
JSX; the table cell just calls it directly.

### Pausing the strip while the cartridge is open

The Slice 17 hover-pause (`animation-play-state: paused` on `:hover`)
only fires while the pointer is still over the strip, which usually
isn't true once someone has clicked a cover and is reading the
cartridge with their mouse elsewhere. Added a second, independent path
to the same CSS property: an inline `animationPlayState: selected ?
"paused" : undefined` alongside the existing `:hover` rule, so the
slide is guaranteed frozen for the entire time a cartridge is open
regardless of pointer position, and resumes the instant it closes.
Verified this the same way Slice 17 verified the hover case: read the
track's computed `animationPlayState` immediately before opening
(`running`), while open (`paused`), and immediately after closing
(`running` again) rather than trusting the CSS rule to behave as
intended.

### Speed and border, the two small direct requests

The slide's duration multiplier dropped from `visible.length * 2.2`
seconds to `* 1.2`. The strip's frame border, previously `3px double
var(--accent)` on all four sides, is now `borderTop`/`borderBottom`
only, with no left/right rule, matching the direct ask for "an upper
and bottom border, however on the side nothing."

### Verification

`npm run lint`, `tsc --noEmit`, `npm run build` all clean; backend's 83
tests untouched and green (this slice is frontend-only). Playwright,
run twice: once confirming the border computes to 3px top/bottom and
0px left/right, a click opens the dialog, the X button closes it, and
`animationPlayState` reads `paused` while open and `running` after
close, at both desktop and mobile viewports with zero console errors
and zero horizontal overflow on `/` and `/catalog`; a second pass
specifically isolating the two other close paths (Escape and a backdrop
click), since "click X on it" was the only close path spelled out
explicitly and the other two were part of the design, not just an
implementation detail. Grepped every new and touched file for em
dashes, en dashes, and non-breaking hyphens; none found outside code
comments.

## Slice 19 — Actually deployed, and two real gaps the deploy itself surfaced

**Date:** 2026-08-29

Everything up to this point had been verified against `localhost` only.
Asked directly "how do I deploy this, and do I still need to run
uvicorn/npm run dev myself" — the answer to the second half is no, once
deployed the platforms run the servers continuously on their own
infrastructure, the local dev commands become a pure local-dev loop.

### Correcting the repo's own advice before following it

The README's Deploying section (written back in Slice 6) recommends
"Render or Fly.io" interchangeably. Rather than trust that unchanged,
checked current terms first, since hosting free tiers are exactly the
kind of fact that goes stale: Fly.io removed its free tier for new
accounts back in 2024 (a 2 VM hour/7 day trial, then a card is
required), while Render's free web service tier still needs no card at
all. Used Render only. Same instinct as verifying the Steam CDN URL
with a live curl before hotlinking it in Slice 16, applied to a
pricing claim instead of a technical one.

### The Health Check Path trap

Render's own new-service form shows `/healthz` as greyed placeholder
text in the Health Check Path field — common convention on other
platforms, but not this app's actual route. The real endpoint is
`GET /health` (`src/api/main.py`). Typing the placeholder literally,
rather than reading it as an example, would have pointed Render's
health monitor at a 404 and likely marked the service unhealthy despite
it actually running fine.

### Build Filters, added because most commits in this repo are irrelevant to the backend

Left unset, every commit to `master` triggers a full Render rebuild,
including a full Docker build and the ~25-minute SteamSpy re-ingestion
step. Looking at this project's own commit history, the overwhelming
majority of commits are frontend-only or documentation-only (this
project's own habit of updating PLAN/DOCEXP/README after every slice).
Added Ignored Paths (`frontend/**`, `*.md`, `.github/**`) so those
commits redeploy the frontend on Vercel (fast, no ingestion involved)
without also triggering a wasted 30-minute backend rebuild that changes
nothing. Chose an ignore-list over an allow-list deliberately: a
forgotten future backend path still deploys correctly under an
ignore-list (fails safe, just an occasional unnecessary rebuild); under
an allow-list, a forgotten path means a real change silently never
ships (fails unsafe, and is a much worse thing to debug later).

### Two real gaps the deploy step itself surfaced, not hypothesized in advance

**Vercel offered to import 26 environment variables it detected across
the whole repo, not just `frontend/`.** These came from `.env.example`
(root, ~25 backend-only vars) plus `frontend/.env.local.example` (the
one real one). All 25 backend vars had empty values, since the actual
`.env` with real values is gitignored and was never visible to Vercel's
GitHub-based import, so nothing sensitive was ever at risk, but
importing them all would have left 25 dead, confusing entries sitting
in a frontend project that never reads any of them. Removed all but
`NEXT_PUBLIC_API_BASE_URL`.

**`CORS_ALLOWED_ORIGINS` was missing entirely from Render's environment,
not merely left at a default value on purpose.** The backend had been
set up via Render's "Add from .env" shortcut against the real local
`.env` file — and that file, unlike `.env.example`, had never actually
included a `CORS_ALLOWED_ORIGINS` line (or `DEBUG`, or the rate-limit/
cache tuning vars — those simply hadn't been needed for local dev,
where the code's built-in defaults already matched what local dev
wanted). Because `src/config.py`'s `Settings` class gives
`cors_allowed_origins` a default of `http://localhost:3000`, the
missing variable didn't cause an error or a crash, which is exactly
what made it easy to miss: the service came up green, `/health` was
fine, and the only visible symptom was the deployed frontend's fetches
failing with a CORS rejection once Vercel was live. Added the variable
explicitly rather than relying on the default resolving correctly by
coincidence.

Picked "Save and deploy" over "Save, rebuild, and deploy" for this
fix specifically: `CORS_ALLOWED_ORIGINS` is read at container startup,
not baked into the image at build time, so a full Docker rebuild (and
therefore a second ~25-minute re-ingestion) would have been pure waste
for a change that only needs a container restart with a new
environment variable.

### Verification, against the live URLs this time, not localhost

`curl https://ai-game-analyst-api.onrender.com/health` confirmed
`db_exists: true` in production, not just that the container booted.
An `OPTIONS` preflight against `/ask` with a real `Origin:
https://full-stack-project-sepia-nine.vercel.app` header confirmed the
response's `access-control-allow-origin` echoes that exact URL, not a
wildcard or a stale `localhost` value left over from before the fix.
Playwright driven against the live Vercel URL (not localhost)
confirmed real Steam cover art loads, a click opens the cartridge and
Escape closes it, `/catalog` renders real rows, and zero console errors
or horizontal overflow at both desktop and mobile viewports. A live
`POST /ask` against the deployed Render backend, not a local one,
returned a correct route classification (`analysis`), correct SQL, and
a correctly formatted markdown-table answer with no em/en dashes. This
is the first slice where "verified" means verified against the actual
public URLs a stranger would hit, not against a machine only I can
reach.

### Open questions (new)

- **No live-URL smoke test runs automatically.** Everything above was
  checked by hand, once, right after deploying. A future code change
  could break the live site (a bad env var, a CORS regression, a
  broken build) with nothing catching it until someone happens to
  visit and notice — there's no scheduled or post-deploy check hitting
  the real URLs the way `test.yml` hits the test suite on every push.
- **The Render free tier's 15-minute spin-down is real and unverified
  under a genuinely cold start.** Every check in this slice happened
  while the service was already warm from the deploy itself; the
  actual 30-60 second cold-start delay after real inactivity hasn't
  been observed directly yet.

## Slice 20 — The answer was correct and completely invisible

**Date:** 2026-08-29

Live-site usage surfaced the first real UX bug found by actually using
the deployed product rather than by reading code: asking a question
gave no visible feedback near the input, and the answer rendered
somewhere off-screen. Worth noting the sequence, since it started as a
misread: while investigating, a Render log full of `huggingface.co`
HEAD requests and an orange `HF_TOKEN` warning looked alarming at
first glance, but turned out to be completely normal (a cold-start
embedding-model cache verification, not a failure — no exception, no
non-2xx from the app itself, and the screenshot alongside it showed the
question had actually answered correctly). Worth recording as a small
lesson on its own: color-coded log severity (Render highlights WARNING
lines in orange) is not the same signal as "something is broken," and
reading the actual HTTP status codes and outcome mattered more than
the visual alarm color.

### The real bug, confirmed by reading the code, not just the report

`app/page.tsx`'s layout put the ask bar near the top, then rendered
`MeetLudo` and the genre showcase in full below it, and only after
both of those did the progress trace and result actually appear.
Clicking Ask changed the button label to "Asking…" and nothing else
visible happened until the user scrolled past two entire sections that
have nothing to do with the question they just asked.

### The fix, and why it needed more than moving one `<div>`

Relocating the trace/result block to sit directly under the ask bar's
example-question pills solves the problem for the main input, but this
app has two other entry points that call the exact same `ask()`
function from much further down the page: Meet Ludo's example
questions and the genre showcase's leaderboard picks. Moving the
result block up without anything else would have just relocated the
bug rather than fixed it for those two paths — triggering a question
from the bottom of the page would then show the answer scrolled far
above the click, same underlying problem, opposite direction. Added a
ref-anchored `scrollIntoView({ behavior: "smooth" })` inside `ask()`
itself, so every entry point converges on the same fix regardless of
where the click originated. Also added an explicit "Ludo is
thinking…" label above the trace dots — the dots alone (five grey
circles, one pulsing) read as decoration at a glance if you don't
already know what they mean; a plain-language label removes that
ambiguity for a first-time visitor.

### The chat feature: recommended against building it as scoped, not silently dropped

The same feedback asked for a bigger change: a ChatGPT-style
multi-turn thread, so a person could ask a follow-up like "compare
that to Elden Ring" and have the agent understand what "that" refers
to. Looked at this honestly rather than defaulting to "sure, more
features": the specific use case named, comparing two games, is
already served today by the existing stateless `run_stats`
`compare_two_groups` tool inside a single question (one of this app's
own example questions already exercises it). Building real
conversational memory to serve that same use case would mean teaching
the router and prompts to resolve cross-turn references, deciding
whether follow-up context should influence SQL generation and how that
interacts with the SQL-safety guardrails, and building genuinely new
frontend state — a real architectural undertaking, not a UI tweak.

Recommended against it specifically because of what it would trade
away, not just its size: this project's actual differentiator among
portfolio projects a technical reviewer will have seen many of isn't
"it's a chatbot" — it's the guardrails (guaranteed SELECT-only SQL, a
router that gates tool access, real statistics instead of an LLM
eyeballing an average, evals with live-computed ground truth). Adding
cross-turn context specifically increases the surface area that safety
story has to cover, for a feature whose stated use case the stateless
pipeline already handles. If full conversational memory is wanted
later specifically to demonstrate that skill for a job application,
that is a legitimate reason on its own — but it deserves its own
brainstorm and its own slice, not to be folded into a loading-state
fix.

### Verification

`tsc --noEmit`, `npm run lint`, `npm run build` all clean. Started a
real local backend (not mocked) and drove the actual UI with
Playwright: confirmed "Ludo is thinking" and the trace panel are both
inside the viewport immediately after clicking Ask, with zero
scrolling, at both desktop and mobile widths; a second run clicked an
example question rendered well down the page and confirmed the page
auto-scrolled and the trace panel became visible, proving the fix
covers all three entry points, not just the one that was reported.
Zero console errors in either run. Grepped the new copy for em dashes,
en dashes, and non-breaking hyphens — the only new punctuation is a
plain ellipsis ("Ludo is thinking…"), not a dash, and none were found.

### Open questions (new)

- **Full multi-turn conversational memory remains unbuilt, on
  purpose** — see the reasoning above. Worth revisiting only if there's
  a specific reason (a target role's job description calling out
  multi-turn agent memory, for instance) that outweighs the guardrail-
  surface-area cost, and it should get its own brainstorming pass
  rather than being added incrementally.

## Slice 21 — Portfolio/hygiene cleanup pass

**Date:** 2026-08-29

A deliberate stock-take rather than a feature: after Slice 19's deploy
and Slice 20's UX fix, took a step back and listed everything still
open before doing more feature work. Two real, self-inflicted findings
came out of just checking git state honestly:

**`render.yaml` wasn't covered by Render's Ignored Paths, and it
mattered immediately, not hypothetically.** The Slice 19 doc-fix commit
touched `render.yaml` (correcting its stale Fly.io comment and
placeholder CORS URL), and because that file wasn't on the ignore
list, it almost certainly triggered a real, wasted Render rebuild —
caught by checking `git show --stat` on the pushed commit, not by
guessing. Added it to the ignore list.

**Dead 3D dependencies were still costing install/bundle weight for
zero functional benefit.** `@react-three/fiber`, `@react-three/drei`,
`three`, and `@types/three` survived Slice 14's WebGL statue hero being
deleted, sitting unused in `package.json` since. Confirmed zero imports
across `app/`, `components/`, `lib/` before removing anything (same
discipline as every other deletion in this project's history — grep
first, never assume). `npm install` afterward dropped 54 packages;
`tsc --noEmit`/`lint`/`build` all still clean, and the build's own
compile step measurably faster (1.3s vs 2.7s) with less to bundle.

### Adding a LICENSE, and being explicit about the name on it

The repo had no LICENSE file at all — a real gap for a public portfolio
project, since without one the legal default is "all rights reserved,"
which is a strange stance for something meant to be read and judged by
strangers. Added MIT, the conventional default for exactly this kind of
project. Used the GitHub-facing identity (`stefhooy`, the name every
commit in this repo is already authored under) as the copyright holder
rather than inserting a real legal name unasked — easy to swap for a
different name if a real one is preferred.

### `frontend-ci.yml`: why it exists and what it actually catches

Every frontend verification in this project's entire history, across
every one of the last several slices, was a manual pass: run `tsc`,
run `lint`, run `build`, run Playwright, by hand, in the same
conversation turn as the change. That's real verification in the
moment, but it proves nothing about the *next* commit. Nothing before
this slice would have caught a future change that broke the frontend
build, introduced a type error, or failed lint, unless someone happened
to run those commands again by hand. `test.yml` already covers exactly
this need for the backend (pytest on every push/PR); `frontend-ci.yml`
is the same idea for the frontend, deliberately mirroring `test.yml`'s
structure and commenting style rather than inventing a different
pattern for the same job.

Scoped to `frontend/**` via `paths:`, for the identical reason Render's
Ignored Paths exist: a backend-only or docs-only commit has no way to
break the frontend build, so re-running it on every unrelated push
would just be wasted Actions minutes for zero additional signal.

Verified the workflow's three steps would actually pass, not just that
the YAML parses: killed lingering local Node processes holding a file
lock, ran a genuinely fresh `npm ci` (not a warm `npm install`, to
match what a CI runner actually does), then ran `lint`, `tsc --noEmit`,
and `build` in that exact order against that fresh install. All three
passed. `NEXT_PUBLIC_API_BASE_URL` is set to a harmless placeholder in
the workflow env — `lib/api.ts` already falls back to
`http://localhost:8000` when the variable is unset, so this is
belt-and-suspenders explicitness rather than a functional requirement;
the build never calls the real backend regardless; it only proves
Next.js's static generation and type checking succeed.

### ruff + mypy for the backend, and fixing what they actually found

Backend static analysis was the next item down Slice 21's list. Chose a
moderate ruff rule set (`E, F, I, UP, B, BLE, RUF`) rather than every
plugin ruff ships: `E/F` is the real correctness baseline, `I` sorts
imports, `UP` modernizes for the 3.12 target, `B` (bugbear) catches
real bug shapes, `RUF` is ruff's own checks. `BLE` (blind-except)
specifically because `src/api/main.py` already had a `# noqa: BLE001`
comment on a broad `except Exception:` before this config existed —
the codebase was already anticipating this exact rule, so enabling it
just made an intent that was already written down actually enforced.
Left `DTZ` (flake8-datetimez, "always pass tzinfo") out on purpose: this
codebase has one consistent convention instead of an oversight —
timestamps are written UTC-aware once, at the single point they're
created (`poll_player_counts.py`), and read back naive everywhere else
because that's genuinely what DuckDB's Python API returns for a
TIMESTAMP column. Enabling DTZ would have flagged the tool and its
tests as buggy for correctly matching their one real data source.

For mypy, deliberately not `--strict`: LangChain, LangGraph, and DuckDB
don't ship complete type stubs, and strict mode on day one would mean
either a wall of blanket `# type: ignore` at every library boundary or
hours spent stub-hunting instead of catching real mistakes. Picked a
level that catches real bugs (`check_untyped_defs`, `warn_unused_ignores`,
`warn_redundant_casts`) while tolerating untyped third-party surfaces
(`ignore_missing_imports`), with room to tighten later once those
boundaries get individually reviewed instead of ignored wholesale.

**What actually turned up, and why every finding got fixed rather than
suppressed:** 33 ruff issues and 13 mypy issues, all real, none of them
"turn off the rule and move on." The two categories worth remembering:

- **Ambiguous-unicode false positives in exactly the two files that are
  supposed to contain literal em/en dashes** (`graph.py`'s
  `_strip_dashes()` and its test) — the one case in this whole pass
  where the right fix genuinely was a per-file ignore, since the
  characters ruff flagged are the entire point of that code, not stray
  punctuation. Scoped the ignore to those two files specifically, not
  project-wide, so the rule still does its job everywhere else.
- **Four functions typed `list[list]` when their only real caller
  (`run_guarded_query`, DuckDB's own `.fetchall()`) actually returns
  `list[tuple]`.** This was a real, if harmless-in-practice, type
  mismatch that would have let a genuine bug (accidentally relying on
  list-only behavior, like item assignment) through undetected.
  Widened the parameter type to `Sequence[Sequence[Any]]` — the
  functions only ever read `rows`, never mutate it, so the more
  permissive, correct type was the actual fix, not a workaround.
- **Two `with_structured_output()` calls** (the router, the eval
  judge) **typed looser than they resolve at runtime.** LangChain's
  stub allows either a `dict` or a `BaseModel` back, since the method
  also accepts a raw JSON-schema dict as its target — but passing a
  Pydantic model class, as both call sites do, always returns an
  instance of that exact class. Used `cast()` with a comment explaining
  why, rather than restructuring working code around a stub's
  generality it doesn't actually need.
- **`ChatGroq`'s `api_key` expects a `SecretStr`, not a plain `str`.**
  A real, if minor, type gap — wrapped it. This one actually touches a
  live code path (every real Groq call), so it got the same live
  verification as the SSE change below, not just a type-checker pass.
- **A duplicated long f-string, not just a long line.** Ruff's line
  length flag on one `yield f"data: {json.dumps(...)}\n\n"` in
  `ask_stream()` led to noticing it was the third near-identical copy
  of the same SSE-framing pattern in that function. Extracted a small
  `_sse()` helper instead of just wrapping the line — a real
  deduplication the line-length check surfaced as a side effect, not
  the fix ruff was actually asking for.

### Verification

`ruff check src/ tests/` and `uv run mypy src/` both clean. Re-ran the
full 83-test pytest suite after every fix landed, not just at the end —
still 83 passed. Because two of the fixes (the `SecretStr` wrapper
around the real Groq API key, the extracted `_sse()` helper inside the
actual `/ask/stream` handler) touch genuine request paths rather than
being pure type-annotation changes, started a real local backend and
drove both `POST /ask` and `POST /ask/stream` with real questions
against the live Groq API afterward — both returned correct answers,
correct SQL, and (for the stream) the expected node-by-node progress
sequence, confirming the refactor didn't silently change runtime
behavior. Also simulated the exact CI workflow order locally (`uv sync
--extra agent --extra dev` fresh, then ruff, then mypy, then pytest, in
that sequence) rather than trusting the YAML would behave the same as
running the commands by hand.

### A small ARCHITECTURE.md accuracy fix, found while checking whether it needed one

Asked directly whether ARCHITECTURE.md needed updating for Slices
18-21's work. It didn't — that document is deliberately scoped to the
reasoning system's shape (the graph, RAG, the safety boundary), not
deployment topology or frontend UI or dev tooling, and none of those
slices touched that shape. But checking it against the real code
surfaced a genuine, pre-existing staleness unrelated to recent work:
the System overview diagram's FastAPI node listed `/ask, /ask/stream,
/genres, /games` without `/catalog`, which has existed for several
slices. Fixed the diagram, and also noticed `catalog.py` — which
bypasses `connection.py` the same way `genre_stats.py` does — wasn't
represented as a box at all. Combined them into one box and rewrote the
explanatory bullet to state their actual, slightly different safety
reasoning accurately rather than imply they're identical: `genre_stats.py`
has no user input at all; `catalog.py` has real user input (search,
sort, genre filter) but never lets it reach a SQL string, routing sort
through a Python allowlist dict and filtering in Python after one
unparameterized fetch instead.

## Slice 22 — A RAG retrieval eval, and confirming the eval itself wasn't fake

**Date:** 2026-08-29

Every eval this project has had so far checks the final answer: is the
number right, is the route right, is the LLM judge satisfied. None of
them check the one step in between — does `retrieve_schema` actually
pull in the right context for a given question. A final answer can
sometimes come out right even when retrieval grabbed the wrong or
incomplete chunks (the LLM might already "know" the schema well enough
from the question itself, or get lucky), so "the answer was correct"
is not the same claim as "retrieval did its job." This slice built a
way to check the second claim directly.

### The free-eval discovery that shaped the design

Before designing anything, read `schema_index.py` and
`schema_corpus.py` rather than assuming how retrieval works. That
surfaced something that changed the whole approach: `SchemaIndex.retrieve()`
only calls `get_embedder()` (the local ONNX model via fastembed) — it
never touches Groq or any paid API. Every other eval in this project
(`run_evals.py`'s golden questions) needs a real LLM call per question,
which is exactly why that harness stays manual-only, cost- and
rate-limit-gated. This one doesn't have that constraint at all. That
single fact moved the whole feature from "another manual report to run
occasionally" to "a real pytest test that gates CI for free, forever."

### Excluding `always_include` chunks from the golden set, on purpose

`schema_corpus.py` marks four chunks (`table:games`, `column:name`,
`column:genre`, `table:player_counts`) as `always_include=True` — they
bypass ranking entirely and come back regardless of the question,
because semantic similarity alone was empirically found (Slice 2/9) to
under-rank generic-sounding columns like "name" against a question
naming a specific game. Including these in the golden set's expected
answers would have made the eval measure nothing real: they'd show up
100% of the time by construction, inflating the score without the
ranking algorithm having done any work. The golden set only tests the
~30 chunks that actually have to compete on similarity to be retrieved.

### Measuring before locking in a bar, and proving the measurement means something

Rather than pick a target recall number, measured what the real system
actually achieves first: **recall@8 (production's real `RAG_TOP_K`)
came back 1.000** across all 15 hand-labeled questions. Before trusting
that as a real baseline rather than an artifact of a too-easy test,
checked whether the eval could actually tell good retrieval from bad by
re-running at a much smaller k: recall@3 dropped to 0.789, recall@1 to
0.522, with real, specific misses reported (e.g. `dev_publisher` scored
0 at k=3 — the developer/publisher columns didn't make the cut at that
tight a limit). That confirmed the eval has real discriminating power,
not just a design that trivially always passes regardless of k, so the
1.000 at the real k=8 is an earned, meaningful result and not a red
flag. Also confirmed the result is deterministic across repeated runs
(no randomness in embedding computation for fixed input text) before
locking `1.0` in as the exact regression bar — a bar grounded in what
was actually measured, not a guessed target picked in advance.

### A quick honesty check on the 15 questions themselves

Cross-checked every chunk ID referenced in the golden set against the
real corpus programmatically (a small script comparing the two id sets)
rather than trusting hand-typed IDs — caught zero typos, but this is
exactly the kind of silent bug (a golden question that can never pass
because its expected chunk ID has a typo, quietly rendering that
question's "pass" meaningless) that's cheap to introduce and easy to
miss by eye in a list of 15 frozensets.

### Two small, unrelated documentation bugs found while checking scope

Asked directly whether ARCHITECTURE.md needed updating for this work
(it doesn't touch the graph's shape, so no) — but checking it against
the real code surfaced two pre-existing staleness bugs unrelated to
this slice: the System overview diagram's FastAPI node was missing
`/catalog` (added several slices ago), and the RAG diagram said "~24
SchemaChunks" when the corpus has grown to 35 real chunks. Fixed both,
and also added `catalog.py` as a second box next to `genre_stats.py`
in the System overview diagram, since it bypasses `connection.py` the
same way but had never been drawn — see the entry just above this one.

### Verification

`ruff check src/ tests/`, `uv run mypy src/`, and the full pytest suite
(now 84 tests, up from 83) all clean. The new test itself was run
individually first to confirm it actually exercises real retrieval
code (not a stub), then as part of the full suite to confirm it
composes cleanly with everything else.

### Open questions (new)

- **15 golden questions is a reasonable start, not exhaustive.** Some
  corpus chunks (e.g. `column:owners_high` specifically, distinct from
  `owners_low`) aren't independently tested — the eval currently checks
  clusters of related chunks together rather than every chunk in
  isolation. Worth expanding if the corpus grows significantly or a
  real retrieval miss shows up in production that this set wouldn't
  have caught.
- **The bar is 1.0 with zero slack.** That's honest given what was
  measured, but it also means any future corpus edit that changes even
  one chunk's wording in a way that shifts its embedding slightly could
  fail this test without retrieval actually being "worse" in any way a
  user would notice — an acceptable trade for a portfolio project's
  regression net, but worth knowing if this pattern is reused somewhere
  with a bigger, noisier corpus.

## Slice 23 — Making the retry loop's self-correction visible, and a real bug found in the design pass

**Date:** 2026-08-29

`execute_tools_node` has had a self-correction retry loop since early in
this project (a tool error gets fed back to the model as a `ToolMessage`,
which gets another try, up to `SQL_MAX_RETRIES`). Nothing outside a
debugger could ever see it happen, though — `attempts` lived in internal
graph state only, never reached the API response, a log line, or
anywhere a person could look and answer "how often does this actually
fire, and does it work."

### The distinction that mattered: `attempts` was already overloaded

Read the existing code before adding anything, rather than assuming
`attempts` meant "retries." It doesn't, quite: `execute_tools_node`
increments it once per tool call in the loop, success or failure alike.
A question needing two legitimate tool calls (a lookup, then a stats
call) increments it exactly the same way a single failed-then-retried
call would. Surfacing the existing `attempts` field alone would have
produced a "self-correction rate" that was actually measuring "how many
tool calls did this question need" — a real but different thing.
Added a second, separate counter, `tool_errors`, incremented only
inside the `except` branch that actually catches a failure. This is the
whole point of the feature: without this split, a genuinely useful
multi-step question and an actual model mistake look identical in the
data, and any rate computed from the conflated number would be
meaningless.

### A real correctness bug caught during the design pass, before it shipped

The natural way to report "how many self-corrections actually
recovered" is `runs_with_tool_error - runs_that_hit_the_retry_cap`,
computed from two independently-incremented counters. Working through
it by hand before writing the code: `attempts` can theoretically reach
`SQL_MAX_RETRIES` through a long chain of *successful* tool calls that
never errored at all (rare in this system, but not impossible, and not
excluded by the type system). If that ever happened, `runs_that_hit_the_retry_cap`
would include a run that was never in `runs_with_tool_error` to begin
with, and the subtraction could go negative — an obviously wrong
"recovered" count with no exception ever raised to reveal it, exactly
the kind of bug that ships quietly and shows up as a nonsensical number
on a dashboard months later. Fixed by tracking the actual intersection
directly (`runs_with_tool_error_that_hit_the_attempts_cap`, incremented
only inside the branch that already knows `had_error` is true) instead
of two independent totals subtracted after the fact — a subset by
construction, so the subtraction can never go negative. This is a
correctness bug that never ran, caught by reasoning about the data
model before writing the implementation, not by a failing test after
the fact.

### What deliberately does NOT count toward these stats, verified live

A semantic-cache hit replays a stored `AgentResult` without running the
graph again — counting it would double-count the same underlying run
under a different question's cache key. A hard provider failure (Groq
rate-limited or erroring before the agent ever reaches the tool-call
loop) is a different failure class entirely, not a self-correction
event — the request never got far enough to try or fail a tool call.
Both exclusions were reasoned about in the design, then actually
confirmed live rather than just argued for: rapid manual testing during
verification genuinely triggered a real Groq 413 (tokens-per-minute
limit exceeded, an authentic free-tier constraint, not a contrived
test), which correctly fell through to the existing
`FRIENDLY_ERROR_MESSAGE` 503 path with no change in behavior, and
`/health`'s `total_runs` confirmed unchanged afterward — the exclusion
held under a real failure, not just an imagined one.

### Verification

`tests/test_graph_tool_errors.py`: four new tests directly against
`execute_tools_node`, using the same real-DuckDB-fixture-over-mocking
approach as `test_catalog.py`/`test_genre_stats.py` — including an
actual `DROP TABLE games` to produce a real `UnsafeQueryError`, not a
simulated one. One test specifically encodes the reason this feature
exists: two successful calls in one turn must leave `tool_errors` at
`0`. `ruff check`, `mypy src/`, and the full suite (88 tests, up from
84) all clean.

Also verified end to end against a real running backend, not just unit
tests: confirmed `/health`'s new `self_correction` block and `/ask`'s
`attempts`/`tool_errors` fields are both present and correctly computed
on genuine live Groq round trips (including catching and killing a
stale backend process from an earlier verification session that was
still bound to port 8000 and silently serving pre-Slice-23 responses —
worth remembering: a "field missing" error can mean the code is wrong,
or it can mean you're not actually talking to the code you think you
are).

### Open questions (new)

- **No real self-correction (an actual model-produced SQL error, not a
  hand-crafted one) has been observed live yet.** Every real question
  tried during verification either succeeded on the first attempt or
  hit a hard Groq rate limit before reaching the tool loop at all. The
  mechanism is proven correct at the unit level against a genuine
  error; the end-to-end "a real live mistake gets logged and counted
  correctly" path is inferred from the wiring, not yet observed
  in production traffic.

## A scoping decision, not a slice: dropping the data-analysis case study

**Date:** 2026-08-29

Slice 21's audit had flagged a standalone data-analysis case study (real
EDA and charts over the 1,000-game catalog) as worth adding for a Data
Scientist/Data Analyst reviewer specifically, distinct from this
project's AI-engineering audience. Asked directly whether it was still
worth building, and it isn't: this project is explicitly positioned for
AI Engineer roles, and separate DS/DA portfolio projects already exist
elsewhere covering that audience. Building it here would spend real time
speaking to a reviewer this project was never meant for, at the cost of
time better spent on the AI-engineering-specific items still open (the
README hook, quantified results) or on applying and interview prep
directly. See PLAN.md's "Dropped" section for the entry.

## Slice 24 — Real numbers, published as they actually came out

**Date:** 2026-08-29

The last audit item asked for quantified results: real eval accuracy,
real `/ask` latency, real cost per question, cache hit rate. The
temptation with a slice like this is to run something until it looks
good, then publish that run. Didn't do that — ran the real eval suite
once, with real instrumentation, and published exactly what came back,
including a real failure.

### Verifying Groq's pricing live, not from memory

Cost per question needs a real $/token rate. Checked Groq's current
on-demand pricing for `openai/gpt-oss-120b` live rather than trusting
training data, which can be stale for something that changes as often
as API pricing: $0.15/M input tokens, $0.60/M output tokens. Same
instinct as verifying the Steam CDN URL or Render's free-tier terms
earlier in this project — a factual claim about an external service
gets checked, not assumed.

### Finding the right instrumentation point before touching any code

The real engineering question here was how to get accurate token counts
across a *whole* agent run (router + agent node, possibly several tool-
call rounds) without invasively changing `classify_question()`'s or
`judge_answer()`'s return types just to smuggle usage data out of
`with_structured_output()` calls, which don't normally expose raw token
usage on their parsed Pydantic return value. Tested
`langchain_core.callbacks.get_usage_metadata_callback()` — a contextvar-
based callback that hooks at `on_llm_end`, before structured-output
parsing consumes the raw response — against a real live `run_agent()`
call before trusting it, and confirmed it correctly captured usage from
*every* LLM call in the graph, structured-output router call included,
with zero changes to any existing function signature. Wired it directly
into `run_evals.py` as a permanent addition (not a disposable script),
since it's exactly the kind of thing worth measuring on every eval run
going forward, not just once.

Deliberately excluded the judge's own LLM call from the tracked block:
the judge is eval-harness overhead, not something a real `/ask` caller
ever pays for, so including it would overstate the real per-question
cost.

### The numbers, published as they came out

Route accuracy: 6/6. Deterministic checks: **5/6**, not 6/6 — one real
failure, and a notable one: `analysis_action_vs_f2p_not_mislabeled` is
the exact regression test Slice 4 built specifically to catch a
free-to-play group being mislabeled without actually being filtered to
`price_usd = 0`. On this real run, the model used
`AVG(CASE WHEN price_usd = 0 THEN price_usd END)` directly in raw SQL
instead of calling `run_stats`'s `compare_two_groups` mode, and the
check correctly flagged it. This is the eval harness doing exactly its
job — catching a live instance of a known failure class — not a
disappointing result to explain away. A quieter, more tempting version
of this slice would have re-run the suite until it came back 6/6 and
reported that instead; publishing the real 5/6 is a more honest signal
about what the harness actually catches, and arguably a better one to
be able to discuss in an interview than a suspiciously clean number.

Avg judge score: 4.2/5. Avg latency: 15.5s over 6 real full-graph runs
— reported the actual range (3.2s to 37.1s) alongside the average
rather than let one number imply more precision than 6 samples support;
the slow outlier was an `analysis` question requiring two sequential
LLM turns plus a heavier query. Avg cost per question: **$0.00057** —
real token counts, real pricing, meaning roughly $0.57 to answer 1,000
questions on Groq's on-demand list price.

### The cache-hit finding: honest and more specific than a single number

There's no real production traffic yet to derive an organic cache "hit
rate" from — inventing one would be exactly the kind of fabricated
number this project has refused to do elsewhere (the em-dash guarantee,
the eval ground truth computed live instead of hardcoded). Instead, ran
a real, small, labeled test against a live backend: a genuinely
different question correctly missed (true negative, no false
positives observed in this or any earlier session's testing); a
natural real-world paraphrase — "Which game costs the most out of
everything in the catalog?" against the cached "What is the most
expensive game in the catalog?" — **also missed**, a genuinely useful
finding that the 0.93 similarity threshold (calibrated empirically back
when the cache was built, see `src/config.py`'s own comment on it) is
more conservative in live practice than a casual assumption would
suggest; a near-identical rewording ("the whole catalog" vs "the
catalog") correctly hit, confirmed via the exact `cache hit: ... ~ ...`
log line, not just the response's `cached: true` flag. The honest
takeaway — precise but conservative — is more useful and more credible
than a single invented percentage would have been.

### A real bug found along the way, deliberately not fixed in this slice

`golden_questions.py`'s `forecast_not_supported` question still carries
`reference_facts` text ("This system has no forecasting tool or
time-series data") that was accurate before Slice 9b's real
`run_forecast` tool existed and is stale now — likely part of why that
question scored a middling 3/5 from the judge, which is grading the
real answer against outdated ground truth. Noticed while reading the
report, not fixed here: this slice was about gathering and publishing
numbers honestly, not re-validating the golden set, and conflating the
two would have meant reporting numbers from a suite that changed mid-
measurement. Logged as an open item instead (see PLAN.md's Slice 24
entry).

### A real environment trap, hit twice now

Killed a stale `uvicorn` process during verification whose port was
still bound from an earlier session — the second time this exact thing
has happened in this project (see Slice 23's entry). `pkill -f "uvicorn
src.api.main"` silently reports nothing and does not actually stop the
process in this environment, for reasons not fully diagnosed (likely a
process-tree/shell-wrapper mismatch under `uv run` on Windows Git Bash).
Confirmed via `netstat -ano | grep ":8000"` and killed it precisely with
`taskkill //F //PID <pid>` instead. Worth remembering as a standing
practice for this project specifically, not just this slice: when a
live check shows unexpected/stale data, checking whether you're
actually talking to fresh code is a real, recurring first step here,
not a hypothetical one.

### Verification

`ruff check`, `mypy src/`, and the full pytest suite (88 tests, all
pre-existing — this slice added instrumentation and ran real evals, no
new test file) all clean. The numbers themselves are the verification
artifact for this slice: real Groq calls, real pricing, real timing, a
real cache test against a real running backend, published without
editing.

### Open questions (new)

- **`forecast_not_supported`'s stale reference facts** (above) should
  be rewritten to match the real post-Slice-9b behavior before the next
  time eval numbers are gathered, or it will keep quietly dragging that
  question's judge score down for a reason that has nothing to do with
  actual answer quality.
- **n=6 is a thin sample for latency and cost claims.** The published
  numbers are real and honestly reported, but six questions is enough
  to be directionally right, not enough to treat as a stable production
  average — worth re-measuring at a larger scale before quoting these
  numbers as if they were production telemetry rather than a single
  eval run's honest result.

## Slice 25 — Building the README hook, and finding a real production bug while taking a screenshot

**Date:** 2026-08-29

The last item on the AI-Engineer-focused audit: the README opened
straight into dense technical prose, with no live demo link, screenshot,
or badges, despite the app being genuinely live since Slice 19. Checked
one thing before writing a word of the hook: whether the GitHub repo is
actually public. A polished hook pointed at a private repo helps nobody
— confirmed via the GitHub API (`private: false`) before investing any
further effort.

### The screenshot had to be real, which is what surfaced the real bug

Decided early that a mockup or a hand-drawn diagram wouldn't do — the
whole point of a "live demo" hook is that it's actually live, so the
screenshot needed to come from driving the real deployed site with
Playwright, not from a local dev server. That decision is what surfaced
a genuine production incident that a static mockup would have hidden
completely.

The first two capture attempts failed for boring reasons (a `wait_for`
that matched a decorative "Show the work" heading in the Meet Ludo
section instead of the real result's `<summary>` element, then a
too-short timeout). The third attempt failed differently: a 90-second
timeout with nothing rendering at all. Rather than assume "just add a
longer timeout and retry," checked the live backend directly with
`curl`, which is what turned this into a real finding instead of a
flaky-test workaround.

### Diagnosing it methodically instead of just retrying until it worked

`curl` against `/ask` for the exact same question returned a hard 502
after ~59 seconds. The instinct at that point could have been "the LLM
call is slow, that's expected" — but the eval numbers from Slice 24,
gathered minutes earlier, already established real latency tops out
around 37s for a heavy question, and a 502 is not what a slow-but-
successful request looks like. Checked `/health` next specifically to
isolate two different possible problems: "the whole service is down"
versus "just the heavy `/ask` path is struggling." `/health` also
returned a 502, in 0.3 seconds — not a timeout, an immediate rejection,
meaning nothing was actually answering behind Render's proxy at all. A
few minutes later, `/health` came back clean with `cache_entries: 0`
(a tell that the process had restarted fresh), and a plain `curl -I`
showed `x-render-origin-server: uvicorn` responding normally again.
This is exactly a real, live production incident living and being
diagnosed in real time, not a hypothetical failure mode.

### What this actually means, and what it doesn't (yet)

The existing README caveat, written back in Slice 19, said the free
tier's spin-down means "the first request after inactivity takes
30-60s to wake it back up" — implying a slow but eventually-successful
response. What was actually observed here is meaningfully worse: a hard
502 across every endpoint during some portion of the wake window, not
a queued request that eventually completes. Corrected the README's own
claim to match what was actually observed rather than leave the more
optimistic (and, it turns out, incorrect) original phrasing standing.
This doesn't yet rise to "the deployment is broken" — the service does
recover on its own within roughly a minute, and every previous live
verification in this project (Slices 19, 23, 24) worked fine once past
that window. The likely cause, not yet confirmed: this app's actual
memory footprint (fastembed's ONNX runtime, numpy, scipy, DuckDB, and
the full LangChain/LangGraph stack all loaded in one process) is
substantial for Render's free-tier 512MB limit, and the boot window
after a spin-down is exactly when memory pressure would be highest —
noted as a real, specific hypothesis to check, not asserted as fact
without evidence.

### The screenshot itself

Once the backend was confirmed stable, drove the real site end to end:
filled the real ask bar, submitted a real question, waited specifically
for the real result's `<summary>` element (disambiguated from the
decorative one), expanded it, and screenshotted that exact panel
element directly — not a manually-computed pixel region, which the
first successful capture attempt got wrong (cropped the top of the
route badge off and missed the question). Element-level screenshotting
fixed that correctly on the first retry. The final image shows a real
route badge (`Lookup`), a real `Cached` label (left in deliberately —
it's evidence the semantic cache actually works, not something to hide
for a cleaner-looking screenshot), the real answer, a real result
table, and the real SQL and retrieved-schema-chunk list under "Show the
work."

### Verification

Confirmed the repo is public via the GitHub API before starting. Every
new badge URL (`test.yml`, `frontend-ci.yml`, the shields.io license
badge) checked with a real `curl` for a 200, not pasted and assumed —
GitHub's workflow badges are dynamic and only exist for a public repo
with that exact workflow file present, so this wasn't a given. The
screenshot file was checked for a reasonable size (70KB). Grepped the
new README copy for em dashes and en dashes; the root README has never
been subject to the frontend's no-dash rule (that rule is explicitly
scoped to `app/`, `components/`, `lib/`, and the agent's generated
answer text — every prior slice's own DOCEXP/PLAN/README prose has used
em dashes freely throughout this whole project), so this was a
consistency check against existing practice, not a new constraint.

### Open questions (new)

- **The 502-during-wake-up root cause (likely memory pressure on the
  free tier's 512MB limit) is a hypothesis, not a confirmed diagnosis.**
  Would need Render's own metrics/logs (not accessible from here) to
  confirm memory is actually the constraint rather than something else
  (a slow health-check registration, a Cloudflare-edge timing issue on
  Render's side). Worth checking directly next time this is
  investigated, rather than continuing to assume.
- **No alerting exists for this.** The only way this incident was
  caught was a screenshot script timing out during unrelated work. A
  portfolio project doesn't need production-grade uptime monitoring,
  but it's worth knowing this failure mode has no visibility outside of
  someone happening to hit it.

## Slice 26 — Finishing token/cost logging on the actual request path

**Date:** 2026-08-29

Closing out the last item from the AI-Engineer audit surfaced something
worth naming honestly: Slice 24's token/cost work only touched
`run_evals.py`. A real `/ask` or `/ask/stream` call still reported
neither — the eval harness had the instrumentation, production didn't.
Declaring the audit item done because the eval harness had it would have
been technically defensible but not actually true to what was asked
("token/cost logging *per request*"), so this slice finishes it on the
path that matters.

### One pricing source, not two

Rather than copy Slice 24's `GROQ_PRICING_USD_PER_MILLION_TOKENS` dict
and cost formula into `graph.py` as a second copy, pulled both out into
a new `src/agent/pricing.py` that `run_evals.py` and the live agent now
both import. Two independently-maintained pricing tables is exactly the
kind of thing that quietly drifts — someone updates the eval harness
after a Groq price change and forgets the production copy, and now the
"measured results" in the README and the numbers a real user's request
actually reports disagree for no reason anyone would notice quickly.

### The same verified mechanism, applied to the path that actually matters

`get_usage_metadata_callback()` was already proven correct against a
real `run_agent()` call in Slice 24. Wrapping `run_agent()` and
`stream_agent()` themselves in it (rather than only wrapping calls made
from `run_evals.py`) was the direct, minimal way to make every real
request carry the same real numbers. `AgentResult` grew `total_tokens`/
`estimated_cost_usd`, following exactly the same shape Slice 23 already
established for `attempts`/`tool_errors`: computed once in
`_result_from_state`, threaded through `AskResponse` so it's visible
per-request at `/docs`, and aggregated in `RunStats` for a live,
production-derived average at `/health` — not just Slice 24's one-off
eval-run snapshot.

### The log line the task actually asked for

The original ask was "token/cost *logging*," and until this slice
nothing was actually logged — only returned in the API response and
folded into an aggregate. Added an explicit `usage: route=...
total_tokens=... estimated_cost_usd=...` log line in
`_record_real_run()`, deliberately unconditional (unlike the
self-correction log line, which only fires when `tool_errors > 0`):
cost is worth a visible trail on every request, not just failures.

### Verification

Ran the exact same live checks Slice 23 used for `attempts`/
`tool_errors`, extended to the new fields: a real `/ask` call and a
real `/ask/stream` call both returned correct `total_tokens`/
`estimated_cost_usd` (in the same range Slice 24 measured — a useful
cross-check that nothing about the refactor silently changed the
number), `/health`'s new `usage` block aggregated correctly after a
real request, and the log line's actual output was read from the
running backend's log file, not assumed from reading the code. Also
re-ran the real eval suite (judge skipped, to avoid spending Groq
quota on what was fundamentally a refactor check) after moving the
pricing constant into its own module — same 5/6 deterministic result,
the same real failure reproduced identically, confirming
`pricing.py`'s extraction didn't change behavior anywhere. `ruff
check`, `mypy src/`, and the full 88-test suite all clean throughout.

## Slice 27 — Finishing what Slice 21 actually started: `run_evals.py` in CI

**Date:** 2026-08-29

Doing a status check surfaced something worth being honest about: the
original plan (Slice 21) was "ruff/mypy → automate `run_evals.py` →
README hook." Ruff/mypy happened. The README hook happened, eventually.
In between, the RAG retrieval eval (Slice 22) got built and *did* land
in CI — but it's a different eval, checking retrieval quality, not
final-answer quality. Slice 24 ran the real `run_evals.py` suite, but
manually, once, to gather numbers. It would have been easy to count
either of those as "the eval automation is done" and move on; neither
one actually was. Worth naming this directly rather than letting it
quietly not happen: reprioritizing mid-project is fine and happened
for good reasons each time, but an item that keeps getting implicitly
superseded by adjacent work still needs to actually get done, or
explicitly dropped — not just forgotten.

### The forecast question: cut, not fixed, and the tradeoff stated plainly

Slice 24 found `forecast_not_supported`'s `reference_facts` were stale.
Asked directly whether to fix it or cut it, given this project's
explicit AI-Engineer-only positioning (Data Scientist/Analyst-facing
work already dropped in an earlier slice for the same reason):
removed the golden question entirely rather than write an accurate
replacement. The honest cost of that choice: the eval suite now has
zero dedicated coverage of the `forecast` route. That's a real
reduction in eval coverage, not a neutral cleanup — recorded here
plainly rather than let the checklist entry imply nothing was lost.

### Why scheduled, not gate-every-push

`run_evals.py`'s own docstring already states the intent: "Exit code
is 0 only if every question's route AND deterministic check passed —
that's the part meant to gate CI later." Gating every push directly
would mean a real Groq API call (router + agent + judge, per golden
question) on every commit. Groq's free tier is rate-limited per
minute, not just billed by the dollar, and this repo has had genuine
multi-commit development days — gating every push risks a
CI failure from a rate limit, not from an actual regression, which
would train the habit of ignoring red X's here. A daily schedule
(plus `workflow_dispatch` for on-demand runs) is enough to catch a
real regression from a prompt edit or dependency bump without that
risk. A red X on this specific workflow on a given day is still
informative rather than embarrassing, by design: the one deterministic
check that's actually failed throughout this whole project's history
is a real, LLM-driven behavior (the free-to-play mislabeling), not a
CI bug — an occasional failure here is the harness doing its job, the
same conclusion Slice 24 reached about the manual run.

### The ephemeral-runner problem, solved the same way `golden_questions.py` was designed to allow

GitHub Actions runners start empty every time, and `data/db/*.duckdb`
is gitignored on purpose (a 1000-game catalog isn't something to
commit). Rather than reach for a cached artifact or restructure
anything, used the property `golden_questions.py` was already built
with back in Slice 5: `build_golden_questions()` queries the *real,
current* DB for its ground truth at eval time, rather than hardcoding
expected values. That means a small, fast, self-consistent catalog
(`--count 100`, built fresh by the workflow itself) is exactly as
valid a target for this suite as the real 1000-game one — the suite is
checking whether the agent reasons correctly over whatever data
exists, not whether this specific dataset matches some fixed
expectation. A larger ingestion would only have cost more CI minutes
for no additional signal.

### Verification

Didn't just write the YAML and trust it. Ran the identical two steps
the workflow runs — a small ingestion, then the real eval suite —
locally, with `DUCKDB_PATH` redirected to a throwaway location
(`/tmp/eval_verify`) specifically so the real local 1000-game catalog
this whole project's local testing depends on was never at risk of
being silently overwritten by a 100-game rebuild. Confirmed 5/5 route
accuracy and 4/5 deterministic checks — the free-to-play mislabeling
bug caught a third time now, across a third distinct catalog (the real
1000-game one in Slice 24, a second 100-game one earlier in this same
slice's exploration, and this one), which is real, accumulating
evidence that it's a genuine, consistent model behavior tied to how
the prompt guides comparison queries, not an artifact of one
particular dataset's contents. Confirmed the real local DB's row count
(1000) afterward to prove it was untouched. Validated the workflow
YAML parses correctly. `ruff check`, `mypy src/`, and the full 88-test
suite all clean (no new test file — this slice changed a golden
question and added a workflow, not application code with new
behavior to unit-test).

### What still needs the user's action

This workflow cannot run at all without a `GROQ_API_KEY` repository
secret — something only the user can add (Settings → Secrets and
variables → Actions), same as `DEPLOY_HOOK_URL` earlier. Until it's
added, the workflow will trigger on schedule and fail cleanly on the
eval step with a clear missing-key error, not silently do nothing and
not do anything destructive either.

### Open questions (new)

- **Zero eval coverage for the `forecast` route now.** A deliberate,
  stated tradeoff, not an oversight — but if forecast quality ever
  needs to be demonstrated again (a future job application emphasizing
  time-series work, say), this is the first gap to fill, with a golden
  question that matches the tool's real current honest-degradation
  behavior instead of the stale one that got removed.
- **The daily schedule has not yet been observed actually firing.**
  Verified the mechanism thoroughly by hand; the first real scheduled
  run (and whether the `GROQ_API_KEY` secret has been added by then)
  is still unconfirmed.

## Slice 28 — The new workflow's first real run found a real bug in itself

**Date:** 2026-08-29

The user added the `GROQ_API_KEY` secret and manually triggered
`run_evals.yml` right away to check it worked, rather than waiting for
the next day's schedule — exactly the right instinct. It failed, with a
genuine `groq.RateLimitError`: "Rate limit reached for model
`openai/gpt-oss-120b` ... on tokens per minute (TPM): Limit 8000, Used
5688, Requested 2367." This is the best possible outcome for a first
run in one specific sense: it found a real bug immediately, in CI,
before it could fail silently or intermittently in a way that looked
like flakiness rather than a real, fixable problem.

### What was actually wrong, and why "scheduled, not on every push" wasn't the whole fix

Slice 27's reasoning for a daily schedule (instead of gating every
push) was about avoiding rate-limit collisions with *other* traffic
during active development bursts. That reasoning was correct as far as
it went, but incomplete: this failure had nothing to do with external
traffic at all. Five golden questions, run back-to-back within one
process, each using 3000-4700+ tokens, can burst past an 8000 TPM cap
entirely on their own, with zero other requests involved. The schedule
change solved a different problem than the one that actually occurred.

### Two mitigations, because either alone has a real weakness

A proactive fixed gap between questions (`SPACING_SECONDS = 5.0`)
reduces how often the limit gets hit, but a fixed number can't
adapt if a future model change increases per-question token usage, or
if Groq's own limit changes. A reactive retry-with-backoff
(`_call_with_retry`) adapts to whatever actually happens, but doing
only that would mean routinely eating a 429 and a 15-30 second wait as
normal operation instead of trying to avoid it in the first place.
Used both: spacing to make hitting the limit uncommon, backoff to
recover cleanly on the rare occasion spacing isn't enough. Same
"defense in depth, not one clever fix" instinct as the two independent
SQL-safety layers in `db/connection.py`.

### A lint rule caught a real design smell in the first draft, not a false positive

The first version of the retry wrapper used a zero-argument lambda
closing over the loop variable (`lambda: run_agent(gq.question)`), with
a mental justification that it was safe because the lambda was invoked
immediately within the same loop iteration, never stored for later.
Ruff's B023 (bugbear's "function definition does not bind loop
variable") flagged it anyway. The instinct to argue the specific usage
was safe and suppress the warning was available and would have worked
functionally — but the actual fix (have `_call_with_retry` accept the
function and its arguments separately, no lambda, no closure) is
strictly simpler code that eliminates the entire bug class the lint
rule exists to catch, not just this one safe-in-practice instance of
it. Took the rule's hint seriously rather than arguing past it.

### Verification

Ran the exact fixed suite for real against a fresh throwaway catalog
(`/tmp/eval_verify2`, never the real local `data/db/games.duckdb`) —
completed with zero rate-limit errors, 5/5 route accuracy, 4/5
deterministic checks. The one failure is the same free-to-play
mislabeling bug, now independently reproduced a fifth time across a
fifth distinct catalog (the real 1000-game one twice across Slices 24
and this verification's baseline, plus three different smaller ones
used for CI/workflow verification across Slices 27 and 28) — this is
now about as strong a signal as this project has for "genuine,
consistent model behavior" versus "one dataset's fluke." Confirmed the
real local catalog's row count (1000) unchanged after the throwaway-DB
runs, same discipline as Slice 27. `ruff check`, `mypy src/`, and the
full 88-test suite all clean.

### Open questions (new)

- **The daily schedule itself still hasn't fired on its own yet** — only
  manual `workflow_dispatch` runs have been observed (the one that
  failed, and the fix verified locally). The actual `cron`-triggered run
  is still unconfirmed, though there's no known reason it would behave
  differently from the manual trigger now that the rate-limit fix is
  in place.

## Slice 29 — A year-old flagged risk finally became a real incident

**Date:** 2026-08-29

The user noticed something looked wrong on the Actions tab: a
scheduled `Poll player counts` run sitting "Pending" seven hours after
it should have started, and a manually-triggered run stuck seven-plus
minutes into a step called "Rebuild the local catalog." Two
screenshots, no logs beyond a truncated build warning — the actual
diagnosis had to come from reading the workflow file and the two
Python scripts it runs, not from the screenshots themselves.

### Reading the code instead of guessing from the symptom

`poll_player_counts.yml`'s "Rebuild the local catalog" step runs
`uv run python -m src.ingestion.ingest` with no `--count` flag, meaning
the real production default of 1000 games. Reading `ingest.py`'s own
docstring spells out exactly what that does: a cheap bulk listing call,
then *two* separate per-game enrichment loops — SteamSpy's own
`appdetails` at roughly 1 request/second, and Steam's storefront
`appdetails` at roughly 1 request per 1.5 seconds. For 1000 games,
that's easily 40+ real minutes, every single time, since a GitHub
Actions runner starts fresh each run with no persisted `data/raw/`
cache to skip re-fetching from.

Then read what `poll_player_counts.py` actually does with all of that:
one line, `SELECT appid FROM {GAMES_TABLE}`. Not genre, not price, not
release date, not any of the ~15 other columns that ~40 minutes of
enrichment calls exist to populate. The entire expensive step existed
solely to let this script learn a list of integers it could have
gotten from the first, cheap step of `ingest.py` alone.

### This was a known risk, written down, and not acted on until it broke

An earlier slice had already flagged this almost exactly: "a narrower
appids-only ingestion path is the likely fix if this becomes a real
problem." That's precisely what happened here — not a new discovery,
a predicted risk finally landing. Worth being honest about the
lesson: flagging a real, understood risk in a document and then not
prioritizing it until a user hits the consequence is a common way
technical debt actually plays out, and this project's own
documentation had the fix already spelled out, just not scheduled.

### Why runs were queuing for hours: the concurrency group, not a bug in it

`poll_player_counts.yml`'s `concurrency: group: poll-player-counts,
cancel-in-progress: false` means only one run executes at a time;
anything else queues behind it. That setting itself isn't wrong — it's
exactly right for a job that commits data back to the repo, where two
overlapping runs writing snapshots simultaneously would be a real
problem. The concurrency group was working as designed; it was
faithfully queuing runs behind a job that individually took far longer
than it needed to, turning a correct safety mechanism into a visible
symptom of the real problem underneath it.

### The fix: remove the dependency, not just speed up the step

Considered adding a `--count` override or an "appids-only" flag to
`ingest.py` itself, but the cleaner fix doesn't touch `ingest.py` at
all: `poll_player_counts.py` now calls `SteamSpyClient.get_all_page()`
directly — the exact same cheap, single-request bulk listing
`ingest.py` already starts with — and never touches DuckDB or requires
a pre-built `games` table in the first place. This removes the "rebuild
the catalog" step from the workflow entirely, not just makes it
faster, and removes a dependency (`get_read_only_connection`,
`GAMES_TABLE`) this script never conceptually needed: it was never
really about the catalog, only ever about a list of appids the catalog
happened to also contain.

### Verification, scoped to what actually changed

Ran the new `_fetch_target_appids()` function directly against the
real live SteamSpy API — no mocking — and got back real appids headed
by `730`, the same game independently confirmed as the top-owned title
in every other real check this project has done this session (the
eval suite's `lookup_top_ccu` golden question, live `/ask` calls, the
README screenshot). Deliberately did not run a full `run_poll()`
locally: that would mean polling 1000 real games at Steam's own real
1-request/second rate limit — about 17 real minutes — to re-verify
polling-loop code that this slice never touched. Scoping verification
to the actual diff, not the whole script, was the right call here.
`ruff check`, `mypy src/`, and the full 88-test suite all clean;
confirmed no stray local files were left behind by the verification
call (it doesn't write a snapshot on its own — only `run_poll()`
does, which wasn't run against the real 1000-game target locally).

### Confirmed on a real run, the same day

Both open questions this entry originally raised got resolved within
hours, not left standing: the user cancelled the old stuck/pending
runs directly from the Actions tab, then manually triggered the fixed
workflow to check it for real rather than waiting for the next `cron`
tick. Run #3 succeeded in **19m29s total** — "Poll live player counts"
alone took 19m22s for all 1000 games, matching the ~20 minute estimate
almost exactly, against the roughly hour-long runs from before this
fix. "Commit the new snapshot" wrote and pushed a real new file
(`data/player_counts_raw/2026-08-29T16-40-29Z.json`, 1004 insertions)
automatically. This confirms the whole pipeline end to end on GitHub's
own infrastructure, not just the appid-fetching function in isolation
the way this slice's own verification section was scoped to.

### Open questions (new)

- **Only a manually-triggered run has been observed, not yet the
  `cron`-scheduled path itself.** No known reason to expect it to
  behave differently — `workflow_dispatch` and `schedule` both invoke
  the identical job definition — but the actual every-6-hours trigger
  firing on its own is still technically unconfirmed.

## Slice 30 — The eval check was wrong, not the model, and four slices of docs said otherwise

**Date:** 2026-08-29

A scheduled `run_evals.yml` run failed on the same check it's failed on
before: `analysis_action_vs_f2p_not_mislabeled`. The instinct at this
point, reinforced by this project's own prior documentation, was "this
is the known Slice 4 bug, seen again" — and that instinct was wrong,
in a way worth writing up in full because of how it was wrong, not
just that it was.

### First pass: diagnosing real competing prompt guidance, correctly

Read `prompts.py` and `schema_corpus.py` rather than guessing, and
found something real: the base system prompt template *and* the
RAG-retrieved `metric:no_union` schema chunk both handed the model a
concrete, worked SQL example for "comparing two groups" directly —
`AVG(CASE WHEN genre LIKE '%Action%' THEN price_usd END) AS
avg_action_price` — using almost the exact domain example
(Action/free-to-play) the failing golden question itself uses. That
competed directly with `ANALYSIS_TOOL_GUIDANCE`'s instruction to
prefer `run_stats` for this exact question class. This diagnosis was
correct as far as it went: two real, contradictory pieces of guidance
existed in the prompt, and removing the dangerous example while
preserving enough domain vocabulary to not regress Slice 22's RAG
retrieval eval (a first attempt did regress it, from 1.000 to 0.967 —
caught by the retrieval eval test itself, not manually) was a
legitimate improvement.

### Verifying the fix instead of trusting it — and it failed

Ran the exact golden question five times with the fix in place (rate
limits cut it to two clean samples before a 429). Both still produced
plain conditional-aggregation SQL, not a `run_stats` call. Reported
this plainly rather than declaring success on a plausible-sounding
fix — this is the exact discipline `verification-before-completion`
exists to enforce, and it's what led to the next, more important step
instead of stopping at "well, I tried."

### The actual discovery: re-reading old evidence instead of gathering new evidence

Rather than immediately trying a third, blunter prompt wording, went
back through every SQL string this check has ever failed on —
Slices 24, 27, 28, and this slice's own two verification runs, six
instances total — and actually read each one instead of trusting the
check's verdict. Every single one correctly filtered the free-to-play
group on `price_usd = 0`. Not "usually." Every time. The original
Slice 4 bug — a group falsely labeled free-to-play without actually
filtering on `price_usd = 0` — has never once recurred in this
project's entire history of testing this question.

The check itself explained why once looked at closely:

```python
if not (result.stats_result and result.stats_result.get("mode") == "compare_two_groups"):
    return CheckResult(False, ...)
```

This is the *first* line of the check. It hard-requires `run_stats`'s
specific structured output shape and returns failure immediately if
that shape isn't present — it never once inspects `result.sql` or the
real returned values when a different tool was used. The check was
never actually testing "is the free-to-play group correctly filtered."
It was testing "was `run_stats` the tool called," and treating that as
a proxy for the real thing. The proxy was wrong, and had been wrong
since whenever this check was last touched.

### The docs said otherwise, four times, and that's worth naming directly

README.md's "Measured results," and DOCEXP.md's Slices 24, 27, 28, and
29 entries all describe this failure as "the same bug reproduced
again" — two, three, four, five times, escalating the claim each time
as if it were accumulating evidence of a real, consistent model
behavior. It was accumulating evidence of something real — just not
what it was labeled as. Every one of those claims was written by
pattern-matching "this specific check failed" to "the historical bug
recurred," without re-verifying the actual SQL each time. That's
exactly the kind of unverified claim this project's own discipline
(evidence before assertions, described in the
`verification-before-completion` skill this whole session has followed)
exists to prevent, and it happened anyway, repeatedly, because the
check's own verdict was trusted as ground truth instead of checked
against the real underlying data. Not correcting the historical
entries themselves — they're a lab notebook, and rewriting past
entries to match current understanding would defeat the purpose of
keeping one — but naming the pattern here, plainly, matters more than
the specific miscount.

### The actual fix: the check, not the model

`_check_action_vs_f2p_not_mislabeled` now accepts two valid paths to a
correct answer: the original `run_stats` `compare_two_groups` result,
or a plain-SQL answer whose free-to-play-labeled column's real
returned value is genuinely ~$0. It checks `result.columns`/
`result.rows` — the actual data that came back — not the SQL's text
and not which tool produced it. This directly serves the original
Slice 4 intent (never let a fake free-to-play group slip through)
without depending on a specific tool being chosen, which prompt
engineering alone had just been shown not to reliably guarantee.

The prompt/schema wording improvements from earlier in this slice
weren't reverted — they're still more precise, correct guidance with
no observed downside (retrieval eval back to 1.000, all other tests
green) — but they're explicitly not credited as "the fix" here. They
were a reasonable attempt at a problem that, once actually understood,
turned out to be smaller than believed.

### Verification

`ruff check`, `mypy src/`, and the full 88-test suite clean throughout
every step of this slice. The real proof is the final eval run against
the real 1000-game local catalog, with the judge, after the check fix:
**5/5 route accuracy, 5/5 deterministic checks** — the first clean
result this specific golden question has ever produced, not because
the model finally got it right, but because the check finally started
looking at whether it actually had been right all along.

### Open questions (new)

- **README.md's "Measured results" table still shows the old 4/5
  numbers** as of this entry and needs republishing with the real,
  current 5/5 result — tracked as the immediate next step, not left
  standing.
- **The prompt/schema changes remain unproven at scale.** They didn't
  cause the fix here, but whether they meaningfully improve real
  production behavior for genuinely ambiguous group-comparison
  questions (distinct from this one golden question, which turned out
  not to need them) is still an open, unverified claim.

## Slice 31 — The Agent Execution Trace artifact catches up to the real graph

**Date:** 2026-08-29

The interactive "Agent Execution Trace" Claude Artifact linked from
`ARCHITECTURE.md` predates a fair amount of this project's own history.
Read it back in full (803 lines) to check it against the current system,
and it had drifted on two independent axes: content and identity.

### Content: a graph node that stopped existing

The artifact's `NODES`/`EDGES` arrays still had a `forecast_not_supported`
box, wired from `router` on the `forecast` route straight to `answer`,
with a fourth "Forecast question" example walking through it: route
classifies as forecast, hits the dead end, answer says "I can't forecast
yet... (planned for a later slice)." That was accurate once, before
Slice 9b gave the graph a real `run_forecast` tool. Since then, forecast
has flowed through the exact same `retrieve_schema` → `agent` ↔
`execute_tools` → `build_chart_spec` loop as `lookup` and `analysis` —
the dedicated dead-end node simply isn't in `src/agent/graph.py` anymore.
Presenting it as current would have been showing a portfolio reviewer an
architecture the code no longer has.

Fixing this meant more than deleting a box. The corrected "Forecast
question" example needed real steps through the real path, and the
instinct to invent a clean "agent calls `run_forecast`, gets a tidy
projection back" transcript had to be resisted — that would be exactly
the kind of fabricated trace this whole artifact exists to be the
opposite of. So the plan was: capture one fresh, real, tool-invoked
forecast run against the live backend, the same way the other four
examples were originally captured.

That didn't work. Five real attempts against the deployed backend
(`ai-game-analyst-api.onrender.com`), spaced 30 to 45 seconds apart to
respect Groq's rate limit, hit real `429`s and `503`s — genuine
production flakiness, not a mocked failure. A local Ollama daemon was
checked as a fallback (`curl http://localhost:11434/api/tags`) and
wasn't running. One real capture did land before the rate limiting set
in: a real question ("How many players will Counter-Strike 2 have next
month?"), a real `route: forecast`, real retrieved schema chunks
(including `table:player_counts` and the note that it needs a JOIN to
`games`), and a real final answer — but on that particular run the model
answered directly from the retrieved schema context without calling
`run_forecast` at all.

That's not a failed capture to discard and retry until a "better" one
appears. It's a real, honest data point about current behavior worth
keeping: on at least one real run, the forecast route reasons from
schema-level context (no historical `player_counts` snapshots exist yet
for most games) rather than empirically checking via the tool first.
The rebuilt example uses exactly that real capture, with a description
that says plainly what happened and why it's shown this way, rather than
presenting a single unbroken "and then it forecasted" narrative that
never actually occurred in any of the six real attempts made to produce
one. Decided to stop spending shared production quota chasing a
cosmetically tidier capture after the fifth failed attempt; this is
illustrative content, not something worth burning more of a rate-limited
free tier on.

While rebuilding the graph, also caught a second, smaller drift: the
retrieval panel's copy still said "ranks ~24 schema chunks," a number
from before Slice 22 grew the RAG corpus to 35. Fixed alongside the
node/edge changes since it was in the same file anyway.

### Identity: amber and Archivo, on a product that's been green and Rajdhani for a while

Separately from content accuracy, the user asked directly for a
dark-green reskin. The artifact's existing palette (`--accent: #a8650f`
light / `#f0a63a` dark, warm amber) and typography (Archivo + IBM Plex
Mono) predate this project's own pivot to a neon-green identity
(`--accent: #39ff88`) and Rajdhani, both already validated and shipped
in the real frontend. Rather than pick a new, unrelated dark-green
palette, reused the live product's own real tokens verbatim
(`--background: #0a0f0b`, `--surface: #101610`, `--accent: #39ff88`,
Rajdhani + IBM Plex Mono) — the artifact *is* this product's
architecture, so it should look like it. Per the `artifact-design`
skill's own guidance on committing to one visual world, built this as a
single dark theme rather than a light/dark toggle, matching the real
frontend's own documented choice not to split light/dark chrome. Kept
semantic status color separate from the brand accent: error/degraded
badges reuse the frontend's actual `--danger` red rather than a second
shade of green, so a failure state doesn't visually blend into "this is
just how the app looks."

### Regenerating the static GIF, for real

`ARCHITECTURE.md`'s static `docs/agent-trace.gif` (the non-interactive
fallback image, captured from the old amber theme) visually mismatched
the new artifact the moment it was rebuilt. First pass at this slice
left it as a flagged follow-up rather than doing it, on the reasoning
that it needed an actual Playwright capture pass, not an HTML edit. The
user asked for it directly right after, so it got done in the same
slice rather than deferred further.

Playwright turned out to already be present on the machine, but not in
the project's own `.venv` (`pip show playwright` found a global
install; the project's venv had no `pip` module at all, this project
manages dependencies with `uv`). Rather than add Playwright to
`pyproject.toml`'s dev extra for what is a one-off documentation-image
capture tool, not something the running system needs, used `uv run
--with playwright` / `--with pillow` to pull both in as ephemeral
dependencies for the single capture script, `UV_LINK_MODE=copy` was
also needed because the repo lives under OneDrive, which breaks uv's
default hardlink-based package install with an "incompatible
hardlinks" error.

The script drives the new artifact's `lookup` example through all 8
real steps (the same steps table shown in the earlier "Content" section
above), screenshotting the `.wrap` container at 2x device scale after
each step's CSS transitions settle, then assembles the frames into a
GIF with Pillow, holding longer on the first and last frames.

Verified by actually reading back individual captured frames rather
than trusting a clean script exit, per this project's own verification
discipline, and that caught a real bug: a stray em dash in the new
artifact's own footer credit line ("...onrender.com) — see
ARCHITECTURE.md..."), against this project's repo-wide no-dash
convention. Fixed, re-captured, and republished to the same artifact
URL before copying the final GIF into `docs/agent-trace.gif`.

(One genuine false alarm along the way, worth naming since it
determined how the frames were checked: a first look at a captured
frame's footer text seemed to show an amber tint against the dark-green
theme, which would have meant the recolor missed a spot. Sampling the
actual pixel values showed the real color matched `--text-faint`
exactly; the apparent tint was a compression artifact from viewing a
downscaled preview of small anti-aliased text on a very dark
background, not a rendering bug. Cropping and re-viewing the region at
native resolution confirmed clean, correctly-colored text. Worth
recording as a reminder that a screenshot preview at reduced size can
manufacture color problems that don't exist in the actual file, so a
finding like that needs the pixel data checked before being reported.)

### Publishing

The user's phrasing ("make a new architecture artifact") was taken at
face value: published as a genuinely new artifact rather than
overwriting the old URL in place, since `ARCHITECTURE.md` doesn't
hard-link the artifact's URL in the repo (it only refers to it as "a
private Claude Artifact"), there was nothing in-repo to break by doing
so.

No open questions left from this slice: the GIF staleness flagged
earlier in this same entry was closed out before the slice ended.

## Slice 32 — A pushback on the forecast example led to a real production finding

**Date:** 2026-08-29

Right after Slice 31 shipped, direct pushback: "why did you include a
forecast question, Ludo doesn't have a forecast capability." The
instinct to defend the example (`run_forecast` is real code, genuinely
bound to the graph) would have missed the actual point, which wasn't
about whether the code exists.

### Checking instead of asserting

Separately, this same slice had already started a broader "is this
project top notch" health pass, and part of that was hitting the live
`/health` endpoint for real. It came back missing the
`self_correction`/`usage` keys Slices 23 and 26 added:

```json
{"status":"ok","db_exists":true,"model_provider":"groq","cache_entries":0}
```

That one fact reframed the forecast question entirely. Chased it
properly instead of treating it as a footnote:

- `data/player_counts_raw/` holds exactly two committed snapshots,
  `2026-08-24` and `2026-08-29`, five days apart — the gap is the
  `poll_player_counts.yml` bug fixed in Slice 29 having sat broken
  before that.
- `player_counts` is only materialized into the queryable database at
  **Docker build time** (`Dockerfile`'s `RUN python -m
  src.ingestion.build_player_counts_table`), not at request time or on
  a schedule. A running deployment only ever sees the snapshots that
  existed *when it was last built*.
- The live `/health` response above means the current deployment
  predates Slices 23/26, so it almost certainly has zero player-count
  history baked in, regardless of what's since been committed to the
  repo.

So `run_forecast` is real, correctly designed code, not fabricated
architecture, but it has, as far as any evidence in this repo shows,
never once produced a working forecast for a real user hitting the live
site. Featuring it as a worked example in the trace artifact presented
a capability that doesn't function in the product a reviewer would
actually try. The user's pushback was right, and for a more concrete
reason than "it just feels off": there's a real, checkable gap behind
it.

### Fixed the artifact, then chased the actual root cause

Removed the Forecast example from the Agent Execution Trace artifact
entirely (offered reframing it around the honest-refusal behavior
instead; the user chose removal). Four examples now, "Five real runs"
corrected to "Four", republished to the same URL, and every remaining
example's node/edge references re-validated programmatically the same
way Slice 31 did, since a manual removal is exactly the kind of edit
that silently breaks a cross-reference.

The more interesting question was why the live deploy is stale at all.
Two real things checked, one ruled out, one left as a genuine open
finding for the user to resolve from the Render dashboard, which this
session has no access to:

**Ruled out**: `.github/workflows/refresh_catalog.yml`, the workflow
whose entire job is periodically poking Render's deploy hook to force a
rebuild, has had zero runs, ever — checked via GitHub's public Actions
API, no auth needed for a public repo. That looked damning until
checking when the workflow was actually added: Monday, August 24th, by
its own git history. Its schedule is Monday 03:00 UTC. The next
Monday, August 31st, hasn't happened yet as of this entry (today is
Saturday the 29th). Zero runs is exactly what a workflow with a
correct weekly Monday schedule looks like four days after being
created, not evidence it's broken. Worth writing down precisely because
the *symptom* (zero runs) looked like a smoking gun and wasn't.

**Left open, genuinely**: this exact symptom, a live backend serving
stale code, already happened once before. Slice 23's own commit message
(`e127dee`) says so directly: "a live backend round trip that caught a
stale process serving old code" — found, apparently fixed by a manual
redeploy at the time, and never written up or root-caused in DOCEXP.md.
It has now recurred. Two concrete, plausible causes remain that only
the Render dashboard can settle:

1. **Auto-Deploy might simply not be enabled** on the service. Slice
   19's original setup used Render's manual "New Web Service" form
   rather than importing `render.yaml` as a Blueprint (documented at
   the top of that file), so whatever Auto-Deploy defaulted to in that
   flow was never explicitly verified either way. A service that only
   ever updates when someone happens to click "Manual Deploy" would
   produce exactly this recurring, unpredictable staleness.
2. **The `DEPLOY_HOOK_URL` GitHub secret may never have been set.**
   `refresh_catalog.yml` no-ops cleanly (a green checkmark, not a
   failure) when it's missing, which is precisely the failure mode that
   hides itself from a casual glance at the Actions tab.

Presented both to the user directly rather than guessing further or
silently doing nothing: this session has no Render API/dashboard
access, so this is a real handoff, not a deferred task the next slice
can just pick back up unassisted.

### Verification

Re-ran the full local suite fresh as part of the same health pass this
slice grew out of: `ruff check` clean, `mypy src/` clean (44 files),
`pytest -q` 88/88, frontend `npm run lint` clean, `npx tsc --noEmit`
clean, `npm run build` clean. Confirmed `git status -sb` shows the
local `master` exactly matching `origin/master`, nothing unpushed. The
artifact's JS was re-checked programmatically after the example removal
(parses, and every remaining step's node/edge id still resolves), the
same discipline Slice 31 established rather than re-reading the whole
file by eye.

### Resolution, confirmed live

The user checked directly: Auto-Deploy was already on for every commit,
and added `DEPLOY_HOOK_URL` as a GitHub secret. Manually firing
`refresh_catalog.yml` (`workflow_dispatch`) triggered a real Render
redeploy — confirmed via GitHub's own public Actions API, no auth
needed, that the run completed successfully. `/health` afterward showed
the `self_correction`/`usage` keys for the first time, confirming
current code is finally live.

The redeploy's own restart then produced a second, smaller scare: the
first live `/ask` request timed out against a 60s client-side limit
with no response, a retry got a real 502, and a third attempt hung for
90s with nothing at all. Diagnosed rather than assumed broken:

- Hit Groq's API directly (bypassing the app) — 200 in 0.92s, so the
  provider itself was fine, not degraded.
- Ran the exact same question through `run_agent()` locally — 9.75s,
  correct answer. Ruled out a code regression.
- Retried the live endpoint once more after the container had a few
  minutes to settle: 200 in 1.2s, `cached: true` — the *original*
  request had actually succeeded server-side all along, just slower
  than the client-side timeouts used to test it, and got cached.
- A genuinely fresh, uncached question then completed in 9.97s with a
  correct answer, matching local performance almost exactly, and
  incidentally re-confirming the Slice 30 fix live in production
  (`avg_price_action: $18.83`, `avg_price_f2p: $0.00`, correctly
  filtered).

Conclusion: no code bug, no lasting incident. A single slow cold start
right after the redeploy (loading the embedding model, DuckDB, and the
compiled graph for the first time on constrained free-tier CPU)
outran two client-side timeouts before finishing on its own. Confirmed
resolved, not just assumed resolved.

One real, separate gap surfaced along the way and left as an open
question rather than fixed reflexively: `src/agent/llm_provider.py`'s
`ChatGroq(...)` sets no explicit `timeout`/`max_retries`, so a slow
cold path or a genuinely degraded provider has no ceiling — which is
exactly what made a one-time slow start look like a hang. Not fixed
in this slice; offered to the user as a small hardening change rather
than applied unprompted to a system that just got confirmed working.

No open questions left from this slice: the timeout question below was
resolved in Slice 33, the next entry.

## Slice 33 — Fixing the missing ceiling, and a second, different Groq limit found along the way

**Date:** 2026-08-30

Two things asked for directly: add the timeout hardening flagged at the
end of Slice 32, and get a fresh green CI run of the answer-quality
evals against current live code.

### The fix

Checked `ChatGroq`'s real defaults before assuming anything — its own
`model_fields` show `request_timeout=None` (the `timeout` constructor
alias) and `max_retries=2`. `None` isn't "a sane default," it's "fall
through to the underlying SDK's own default," which for an
OpenAI-style client is generous enough to be effectively unbounded for
this app's purposes. That's the actual gap: not that retries never
stop, but that a single attempt has no ceiling.

Added `groq_request_timeout_seconds` (45.0) and `groq_max_retries` (1)
to `Settings`, passed through as `timeout=`/`max_retries=` in
`ChatGroq(...)`. Chose 45s deliberately generous rather than tight — a
real request completes in single-digit seconds normally (confirmed
repeatedly this session, both locally and live), so 45s is headroom for
a legitimately slow moment, not a trap that fires on ordinary variance.
`max_retries=1` keeps the worst case at roughly two attempts × 45s
instead of stacking the library's own default of two retries on top of
an unbounded per-attempt timeout. Documented both as commented-out
overrides in `.env.example` alongside the existing `GROQ_*` vars.

Verified without spending any more of the day's Groq token budget,
which mattered given what happened next: confirmed first that nothing
under `tests/` references `ChatGroq` or `run_agent` (the suite is
fully mocked at that boundary), so re-running it wouldn't touch the
real API. `ruff check`, `mypy src/` (44 files), and `pytest -q`
(88/88) all came back clean.

### A second, different Groq limit, found by actually trying

Asked to re-run `run_evals.yml` for a clean CI result. It failed again,
but not with the per-minute (TPM) limit `SPACING_SECONDS` and
`_call_with_retry` already handle (Slices 27/28) — a different one:

```text
groq.RateLimitError: Error code: 429 - {'error': {'message': 'Rate
limit reached for model `openai/gpt-oss-120b` in organization
`org_...` service tier `on_demand` on tokens per day (TPD): Limit
200000, Used 199865, Requested 656. ...'}}
```

200,000 tokens/day on the on-demand tier, with 199,865 already used —
essentially exhausted, not almost hit. This wasn't hypothesized in
advance; it surfaced because the user actually ran the workflow and
pasted the real log, exactly the kind of evidence this project's own
verification discipline depends on rather than a check that would have
been guessed at and gotten wrong.

Read it plainly rather than reflexively retrying: this is real, shared,
daily account usage, not a per-run problem `run_evals.py` itself could
fix. A meaningful share of it plausibly came from this session's own
testing earlier the same day — a direct Groq API ping used for the
production-incident diagnosis, two local `run_agent()` calls, three
live `/ask` calls against the production backend, on top of the eval
suite's own five golden questions plus five judge calls. With ~135
tokens of headroom left at the time of the error, an immediate re-run
would almost certainly fail the exact same way, so didn't retry
immediately — the honest fix here is patience (the window rolls off,
or a calendar-day reset, depending on how Groq actually buckets "day"),
not a code change.

Recorded as a genuinely new, previously-undocumented constraint on this
project (distinct from the TPM limit already known and handled), not a
regression and not something Slice 27/28's existing retry logic was
wrong to not cover — a daily cap needs waiting, not backoff.

### Verification

`ruff check`, `mypy src/`, `pytest -q` (88/88) all clean, run fresh
after the config/provider change, without needing any real Groq call to
confirm correctness — the change is a pass-through of two typed
settings into a constructor the existing test suite already exercises
in its mocked form.

### Open questions (new)

- **Whether today's TPD exhaustion clears in the "please try again in
  3m45s" sense Groq's own error reported, or only at a calendar-day
  boundary.** Not verified either way yet; matters for deciding when a
  clean `run_evals.yml` run is worth attempting again.

## Slice 34 — "Frontend CI has no runs" turned into a real bug hunt

**Date:** 2026-08-30

Noticed while looking at the Actions tab: `frontend-ci.yml` showed zero
runs, ever. Checked before assuming anything was wrong: the workflow
was registered and `active` (GitHub's own Actions API), correctly
scoped to `paths: [frontend/**]` — it had simply never had a matching
commit since it was added. `git log` confirmed it precisely: the last
commit touching `frontend/**` landed at 15:20:19, the workflow itself
was added at 15:27:50, seven minutes later. Not a misconfiguration —
there had been nothing for it to react to yet.

### Proving it works, instead of asserting it

The honest way to confirm a never-run workflow actually works is to
give it a real trigger, not to read the YAML and declare it correct.
Added a small, genuinely-informative comment to `next.config.ts`
(documenting exactly why the file was being touched) and had the user
commit and push it.

It failed. For real, and usefully:

```text
app/layout.tsx(36,50): error TS2304: Cannot find name 'LayoutProps'.
```

### Diagnosing instead of patching around it

`LayoutProps<"/">` is not an import — it's a Next.js-generated ambient
type (the typed-routes feature), written to `.next/types` only after
`next dev`, `next build`, or `next typegen` has actually run. This
project's own local `tsc --noEmit` had been run dozens of times this
session and always came back clean, which made the CI failure
confusing for a moment — until realizing why: every one of those local
runs happened in a working directory that already had a `.next/`
directory left over from an earlier `npm run build` or `npm run dev`.
Local `tsc` was silently riding on stale build artifacts every single
time; nothing had ever actually exercised a genuinely clean checkout's
type-check step before this workflow's first real run did.

Confirmed the diagnosis by reproducing it locally rather than trusting
the theory: `rm -rf .next`, then `npx tsc --noEmit` — the exact same
`Cannot find name 'LayoutProps'` error, on this machine, on demand.
`frontend-ci.yml`'s own step order made it worse: Type check runs
*before* Build, so even CI's own later build step (which would have
generated the type) never gets a chance to save it.

### The fix, verified before trusting it

Next.js 16 ships `next typegen` specifically for this: generates only
the route/page/layout ambient types, no full build. Verified directly:
`rm -rf .next` again, `npx next typegen`, then `npx tsc --noEmit` —
clean. Added a "Generate route types" step to `frontend-ci.yml` between
Lint and Type check, with a comment explaining why it's there (the
"why," not just the "what" — the next person reading this workflow
shouldn't have to rediscover the stale-`.next/`-artifact trap to
understand the step's purpose).

Verified the whole corrected sequence end to end afterward, not just
the one step that had failed: `rm -rf .next` once more, then lint →
typegen → type check → build in the exact order `frontend-ci.yml` now
runs them, all four green.

### Why this matters beyond the one bug

This is a second real instance (after Slice 30's eval-check discovery)
of this project's own local verification silently passing for a reason
that had nothing to do with correctness — stale build artifacts masking
a real gap, the same way a wrongly-designed check masked a real bug
fix earlier. `npm run lint`/`npx tsc --noEmit`/`npm run build`, run by
hand after nearly every frontend slice this whole project, had never
once caught this, because none of those runs ever started from a truly
clean checkout. A workflow's first real run against a genuinely fresh
environment is exactly the kind of check this project's own "automate
the answer-quality suite" and "automate the RAG eval" slices were
chasing — this is the frontend getting the equivalent for free, the
moment it actually ran.

### Verification

Full corrected sequence reproduced locally from a clean `.next/` state:
lint, `next typegen`, `tsc --noEmit`, and `npm run build` all clean, in
the same order the workflow now runs them.

## Slice 35 — The fix worked; confirming it hit one more GitHub Actions surprise

**Date:** 2026-08-30

The user's own instinct to actually verify the Slice 34 fix, rather than
trust a green write-up, surfaced one more real thing worth knowing.
Clicked "Re-run jobs" on the failing run — it failed identically, same
`Cannot find name 'LayoutProps'`, and critically, the log showed no
"Generate route types" step at all, meaning the fix genuinely hadn't
run, not that it had run and failed anyway.

Diagnosed rather than assumed the fix was wrong: GitHub Actions pins a
`push`-triggered run's workflow *definition* to the commit SHA that
originally triggered it. "Re-run jobs" replays that same pinned
snapshot — it does not pick up whatever is currently on `master`, even
when the fix has already been pushed. `git log`/`git status` confirmed
the fix commit was in fact already on `origin/master`; it had simply
never had a run of its own yet.

A second, compounding reason it hadn't: the fix commit only touched
`.github/workflows/frontend-ci.yml` plus docs, nothing under
`frontend/**` — so even a fresh push of it would never have satisfied
this workflow's own `paths` filter and self-triggered.

Fixed the actual gap rather than working around it with another
throwaway frontend edit: added `workflow_dispatch: {}` to
`frontend-ci.yml`, the same manual-trigger pattern `poll_player_counts.yml`
and `refresh_catalog.yml` already use. Verified the YAML still parses
(`yaml.safe_load`) and re-ran the clean-state lint/typegen/`tsc` sequence
once more before handing it back.

### Verification

`yaml.safe_load` confirmed `workflow_dispatch: {}` parses correctly
alongside the existing `push`/`pull_request` triggers. Clean-state
sequence (`rm -rf .next`; lint, `next typegen`, `tsc --noEmit`) reran
green.

## Slice 36 — The same re-run gotcha, a second time, and the real Slice 30 confirmation it was blocking

**Date:** 2026-08-30

The user re-ran `Answer-Quality Evals` (the workflow behind the whole
Slice 24/27/28/29/30 saga) and got a confusing result: 4/5, failing on
`analysis_action_vs_f2p_not_mislabeled` again, with a failure message
that didn't match the current code at all — "expected a
compare_two_groups stats_result, got None" is the *original*, pre-Slice-30
check's wording, not the rewritten one's ("expected either a
compare_two_groups stats_result, or a plain-SQL column..."). The SQL in
the failure was genuinely correct
(`AVG(CASE WHEN price_usd = 0 THEN price_usd END) AS avg_freetoplay_price`)
— exactly the false-failure pattern Slice 30 fixed.

Recognized the shape immediately, from having just found it an hour
earlier in Slice 35: this was the identical GitHub Actions "re-run
pins the workflow to its original commit" behavior, in a second
workflow. Confirmed rather than assumed — the Actions API for this run
showed `run_attempt: 4` and a `head_sha` of `5f783fa`, the commit that
*added* `run_evals.yml` at 19:04, a full two hours before the real fix
landed in `4507500` at 21:14. Every "Re-run jobs" click had been
replaying that original, pre-fix commit the whole time.

A genuine bonus surfaced in the same run: it completed end to end with
zero Groq rate-limit errors, meaning the Slice 33 daily-quota
exhaustion had fully reset overnight — the first time this session the
full 5-question-plus-judge suite ran without hitting some real Groq
limit along the way.

Told the user to use "Run workflow" instead of "Re-run jobs" — a
genuinely fresh run against current `master`. Result: **5/5 route
accuracy, 5/5 deterministic checks**. This is the real close of the
loop that started back in Slice 24: the eval-check fix from Slice 30
had only ever been confirmed locally until this run: the first
CI-verified, ephemeral-runner, fresh-catalog confirmation that it
actually works, not just that it works on this machine.

### Verification

The run itself is the verification — a real GitHub Actions run, fresh
`--count 100` catalog, real Groq calls, 5/5 and 5/5, `head_sha`
confirmed via the Actions API to be current `master`.

## Slice 37 — The README stopped being a pitch somewhere around Slice 20

**Date:** 2026-08-30

Direct user feedback: "it's a bit too long, it doesn't really hook, it's
just very long." True, and worth understanding why before just cutting
text. The README had grown, slice by slice, the same way DOCEXP.md
grows, one honest, heavily-qualified addition at a time, and nobody had
ever stepped back to ask whether a document meant to be read cold by a
recruiter in under a minute should look like a document meant to be read
slowly by an engineer auditing every decision. Both documents existed.
Only one of them should read like a lab notebook.

### Process, not just output

Loaded `superpowers:brainstorming` before touching the file, classified
Bounded (an existing file, a well-scoped rewrite, no new subsystem), and
actually used the gate rather than treating it as ceremony: read the
whole current README (620 lines) and ARCHITECTURE.md's existing "Design
choices, named" section first, then presented a concrete section-by-
section outline in chat and stopped. Two real judgment calls surfaced
that were worth the user's input rather than assuming: where the cut
engineering-justification content should go (moved into ARCHITECTURE.md
vs. just cut), and how aggressively to trim (150 to 200 lines vs. a more
moderate pass keeping more of the existing depth). Both answered before
a single line changed.

### What actually changed

The old README's problem wasn't inaccuracy, everything in it was true
and verified, it was that the *pitch* (what is this, why does it matter,
proof it works) was buried under the *justification* (why each internal
decision was made, with slice citations). A recruiter skimming a
portfolio project needs the first two sentences to answer "is this
worth 30 more seconds," not to have already read past a parenthetical
about `INGEST_GAME_COUNT`'s SteamSpy pagination ceiling.

Rewrote around a direct hook: what separates this from a chatbot-wrapper
portfolio project, stated in the first paragraph, not implied by a
feature list. Replaced the old "What's here right now" wall of
justification-heavy bullets with six quantified "why this matters for an
AI Engineer role" bullets, each naming a real, checkable number (the
measured recall@8, the CI-run route accuracy, the real cost-per-question
figure) rather than a claim. Kept the Measured results table close to
verbatim, it was already the strongest, most concrete section in the
document. Condensed Quickstart, Architecture, MCP server, and Deploying
down to what's needed to actually use each thing, cutting the extended
"why" prose that DOCEXP.md and ARCHITECTURE.md already carry properly.
Cut the "Not in this slice" section entirely, it duplicated PLAN.md's
own "Dropped" section and mostly existed to correct claims an even
older README snapshot had made, a correction ARCHITECTURE.md/DOCEXP.md
don't need repeated a third time.

Didn't just delete the cut content: pulled the genuinely new, non-
duplicate design points into ARCHITECTURE.md's existing "Design choices,
named" section (live-computed golden-question ground truth, the eval
that checks a fact the label implies rather than parsing the model's
prose, the LLM judge never gating CI's exit code, SSE per-node
streaming over token-level streaming), so the reasoning didn't just
disappear, it moved to the document that already exists specifically to
hold it.

### A real bug the rewrite surfaced

While rewriting the example-questions list, noticed the old README's own
internal inconsistency: that list still described the forecast route as
"routed to an honest 'not supported yet'", while two other sections of
that same README correctly described `run_forecast` (Slice 9b) as a real
tool. Nobody had caught this because nobody had read the whole document
end to end in one pass in a long time, exactly the kind of thing a
620-line file makes easy to miss and a 227-line one doesn't. Fixed to
describe the actual current behavior.

### Repo cleanliness, checked rather than assumed

The user also asked about cleaning up the repo more broadly. Checked
before doing anything: `git ls-files` against the repo root showed
nothing that shouldn't be there, `desktop.ini`, `.mypy_cache`,
`.pytest_cache`, `.ruff_cache`, and `.venv` are all either already
gitignored or simply never added, never actually tracked. `data/` and
`docs/` are already minimal, exactly the files they need to be. There
was no real cleanup work hiding here; the README was the actual, entire
ask, correctly identified this time rather than manufacturing busywork
to look thorough.

### Verification

Every link and referenced file checked to actually exist before
publishing (`PLAN.md`, `ARCHITECTURE.md`, `DOCEXP.md`,
`frontend/README.md`, `LICENSE`, `.env.example`,
`frontend/.env.local.example`), and the new file scanned for em/en
dashes (none), matching the project's own no-dash convention. Line count:
620 to 227.

## Slice 38 — README follow-up: stack and framing

**Date:** 2026-08-30

Immediate feedback on the just-shipped Slice 37 README, two small,
concrete asks: add a real tech-stack section, and rename "Why this
matters for an AI Engineer role" to "AI Engineer Features." The second
one is a real, worthwhile distinction, not just wording: "why this
matters" argues at the reader; "features" is a list a reviewer scans
and judges for themselves. The bullets underneath already read as
concrete, quantified feature statements rather than persuasive prose, so
only the header needed to change, not the content.

Sourced the tech-stack section from the actual manifests rather than
writing it from memory, `pyproject.toml`'s base and `agent` extra for
the backend, `frontend/package.json`'s real `dependencies`/
`devDependencies` for the frontend, so every named tool (LangGraph,
Groq, DuckDB, fastembed, scipy, sqlglot on the backend; Next.js 16,
React 19, Tailwind v4, Motion, Recharts on the frontend) is something
actually installed and used, not an aspirational list.

### Verification

244 lines (up from 227, a real addition, not padding), scanned for
em/en dashes (none introduced).

### Same-slice follow-up: badges instead of prose

The user then asked for the stack shown as visual "squares" like a
common portfolio-site tech-stack widget, colored pill badges grouped by
category, not the paragraph form just shipped.

Rebuilt as shields.io `for-the-badge` images, grouped under real `###`
subheadings (the earlier bold-text labels had already tripped a real
markdownlint warning, `MD036`, emphasis used instead of a heading, worth
fixing properly rather than leaving a linter complaint in a document
meant to look polished).

Didn't trust that a badge URL resolving (HTTP 200) means the logo
actually renders, shields.io silently omits an unrecognized `logo=`
slug rather than erroring, so a broken icon wouldn't show up as a fetch
failure. Built a local HTML page with every candidate badge and rendered
it through a real headless-Chromium screenshot (Playwright, the same
ephemeral `uv run --with playwright` pattern from Slice 31/33, hit the
same TLS-interception wall from DOCEXP's Slice 9b history along the way
and fixed it the same documented way, `--native-tls`) before trusting
any of them. Caught one real inaccuracy this way: the only available
"SQL" badge icon was PostgreSQL's elephant, and this project uses
DuckDB, not Postgres, keeping it would have been a small but real,
avoidable misrepresentation. Dropped rather than kept for badge-count's
sake.

### Verification

Every candidate badge visually confirmed rendering its correct icon in
the screenshot before being added. The markdown linter's `MD036`
warnings on the Tech stack section resolved by using real subheadings.
275 lines, no em/en dashes introduced.

## Slice 39 — "Let's try some cool glass" turned into two real dependency-integration investigations

**Date:** 2026-08-30

The request started simply: an Apple-style liquid-glass effect on the
Ask button, specified precisely (`feTurbulence type="fractalNoise"` into
`feDisplacementMap`, fed into `backdrop-filter`, with an edge highlight).
Built exactly that first: a `#liquid-glass` SVG filter defined once in
`app/layout.tsx`, a `.glass` utility in `globals.css` applying it via
`backdrop-filter: url(#liquid-glass) blur(8px) saturate(150%)`, with a
plain `blur()`/`saturate()` value declared *first* as the real fallback
-- Safari and Firefox don't support an SVG filter reference inside
`backdrop-filter` at all, and CSS discards an entire declaration it
can't parse, not just the unsupported part, so the fallback has to be a
separate, earlier declaration of the same property, never merged into
one value. Applied to the ask bar and the sticky nav. Lint clean, done.

Then came the actual pivot: two named GitHub repos, `liquid-glass-js`
for the Ask button specifically, `shadergradient` for an animated
green/turquoise/black background. Neither got installed on the strength
of the user naming them -- both got read first.

### Checking fit before integrating, twice

`liquid-glass-js` turned out not to be an npm package at all: plain
`<script>` tags, WebGL 2.0, `html2canvas` sampling the DOM into a
texture. Fetched its actual README, then (once the README's own summary
raised real questions) its actual source files directly, not another
paraphrase. `shadergradient` was the opposite story: a real, standard
React package, but pulling in three.js + `@react-three/fiber` +
`camera-controls`, roughly 600KB of new dependencies in a project that
had shipped zero 3D/WebGL libraries until this slice, and whose own
README (rewritten just one slice ago) leads with restraint as a
differentiator.

Presented both trade-offs plainly rather than picking silently: for the
glass effect, keep the already-built native version, or attempt the
real library despite the DOM-sampling mismatch with a continuously
scrolling, dynamically updating page; for the background, a lightweight
CSS gradient, or the real three.js-backed shader. The user chose the
heavier option both times. Both got built for real, not built partway
and hand-waved.

### shadergradient: a real dependency, tuned against a real screenshot

Installed `@shadergradient/react@2.4.20`, `@react-three/fiber@9.7.0`
(the v9 line specifically -- the package's own docs are explicit that
R8 is structurally incompatible with the App Router's React 19), `three`,
`three-stdlib`, `camera-controls`. Read the installed package's own
shipped `index.d.mts` for the real `ShaderGradient`/`ShaderGradientCanvas`
prop surface (`type`, `color1`/`color2`/`color3`, `uSpeed`/`uStrength`,
`cDistance`/`cPolarAngle`, `grain`, etc.) rather than guessing at what a
"waterPlane" preset needed.

The first real screenshot of it running (not assumed correct from the
code) showed a real problem: full-brightness, saturated stock colors
made the hero's own body copy close to unreadable against the moving
background. Fixed with two independent changes, deliberately not just
one: dimmed and darkened the shader itself (`brightness` down from 1.1
to eventually 0.85, `color1`/`color2` swapped for deep forest-green and
teal instead of the bright accent hex directly), and added a second,
separate layer -- a semi-transparent radial-gradient scrim between the
canvas and the actual page content, shaped darkest directly behind the
text column and lighter toward the edges, so contrast doesn't depend on
catching the shader at a dim frame. Re-screenshotted after each real
change until the hero text was cleanly legible with the gradient still
visibly present and moving, not tuned once and assumed good.

### liquid-glass-js: read the real source before writing a line of integration code

This is the slice's real depth. Rather than trust the README's own
AI-generated summary of "how to use it," fetched `container.js` and
`button.js` directly and read them in full. Three concrete facts came
out of that read that changed the plan:

1. **`addChild()` re-parents whatever DOM element you pass it** into the
   library's own tree. Doing that to an existing React-rendered node is
   exactly the kind of DOM surgery that crashes React's reconciler later
   (`NotFoundError: Failed to execute removeChild`) when React tries to
   remove a node from where it thinks that node still lives.
2. **The glass refracts a single `html2canvas` snapshot of the page**,
   cached statically across every instance (`Container.pageSnapshot`),
   not a live re-render -- `startRenderLoop()` renders once, then only
   again on `scroll` events, not every frame. Cheaper than first feared,
   but also meant a naive "wrap the ask bar" plan would go visibly stale
   the moment a result streamed in below it.
3. **There is no public destroy or cleanup method at all.** The scroll
   listener the render loop registers on `window` is never removed by
   the library itself, for the life of the page.

Surfaced all three to the user with a concrete recommendation (skip it,
or use it somewhere genuinely standalone) before writing any React code
-- this was a case where the *first* answer ("try it anyway") had been
given before these specific facts were known, so it was worth asking
again with the real information rather than silently building around it
or silently overriding the user's stated preference. Told again to force
it onto the Ask button anyway, with the risks understood.

### Building around the constraints instead of ignoring them

Given `addChild()`'s re-parenting behavior, `LiquidGlassAskButton.tsx`
never wraps the existing `<button>`. React owns exactly one thing: an
empty mount `<div>` via a real `useRef<HTMLDivElement>`. The library's
`Button` instance is constructed imperatively inside a `useEffect` and
appended into that div; every other effect that needs to touch it
queries fresh via `mountRef.current?.querySelector(...)` rather than
threading a stored class-instance ref across effects.

That last detail wasn't stylistic. The project's installed
`eslint-plugin-react-hooks@7.1.1` ships the newer, stricter React-
Compiler-era rules (`react-hooks/refs`, `react-hooks/immutability`), and
a first draft that stored the `Button` instance in its own ref and
mutated properties on it from a second effect hit real lint *errors*,
not warnings: "Cannot access refs during render" and "This value cannot
be modified." Restructured to the single-DOM-ref-plus-querySelector
pattern specifically because that's the one the rule is built to trust,
not because it was the only way to write the code.

Vendored `container.js`/`button.js` under
`public/vendor/liquid-glass/` with a real `LICENSE` file (fetched the
actual MIT text and copyright holder from the repo, not assumed),
loaded in the strict order the library actually requires -- html2canvas,
then container.js, then button.js -- by a new `lib/loadLiquidGlass.ts`
that memoizes the loading promise so React Strict Mode's double-invoked
effects in dev don't inject the scripts twice. `glass.css` went into the
real bundled source tree (`app/liquid-glass.css`) instead of staying a
vendored static asset loaded via `<link>`, once Next's own lint rule
(`@next/next/no-css-tags`) objected to a raw stylesheet tag and a plain
`import` turned out to be the correct fix anyway, since CSS has none of
the load-order constraints the two scripts do.

### Two real bugs, found by actually running it, not by re-reading the code

**Bug one**: after wiring everything up, `.glass-button` count was zero
in a real Playwright run. `window.Button` was `undefined` even though
the console showed all three scripts loading successfully. The actual
cause, confirmed by checking the vendored source's first line
(`class Button extends Container`, at the top level of a classic,
non-module script): a top-level `class` declaration creates a binding in
the shared global *lexical* scope, reachable as a bare identifier from
another classic script in the same document, but it is NOT added as a
property of `window` -- unlike `var`, which is. Fixed with a small
inline bridging script (`window.Container = Container; window.Button =
Button;`) that runs in that same shared scope and explicitly assigns
them where the app's real ES modules can reach them.

**Bug two**, found immediately after fixing the first one and actually
looking at the result: the button displayed "Asking..." (the long
placeholder text used purely for initial pill sizing) on first page
load, instead of "Ask." The `useEffect(() => {...}, [label])` meant to
keep the button's text synced ran synchronously right after mount, long
before the async script-loading chain (html2canvas, then two more
scripts, then a bridge script) had actually created the button element
-- so it found nothing in the DOM and no-opped, and never got a second
chance to run since `label` itself never changed again until the user's
first real interaction. Fixed by keeping `label`/`disabled` in
always-current refs and applying them immediately once the button is
actually created, not relying solely on the prop-change effect to catch
up.

### A real accessibility regression, caught before it shipped

The `<button>` this replaced was keyboard-accessible for free. The
library's generated `<div class="glass-button">` has no `tabindex`, no
`role`, and no keyboard handling of any kind -- confirmed by reading the
source, not assumed from the demo working with a mouse. Patched
`tabIndex`, `role="button"`, `aria-label` (kept in sync with the visible
label on every change), `aria-disabled`, and a real `keydown` handler
for Enter/Space directly onto the generated element, sharing the exact
same `activate()` function the click handler uses, itself gated on a
`disabled` check -- `pointer-events: none` alone blocks a mouse click
while disabled but does nothing to stop a keyboard Enter press, so the
guard had to live in the activation function itself, not just in a
style.

### Verification

Full local stack actually running (real backend, real frontend dev
server), not just `lint`/`build`: a real Playwright click through the
actual liquid-glass button, through the real streaming `ask()` flow, to
a real Groq-backed answer, zero page errors, button text confirmed
cycling Ask -> Asking... -> Ask correctly. A second, separate Playwright
run confirmed real keyboard access: `tabIndex: 0`, `role: button`,
`aria-label: Ask`, focus landing on the correct element, Enter
triggering the same activation path. `ruff`, `mypy`, `eslint`, `tsc`,
and a full `next build` all clean throughout. Both libraries credited
with real links and real explanations in `frontend/README.md`, which
also had its own now-false claim ("No WebGL/3D, no fixed animated
background layer") corrected to match what actually shipped, and the
root README's Tech Stack gained a Three.js badge, verified rendering
the same way every other badge in that section was.

### Open questions (new)

- **liquid-glass-js's permanent per-mount scroll-listener leak.** Real,
  documented, and left as-is: the library has no destroy method to call.
  Bounded for a page most users won't repeatedly navigate away from and
  back to within one session, but worth knowing if this pattern is ever
  reused elsewhere in the app.

## Slice 40 — "Should we remove forecast?" turned into finding a real, fixable bug instead

**Date:** 2026-09-04

A screenshot showed the actual failure this time, not a description of
one: "How many players will Counter Strike have next year?" came back
"there are no historical live-player snapshots for that game" -- a
direct contradiction of Slice 33's own finding that the poller had
already accumulated 24 real snapshots by this point. The user's
conclusion, reasonably, was "it doesn't work, should we remove it," and
separately, a fair complaint that the example-question pill "How many
players will this game have next year?" is a bad canned example -- it
doesn't name a game, so it's unanswerable as written.

### Diagnosing instead of accepting the surface reading

The instinct to defend the forecast tool (it had just been proven
working, twice, a few messages earlier) would have missed the point.
Reproduced the exact failing question locally instead, where the real
tool-call arguments and errors are visible, not just the final answer
text a screenshot shows. Got `tool_errors: 1` -- a real failed tool
call, not a false claim manufactured from nothing.

### Two hypotheses, one disproven by actually testing it

First hypothesis: multiple real "Counter-Strike"-named games (there are
six: the original, Source, two rows both labeled Condition Zero, Global
Offensive, and a Nexon spinoff) make the prompt's own scalar-subquery
pattern (`WHERE appid = (SELECT appid FROM games WHERE name ILIKE
'%...%')`) ambiguous, and DuckDB silently resolving to the wrong one.
Tested this directly against the real DB rather than trusting the
theory -- and it was wrong: `SELECT appid FROM games WHERE name ILIKE
'%Counter Strike%'` returns zero rows, not six. The real cause is
simpler and more fundamental: `ILIKE` is a literal substring match, and
the real title is `'Counter-Strike: Global Offensive'` -- a hyphen and
a colon, not spaces. "Counter Strike" (what a person naturally types)
never matches it at all. Not a data problem (the 24 snapshots for
appid 730 are real and correct, confirmed directly), not fundamentally
an ambiguity problem -- a punctuation-literalism bug, present in
`FORECAST_TOOL_GUIDANCE`'s own worked example, which taught exactly
this fragile pattern.

### Verifying the fix before writing a word of prompt text

Two claims, both tested against the real database before being
written into guidance meant to shape a model's behavior:

1. `REPLACE(REPLACE(name, '-', ' '), ':', ' ') ILIKE '%counter
   strike%'` -- confirmed it finds all six real "Counter-Strike" rows,
   where the naive version found zero.
2. That reintroduces the original six-way ambiguity, so also confirmed
   `ORDER BY peak_ccu DESC LIMIT 1` reliably picks the right one:
   Counter-Strike: Global Offensive at 1,013,936 peak_ccu, an order of
   magnitude past the next-highest real contender (Counter-Strike:
   Source, 7,323). Not a close call this heuristic could get wrong for
   this game or, plausibly, most others: the currently-popular variant
   of a franchise is almost always what "the game" means in a casual
   question.

### Fixed in the leveraged place, not just the obvious one

`FORECAST_TOOL_GUIDANCE`'s worked example got the direct fix. But the
same fragile `ILIKE '%name%'` pattern is exactly what `run_sql`/
`run_stats` would reach for too, for any lookup or analysis question
naming a specific game -- this was never actually forecast-specific,
just first noticed there. Fixed it in `column:name`'s schema chunk
instead of only in the forecast guidance: that chunk is marked
`always_include=True` (added in an earlier slice specifically because
semantic retrieval alone under-ranks a plain "Game title" description
against a question naming a specific title), meaning every single
question, regardless of route, sees this guidance now, not just
forecast-routed ones. One fix, reused everywhere the same failure mode
could otherwise recur silently.

Checked before assuming the schema chunk edit was safe: Slice 30 (and
Slice 22's own retrieval eval) already learned the hard way that
editing schema-chunk text can shift its embedding enough to change
retrieval ranking. Re-ran `tests/test_retrieval_eval.py` after the
edit -- still at its locked baseline, not regressed.

### Confirmed fixed, not assumed fixed

Re-ran the exact original failing question after the prompt change:
`tool_errors: 0`, the real query now includes the normalize-and-
disambiguate pattern verbatim, resolves to the correct game, and
returns a real forecast. Pushed the test further on purpose, asking for
a 365-day projection from ~11 days of real history (a genuinely
unreasonable extrapolation) specifically to check the tool's *honesty*
under a harder case, not just its happy path: it correctly floors the
negative-slope projection at 0 and reports `low_confidence: true` with
the specific, correct reason ("projecting 365 day(s) ahead from only
10.81 day(s) of observed history"), and the model's own answer
faithfully passed that caveat through rather than stating 0 with false
confidence. That is the tool behaving exactly as designed under stress,
not a new failure to explain away.

### The example-question fix

Replaced "How many players will this game have next year?" (unanswerable
as written) with "How many players will Counter-Strike: Global
Offensive have next month?" -- the real catalog name, a horizon close
enough to the real observed history to give a confident answer rather
than an immediate low-confidence caveat, and now genuinely verified
working, not just plausible-sounding. Left `router.py`'s similarly-
worded "will this game have next month" line untouched -- it is a
classification-pattern example teaching the router what forecast-intent
phrasing looks like, not a question anyone is meant to literally send,
a different and legitimate use of near-identical wording that doesn't
share the pill's problem.

### Verification

`ruff`, `mypy`, the full 88-test `pytest` suite, and
`tests/test_retrieval_eval.py` all clean/at-baseline after the prompt
and schema-chunk changes. Frontend `eslint`/`tsc` clean after the
example-question change. The real end-to-end re-test against the actual
24-snapshot local catalog is the substantive verification, not the
linters -- a real tool call, zero errors, a real number, an honest
confidence caveat.

## Slice 41 — Explaining a "0", then fixing why the live site was stale at all

**Date:** 2026-09-04

The user tried the just-fixed forecast question live, on the deployed
Vercel frontend, and got a real result: "Counter-Strike: Global
Offensive is projected to have about 0 concurrent players... based on
only eight live-player snapshots covering roughly 6.3 days." Two
things worth separating in that one screenshot, and the second turned
out to matter more than the first.

### Explaining the "0" honestly, not defensively

A big "0 players" for one of Steam's most-played games looks alarming.
Explained the actual mechanism rather than reflexively reassuring:
`_forecast()` fits a line through the real (short, noisy) history,
found a negative slope, and `projected = max(0.0, fit.slope * x_target
+ fit.intercept)` floors an extrapolation that would otherwise go
negative, an impossible player count. Paired with R²=0.063 and
`low_confidence: true`, the "0" is the model saying "this trend, taken
literally 30 days out, breaks down," not "this game will have no
players." That's the tool being honest about a bad fit, exactly its
job description, not a new failure to explain away.

### The more important question: why only 8 snapshots?

By the time this screenshot was taken, the real, committed history was
already 24+ snapshots spanning 10.8+ days (confirmed directly, Slice
40). 8 snapshots over 6.3 days is real, but stale data. Asked directly
which environment produced it: the live Vercel-deployed frontend.
Worth naming precisely, since it's an easy thing to conflate: Vercel
only serves the frontend. The forecast computation and `player_counts`
itself live entirely on the Render backend, which only ever sees new
snapshots at Docker build time. The frontend having already picked up
the new example-question pill (a real, recent push) said nothing about
whether the *backend* had rebuilt recently enough to reflect the newer
snapshot commits.

### Root-caused the actual mechanism, not just noted it as expected

Read `refresh_catalog.yml` again rather than assume it was already
tuned for this: it's the *only* thing that ever triggers a Render
rebuild, and its own comment explains why it was set to weekly --
catalog facts like price and reviews change slowly. That reasoning was
sound when written (Slice 7), but `player_counts` was folded into the
same trigger without revisiting the cadence, and player counts change
far faster than catalog metadata. A live forecast could legitimately
lag the real accumulated history by up to 7 days, exactly what this
screenshot caught in the act.

### Checking real costs before recommending a fix

The instinct to just match `poll_player_counts.yml`'s own 6-hour
cadence (freshest data, simplest mental model) got checked against
Render's actual free-tier documentation first, not assumed free the
same way GitHub Actions is on this public repo. Real finding: Render's
free tier has an *undisclosed* monthly cap on build-pipeline minutes,
and exceeding it bills for the overage unless a spend limit is set.
Each rebuild is a full Docker build plus ingestion, roughly 20 to 30
real minutes. Six-hourly would mean roughly 4x the build-minute spend
of daily, against a cap whose actual size isn't published anywhere
this session could find. Presented three real cadences (weekly/as-is,
daily, every 6h) with that specific cost tradeoff named plainly, rather
than picking the technically-purest option and hoping the free tier
absorbed it.

### The fix

Changed `refresh_catalog.yml`'s schedule from `0 3 * * 1` (weekly,
Monday) to `0 4 * * *` (daily, 04:00 UTC). Deliberately not matched to
the poller's 6-hour cadence, for the cost reason above -- daily closes
the real staleness window from up to a week down to at most a day,
at a small, predictable ~30 build-minutes/day instead of scaling
linearly with every single poll.

A small, worth-naming verification wrinkle: a first `yaml.safe_load`
check of the edited file returned `None` for the `on:` key, which
looked like a real parse failure for a second. It wasn't -- PyYAML
(like most YAML 1.1 parsers) coerces the bare, unquoted key `on` to the
boolean `True`, a well-known quirk of that spec's implicit typing
(`on`/`off`/`yes`/`no`/`true`/`false` are all boolean literals unless
quoted). Re-checked against `d.get(True)`, which returned the real
`schedule`/`workflow_dispatch` structure correctly. GitHub's own
Actions YAML parser is unaffected by this, it treats `on:` as the
trigger keyword regardless, so the workflow itself was never actually
broken, only this particular verification script's naive read of it.
Worth remembering the next time a `.github/workflows/*.yml` file needs
a Python-side sanity check.

### Verification

YAML re-validated correctly via `d.get(True)`. No other file in the
repo referenced the old weekly cadence in prose that would now be
stale (checked directly, not assumed). The real test of this fix is
external and can't be verified from inside this session: whether
tomorrow's 04:00 UTC run actually redeploys the backend with fresher
data, since this session has no Render dashboard access to confirm a
build's real outcome.

### Open questions (new)

- **Render's actual build-pipeline-minute allowance is still unknown.**
  Not found in public docs; only checkable from the account's own
  billing page. Worth confirming directly at some point that daily
  rebuilds comfortably fit within it, rather than assuming the
  conservative choice is automatically safe.

## Slice 42 — Forecast latency traced to a real crash, and a full honest audit

**Date:** 2026-09-05

Asked to improve forecast latency and, separately, for a full structural
audit with a score out of 10 against a 9/10 goal. Both turned out to be
more connected than expected, and the audit is recorded here in full,
not just summarized in PLAN.md.

### Measuring instead of guessing at latency

Ran all three routes for real before touching anything: lookup 16.3s,
analysis 3.8s, forecast 56.4s (`attempts: 2, tool_errors: 1`). The gap
wasn't raw model speed, it was forecast's own self-correction loop
firing, a real, avoidable failed-attempt-plus-full-retry cost, not
inherent slowness.

### The real root cause, and how it traces back to this project's own last slice

Captured the raw message trace (invoking the compiled graph directly,
not just reading `AgentResult`'s final state) to see the actual failed
tool call, not just its aftermath: `groq.BadRequestError: Failed to
parse tool call arguments as JSON`, with the failed generation showing
the model backslash-escaping single quotes inside a SQL string value.
JSON never requires escaping a single quote, and doesn't allow a bare
`\'` at all, that's what failed to parse.

Directly traceable to Slice 40's own name-matching fix: the taught query
pattern (`REPLACE(REPLACE(name, '-', ' '), ':', ' ')`) has five separate
single-quoted literals in one tool call, real quote density that
apparently raises the odds of the model mis-encoding at least one of
them. Fixing a real correctness bug two slices ago introduced a real
latency regression this slice, worth naming plainly rather than only
crediting the fix and ignoring its cost.

Simplified to `regexp_replace(name, '[-:]', ' ', 'g')` (verified
identical correctness against the real DB first) and added an explicit
instruction about JSON's actual escaping rules, in both
`FORECAST_TOOL_GUIDANCE` and the always-included `column:name` schema
chunk (the same chunk every route reads, so this reasoning benefits
lookup/analysis too, same leverage decision as Slice 40).

### A second, more serious bug the retest surfaced

Re-testing to check the fix, a *different* Groq rejection appeared:
`Tool choice is none, but model called a tool` -- and this one crashed
`run_agent()` outright, an unhandled exception, not a caught
self-correction retry. Traced to the actual code: `agent_node`'s
`model.invoke()` call had no error handling at all.
`execute_tools_node`'s self-correction try/except only ever covered a
tool that ran and failed (bad SQL, etc.); a failure of the model call
itself, before any tool call was even validly formed, had never had a
path back into the retry loop, for the entire history of this project.
In production that means a real user hitting this exact failure got a
generic "high demand" 503 from `main.py`'s broad exception handler,
with no chance for the self-correction system built for exactly this
kind of recoverable friction to actually engage.

Fixed directly: `agent_node` now catches that failure, retries once
with no tools bound (sidestepping the exact JSON-escaping failure mode
by forcing a plain-text response), and degrades to a clean, honest
answer if that also fails, incrementing `attempts`/`tool_errors`
consistently with how `execute_tools_node` already does, so `/health`'s
self-correction statistics keep meaning what they claim to mean. Used a
deliberately broad `except Exception` here (with `# noqa: BLE001` and a
comment explaining why), unlike the narrower, specific exception types
`execute_tools_node` catches: there's no portable exception hierarchy
across LLM providers to catch instead, and every failure at this exact
point is equally unrecoverable without a retry regardless of its type.

Added 4 real regression tests (`test_agent_node_llm_errors.py`), mocking
`get_llm()` rather than hitting a real provider, matching this
project's own `live`-test-exclusion convention: a model that fails then
recovers, a model that fails twice and degrades honestly, a clean call
that never touches the counters, and the already-at-the-retry-cap case.
92/92 tests pass, `ruff`/`mypy` clean.

### Verified honestly, not oversold

Confirmed the crash is genuinely fixed: a real repeat run against the
exact original failing question completed without an exception,
`tool_errors: 2` on both of two repeat runs but degraded gracefully
each time rather than crashing. Worth stating plainly rather than
declaring victory: the prompt simplification did NOT eliminate the
underlying JSON-escaping tendency, only reduced its quote-density
surface somewhat. This is a probabilistic model behavior, not a
deterministic bug a prompt tweak alone can fully stamp out. What's
actually fixed is that this failure mode can no longer crash the
system, not that forecast questions are now uniformly as fast as
lookup/analysis. A bigger structural fix (e.g. not using native
tool-calling JSON for this specific query, parsing a plain-text
response instead) exists as a real option but wasn't undertaken here,
a bigger architectural change than this slice's scope, and one this
session did not have standing approval for.

### The full audit

The user asked to be treated like a senior AI engineer giving an honest
score, not a reassuring one. This is that assessment, in full, not
summarized:

**Real, genuinely uncommon strengths:**

- A real SQL safety boundary (`sqlglot`-parsed, SELECT-only,
  table-allowlisted) independent of the prompt, not "the model was told
  not to." Verified directly with a real `DROP TABLE` attempt.
- Measured RAG retrieval quality (1.000 recall@8), gated in CI as a
  free, no-LLM-call regression test, not just "we added RAG."
- A genuinely real eval methodology: golden questions with ground truth
  computed live from the DB (not hardcoded), deterministic checks, and
  an LLM judge that never gates the exit code, automated in CI.
- Real, live cost and token accounting per request, not an
  after-the-fact estimate.
- A self-correction loop with a real, meaningful metric
  (`tool_errors` vs `attempts`) distinguishing genuine multi-step
  tool use from actual recovered failure, not just a retry counter.
- Two real interfaces (the web app and an MCP server) sharing one
  safety boundary, not two implementations to keep in sync.
- Documentation discipline (PLAN.md/DOCEXP.md/ARCHITECTURE.md) that is
  genuinely rare even in production codebases, let alone portfolio
  projects, including a real, visible track record of naming its own
  mistakes rather than quietly fixing and hiding them.

**Real, structural gaps, measured, not guessed:**

- **Test coverage is 46% overall** (measured via an ephemeral
  `pytest-cov` install, not assumed from "92 tests" alone), and the
  gap isn't evenly distributed: the entire API layer (`main.py`,
  `rate_limit.py`, `run_stats.py`, `schemas.py`), the semantic cache,
  the MCP server, and the *entire eval harness itself*
  (`checks.py`, `golden_questions.py`, `judge.py`, `run_evals.py`) sit
  at 0% direct unit coverage. The parts of this system doing the most
  "interesting" reasoning (the agent graph, the tools) are reasonably
  tested; the parts every real request and every eval run actually
  passes through are not tested at all.
- **This gap is not hypothetical**, it already caused a real incident:
  Slice 30's whole saga (a wrong eval check silently failing the same
  golden question across four separate slices before the check itself,
  not the model, was found to be the bug) happened specifically because
  no unit test asserted the check function's own correctness in
  isolation. A test asserting `_check_action_vs_f2p_not_mislabeled`
  passes a plain-SQL correct answer would have caught this the first
  time, not the fourth.
- **The golden question set is 5 questions.** The methodology (live
  ground truth, deterministic + judge) is sound, but 5/5 route accuracy
  is a single flipped answer away from 4/5 either direction, a smoke
  test's statistical weight, not a real eval suite's. Real confidence
  in "does this agent work" needs a meaningfully larger, more diverse
  set.
- **Production resilience had a real, un-tested blind spot until this
  exact slice.** The self-correction story only ever covered tool
  execution failures; a model-call failure crashed the whole request,
  for this project's entire history, until today. The eval harness's
  own retry logic (`_call_with_retry` in `run_evals.py`) was never
  shared with the production path at all, meaning the thing that tests
  reliability and the thing that actually serves users had different,
  disconnected resilience stories.
- **A recurring process gap, not a one-off bug**: the GitHub Actions
  "re-run pins to the original commit" gotcha was independently
  rediscovered twice (Slices 34/35 and 36), and the Render deploy-
  staleness issue was independently rediscovered twice (Slices 32/33
  and 40/41). Both times, the underlying lesson was documented after
  the fact, but nothing was built to *detect* the same class of
  staleness proactively (e.g. `/health` reporting the git SHA/build
  time it was actually built from) — both recurrences were found by a
  human noticing something looked off, not by an automated check.
- **The frontend just took on a real, quantified regression against its
  own stated design values.** Slice 39 added three.js/shadergradient:
  measured at 2.4MB total client JS with a single 1.1MB chunk, for a
  homepage background flourish, loaded unconditionally for every
  visitor (no lazy-loading or below-the-fold deferral applied). Paired
  with liquid-glass-js's own documented, permanent per-mount memory
  leak (no destroy method exists), both were deliberate, informed
  trade-offs at the time, not accidents, but a senior review has to name
  the tension directly: a project whose own README leads with restraint
  as a differentiator just spent real, measured weight and a known leak
  on two decorative flourishes.
- **No authentication on `/ask`**, a real, cost-incurring LLM endpoint,
  reachable directly, bypassing the frontend entirely. Per-IP rate
  limiting exists but is in-memory (reset by every redeploy, now daily)
  and trivially bypassed by multiple source IPs. Given this project
  already had one real incident where a shared daily Groq quota got
  exhausted (Slice 33), this is a live, not hypothetical, exposure.

**Score: 7/10.**

Genuinely above-average, thoughtfully engineered, with real,
verifiable rigor in the areas its author chose to focus on (agent
design, RAG, the eval concept, documentation). Not yet a 9: the
sophistication of the "interesting" parts of this system is
meaningfully ahead of the rigor applied to the "boring but critical"
parts, the actual serving path, its test coverage, and its production
resilience, and that gap is exactly where a real incident (Slice 30)
already happened once and where this exact slice found a live,
previously-unhandled crash path.

**Concrete path to 9/10, in priority order:**

1. Test the serving layer, not just the reasoning layer: real coverage
   on `main.py`, `rate_limit.py`, `run_stats.py`, `cache.py`, and the
   eval harness's own check functions. This single item closes the
   biggest, most consequential gap and directly prevents a Slice-30-
   shaped incident from recurring in a part of the codebase that
   hasn't been audited this way yet.
2. Grow the golden question set from 5 to something like 20 to 30 real,
   diverse questions, so route accuracy and deterministic-check numbers
   carry real statistical weight instead of smoke-test weight.
3. Add a deploy-freshness guard (`/health` reporting the actual git SHA
   and build timestamp it was built from, checked by a scheduled
   workflow) so the staleness class of bug that recurred twice gets
   caught automatically, not by a human noticing something looked off
   a third time.
4. Share one real retry/backoff utility between `run_evals.py` and
   `agent_node`'s new resilience path instead of two separate,
   differently-mature implementations of "handle a transient provider
   failure."
5. Lazy-load or defer the shadergradient canvas (an intersection
   observer, or the library's own `lazyLoad` prop) instead of paying
   its full weight unconditionally for every homepage visitor.
6. Add a lightweight cost ceiling on `/ask` beyond per-IP rate limiting
   (a global daily token/request budget, or a simple API key), given
   the endpoint is directly reachable and this project has already
   once run into a shared quota limit for real.

### Verification

Latency measurements, the raw trace capture, the fix's own correctness,
and the coverage/bundle-size numbers underlying this audit were all
measured directly against the real local environment this session, not
estimated or recalled from memory. `ruff`, `mypy`, and the full 92-test
suite (88 existing + 4 new) all clean after the `agent_node` fix.

## Slice 43 — Working the 9/10 list, item one: the serving layer's tests

**Date:** 2026-09-05

Chose to work the audit's path-to-9/10 list one item at a time rather
than everything at once, starting with test coverage, the item with the
most concrete evidence behind it (the Slice 30 incident) and the
clearest scope (the specific files the audit named at 0%).

### Starting where the incident already happened

`evals/checks.py` got the first, and most carefully written, tests.
Slice 30's whole saga was a wrong check failing the same golden question
across four slices before anyone realized the check itself, not the
model, was broken — and it took that long specifically because nothing
had ever asserted the check function's own correctness independent of a
real LLM run. Wrote both real paths of
`_check_action_vs_f2p_not_mislabeled` directly: `compare_two_groups`
correct and mislabeled, plain-SQL correct and mislabeled, and the
fallback case, all via a hand-built `AgentResult`, no LLM, no DB. This
is, concretely, the test that would have caught the Slice 4 bug (or
proven the Slice 24/27/28 "failures" were false alarms) on the very
first run, not the fourth.

### The rest of the serving layer

`run_stats.py`, `rate_limit.py`, and `cache.py` each got their own
focused test file, pure logic or a real local embedder, no mocks of
this project's own layers, same standing rule the DB-fixture tests
already followed. `run_stats.py`'s tests deliberately pin down the exact
edge case its own docstring calls out (a long, error-free tool chain
that happens to hit the attempts cap must never be misclassified as a
self-correction failure) and the "recovered" rate's real intersection
semantics (never able to go negative by construction, now actually
verified, not just argued for in a comment).

`test_api_main.py` was the biggest piece: FastAPI's `TestClient`
against the real app, `run_agent`/`stream_agent` mocked (no real LLM
calls, matching this project's `live`-exclusion convention), covering
`/health`, every DB-backed route's missing-DB 503, `/ask`'s real
response shape, both branches of debug-mode error detail, and
`/ask/stream`'s SSE framing for the normal and the error case.

### A second real bug, found the same way the first one was

Writing MCP server tests (the last 0% item on the list) surfaced
something genuinely unplanned: a query too malformed to even parse
(`"SELECT this is not valid sql at all"`) crashed both `run_sql`'s test
and, on inspection, would have crashed the real MCP tool call and the
real agent's `execute_tools_node` identically. Checked why directly
rather than patching the test to dodge it: `sqlglot.parse()` raises its
own `sqlglot.errors.ParseError`, confirmed via its actual `__mro__` to
not be a `ValueError` subclass, so it passed straight through
`execute_tools_node`'s and the MCP server's identical
`except (UnsafeQueryError, duckdb.Error, ValueError)` catches, both
uncaught, the same class of gap Slice 42 found and fixed a few hours
earlier in a different node (`agent_node`'s model-call failure).

Fixed once, at the root, not at each of the two call sites that happened
to be found: `_parse_single_select` in `db/connection.py` now catches
`sqlglot.errors.ParseError` and re-raises as `UnsafeQueryError`, which
already *is* a `ValueError` subclass, so every existing catch site
(and any future one) gets the fix automatically. This is exactly the
"fix it once where the safety boundary already lives" pattern this
module's own docstring already commits to; extending that principle to
error handling, not just the parsing logic itself, was the right call
rather than special-casing this one exception type at each caller.

Added the direct regression test at the actual source
(`test_sql_guard.py`), not just the MCP test that happened to surface
it, so the fix is pinned down where it structurally belongs.

### Verified with real numbers, not the "add tests until it feels done" instinct

Installed `pytest-cov` ephemerally (not a new permanent dependency) and
ran a real coverage report before and conceptually compared it to the
audit's own numbers: **46% to 69% overall**, and specifically, every
file the audit named at 0% is now at 79-100%
(`main.py` 79%, `rate_limit.py`/`run_stats.py`/`schemas.py`/`cache.py`
100%, `mcp_server/server.py` 96%, `evals/checks.py` 100%). 153/153 tests
pass (88 to 153, all new tests additive, nothing removed or weakened),
`ruff`/`mypy` clean throughout.

Named what this did NOT close, rather than letting a big percentage
jump imply everything was solved: `evals/judge.py` (needs a real LLM
call to test meaningfully) and `evals/run_evals.py` (orchestration,
harder to unit test in isolation) are still at 0%, along with a few
ingestion scripts. These weren't the specific files the audit named,
so treated as a real, separate, lower-priority follow-up rather than
folded into this item's "done" claim.

### Open questions (new)

- **Whether `evals/judge.py`/`run_evals.py` and the remaining 0%
  ingestion scripts are worth a dedicated follow-up item**, or
  acceptable as-is given their harder-to-unit-test nature (real LLM
  calls, thin orchestration). Not decided; flagged for whenever the
  9/10 list's first six items are otherwise exhausted.

## Slice 44 — Working the 9/10 list, item two: growing the golden set found three more real bugs

**Date:** 2026-09-05

Item two on the audit's list: 5 golden questions "carries real statistical
weight as a smoke test, but not much more." Chose 15 as the target size
(4 lookup / 4 analysis / 4 needs_clarification / 3 forecast) after being
upfront about the real cost tradeoff — more questions means more real
Groq calls, and this account runs on the free tier's daily token cap.

### Growing the set the same way the original one was built

Every new reference fact is computed live from the DB inside
`build_golden_questions()`, never hardcoded — Strategy's real average
price, Linux game count, the free-to-play/paid review-score comparison
(deliberately chosen over Action-vs-Strategy after checking both: p≈1e-10
and Cohen's d=-1.13 for f2p-vs-paid is numerically robust in a way a
p=0.03 borderline result isn't, so the golden question won't flip on
re-ingestion), and — new in this slice — dynamically picking *which*
untracked game to use for the "insufficient history" case, so it stays
correct as the poller's tracked set grows rather than naming one game
that could eventually get tracked and silently invalidate the question.

Added two new check builders, `forecast_has_real_projection()` and
`forecast_reports_insufficient_history()`, both asserting on
`forecast_result` directly rather than just the answer text — the
specific thing they guard against is the model answering a forecast
question correctly *in prose* without the tool having actually run,
which turned out, later in this same slice, to not be a hypothetical
concern at all.

### First real run: the rate limit fix from Slice 42 wasn't enough on its own

The first live 15-question run came back route accuracy 15/15 but
deterministic checks only 7/15, judge 3.1/5. Looked at the actual
failures rather than assuming "the new questions are just harder":
eight of them showed `agent_node`'s own hardcoded degraded-fallback text
("I ran into a repeated error...") sitting next to suspiciously small
token counts (500-600 vs. a normal 3000-8000+) — the exact signature of
a call that got rate-limited before it ever generated a real answer, not
a quality problem with the questions themselves.

Root cause: `run_evals.py`'s `SPACING_SECONDS` was still 5.0, tuned back
when the set had 5 questions. Several of the new ones run 5000-8000+
tokens, well above the old set's average, so two heavy questions landing
in the same rolling 60-second window was now common instead of rare.
Raised it to 20.0. Also gave `agent_node`'s own retry (the one Slice 42
added for a failed model call) a real backoff — `agent_retry_backoff_seconds`,
3.0s — since a same-instant retry has zero chance against a TPM window
that's still exceeded; it needs a moment to actually roll forward.

Re-ran for real: **12/15 deterministic checks (up from 7/15), judge
4.1/5 (up from 3.1)**, and not one of the 12 passing results carries the
degraded-fallback signature anymore. A real, measured improvement, not
an assumption that raising a constant would obviously help.

### The other three failures were not one bug wearing three disguises

The temptation after a fix like that is to assume the remaining failures
are just "more of the same, needs more tuning." Checked each individually
instead, and they turned out to be three unrelated things:

**1. A whitespace bug in the check itself, not the model.** The price-
outliers question's answer correctly named "EA SPORTS FC 24" — but the
model rendered it with a Unicode narrow-no-break-space (U+202F) between
words instead of a plain space, which `contains_text`'s literal substring
check missed even though the answer was factually right. Fixed by
normalizing whitespace on both sides of the comparison
(`re.sub(r"\s+", " ", text)` — confirmed empirically that `\s` matches
U+202F even though `in` doesn't). Regression test built the stylized
string with `chr(0x202F)` rather than typing the literal character,
since the literal character itself tripped `ruff`'s own
ambiguous-character lint (`RUF001`/`RUF003`) the first two times this
was attempted — worth a note since it wasted real time before landing.

**2. A real, live-confirmed bug in Slice 42's own forecast-guidance fix.**
The CS:GO next-month forecast failed with `tool_errors: 2`, forecast
tool never producing a result. Rather than guess, captured the actual
message trace directly (same technique as Slice 42's original crash
diagnosis): the model correctly wrote
`regexp_replace(name, '[-:]', ' ', 'g') ILIKE '%counter strike global
offensive%'` — the exact pattern Slice 42 taught it — but that pattern
turns `"Counter-Strike: Global Offensive"` into `"Counter Strike  Global
Offensive"`, TWO spaces before "Global" (the colon becomes a space
immediately next to the literal space that already followed it in the
real title), which then fails to match the model's own single-spaced
search phrase. Zero rows, `execute_run_forecast` raises `ValueError`,
self-correction retries exhaust, honest degradation. Fixed by matching
*runs* of punctuation/whitespace instead of single characters —
`'[-:\s]+'` — in both `FORECAST_TOOL_GUIDANCE` and the `column:name`
schema chunk. Verified live, directly, not just by reasoning about the
regex: re-ran the exact same question, got 1 attempt, 0 tool errors, and
a real projection (`n_snapshots: 24`, `low_confidence: true` — correctly
flagged, since 30 days out from only ~11 days of real history is exactly
the low-confidence case the tool's own design already accounts for).

**3. Genuinely inconclusive — and said so rather than guessed.** The
third failure (`forecast_insufficient_history_is_honest`) showed the
same hardcoded degraded-fallback text. First assumed this was leftover
rate-limiting from testing the CS:GO fix moments earlier, and re-ran it
in isolation after a real 65-second wait. Same failure. Bypassed
`agent_node`'s swallowed exception directly (called `model.invoke()`
outside its try/except) to see the actual error rather than continuing
to guess, and found: `groq.RateLimitError`, but on **tokens per day**
(TPD 200,000, 198,557 already used), not the per-minute limit Slice 42's
fix targets — this session's own cumulative live-verification testing
today, not a code bug. This means the original design concern the check
was written to catch (the model answering correctly in prose without
the tool having actually run) is still **unverified**, not confirmed
either way, and is written up here as exactly that rather than silently
folded into "item 2 is done" or guessed at without evidence.

### What this slice's own experience says about the audit's underlying finding

Two of three remaining failures here were bugs in the verification layer
itself (the check's whitespace handling) or in a previous slice's own
prompt-engineering fix (the double-space regex) — not in the questions
newly added. This is the same shape as Slice 30's incident and Slice 43's
`sqlglot.ParseError` find: expanding real test/eval surface keeps finding
real bugs that were already there, just previously unexercised. The
audit's coverage argument is, once again, borne out by its own remedy.

### Verified for real before writing any of this down

`ruff check .` and `uv run mypy src` clean throughout every step, not
just at the end. 160/160 tests pass (159 from Slice 43, +1 new regression
test for the whitespace-normalization fix). The two live-confirmed fixes
(rate-limit spacing/backoff, and the forecast regex) were each verified
against a real Groq call showing the specific failure gone, not inferred
from the code change alone.

### Open questions (new)

- **Whether the 3rd failure (`forecast_insufficient_history_is_honest`)
  is a real design gap or was always going to pass** — genuinely
  unknown until the daily Groq quota resets and it can be re-run in
  isolation, ideally as close to the top of the daily quota as possible
  so a real TPD-related false failure can be ruled out cleanly.
- **Whether item 2 should be considered "done"** given this open
  question, or held open until the third failure is actually resolved
  one way or the other — a call for whoever picks this list back up,
  not assumed here.

## Slice 45 — Working the 9/10 list, item three: the router had agent_node's old blind spot

**Date:** 2026-09-05

Item 3 on the audit's list was "production resilience blind spot (model-
call failure path never tested/handled)." At first glance this looked
already closed — Slice 42 fixed exactly that, in `agent_node`, with 4
regression tests. Rather than mark it done on that assumption, went
looking for whether Slice 42's fix was actually the *only* unprotected
LLM call in the production request path, or just the first one found.

### The router has its own LLM call, and it had never been touched

`classify_question` (`src/agent/router.py`) is the very first LLM call
in the entire graph — a structured-output classification into one of
four routes — and it's a completely separate call from `agent_node`'s.
Checked it directly: zero error handling, no try/except at all. Any
Groq failure here (a rate limit, a malformed structured-output parse,
the same class of thing Slice 42 found for the tool-calling turn) would
propagate straight out of `router_node` unhandled.

Checked whether this was actually reaching users as a crash before
treating it as urgent: it wasn't, quite — `main.py`'s `/ask` and
`/ask/stream` both wrap the whole agent run in a blanket
`except Exception`, turning anything unhandled into a clean 503 rather
than a leaked stack trace. So this was never a literal crash reaching a
client. But it was a real asymmetry: the exact same class of failure
that `agent_node` recovers from gracefully (retry once, degrade to an
honest answer) failed the *entire* request for `router_node`, relying
entirely on a generic "high demand" 503 instead of the better behavior
already built for the other node. Presented this finding and the fix to
the user directly before implementing (this is a production error-
handling behavior change, not a pure bug fix) and got explicit
confirmation to proceed.

### The fix, deliberately identical in shape to Slice 42's

`router_node` now catches a `classify_question` failure, sleeps
`agent_retry_backoff_seconds` (the same setting Slice 44 added for
`agent_node`, now shared and its docstring updated to say so), retries
once, and — if that also fails — degrades to `needs_clarification` with
an honest message asking the user to rephrase, instead of letting the
API's blanket catch-all be the only thing standing between a router
hiccup and a bare 503. Reusing the exact same setting and the exact same
shape (try, sleep, retry, degrade) rather than inventing a parallel
mechanism was a deliberate choice: this is the same problem in a second
location, not a new problem needing its own solution.

Added 4 new regression tests (`test_router_node_llm_errors.py`), mocking
`classify_question` directly rather than `get_llm()` — cheaper and more
direct, since `router_node` calls `classify_question` by name and
`graph.py` imports it into its own namespace, so patching
`graph_module.classify_question` is enough, no need to reach through to
the LLM object itself. Covers the recovery path, the fully-degraded
path, a clean call never retrying, and — the one case Slice 42's test
file didn't need an equivalent for — a *real*, successful
`needs_clarification` decision still passing through completely
unaffected by the new error-handling wrapper around it.

### Verified for real

`ruff check .` and `uv run mypy src` clean. 164/164 tests pass (160 from
Slice 44, +4 new). No live Groq calls were made to verify this — the
daily quota was still near-exhausted from Slice 44's own live
verification earlier the same day, and unlike that slice's forecast-
regex fix, this one didn't need a live call to confirm: the failure mode
being tested (an LLM call raising an exception) is exercised directly
and deterministically via a mocked `classify_question`, the same
reasoning that justified `test_agent_node_llm_errors.py` never needing
a live call either.

### Open questions (new)

- **Whether any other LLM call sites in the production request path
  remain unaudited.** Checked router_node and agent_node specifically
  (the two direct chat-model calls in the graph); `retrieve_schema_node`
  calls a local embedder (no network, a different failure class) and
  wasn't in scope for this pass. Worth a deliberate sweep if "production
  resilience" is revisited again, rather than assuming these two were
  the only two.

## Slice 46 — Working the 9/10 list, item four: two recurring process gaps, closed for real this time

**Date:** 2026-09-05

Item 4 was "two process gaps rediscovered twice each" — the GitHub
Actions re-run staleness trap (Slices 35, 36) and Render deploy
staleness (Slices 32, 41). Both had real fixes already landed, but
neither had been closed in a way that actually prevents a third
recurrence — one was patched per-workflow as each was individually
caught, the other fixed its symptom (cadence) without ever answering
the actual open question underneath it.

### Gap A: checked whether the fix pattern had actually been applied everywhere

Slice 35 added `workflow_dispatch: {}` to `frontend-ci.yml` after "Re-run
jobs" was found to replay the pinned original commit instead of picking
up a fix already on master. Slice 36 hit the identical thing in
`run_evals.yml` the very same day. Rather than assume a third workflow
couldn't have the same gap just because it hadn't been *caught* yet,
checked every workflow file directly: `poll_player_counts.yml` and
`refresh_catalog.yml` already had `workflow_dispatch` (added when they
were first written, unrelated to this bug). `test.yml` — the workflow
that actually gates every push and PR, arguably the most important one
in the repo — did not. This is exactly the shape of "rediscovered
twice": the fix pattern was known, just not applied systematically.
Added it there too, with a comment pointing back at Slices 35/36 so the
next person (or the next slice) doesn't have to re-derive why it matters.

### Gap B: the real open question from Slice 32 finally gets an automated answer

Slice 41 fixed `refresh_catalog.yml`'s cadence (weekly to daily), which
closed the *specific* symptom that slice caught (a live forecast lagging
real history by up to 7 days). It didn't touch the actual open question
Slice 32 left sitting: is Auto-Deploy even enabled on Render, is
`DEPLOY_HOOK_URL` set at all — both literally unverifiable from inside
any session or CI run, since this project has no Render dashboard/API
access. A tighter cadence makes staleness *less likely*, but says
nothing about whether the mechanism causing it is actually fixed or
just hasn't happened to fail again recently.

Rather than accept "we can't check this" as final, looked for a real
signal that doesn't need dashboard access. Checked Render's own
documentation directly (not assumed): it automatically injects
`RENDER_GIT_COMMIT` (and `RENDER_GIT_BRANCH`, `RENDER_GIT_REPO_SLUG`)
into any service deployed from a connected git repo — no Dockerfile
change, no build-arg plumbing needed. Added `deploy_commit` to `/health`
reading that env var directly.

That alone doesn't close the loop — someone still has to look at
`/health` and notice a stale commit. So added `check_deploy_freshness.yml`:
queries the live `/health`, takes whatever commit it reports, looks up
that commit's *real* committed-at timestamp via GitHub's API (`gh api
repos/.../commits/$SHA`, using the workflow's automatic token — no new
secret needed), and fails the job if it's older than 48 hours. That
threshold is deliberately generous over `refresh_catalog.yml`'s daily
cadence (real slack for a slow build or scheduling jitter) while still
being tight enough to catch a genuinely broken Auto-Deploy within a
day or two, not weeks.

Presented the whole design to the user before writing any of it — a new
scheduled workflow with a specific staleness threshold is a real,
ongoing behavior decision, not a pure bug fix — and got explicit
confirmation, same as Slice 45's router-resilience change.

### A small but real verification step: testing the shell logic against real input first

The workflow's own logic (extract `deploy_commit` from a JSON response,
compute an age in hours from an ISO timestamp) is exactly the kind of
thing that looks obviously correct and then breaks the first time it
actually runs in CI, with no local feedback loop to catch it beforehand.
Tested both pieces directly against real sample input before trusting
them inside the workflow: the JSON-extraction one-liner against a
literal `{"deploy_commit": "abc123"}` and a `null` case, and the date-
diff arithmetic (`date -u -d "<iso-timestamp>" +%s`) against a real
timestamp, confirming Git Bash's `date` here behaves the same way the
GNU `date` on GitHub's Ubuntu runners will. Cheap to do, and exactly the
kind of thing Slice 41's own `on:`-as-boolean YAML surprise already
proved is worth checking rather than assuming.

### Verified for real

`ruff check .` and `uv run mypy src` clean. 166/166 tests pass (164 from
Slice 45, +2 new for `/health`'s `deploy_commit` field). Both new/changed
YAML files re-validated with `yaml.safe_load` (handling PyYAML's `on:` →
boolean `True` coercion the same way Slice 41 already learned to). What
could NOT be verified from inside this session, named honestly rather
than assumed: whether `check_deploy_freshness.yml`'s first real
scheduled run actually reaches the live `/health` endpoint and reports a
sensible commit — that only happens on Render's own infrastructure,
tomorrow at 07:00 UTC.

### Open questions (new)

- **Whether the live deploy is currently fresh at all** — genuinely
  unknown until `check_deploy_freshness.yml`'s first real run. If it
  fails immediately, that's itself a real, useful answer to Slice 32's
  long-open question (Auto-Deploy/`DEPLOY_HOOK_URL` may in fact not be
  configured), not a bug in this new workflow.
- **The 48h threshold is a first real guess, not a tuned constant.**
  Worth revisiting once a few real runs' worth of actual staleness
  numbers exist, the same way `SPACING_SECONDS` (Slice 44) started as a
  guess and got corrected against real evidence.

## Slice 47 — Working the 9/10 list, item five: the frontend regression the audit named directly

**Date:** 2026-09-05

Item 5 was the audit naming its own prior work as a real tension: "a
project whose own README leads with restraint as a differentiator just
spent real, measured weight and a known leak on two decorative
flourishes" — Slice 39's shadergradient integration (2.4MB total client
JS, a single 1.1MB chunk, loaded unconditionally) and liquid-glass-js's
permanent per-mount scroll-listener leak, both accepted at the time as
deliberate, documented trade-offs. Working this item meant deciding
whether "documented and accepted" was actually the right final answer,
or just where the investigation had stopped.

### The leak wasn't actually as fixed-forever as it sounded

Slice 39's own writeup said "the library has no destroy method to call"
— true, but worth checking what "the library" actually meant here.
`liquid-glass-js` isn't an npm dependency; it's vendored, plain
`<script>`-global source checked directly into
`public/vendor/liquid-glass/`. There is no upstream release cadence to
wait on, no maintainer to file an issue with — it's this repo's own
file, editable the same as anything else in `src/`.

Read `container.js` directly to find the actual leak mechanism rather
than trusting the prior description alone: `startRenderLoop()` registers
`window.addEventListener('scroll', handleScroll, ...)` and never stores
a reference to `handleScroll` anywhere, so nothing could ever call
`removeEventListener` on it even in principle. A second, independent
leak source sat right next to it, previously unmentioned: `Container`'s
constructor pushes `this` onto a static `Container.instances` array that
nothing else in either vendored file ever reads — dead weight whose only
real effect is keeping every instance ever constructed reachable
forever, regardless of the listener.

Added a real `destroy()` to `Container`: removes the (now-stored)
scroll listener, explicitly frees the WebGL context via the standard
`WEBGL_lose_context` extension rather than waiting on GC, removes the
instance from that static array, recursively destroys children, detaches
the DOM element. Wired `LiquidGlassAskButton`'s existing unmount cleanup
to call it, optional-chained so a future re-vendoring that drops
`destroy()` again degrades to the old (already-documented) leak instead
of throwing.

### The bundle-size fix, and being honest about what it actually changes

`GradientBackground`'s dynamic import (`next/dynamic`, `ssr:false`) was
already correctly code-split — confirmed directly via the build's
`react-loadable-manifest.json`, which showed the 1.1MB chunk tied only to
the home page, never leaking into `/catalog`'s bundle. What was missing
wasn't code-splitting; it was *scheduling*. Nothing deferred *when* the
client fetched that chunk once the component rendered, so it competed
with the actual hero content for bandwidth and main-thread time on every
single visit to "/", for a background flourish with zero functional
weight.

Added `useDeferredMount()` (`lib/`): flips a boolean once
`requestIdleCallback` fires (falling back to `setTimeout` where
unsupported, notably Safari), and `page.tsx` gates `GradientBackground`'s
render behind it instead of mounting unconditionally. Deliberately not
framed as a size reduction anywhere in this writeup or in PLAN.md's
entry: the chunk is still 1.1MB, the total build is still 2.4MB. What
changed is *when* the browser has to deal with it, not how much there is
to deal with — claiming otherwise would be exactly the kind of
overclaim this project's own review process exists to catch.

### Verified in a real browser, not reasoned about from the source

Installed Playwright ephemerally (`npm install --no-save`, same pattern
as Python's ephemeral `pytest-cov` installs elsewhere in this project;
removed afterward) against a real `next build` + `next start`, not the
dev server, so timing numbers reflect production behavior.

Confirmed the deferral directly: the known shadergradient chunk's
network request lands ~1.4 seconds after the page's `load` event fires,
not before or during it — a real number captured from a real response
event, not inferred from the `useDeferredMount` timeout value.

Confirmed the leak fix by testing across real client-side navigations
(`next/link`, clicking through "Catalog" and back to "Ask Ludo" in the
actual rendered nav) rather than `page.goto()` reloads — a hard reload
would trivially "pass" this test by destroying the whole JS realm along
with any leaked listener, proving nothing about whether the component's
own cleanup actually works. A `window.addEventListener`/
`removeEventListener` monkeypatch tracked the live count across 5 real
navigate-away-and-back cycles: flat, never growing.

A genuinely useful thing surfaced by instrumenting this broadly rather
than narrowly: the shadergradient chunk turned out to register its
*own*, entirely separate `window` scroll listener (visible directly in a
captured stack trace: `Button.startRenderLoop` for the known
liquid-glass one, and a separate, minified `@react-three/fiber`-internal
call chain for the other). Worth checking whether this was a *second*
undiscovered leak rather than assuming the audit's finding was
exhaustive — it wasn't: the count went 2 (both mounted on "/") to 0
(navigated to "/catalog", where neither component mounts) back to 2
(returned to "/"), never 3 or 4. `@react-three/fiber`'s own Canvas
lifecycle already disposes correctly; only liquid-glass-js's listener
was ever actually leaking.

### Verified for real

`npx tsc --noEmit` and `npx eslint .` clean across every changed file.
A real `next build` still succeeds, same 2.4MB total (unchanged, as
expected and as reported above). No backend files touched this slice,
so the Python suite (166/166, `ruff`/`mypy` clean) is unaffected and
wasn't re-run needlessly.

### Open questions (new)

- **Whether the `WEBGL_lose_context` call in the new `destroy()` is
  strictly necessary** versus letting the browser's own GC reclaim the
  GPU context once nothing references it. Included as defensive best
  practice (explicit is better than waiting on GC for a scarce resource
  like a WebGL context, and browsers do cap how many can exist
  simultaneously), not because a specific failure was observed without
  it.
- **Whether `useDeferredMount`'s default 200ms timeout is the right
  number**, or whether a longer one would defer even more aggressively
  without a real user-perceivable downside. Not tuned against real field
  data, the same honest caveat Slice 44's `SPACING_SECONDS` and Slice
  46's 48h threshold both got.

## Slice 48 — Working the 9/10 list, item six (the last one): a global budget, not a login wall

**Date:** 2026-09-05

Item 6, the last of the six: "no authentication on /ask, a real,
cost-incurring LLM endpoint, reachable directly, bypassing the frontend
entirely... given this project already had one real incident where a
shared daily Groq quota got exhausted (Slice 33), this is a live, not
hypothetical, exposure."

### Naming what "fixing" this actually means before building anything

The word "authentication" in the audit's own finding could mean several
very different things, with very different costs. Worked through the
real options rather than reaching for the first one:

- **A real login wall** would technically satisfy "add auth," but
  directly defeats the point of a public portfolio piece a recruiter
  should be able to try in ten seconds, no signup. Rejected immediately,
  not seriously considered.
- **A shared secret shipped from the frontend** (an API key in a request
  header) sounds like a fix but isn't one on its own: anything sent from
  browser JS is visible in the Network tab to anyone who opens DevTools,
  so it stops nothing but the most casual accidental discovery. A *real*
  version of this needs a server-side proxy (a Next.js API route holding
  the actual secret, forwarding to the FastAPI backend) — a genuinely
  bigger architectural change, and even then it only gates *who* can
  spend the budget, not *how much* gets spent by legitimate frontend
  traffic itself.
- **Per-IP rate limiting** already exists (`InMemoryRateLimiter`,
  10 requests/60s) — but the audit's own wording is precise about why
  that's not enough: "trivially bypassed by multiple source IPs." It
  protects against one abusive client hammering the endpoint, not
  against the actual shared resource (Groq's real 200,000 tokens/day
  cap, already exhausted for real once, Slice 33) being exhausted by
  *any* combination of traffic.

Presented this reasoning directly to the user rather than picking one
silently: a global (not per-IP) daily token budget is the option that
protects the actual thing at risk, regardless of how many IPs, whether
traffic comes through the real frontend or a direct `curl`, or any other
detail about *who* is asking — it closes the specific, real, previously-
experienced failure mode named in the finding, without adding login
friction a demo can't afford. Got explicit confirmation before building.

### The implementation, and one real subtlety in getting the error message right

`RunStats` (Slice 26) already tracks `total_tokens` cumulatively across
every real (non-cached) run — the exact number needed, already there,
nothing new to instrument. Added `daily_token_budget=180_000` to
`Settings` (real headroom under Groq's actual 200,000 cap, room for the
router's own small classification calls and other traffic sharing the
same key) and checked it in both `/ask` and `/ask/stream`, specifically
*after* a confirmed cache miss — checking before the cache lookup would
have blocked cached answers too, which cost nothing and have no reason
to stop working just because the *real* budget is exhausted.

The one real subtlety: `/ask`'s existing `except Exception as exc:`
block already exists specifically to turn any unexpected failure into a
clean 503 with a generic "high demand" message in production (hiding
real internal error detail from users, per `settings.debug`). Raising a
plain `HTTPException` with the honest daily-budget message directly at
the check site would have been caught by that *same* generic handler and
had its message overwritten — silently turning an honest, specific,
non-error condition into the same vague message as a genuine crash.
Fixed by giving it its own exception type, `DailyBudgetExceeded`, caught
explicitly before the generic handler. Wrote the regression test for
exactly this (`test_ask_daily_budget_message_is_returned_even_with_debug_on`)
rather than trusting the reasoning alone — this is precisely the kind of
"looks obviously right" code that silently breaks the first time an
edge case (debug=True, here) is actually exercised.

### A real, pre-existing bug the new tests surfaced, unrelated to the budget logic itself

Adding 4 new tests that each make a real `/ask`/`/ask/stream` call
pushed `tests/test_api_main.py`'s total call count in one run high enough
to trip something that had never actually fired before: `src/api/
rate_limit.py`'s `_limiter` is a genuine module-level singleton, created
once at import time and shared by every test in the process (not just
every request in production) — its own dedicated test file
(`test_rate_limit.py`) covers its behavior directly, but nothing had
ever reset its accumulated state between tests in *this* file, because
the total request count across all of `test_api_main.py`'s pre-existing
tests had simply never crossed the real 10-requests/60-second threshold
before. Adding 4 more tipped it over for the first time, and two tests
later in the file started failing with a genuine 429 that had nothing to
do with what they were actually testing (an unrelated agent failure, and
the new budget check itself).

Fixed at the root, not by trimming test count to dodge the threshold:
added an autouse fixture clearing `rate_limit._limiter._hits` before
each test, the same "this behavior is covered by its own file, isolate
it here" pattern this file already used for the semantic cache
(`_disable_cache`). Re-ran the full suite twice after the fix to confirm
it was real and not a fluke of test ordering.

### Verified for real

`ruff check .` and `uv run mypy src` clean. 170/170 tests pass (166 from
Slice 47, +4 new). `.env.example` updated with the new setting so a
fresh clone's documented configuration surface stays complete, matching
how every other tunable in `Settings` is already mirrored there.

### Open questions (new)

- **Whether 180,000 is the right number**, or whether it should account
  more precisely for the router's own per-question token cost rather
  than a round-number guess at headroom. Same honest caveat as every
  other first-guess threshold this project has shipped and later
  revisited against real evidence (`SPACING_SECONDS`, the 48h deploy-
  staleness threshold).
- **Whether a server-side proxy (the heavier option presented and not
  chosen) is worth adding later**, specifically to stop non-frontend
  traffic from consuming the shared budget at all, rather than just
  capping total spend once it's already being consumed. Not needed to
  close this item honestly, but a real remaining gap if direct API abuse
  (as opposed to shared-budget exhaustion) becomes an actual observed
  problem rather than a named risk.
- **This closes the 9/10 audit's sixth and final named item.** Whether
  the project is now actually at a 9/10, or whether working through
  these six surfaced enough *new* real findings (the router resilience
  gap, the forecast regex bug, the shadergradient scroll-listener
  non-leak, this file's own rate-limiter test-isolation gap, among
  others) that a fresh review is warranted before calling it settled, is
  a real open question for whoever picks this up next -- not assumed
  here either way.

## Slice 49 — "There's a problem with Render" — a real incident, found via actual logs, not guessed at

**Date:** 2026-09-07

The user reported a problem and asked for a direct test, no more detail
than that. Used the `systematic-debugging` skill explicitly rather than
jumping to a fix: reproduce first, read the real evidence, form one
hypothesis, test it, only then implement.

### Reproducing directly against the live URL

Curled the live Render backend's `/health` and `/ask` directly, bypassing
the frontend entirely -- the fastest way to know whether this is a
frontend issue, a backend issue, or a network issue. First real finding,
purely local: this machine's curl (Windows schannel) failed the TLS
handshake outright with `CRYPT_E_NO_REVOCATION_CHECK` -- a local
certificate-revocation-check failure, not evidence about Render at all.
Confirmed by retrying with `--ssl-no-revoke`: the real connection
succeeded fine once that local artifact was bypassed. Named this
explicitly rather than let a local networking quirk get mistaken for the
actual reported problem.

### `/health` always worked; every real question always failed

`/health` responded correctly every time (once past the local TLS
issue), including a real `deploy_commit` (Slice 46's own fix, now
genuinely paying off) confirming the live backend was running a commit
that includes all of Slices 42-48. Three separate real `/ask` calls,
different questions, different routes -- all failed: a 502 after 14-20s
of real processing twice, and once an instant, empty-body 503. Tested
the fastest possible path deliberately (`"Is this game good?"`, which
routes to `needs_clarification` -- a single quick router call, no schema
retrieval, no tool-calling turn at all) specifically to rule out "the
slow routes are just timing out": it failed too, just as fast/hard as
the others. This single test is what ruled out a route-specific latency
problem and pointed at something shared by literally every real
question, not just the expensive ones.

### The one thing every failing request has in common

`_run_with_cache()` (`src/api/main.py`) calls the semantic cache's
`.get(question)` before any routing happens at all, for every real
`/ask` call -- and that call embeds the incoming question via the local
ONNX model to check for a similarity hit. `/health` never touches this
path. That's the one piece of code every failure shares and the one
success (`/health`) never runs.

### Getting the evidence this session doesn't have access to

No Render dashboard/log access from inside this session -- the same
standing limitation named repeatedly since Slice 32. Asked the user
directly for the real service logs rather than keep guessing from
outside evidence alone. They pasted them. The logs settled it precisely:

```
07:06:09  a real Groq call succeeds (200 OK)
07:06:11-07:06:16  the app downloads the embedding model live from
                    HuggingFace (5 files, ~5s of network + disk work)
07:06:21  last log line
07:07:00  "Started server process [1]" -- a brand new, clean process
          start, with NOTHING logged in between
```

No Python traceback, no error line, no graceful-shutdown message --
just silence, then a fresh restart, exactly the pattern a genuine
Python exception would NOT produce (uvicorn logs those) but an OOM kill
(the kernel/container runtime SIGKILLs the process outright) would. The
same silence-then-restart shape repeated a second time a few minutes
later. This was the single piece of evidence that turned "the embedder
is somehow involved" into an actual, confident root cause.

### Root-causing precisely, not stopping at "it's the embedder"

The real question was why a model that's supposedly pre-warmed at
Docker *build* time was downloading again at *runtime*. Read fastembed's
own installed source directly rather than guess: its default cache
directory is `tempfile.gettempdir() + "/fastembed_cache"` (`/tmp/...`),
overridable via `FASTEMBED_CACHE_PATH`. Checked Render's own docs: "by
default, Render services have an ephemeral filesystem... any changes...
are lost every time the service redeploys or restarts" -- confirming
`/tmp` populated during the Dockerfile's build-time pre-warm step has no
guarantee of surviving into a fresh runtime container, unlike `/app`
(where the app's own source and venv demonstrably do persist and work).
Every restart -- and this app had clearly been restarting on essentially
every real request -- was re-paying the full network-download-plus-
ONNX-init cost that was supposed to be a one-time build-time expense,
right at the exact moment (freshly started, before anything else has
settled) the process is least able to absorb the extra memory pressure.

### The fix, verified locally before trusting it in production

Pinned `FASTEMBED_CACHE_PATH=/app/.fastembed_cache` in the Dockerfile,
set before the existing pre-warm `RUN` step so both build time and
runtime agree on the same real, persistent path. Verified the actual
mechanism directly rather than trusting the reasoning alone: set the
env var locally, confirmed the model files landed exactly under the
custom path, then ran a second time against the same path and confirmed
no download bar appeared at all -- a real, observed cache hit, not an
assumed one. Docker itself isn't available in this environment, so a
full local image rebuild couldn't be used to verify the complete
build-to-runtime path the way the real Render build will -- named
honestly as the one thing that still needs the next real deploy to
fully confirm.

### Verified for real

`ruff check .`, `uv run mypy src`, and the full test suite (170/170)
all clean -- confirming the Dockerfile-only change touched nothing else
in the codebase, not that it fixes the incident by itself (that needs a
real redeploy and a real retest, not a local check).

### Open questions (new)

- **Whether this fully resolves the crash, or only reduces its
  frequency/severity.** The evidence points squarely at the re-download-
  and-reinit cycle as the trigger, but without real Render memory
  metrics, it's not fully ruled out that loading the ONNX model into
  memory for the first time is *itself* enough to approach the free
  tier's undocumented memory ceiling, cache hit or not. The real test is
  the next live redeploy plus a real retest against `/ask`, not this
  local verification alone.
- **How long this incident had actually been live.** Given
  `refresh_catalog.yml` rebuilds (and therefore restarts) the backend
  daily, and every fresh restart appears to have hit this same
  redownload-then-possible-OOM cycle, this may have been silently
  breaking the very first real question after every single day's
  rebuild for a while -- not something introduced by Slices 42-48,
  since none of that work touched the embedding/cache path at all. Worth
  a genuine "how long has this been broken" retrospective once it's
  confirmed fixed, rather than assuming it was newly introduced today.

## Slice 50 — The real root cause: one long schema chunk, batch-embedded, blowing past Render's actual 512MB ceiling

**Date:** 2026-09-07

Slice 49's fix (the fastembed cache path) was real and worth keeping,
but the very next redeploy proved it wasn't the whole story: a real
`/ask` question still failed, 502 after 12s, and this time with no
re-download in the logs at all -- the crash was happening somewhere
else. Slice 49's own "Open questions" already flagged this as a live
possibility rather than assuming victory; it turned out to be the right
call to stay skeptical.

### Getting the real ground truth, not another guess

Rather than propose fix #2 blind, asked the user to check two things
only Render's own dashboard can show: the Metrics tab and the Events
tab. The Events tab gave a direct, unambiguous answer that ended all
speculation about *whether* this is memory-related:

> **Instance failed: Sep 7 10:42:19 AM**
> "Ran out of memory (used over 512MB) while running your code."

This is the single most valuable piece of evidence in the whole
investigation -- a real, first-party confirmation of both the mechanism
(OOM, not a native crash or something else entirely) and the exact
number (Render's free tier really is 512MB, not the "undocumented"
figure earlier slices had to guess at).

### Instrumenting precisely, then hitting a real practical wall

Added `MEMORY [...]` checkpoint logging (peak RSS via stdlib `resource`)
at the request boundary in `main.py`, then a second round in
`graph.py`'s `retrieve_schema_node`/`agent_node`/`execute_tools_node`
once the first round's evidence (the semantic cache's own embed call
showing a completely flat 194.7MB -> 194.7MB) ruled out the cache/
embedder-loading step and pointed at something further into the
pipeline. Each round needed a real commit -> push -> redeploy -> ask a
real question -> read Render's logs cycle -- several real minutes each,
plus, once a real question was asked, real Groq quota.

The user asked directly, reasonably: is there a faster way? There was.
This also surfaced a second real, independent gap worth fixing on its
own: the Dockerfile never set `PYTHONUNBUFFERED=1`, meaning an OOM
kill's instant `SIGKILL` could silently discard whatever log lines were
still sitting in Python's output buffer, unflushed, at the exact moment
that matters most. Fixed alongside the diagnosis, not left for later --
a diagnostic that can silently lose its own most important evidence is
a real bug in the diagnostic itself.

### Moving the investigation local, for real evidence at zero cost

Wrote a throwaway script using `psutil` (cross-platform; the stdlib
`resource` module used in the production checkpoints doesn't exist on
this Windows dev machine at all) to reproduce the same measurements
locally. Absolute numbers wouldn't match Render's Linux container
exactly, but the *relative* jump at each step -- the actual thing being
hunted -- would still be real signal, at zero redeploy cost and zero
Groq spend.

First local run (full pipeline: import `graph.py`, run `router_node`,
`retrieve_schema_node`, `agent_node`) found a massive, unambiguous jump:

```
after router_node:                178.1 MB
after retrieve_schema_node:       755.4 MB   (+577.3 MB)
```

Tempting to stop there and ship a fix. Didn't: that single number
conflates two different costs (loading the ONNX runtime for the first
time in this process, AND `SchemaIndex`'s one batch `embed_texts()` call
over the whole schema corpus), and the earlier Render evidence (the
cache's own embed call costing ~0MB) already suggested those two costs
don't behave the same way. Ran a second, isolated test: load the
embedder fresh, embed one query, then batch-embed 20 short *placeholder*
strings. Result: model load ~138MB (real, but not the incident), one
query ~3MB (matches Render exactly), batch-of-20-short-strings only
~7.6MB more. That flatly contradicted "it's the batch size" -- so the
placeholder test's texts must not have matched the real corpus's actual
shape.

### The precise, decisive test

Re-ran the same isolation, in the same heavy-import context as
production (`import src.agent.graph` first, matching Render's real
module-load shape), but swapped the fake placeholder texts for the
*real* schema corpus (`SCHEMA_CHUNKS`), with zero Groq calls needed at
all for this specific measurement:

```
after get_embedder():                              298.6 MB
after ONE query embed:                              302.0 MB   (+3.4 MB)
after batch-embedding the REAL 35-chunk corpus:     747.6 MB   (+445.6 MB)
```

Then the confirming counter-test -- same real corpus, same real content,
embedded one chunk at a time instead of one batch call:

```
after embedding the same 35 real chunks ONE AT A TIME:  310.4 MB   (+11.5 MB)
```

Decisive: not the model, not the corpus size, specifically *batching*
the real corpus together in one call. The real corpus's chunk lengths
explain why: 54 to 1,465 characters, median 118 -- one genuine long
outlier (almost certainly `column:name`'s detailed regex/JSON-escaping
guidance, itself grown across Slices 42 and 44's own fixes) sitting in a
batch with 34 much shorter chunks. Batched inference pads every item in
a batch to the longest one, and attention cost scales with sequence
length -- one long chunk drags the whole batch's processing (and,
evidently, memory allocation) up to its own scale, not the corpus's
typical scale.

### A permanent tool, not another throwaway script

The user asked for exactly the right thing: a real, reusable file to
run this check locally going forward, instead of a scratch script that
gets deleted after one investigation. Added
`src/diagnostics/memory_probe.py` (`python -m src.diagnostics.memory_probe`):
mirrors production's real import shape, measures the same checkpoints,
and -- since it exercises the *real* `get_schema_index()` call rather
than reimplementing both the batch and one-at-a-time variants inline --
it stays a true regression guard against whatever the actual code does,
not a simulation that could drift out of sync with it. Ends with a real
pass/fail check (schema-index build cost budgeted at 50MB, real headroom
above the ~11.5MB one-at-a-time cost, tight enough to catch a reversion
back toward batch-embedding) and a non-zero exit code on failure, so
it's usable as an actual gate, not just a printout to eyeball. `psutil`
added as a real `dev` dependency (not ephemeral this time) since this
tool is meant to be run repeatedly going forward, not once.

Confirmed the tool actually reproduces the incident against the current
(still-unfixed) code before trusting it: ran it, got the same ~449MB
schema-index-build cost and an explicit FAIL, exit code 1 -- the tool
catching the real, still-live bug on its very first run.

### Verified for real

`ruff check .`, `uv run mypy src`, and the full test suite (170/170)
all clean. The diagnostic tool itself verified by running it against the
known-bad current code and confirming it reports the failure
correctly -- not just that it runs without crashing.

### The fix, applied and verified against the diagnostic tool itself

Presented the real options rather than assuming one: embed one chunk at
a time (the minimal, directly-evidenced fix); precompute the corpus's
embeddings entirely offline, going further than just avoiding the batch
call; or shorten the one long outlier chunk. User chose to start with
the first, smallest, most directly-evidenced fix.

Changed `SchemaIndex.__init__` to build `self._vectors` from a loop of
individual `embed_query()` calls instead of one `embed_texts()` batch
call over the whole corpus -- each chunk's own embedding is
mathematically identical either way (nothing about batching changes
what a single text embeds to), only the memory profile of computing
them changes. Ran `src/diagnostics/memory_probe.py` again, against the
fixed code, as the actual verification rather than trusting the
reasoning alone:

```
before: schema index build cost = 449.2 MB  (FAIL)
after:  schema index build cost =  10.2 MB  (PASS)
```

Total peak dropped from 748.6 MB to 309.6 MB -- comfortably under
Render's real 512MB ceiling, with real headroom left for the rest of a
question's actual work (the agent's own LLM/tool-calling turn, DuckDB
queries). Retrieval quality itself confirmed unaffected via the existing
real recall@top_k regression test (`test_retrieval_recall_at_production_top_k_stays_at_the_measured_baseline`),
which still passed -- expected, since no single chunk's own embedding
depends on what else happened to be in its batch, but checked directly
rather than assumed.

Precomputing the corpus offline (option 2) and a chunk-length guardrail
(option 3 from the conversation) are real, deliberately deferred next
steps, not abandoned -- see PLAN.md and whichever slice follows.

### Open questions (new)

- **Whether other batch-embedding call sites exist anywhere else in the
  codebase** that could hit the same class of bug with a different
  (currently short, could grow) batch of texts. Only `SchemaIndex.__init__`
  was found and fixed in scope here; not an exhaustive audit.
- **Whether onnxruntime's actual behavior here (padding to the batch's
  longest sequence, then allocating disproportionately) is fastembed/
  onnxruntime-specific or a more general transformer-embedding-library
  pattern.** Diagnosed empirically (measured, not read from
  onnxruntime's own internals or documentation) -- the *fix* is solid
  regardless, but the deeper *why* is inferred from behavior, not
  confirmed from the library's own source.

## Slice 51 — Two more real CI bugs, surfaced while checking whether everything else was actually green

**Date:** 2026-09-07

After confirming the live memory fix worked, asked for a status overview
rather than assuming everything else was fine -- and checking rather
than assuming immediately surfaced two more real, previously-unnoticed
bugs, both in CI, both genuine, neither a flake.

### mypy: the exact same code, two different verdicts on two different OSes

Pushing the memory-diagnostic code (Slice 49/50) turned `test.yml` red
in CI with `Unused "type: ignore" comment` on both
`resource.getrusage`/`RUSAGE_SELF` calls -- despite `uv run mypy src`
being clean on this local machine moments before. Not a flaky CI runner;
a real, reproducible platform disagreement: `resource` is a Unix-only
stdlib module, and its typeshed stub is fully populated on Linux but
empty on Windows. The exact same `# type: ignore[attr-defined]` was
simultaneously *required* here (Windows) and an *error* there
(`ubuntu-latest`, where `warn_unused_ignores = true` turns an
unnecessary ignore into a hard failure).

Fixed at the actual source of the disagreement, not by picking whichever
OS's answer to trust: pinned `platform = "linux"` in `[tool.mypy]`,
since that's genuinely where this code runs in production (Render).
Verified the fix the same rigorous way as everything else this
session -- reproduced CI's exact error *locally* first (confirming the
platform pin actually changed local mypy's verdict to match CI's),
*then* removed the now-genuinely-unnecessary ignore comments, rather
than removing them first and hoping.

### `run_evals.yml`: a missing setup step, invisible until the next scheduled run

Asked for an overview of what's left, and checking GitHub's own Actions
API for real, current status (rather than trusting memory) surfaced a
second real failure: the scheduled Answer-Quality Evals workflow had
crashed the one time it ran since the golden set expanded to 15
questions -- `duckdb.CatalogException: Table with name player_counts
does not exist`.

Root-caused precisely, not just from the traceback alone: Slice 44
added three forecast golden questions that query `player_counts`
directly (`csgo_snapshots`, `hoi4_snapshots`, `untracked_game`), but
`run_evals.yml`'s own catalog-build step only ever ran
`src.ingestion.ingest` (builds `games`), never
`src.ingestion.build_player_counts_table`. This is exactly the kind of
gap that's invisible from a developer's own machine: this machine's
local DB already had `player_counts` built from earlier work, and every
eval run triggered by hand during Slice 44's own session ran locally,
never through this specific CI job's actually-fresh checkout. The
workflow only runs once a day, so it took a full extra day after Slice
44 landed for the gap to ever actually get exercised.

Reproduced before fixing, the same way as every other real bug this
session: built a fresh, `games`-only DB locally (mirroring exactly what
the CI checkout had), confirmed the identical `CatalogException` at the
identical line, then added the missing
`python -m src.ingestion.build_player_counts_table` step (mirroring the
Dockerfile's own two-step pattern, which already gets this right) and
confirmed `build_golden_questions()` succeeds end to end, all 15
questions, against that same reproduction DB.

### The pattern across both

Neither bug was found by looking for bugs -- both surfaced from
actually checking real, current status (GitHub's Actions API, a fresh
local mypy run against the exact CI platform) instead of assuming
"probably fine" after the main incident got fixed. Same lesson as
Slice 30, Slice 43's `sqlglot.ParseError`, and Slice 46's `test.yml`
gap: real verification keeps finding real, previously-invisible gaps,
specifically because most of this project's own testing happens on a
developer's already-warmed-up local machine, not the genuinely fresh
environment CI (and production) actually run in.

### Verified for real

Both fixes verified by direct reproduction before and after, not by
reasoning alone: the mypy platform mismatch reproduced locally then
resolved; the `run_evals.yml` gap reproduced against a real fresh DB
then resolved, with `build_golden_questions()` confirmed to return all
15 questions afterward. `ruff check .`, `uv run mypy src`, and the full
test suite (170/170) all clean throughout.

### Open questions (new)

- **Whether any other CI workflow has a similar "works locally, never
  actually exercised fresh" gap.** Only `run_evals.yml`'s missing
  `player_counts` step was found and fixed here; not an exhaustive audit
  of every workflow's setup completeness.
- **Whether the scheduled eval run's actual pass rate (not just "did it
  crash") has improved** now that it can run to completion again --
  genuinely unknown until the next real scheduled run (or a manual
  `workflow_dispatch` trigger) completes.

## Slice 52 — RAG hardening: precompute, a real length ceiling, and CI wiring

**Date:** 2026-09-07

With the live incident resolved and both real CI bugs fixed, came back to
the two follow-up items flagged back when Slice 50 first found the root
cause: stop computing the schema corpus's embeddings live at all, and
put a real, enforced ceiling on individual chunk length so the exact
failure mode that caused the incident can't quietly grow back.

### Precomputing, with real persistence and real staleness handling

The schema corpus is static text baked into `schema_corpus.py` -- its
embeddings are the same every time unless that file itself changes, so
recomputing them live on every fresh process start was pure waste on top
of being the actual incident trigger. Added a disk cache to
`SchemaIndex`: a content-hashed `.npz` file (the hash covers every
chunk's actual text, not just a count or a timestamp, so an edited chunk
is detected precisely) at `settings.schema_index_cache_path`, resolved
under the project root -- which in the Docker image means `/app`, the
same real-persistence reasoning `FASTEMBED_CACHE_PATH` already
established in Slice 49 (not `/tmp`, which doesn't survive a restart).

The Dockerfile needed no new step at all: its existing pre-warm line
(`get_schema_index()`, added back when this was first needed to trigger
fastembed's own model download) already calls the exact code path that
now also builds and saves this cache, for free, as a side effect.

Verified every real behavior directly rather than trusting the design:
built a fresh cache (confirmed the file appears), hit it on a second run
(confirmed via timing -- 8.07s cold vs. 4.72s warm, locally -- and by
replacing the embedder with one that raises `AssertionError` if called,
proving the second run never touches it), corrupted the saved hash to
simulate a stale cache (confirmed it rebuilds rather than silently
serving mismatched vectors), and wrote garbage into the cache file
entirely (confirmed it degrades to recomputing instead of crashing).
Each of these four became a permanent test in `test_schema_index.py`,
using the real local embedder throughout (matching this project's
established convention, see `test_cache.py`) rather than mocking the one
thing actually being tested.

### A real ceiling, which meant actually shrinking the chunk that caused this

A guardrail that permits the exact chunk that already caused an
incident isn't much of a guardrail. `column:name` was 1,465 characters
-- split into four focused pieces (the core column description stays
`column:name`; the double-space regex gotcha, the multi-game
disambiguation note, and the JSON-escaping rule each became their own
chunk), bringing the longest chunk down to 558 characters, back in line
with the corpus's other legitimately-detailed chunks (max 478
elsewhere). Set the actual ceiling at 600 -- real headroom above today's
content, nowhere near the 1,465 that caused the incident -- and wrote
`test_schema_corpus.py` to enforce it on every chunk, plus a cheap
bonus check (every chunk id actually unique, worth having once a corpus
is being edited by hand into more, smaller pieces).

Confirmed the split didn't cost anything real: the existing recall@top_k
regression test (`test_retrieval_recall_at_production_top_k_stays_at_the_measured_baseline`)
still passes at the same 1.0 baseline -- more, smaller always-included
chunks changed nothing about what information reaches the model, only
how it's organized for the embedding step's sake.

### Wiring the actual measurement into CI, not just the corpus's shape

`test_schema_corpus.py`'s length check guards the corpus's shape; it
doesn't guard against a *different* expensive call site appearing
somewhere else in the RAG pipeline. Added a real step to `test.yml`
running `src.diagnostics.memory_probe` on every push/PR -- the same tool
built during Slice 50's own investigation, now actually exercised
automatically instead of only when someone remembers to run it by hand.

### Verified for real

`ruff check .` and `uv run mypy src` clean. 178/178 tests pass (170 +
6 new in `test_schema_index.py` + 2 new in `test_schema_corpus.py`).
`memory_probe.py` re-run against the fully restructured corpus: schema
corpus now 38 chunks, 54-558 characters, PASS.

### Open questions (new)

- **Whether 600 characters is the right ceiling long-term**, or whether
  it should be tuned down further once there's more experience with how
  detailed a chunk actually needs to be to be useful -- picked with real
  headroom above today's content, not derived from a formal analysis of
  the tradeoff between chunk detail and embedding cost.
- **Whether the disk cache should also be exercised by a CI step that
  simulates a genuinely fresh Render deploy** (empty cache, real build
  sequence) rather than only unit-tested in isolation -- the unit tests
  cover the caching *logic* thoroughly, but the actual Dockerfile
  integration is still only verified by reasoning about the file paths
  involved, the same category of gap Slice 51 found in `run_evals.yml`.

## Slice 53 — Closing the forecast honesty gap surfaced a real bug it had been hiding

**Date:** 2026-09-07

The last real, product-facing item from the golden-eval investigation:
the model sometimes answered a forecast question correctly in prose
without ever calling `run_forecast`, for a game with no tracked history.
Not a data problem (more history won't fix a process gap) and not
fixable by waiting -- a direct instruction was the right tool.

### The fix, and what it immediately uncovered

Added one explicit rule to `FORECAST_TOOL_GUIDANCE`: always call
`run_forecast`, even when the model is confident the game isn't
tracked -- an untested "sounds right" answer is indistinguishable from a
lucky guess the one time it's actually wrong, which is exactly the
failure mode this project's own design principle (never let the model
guess when a tool can check) exists to prevent.

Verified live rather than assumed: asked the real, deployed graph about
Grand Theft Auto IV: Complete Edition (a real, currently-untracked
game). The model now correctly called `run_forecast` first -- genuine
progress -- but the tool call itself then failed:
`ValueError: Query returned no non-null (timestamp, value) rows to
forecast from`. A new bug, immediately, from fixing the first one.

### Root-caused directly, and found exactly why it had been invisible

Read `_forecast()` precisely: when a query returns zero real
(timestamp, value) pairs (exactly what an untracked game's real, valid,
well-formed query returns), the function raised `ValueError` before
ever reaching its own `if n_snapshots < 2: return {insufficient_history:
True, ...}` branch -- unreachable with `n_snapshots == 0`, since that
branch is only ever reached with `n_snapshots >= 1` once you're past the
raise. The tool's entire honest-degradation design (its own docstring's
core promise) only ever actually worked for the "exactly one snapshot"
case, never true zero.

The reason this had never been caught before is the same reason it
existed at all: the model itself was routinely skipping the tool call
for exactly this scenario (the bug this slice started by fixing), so
this code path had, in practice, never actually run against a real
zero-row query before today. Fixing the prompt gap didn't just close
that gap -- it was the first thing to ever genuinely exercise this part
of the tool, and it broke immediately.

Also found, while fixing it: the existing test suite had encoded this
exact bug as correct behavior --
`test_reports_insufficient_history_with_zero_real_rows` asserted
`pytest.raises(ValueError, match="no non-null")`. Not a test that missed
a bug; a test that actively certified the wrong behavior as right,
because whoever wrote it (this session, Slice 9b) reasoned from the
code's actual behavior rather than the tool's own stated design intent.
A good reminder that a passing test only proves the code matches the
test, not that either one is correct.

### The fix, and re-verification end to end

Removed the early `raise`, let zero real points fall through to
`n_snapshots = 0`, and handled the one real edge case that introduces
(`earliest_snapshot` has nothing to report when there are zero
snapshots, not just one) by making that field `None` rather than
crashing on an empty list's `[0]`. Fixed the misencoded test to assert
the correct behavior, and added a second case (a query returning
literally zero rows, not rows with null values) to cover both real
shapes this takes.

Re-ran the exact same live question after both fixes: clean single
attempt, zero tool errors, a real `insufficient_history: True,
n_snapshots: 0` result, and a final answer genuinely grounded in that
tool result -- not reasoning about what should happen, watching it
happen.

### Verified for real

`ruff check .` and `uv run mypy src` clean. 179/179 tests pass (178 + 1
new). Both the prompt fix and the tool fix verified against a real,
live Groq call end to end, not just unit-tested in isolation -- the
kind of bug (a model skipping a tool call, a code path never actually
exercised) that a mocked test could easily miss even with the fix
correctly in place.

### Open questions (new)

- **Whether any other tool has a similar "the error path was never
  actually reachable because callers avoided triggering it" gap.** Only
  `forecast_tool.py`'s zero-rows case was found and fixed here, found by
  accident (fixing an unrelated prompt gap), not by a deliberate audit
  of every tool's own error-handling paths.
- **Whether other existing tests silently encode other real bugs as
  correct behavior**, the way this one did. Worth a skeptical pass over
  the test suite at some point -- a test asserting current behavior is
  not the same claim as a test asserting correct behavior, and this
  session found one real, if narrow, case where the two had quietly
  diverged.

## Slice 54 — Two real CI eval fixes, both traced from a fresh live run, not guessed at

**Date:** 2026-09-07

With the honesty gap closed, ran the real scheduled eval suite twice
more to see the actual effect. Both runs kept failing, but on different
questions each time -- `price_outliers`/`hoi4`/`insufficient_history`
once, then `csgo`/`hoi4`/`insufficient_history` the next. That pattern
itself was the clue: different failures on different runs of the same
suite means the *environment* is varying, not the code under test.

### Confirming HOI4's fragility precisely, not just suspecting it

`run_evals.yml`'s catalog-build step runs `ingest.py --count 100`.
Checked that file's own comment rather than assuming: SteamSpy's bulk
listing is sorted by *owners* descending, so `--count 100` is
specifically the 100 most-owned games on Steam, not the 100 most
popular by concurrent players or by whatever the poller happens to
track. Rebuilt that exact catalog locally (same command, same day) and
queried it directly: Hearts of Iron IV, a real, solid, but far smaller
game than CS:GO, genuinely isn't in it. Confirmed, not inferred --
`forecast_hoi4_next_week`'s hardcoded golden question was never going to
reliably survive a small owners-ranked sample, no matter how many times
it was retried.

### CS:GO's failure looked identical but had a different cause

`forecast_csgo_next_month` failed too, in a run where CS:GO almost
certainly *was* present (it's reliably one of the most-owned games on
Steam) -- but with only 509 tokens and 7.8 seconds, the exact signature
of a rate-limited call that never got to do real work, not a missing-
game error. Looked at what ran immediately before it:
`analysis_price_outliers` alone had used 8,380 tokens -- by itself,
almost the entire 8,000-tokens-per-minute cap Groq enforces. The
existing 20-second spacing (Slice 44) was sized for the *previous*
golden set's typical question size, not for one question nearly
saturating the whole per-minute budget on its own.

### Two independent fixes, not one, because they were two independent problems

**The hardcoded-game problem**: replaced both `forecast_has_real_projection()`
golden questions with a live query -- the two most-popular *currently-
tracked, sufficiently-historied* games, `ORDER BY peak_ccu DESC`,
`HAVING COUNT(DISTINCT polled_at) >= 2`. Written to produce 0, 1, or 2
questions gracefully depending on what actually qualifies that day,
never assumed to be exactly 2 -- a hardcoded expectation of "exactly 2"
would just be a smaller, subtler version of the same bug. Verified all
three counts directly (`test_golden_questions_forecast_selection.py`),
using the real `games_db` fixture with `player_counts` rows inserted for
real, not mocked, plus the one edge case that actually matters here (a
game with exactly one snapshot must not qualify -- `_forecast()` itself
needs two to fit any trend).

**The rate-limit problem**: raised `SPACING_SECONDS` again, 20.0 to
45.0. Reasoned honestly about why 20s couldn't have been enough this
time: when a single question can use nearly the entire per-minute
budget by itself, no amount of spacing short of nearly the full rolling
window guarantees the next call won't overlap it. A precise token-
budget-aware spacing scheme would be more elegant, but this is a
scheduled, once-a-day job -- an extra ~10 minutes of total runtime is a
real cost worth naming, but a small one next to correctness.

### Verified for real

`ruff check .` and `uv run mypy src` clean. 184/184 tests pass (179 +
5 new). The dynamic game-selection logic verified against real inserted
`player_counts` rows for all three real counts (0/1/2), not just the
common case.

### Open questions (new)

- **Whether 45s is actually enough, or whether this needs to become
  token-aware eventually** (space proportionally to the previous
  question's real usage rather than a fixed constant) -- same honest
  "first real guess, not a formally-derived number" caveat every other
  threshold in this project has gotten (`agent_retry_backoff_seconds`,
  the 48h deploy-staleness window, the 600-character chunk ceiling).
  Only a future real run will show whether 45s actually closes this for
  good.
- **Whether CI's `--count 100`-by-owners catalog is the right sampling
  strategy at all** for an eval suite whose golden questions increasingly
  depend on specific games being present. Worked around here by making
  the golden questions adapt to whatever catalog exists; the alternative
  (change what CI ingests, e.g. by owners *and* peak_ccu, or a larger
  count) was not explored.

## Slice 55 — Item #3: the last audit-adjacent fix, and a real lesson about small samples

**Date:** 2026-09-07

The last of the three real eval findings: `analysis_price_outliers`
assumed the single highest-priced game in the catalog is always a real
statistical outlier. On production's real ~1000-game catalog that's
usually true; on CI's smaller, differently-shaped sample it sometimes
isn't, and a model correctly reporting "nothing stands out" was being
marked wrong for being right -- the same shape of bug as the forecast
golden questions fixed in Slice 54, just in the analysis route instead
of forecast.

### Matching the check to the same math the tool actually uses

Read `stats_tool.py`'s own `_outliers()` function to get the exact
formula right rather than approximating it: sample standard deviation
(`ddof=1`), `z_score = (value - mean) / std`, flagged only if
`abs(z_score) > 2.5` (the same default `z_threshold` the real tool
uses). Replicated that precisely in `golden_questions.py`'s own live
query, computing the top-priced game's real z-score at eval time instead
of assuming it's automatically significant.

Branched the check itself on the real result: if the top price genuinely
clears 2.5, the model is still expected to name it (the original,
correct assertion for that case). If it doesn't, there's no single
positive fact left to assert -- "nothing stands out" has no specific
name to check for, and writing a check for what the answer must *not*
say would just be a subtler, harder-to-get-right version of the same
mistake. Left that case checking only the route, and reporting the real
z-score honestly in the reference facts the judge already sees -- which
the actual failed run's own judge output already showed handles this
correctly.

### A real lesson from trying to write the regression test

Building a test for the "genuine outlier" branch surfaced something
worth keeping: the `games_db` fixture's own 4 rows are too few to
reliably demonstrate this. Setting one game to $999.99 against three
normal-ish prices produced a z-score of only 1.50 -- an extreme, obviously-
absurd price, and it still didn't clear 2.5, purely because sample
standard deviation with n=4 is itself wildly unstable (one extreme value
inflates the very statistic meant to detect it). Had to insert a
properly-sized synthetic sample (20 tightly-clustered games plus one
real outlier) to get a statistically meaningful result either way. This
is, concretely, the exact same phenomenon the fix itself is about --
small samples make z-score outlier detection genuinely unreliable, not
just "sometimes surprising" -- encountered directly while testing the
fix, not just reasoned about in the abstract.

A second, smaller thing the same test-writing surfaced: naively
resetting all of the fixture's own prices for a clean test dataset
briefly zeroed out its only free-to-play row, which crashed a
*different* golden question's own reference-fact computation
(`f2p_review_mean`'s `AVG()` over zero matching rows returns `NULL`,
and formatting `None` with `:.3f` raises). Fixed by normalizing only the
non-free rows, leaving the fixture's real free-to-play game alone --
a reminder that golden-question fixtures interact with each other more
than they look like they do.

### Verified for real

`ruff check .` and `uv run mypy src` clean. 186/186 tests pass (184 + 2
new), both new tests exercising a real, correctly-sized synthetic
dataset rather than the fixture's own too-small one.

### Open questions (new)

- **Whether the same "assumes a fixed fact always holds against any
  catalog shape" pattern exists in any of the other golden questions**
  not touched by Slices 54/55. `lookup_avg_strategy_price`,
  `lookup_linux_count`, and similar lookup questions use
  `contains_number` with a live-computed value, which is inherently
  shape-independent (there's no "is this significant" judgment call
  involved) -- lower risk, but not formally audited for this specific
  failure mode the way the two fixed questions now have been.

## Slice 56 — Quota-free work while waiting: run_evals.py's own crash-on-crash gap

**Date:** 2026-09-07

Triggered `run_evals.yml` for real to check Slices 54/55's fixes
together. It failed -- not from any of today's code, but from a real
`groq.RateLimitError` on the **daily** token cap (200,000 TPD, 199,683
already used, purely from this session's own heavy live-verification
testing across the whole day). Nothing to fix about that directly; it's
just quota, and the honest move was to wait for the reset rather than
burn more of it chasing a clean run today.

### The real thing worth fixing while waiting

The actual traceback showed something different, though: the failure
happened on the LLM-judge call for a question *partway through* the
suite, and because `run_evals()` had no handling for it, the entire
script crashed and printed **nothing** -- not even the results for
every question that had already completed successfully before that
point. A real quota exhaustion (or any other transient failure) landing
late in a 15-question run meant losing all of it, not just the one
affected question.

This was fixable, and verifiable, without spending any more of today's
exhausted quota -- exactly the kind of work worth doing in the gap
before tomorrow's reset rather than sitting idle.

### The fix: isolate failures per-question, not per-run

Two separate try/excepts, deliberately scoped differently:

- Around the **agent call**: a failure here means there's no real answer
  for that question at all. Recorded as a genuine failure
  (`agent_result=None`, a synthetic `CheckResult(False, ...)` explaining
  why) and the loop moves on to the next question, rather than crashing.
- Around the **judge call**, separately and more narrowly: this suite's
  own docstring already states the judge score never gates pass/fail --
  so a judge failure degrades to `judge_verdict=None` while keeping the
  real `agent_result`/`check_result` that were already obtained. Losing
  a nice-to-have score shouldn't cost the actual result.

Fixing this surfaced one more small, real bug for free: `print_report`'s
route-accuracy count did `r.agent_result.route` unconditionally, which
would have raised `AttributeError` the moment a `None` agent_result
(the exact case this slice just introduced) reached it. Excluded those
questions from the route-accuracy denominator instead -- there's no
route to have gotten right or wrong when the call never produced a
result.

### Verified without touching Groq at all

Wrote `test_run_evals_resilience.py`: mocks `run_agent`, `judge_answer`,
and `_call_with_retry` directly, matching this project's own
`live`-exclusion convention. Covers all four real shapes this needed to
handle -- an agent failure recorded without stopping later questions, a
judge failure that doesn't cost the real result underneath it,
`print_report` not crashing on the new `None` case, and the synthetic
failure actually being a real `CheckResult`, not an ad hoc stand-in.
Four tests, zero live LLM calls, real coverage added to a file that
Slice 43's audit had specifically named as sitting at 0%.

### Verified for real

`ruff check .` and `uv run mypy src` clean. 190/190 tests pass (186 + 4
new).

### Open questions (new)

- **Whether `run_evals.yml` should also alert or retry differently on a
  TPD-specific rate limit** (as opposed to the TPM limit `SPACING_SECONDS`
  is tuned for) -- a daily cap can't be waited out with a short backoff
  the way a per-minute one can; this fix stops it from destroying the
  whole run's report, but doesn't make the affected question itself
  succeed. Not addressed here, since the honest fix for "the daily quota
  ran out" is "wait for the daily quota," not a code change.
- **Still pending, unchanged from before this slice**: a genuinely
  comprehensive `run_evals.yml` run covering Slices 53/54/55's fixes
  together has not yet completed, and won't until tomorrow's quota
  reset.

---

## Slice 57 — The same bug found a second time, a shared fix instead of a second copy, and a quota-free waiting-game

**Date:** 2026-09-07

### Auditing Slice 55's fix for siblings, found one

Slice 55 fixed `analysis_price_outliers`'s assumption that the single
highest-priced game is automatically a real z-score outlier. Going back
to check whether that same shape of bug existed anywhere else in the
golden question set (rather than assuming it was a one-off) turned up
`analysis_ccu_outliers`, unconditionally asserting the single highest
`peak_ccu` game "should be flagged as a clear outlier" -- the exact
same unproven assumption, never computing a real z-score.

Lower real-world risk than the price case: CS:GO's peak_ccu is so
extreme (millions of concurrent players against a catalog where most
games have low hundreds) that it clears any reasonable z-score
threshold in every catalog shape actually seen so far. But "hasn't bitten
us yet" isn't the same as "structurally sound," and Slice 55's own
DOCEXP entry existed specifically because an *unlikely* catalog shape
had already broken an *unlikely*-seeming assumption once.

### Fixing it as a shared helper, not a second copy-paste

The tempting fast path was to paste Slice 55's z-score block a second
time with `peak_ccu` swapped in for `price_usd`. Rejected that
specifically because two independent copies of the same statistical
logic is exactly the failure mode that let this bug exist twice in the
first place -- a fix applied to one copy and not verified against the
other. Instead, extracted `_outlier_check_and_reference(conn, column,
noun)` in `golden_questions.py`: computes the real z-score for the
single highest value of any given column, with the identical
branching logic Slice 55 established (a real outlier demands the model
name it; a non-outlier only requires the right route, since "nothing
stands out" has no one name to assert and inventing a check for what
the answer must *not* say would just be a subtler version of the same
original bug). Both `analysis_price_outliers` and
`analysis_ccu_outliers` now call the same function with different
columns -- one place to get right, one place a future audit needs to
re-check.

`column` is interpolated directly into the SQL string rather than
parameterized -- safe here specifically because it's always one of this
module's own two hardcoded call sites, never anything derived from user
input; the docstring says so explicitly rather than leaving it to be
rediscovered as "wait, is this a SQL injection risk?" on a future read.

### Verified for real

Added `test_golden_questions_ccu_outlier.py`, mirroring
`test_golden_questions_price_outlier.py`'s structure exactly (real
DuckDB fixture, fixture rows normalized into a tight cluster, a large
enough synthetic sample for the z-score to be meaningful either way).
`ruff check .` and `uv run mypy src` clean; 192/192 tests pass (190 +
2 new).

### The dinosaur waiting-game

Separately, while today's Groq daily quota was still exhausted: built
the opt-in "🦖 Play while you wait" feature discussed earlier
(`frontend/components/DinoGame.tsx`), the design approved via
`AskUserQuestion` -- a small button next to the existing "Ludo is
thinking…" trace panel, never shown automatically, never replacing
`TraceSteps` (the real execution trace stays exactly as it was for
anyone who wants to watch it instead).

Pure frontend, zero Groq/backend calls, and -- given everything Slice
50 taught about Render's real 512MB ceiling that same day -- a tiny
canvas endless-runner is nowhere near the scale of what actually caused
that incident (one long batch-embedded RAG chunk). Worth naming
explicitly: this feature exists *because* of that day's memory
incident, as a deliberately memory-trivial contrast, not despite it.

Implementation notes worth keeping:

- Game state (dino position/velocity, obstacles, distance, speed) lives
  in refs mutated every animation frame, not `useState` -- routing
  ~60fps updates through React state would re-render the whole subtree
  that often for no reason. Only score and phase (idle/playing/over) --
  what actually needs to reach the DOM or a screen reader -- are
  mirrored into state, and only when they visibly change.
- `prefers-reduced-motion` is respected by not showing the opt-in button
  at all in that case, reusing the existing `useReducedMotion` hook
  (`lib/useReducedMotion.ts`) rather than writing a second media-query
  subscription.
- `requestAnimationFrame` and both event listeners (`keydown` at the
  window level, filtered so typing in an input doesn't trigger a jump;
  `pointerdown` on the canvas for tap/click) are explicitly torn down on
  unmount. Slice 47's liquid-glass memory-leak fix (the vendored
  `destroy()` addition) was the direct reminder to get this right the
  first time here instead of finding it live later.
- High score persists via `localStorage`, wrapped in try/catch (private
  browsing / storage-disabled tabs just start at 0 instead of throwing).

A real TypeScript gotcha hit along the way, worth a note for next time:
`const ctx = canvas.getContext("2d"); if (!ctx) return;` does NOT keep
`ctx` narrowed to non-null inside a `function frame() {...}` declared
later in the same scope -- TS's control-flow narrowing of a `const`
doesn't carry into a hoisted function declaration, since that function
could in principle be invoked before the narrowing check runs. Fixed by
re-binding to a second, explicitly-typed `const ctx: CanvasRenderingContext2D
= context2d` right after the null check.

### Verified live, not just compiled

`tsc --noEmit`, `eslint`, and `next build` all passed clean on the real
change -- but per the `run` skill's standard (launching and driving the
actual app, not just checking it compiles), also built a temporary,
throwaway isolated test route (`app/dino-test/page.tsx`, deleted before
finishing) that mounted `DinoGame` on its own, started the real Next.js
dev server, and drove a full session against it with Playwright under a
real headless Chromium (installed fresh into the scratchpad for this,
not touching the project's own dependencies): idle screen → start on
click → repeated jumps → a genuine collision with a spawned obstacle →
"game over" state → restart, with the high score correctly persisted
across the restart via `localStorage`. Zero console or page errors the
whole way. Screenshots confirmed the canvas actually renders (dino
shape, scrolling ground dashes, accent-colored obstacles) using the
site's real theme colors, not just that the component doesn't throw.

### Real feedback from the live deploy, closed the same day

The frontend change had already gone live (Vercel auto-deploys on
push) by the time real feedback came back on it: a screenshot of the
actual shipped sprite -- a rounded blob with two leg ticks -- next to
"it doesn't look like a dinosaur." Fair: that's genuinely all it was.

Rebuilt it as a real blocky T-rex silhouette instead of a smoother
`roundRect` blob, staying in the same cheap `fillRect`-block style: a
tail block, the main body, a small back-ridge bump, a raised neck and
head/snout, a small arm stub, and an eye -- drawn as a cutout back to
the canvas's own background color (`--background`) rather than a
separate fill, so it reads against the head without needing a second
color. Legs keep the same alternating-stride idea as before, now tucked
to an even, shorter length while airborne instead of continuing to
alternate mid-jump.

Re-running the exact same Playwright verification loop from earlier in
this slice -- close-up screenshots at idle, mid-run, and mid-jump, at
3x device scale to actually see the shape -- caught two more real bugs
along the way, neither about the sprite's look at all:

1. **The first tap that starts a run also immediately jumped.**
   `jump()`'s idle-branch set the phase to "playing" and fell through
   (no `return`) into the onGround check right below it, which now
   read "playing" and immediately fired a jump. Every run started
   already mid-air. Fixed with the missing `return`.
2. **The jump's apex clipped past the canvas's top edge** -- not a
   theoretical worry, an actual screenshot showed the sprite cut off
   against the play area's top border at peak height. The arithmetic
   confirms it: `JUMP_VELOCITY=-620` gives an apex height of
   `620²/(2×1800) ≈ 107px`, and `GROUND_Y (110) - DINO_SIZE (22) -
   107 ≈ -19` -- genuinely negative, above the canvas's own y=0.
   Tuned `JUMP_VELOCITY` to -520 (apex ≈75px), leaving a real ~11px of
   clearance at peak, confirmed again with a fresh screenshot at the
   new apex.

Worth naming directly: neither bug was something a design review would
have caught -- both only showed up because the verification loop
actually pressed the keys and looked at the resulting frames, not just
confirmed the component renders without throwing.

### A second, separate real gap: the Groq-quota message never actually reaches a live user

Prompted by a direct ask: when Groq's real usage limit is hit, tell
users honestly to check back tomorrow, instead of a message that reads
like a quick retry will fix it.

The obvious place to add this looked like `src/api/main.py`'s
exception handling around the agent call -- and initially, that's where
the fix started. But tracing where a real `groq.RateLimitError` would
actually surface from found that it never gets there at all in
practice: `router_node` and `agent_node` (both fixed in earlier slices,
45 and 42) already catch *every* exception from their own LLM calls,
retry once, and degrade to a generic "try rephrasing or ask again in a
moment" AIMessage/clarifying_question if the retry also fails --
`except Exception`, not scoped to any particular failure type. A real
`groq.RateLimitError` gets silently absorbed into that same generic
text before it can ever reach `main.py`'s own exception handlers. The
`main.py`-level fix would have been effectively dead code for the exact
scenario it was meant to handle.

Fixed at the actual source instead: both nodes' final (post-retry)
`except` now checks `isinstance(exc, RateLimitError)` specifically and
returns a new, honest `GROQ_RATE_LIMIT_MESSAGE` in that case, distinct
from the generic fallback text used for every other failure type. The
retry itself is untouched -- it's still correct and useful for a
transient per-minute (TPM) blip -- the message only changes for the
case where a real rate limit survived that retry too, which is a much
stronger signal of a persistent (very likely daily/TPD) constraint.
Groq's SDK doesn't distinguish TPM from TPD at the exception-type
level, so the message is worded to stay honest either way rather than
overclaiming a specific cause: "needs a moment to recover... if this
keeps happening, check back tomorrow."

`GROQ_RATE_LIMIT_MESSAGE` is defined once, in `graph.py`, and
`main.py` imports it rather than keeping its own independently-worded
copy -- the same "one shared source, not two copies that can drift"
lesson from this same slice's `_outlier_check_and_reference()` fix.
`main.py`'s own `except RateLimitError` stays in place as a defensive
backstop (for a RateLimitError raised somewhere outside these two node
call sites, now or in some future change), just no longer the primary
path -- its comment says so explicitly.

Added 2 new tests, one per node, each constructing a real
`groq.RateLimitError` (it requires an actual `httpx.Response`, not a
bare exception -- a stand-in `RuntimeError` wouldn't exercise the
`isinstance` check this fix depends on) and asserting the honest
message wins over the generic one when both the call and its retry
fail with it.

### Verified for real

`ruff check .` and `uv run mypy src` clean. 194/194 tests pass (192 +
2 new). Frontend: `tsc --noEmit`, `eslint`, and `next build` all clean
again after the sprite/physics fixes, verified live a second time with
the same temporary-test-route-plus-Playwright approach (close-up
screenshots at 3x scale, idle/running/jumping frames), then the test
route deleted again.

### Open questions (new)

- None new from this slice specifically. The two items already tracked
  from Slice 56 (TPD-specific rate-limit handling; the still-pending
  comprehensive `run_evals.yml` run) remain open, unaffected by this
  slice's changes. (The TPD-specific item there is about `run_evals.py`
  specifically -- this slice's Groq-rate-limit fix is the live `/ask`
  path, a different piece of code, and doesn't close that item.)

---

## Slice 58 — The first real run_evals.yml run after Slice 57, and what it actually found

**Date:** 2026-09-08

Today's scheduled `run_evals.yml` run was the first to cover Slices
53-57's fixes together -- the comprehensive run that's been listed as
"still pending" since Slice 56. It mostly passed (14/15 route accuracy,
14/15 deterministic checks, avg judge 4.3/5), and reading the actual
report line by line turned up one real, CI-failing bug and one real,
lower-severity prompt gap -- plus one thing that looked like a bug and
wasn't.

### The real failure: analysis_price_outliers routed to lookup

`analysis_price_outliers` ("Are there any games with an unusually high
price compared to the rest?") came back `expected route='analysis', got
'lookup'`. Routed to `lookup`, the model has no `run_stats` tool bound
at all -- and rather than saying so, it answered anyway, inventing a
"$59.99" price and a "$16.45" average that don't exist in the real
data. That's the actual reason the workflow exited 1.

The useful clue sat right next to it in the same report:
`analysis_ccu_outliers` -- the identically-shaped question, just about
`peak_ccu` instead of `price_usd` -- routed correctly to `analysis` in
this same run. Same phrasing pattern, same day, same model, opposite
outcome. That rules out "the question is inherently unanswerable by the
router" and points at something genuinely borderline: the router's own
system prompt gives the `analysis` category exactly one example, and
it's a group-comparison question ("How does the average price of Action
games compare to free-to-play games?"), not an anomaly/outlier-shaped
one -- even though `router.py`'s own module docstring names "anomalies"
as a canonical `analysis` case. The category was well-documented in
prose and completely undemonstrated in the one place that actually
shapes the classifier's behavior.

Fixed by adding a second example to the `analysis` bullet in
`ROUTER_SYSTEM_PROMPT`, deliberately about a metric neither golden
question touches (negative review counts, not price or peak_ccu) --
using either existing golden question's own exact wording as the new
example would have let the router pattern-match that literal phrasing
rather than generalizing the category, quietly turning the golden set
into something closer to a memorized answer key than an independent
check.

### The related, lower-severity finding: an outlier answer padded with an unverified extra

`analysis_ccu_outliers` passed its deterministic check (it named
Counter-Strike: Global Offensive, the real outlier) but scored only 2/5
from the judge: the model also volunteered PUBG: BATTLEGROUNDS as a
second outlier with a specific number, which `run_stats` never actually
flagged. Not a routing problem this time -- the model got to the right
tool and the right primary answer, then editorialized past what the
tool returned, presumably because PUBG's peak_ccu also looks large in
absolute terms even though its real z-score doesn't clear the
threshold.

`ANALYSIS_TOOL_GUIDANCE` already had a near-identical guardrail for
`compare_two_groups` (Slice 4's group-mislabeling fix: don't label a
group with a name implying a filter you didn't apply). Added the
sibling instruction for `outliers`: report only the rows the tool
actually flagged, not other rows that merely look large or small --
"go by what the tool returned, not by which numbers happen to catch
your eye."

### What looked like a bug and wasn't: a judge false negative

`forecast_tracked_game_1_next_month` passed its deterministic check but
scored 1/5 from the judge, which flagged the model's numbers as
unsupported by the reference facts. Checked the real
`forecast_result` from that exact run: `projected_value=541617.9` and
`observed_span_days=14.598` -- which is exactly what the model reported
("~542,000" and "~15 days"). The judge read "not present in the
reference facts text" as "wrong," when the reference facts (deliberately
prose, not exact numbers -- see `golden_questions.py`'s own module
docstring) were never meant to spell out a specific projected value for
the judge to string-match against.

Left alone rather than "fixed": the judge score has never gated
pass/fail, specifically because an LLM judge is itself fallible in
exactly this way (Slice 43's own reasoning for keeping deterministic
checks as the real gate). Worth a second look only if this same pattern
recurs against the same question shape.

### Verifying a prompt-wording fix without a red/green cycle

Both real changes this slice (the router example, the outliers
guidance) are wording changes to an LLM's own instructions, not
deterministic code. There's no meaningful TDD cycle for "does the
classifier route this correctly" -- the existing `test_router_node_llm_errors.py`
suite mocks `classify_question` entirely, so it verifies the graph's
error-handling wiring, not classification quality; it passed unchanged,
which only confirms this fix didn't break anything, not that it
improved anything.

`ruff check .` and `uv run mypy src` clean, 194/194 tests pass, no
regressions. But the real verification -- does `analysis_price_outliers`
actually route correctly now, more reliably than "the one run where it
didn't" -- needs a live `run_evals.yml` run, and today's Groq quota is
already spent from this same investigation's own context (this session
started from reading the failed run's logs, not from making new live
calls, so no quota was burned confirming any of this -- but none is
left over to *re-run* the suite today either). Pending tomorrow's reset,
same as the comprehensive run itself was pending until today.

### Open questions (new)

- Whether the router-prompt fix actually resolves the misroute
  reliably, or just shifts which borderline question flakes on a given
  day, won't be known until at least one more live run. LLM
  classification at temperature 0 reduces but doesn't eliminate
  run-to-run variance on a hosted model -- a single passing re-run
  would be reassuring, not conclusive.
- The judge false-negative pattern (flagging a correct answer because
  its specific numbers aren't spelled out in prose reference facts) may
  recur for other forecast questions with real, data-dependent
  projected values. Not fixed here since it doesn't gate anything;
  worth a dedicated look if it keeps showing up and starts masking a
  judge score that would otherwise catch something real.

### Follow-up, same day: is 15 questions too few?

Asked directly, after walking through the above. The honest answer had
two halves. Yes, statistically: one flaky classification had just moved
the headline score by 6.7% (1/15) -- that's real noise, not a stable
signal of regression vs. one-off. But "just add more" wasn't the right
instinct either -- every question costs real Groq tokens and ~45s of
spacing, runs daily in CI indefinitely, and shares a daily token budget
with the live site's own real visitors. Growing the set is a recurring
cost, not a one-time one, and past a point more questions mostly buy
statistical smoothing, not new bug-finding surface.

So: targeted, not blanket. Two real motivations, not one: diluting the
one route's flakiness, and a genuine coverage gap found while looking
for where extra depth would actually mean something -- `run_stats` has
three modes (`compare_two_groups`, `outliers`, `describe`) and the
golden set had end-to-end coverage of exactly two of them. Nobody had
noticed `describe` had zero coverage until asked "where would growing
this actually help."

Added three questions: `analysis_metacritic_score_distribution` (closes
the `describe` gap, and specifically chose `metacritic_score` because
its real NULL-for-unscored-games quirk exercises `_describe()`'s NULL-
filtering, not just its arithmetic), `analysis_achievements_vs_review_scores`
(a second `compare_two_groups` question on category tags, computed live
via a real Welch's t-test rather than assumed -- deliberately not
repeating `analysis_f2p_vs_paid_review_scores`'s own pre-existing "p <<
0.001" hand-wave, though that question's fragility is the same kind and
wasn't touched here), and `analysis_discount_outliers` (a third outlier
question, via the already-shared `_outlier_check_and_reference()`
helper, on a column neither existing outlier question touches -- proof
the helper generalizes, and one more independent analysis-route sample).
Added `stats_result_has_mode()` to `checks.py` for the `describe`
question to verify the real tool actually ran in that mode, not just
that a plausible number showed up in the answer.

### A second real bug, this time in code written five minutes earlier

Writing the achievements question's live significance check, the first
version hard-asserted at least 2 games in each achievement group before
computing anything. Running the test suite immediately after: 9 failures,
all in files that have nothing to do with achievements at all --
`test_golden_questions_ccu_outlier.py`, `test_golden_questions_forecast_selection.py`,
`test_golden_questions_price_outlier.py`. The reason: the small, 4-row
`games_db` fixture those tests use for entirely unrelated purposes has
exactly one game tagged "Steam Achievements" -- one short of the two a
real t-test needs -- and `build_golden_questions()` is called by all of
them, so a hard assert on a narrow catalog-shape assumption crashed
construction for every single one, not just the one new question that
couldn't be answered.

This is precisely the same class of fragility this whole slice already
existed to fix (a hardcoded assumption about catalog shape), just caught
in code that was minutes old instead of a year old. Fixed the same way
the forecast questions already handle it: `achievements_question` (and,
applying the same reasoning proactively, `metacritic_question`) are now
built as `GoldenQuestion | None`, appended to the list only when the
real data supports them, instead of asserted to always exist. Added a
direct regression test for the exact scenario that broke
(`test_achievements_question_absent_with_the_fixtures_own_single_achievement_game`).

### Verified for real, twice

`ruff check .` and `uv run mypy src` clean, 202/202 tests pass (194 + 8
new: 3 for `stats_result_has_mode()`, 5 in a new
`test_golden_questions_metacritic_and_achievements.py`). Then, beyond
the test fixtures, sanity-checked all three new questions against the
real, full local catalog (`data/db/games.duckdb`, not the tiny 4-row
fixture) -- 18 total questions now, and the real numbers are sane: 568
games with a real Metacritic score (mean 80.6, median 82.0), a real
p=0.0007 on the achievements comparison, a real z-score-clearing
discount outlier. Nothing degenerate, nothing that only works on a toy
sample.

### Two real runs later: the first fully clean comprehensive run, and two more real findings

Two live `run_evals.yml` runs followed this slice's own work, same day.
The first (a re-run of an existing workflow run, which GitHub Actions
ties to the commit that originally triggered it) confirmed the router
fix directly: `analysis_price_outliers` now shows `got 'analysis'`,
15/15 route and deterministic accuracy. The second, a genuinely fresh
trigger against current `master`, ran all 18 questions for the first
time and came back **18/18 route accuracy, 18/18 deterministic checks**
-- the first fully clean, comprehensive run this project has had,
closing an item that had been "still pending" since Slice 56.

The judge scores (4.1/5 avg) on that second run were worth reading
closely rather than accepting as "good enough" -- they split into two
genuinely different problems.

**A gap in reference_facts, not the model.** `analysis_metacritic_score_distribution`
got dinged for reporting real percentiles (81, 89) as "unsupported" --
but `_describe()` always computes `p25`/`p75` as part of its real
output; `metacritic_reference_facts` just never restated them. Both
forecast questions got the same treatment: the model's specific numbers
(699,000 / 306,000) were the actual `projected_value` from that run's
real `forecast_result`, flagged as fabricated purely because
`reference_facts` never stated a number to check against -- deliberately
left out originally, since a forecast's real value is data-dependent and
computed live, not known at golden-question-authoring time.

Fixed by computing that real value at construction time too, the same
"call the real tool" principle `_outlier_check_and_reference()` already
established: `execute_run_forecast()` called directly (needed
`tracked_forecastable_games`'s query to also select `appid`, not just
`name`). Safe to precompute specifically because `_forecast()` is a pure
function of `player_counts`' own stored timestamps/values and
`horizon_days` -- never wall-clock `datetime.now()` -- and nothing
writes to that table between golden-question construction and the live
agent's own call moments later in the same CI job, so this is the exact
number the tool will independently recompute, not a guess at it. Same
fix, same reasoning, for the metacritic question's own p25/p75.

**A real model behavior the Slice 58 guidance sentence didn't actually
fix.** `analysis_ccu_outliers` (judge 2/5, both runs: adds PUBG
unprompted) and the brand-new `analysis_discount_outliers` (judge 1/5:
"lists multiple games as outliers" when only one is real) both show the
identical pattern Slice 58 already tried to close with one guidance
sentence -- and that sentence plainly didn't work, confirmed now across
three different outlier questions and two separate live runs.

Reading the model's own answer format was the useful clue: both
failures rendered as a markdown table of several games with their raw
values, not a single flagged name -- the signature of a `run_sql`
"top N" query built for display, then presented as if every row in it
were a real outlier, alongside (or instead of) what `run_stats` actually
flagged. The original guidance sentence never said anything about that
specific combination.

Rewrote `ANALYSIS_TOOL_GUIDANCE` to be concrete instead of principled:
names the exact `"outliers"` field as a strict allow-list ("if it has
one entry, your answer names exactly that one game"), and explicitly
forbids the exact mechanism just found ("do NOT also run a separate
'top N by value' run_sql query to build a table for this kind of
question"). Whether this actually changes the behavior -- a
prompt-wording fix has no red/green unit test, the same limitation as
the router-prompt fix earlier in this slice -- is only checkable by
another live run.

### Verified for real

`ruff check .` and `uv run mypy src` clean, 202/202 tests pass (no
regressions from either fix). The updated forecast and metacritic
reference facts were also sanity-checked against the real, full local
catalog: real numbers came back (e.g. a real ~56,310-player projection
for CS:GO, real p25/p75 of 76.0/86.0 for Metacritic scores), nothing
degenerate. The outlier-guidance rewrite's actual effectiveness remains
open until the next live `run_evals.yml` run.

---

## Slice 59 — Stop asking the model to be accurate; make it structurally unable to be wrong

**Date:** 2026-09-09

### The question that reframed the whole approach

Asked directly: "is there a better solution? What would an AI engineer
do?" -- a fair challenge, because the honest answer was that Slice 58's
fix was still the *weaker* of two well-known approaches to this problem
class. It made the request for accuracy more specific (name the exact
field, forbid the exact mechanism observed), but it was still a request
-- a probabilistic ask the model could, and twice did, fail to honor.

The reframe: if a fact already exists in code (`stats_result`, straight
from `run_stats`), never let the model regenerate that fact in free
text at all. Detection-after-the-fact (the original plan discussed
before this reframe: a new graph node, a retry loop, a string-matching
heuristic) is strictly weaker than prevention, for one concrete reason
-- detection has to correctly judge a moving target (arbitrary model
phrasing) and can always miss a paraphrase or false-positive on an
honest hedge. Prevention has no target to miss: if the model never
writes the sentence, there is no sentence to get wrong.

This is "make invalid states unrepresentable," applied to LLM output
instead of a type system. The general pattern -- **render facts
deterministically from structured tool output; let the LLM narrate,
never restate** -- is exactly how a lot of real production
analytics/BI copilots handle this same class of risk, and it's the
answer worth having ready for an interview: prompting reduces
hallucination *rate*; only removing the model's opportunity to generate
the fact removes the *failure mode*.

### Why simply appending a correct block isn't enough

The naive version of "render facts in code" -- print the correct
outliers list, then just tack the model's full original paragraph on
after it -- doesn't actually fix anything: if the model's own paragraph
*still* says "PUBG is also an outlier," the user now sees both the
correct block and the wrong claim sitting side by side. The fix has to
shrink what the model is even asked to produce, not just add a correct
answer next to a possibly-wrong one.

### The actual design

`ANALYSIS_TOOL_GUIDANCE` (`prompts.py`) now tells the model, for
`mode="outliers"` specifically, that its final answer must not name any
row, value, or z-score at all -- only a short (1-2 sentence),
completely name-free interpretive comment. The prompt says outright
that asking it to restate the list accurately has already failed twice
for real, so it's simply not the model's job anymore. This shrinks the
model's writing surface from "the whole factual answer" to "one
sentence of color commentary with nothing left in it to hallucinate."

`_render_outliers_fact_block()` (`graph.py`) is the actual fact: an
f-string over `stats_result`'s real `outliers` list and `z_threshold`,
naming the real outlier(s) with their real z-scores, or honestly
stating nothing clears the threshold. This text is never generated by
an LLM call -- it cannot be wrong in a way a prompt could fail to
prevent, because there is no prompt involved in producing it at all.

`_compose_outliers_answer()` combines the two: fact block first, then
the model's commentary. One defensive backstop, since prompt compliance
still isn't literally 100%: if the model's commentary *still* names an
unauthorized row from a companion `run_sql` query (the exact mechanism
behind the original hallucination -- the model running a "top N" query
to build a display table, then treating every row in it as a flagged
outlier), the commentary is swapped wholesale for a safe, generic,
name-free sentence rather than surgically edited. Wholesale replacement
over clause-level editing was a deliberate simplicity choice: trying to
remove just the offending clause from free text risks leaving broken
grammar behind (a dangling "and," an incomplete sentence), and the fact
block above it is completely unaffected either way, so the backstop
firing costs nothing but a slightly blander sentence, never
correctness.

Wired into `_result_from_state()` -- the one real choke point both
`/ask` and `/ask/stream` already funnel every answer through -- so both
endpoints get the guarantee automatically, no duplicated logic per
endpoint.

### A genuinely nice side effect: zero eval-harness changes needed

`analysis_ccu_outliers`/`analysis_price_outliers`/`analysis_discount_outliers`'s
existing golden-question checks (`contains_text(name)` for a real
outlier, `route_is` alone for the honest "nothing clears the threshold"
case) check the final `answer` string -- which still contains the real
name, now via the fact block instead of the model's own prose. These
checks are, if anything, now *more* reliably satisfied than before,
since the name's presence in the answer no longer depends on the model
writing it correctly at all.

### Verified for real

Added `test_graph_outliers_answer_composition.py`: 7 tests, pure
functions of `stats_result` and plain strings, zero LLM calls needed --
the fact block's exact rendering (one outlier, several, the honest
empty case) and the composition/backstop logic (clean commentary passes
through untouched; an unauthorized name gets swapped for the safe
fallback while the fact block stays intact regardless; a real outlier's
own name mentioned in commentary is never mistakenly flagged; no
companion query means nothing to check against). `ruff check .` and
`uv run mypy src` clean, 209/209 tests pass (202 + 7 new).

### What's actually guaranteed versus what still needs a live check

Worth being precise about the difference: the *naming* failure mode is
now architecturally impossible for the primary path, not just less
likely -- that text is simply not something the model writes anymore,
so there's nothing left to verify live for that part specifically. The
defensive backstop is different: it still depends on the model's
short commentary text and a companion-query heuristic, so *that* path's
real effectiveness is exactly as unconfirmed as any other prompt-facing
logic until a live run exercises it. The next `run_evals.yml` run should
show `analysis_ccu_outliers`/`analysis_discount_outliers` with clean
judge scores now that the model has nothing factual left to get wrong
in its own text -- if the backstop fires at all, that's itself a signal
worth reading closely, not just accepting silently.

### Open questions (new)

- Whether the same "render facts, narrate around them" pattern should
  extend to `compare_two_groups` and `describe` too. Not done here,
  scoped deliberately to `outliers` only (the one mode with an actual
  observed, repeated hallucination) -- `compare_two_groups`'s risk
  shape is different (mislabeling an existing group, already covered by
  Slice 4/30's own check) and `describe`'s only real gap found so far
  was a reference-facts completeness issue (Slice 58/59's own p25/p75
  fix), not a naming hallucination. Worth revisiting only if either mode
  shows this same failure pattern for real.