Community benchmark: measure the Living Wiki against normal repository search

[Phelan164]

Goal

Measure whether an index-first, review-first Codex wiki improves repeated engineering and documentation tasks enough to justify its context and maintenance cost.

Karpathy's LLM Wiki pattern is promising, but the repository will not claim token or productivity gains without comparable runs. The planned protocol is documented in knowledge/experiments/wiki-efficiency-baseline.md.

Three conditions

Run the same bounded question or maintenance task in fresh copies of the same repository revision:

1. Normal search — existing repository documentation, with no wiki instruction.
2. Index first — start at knowledge/index.md, then read only relevant pages.
3. Wiki skill — use maintain-codex-wiki for a query or explicitly authorized write operation.

Hold the Codex model, reasoning effort, repository revision, tools, permissions, prompt, and done conditions constant. Rotate condition order across tasks when practical.

Useful task shapes

• find a prior architecture decision;
• explain why an engineering rule exists;
• identify every page affected by an updated source;
• detect a stale or contradicted claim;
• answer a multi-hop question spanning several documents;
• perform exact identifier or file lookup; and
• correctly state that the repository has no evidence.

Exact lookup and no-evidence cases are important: a wiki is not expected to win every retrieval shape.

Record

For every run, preserve:

• task and acceptance criteria;
• model, reasoning effort, Codex surface, and tool profile;
• repository and skill revisions;
• answer correctness and completeness;
• citation precision and unsupported claims;
• relevant files found and affected pages identified;
• input/output tokens when the surface exposes them;
• elapsed time and tool calls;
• stale-claim or contradiction detection;
• duplicate prose introduced;
• human corrections and review time; and
• anything that remained unverified.

Do not estimate hidden token or cost data. Remove credentials, proprietary content, customer data, private conversations, and identifying information.

How to contribute

Either:

• post a compact sanitized result in this issue; or
• fork the repository and open a PR adding a reproducible report under examples/measurements/.

Negative and neutral results are welcome. If normal search is better, that is useful evidence for simplifying or removing wiki behavior.

Decision rule

Recommend the wiki only when repeated runs preserve answer quality, improve evidence recall or stale-claim detection, and reduce repeated context discovery without creating factual drift. A 30% reduction in files loaded or input tokens is a useful target, not a promised result.

This experiment becomes directionally useful after at least five independent reports covering three repository or task contexts.