Community benchmark: measure the engineering loop across real tasks

[Phelan164]

Goal

Collect reproducible evidence about when a Codex engineering-loop skill improves
software work—and when it only adds overhead.

The repository currently includes a
three-way measurement protocol
for comparing:

1. no skill;
2. the full engineering-loop skill; and
3. a lean version of the loop.

One example is not enough to establish efficiency. We need results from
different languages, repositories, task types, and Codex configurations.

How to contribute

Run the same bounded engineering task under the three conditions above. Keep
the model, reasoning effort, repository revision, tools, task prompt, and done
conditions constant.

Record:

• task type and stack;
• Codex model and reasoning effort;
• completion time;
• tool calls or context usage when available;
• focused and required checks;
• regressions or review findings caught;
• human corrections required;
• result quality; and
• anything that could not be verified.

Remove credentials, proprietary source, customer data, and identifying
information.

Then either:

• post a concise result in this issue; or
• fork the repository and open a PR containing a sanitized Markdown report
  based on the measurement protocol.

Negative and neutral results are especially valuable. The goal is to discover
where the workflow earns its cost, not to manufacture a productivity claim.

Useful comparisons

• bug fix versus new feature;
• frontend versus backend;
• small repository versus monorepo;
• full loop versus one specialist skill;
• single-agent execution versus bounded subagent delegation; and
• low, medium, and high reasoning effort on the same task.

Done

This experiment becomes useful once we have at least five independently
reproducible reports covering three or more engineering contexts. Findings will
be summarized with limitations, and the skills will be simplified where the
evidence shows redundant steps.

[Phelan164]

First controlled three-way seed run is ready in #19.

GPT-5.6-sol at medium reasoning completed the same backend boundary task under no repository skill, engineering-loop v0.2.0, and engineering-loop v0.4.0. All three passed the quality gates and produced the same six-line fix. The no-repository-skill control used the fewest reported tokens and retries on this bounded task; the lean skill used fewer tokens than v0.2.0 but did not beat the control.

The report preserves CLI usage fields, routing behavior, limitations, and exact setup. It is one seeded task, not a general efficiency claim. Frontend, infrastructure, and independent repository replications remain open.