More Effort, No Miracle

Reproducible companion code and privacy-minimized derived data for the research note:

• English: More Effort, No Miracle
• 中文：大力出不了奇迹

The analysis studies adjacent reasoning-effort comparisons on the frozen DeepSWE v1.1 snapshot from 2026-08-18. It asks whether additional test-time resources above High produce task-success changes that are distinguishable from four-run variation.

Main result

Across 12 High→XHigh and XHigh→Max comparisons covering 1,279 matched comparison-instances:

• 11 of 12 observed score differences stayed inside the 95% interval from 100,000 within-task random regroupings;
• 579 instances changed their four-run success count, and 416 of those changes (71.8%) differed by only one success;
• no instance moved from 0/4 to 4/4;
• the mean pairwise overlap of regressed-task sets was only 0.140 for High→XHigh and 0.083 for XHigh→Max;
• measured steps, tokens, and recorded cost continued to rise.

This does not prove that every observed change is random, that effort above High has zero effect, or that High is a universal optimum. It shows that most High+ changes in this snapshot are not reliably separable from finite-repeat variation under this analysis.

Reproduce

Requirements: uv and Python 3.12.

uv sync --extra dev --frozen
uv run deepswe-effort
uv run pytest

The command rewrites:

• results/summary.json
• results/permutation_results.csv
• results/comparison_metrics.csv
• results/figures/*.png

The Monte Carlo procedure uses a fixed base seed (20260819) and a stable per-comparison seed derived from the model and effort pair.

Data boundary

The official DeepSWE v1.1 trial index is public at https://deepswe.datacurve.ai/artifacts/v1.1/trials.json, but that URL is mutable. The article used a frozen 2026-08-18 copy with SHA-256:

13d6f7563330110231b008ae4eb38e03de24af08acead840de296d1127144971

This repository does not redistribute the 43 MB trial export, trajectories, model patches, verifier output, logs, trial IDs, timestamps, or task names. Instead, data/ contains only the minimum derived values required to audit the published claims. Task identifiers are stable one-way hashes.

See METHODS.md and data/README.md for the exact population, unit of analysis, health rule, randomization test, and fields.

Repository map

data/                         privacy-minimized frozen inputs
src/deepswe_effort_saturation analysis and figure code
tests/                        article-number regression tests
results/                      regenerated tables, summary, and figures

License

Code is released under the MIT License. The derived tables are provided for auditing this analysis; they do not grant rights to any upstream benchmark task content, which is not included here.