Set up outer loop gen-1 run with expanded gate and holdout validation

[lukeinglis]

Context

Gen-0 outer loop run ($521, 82 hours) resulted in a full revert — see docs/outer-loop-retrospective.md. Remote-factory has since shipped fixes for the two critical problems: worktree isolation per candidate (no more auto-merge baseline confounding) and holdout validation support. Time to test whether gen-1 produces better results.

Setup checklist

1. Update factory CLI

Currently installed: 0.3.1.dev162 (46 commits behind). Reinstall to pick up worktree isolation, holdout validation, MAP-Elites, and knob/prompt mutation improvements:

uv tool install git+https://github.com/akashgit/remote-factory.git --force

2. Expand gate rounds

Currently pinned: 2026-03-01, 2026-04-12 (2 rounds — too narrow, allowed gen-0 to overfit).

Proposed split across 6 rounds:

• Training (4 rounds): 2026-03-01, 2026-04-12, 2026-05-10, 2026-06-07
• Holdout (2 rounds): 2026-07-05, 2026-08-02

Update gate_rounds.json to include the 4 training rounds (the gate scores these). Holdout rounds are configured in the outer loop config, not the gate.

~155s per round, so 4 training rounds ≈ 10 min per candidate gate evaluation.

3. Update outer loop config

Update .factory/outer_loop/config.json:

• Set training_instances and holdout_instances with the round split above
• Keep population_size: 3, budget: 9 (small test run)
• Verify test_command: "uv run pytest gate/ -q" still correct

4. Trim the backlog

Current .factory/strategy/backlog.md has 4 items, some stale. Trim to forecaster-relevant items only using .factory/scripts/trim-backlog.sh (directives still aren't wired in the outer loop, so sub-CEOs scan the full backlog).

5. Reset outer loop state

Clear .factory/outer_loop/state.json (still has gen-0 data).

6. Re-baseline the gate

After expanding rounds, run make_manifest.py --set-baseline to update gate_baseline.json. This makes real API calls (~$8-12 for 4 rounds).

7. Run gen-1

factory outer-loop calibrate ~/projects/work/forecastbench --benchmark forecastbench --population-size 3 --budget 9

What we're testing

• Does worktree isolation prevent auto-merge baseline confounding?
• Does holdout validation catch overfitting?
• Do knob/prompt mutations produce more targeted changes than gen-0's random backlog picks?
• Any new issues to flag upstream on remote-factory?

Related

• Gen-0 retrospective: docs/outer-loop-retrospective.md
• Remote-factory rename issue: Rename FeatureBench-branded outer loop classes to generic names akashgit/remote-factory#1421
• Remote-factory outer loop v2: feat: Outer Loop v2 — evolutionary workflow search with E2E validation akashgit/remote-factory#1284