Set up outer loop with multi-benchmark support from remote-factory #1332

[lukeinglis]

Context

Generation 0 of the outer loop ran with --benchmark featurebench (copied from docs) instead of a proper forecastbench benchmark config. Sub-CEOs received no task instruction and picked random backlog items, making topology comparison meaningless.

remote-factory PR akashgit/remote-factory#1332 adds multi-benchmark support with TOML configs, pluggable evaluators, and a built-in forecastbench.toml. That PR is now merged.

Work

1. Update remote-factory/ — pull the latest remote-factory to pick up #1332

2. Create project-local benchmark config at .factory/benchmarks/forecastbench.toml matching our actual gate setup:

   [meta]
   name = "forecastbench"
   description = "ForecastBench — Brier Index gate with graded pytest ladder"
   
   [test]
   format = "pytest"
   command = "uv run pytest gate/ -q"
   timeout = 600
   
   [instances]
   format = "directory"
   
   [seed_workflow]
   name = ""
   
   [scoring]
   method = "partial_credit"

3. Re-run calibrate with the correct benchmark:

   factory outer-loop calibrate ~/projects/work/forecastbench \
     --benchmark forecastbench \
     --population-size 3 \
     --budget 9

   The test command and format should auto-resolve from the TOML.

4. Evaluate and compare against generation 0 results

Known limitation

InnerLoop.step() still doesn't pass --focus to sub-CEOs — tracked at akashgit/remote-factory#1338. Until that lands, sub-CEOs will still pick work from the backlog rather than being directed to improve the Brier Index. A workaround is to trim the backlog to only contain the target task before running evaluate.

Triage: open PRs from generation 0

• Improve observability: add structured logging to tournament, analyze, and verify_parity #158 and Improve observability: add structured logging to tournament, analyze, and verify_parity #160 — duplicate logging PRs from two candidates that independently picked the same backlog item. Close Improve observability: add structured logging to tournament, analyze, and verify_parity #160, review Improve observability: add structured logging to tournament, analyze, and verify_parity #158 if the logging is useful.
• Integrate CutoffEnvironment into eval pipeline #159 — CutoffEnvironment integration. Genuine forecasting change but needs controlled evaluation outside the factory (cache interaction unclear). Keep open.