Add continuous agent research with isolated execution - #77
Conversation
|
Important Review skippedToo many files! This PR contains 118 files, which is 18 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry. ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (118)
You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…ctive limits Remove cache-dependent base substitution, share reviewed public artifact declarations, and add pin consistency checks. Add architecture budgets for the two policy declaration modules.
Keep real operator and evaluator scripts, replacing only Harbor execution with deterministic local artifacts. Block network sockets throughout descendant processes and verify gate, record, champion, analysis and GEPA admission outcomes.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5dbf7a7d36
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if receipt.get("status") == "queued" and receipt["action"]["type"] == "finish_research": | ||
| drain_action(workspace) | ||
| else: | ||
| _close_budget(workspace, reasons) |
There was a problem hiding this comment.
When the last permitted controller attempt queues any action other than finish_research, that attempt immediately makes controller_status(...)["exhausted_reasons"] contain attempts, so this branch calls _close_budget before drain_action. _close_budget then marks the queued request resolved without executing it and finishes research with the previous champion. Thus, with the default finite --max-attempts (or a token/controller-cost limit crossed by the attempt), the controller's final fork, commit, or evaluation decision is silently discarded even though executing it requires no further controller attempt; drain eligible queued work before applying controller-only exhaustion.
Useful? React with 👍 / 👎.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
1 participant
An outer researcher needs to change its method and publish candidates without losing its notes, budget history, or recovery state. This adds continuous
evolve agentsessions and opt-in container execution for research, operators, and target candidates.Changes
Agent Driven has one continuous lifecycle: require an objective and immutable optimizer bundle; publish a best candidate and continue, or finish research explicitly. Remove batch mode,
submit_champion,--mode,--continuous, and controller prompt fallback branches. Fixed recipe drivers remain independent.Rename sealed acceptance to
seal-research; evaluate the initial and final candidates after research finishes. Old batch manifests are unsupported and require a fresh research workspace; existing continuous sessions retain their history.Add typed, receipted actions; persistent research notes; immutable method activations; publication without ending the session; cumulative usage and interrupted-action recovery.
Export exact target-only candidate packages and run them through a Docker sandbox or an isolated Harbor adapter. Keep evaluator files, private Git state, and host credentials outside the research container; expose an explicit aggregate feedback projection.
Add a host-owned model broker, local HTTP bridge, and Codex controller file transport. Unknown usage blocks further work.
Enforce bounded tmpfs output, archive validation and log capture. An independent supervisor owns deadlines and removes containers after launcher death. Preserve executable modes in method snapshots and worker downloads; new method identities include executable metadata.
Centralize host resource policy and record effective limits in receipts. Use a runtime pin manifest with consistency checks; setup validates and reuses local images by immutable ID and actual tool versions, with explicit --rebuild replacement. Consolidate the existing public artifact/metric allowlists into a versioned host-owned declaration.
Include the supporting Harbor cancellation, recovery, runtime-plan and exception-accounting changes, two full Terminal-Bench profiles, and recipe-selected Codex versions. Existing recipes retain mutation Codex 0.146.0 and seed Codex 0.143.0; the new full Codex profile explicitly selects 0.149.0. Setup accepts custom recipe paths and passes their mutation version and image tag to Docker. Target evaluator versions and the outer controller version remain independent. Existing benchmark model settings from current main are preserved.
Review the session/action modules first, then candidate/runtime isolation, then Harbor integration and recipe changes. The implementation is consolidated onto current main; experiment output and the obsolete qualification plan are excluded.
agent preparefor development-only research baselines; continuous sealed acceptance evaluates initial and final candidates after finish.Validation
After simplifying to the continuous lifecycle: 104 related tests passed, including two candidate cycles, method adoption, publish-and-continue, pause/resume, process interruption recovery, the Codex wrapper, and all nine legacy recipe regressions. Runtime code decreased by 143 lines. No real model calls.
Nine legacy recipes complete a generation with unchanged recipe/operator configuration and real evaluator scripts; only Harbor execution is substituted with deterministic local artifacts. Linux seccomp blocks network sockets in the CLI and descendants. Assertions cover mutation, analyzer success, GEPA admission, certified evaluation, gate, record, and champion update. Separate native local-Harbor GEPA regression also passed without models. These are execution regressions, not benchmark quality measurements.
Five real child-process exit points through directory replacement and enqueueing recover exactly one published action; related slow recovery suite: 37 passed.
Existing Codex 0.146 image passed an offline read-only CLI/tool probe without rebuilding.
Default suite: 1,542 passed, 17 skipped (nine additional full-generation cases are in the opt-in slow tier).
Prior isolation implementation, unchanged by the recipe version update — real Docker regressions: 4 passed (output quota, timeout, broker round trip, launcher SIGKILL cleanup).
Built wheel and sdist; release-artifact checks: 2 passed, including policy declarations and the runtime pin manifest.
Ruff, formatting, ty, lockfile, diff checks, and strict MkDocs build passed.
A real Docker controller run completed three turns with method adoption and continued research after publication, using the copied framework and a fixture model broker. Earlier integration checks also exercised native Harbor evaluation and recovery after an actual SIGKILL without duplicate evaluation. Model responses were fixtures: these checks establish execution paths, not autonomous research improvement. No paid model calls were made.
Current limits
Version selection is covered by setup command-contract tests and seed override tests; arbitrary Codex releases have not been benchmarked for compatibility. Setup reuses validated local images and builds missing ones; mismatches require explicit --rebuild. Validation checks the tool/version contract, not complete environment equivalence.
Isolation requires explicit configuration; legacy host execution remains available. The candidate adapter currently supports single-step trials, and isolated operators are offline. Candidate usage is adapter-reported rather than independently metered. Isolated controller imports use a flushed host journal with roll-forward directory recovery and idempotent enqueueing. Unknown controller usage still requires reconciliation. Operator output import remains non-transactional. Model budgets are checked between calls, and the HTTP bridge buffers streaming responses.
Sandbox images require
/bin/sh,tar,sleep,catandtouch. Output is capped at 64 MiB (32 MiB per file, 4096 handoff entries); logs are bounded and truncated. Snapshot transport adds copying overhead. Cleanup after launcher death requires a surviving supervisor and reachable daemon; simultaneous host/supervisor failure still needs reconciliation. Seedocs/reference/operators.mdfor the exact limits.Ready for review. Deployment and merge are outside this change's delivery scope.