feat: tool_discovery solver — agent learns the API through tools - #412
Open
JackHopkins wants to merge 16 commits into
Open
feat: tool_discovery solver — agent learns the API through tools#412JackHopkins wants to merge 16 commits into
JackHopkins wants to merge 16 commits into
Conversation
SimpleServerPool assumed every run_idx in range had a container behind it. With fewer real servers than epochs, most rollouts crashed inside make_factorio_env and Inspect recorded them as errored samples while the eval completed — e.g. pass_at_8 scored over 1 real rollout out of 8. - Probe server reachability at pool init and only allocate real servers; raise if none are reachable. Rollouts now wait for a free server (30min timeout), so 8 epochs on 1 container run sequentially instead of erroring. - fle inspect-eval passes --fail-on-error by default so any errored sample fails the eval; --no-fail-on-error restores the old behavior. Verified: production_science_pack_throughput with epochs=8 on a single container now completes 8/8 samples (was 1/8).
Every existing solver injects the full SystemPromptGenerator manual into the system prompt and ablates observations/images. tool_discovery inverts the ablation: a minimal system prompt plus Inspect-native tools (list_methods, manual, observe, run_code) sourced from the same per-tool agent.md files the MCP server serves, so acquiring the rules is part of the measured behavior. Only run_code counts against the trajectory_length budget; manuals and observation are free, keeping scores comparable with controlled. Server allocation, TrajectoryData scoring, and pass@k wiring are unchanged. Registered in SOLVER_MAP and the --solver CLI choices (sync test passes).
run_code returned only the formatted observation; the agent could not see what its program printed or why it failed. Surface info['result'] (the same source the controlled solver uses) ahead of the game state.
Three epochs of a production run hung indefinitely inside generate() against a stalled provider connection. 300s timeout with 3 retries turns a hung request into a bounded delay.
Trimming could cut between an assistant tool_call and its tool result; OpenAI rejects such histories (No tool call found for call_id). Drop leading orphaned tool results from the kept tail.
A single transient get_entities failure (engine busy, RCON hiccup) killed an entire multi-hour eval: one rollout errored ~1h in and fail-on-error cancelled the other three healthy epochs. The action sequence replayed clean, confirming the failure was transient. Retry up to 3 times with short backoff, and include the underlying error in the raised message — the previous wrapper discarded it, making the failure undiagnosable from the eval log.
run_code streamed the full formatted game state back after every step — the controlled-solver habit leaking into the tool harness. Now run_code returns only the program's STDOUT/STDERR plus a step counter, and the agent pulls state deliberately through dedicated free tools: - task(): objective, quota, current score, remaining budget - entities(): entities on the map (compact reprs) - inventory(): carried items - summary(): one-glance position/score/budget/counts overview observe() is superseded. Documentation tools unchanged.
The RCON client rejects concurrent calls (ClientBusy: 'The client is already busy with another call'), and Inspect may execute multiple tool calls from a single assistant message in parallel — entities()/ inventory() racing a run_code() step corrupted responses and killed epochs. All env-touching tools now share an asyncio.Lock.
score() after a step could fail transiently (e.g. KeyError 'player' from a corrupted RCON response) and kill the epoch. Apply the same retry-with-backoff used for get_entities, via a shared _call_with_retry helper used by both sites.
The per-call sliding window (head + last-40) shifted the prompt prefix on every generation once history exceeded the window, defeating provider prompt caching: measured trajectories cache-hit only the ~784-token system head, re-billing ~25-40k input tokens on every call. Trim only when crossing a high-water mark (120), cutting to a low-water mark (60), so the prefix stays stable for ~60 generations between trims. Trimming mutates the shared list so the stable prefix persists across calls.
trim_messages re-trimmed every 1-2 steps with the configured windows (e.g. 12/8, 25/16), shifting the prompt prefix on nearly every generation and defeating provider prompt caching, same pathology as the tool_discovery sliding window. Trigger trimming only above max(max_messages, 2*trim_to + 1); the kept context after a trim is unchanged so each variant's context ablation semantics are preserved — the context ceiling between trims grows slightly in exchange for a prefix that stays cache-stable for many steps.
Unpinned OpenRouter routing bounces requests across providers, splitting or missing their prompt caches. Resolve the model's top tool-capable provider from the endpoints API at CLI startup and pin routing to it (order + allow_fallbacks=false). --pin-provider overrides the choice; --no-pin-provider disables pinning. Best-effort: any resolution failure falls back to unpinned routing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A new Inspect solver variant,
tool_discovery, that inverts the standard prompt design: instead of injecting the full SystemPromptGenerator manual into the system prompt, the agent gets a minimal prompt plus Inspect-native tools to discover the rules itself:list_methods()— enumerate the environment APImanual(method)— full documentation for one method (same per-tool agent.md files the MCP server serves)observe()— current inventory/position/entities (free)run_code(code)— execute Python in the environment (counts against the step budget)Only
run_codeconsumes trajectory_length; manuals and observation are free, so scores stay comparable withcontrolled— the cost of acquiring the rules shows up as tokens/latency, not lost game actions. A 3×-budget generation cap halts models that read forever without acting.Server allocation (SimpleServerPool), TrajectoryData scoring, and pass@k wiring are unchanged. Registered in SOLVER_MAP, the
--solverCLI choices, and gains anopen_play_tool_discoverytask via the existing variant machinery. The solver-choices sync test passes.Why
None of the 12 existing solver variants ablate the manual — they ablate observations/images/HUD. The MCP server implements rules-discovery interactively but has no eval harness; PR #158 tried a RAG variant years ago (unmergeable). This makes "can the agent learn the environment?" a measurable question.
Smoke test (live server, OpenRouter, gpt-4o-mini, 8-step budget)
iron_ore_throughput: score 21.0/16 → scorer 1.0. Trajectory shows the intended behavior: list_methods ×1, manual ×7, observe ×1, run_code ×8 (exact budget), clean server release. 2m06s, 58k tokens.
Known gaps (follow-ups)