fix(solver): don't send OpenRouter transforms to every provider - #390
Open
phaestos2501 wants to merge 1 commit into
Open
fix(solver): don't send OpenRouter transforms to every provider#390phaestos2501 wants to merge 1 commit into
phaestos2501 wants to merge 1 commit into
Conversation
The controlled solver hardcoded 'transforms': ['middle-out'] into GenerateConfig for every generate call. transforms is an OpenRouter-specific option, and current inspect-ai validates GenerateConfig strictly, so every step of every lab-play run fails with "Unknown GenerateConfig field(s): transforms" regardless of provider — the model is never called and tasks score 0. Guard it behind an openrouter model-name check, matching the pattern already used at the other generate sites in solver.py and solver_variants.py. (Those guarded sites will still need migrating to extra_body for OpenRouter runs on current inspect-ai, per the validation error's guidance — left out of scope here since it changes OpenRouter behavior.)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
factorio_controlled_solverhardcodes'transforms': ['middle-out']into theGenerateConfigfor every generate call.transformsis an OpenRouter-specific option, and current inspect-ai validatesGenerateConfigstrictly, so every step of every lab-play run fails with:The model is never called and every task scores 0, regardless of provider.
Fix
Guard the option behind an openrouter model-name check, matching the pattern already used at the other generate sites in
solver.pyandsolver_variants.py.Note: on current inspect-ai the guarded sites will also need migrating to
extra_bodyfor OpenRouter runs (per the validation error's guidance). Left out of scope here since it changes OpenRouter-path behavior; happy to follow up.Verification
Before: an
fle inspect-evalrun (iron_plate_throughput, deepseek-v4-flash viaopenai-api/) logged the validation error at every step, zero assistant messages, score 0. After: the same run completes with the model responding normally (score 1.0).🤖 Generated with Claude Code