fix(eval): pass the deep-swe prompt after -- so a leading dash is not a flag - #134
Merged
Merged
Conversation
…ot a flag The four deep-swe adapters passed the task instruction to each agent CLI as a bare positional argument. When the instruction text begins with `- `, the argument parser treats it as a flag, the CLI exits with a usage error, and the agent never runs. The trial is scored zero and marked invalid on every arm, so the task is silently voided rather than failed. This is not specific to one task: all four adapters built the command the same way, with no separator between the flags and the prompt. Any future instruction that happens to open with a dash would be voided the same way, on every arm, on every run. In one 20-task run it cost a whole task across all four arms, leaving 19 valid tasks instead of 20. Insert `--` before the prompt in all four `run_command` implementations, and record why it is there so it is not later removed as noise. Verified per CLI rather than assumed, since a separator the parser does not honour would look like a fix and still void the task: - `amplifier-agent run` and `amplifier run` are stock click with a plain positional argument, no `ignore_unknown_options` and no argv pre-parsing. With `--` the prompt parses intact; without it click reports `No such option '- '`. - `amplifier-opencode launch` declares `nargs=-1, type=click.UNPROCESSED` and execs via `os.execvp` with an argument list, no shell. Click consumes only the first `--`, so the second survives verbatim into the opencode argv. - `opencode run` is yargs with `populate--: true`, which routes anything after `--` to `argv["--"]` rather than the positional. Its run command merges `argv["--"]` back into the message, so the prompt still arrives. Confirmed against the binary: with `--` the message reaches the server, without it opencode prints its usage banner and drops the prompt. Behaviour for prompts that do not begin with a dash is unchanged on all four. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The four deep-swe adapters pass the task instruction to each agent CLI as a bare positional argument. When the instruction text begins with
-, the argument parser treats it as a flag, the CLI exits with a usage error, and the agent never runs. The trial is scored zero and marked invalid on every arm, so the task is silently voided rather than failed.This is not specific to one task. All four adapters built the command the same way, with no separator between the flags and the prompt:
The shell quoting was already correct, so each CLI received exactly one argument. The problem was only that the argument started with
-.opencode_amplifier.pyalready contained a--, but it separateslaunchfromrunrather than the flags from the prompt, so it did not help.Any future instruction that happens to open with a dash would be voided the same way, on every arm, on every run. In one 20-task run this cost a whole task across all four arms, leaving 19 valid tasks instead of 20.
Change
Insert
--before the prompt in all fourrun_commandimplementations, and record why it is there so it is not later removed as noise.Verification
Checked per CLI rather than assumed, since a separator the parser does not honour would look like a fix and still void the task.
amplifier-agent runandamplifier runare stock click with a plain positional argument, noignore_unknown_optionsand no argv pre-parsing. With--the prompt parses intact; without it click reportsNo such option '- '.amplifier-opencode launchdeclaresnargs=-1, type=click.UNPROCESSEDand execs viaos.execvpwith an argument list, no shell. Click consumes only the first--, so the second survives verbatim into the opencode argv.opencode runis yargs withpopulate--: true, which routes anything after--toargv["--"]rather than the positional. Its run command mergesargv["--"]back into the message, so the prompt still arrives. Confirmed against the binary: with--the message reaches the server; without it opencode prints its usage banner and drops the prompt.Behaviour for prompts that do not begin with a dash is unchanged on all four.
Lint, format and type checks pass on the touched package.
Not done here
The affected task has not been re-run on the four arms. Confirmation that this holds end to end is that the task produces a real agent run rather than a usage error, and the run reports 20 valid tasks.
Separate pre-existing defect, not addressed here
opencode's run command wraps any argument containing a space in literal double quotes before sending it as the prompt. Both opencode arms have therefore been sending the model"<instruction>"with literal quote characters on every task, not only dash-leading ones. This change neither causes nor worsens that: the behaviour is identical with and without--. Fixing it means either piping the prompt on stdin or patching opencode upstream, and both shift opencode-arm results, so it belongs in its own change.