Skip to content

fix(eval): pass the deep-swe prompt after -- so a leading dash is not a flag - #134

Merged
David Koleczek (DavidKoleczek) merged 1 commit into
mainfrom
fix/deep-swe-dash-prompt
Aug 18, 2026
Merged

fix(eval): pass the deep-swe prompt after -- so a leading dash is not a flag#134
David Koleczek (DavidKoleczek) merged 1 commit into
mainfrom
fix/deep-swe-dash-prompt

Conversation

@DavidKoleczek

Copy link
Copy Markdown
Collaborator

Problem

The four deep-swe adapters pass the task instruction to each agent CLI as a bare positional argument. When the instruction text begins with - , the argument parser treats it as a flag, the CLI exits with a usage error, and the agent never runs. The trial is scored zero and marked invalid on every arm, so the task is silently voided rather than failed.

This is not specific to one task. All four adapters built the command the same way, with no separator between the flags and the prompt:

amplifier_agent.py        amplifier-agent run -y --config … --output json "$(cat …)"
amplifier_foundation.py   amplifier run --bundle … --output-format json "$(cat …)"
opencode_vanilla.py       opencode run --model … --auto "$(cat …)"
opencode_amplifier.py     amplifier-opencode launch -- run --auto --model … "$(cat …)"

The shell quoting was already correct, so each CLI received exactly one argument. The problem was only that the argument started with -. opencode_amplifier.py already contained a --, but it separates launch from run rather than the flags from the prompt, so it did not help.

Any future instruction that happens to open with a dash would be voided the same way, on every arm, on every run. In one 20-task run this cost a whole task across all four arms, leaving 19 valid tasks instead of 20.

Change

Insert -- before the prompt in all four run_command implementations, and record why it is there so it is not later removed as noise.

Verification

Checked per CLI rather than assumed, since a separator the parser does not honour would look like a fix and still void the task.

  • amplifier-agent run and amplifier run are stock click with a plain positional argument, no ignore_unknown_options and no argv pre-parsing. With -- the prompt parses intact; without it click reports No such option '- '.
  • amplifier-opencode launch declares nargs=-1, type=click.UNPROCESSED and execs via os.execvp with an argument list, no shell. Click consumes only the first --, so the second survives verbatim into the opencode argv.
  • opencode run is yargs with populate--: true, which routes anything after -- to argv["--"] rather than the positional. Its run command merges argv["--"] back into the message, so the prompt still arrives. Confirmed against the binary: with -- the message reaches the server; without it opencode prints its usage banner and drops the prompt.

Behaviour for prompts that do not begin with a dash is unchanged on all four.

Lint, format and type checks pass on the touched package.

Not done here

The affected task has not been re-run on the four arms. Confirmation that this holds end to end is that the task produces a real agent run rather than a usage error, and the run reports 20 valid tasks.

Separate pre-existing defect, not addressed here

opencode's run command wraps any argument containing a space in literal double quotes before sending it as the prompt. Both opencode arms have therefore been sending the model "<instruction>" with literal quote characters on every task, not only dash-leading ones. This change neither causes nor worsens that: the behaviour is identical with and without --. Fixing it means either piping the prompt on stdin or patching opencode upstream, and both shift opencode-arm results, so it belongs in its own change.

…ot a flag

The four deep-swe adapters passed the task instruction to each agent CLI as a
bare positional argument. When the instruction text begins with `- `, the
argument parser treats it as a flag, the CLI exits with a usage error, and the
agent never runs. The trial is scored zero and marked invalid on every arm, so
the task is silently voided rather than failed.

This is not specific to one task: all four adapters built the command the same
way, with no separator between the flags and the prompt. Any future instruction
that happens to open with a dash would be voided the same way, on every arm, on
every run. In one 20-task run it cost a whole task across all four arms, leaving
19 valid tasks instead of 20.

Insert `--` before the prompt in all four `run_command` implementations, and
record why it is there so it is not later removed as noise.

Verified per CLI rather than assumed, since a separator the parser does not
honour would look like a fix and still void the task:

- `amplifier-agent run` and `amplifier run` are stock click with a plain
  positional argument, no `ignore_unknown_options` and no argv pre-parsing.
  With `--` the prompt parses intact; without it click reports
  `No such option '- '`.
- `amplifier-opencode launch` declares `nargs=-1, type=click.UNPROCESSED` and
  execs via `os.execvp` with an argument list, no shell. Click consumes only
  the first `--`, so the second survives verbatim into the opencode argv.
- `opencode run` is yargs with `populate--: true`, which routes anything after
  `--` to `argv["--"]` rather than the positional. Its run command merges
  `argv["--"]` back into the message, so the prompt still arrives. Confirmed
  against the binary: with `--` the message reaches the server, without it
  opencode prints its usage banner and drops the prompt.

Behaviour for prompts that do not begin with a dash is unchanged on all four.

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@DavidKoleczek
David Koleczek (DavidKoleczek) merged commit 715e3dd into main Aug 18, 2026
3 checks passed
@DavidKoleczek
David Koleczek (DavidKoleczek) deleted the fix/deep-swe-dash-prompt branch August 18, 2026 21:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant