feat(agent-cli): add ori code harness for terminal-bench - #76
feat(agent-cli): add ori code harness for terminal-bench#76devin-ai-integration[bot] wants to merge 1 commit into
Conversation
Co-Authored-By: Louis Vichy <louis@openrouter.ai>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
Original prompt from Louis
|
TL;DR
Adds
codetoORI_AGENTSso terminal-bench can drive Ori's native coding agent (ori code) instead of only the wrapped third-party agents (pi,claude,prime-agent,omp).What changed?
ORI_CODE_HARNESSinsrc/benchmarks/agent-cli/harness.ts:ori code --model "$TB_MODEL" --reasoning-effort <effort> --approvals self-drive --output jsonl --prompt-file /instruction.md, stdin from/dev/null,reasoning.delta/tool.output.deltaevents filtered beforeteeto/logs/agent/ori-code.txtoriChannelcontrols the version) and checksori --versiononce.agentPackageother than the"ori"sentinel throws, sinceori codeships inside the CLI.parseOriCodeStreamreads the{kind:"event"|"result"}envelopes: assistant text fromassistant.text.delta, tool calls fromtool.started, usage/cost/generationIdsfrom theturn.succeededpayload, errors fromturn.failed/session.failed/runtime.errorand a non-okresult line.inputTokensis reported as-is (ori'sinputTokensalready includes cached tokens).systemPrompt/appendSystemPrompt/allowedTools/disallowedToolsexit 2 before launch —ori codehas no flags for them.isolateAgentConfigis a no-op (the sandbox has no user config).ORI_AGENTSgains"code"; fake sandbox helper recognisesori codeas the agent command.Why?
We want terminal-bench numbers for the actual
ori codepath (requested by Louis in #agents-ori), and no existing harness invokes it.How to test
Event shape was captured from a real headless run against
meta/muse-spark-1.3(bun framework/cli/bin/ori.ts code --model meta/muse-spark-1.3 --reasoning-effort medium --approvals self-drive --output jsonl --prompt-file instruction.md </dev/null), exit 0, 21 NDJSON lines,turn.succeededcarryingusage.{inputTokens,outputTokens,costUsd,generationIds,model}and a final{"kind":"result","ok":true,"sessionId":...}line; the test fixture mirrors it.Live smoke:
Expect the agent log at
/logs/agent/ori-code.txtin the sandbox to start with{"event":{"audit":...and metadataagent: "code",agentTurns: 1, non-emptygenerationIds.Benchmark impact
New agent option only; existing agents' scripts and parsers are untouched.
Reviewer focus
ori codewrites.ori/under the task cwd (/app); tasks whose verifier inspects the directory listing could be affected. Same class of risk as other agents' scratch files, flagging in case we want a different cwd.ori codeexits non-zero withinteraction_pendingif the model asks a question; we parse that as an error. ori#2557 removes that path upstream.Checklist
Link to Devin session: https://openrouter.devinenterprise.com/sessions/c8c55d9cfa2e43569688382fc7deb436
Open in Devin Desktop: https://openrouter.devinenterprise.com/desktop/session/c8c55d9cfa2e43569688382fc7deb436?variant=devin
Requested by: @louisgv