Skip to content

feat(agent-cli): add ori code harness for terminal-bench - #76

Open
devin-ai-integration[bot] wants to merge 1 commit into
mainfrom
devin/1788844974-ori-code-harness
Open

feat(agent-cli): add ori code harness for terminal-bench#76
devin-ai-integration[bot] wants to merge 1 commit into
mainfrom
devin/1788844974-ori-code-harness

Conversation

@devin-ai-integration

Copy link
Copy Markdown
Contributor

TL;DR

Adds code to ORI_AGENTS so terminal-bench can drive Ori's native coding agent (ori code) instead of only the wrapped third-party agents (pi, claude, prime-agent, omp).

What changed?

  • ORI_CODE_HARNESS in src/benchmarks/agent-cli/harness.ts:
    • run script: ori code --model "$TB_MODEL" --reasoning-effort <effort> --approvals self-drive --output jsonl --prompt-file /instruction.md, stdin from /dev/null, reasoning.delta/tool.output.delta events filtered before tee to /logs/agent/ori-code.txt
    • image steps install only curl/ca-certificates/git; the bootstrap installs the ori CLI (oriChannel controls the version) and checks ori --version once. agentPackage other than the "ori" sentinel throws, since ori code ships inside the CLI.
    • parseOriCodeStream reads the {kind:"event"|"result"} envelopes: assistant text from assistant.text.delta, tool calls from tool.started, usage/cost/generationIds from the turn.succeeded payload, errors from turn.failed/session.failed/runtime.error and a non-ok result line. inputTokens is reported as-is (ori's inputTokens already includes cached tokens).
    • systemPrompt/appendSystemPrompt/allowedTools/disallowedTools exit 2 before launch — ori code has no flags for them. isolateAgentConfig is a no-op (the sandbox has no user config).
  • ORI_AGENTS gains "code"; fake sandbox helper recognises ori code as the agent command.

Why?

We want terminal-bench numbers for the actual ori code path (requested by Louis in #agents-ori), and no existing harness invokes it.

How to test

Event shape was captured from a real headless run against meta/muse-spark-1.3 (bun framework/cli/bin/ori.ts code --model meta/muse-spark-1.3 --reasoning-effort medium --approvals self-drive --output jsonl --prompt-file instruction.md </dev/null), exit 0, 21 NDJSON lines, turn.succeeded carrying usage.{inputTokens,outputTokens,costUsd,generationIds,model} and a final {"kind":"result","ok":true,"sessionId":...} line; the test fixture mirrors it.

Live smoke:

bun run bench -- --benchmark terminal_bench --model meta/muse-spark-1.3 --reasoning-effort medium --limit 1 --solver-config '{"agent":"code"}'

Expect the agent log at /logs/agent/ori-code.txt in the sandbox to start with {"event":{"audit":... and metadata agent: "code", agentTurns: 1, non-empty generationIds.

Benchmark impact

New agent option only; existing agents' scripts and parsers are untouched.

Reviewer focus

  • ori code writes .ori/ under the task cwd (/app); tasks whose verifier inspects the directory listing could be affected. Same class of risk as other agents' scratch files, flagging in case we want a different cwd.
  • Headless ori code exits non-zero with interaction_pending if the model asks a question; we parse that as an error. ori#2557 removes that path upstream.

Checklist

  • Tests cover changed behavior
  • Public API or configuration changes are backward compatible, or the break is documented
  • Benchmark changes document dataset provenance and licensing (n/a)
  • No credentials, private results, or restricted dataset contents are included
  • Documentation is updated where needed (no README agent list exists)

Link to Devin session: https://openrouter.devinenterprise.com/sessions/c8c55d9cfa2e43569688382fc7deb436
Open in Devin Desktop: https://openrouter.devinenterprise.com/desktop/session/c8c55d9cfa2e43569688382fc7deb436?variant=devin
Requested by: @louisgv

Co-Authored-By: Louis Vichy <louis@openrouter.ai>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR that start with 'DevinAI' or '@devin'.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

Original prompt from Louis

SYSTEM:
<latest_message>
Louis Vichy (U0ACWETHNLT) [ts=1788844310.775339]: @Devin with ori cli against terminal bench on it, using the latest muse model -- i.e make sure we're running ori code!
</latest_message>

=== BEGIN THREAD HISTORY (in #agents-ori) ===
Louis Vichy (U0ACWETHNLT) [ts=1788844310.775339]: @Devin with ori cli against terminal bench on it, using the latest muse model -- i.e make sure we're running ori code!
=== END THREAD HISTORY ===
Channel ID: C0BS37QUK7T
Thread URL: https://openrouter.slack.com/archives/C0BS37QUK7T/p1788844310775339?thread_ts=1788844310.775339&amp;cid=C0BS37QUK7T

The <latest_message> is the message that you should use to guide your goals + task for this session, and you should use the rest of the slack thread as context.
A [ts=...] marker on a Slack message is that message's timestamp. To act on a specific message with the slack tool (e.g. adding an emoji reaction via the reaction command), pass that value as timestamp along with the Channel ID — no extra lookup call is needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant