feat(omp): add Oh-My-Pi as a native benchmark agent - #3
Open
arielnabavian wants to merge 1 commit into
Open
arielnabavian wants to merge 1 commit into
arielnabavian wants to merge 1 commit into
Conversation
Adds Omp (https://github.com/can1357/oh-my-pi), a TypeScript/Rust coding agent — fork of Mario Zechner's Pi with LSP, DAP, and Rust-core internals. Pairs with the Harbor adapter added in notowen333/harbor#1 (and the matching upstream PR harbor-framework/harbor#2715). Changes mirror the Codex PR (notowen333#2) in shape: * `run-benchmark.sh`: - New `omp)` agent case; `OMP_VERSION` defaults to 17.2.15 (current npm `@oh-my-pi/pi-coding-agent@latest` at PR time). Pinning matches the pattern for CLAUDE_CODE_VERSION and OPENCODE_VERSION. - New `omp)` model-routing case: `amazon-bedrock/<id>` for the common case (Anthropic + Kimi + other Bedrock-native models — auth via the fleet's IAM instance profile through IMDS, no bearer token gymnastics since Omp's Bedrock provider does its own SigV4 signing), or `openai/<id>` for Mantle GPT. - New `configure_omp_model` helper that only fires for Mantle GPT. * `run.py`: - Omp added to `_native_agents`. - New env-forwarding branch: for `openai/` model routing forwards `OPENAI_API_KEY` and `OPENAI_BASE_URL` (from `OMP_OPENAI_BASE_URL`); for `amazon-bedrock/` routing the fleet's IAM instance profile covers auth. * `run-matrix.sh`: - Omp recognized as native for the `HAS_NATIVE` flag. - `OMP_VERSION` gets baked into the matrix state id. * Docs (README.md, ORCHESTRATOR.md): agent tables include `omp`, `OMP_VERSION` documented, native-competitor section describes the Bedrock + Mantle wiring. Verified locally: * `bash -n` clean on both shell scripts; run.py parses under Python 3.12. * `BENCHMARK_DRY_RUN=1` on Anthropic Bedrock produces `codex@17.2.15--opus-4.8--*` (job name) and `amazon-bedrock/global.anthropic.claude-opus-4-8` (Harbor model). * `BENCHMARK_DRY_RUN=1` on Mantle GPT produces `openai/openai.gpt-5.6-sol` and the correct Mantle base URL. * `HARBOR_DRY_RUN=1` inspection confirms the `harbor run` argv includes `-a omp -m <resolved> --ak version=17.2.15` and — for Mantle GPT — the correct `--ae OPENAI_API_KEY=${OPENAI_API_KEY}` templates. Depends on notowen333/harbor#1 (Harbor adapter for omp).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Omp (
can1357/oh-my-pi) — a TypeScript/Rust coding agent, fork of Mario Zechner's Pi with LSP, DAP, and native Bedrock providers — as a fourth native agent alongside Stan/Claude Code/OpenCode. Motivation: broader native-agent set to compare Stan against.Depends on notowen333/harbor#1 (the Harbor
ompadapter). Matching upstream PR: harbor-framework/harbor#2715.Approach
Since Omp shares Pi's CLI shape and has its own SigV4-signing Bedrock provider that reads AWS credentials via IMDS, the wrapper diff is small and mirrors the Codex PR:
amazon-bedrock/*routing — fleet's IAM instance profile handles auth. No bearer-token forwarding needed.openai/*routing (Mantle GPT) — reuses the samebedrock_api_keysecret OpenCode and TAU3 already consume; exports asOPENAI_API_KEY+OMP_OPENAI_BASE_URL.Changes
run-benchmark.sh— newomp)agent case + model-routing case +configure_omp_modelhelper.OMP_VERSIONdefaults to17.2.15(current npm@oh-my-pi/pi-coding-agent@latest).run.py— Omp added to_native_agents; env-forwarding branch handles both Bedrock and Mantle GPT paths.run-matrix.sh— Omp recognized as native;OMP_VERSIONbaked into the matrix state id.README.md,ORCHESTRATOR.md) — agent tables listomp, env-var table documentsOMP_VERSION, native-competitor section describes the Bedrock + Mantle wiring.Verified
bash -nclean on both shell scripts;run.pyparses under Python 3.12.BENCHMARK_DRY_RUN=1 bash strands-infra-runner/run-benchmark.sh omp opus-4.8 swe-bench/swe-bench-verified 4→ job nameomp@17.2.15--opus-4.8--*, Harbor modelamazon-bedrock/global.anthropic.claude-opus-4-8.BENCHMARK_DRY_RUN=1 bash strands-infra-runner/run-benchmark.sh omp openai.gpt-5.6-sol strands-harness-benchmark-index→ job nameomp@17.2.15--openai.gpt-5.6-sol--*, Mantle URLhttps://bedrock-mantle.us-east-1.api.aws/openai/v1.HARBOR_DRY_RUN=1inspection ofrun.pyconfirmsharbor runargv includes-a omp -m <resolved> --ak version=17.2.15and, for Mantle GPT, the correct--ae OPENAI_API_KEY=${OPENAI_API_KEY}templates.Usage
Note
Doesn't touch
run-full-native-suite.sh's defaultAGENTS="claude-code,opencode"— same reason as the Codex PR: adding Omp there would kick off unplanned baseline runs; separate opt-in when the team wants to expand the default sweep.