feat(bench): Harbor TB2.1 plan + TkHarborAgent stub - #28
Open
undivisible wants to merge 3 commits into
Open
undivisible wants to merge 3 commits into
undivisible wants to merge 3 commits into
Conversation
Land bench/harbor scaffold (PLAN, README, installed-agent stub) for local tk/rx4 Terminal-Bench 2.1 comparison. Docker not required for this PR; Phase A/C smoke gated separately. Does not touch tk-cloud.
|
Capy couldn't review this pull request because Max's workspace is out of credits, add credits or enable auto-reload to resume automatic reviews. |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_19af11bc-ca7d-4d83-b336-ff5e57ab63f6) |
Max: no OpenAI API spend; Codex via ChatGPT OAuth; Amp in comparison table.
Per Max/CoS: Harbor Codex via CLI OAuth or custom codex-exec wrapper.
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
bench/harbor/PLAN.md— Harbor × Terminal-Bench 2.1 design (phases A–E, TkHarborAgent contract, model lock)bench/harbor/agents/tk_harbor_agent.py— installed-agent stub (name/install/run/populate_context_post_run) without importing harbor at load timebench/harbor/README.md— run commands + Docker gate (smoke blocked until Docker; this PR is plan+stub only)Notes
tk/rx4) only — does not touch tk-cloud / CF Computer / orbsTest plan
bench/harbor/Note
Low Risk
Documentation and a non-functional stub only; no production paths, auth, or tk-cloud changes.
Overview
Introduces a new
bench/harbor/scaffold for comparing Terminal-Bench 2.1 pass@1 across Codex, OpenCode/omp, and headlesstk execvia Harbor—explicitly as a local CLI harness, not tk-cloud/orbs.PLAN.mdadds the phased roadmap (A–E),TkHarborAgentinstall/run contract (BaseInstalledAgent, model lock, logging to/logs/agent/tk.txt), exampleharbor runcommands, and current blockers (no Docker on box; fixed model TBD).README.mdmirrors layout, prerequisites, Docker gate, and stub status.agents/tk_harbor_agent.pyis a Phase C placeholder:name,install,run, andpopulate_context_post_runraiseNotImplementedErrorand avoid importing harbor at module load so CI stays green without the package. No runtime bench behavior lands in this PR—smoke waits on Docker and a follow-up to flesh out the adapter.Reviewed by Cursor Bugbot for commit 11b85be. Configure here.