Leaderboard entry: Opencode / DeepSeek V4 Pro

[ajayr4j]

Summary

• Adds leaderboard/entries/2026-08-11__opencode__deepseek-v4-pro.yaml: a 140-trial public Harbor run of Opencode (agent org: Anomaly) driving DeepSeek V4 Pro (model org: High-Flyer) against Enterprise-Bench/l1-l2-bench.
• Public Hub job: https://hub.harborframework.com/jobs/336fe762-f88c-420f-b735-1b52ba817b96 (verified viewable and trajectories accessible while logged out).

Run details

• Agent: Opencode v1.18.16
• Model: DeepSeek V4 Pro (High-Flyer), via a LiteLLM proxy
• Harbor: v0.20.0
• Dataset: Enterprise-Bench/l1-l2-bench, ref sha256:834e40f54800287c913cebc9fa8e8273292940436168a56c1c5c72ff8a687e8d (unchanged since dataset.toml's last update, 2026-07-27)
• Command shape: harbor run with n_attempts=10, n_concurrent_trials=5, agent_timeout_multiplier=1.0
• 140/140 planned trials completed; 9 exceptions (8 AgentSetupTimeoutError, 1 AgentTimeoutError) counted as reward 0 in the denominator, consistent with how prior entries with errored trials (e.g. 2026-07-04__computer__opus-4-6.yaml) are scored.

Metrics

• accuracy: 40.71% (57/140)
• pass@2/3/4/5/8/10: 0.5667 / 0.6446 / 0.6939 / 0.7307 / 0.8127 / 0.8571
• avg_trial_duration_sec: 339.43 (derived from minute-granularity Hub timestamps — coarser precision than entries using full datetime timestamps; flagging for reviewer awareness)
• Cost suppressed (total_cost_usd: 0.0, display_total_cost_usd: "-"): the LiteLLM-proxy gateway path used for this run does not report per-trial cost, confirmed both in the Hub trial export and in local result.json (cost_usd: null). Same suppression pattern as 2026-07-28__computer__glm-5-2.yaml.

Known infra artifact (disclosed)

OpenCode has a hardcoded-provider bug that requires registering the model under a custom local provider alias in its config rather than the model's real upstream provider name. The run was executed with this alias correctly labeled high-flyer (not a placeholder), so agent_info.model_info.provider on Hub reads High-Flyer accurately. No result, reward, token, or trajectory data was altered — only the recorded provider label, which is a local client-side routing name, not part of the graded outcome.

Validation

• Entry copied from leaderboard/row-template.yaml with every field completed.
• All 140 trial_ids are unique and verified with zero overlap against the 1,120 trial IDs already used across the 8 existing leaderboard/entries/*.yaml files.
• n_trials (140) equals len(trial_ids).
• make validate passes (pre-existing local pyproject.toml/uv.lock diffs from an unrelated environment change are not part of this PR).
• Job and trajectories confirmed publicly accessible while logged out.

Notes for reviewers

• PR feat: Add CI check and PR template for leaderboard submissions. #16 (CI leaderboard validator) was still open/unmerged at submission time, so make validate-leaderboard wasn't run — happy to re-run once it lands.

[nimit2801]

Thanks for the leaderboard submission. First-touch impression: this is a single leaderboard entry for Opencode v1.18.16 + DeepSeek V4 Pro, sourced from Harbor job https://hub.harborframework.com/jobs/336fe762-f88c-420f-b735-1b52ba817b96.

Checks: GitHub currently reports no checks on this branch, so I will reassess once validation/CI is available.

Score/accessibility: reported accuracy is 40.71% (57/140), with pass@10 0.8571; the PR says the job and trajectories were verified publicly accessible while logged out.

Risk: moderate, mostly around Hub export provenance, cost suppression, exception handling in the denominator, and validator coverage not yet reflected in CI.

Maintainer verification points:

1. Confirm the public Hub job and trajectories are accessible logged out.
2. Spot-check the score, pass@ values, and 140 unique trial IDs against the Hub export.
3. Re-run make validate / leaderboard validation once the validator workflow is available.