Add leaderboard entry: Trinity / DeepSeek V4 Pro (78.6% accuracy)

[ajayr4j]

Summary

• Trinity (11-skill Claude Code skill pack) + DeepSeek V4 Pro, full 140-trial run across all 14 Enterprise-Bench l1-l2-bench tasks (10 trials/task).
• Overall accuracy: 78.57% (110/140 successful trials).
• Source job (public): https://hub.harborframework.com/jobs/02b0a6e5-ce65-4a5e-9494-feab0f5f5ed4

Known issues disclosed

1. eng-l1-c__MjQfY3J: litellm.BadRequestError — DeepSeek V4 Pro thinking-mode/reasoning_content litellm gateway compatibility bug after 26 real conversation turns. Not an infra failure.
2. eng-l2-b__GjkKAcC: litellm.BadRequestError 402 "Insufficient Balance" — transient upstream billing event, confirmed resolved mid-run (balance topped up, rest of run completed normally). Not a model/task failure.

Both errors score 0 in the accuracy computation (no exclusion).

Cost reporting note

This run routed through litellm to a non-Anthropic model (DeepSeek V4 Pro, via High-Flyer), and Claude Code's stream-json output did not populate trustworthy usage/cost stats — the same instrumentation gap already disclosed in PR #11 (Claude Code / DeepSeek V4 Pro bare harness) and PR #9 (GLM-5.2). An independent tiktoken cl100k reconstruction from the raw transcripts counts ~4.33M input tokens vs. the ~42.4M self-reported by litellm here (~10x discrepancy), so the self-reported cost is not trustworthy. Following the PR #11 precedent, total_cost_usd: 0.0 / display_total_cost_usd: "-". Token counts (uncached_input/output/total) are left as the raw litellm-reported figures, consistent with what's shown on the public Hub job page, with this caveat disclosed here for reviewers.

Test plan

• YAML validated against leaderboard/leaderboard.yaml's metrics_schema (all 16 required fields present, no unknown keys)
• trial_ids count (140) matches n_trials
• No duplicate trial IDs within this file or against other merged leaderboard entries
• Source Harbor Hub job is public and trial trajectories are viewable while logged out

[ajayr4j]

cc @jonmaaku-rev

[jonmaaku-rev]

@ajayr4j

We just added a PR checklist for every Leaderboard entry, so far so good for this job, I would have spend some time to check random traces manually.

PR Checklist — Leaderboard Row Entry

Verified against local traces 20260803__095254_trinity-deepseek-v4-pro (140 trial dirs) and 2026-08-03__trinity__deepseek-v4-pro.yaml.

Provenance & access

• Source Harbor job is PUBLIC (openable + trajectories viewable while logged out)
• Submitter owns the job / job was uploaded via harbor job upload
• Row entry YAML links to the public job URL (hub.harborframework.com/jobs/)
• Job ID in the header comment matches the public job

Row file conforms to the template

• Copied from row-template.yaml; no unknown keys (schema is additionalProperties: false)
• Filename follows convention: .yaml
• All required metrics present: accuracy, display_accuracy, token breakdown (uncached_input, cached_input, output, total), avg_trial_duration_sec, pass_at_2/3/4/5/8/10, n_trials
• metadata block complete: agent_display_name, model_display_name, agent_org_display_name, model_org_display_name
• n_trials == number of trial_ids listed (140 == 140)
• trial_ids are unique — no duplicates within the file or across existing rows

Metrics integrity

• total_tokens reconciles with uncached_input + cached_input + output (43,123,643 = 42,386,558 + 0 + 737,085)
• display_accuracy matches accuracy — verified: traces give 110/140 = 78.57%, matching the YAML; pass@k reproduces to 4 decimals (pass@2 0.8937, pass@3 0.9375, pass@4 0.9622, pass@5 0.9779, pass@8 0.9984, pass@10 1.0)
• If cost fields are included, display_total_cost_usd matches total_cost_usd (both suppressed: "-" / 0.0, disclosed)
• status is display (template default)

Public job contents (each trial)

• trajectory.json per trial (+ reward/result file per trial)
• Token usage present per trial — figures are litellm-reported (~42.4M input); offline audit counts ~4.33M (~10x lower). Disclosed; reconcile which figure is published for cross-row comparability.
• Provider / model / agent surfaced on the job

Run conditions (comparability)

• Timeout multipliers that affect scored work at default (== 1.0) — agent_timeout_multiplier is 2.5 on every trial (must be 1.0). Global timeout_multiplier is 1.0 (OK). Note: 0 trials actually exceeded the 600s base agent timeout, so no result depended on the raised multiplier.
• Setup/build timeouts only adjusted if needed (agent_setup / environment_build) — not modified
• No CPU / memory / storage overrides
• Agent did not access benchmark site/repo or answers during the run (ideal_response/reward/verifier = 0 hits in transcripts; answer delivered only via /submit_agent_response)
• Dataset + version pinned (enterprise-bench/l1-l2-bench) and harness recorded

Review

• Reviewer can independently re-verify: trajectories replay and reward matches the submitted score (110/140 = 78.57% reproduced from traces)
• Confirm reviewers agree with no-exclusion of the 2 disclosed error trials (eng-l1-c__MjQfY3J, eng-l2-b__GjkKAcC), which score 0.

[ajayr4j]

Hi @jonmaaku-rev, thanks for the checklist, really useful format. Here's the info for the two open items, pulled directly from our local run directory and cross-checked against the live job.

Live job, for reference (harbor hub job show 02b0a6e5-ce65-4a5e-9494-feab0f5f5ed4): 140 trials, 2 errors, reward 0.786, input tokens 42,386,558, output tokens 737,085, cost $0.00, matching the submitted YAML exactly.

1. Token usage per trial (litellm ~42.4M vs audit ~4.33M)

Same instrumentation gap already disclosed in the PR body and in AGGREGATE_SUMMARY.json/.md in our run directory: this job routed through litellm to a non-Anthropic model (DeepSeek V4 Pro), and Claude Code's stream-json output did not populate real usage stats, so litellm's self-reported token counts are inflated. We ran an independent tiktoken cl100k reconstruction directly from the trial transcripts as a cross-check: 4,330,479 input / 496,409 output, versus litellm's self-reported 42,386,558 input / 737,085 output, a roughly 10x gap. Same gap flagged in PR #11 (Claude Code / DeepSeek V4 Pro) and PR #9 (GLM-5.2), not specific to this submission.

Honestly not sure which number should be treated as authoritative here. We already suppress total_cost_usd to 0.0/"-" because the litellm figure is unreliable for this route, so it feels inconsistent to publish the litellm token counts as-is while distrusting the litellm cost counts from the same instrumentation gap, but I don't want to unilaterally swap the published uncached_input_tokens/output_tokens/total_tokens in the entry YAML to the audit numbers without a steer, since that's a bigger change than a comment reply and will keep coming up for every non-Anthropic-model submission that goes through litellm.

Could you (or anyone else who's hit this) advise which figure the leaderboard wants standardized for these cases, litellm-reported, audit-reconstructed, or something else entirely? And separately, is there a better-supported way to get accurate token/cost accounting for non-Anthropic models through litellm that we should be using instead of this manual audit workaround? Genuinely asking for guidance here, want to get this right rather than guess.

2. agent_timeout_multiplier: 2.5

Confirmed present in our config.json, same issue as flagged on PR #15. Checked actual trial durations from started_at/finished_at across all 140 trials in the local run directory: longest trial ran 445s, well under the 600s base agent timeout, and no trial in the set exceeded it. Same situation you noted for the GLM-5.2 run, no result depended on the raised multiplier.

The only 2 non-reward-1.0 outcomes in this run are the 2 disclosed error trials (eng-l1-c__MjQfY3J, eng-l2-b__GjkKAcC), and both are confirmed API errors unrelated to timeout: a litellm/DeepSeek thinking-mode compatibility bug ("reasoning_content must be passed back to the API") and a transient upstream "Insufficient Balance" billing error. Neither is a timeout.

Happy to do a clean rerun at the compliant setting if you'd like one for this submission too, even though the data here shows it didn't affect the result.

Let me know if you want the clean rerun for this one too, or if the "didn't affect any result" evidence is enough to leave this PR as is.

[ajayr4j]

One more note on the token question above: our GLM-5.2 submission (PR #15) actually already published the audit-reconstructed numbers rather than the raw gateway ones, for a related reason. Harbor Hub's live trials table shows $0.00/0 tokens for ~19 of the 140 GLM-5.2 trials because the litellm gateway only returns real usage on chatcmpl--shaped responses, not the more common streamed msg_-shaped ones. So that entry's uncached_input_tokens/output_tokens are already the cl100k-audited figures, not the gateway's.

[ajayr4j]

cc @roycabezas @mira-mohammad , you're both listed as requested reviewers here, would appreciate a look when you get a chance. I've addressed @jonmaaku-rev's checklist items above (token accounting question + timeout multiplier confirmation).

cc @nimit2801

[ajayr4j]

Hi @jonmaaku-rev, following up here, any rough ETA on the token-accounting question above? No rush, just want to plan around it. Thanks!