Add leaderboard entry: Trinity / GLM 5.2 - #13
Closed
ajayr4j wants to merge 1 commit into
Closed
Conversation
Trinity: a custom Claude Code skill harness (11 atomic skills targeting known silent-failure modes in the CRM/PM/file-server MCP tools) routed to GLM-5.2 via litellm. Full Enterprise-Bench/l1-l2-bench run: 14 tasks x 10 attempts = 140 trials, 0 errors, 89.29% accuracy. Token/cost telemetry: this run routed through litellm to a non-Anthropic model, and Claude Code's stream-json output does not populate real usage stats for this route (same gap noted in devrev#9/devrev#11). Instead of using those near-zero self-reported numbers, token counts were independently reconstructed from each trial's own claude-code.txt transcript (every system-prompt/tool/skill payload, user turn, and tool_result fed to the model as input; every assistant text block and tool_use payload as output), tokenized with tiktoken cl100k_base. Reconstruction script: enterprise-bench-eval/scripts/audit_tokens.py in the source repo. Cost is computed from those reconstructed input/output tokens at GLM-5.2 list pricing ($1.40 / $4.40 per 1M). Source job: https://hub.harborframework.com/jobs/32e17142-e309-4265-b359-50804adc6b06
ajayr4j
requested review from
jonmaaku-rev,
mira-mohammad and
roycabezas
as code owners
August 1, 2026 10:01
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Leaderboard entry for Trinity / GLM 5.2 — a custom Claude Code skill
harness (11 atomic skills targeting known silent-failure modes in the
CRM/PM/file-server MCP tools: field-value verification, leaf-level
grouping, filter-before-rank, complete join chains, exact scope
boundaries, exhaustive entity coverage, named-record resolution, file
search broadening, pagination, and real submission calls) routed to
GLM-5.2 via a litellm gateway.
Full
Enterprise-Bench/l1-l2-benchrun: 14 tasks x 10 attempts = 140trials, 0 errors, 89.29% accuracy.
Token/cost telemetry: this run routed through litellm to a
non-Anthropic model, and Claude Code's stream-json output does not
populate real usage stats for this route (the same instrumentation gap
noted in #9 / #11). Rather than use those near-zero self-reported
numbers, token counts were independently reconstructed from each
trial's own
claude-code.txttranscript — every system-prompt/tool/skill payload, user turn, and tool_result fed to the model as input;
every assistant text block and tool_use payload as output — tokenized
with tiktoken
cl100k_base. Cost is computed from those reconstructedinput/output tokens at GLM-5.2 list pricing ($1.40 / $4.40 per 1M
tokens).
Source job: https://hub.harborframework.com/jobs/32e17142-e309-4265-b359-50804adc6b06
Type of contribution
Validation
Enterprise-Bench/l1-l2-benchpackageleaderboard/row-template.yaml