Skip to content

Add leaderboard entry: Trinity / GLM 5.2 - #13

Closed
ajayr4j wants to merge 1 commit into
devrev:mainfrom
ajayr4j:leaderboard/trinity-glm-5.2
Closed

Add leaderboard entry: Trinity / GLM 5.2#13
ajayr4j wants to merge 1 commit into
devrev:mainfrom
ajayr4j:leaderboard/trinity-glm-5.2

Conversation

@ajayr4j

@ajayr4j ajayr4j commented Aug 1, 2026

Copy link
Copy Markdown

Summary

Leaderboard entry for Trinity / GLM 5.2 — a custom Claude Code skill
harness (11 atomic skills targeting known silent-failure modes in the
CRM/PM/file-server MCP tools: field-value verification, leaf-level
grouping, filter-before-rank, complete join chains, exact scope
boundaries, exhaustive entity coverage, named-record resolution, file
search broadening, pagination, and real submission calls) routed to
GLM-5.2 via a litellm gateway.

Full Enterprise-Bench/l1-l2-bench run: 14 tasks x 10 attempts = 140
trials, 0 errors, 89.29% accuracy.

Token/cost telemetry: this run routed through litellm to a
non-Anthropic model, and Claude Code's stream-json output does not
populate real usage stats for this route (the same instrumentation gap
noted in #9 / #11). Rather than use those near-zero self-reported
numbers, token counts were independently reconstructed from each
trial's own claude-code.txt transcript — every system-prompt/tool/
skill payload, user turn, and tool_result fed to the model as input;
every assistant text block and tool_use payload as output — tokenized
with tiktoken cl100k_base. Cost is computed from those reconstructed
input/output tokens at GLM-5.2 list pricing ($1.40 / $4.40 per 1M
tokens).

Source job: https://hub.harborframework.com/jobs/32e17142-e309-4265-b359-50804adc6b06

Type of contribution

  • Agent result submission

Validation

  • Dataset points at the official Enterprise-Bench/l1-l2-bench package
  • Trial IDs verified unique (140/140) and public on Harbor Hub
  • All 16 required metrics present per leaderboard/row-template.yaml

Trinity: a custom Claude Code skill harness (11 atomic skills targeting
known silent-failure modes in the CRM/PM/file-server MCP tools) routed
to GLM-5.2 via litellm. Full Enterprise-Bench/l1-l2-bench run: 14 tasks
x 10 attempts = 140 trials, 0 errors, 89.29% accuracy.

Token/cost telemetry: this run routed through litellm to a
non-Anthropic model, and Claude Code's stream-json output does not
populate real usage stats for this route (same gap noted in devrev#9/devrev#11).
Instead of using those near-zero self-reported numbers, token counts
were independently reconstructed from each trial's own claude-code.txt
transcript (every system-prompt/tool/skill payload, user turn, and
tool_result fed to the model as input; every assistant text block and
tool_use payload as output), tokenized with tiktoken cl100k_base.
Reconstruction script: enterprise-bench-eval/scripts/audit_tokens.py
in the source repo. Cost is computed from those reconstructed
input/output tokens at GLM-5.2 list pricing ($1.40 / $4.40 per 1M).

Source job: https://hub.harborframework.com/jobs/32e17142-e309-4265-b359-50804adc6b06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant