feat(evals): combined offline SDK harness + live behavioral eval suite by khaliqgant · Pull Request #1109 · AgentWorkforce/relay

docs(evals): add master plan for relay SDK eval suite

Proactive Runtime Bot

and

claude

committed

docs(evals): add op-arg reference + executor result contract

Proactive Runtime Bot

and

claude

committed

docs(evals): mark op vocabulary LOCKED per W1 confirmation

Proactive Runtime Bot

and

claude

committed

Add messaging relay eval suites

Proactive Runtime Bot

committed

Add session listener facade eval suites

Proactive Runtime Bot

committed

Add channel workspace agent directory eval suites

Proactive Runtime Bot

committed

Add delivery actions eval suites

Proactive Runtime Bot

committed

Relax W3 eval content expectations

Proactive Runtime Bot

committed

Align W3 eval duplicate error codes

Proactive Runtime Bot

committed

Relax delivery action eval expectations

Proactive Runtime Bot

committed

Align W5 eval cases with runner output

Proactive Runtime Bot

committed

Align workspace eval assertions with runner output

Proactive Runtime Bot

committed

Mark capability eval hooks pending

Proactive Runtime Bot

committed

Relax unknown agent removal eval assertion

Proactive Runtime Bot

committed

feat(evals): add relay eval harness

Proactive Runtime Bot

committed

chore: apply pr-reviewer fixes for #1092

agent-relay-code[bot]

authored and

Proactive Runtime Bot

committed

Add local workflow run commands

willwashburn

authored and

Proactive Runtime Bot

committed

Avoid log read file race

willwashburn

authored and

Proactive Runtime Bot

committed

Use Relayflows for local workflow runs

willwashburn

authored and

Proactive Runtime Bot

committed

Add agent messaging eval suite and reports

willwashburn

authored and

Proactive Runtime Bot

committed

feat(evals): add spawn/release reliability eval suite with onboarding variants

Proactive Runtime Bot

and

claude

committed

chore: apply pr-reviewer fixes for #1109

agent-relay-code[bot]

authored and

Proactive Runtime Bot

committed

feat(evals): multi-model lifecycle evals, HTML report redesign, 4 new scenarios

Proactive Runtime Bot

and

claude

committed

feat(evals): s04 native-subagent detection scenario

Proactive Runtime Bot

and

claude

committed

feat(broker): model-aware relay skill injection for small-tier models

Proactive Runtime Bot

and

claude

committed

feat(evals): comprehensive opencode model lifecycle eval scripts

Proactive Runtime Bot

and

claude

committed

feat(evals): s05 phrasing variants, auto-routing spec, skill text fix, opus timeout fix

Proactive Runtime Bot

and

claude

committed

feat(auto): add auto-routing Phase 1–3 — classifier, composer, Director prompt

Proactive Runtime Bot

and

claude

committed

feat(evals): add s06 auto-routing Director scenario and auto-routing eval group

Proactive Runtime Bot

and

claude

committed

docs(specs): update auto-routing spec with confirmed timeout fix and s05 early phrasing data

Proactive Runtime Bot

and

claude

committed

docs(specs): update auto-routing spec with skill text analysis and phrasing finding

Proactive Runtime Bot

and

claude

committed

fix(evals): strengthen skill text with explicit relay-tool disambiguation

Proactive Runtime Bot

and

claude

committed

docs(specs): add routing table rationale — conditional guidance hurts capable models

Proactive Runtime Bot

and

claude

committed

fix(evals): revise brief onboarding to remove conditional spawn guidance

Proactive Runtime Bot

and

claude

committed

docs(specs): update opus s03 results table with confirmed timeout fix findings

Proactive Runtime Bot

and

claude

committed

fix(broker): fix grok mcp add — NAME-first ordering and embed flag-args in command

Proactive Runtime Bot

and

claude

committed

docs(specs): add multi-harness eval findings to auto-routing spec

Proactive Runtime Bot

and

claude

committed

docs(specs): update multi-harness phrasing table with 5-run results

Proactive Runtime Bot

and

claude

committed

docs(specs): clarify grok failure mode post MCP fix + confirm opencode phrasing complete

Proactive Runtime Bot

and

claude

committed

docs(specs): add s01/s02/s03 lifecycle breakdown tables for non-Claude harnesses

Proactive Runtime Bot

and

claude

committed

docs(specs): update phrasing table with droid/gemini/grok/cursor data

Proactive Runtime Bot

and

claude

committed

docs(specs): update s03 lifecycle table with codex/opencode confirmed results

Proactive Runtime Bot

and

claude

committed

fix(evals): emphasize parallel spawn in s06 Director prompt

Proactive Runtime Bot

and

claude

committed

docs(specs): update auto-routing spec with complete s03/s05 non-Claude results

Proactive Runtime Bot

and

claude

committed

docs(specs): update s03 with droid/gemini confirmed results

Proactive Runtime Bot

and

claude

committed

fix(evals): inject fake worker ACK in s06 to unblock Director second spawn

Proactive Runtime Bot

and

claude

committed

fix(evals): extend s06 spawn window to 180s and simplify ACK text

Proactive Runtime Bot

and

claude

committed

fix(evals): change s06 injection to Orchestrator nudge after first spawn

Proactive Runtime Bot

and

claude

committed

docs(specs): document s06 multi-spawn finding and add to open questions

Proactive Runtime Bot

and

claude

committed

fix(evals): s06 v6 — Director executes from meta-prompt alone, no trigger message

Proactive Runtime Bot

and

claude

committed

docs(specs): update s04 table with codex/gemini/droid results

Proactive Runtime Bot

and

claude

committed

fix(evals): s06 v7 — count startup spawns before clearEvents

Proactive Runtime Bot

and

claude

committed

docs(evals): complete s05 phrasing data for all 7 harnesses; fix s06 scoring; document PTY multi-spawn limitation

Proactive Runtime Bot

and

claude

committed

fix(cursor): resolve cursor CLI to cursor-agent binary, not grok's agent binary

Proactive Runtime Bot

and

claude

committed

docs(evals): cursor-agent confirmed 0% — not viable relay worker

Proactive Runtime Bot

and

claude

committed

feat(evals): add codex model tier comparison results and fix parseHarnessSpec for codex models

Proactive Runtime Bot

and

claude

committed

feat(evals): add opencode batch eval script and codex tier constants to composer

Proactive Runtime Bot

and

claude

committed

fix(evals): harden opencode batch script — tolerate per-model failures, add missing claude variants

Proactive Runtime Bot

and

claude

committed

fix(evals): scope opencode batch to alternative/Chinese models only

Proactive Runtime Bot

and

claude

committed

feat(auto): add role-fit classification layer to composer and auto-routing spec

Proactive Runtime Bot

and

claude

committed

feat(evals): add Phase 1 opencode batch results to role-fit map

Proactive Runtime Bot

and

claude

committed

feat(evals): add role-fit rankings and communicator sections to master summary

Proactive Runtime Bot

and

claude

committed

feat(evals): add @agent-relay/evals package with publish pipeline

Proactive Runtime Bot

and

claude

committed

style: auto-format Rust code with cargo fmt

github-actions[bot]

authored and

Proactive Runtime Bot

committed

feat(evals): add s07-lead-delegation scenario group

Proactive Runtime Bot

and

claude

committed

fix(mcp): pass model via metadata in add_agent spawn call

Proactive Runtime Bot

and

claude

committed

fix(evals): bump @agent-relay/evals to 8.6.0 and regenerate lock file

Proactive Runtime Bot

and

claude

committed

style: auto-format all files with prettier; add broker gitignore for eval artifacts

Proactive Runtime Bot

and

claude

committed

fix(evals): use rk_live_ substring check instead of glob in workspaces eval

Proactive Runtime Bot

and

claude

committed

style: auto-format with Prettier

github-actions[bot]

committed

fix(ci): resolve three CI failures on PR #1109

Proactive Runtime Bot

and

claude

committed

fix(evals): re-apply rk_live_ substring fix after prettier bot revert

Proactive Runtime Bot

and

claude

committed

style: auto-format with Prettier

github-actions[bot]

committed

fix(evals): use exact mock key in workspace contentIncludes check

Proactive Runtime Bot

and

claude

committed

fix(lock): remove stale broker@8.2.0 entries from packages/cli/node_modules

Proactive Runtime Bot

and

claude

committed

Uh oh!

Commits

Commits on Jun 14, 2026