Skip to content

Protocol Evals #1066

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
willwashburn wants to merge 5 commits into from
Closed

Protocol Evals #1066

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
37b510d
Add local workflow run commands
willwashburn Jun 3, 2026
25f5e2f
Avoid log read file race
willwashburn Jun 4, 2026
d27feec
Merge branch 'main' into feature/local-cloud-workflow-files
willwashburn Jun 5, 2026
5012154
Use Relayflows for local workflow runs
willwashburn Jun 5, 2026
431c227
Add agent messaging eval suite and reports
willwashburn Jun 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion .agentworkforce/trajectories/active/traj_b1jrutolckfb/trajectory.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,18 @@
"reasoning": "User requested Relaycast request attribution, install/update events, and MCP action-call telemetry while preserving UA-like harness values."
},
"significance": "high"
},
{
"ts": 1780761935387,
"type": "decision",
"content": "Repaired broker-harness.ts to current SDK/harness-driver API and built eval suite on it: Repaired broker-harness.ts to current SDK/harness-driver API and built eval suite on it",
"raw": {
"question": "Repaired broker-harness.ts to current SDK/harness-driver API and built eval suite on it",
"chosen": "Repaired broker-harness.ts to current SDK/harness-driver API and built eval suite on it",
"alternatives": [],
"reasoning": "Broker integration suite was pre-existingly broken: SDK narrowing moved BrokerEvent/HarnessDriverClient/SendMessageInput to @agent-relay/harness-driver and RelayCast to @relaycast/sdk; AgentRelay facade no longer does broker lifecycle. Fixed imports + removed the unused AgentRelay facade from BrokerHarness; added a dedicated evals/tsconfig.json compiling only evals/ + utils/ so eval:build is green without rewriting the still-broken sibling test files."
},
"significance": "high"
}
]
}
Expand All @@ -55,4 +67,4 @@
"startRef": "bd42f4f84f41821e33879618d114941d6eabe835",
"endRef": "bd42f4f84f41821e33879618d114941d6eabe835"
}
}
}
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,3 +93,6 @@ web/.open-next
!/workflows/refactor/
!/workflows/relayauth-integration/
!/workflows/cloud-connect/

# Eval harness JSON reports (generated per run)
tests/integration/broker/evals-reports/
4 changes: 4 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- Agent messaging eval suite (`npm run eval`, `eval:matrix`, `eval:unit`, `eval:selftest`, `eval:toolcheck`) spawns real agent CLIs and scores, from broker events, whether agents actually used the MCP/CLI to message — reporting message-sent rate, phantom-message rate (intent stated in prose but no send), ACK/DONE protocol adherence, and wrong-channel replies. Includes a `realistic` tier (natural-language tasks where the protocol must come from the injected onboarding) and a `smoke` tier (leading prompts as a plumbing canary), a negative-control self-test, and a deterministic wrong-tool-name trap that flags onboarding referencing tools the MCP server doesn't register. Each run emits a self-contained HTML viewer (overview, per-scenario prompts, full message transcript, phantom call-outs) alongside JSON reports with baseline regression diffing.

@gemini-code-assist gemini-code-assist Bot Jun 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The changelog entry is extremely detailed and contains implementation specifics. According to the repository style guide, changelog entries should be concise and impact-first.

Suggested change
- Agent messaging eval suite (`npm run eval`, `eval:matrix`, `eval:unit`, `eval:selftest`, `eval:toolcheck`) spawns real agent CLIs and scores, from broker events, whether agents actually used the MCP/CLI to message — reporting message-sent rate, phantom-message rate (intent stated in prose but no send), ACK/DONE protocol adherence, and wrong-channel replies. Includes a `realistic` tier (natural-language tasks where the protocol must come from the injected onboarding) and a `smoke` tier (leading prompts as a plumbing canary), a negative-control self-test, and a deterministic wrong-tool-name trap that flags onboarding referencing tools the MCP server doesn't register. Each run emits a self-contained HTML viewer (overview, per-scenario prompts, full message transcript, phantom call-outs) alongside JSON reports with baseline regression diffing.
- Add agent messaging eval suite to spawn real agent CLIs and score protocol adherence, phantom messages, and wrong-channel replies, with HTML and JSON reporting.
References
  1. Changelog entries should be concise and impact-first, preferring one short bullet per user-visible change. (link)

@coderabbitai coderabbitai Bot Jun 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Condense this [Unreleased] bullet into short impact-first entries.

This entry is too long and mixes multiple details; split it into concise one-line user-visible bullets per change.

As per coding guidelines, “Changelog entries should be concise and impact-first, with one short bullet per user-visible change. Omit issue/PR links, internal notes, and implementation details.”

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@CHANGELOG.md` at line 12, The long single `[Unreleased]` changelog bullet in
CHANGELOG.md should be replaced by several concise, impact-first one-line
bullets: extract each user-visible change (e.g., "Adds agent messaging
evaluation suite with HTML and JSON reports", "Supports realistic and smoke
tiers for protocol-driven tasks", "Reports message-sent rate, phantom-message
rate, ACK/DONE and wrong-channel detection", "Includes negative-control
self-test and deterministic wrong-tool-name trap") and write each as a short
standalone line; remove implementation details and internal notes, avoid
PR/issue links, and ensure each bullet is brief and focused on user-facing
impact.

Source: Coding guidelines

- `@agent-relay/harnesses` adds a `grok` PTY harness for the Grok CLI, including Relaycast MCP support for spawned agents.
- `@agent-relay/harnesses` is now published to npm, so SDK consumers can install the prebuilt PTY harnesses and harness-authoring helpers.
- `agent-relay drive` and `agent-relay passthrough` add adaptive predictive echo so typing stays responsive when driving a high-latency or remote agent, and stays invisible on fast local links.
Expand Down Expand Up @@ -39,6 +40,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- `@agent-relay/sdk` agent clients send via `sendMessage({ to })` (`#channel`, `@handle`, or an array of `@handle`s for a group DM), `reply({ messageId })`, and `react({ messageId, emoji })`; every message exposes `messageId`.
- `@agent-relay/harnesses` adds `createHuman({ relay, name })` (self-registers a human, returns the live client) and re-exports `defineHarness` plus the harness contract types.
- `agent-relay` forwards CLI origin, orchestrator harness, and distinct client identity context to hosted Relaycast so backend telemetry can distinguish CLI/SDK traffic from raw API calls.
- `agent-relay local run|logs|sync` starts executable workflow files on the local machine, stores run metadata and logs under `.agentworkforce/relay/local-runs`, and mirrors the cloud run/logs/sync command shape for laptop-hosted workflows.
- `agent-relay local run` supports Relayflows YAML workflows through the same background logs and sync wrapper used for local script workflows.

### Changed

Expand All @@ -57,6 +60,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Root builds now validate the simplified core package set: config, utils, SDK, harness-driver, harnesses, and CLI.
- `@agent-relay/sdk` no longer emits client-side analytics or depends on `@agent-relay/telemetry`; SDK/API attribution uses Relaycast origin metadata instead.
- `agent-relay` CLI telemetry now posts through the hosted ingestion proxy at `https://i.agentrelay.com` by default.
- `agent-relay local run` delegates YAML, TypeScript, and Python workflow execution to `@relayflows/cli` instead of bundling TypeScript workflows inside the Relay CLI.

### Deprecated

Expand Down
Loading
Loading