AgentWorkforce · khaliqgant · Jun 9, 2026 · Jun 9, 2026 · Jun 9, 2026 · Jun 9, 2026
@@ -0,0 +1,75 @@
+name: Relay Evals
+
+on:
+  pull_request:
+    paths:
+      - ".github/workflows/relay-evals.yml"
+      - "AGENTS.md"
+      - "README.md"
+      - "evals/**"
+      - "packages/**"
+      - "scripts/evals/**"
+      - "package.json"
+      - "package-lock.json"
+  push:
+    branches:
+      - main
+    paths:
+      - ".github/workflows/relay-evals.yml"
+      - "AGENTS.md"
+      - "README.md"
+      - "evals/**"
+      - "packages/**"
+      - "scripts/evals/**"
+      - "package.json"
+      - "package-lock.json"
+  workflow_dispatch:
+
+concurrency:
+  group: relay-evals-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+env:
+  NODE_VERSION: "22"
+  NPM_CONFIG_FUND: "false"
+
+jobs:
+  evals:
+    name: Offline evals
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: ${{ env.NODE_VERSION }}
+          cache: npm
+          cache-dependency-path: package-lock.json
+
+      - name: Install dependencies
+        run: npm ci
+
+      - name: Build SDK
+        run: npm run build:sdk
+
+      - name: Run offline evals
+        run: npm run evals:offline
+
+      - name: Summarize evals
+        if: always()
+        run: node scripts/evals/ci-summary.mjs
+
+      - name: Upload eval artifacts
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: relay-eval-run
+          path: .relay/evals/runs/
+          retention-days: 14
+          if-no-files-found: ignore
@@ -0,0 +1,31 @@
+# Trajectory: Review and fix PR #1092
+
+> **Status:** ✅ Completed
+> **Confidence:** 70%
+> **Started:** June 11, 2026 at 08:29 AM
+> **Completed:** June 11, 2026 at 08:30 AM
+
+---
+
+## Summary
+
+Fixed eval parser, runner, summary, executor, and relay-check issues for PR #1092; scoped verification passed, full verification blocked by incomplete dependency install and GitHub mergeability is dirty.
+
+**Approach:** Standard approach
+
+---
+
+## Key Decisions
+
+### Kept fixes scoped to relay eval harness
+- **Chose:** Kept fixes scoped to relay eval harness
+- **Reasoning:** Validated current PR comments and changed only PR eval scripts/checks; full repo verification is blocked by killed npm ci leaving missing dependencies.
+
+---
+
+## Chapters
+
+### 1. Work
+*Agent: default*
+
+- Kept fixes scoped to relay eval harness: Kept fixes scoped to relay eval harness
@@ -0,0 +1,53 @@
+{
+  "id": "traj_o61z0ze6kvla",
+  "version": 1,
+  "task": {
+    "title": "Review and fix PR #1092"
+  },
+  "status": "completed",
+  "startedAt": "2026-06-11T08:29:58.788Z",
+  "completedAt": "2026-06-11T08:30:00.119Z",
+  "agents": [
+    {
+      "name": "default",
+      "role": "lead",
+      "joinedAt": "2026-06-11T08:29:59.481Z"
+    }
+  ],
+  "chapters": [
+    {
+      "id": "chap_f292s4revkwp",
+      "title": "Work",
+      "agentName": "default",
+      "startedAt": "2026-06-11T08:29:59.481Z",
+      "endedAt": "2026-06-11T08:30:00.119Z",
+      "events": [
+        {
+          "ts": 1781166599482,
+          "type": "decision",
+          "content": "Kept fixes scoped to relay eval harness: Kept fixes scoped to relay eval harness",
+          "raw": {
+            "question": "Kept fixes scoped to relay eval harness",
+            "chosen": "Kept fixes scoped to relay eval harness",
+            "alternatives": [],
+            "reasoning": "Validated current PR comments and changed only PR eval scripts/checks; full repo verification is blocked by killed npm ci leaving missing dependencies."
+          },
+          "significance": "high"
+        }
+      ]
+    }
+  ],
+  "retrospective": {
+    "summary": "Fixed eval parser, runner, summary, executor, and relay-check issues for PR #1092; scoped verification passed, full verification blocked by incomplete dependency install and GitHub mergeability is dirty.",
+    "approach": "Standard approach",
+    "confidence": 0.7
+  },
+  "commits": [],
+  "filesChanged": [],
+  "projectId": "/home/daytona/workspace",
+  "tags": [],
+  "_trace": {
+    "startRef": "5e63ef398e6376b6c96d9cffeae9c1b668ab45cf",
+    "endRef": "5e63ef398e6376b6c96d9cffeae9c1b668ab45cf"
+  }
+}
@@ -0,0 +1,193 @@
+# Relay Evals — Master Plan
+
+Goal: an extensive, deterministic eval suite that proves **agents using the
+Relay protocol + `@agent-relay/sdk` behave as expected across every SDK
+surface**. Sibling projects `../agent-assistant` and `../relayfile` already run
+this style of suite; we adopt the same substrate so tooling, CI, and artifacts
+are consistent.
+
+## Architecture (mirror `../relayfile`)
+
+`../relayfile` consumes the reusable eval substrate published as
+`@agent-assistant/telemetry/evals` and adds product-owned cases + an executor.
+We do the same for relay.
+
+```
+evals/
+  README.md
+  PLAN.md                      # this file (source of truth)
+  suites/
+    <suite>/
+      cases.md                 # HUMAN-AUTHORED source of truth
+      cases.jsonl              # GENERATED by compile-cases.mjs — do not edit
+      rubric.md                # what "passing" means for this suite
+scripts/evals/
+  compile-cases.mjs            # cases.md -> cases.jsonl (port from relayfile)
+  run-relay-evals.mjs          # loads suites, runs executor, writes artifacts
+  relay-executor.mjs           # exercises @agent-relay/sdk in-memory, returns observed state
+  relay-checks.mjs             # deterministic expectation checks
+  ci-summary.mjs               # CI summary
+.github/workflows/relay-evals.yml
+.relay/evals/runs/             # run artifacts (gitignored)
+```
+
+npm scripts (root `package.json`):
+```
+"evals:compile": "node scripts/evals/compile-cases.mjs",
+"evals":         "npm run evals:compile && node scripts/evals/run-relay-evals.mjs",
+"evals:list":    "npm run evals:compile && node scripts/evals/run-relay-evals.mjs --list",
+"evals:offline": "npm run evals:compile && node scripts/evals/run-relay-evals.mjs --mode offline"
+```
+
+## cases.md format (verbatim from relayfile — do not invent a new one)
+
+```
+# <Suite Title>
+<one-paragraph description>
+
+## <suite>.<case-id>
+Executor: relay
+Kind: regression | capability
+Tags: a, b
+Human Review: false
+
+### Message
+<agent-facing scenario sentence>
+
+### Mock
+```json
+{ "...seed state for the in-memory relay..." }
+```
+
+### Operations
+```json
+[ { "op": "<verb>", "...": "..." } ]
+```
+
+### Deterministic Checks
+ok: true
+contentIncludes:
+- ...
+must:
+- ...
+mustNot:
+- ...
+```
+
+## Executor operation vocabulary (W1 owns + extends; authors target these)
+
+Each operation maps to a real SDK / MCP surface call run against an in-memory
+relay (NO live broker). Authors: if you need an op not listed, request it in
+`#relay-evals` so W1 adds it before you depend on it.
+
+- messaging: `post_message`, `send_dm`, `send_group_dm`, `reply_to_thread`,
+  `add_reaction`, `remove_reaction`, `list_messages`, `search_messages`,
+  `mark_read`, `get_thread`, `get_readers`, `check_inbox`
+- channels: `create_channel`, `join_channel`, `leave_channel`,
+  `invite_to_channel`, `archive_channel`, `set_topic`, `list_channels`,
+  `list_dms`, `create_workspace`, `set_workspace_key`
+- agents: `register_agent`, `add_agent`, `remove_agent`, `list_agents`
+- delivery: `deliver` with `mode: wait|steer`
+- actions: `register_action`, `invoke_action`
+- session: `define_harness`, `resume_session`
+
+`relay-checks.mjs` deterministic keys (port from relayfile, extend as needed):
+`ok`, `contentIncludes`, `toolCallsInclude`, `minToolCalls`, `must`, `mustNot`,
+plus relay-specific: `messageExists`, `threadReplyCount`, `reactionCount`,
+`channelMembers`, `agentPresence`, `errorCode`, `eventEmitted`.
+
+## Construction guidance for the executor
+
+The executor builds clients exactly like the existing vitest suites do. Read
+these first — they are the template for the in-memory harness:
+`packages/sdk/src/__tests__/{messaging,delivery-actions,facade,listeners,register-action-relay,relaycast-errors}.test.ts`.
+Prefer the SDK's existing in-memory primitives (`InMemoryAgentRelayActions`,
+`createWorkspaceFacade`, mock messaging client) over standing up a real broker.
+
+## Surface ownership (workers)
+
+| Worker | Surfaces / suites |
+|---|---|
+| **W1 eval-harness** | substrate: compile/run/executor/checks/ci-summary, npm scripts, CI workflow, `evals/README.md`, + reference suite `protocol-framing` (envelope encode/decode, protocol↔SDK type conformance) proving the harness end-to-end |
+| **W2 messaging** | `messaging` (post/dm/group-dm), `threads` (reply_to_thread, get_thread, reply counts), `reactions` (add/remove, counts), `read-receipts` (mark_read, get_readers, inbox), `search` (search_messages, list_messages) |
+| **W3 channels** | `channels` (create/join/leave/invite/archive/set_topic), `workspaces` (create_workspace/set_workspace_key), `agent-directory` (register/add/remove/list_agents, list_channels, list_dms, presence) |
+| **W4 delivery-actions** | `delivery-modes` (wait vs steer, retries/backoff via DeliveryRunner), `actions` (register/invoke via InMemoryAgentRelayActions/ActionRegistry), `action-schema` (json-schema-lite validation, actionSchemaToJsonSchema), `action-errors` (Registration/NotFound/Validation), `capabilities` (RelayCapabilityError) |
+| **W5 session-listeners-errors** | `session` (defineHarness, nextHarnessName, normalizeAgentIdentity, resume/continuity, MINIMAL_AGENT_SESSION_CAPABILITIES), `listeners` (hub + MessageCreated/Read/Reacted/Action/Status/ToolCalled predicates, matchesSelector, toPublicMessagingEvent), `facade` (createWorkspaceFacade, notify handler, agent client), `auth-errors` (invalid agent token detect/recover) |
+
+Target **≥ 15–25 cases per worker** (mix regression + capability). Total goal:
+a large suite (100+ cases) covering every surface above.
+
+## Workflow rules (CLAUDE.md)
+
+- Branch: `feature/relay-evals`. NEVER push to main.
+- Commit to the feature branch only; the operator merges.
+- Keep `cases.jsonl` generated (never hand-edit); commit both `cases.md` and the
+  compiled `cases.jsonl`.
+- Run `npm run evals:offline -- --suite <yours>` and ensure it is green before
+  reporting your suite done.
+- Coordinate format/op questions in `#relay-evals`. Progress to the operator is
+  relayed by the **slack-comms** agent — post status to `#relay-evals` and it
+  will surface it; do not DM the operator directly.
+
+## Op argument reference (LOCKED by W1)
+
+Confirmed canonical by W1 (eval-harness). Verb names below are stable — author
+against them. Clarifications W1 added: `get_thread` accepts `{messageId}` or
+`{parent}`; `reply_to_thread` is `{as, parent, text, id?}`; `register_agents` is
+`{agents:[...]}`; all create/message ops accept optional `id` for deterministic
+checks. SDK-export aliases the executor maps: `format_handle`→`formatAgentHandle`,
+`token_recovery_message`→`agentTokenRecoveryMessage`. Every op takes `op` plus an
+acting identity via `as` where an agent context is required.
+
+Messaging:
+- `post_message` {as, channel, text, id?, idempotencyKey?, attachments?}
+- `send_dm` {as, to, text, id?}
+- `send_group_dm` {as, participants:[], name?, text, id?}
+- `reply_to_thread` {as, parent:<msgId>, text, id?}
+- `get_thread` {messageId}
+- `add_reaction` / `remove_reaction` {as, messageId, emoji}
+- `mark_read` {as, messageId}; `get_readers` {messageId}; `check_inbox` {as}
+- `list_messages` {channel, limit?}; `search_messages` {query, channel?}
+
+Session/listeners/facade/errors (thin wrappers over exported SDK fns):
+- session: `define_harness`{name,version?,input?}, `next_harness_name`{base,explicit?},
+  `normalize_identity`{input}, `format_handle`{name}, `read_capabilities`{},
+  `resume_session`{...}
+- listeners: `add_listener`{selector}, `on_predicate`{predicate}, `emit_event`{raw},
+  `emit_session_event`{agentId,event}, `match_selector`{selector,type}, `to_public_event`{raw}
+- facade: `register_agent`/`register_agents`{[...]}, `reconnect`{apiToken},
+  `notify`{target,options}, `workspace_info`{}
+- auth-errors: `is_invalid_token_error`{error}, `is_invalid_token_tool_result`{result},
+  `token_recovery_message`{}
+
+**Create ops accept optional `id`** to pin the created message's id so later ops
+and checks reference it deterministically; otherwise only seeded ids are stable.
+
+Mock seed shape:
+```json
+{ "agents": [{"name":"...","type":"agent"}],
+  "channels": [{"name":"...","members":["..."]}],
+  "messages": [{"id":"...","channel":"...","from":"...","text":"...","threadParent":"?"}] }
+```
+
+## Executor observed-result contract (W1 implements)
+
+So pure-function and event ops are checkable, the executor populates per run:
+- `observed.content` — stringified return value of each op (drives `contentIncludes`)
+- `observed.events[]` — emitted events (drives `eventEmitted`)
+- `observed.error.code` — error code when an op throws (drives `errorCode`)
+- `observed.toolCalls[]` — op/verb trace (drives `toolCallsInclude`/`minToolCalls`)
+- `register_action` uses JSON-safe `handlerFixture` names instead of inline
+  functions. Supported fixtures: `echo_text`, `sum_numbers`, `throw_error`,
+  `invalid_output`, and `policy_deny`. The executor also accepts `fixture` as
+  an alias for `handlerFixture`.
+
+Relay-specific check keys (bullet arrays of objects):
+- `messageExists`: {channel?|kind?, text, from?} — an observed msg matches all fields
+- `threadReplyCount`: {parent:<id>, count}
+- `reactionCount`: {messageId, emoji, count}
+- `channelMembers`: {channel, members:[...]}
+- `agentPresence`: {name, status}
+- `errorCode`: string | string[]
+- `eventEmitted`: string | {type, ...}
+Plus shared `ok`, `contentIncludes`, `must`, `mustNot`, `toolCallsInclude`, `minToolCalls`.