From 964c37ec6c6b6a8caa69bf8e0ba289a922817001 Mon Sep 17 00:00:00 2001 From: Awesome Rustacean Date: Mon, 17 Aug 2026 13:00:43 +0300 Subject: [PATCH 1/4] docs(rfd): proposed agent interaction testing. Co-authored-by: Codex --- md/SUMMARY.md | 2 + md/rfds/agent-interaction-testing/README.md | 211 ++++++++++++++++++ .../proposed-agent-interaction-testing.md | 80 +++++++ 3 files changed, 293 insertions(+) create mode 100644 md/rfds/agent-interaction-testing/README.md create mode 100644 md/rfds/agent-interaction-testing/proposed-agent-interaction-testing.md diff --git a/md/SUMMARY.md b/md/SUMMARY.md index 0ed860b4..a505ec54 100644 --- a/md/SUMMARY.md +++ b/md/SUMMARY.md @@ -96,6 +96,8 @@ - [Discovery & sync](./rfds/registry-centric-plugins/discovery-sync/README.md) - [User-managed plugins](./rfds/registry-centric-plugins/user-managed-plugins/README.md) - [Predicate caching](./rfds/predicate-caching/README.md) + - [Agent interaction testing](./rfds/agent-interaction-testing/README.md) + - [Proposed: Agent interaction tests](./rfds/agent-interaction-testing/proposed-agent-interaction-testing.md) - [Completed](./rfds/completed.md) - [Configuration parsing and normalization](./rfds/config-normalization/README.md) - [RFD Process](./rfds/rfd-process/README.md) diff --git a/md/rfds/agent-interaction-testing/README.md b/md/rfds/agent-interaction-testing/README.md new file mode 100644 index 00000000..baee2af5 --- /dev/null +++ b/md/rfds/agent-interaction-testing/README.md @@ -0,0 +1,211 @@ +# Agent interaction testing + +## TL;DR + +- Add an experimental integration-test suite that drives real coding agents through realistic, multi-turn user journeys. +- Keep the existing deterministic integration tests; the new suite tests the combined behavior of Symposium and an agent. +- Define scenarios independently of any agent, with Claude as the first adapter. +- Support a fast host runner for development and an authoritative Linux-container runner for production conformance. +- Record exact system evidence and use narrow semantic checkpoints instead of matching entire model responses. + +## Motivation + +Symposium aims to be a one-stop shop that helps coding agents write great Rust code. Its value does not come from one isolated command. It comes from the interaction between discovery, user consent, skills, hooks, MCP servers, configuration, caching, and the coding agent itself. + +The existing integration tests are valuable for deterministic CLI and registry behavior. They cannot answer the larger question: given a realistic Rust task, does a real agent receive and use what Symposium provides, and does the resulting interaction behave as intended? + +We need a production rehearsal that starts from a controlled fixture, isolates agent and user state from the developer's machine, drives a realistic conversation, and captures enough evidence to explain both success and failure. + +## Product contract + +The suite distinguishes guarantees from hypotheses. + +Symposium's platform guarantee is that it: + +- discovers extensions relevant to the current project; +- respects trust and explicit user choices; +- delivers enabled skills, hooks, and MCP servers to supported agents; +- behaves consistently across agent adapters where their capabilities overlap; and +- makes failures visible and contains their effects. + +Our initial outcome hypothesis is that this platform helps an agent produce more correct, idiomatic, and current Rust. Version one checks representative journeys and concrete outcomes. It is not a statistical model benchmark and does not claim that every successful run proves a general improvement over a baseline agent. + +The first audience is an individual Rust developer using an agent in a repository. Organization policy, team-managed configuration, and crate-author publishing journeys are valuable later extensions, but they are not required to validate the first design. + +## Change in a nutshell + +Add a separate `cargo xtask agent-test` suite built around four replaceable boundaries: + +1. a serializable, agent-neutral scenario description; +2. an agent adapter that exposes capabilities and a persistent conversation; +3. an environment backend that runs either on the host or in an isolated container; and +4. graders that inspect exact state, normalized events, and bounded semantic outcomes. + +The suite is additive. Existing fixtures, `TestContext`, simulations, and deterministic tests remain the primary way to cover the combinatorial corners of the registry and CLI. Real-agent journeys cover a smaller set of representative end-to-end interactions. + +## Detailed plans + +### Scenario model + +Scenarios are authored with typed Rust builders at first. The underlying model remains pure data: steps cannot contain arbitrary Rust closures. This keeps scenarios serializable and leaves open a later YAML or TOML representation without committing to a DSL before the vocabulary is understood. + +A scenario contains: + +- a fixture describing project files, registry data, and local services; +- required capabilities such as persistent conversation, PTY input, hooks, or MCP; +- environment and authentication requirements; +- ordered user, CLI, mutation, restart, and checkpoint steps; +- deadlines and resource limits; and +- graders and artifact-retention policy. + +There are two interaction channels: + +- PTY steps drive an interactive CLI by waiting for an observable prompt or state, sending a line or key, and checking exit status. +- Agent turns send a user message to a persistent structured agent session and wait for a protocol completion event. + +Tests never synchronize with fixed sleeps. Every wait targets an observable condition and has a deadline. + +### Agent-neutral adapters + +An `AgentDriver` reports its capabilities, prepares an isolated runtime, starts and stops a persistent session, sends turns, and returns both normalized events and raw provider artifacts. + +Scenarios select capabilities, not brand names. If the selected driver or environment cannot provide a required capability, the result is `Unavailable`, not a misleading test failure. + +Claude is the first adapter because it is already used by Symposium developers. Its structured SDK is used for the main journeys so that completion and tool activity are observable. One narrow PTY smoke test covers the real interactive Claude entry point. Claude-specific protocol details must remain inside the adapter. + +The existing `AgentSession::ClaudeSdk` path starts a new provider query for every prompt. The new adapter must maintain one conversation across turns so that confirmation, follow-up work, and later-session behavior are genuine interactions. + +### Environment backends + +The host backend is optimized for fast local iteration. It creates fresh project, home, configuration, cache, and temporary directories; filters inherited environment variables; and may reuse the developer's local agent authentication. It is useful but not authoritative because the host OS and installed tools can still affect results. + +The container backend is the production-conformance environment. Version one uses Linux containers through Docker, behind an environment abstraction that can later support another container runtime, a VM, or a remote worker. + +Each scenario receives a fresh container, while all turns and deliberate agent restarts within that scenario share its writable state. The fixture is copied into the container rather than mounting the repository. The container runs as a non-root user with a read-only root filesystem, dropped capabilities, no Docker socket, explicit writable directories, and CPU, memory, process, and time limits. + +The container image is layered and cached. A stable base contains the agent runtime and ordinary tools. The current compatible Symposium Linux binary is built once per revision, or supplied explicitly with `--symposium-bin`, and copied into a thin test layer. We do not install Symposium from a package manager: doing so would test a released artifact rather than the code under development and would make the suite depend on registry and network speed. + +This also avoids copying a binary during every scenario. Image preparation is incremental; scenarios start only after the revision-specific layer exists. + +### Network and authentication + +Authoritative container runs use a restricted CI API key. Host runs may reuse local Claude authentication for developer convenience. + +The scenario container has no direct external egress. Provider traffic passes through a controlled forward-proxy sidecar connected to both an internal scenario network and an egress network. The proxy permits only provider endpoints and any endpoints explicitly declared by the scenario. Fixture services and local MCP servers stay on the internal network. + +The CI credential is available only to trusted scheduled or manually dispatched jobs, never to fork pull requests. Version one is experimental and non-blocking while cost, stability, and diagnostic quality are measured. + +### Evidence and grading + +The runner writes a canonical, coarse event journal while retaining raw agent and process artifacts. The canonical vocabulary includes events such as process start and exit, prompt observed, input sent, agent turn completed, tool invoked, file changed, configuration changed, and grader completed. + +Authoritative assertions prefer: + +- exact files and configuration state; +- exact process status and normalized system events; +- protocol-level completion and tool activity; and +- task-specific graders such as compilation, tests, or targeted source inspection. + +Model prose is checked only through narrow, stable semantic anchors when it is itself part of the contract. Full-response snapshots and exact wording are diagnostic, not gating. + +A run has one of four results: + +- `Passed`: the requested journey completed and all graders passed; +- `Failed`: the environment ran correctly but the behavior violated an assertion; +- `InfrastructureError`: setup, credentials, provider access, or the runner failed; +- `Unavailable`: the selected agent or environment lacks a required capability. + +An explicitly requested unavailable run exits unsuccessfully and explains the missing capability. Ordinary `cargo test` is unaffected because this suite is opt-in. + +Artifacts live under `target/agent-tests//`. Every run keeps a compact summary. Failures keep sanitized journals, transcripts, logs, diffs, and relevant final state. Passing runs retain full artifacts only with `--keep-artifacts`. Secrets must be removed before artifacts are persisted. + +### Initial journeys + +The first suite should contain a few high-value vertical journeys: + +1. A fresh agent enters a Rust fixture with a trusted dependency, receives the relevant Symposium guidance, completes a controlled task, and passes Rust-specific graders. +2. An untrusted dependency triggers consent. Separate variants enable and decline it, and a later session honors the recorded choice. +3. A hook affects agent behavior and leaves the expected observable evidence. +4. An MCP-assisted task demonstrates that the configured server is available and useful to the agent. +5. A registry resynchronization is observed, followed by one deliberately broken extension whose failure is visible and contained. + +Combinatorial cases such as every predicate permutation, cache boundary, or malformed registry entry remain in deterministic tests. A real-agent journey is added when the interaction between user, Symposium, and agent is what could fail. + +### Command-line interface + +The proposed entry point is: + +```console +cargo xtask agent-test [OPTIONS] +``` + +Initial options are: + +```text +--list +--agent +--environment +--scenario +--symposium-bin +--keep-artifacts +``` + +Running container scenarios without a working runtime, compatible binary, or required credential produces an explicit `Unavailable` result. It must never silently fall back to the host backend. + +### Time and cost + +Containers add image preparation and startup time, but real-agent latency will usually dominate. The runner records cold image preparation, warm environment startup, agent time, and grading time separately so that optimization is based on measurements. + +Fast deterministic tests continue to run on every change. Host journeys are for focused development. A small container suite runs on a trusted schedule or manual dispatch. This layering avoids multiplying expensive agent calls across the full deterministic matrix. + +## Frequently asked questions + + + +### Does this replace the current integration tests? + +No. The current tests give faster, deterministic, exhaustive coverage of Symposium's own logic. The new suite adds evidence about real interactions. Existing infrastructure should only change where a small reusable seam makes both suites clearer. The old one-shot real-agent path may be removed after the persistent adapter supersedes it. + +### Why typed Rust scenarios instead of YAML or TOML? + +Typed builders provide compiler-assisted refactoring, good IDE discovery, and direct reuse of test helpers while the scenario vocabulary is still changing. The cost is that non-Rust contributors cannot edit a data file and scenarios must be recompiled. Keeping the scenario model serializable and closure-free preserves an escape hatch: once the vocabulary stabilizes, a data format can be added as another frontend. + +### Can deterministic conversation checkpoints work with a nondeterministic model? + +Yes, if the checkpoints target deterministic boundaries. We can exactly check which configuration changed, whether a prompt was answered, which capability became available, what files were produced, and whether the Rust task passes. We should not require the model to emit an exact paragraph. + +### Why not copy the developer's installed Symposium executable into every test? + +The installed executable may not match the checkout and copying it per scenario wastes time. The default authoritative path builds or accepts one compatible Linux binary and caches it in a thin image layer. An explicit `--symposium-bin` remains useful for testing a known artifact. + +### Why are containers not the only backend? + +The host backend shortens the edit-test-debug loop and can use local authentication. The container backend answers the stronger production-conformance question. Treating them as implementations of one environment interface keeps scenarios portable without pretending that host isolation is complete. + +### What do the linked CLI testing projects contribute? + +`cli-testing-library` demonstrates a useful interaction model: wait for observable output, query the screen, send user events, and avoid hand-written timing. Its Node implementation and reported platform constraints make it a reference rather than a foundation for this Rust, cross-platform suite. + +`cli-testing-specialist` is oriented toward generic, generated CLI validation. Our journeys need persistent agents, Symposium-specific state, hooks, MCP, consent, and outcome graders, so adopting it would not remove the hard integration work. + +### How will we know whether Symposium caused an idiomatic Rust result? + +Version one proves delivery and checks bounded task outcomes: the relevant extension was selected, the agent could use it, and the fixture satisfies concrete Rust graders. Strong causal claims require repeated paired runs against a no-Symposium baseline and statistical analysis. That is a future evaluation layer, not a prerequisite for integration testing. + +## Implementation plan + +1. Introduce the scenario, capability, event, artifact, and result types with fake drivers and runner tests. +2. Add the host backend and a persistent Claude adapter. +3. Add the PTY driver for interactive Symposium and one Claude entry-point smoke test. +4. Add the Linux container backend, revision-layered Symposium binary, proxy, isolation rules, and infrastructure diagnostics. +5. Implement the trusted-dependency guidance journey as the first production rehearsal. +6. Add consent, hook, MCP, resynchronization, and contained-failure journeys. +7. Add the trusted scheduled/manual CI workflow, document operation and cost, correct stale design documentation, and retire superseded one-shot agent-test code. + + + +## Implementation status + +This RFD describes proposed experimental infrastructure. Implementation has not begun. + +See [Proposed: Agent interaction tests](./proposed-agent-interaction-testing.md) for the intended operator workflow. \ No newline at end of file diff --git a/md/rfds/agent-interaction-testing/proposed-agent-interaction-testing.md b/md/rfds/agent-interaction-testing/proposed-agent-interaction-testing.md new file mode 100644 index 00000000..386af031 --- /dev/null +++ b/md/rfds/agent-interaction-testing/proposed-agent-interaction-testing.md @@ -0,0 +1,80 @@ +# Agent interaction tests + +Agent interaction tests exercise Symposium together with a real coding agent in a controlled project. They complement the ordinary test suite: use ordinary tests for exhaustive CLI and registry logic, and use these journeys when the interaction among the user, Symposium, and the agent is the behavior under test. + +The feature is experimental. Real-agent runs consume provider capacity and may expose provider variability, so they are opt-in and separate from `cargo test`. + +## Discovering scenarios + +List the scenarios and their required capabilities: + +```console +cargo xtask agent-test --list +``` + +The output identifies whether the selected agent and environment can run each scenario. A missing capability is reported as unavailable rather than being mistaken for a product failure. + +## Running locally + +For a fast development run, use the host environment: + +```console +cargo xtask agent-test --agent claude --environment host --scenario trusted-dependency-guidance +``` + +The host runner creates isolated project, home, configuration, cache, and temporary directories. It may use the agent authentication already present on the machine. Host results are useful for debugging but are not authoritative because installed tools and the operating system can still influence the run. + +## Running a production-conformance journey + +Use the container environment for the authoritative Linux rehearsal: + +```console +$env:ANTHROPIC_API_KEY = "..." +cargo xtask agent-test --agent claude --environment container --scenario trusted-dependency-guidance +``` + +The runner prepares a cached base image and a thin layer containing the Symposium binary for the current revision. It then creates a fresh, restricted container for the scenario. All turns and deliberate restarts in that scenario share its state. + +To test an already-built compatible Linux artifact, select it explicitly: + +```console +cargo xtask agent-test --agent claude --environment container --symposium-bin ./artifacts/symposium-linux-x86_64 --scenario trusted-dependency-guidance +``` + +The runner never silently substitutes a released package or falls back from a requested container to the host. + +## Reading a result + +Each run ends as one of: + +* `Passed` — the journey and its graders succeeded; +* `Failed` — the environment worked, but observed behavior violated the scenario; +* `InfrastructureError` — setup, authentication, provider access, or the runner failed; +* `Unavailable` — a requested driver or environment lacks a required capability. + +The console summary names the failed step and points to `target/agent-tests//`. Failure artifacts include a sanitized event journal, relevant logs, the conversation transcript, file diffs, and final inspected state. Use `--keep-artifacts` to retain the same detail after a successful run: + +```console +cargo xtask agent-test --agent claude --environment container --scenario consent-enable --keep-artifacts +``` + +## Writing a scenario + +Scenarios are initially written with typed Rust builders, but contain portable data rather than arbitrary closures. A typical scenario describes this sequence: + +1. Create a Rust fixture whose dependency has a trusted Symposium extension. +2. Start a persistent agent session in the fixture. +3. Ask the agent to implement a small, controlled Rust task. +4. Wait for protocol completion rather than sleeping for a guessed duration. +5. Assert that Symposium selected and delivered the extension. +6. Inspect the resulting files and run targeted Rust checks. + +Check exact state at deterministic boundaries. For example, check that consent was recorded, a hook event occurred, an MCP tool was invoked, or `cargo test` passed. Do not snapshot an entire model answer or require incidental wording. + +A scenario declares capabilities such as `persistent-conversation`, `pty-input`, `hooks`, and `mcp`. It does not contain Claude-specific branching. Agent-specific behavior belongs in the adapter. + +## CI operation + +Container journeys run only in a trusted scheduled or manually dispatched workflow. The workflow uses a restricted spending credential and does not expose it to fork pull requests. While the suite is experimental, failures are visible but do not block ordinary pull requests. + +When investigating runtime, compare the recorded phases separately: image preparation, environment startup, agent execution, and grading. A slow provider turn should not be diagnosed as slow container startup. From 3bd6a2ed2dd922e58ed6d57715225ef884105d97 Mon Sep 17 00:00:00 2001 From: Awesome Rustacean Date: Fri, 21 Aug 2026 01:15:56 +0300 Subject: [PATCH 2/4] docs(rfd): refine agent interaction testing design Scope the proposal to a tracer implementation and define isolation, evidence, reliability, token budget, and follow-on work. Co-authored-by: Codex --- md/SUMMARY.md | 7 +- md/rfds/agent-interaction-testing/README.md | 216 ++++-------------- .../agent-adapters/README.md | 51 +++++ .../coverage-and-ci/README.md | 155 +++++++++++++ .../environments/README.md | 62 +++++ .../evidence/README.md | 85 +++++++ .../proposed-agent-interaction-testing.md | 80 ------- .../proposed-guide/README.md | 117 ++++++++++ .../scenario-model/README.md | 62 +++++ 9 files changed, 584 insertions(+), 251 deletions(-) create mode 100644 md/rfds/agent-interaction-testing/agent-adapters/README.md create mode 100644 md/rfds/agent-interaction-testing/coverage-and-ci/README.md create mode 100644 md/rfds/agent-interaction-testing/environments/README.md create mode 100644 md/rfds/agent-interaction-testing/evidence/README.md delete mode 100644 md/rfds/agent-interaction-testing/proposed-agent-interaction-testing.md create mode 100644 md/rfds/agent-interaction-testing/proposed-guide/README.md create mode 100644 md/rfds/agent-interaction-testing/scenario-model/README.md diff --git a/md/SUMMARY.md b/md/SUMMARY.md index a505ec54..ef43a934 100644 --- a/md/SUMMARY.md +++ b/md/SUMMARY.md @@ -97,7 +97,12 @@ - [User-managed plugins](./rfds/registry-centric-plugins/user-managed-plugins/README.md) - [Predicate caching](./rfds/predicate-caching/README.md) - [Agent interaction testing](./rfds/agent-interaction-testing/README.md) - - [Proposed: Agent interaction tests](./rfds/agent-interaction-testing/proposed-agent-interaction-testing.md) + - [Scenario model](./rfds/agent-interaction-testing/scenario-model/README.md) + - [Agent adapters](./rfds/agent-interaction-testing/agent-adapters/README.md) + - [Execution environments](./rfds/agent-interaction-testing/environments/README.md) + - [Evidence and results](./rfds/agent-interaction-testing/evidence/README.md) + - [Coverage and CI](./rfds/agent-interaction-testing/coverage-and-ci/README.md) + - [Proposed: Agent interaction tests](./rfds/agent-interaction-testing/proposed-guide/README.md) - [Completed](./rfds/completed.md) - [Configuration parsing and normalization](./rfds/config-normalization/README.md) - [RFD Process](./rfds/rfd-process/README.md) diff --git a/md/rfds/agent-interaction-testing/README.md b/md/rfds/agent-interaction-testing/README.md index baee2af5..32c051bd 100644 --- a/md/rfds/agent-interaction-testing/README.md +++ b/md/rfds/agent-interaction-testing/README.md @@ -2,210 +2,86 @@ ## TL;DR -- Add an experimental integration-test suite that drives real coding agents through realistic, multi-turn user journeys. -- Keep the existing deterministic integration tests; the new suite tests the combined behavior of Symposium and an agent. -- Define scenarios independently of any agent, with Claude as the first adapter. -- Support a fast host runner for development and an authoritative Linux-container runner for production conformance. -- Record exact system evidence and use narrow semantic checkpoints instead of matching entire model responses. +- Extend the existing test infrastructure with scripted user, CLI, and real-agent journeys. +- Start every journey from controlled fixtures and isolated user and agent state. +- Run production-facing `cargo agents` processes under a real PTY. +- Keep exhaustive registry logic deterministic; use real agents only where delivery into an agent can fail. +- Use Claude first without placing Claude-specific concepts in scenarios. +- Use a Linux container for isolated Linux conformance and native lanes for Windows and macOS behavior. ## Motivation -Symposium aims to be a one-stop shop that helps coding agents write great Rust code. Its value does not come from one isolated command. It comes from the interaction between discovery, user consent, skills, hooks, MCP servers, configuration, caching, and the coding agent itself. +Symposium's value comes from the interaction between discovery, consent, configuration, skills, hooks, MCP servers, caches, the user, and the coding agent. Existing integration tests cover much of Symposium's logic, but they do not consistently exercise the complete production boundary. For example, report events currently can reach a hook's stdout before its protocol payload, while an in-process dispatch assertion can still observe the correct event. The accepted `disable` precedence also disagrees with one current enablement path. Neither boundary failure is made obvious by the current integration suite. -The existing integration tests are valuable for deterministic CLI and registry behavior. They cannot answer the larger question: given a realistic Rust task, does a real agent receive and use what Symposium provides, and does the resulting interaction behave as intended? +We want to begin with fixture directories and files, run real commands as a user would, answer interactive prompts, start real agents, and inspect what happened. The harness must make failures reproducible and distinguish a Symposium contract violation from an unavailable runtime or provider failure. -We need a production rehearsal that starts from a controlled fixture, isolates agent and user state from the developer's machine, drives a realistic conversation, and captures enough evidence to explain both success and failure. +This is integration testing, not agent evaluation. Measuring whether Symposium makes agents write better Rust requires baselines and statistical analysis and is outside this RFD. A future evaluation system may reuse these fixtures, adapters, and environments. -## Product contract +## Behavioral contract -The suite distinguishes guarantees from hypotheses. - -Symposium's platform guarantee is that it: +The tests verify that Symposium: - discovers extensions relevant to the current project; - respects trust and explicit user choices; -- delivers enabled skills, hooks, and MCP servers to supported agents; -- behaves consistently across agent adapters where their capabilities overlap; and +- delivers enabled skills, hooks, MCP servers, and subcommands; +- expresses journeys through an adapter-neutral contract; and - makes failures visible and contains their effects. -Our initial outcome hypothesis is that this platform helps an agent produce more correct, idiomatic, and current Rust. Version one checks representative journeys and concrete outcomes. It is not a statistical model benchmark and does not claim that every successful run proves a general improvement over a baseline agent. - -The first audience is an individual Rust developer using an agent in a repository. Organization policy, team-managed configuration, and crate-author publishing journeys are valuable later extensions, but they are not required to validate the first design. - -## Change in a nutshell - -Add a separate `cargo xtask agent-test` suite built around four replaceable boundaries: - -1. a serializable, agent-neutral scenario description; -2. an agent adapter that exposes capabilities and a persistent conversation; -3. an environment backend that runs either on the host or in an isolated container; and -4. graders that inspect exact state, normalized events, and bounded semantic outcomes. - -The suite is additive. Existing fixtures, `TestContext`, simulations, and deterministic tests remain the primary way to cover the combinatorial corners of the registry and CLI. Real-agent journeys cover a smaller set of representative end-to-end interactions. - -## Detailed plans - -### Scenario model - -Scenarios are authored with typed Rust builders at first. The underlying model remains pure data: steps cannot contain arbitrary Rust closures. This keeps scenarios serializable and leaves open a later YAML or TOML representation without committing to a DSL before the vocabulary is understood. - -A scenario contains: - -- a fixture describing project files, registry data, and local services; -- required capabilities such as persistent conversation, PTY input, hooks, or MCP; -- environment and authentication requirements; -- ordered user, CLI, mutation, restart, and checkpoint steps; -- deadlines and resource limits; and -- graders and artifact-retention policy. - -There are two interaction channels: - -- PTY steps drive an interactive CLI by waiting for an observable prompt or state, sending a line or key, and checking exit status. -- Agent turns send a user message to a persistent structured agent session and wait for a protocol completion event. - -Tests never synchronize with fixed sleeps. Every wait targets an observable condition and has a deadline. - -### Agent-neutral adapters - -An `AgentDriver` reports its capabilities, prepares an isolated runtime, starts and stops a persistent session, sends turns, and returns both normalized events and raw provider artifacts. - -Scenarios select capabilities, not brand names. If the selected driver or environment cannot provide a required capability, the result is `Unavailable`, not a misleading test failure. - -Claude is the first adapter because it is already used by Symposium developers. Its structured SDK is used for the main journeys so that completion and tool activity are observable. One narrow PTY smoke test covers the real interactive Claude entry point. Claude-specific protocol details must remain inside the adapter. - -The existing `AgentSession::ClaudeSdk` path starts a new provider query for every prompt. The new adapter must maintain one conversation across turns so that confirmation, follow-up work, and later-session behavior are genuine interactions. - -### Environment backends - -The host backend is optimized for fast local iteration. It creates fresh project, home, configuration, cache, and temporary directories; filters inherited environment variables; and may reuse the developer's local agent authentication. It is useful but not authoritative because the host OS and installed tools can still affect results. - -The container backend is the production-conformance environment. Version one uses Linux containers through Docker, behind an environment abstraction that can later support another container runtime, a VM, or a remote worker. - -Each scenario receives a fresh container, while all turns and deliberate agent restarts within that scenario share its writable state. The fixture is copied into the container rather than mounting the repository. The container runs as a non-root user with a read-only root filesystem, dropped capabilities, no Docker socket, explicit writable directories, and CPU, memory, process, and time limits. - -The container image is layered and cached. A stable base contains the agent runtime and ordinary tools. The current compatible Symposium Linux binary is built once per revision, or supplied explicitly with `--symposium-bin`, and copied into a thin test layer. We do not install Symposium from a package manager: doing so would test a released artifact rather than the code under development and would make the suite depend on registry and network speed. - -This also avoids copying a binary during every scenario. Image preparation is incremental; scenarios start only after the revision-specific layer exists. - -### Network and authentication - -Authoritative container runs use a restricted CI API key. Host runs may reuse local Claude authentication for developer convenience. - -The scenario container has no direct external egress. Provider traffic passes through a controlled forward-proxy sidecar connected to both an internal scenario network and an egress network. The proxy permits only provider endpoints and any endpoints explicitly declared by the scenario. Fixture services and local MCP servers stay on the internal network. - -The CI credential is available only to trusted scheduled or manually dispatched jobs, never to fork pull requests. Version one is experimental and non-blocking while cost, stability, and diagnostic quality are measured. - -### Evidence and grading +Accepted RFDs and reference documentation define expected behavior, even when the implementation currently disagrees. For example, the accepted registry contract says `disable` overrides `use` and `auto-enable`; one current code path does not yet enforce that rule consistently. The coverage table marks this as follow-on direction instead of copying the bug into the expected result or claiming coverage. It becomes `Gap(issue)` only when a linked issue and executable reproducer exist. -The runner writes a canonical, coarse event journal while retaining raw agent and process artifacts. The canonical vocabulary includes events such as process start and exit, prompt observed, input sent, agent turn completed, tool invoked, file changed, configuration changed, and grader completed. +## First journey -Authoritative assertions prefer: - -- exact files and configuration state; -- exact process status and normalized system events; -- protocol-level completion and tool activity; and -- task-specific graders such as compilation, tests, or targeted source inspection. - -Model prose is checked only through narrow, stable semantic anchors when it is itself part of the contract. Full-response snapshots and exact wording are diagnostic, not gating. - -A run has one of four results: - -- `Passed`: the requested journey completed and all graders passed; -- `Failed`: the environment ran correctly but the behavior violated an assertion; -- `InfrastructureError`: setup, credentials, provider access, or the runner failed; -- `Unavailable`: the selected agent or environment lacks a required capability. - -An explicitly requested unavailable run exits unsuccessfully and explains the missing capability. Ordinary `cargo test` is unaffected because this suite is opt-in. - -Artifacts live under `target/agent-tests//`. Every run keeps a compact summary. Failures keep sanitized journals, transcripts, logs, diffs, and relevant final state. Passing runs retain full artifacts only with `--keep-artifacts`. Secrets must be removed before artifacts are persisted. - -### Initial journeys - -The first suite should contain a few high-value vertical journeys: - -1. A fresh agent enters a Rust fixture with a trusted dependency, receives the relevant Symposium guidance, completes a controlled task, and passes Rust-specific graders. -2. An untrusted dependency triggers consent. Separate variants enable and decline it, and a later session honors the recorded choice. -3. A hook affects agent behavior and leaves the expected observable evidence. -4. An MCP-assisted task demonstrates that the configured server is available and useful to the agent. -5. A registry resynchronization is observed, followed by one deliberately broken extension whose failure is visible and contained. - -Combinatorial cases such as every predicate permutation, cache boundary, or malformed registry entry remain in deterministic tests. A real-agent journey is added when the interaction between user, Symposium, and agent is what could fail. - -### Command-line interface - -The proposed entry point is: - -```console -cargo xtask agent-test [OPTIONS] -``` - -Initial options are: +The tracer journey starts with an empty Symposium home, empty agent configuration, and a Rust project whose dependency embeds a plugin awaiting consent. ```text ---list ---agent ---environment ---scenario ---symposium-bin ---keep-artifacts +run cargo agents init --add-agent claude +run cargo agents sync under a PTY +wait for the dependency suggestion +choose Enable +assert the visible prompt, structured events, exit status, config, and files +start a persistent Claude session +assert a fixture capability witness ``` -Running container scenarios without a working runtime, compatible binary, or required credential produces an explicit `Unavailable` result. It must never silently fall back to the host backend. +A separate decline scenario begins from fresh state, selects “No, don't ask again,” restarts the CLI, and proves that the decision persists and the prompt does not return. Ask-later and Escape variants record nothing. -### Time and cost +The same typed scenario model runs through deterministic, native process, Linux-container, and selected real-agent layers. Unsupported combinations are reported explicitly rather than silently weakened. -Containers add image preparation and startup time, but real-agent latency will usually dominate. The runner records cold image preparation, warm environment startup, agent time, and grading time separately so that optimization is based on measurements. +## Design chapters -Fast deterministic tests continue to run on every change. Host journeys are for focused development. A small container suite runs on a trusted schedule or manual dispatch. This layering avoids multiplying expensive agent calls across the full deterministic matrix. +- [Scenario model](./scenario-model/README.md) defines the shared engine, typed linear scenarios, production process boundary, PTY scripting, and explicit time-state fixtures. +- [Agent adapters](./agent-adapters/README.md) defines Claude, ACP, fake adapters, persistent sessions, capability witnesses, permissions, and runtime pinning. +- [Execution environments](./environments/README.md) defines host and container isolation, native OS coverage, binary provenance, initialization, networking, trust, and authentication. +- [Evidence and results](./evidence/README.md) defines dual observation, canonical events, assertions, results, retries, cleanup, and artifact safety. +- [Coverage and CI](./coverage-and-ci/README.md) defines the contract table, coverage obligations, tracer journeys, command interface, budgets, reliability policy, and implementation steps. +- [Proposed guide](./proposed-guide/README.md) shows how developers would list, run, inspect, and author scenarios. -## Frequently asked questions - - - -### Does this replace the current integration tests? - -No. The current tests give faster, deterministic, exhaustive coverage of Symposium's own logic. The new suite adds evidence about real interactions. Existing infrastructure should only change where a small reusable seam makes both suites clearer. The old one-shot real-agent path may be removed after the persistent adapter supersedes it. - -### Why typed Rust scenarios instead of YAML or TOML? +## Key boundaries -Typed builders provide compiler-assisted refactoring, good IDE discovery, and direct reuse of test helpers while the scenario vocabulary is still changing. The cost is that non-Rust contributors cannot edit a data file and scenarios must be recompiled. Keeping the scenario model serializable and closure-free preserves an escape hatch: once the vocabulary stabilizes, a data format can be added as another frontend. +The new engine is additive. Existing fixtures, `TestContext`, simulations, and deterministic tests remain. `cargo test` handles fast coverage; `cargo xtask agent-test` is a thin orchestration frontend for selecting expensive environments and agents. -### Can deterministic conversation checkpoints work with a nondeterministic model? +Authoritative user journeys execute the compiled binary through `cargo agents`; they do not substitute an in-process call. PTY output proves that a user can see and answer a prompt, while a structured side channel and final state prove the underlying decision. -Yes, if the checkpoints target deterministic boundaries. We can exactly check which configuration changed, whether a prompt was answered, which capability became available, what files were produced, and whether the Rust task passes. We should not require the model to emit an exact paragraph. +Scenario fixtures are reviewed repository content. “Untrusted” means awaiting user consent, not hostile code. The container improves reproducibility and least privilege but is not claimed as a sandbox for malicious extensions. -### Why not copy the developer's installed Symposium executable into every test? +Every real-agent journey has a bounded capability witness. General prose and code quality are not graded. Claude is the first production adapter. Fake and ACP conformance are required follow-ups before the provisional driver interface can be called stable. -The installed executable may not match the checkout and copying it per scenario wastes time. The default authoritative path builds or accepts one compatible Linux binary and caches it in a thin image layer. An explicit `--symposium-bin` remains useful for testing a known artifact. +## Scope and milestones -### Why are containers not the only backend? +This RFD is implemented when the tracer is proven: the consent journey works through real native processes, a parsed PTY, structured evidence, a fresh Linux container, and a persistent-Claude witness, with useful failure artifacts and measured runtime. -The host backend shortens the edit-test-debug loop and can use local authentication. The container backend answers the stronger production-conformance question. Treating them as implementations of one environment interface keeps scenarios portable without pretending that host isolation is complete. +Broader catalog automation, consent branches, cross-platform process lanes, fake and ACP conformance, hook and MCP witnesses, and trusted release CI are follow-on direction rather than acceptance criteria for this RFD. They require tracked issues or follow-on RFDs after the tracer informs the interfaces. See [Coverage and CI](./coverage-and-ci/README.md#milestones-and-follow-on-direction). -### What do the linked CLI testing projects contribute? - -`cli-testing-library` demonstrates a useful interaction model: wait for observable output, query the screen, send user events, and avoid hand-written timing. Its Node implementation and reported platform constraints make it a reference rather than a foundation for this Rust, cross-platform suite. - -`cli-testing-specialist` is oriented toward generic, generated CLI validation. Our journeys need persistent agents, Symposium-specific state, hooks, MCP, consent, and outcome graders, so adopting it would not remove the hard integration work. - -### How will we know whether Symposium caused an idiomatic Rust result? - -Version one proves delivery and checks bounded task outcomes: the relevant extension was selected, the agent could use it, and the fixture satisfies concrete Rust graders. Strong causal claims require repeated paired runs against a no-Symposium baseline and statistical analysis. That is a future evaluation layer, not a prerequisite for integration testing. +## Frequently asked questions -## Implementation plan +### Does this replace the current integration tests? -1. Introduce the scenario, capability, event, artifact, and result types with fake drivers and runner tests. -2. Add the host backend and a persistent Claude adapter. -3. Add the PTY driver for interactive Symposium and one Claude entry-point smoke test. -4. Add the Linux container backend, revision-layered Symposium binary, proxy, isolation rules, and infrastructure diagnostics. -5. Implement the trusted-dependency guidance journey as the first production rehearsal. -6. Add consent, hook, MCP, resynchronization, and contained-failure journeys. -7. Add the trusted scheduled/manual CI workflow, document operation and cost, correct stale design documentation, and retire superseded one-shot agent-test code. +No. It reuses them and adds missing process, PTY, isolation, observation, and persistent-agent seams. A real-agent call is added only when activation inside the agent is the behavior under test. +### Why not adopt one of the linked CLI testing projects? +`cli-testing-library` provides a useful screen-query and user-event model, which this design borrows. Its Node implementation and platform constraints are not a good foundation for this Rust, cross-platform harness. `cli-testing-specialist` targets generic generated CLI validation and does not provide Symposium-specific state, consent, hooks, MCP, or persistent-agent behavior. ## Implementation status -This RFD describes proposed experimental infrastructure. Implementation has not begun. - -See [Proposed: Agent interaction tests](./proposed-agent-interaction-testing.md) for the intended operator workflow. \ No newline at end of file +This RFD describes proposed experimental infrastructure. Implementation has not begun, and the tracer milestone has not been reached. diff --git a/md/rfds/agent-interaction-testing/agent-adapters/README.md b/md/rfds/agent-interaction-testing/agent-adapters/README.md new file mode 100644 index 00000000..c6ec245d --- /dev/null +++ b/md/rfds/agent-interaction-testing/agent-adapters/README.md @@ -0,0 +1,51 @@ +# Agent adapters + +## Driver contract + +An `AgentDriver`: + +- reports its capabilities and supported witness forms; +- prepares only agent-specific runtime and authentication state; +- starts and stops a persistent session; +- sends turns and waits for protocol completion; +- reports input, cache-read, cache-write, and output usage for every provider request; +- enforces supported output and operation limits and responds to runner cancellation; +- applies scenario-declared permission policy; and +- returns normalized events plus sanitized raw provider artifacts. + +The driver interface is capability-based. A scenario requiring an unsupported capability or witness is `Unavailable` for that adapter rather than weakened to a filesystem-only check. An adapter without trustworthy usage accounting cannot run in scheduled paid CI. + +## Initial adapters + +Claude is the first production adapter because Symposium developers already use it. Main journeys use its structured SDK so completion and tool activity are observable. One narrow PTY smoke test covers the interactive Claude entry point. + +The existing `AgentSession::ClaudeSdk` path starts a fresh query for each prompt. Its replacement maintains one conversation across turns so that confirmation, restarts, and follow-up behavior are genuine interactions. + +The tracer implements Claude behind a provisional capability-based interface. It does not claim cross-agent behavioral consistency from one production adapter. + +Before the interface is declared stable, a follow-up adds deterministic fake adapters for success, failure, timeout, malformed-event, and missing-capability paths. The existing persistent ACP path then becomes a second adapter and is contract-tested against a fixture ACP agent. Only Claude belongs to this RFD's scheduled real-agent tracer. + +## Capability witnesses + +Every real-agent journey defines bounded evidence that an installed capability crossed into the running agent: + +- A skill witness exposes a scenario nonce through a structured load or tool event, with a narrow exact response as fallback. +- A hook witness is the corresponding hook trace. +- An MCP witness is the fixture server's initialization, tool-list, or explicitly requested tool-call log. +- A subcommand witness is the observed subprocess invocation and exit status. + +Capability witnesses do not grade general prose, code quality, or whether Symposium makes an agent write better Rust. Those are effectiveness-evaluation questions outside this RFD. + +## Permissions + +Every real-agent scenario declares allowed read and write roots, executable commands, MCP tools, and non-provider network access. Unexpected requests are denied and fail the scenario. Adapters normalize permission requests and outcomes into the event journal and may not automatically select an arbitrary approval option. + +The runner also verifies that the agent did not write outside allowed roots or leave undeclared processes. A denial scenario names the forbidden operation and expected result explicitly. Witnesses use the least powerful operation available. + +## Runtime reproducibility + +Pinned conformance runs fix the agent CLI or SDK version, dependency lock, base-image digest, and model identifier where the provider supports one. Results record requested and provider-returned model metadata, agent version, Symposium revision, and scenario version. + +A follow-up may add a smaller latest-agent canary for current supported releases. Canary failures indicate upstream compatibility work and do not rewrite pinned conformance results. Advancing a pin is a reviewed compatibility change. + +Provider behavior cannot always be frozen completely. Assertions therefore remain limited to stable protocol boundaries and capability witnesses. diff --git a/md/rfds/agent-interaction-testing/coverage-and-ci/README.md b/md/rfds/agent-interaction-testing/coverage-and-ci/README.md new file mode 100644 index 00000000..1a2fa6b0 --- /dev/null +++ b/md/rfds/agent-interaction-testing/coverage-and-ci/README.md @@ -0,0 +1,155 @@ +# Coverage and CI + +## Contract table + +The tracer begins with a reviewed Markdown table of the Symposium promises it exercises. Each row has a stable rule identifier, a behavioral statement, a link to the accepted specification, its required test layers, and one state: + +- `Committed(step)`: this RFD commits to implementing the row in the named tracer step. It becomes `Covered` after the required scenarios pass. +- `Covered`: every required tracer scenario exists and passes. +- `Gap(issue)`: the implementation is known to violate the specification, a linked issue owns the discrepancy, and an executable reproducer returns `Failed` when run directly. The reproducer reports on its schedule but is excluded from the release gate. +- `Direction(follow-up)`: the rule is outside this RFD's tracer commitment and must be carried into a closing follow-up issue or RFD. It is not counted as tracer coverage. + +Accepted RFDs and current reference documentation remain authoritative. The table must not copy an implementation bug into the expected result. Adding a gap, removing its reproducer, or increasing the gap count requires review. + +Typed Rust scenarios name the rules they prove. After enough journeys exist to expose stable catalog requirements, a follow-up may make the table machine-readable, validate layer and operating-system obligations, and generate a coverage report. This RFD does not build that meta-tool before the first journey. + +## Coverage layers + +Every `Covered` contract rule has deterministic coverage. Covered user-visible branches have real-process scenarios. A PTY is required only when the production command is interactive; hooks and other noninteractive subprocesses use piped stdin, stdout, and stderr. Covered agent-delivery mechanisms have selected real-agent witnesses. The matrix below records intended obligations, including follow-on direction; it does not claim those rows are implemented by the tracer. Linux containers rehearse representative production paths rather than duplicating the deterministic matrix. + +| Rule ID | Contract | State | Deterministic | Real process | Real agent | +|---|---|---|---:|---:|---:| +| `consent.accept` | Undecided candidate is accepted | `Committed(steps 1, 4)` | required | required, PTY | one delivery smoke | +| `consent.decline` | Undecided candidate is declined | `Committed(step 1)` | required | required, PTY | not required | +| `consent.defer` | Ask later records nothing | `Direction(follow-up)` | required | required, PTY | not required | +| `cli.noninteractive` | Noninteractive execution never prompts | `Direction(follow-up)` | required | required, pipes | not required | +| `enablement.disable-precedence` | Disable overrides other enablement | `Direction(follow-up)` | required | representative, pipes | not required | +| `cache.expiration` | Cache expiration reevaluates its input | `Direction(follow-up)` | required | required, pipes | not required | +| `hook.stdout-protocol` | Hook stdout contains only protocol output | `Committed(step 1)` | required | required, pipes | not required | +| `isolation.skill-inventory` | Isolated custom-skill inventory exactly matches the fixture | `Committed(steps 1, 3)` | required | required, pipes | required | +| `use.search-endpoint` | Non-workspace `use` search uses only its declared fixture endpoint | `Direction(follow-up)` | required | required, pipes | not required | +| `delivery.hook` | Hook delivery reaches an agent | `Direction(follow-up)` | required | representative, pipes | required | +| `delivery.mcp` | MCP registration reaches an agent | `Direction(follow-up)` | required | representative, pipes | required | + +Operating-system applicability is recorded separately. A Linux-container pass cannot satisfy a Windows-native or macOS-native requirement. + +The first audit must include conflicting enablement entries. The accepted registry contract says `disable` wins over `use` and `auto-enable`, even though one current implementation path checks `use` first. + +## Tracer journeys + +This RFD commits to two fresh-state consent journeys. The accepted branch runs real `init` and `sync`, answers the prompt, verifies installation, and obtains a capability witness from a persistent Claude session. The declined branch verifies that nothing is installed, the decision persists, and a later sync does not ask again. + +The tracer also adds the deterministic hook-stdout regression because the structured side channel must not contaminate an agent protocol. It plants an isolation canary and verifies the exact custom-skill inventory owned by the fixture. + +Ask-later and Escape, custom predicates, cache reuse and expiration, malformed registries, enablement precedence, non-workspace `use`, hook delivery, MCP delivery, and registry resynchronization remain stated follow-on families. They use deterministic or real-process coverage by default. A real agent is added only where delivery into the agent could fail. + +## Command interface + +The orchestration entry point is: + +```console +cargo xtask agent-test [OPTIONS] +``` + +Initial options are: + +```text +--list +--agent +--environment +--scenario ... +--symposium-bin +--auth +--max-agent-turns +--max-input-tokens +--max-output-tokens +--max-tool-calls +--keep-artifacts +``` + +`--scenario` is repeatable. No scenario means “print the execution plan,” not “start an agent.” A selection containing a real-agent journey requires an explicit agent. Missing runtime, credentials, or capability yields `Unavailable`; a requested container never silently falls back to the host. + +The plan reports CLI-only and real-agent scenarios, scenario and operator token limits, maximum turns and tool calls, remaining daily and monthly capacity, environment, binary provenance, and pinned runtime before execution. + +## Cost and runtime controls + +Each real-agent scenario declares maximum cumulative input, cache-read, cache-write, and output tokens; provider requests; user turns; tool calls; real-time deadline; and usage class. Scheduled paid execution requires trustworthy provider accounting. Cached tokens remain visible and count toward token limits even when their billable price is lower. + +Scenario-owned limits define the product contract. Exceeding one is `Failed`. Operator flags and run-wide limits are protective ceilings. If a lower `--max-agent-turns` or run-wide cap stops an otherwise valid scenario, the result is `InfrastructureError` owned by `runner.budget`, never a Symposium failure. The execution plan shows both limits and their effective minimum before paid work begins. + +Before any provider request, the runner estimates harness-controlled prompt, fixture, skill, and tool context and rejects an oversized request as `InfrastructureError` owned by `runner.budget`. The adapter accounts for agent-owned context that can only be measured by the provider. A changed agent or model pin returns to manual calibration rather than inheriting the previous allowance. + +The runner reserves the complete scenario ceiling from daily and monthly token ledgers before starting paid work. If either ledger lacks capacity, no provider request is made. Actual usage is charged after the run and unused capacity is released. A retry requires a second complete reservation; a billable first attempt is never retried when the remaining ledger cannot cover it. + +CI uses a dedicated restricted provider key with a $5 monthly provider-side spending limit as the final backstop. Real-agent concurrency begins at one. Any follow-up latest-agent canary receives a smaller budget than pinned conformance. Credentials alone never enable paid tests; ordinary `cargo test` retains its explicit agent-testing gate. + +Runtime reporting separates checkout build or image preparation, warm environment startup, agent execution, and assertion/evidence processing. This makes container overhead distinguishable from provider latency. + +The tracer begins with five manually triggered calibration runs. Its provisional guard permits one user turn, at most four provider requests, three tool calls, 25,000 total input-side tokens across base input, cache reads, and cache writes, and 1,000 cumulative output tokens. It reserves 30,000 total tokens per day and 750,000 per month, allowing at most one paid run per day and 25 per month. If the pinned runtime cannot produce the nonce witness within that guard, the prompt, tools, fixture, and context are reduced before any limit is raised. + +The initial scheduled limit is the maximum observed calibration usage plus 20 percent, never more than the provisional guard. After 20 eligible runs, it may be reviewed against P95 usage plus 20 percent. Limits never rise automatically after an agent upgrade or unusual run. + +At the standard Sonnet base price of $3 per million input tokens and $15 per million output tokens, the provisional base-token estimate is approximately $0.09 per run. The runner allows up to $0.20 per run for cache-price differences while the provider key caps the month at $5. Actual cost should be lower after calibration. The estimate is recalculated when the model pin or [provider pricing](https://www.anthropic.com/news/claude-sonnet-5) changes; tokens remain the primary limit and dollars are derived reporting. + +## CI lanes in this RFD + +- Fast deterministic tests block every pull request. +- Stable native agent-free process and PTY scenarios may become PR-blocking. +- A small agent-free Linux-container suite may graduate if its measured runtime is acceptable. +- Real-agent tracer journeys run only in trusted scheduled or manual jobs during this RFD. + +The nightly cadence produces the first 20 eligible samples in about 20 days when infrastructure is available. + +## Post-tracer release policy + +Assertion and Symposium failures are never retried to improve a product result. Each scheduled execution is an independent sample from fresh state. A real-agent journey becomes release-gating only after at least 19 of its latest 20 eligible scheduled executions pass for the same scenario contract and pinned adapter/runtime manifest. At least one eligible execution in that window must exercise the release-candidate revision. `InfrastructureError`, `Unavailable`, and `non-authoritative` runs do not enter the product pass-rate denominator; their rates are reported separately and they cannot satisfy the release-candidate requirement. + +Falling below the target triggers a quarantine decision. Until reviewed, the release gate remains blocked. An approved quarantine requires a linked issue, continues running and reporting the journey, and excludes it from the gate. Adding or extending a quarantine requires review. A release gate evaluates the rolling target, not whether someone manually reran the latest failure. + +Agent-free tests graduate after an observation period with no unexplained flakes, acceptable runtime, actionable failure artifacts, reliable cleanup, and consistently successful secret-canary validation. Provider credentials are never exposed to fork pull requests. + +## Milestones and follow-on direction + +### Tracer proven + +The dependency-consent journey reuses the current fixture infrastructure and runs real `init` and `sync` processes. Accept and decline work through a parsed PTY, structured events agree with final state, the scenario runs in a fresh Linux container, and the accepted branch produces a persistent-Claude capability witness. Failure artifacts, cleanup, cost, and phase timing are demonstrated. + +This validates the architecture and completes this RFD. + +### Post-tracer direction + +After the tracer, tracked follow-ups can expand the contract table across registry, discovery, predicate, cache, and delivery behavior. They can add every consent branch, representative Linux-container and native Windows/macOS lanes, fake and fixture-ACP adapter contracts, and selected hook and MCP witnesses. + +Catalog automation, full release reporting, a latest-agent canary, and broader CI graduation are separate commitments informed by tracer evidence. They are not acceptance criteria for this RFD. + +## Implementation plan + +### Step 1: Run the first black-box host journey + +From an empty user configuration, run compiled `cargo agents init --add-agent ` and `cargo agents sync` processes under a PTY for one dependency candidate. Add only the scenario steps, side-channel events, terminal anchors, assertions, and artifacts required for initialization plus accept and decline. + +Verify both branches against terminal output, structured events, exit status, configuration, filesystem state, the host-state canary, and the exact fixture-controlled custom-skill inventory. Add the deterministic assertion that invoking a hook through pipes writes only valid protocol output to stdout. + +### Step 2: Record the tracer contracts + +Write the initial Markdown contract table from the behavior exercised by step 1. A discrepancy becomes `Gap(issue)` only when it has a product issue and executable reproducer; otherwise it remains follow-on direction. Do not add catalog code generation or validation. + +Verify manually that every `Covered` row names an executable scenario and every `Gap(issue)` row names both an issue and a reproducer. + +### Step 3: Isolate the journey in Linux + +Add Docker execution, the content-addressed Symposium binary, fixture services, least-privilege rules, and infrastructure diagnostics. Run the existing consent scenario unchanged. + +Verify cold preparation, warm startup, the same host-state canary and custom-skill inventory assertions used by the host backend, and parity with the remaining host assertions. + +### Step 4: Add the first real-agent witness + +Add the persistent Claude adapter and extend the accepted branch with a fixture-skill witness. Pin its runtime and retain one interactive Claude smoke test. + +Verify the capability nonce, persistent session, installation and hook-registration evidence, redaction, and error classification. + +Record scheduled outcomes without automatic assertion retries. The tracer remains non-gating while it accumulates reliability evidence. + +Before closing the RFD, correct `md/design/running-tests.md` so it documents the `SYMPOSIUM_ENABLE_AGENT_TESTING` gate. Keep `TestMode::AgentOnly`, `test-agents.toml`, and `tests/agent_harness/run_scenario.py` temporarily for existing Claude and ACP coverage, but mark that path as superseded and add no new scenarios to it. File its removal with the ACP follow-up, after remaining scenarios migrate. + +Also file follow-up issues or RFDs for catalog automation, fake and ACP conformance, remaining scenario families, native operating-system expansion, and release CI graduation. diff --git a/md/rfds/agent-interaction-testing/environments/README.md b/md/rfds/agent-interaction-testing/environments/README.md new file mode 100644 index 00000000..6825ccf5 --- /dev/null +++ b/md/rfds/agent-interaction-testing/environments/README.md @@ -0,0 +1,62 @@ +# Execution environments + +## Environment backends + +The host backend is for fast local iteration. It creates fresh project, home, configuration, cache, and temporary directories and passes an explicit filtered environment to every child process. Host results remain non-authoritative because installed tools and the operating system can influence them. + +The Linux container backend is the isolated conformance environment. The tracer uses Docker behind an interface that can later support another container runtime, a VM, or a remote worker. Linux-container results are authoritative only for Linux. + +Windows and macOS use native deterministic and real-process/PTY CI lanes with fresh test directories. These lanes cover platform-specific paths, command dispatch, shell behavior, PTYs, permissions, and process handling without claiming container-strength isolation. Native real-agent smokes may be added when trusted runners and credentials are available. + +## Scenario isolation + +Every scenario and retry attempt receives unique workspace, home, cache, network, container, service, trace, and artifact identifiers. A fresh container is created per scenario, while deliberate CLI and agent restarts within that scenario retain its writable state. + +Fixtures are copied into the environment rather than mounting the repository. Containers run as non-root with a read-only root filesystem, dropped capabilities, no Docker socket, explicit writable directories, and CPU, memory, process, and time limits. + +Scenarios never mutate the runner's process-global environment. Concurrent scenarios share only immutable or content-addressed build assets. Image and binary preparation use an interprocess lock and publish a completed read-only artifact. Services use isolated networks and dynamic host ports. Resources are labelled by run and scenario, and cleanup is idempotent. + +Every host and container run tests host-state exclusion. The runner places a harmless synthetic capability canary in a harness-owned decoy host configuration that is outside the fresh scenario home and, for local-auth tests, exercises the adapter's filtered credential bridge. It never writes a canary into the developer's real home. The runner asserts that the canary is absent from copied homes, agent-visible custom capabilities, and persisted artifacts. + +The runner also asserts that the custom-skill inventory in every harness-controlled user and project scope exactly matches the fixture, including before every real-agent session. Agent-provided built-in capabilities are recorded separately and are not claimed as host state. + +Agent concurrency is capped separately from CLI-only concurrency. A scenario may request an exclusive resource only for an external tool that genuinely cannot be isolated. + +## Symposium binary provenance + +By default, the runner builds one Linux `cargo-agents` artifact from the current checkout and reuses it across selected container scenarios. Its content-addressed key includes: + +- source revision or dirty-source digest; +- `Cargo.lock` digest; +- Rust toolchain; +- target triple; +- profile and feature set; and +- container base-image digest. + +Dependency and intermediate build layers are reused, so preparation is incremental rather than a per-scenario copy pause. + +`--symposium-bin` is an explicit override, never an automatic PATH lookup. The runner checks its executable format, operating system, architecture, and available version metadata. Results record the binary digest and provenance as a checkout build or explicit override. Host runs apply the same checks to local artifacts. + +## Provisioning and initialization + +Placing the checkout binary on the isolated PATH is harness preparation, not an installer assertion. The tracer does not test `cargo install`, `cargo binstall`, release archives, or a future package manager. + +Symposium initialization is product behavior. Fresh-user scenarios run the real `cargo agents init` process and assert configuration creation, agent setup, hook installation, optional choices, repeated-init idempotency, and later hook activation. Distribution paths can later feed the same post-provisioning scenarios. + +## Network and fixture trust + +Tracer fixture content is repository-owned and reviewed. An extension marked untrusted is awaiting Symposium consent; it is not arbitrary hostile code. Testing malicious hooks, MCP servers, agents, or fixtures requires a separate security-testing design. + +Scenario runtime is hermetic except for the selected real-agent provider. Rust projects use path dependencies or a controlled local registry. Registry, git-source, MCP, hook, predicate, and failure scenarios use local fixture processes or pinned image tools. A `cargo agents use` scenario that searches for a non-workspace crate must route `CargoPm::search` to a declared local fixture endpoint; it never queries crates.io. Fixture services can return exact versions, delays, disconnects, malformed data, and cache validators. + +The scenario container has no direct external egress. HTTPS provider traffic crosses a CONNECT proxy with an allowlist of destination hosts and ports. The proxy does not terminate TLS and the harness installs no interception certificate authority. Local registries, MCP servers, and other fixture services remain on the internal network. Any additional external endpoint must be declared in the scenario and recorded in the execution plan and manifest. + +Image construction and dependency acquisition finish before scenario execution and are identified by lockfiles and digests. Network failures are simulated through fixture services rather than public outages. + +## Authentication + +Container conformance uses a restricted API key available only to trusted jobs and never mounts local agent state. + +Host authentication is explicit: `--auth api-key` or `--auth local`. API-key mode uses a fresh agent home. Local mode bridges only the minimum adapter-supported credential material, read-only, while agent settings, Symposium configuration, skills, hooks, MCP configuration, caches, and conversations remain fresh. + +If an agent cannot separate credentials from user configuration, the base result carries the `non-authoritative(contaminated-auth-context)` modifier. The manifest records the mode and inherited credential paths without their contents. diff --git a/md/rfds/agent-interaction-testing/evidence/README.md b/md/rfds/agent-interaction-testing/evidence/README.md new file mode 100644 index 00000000..e6f93881 --- /dev/null +++ b/md/rfds/agent-interaction-testing/evidence/README.md @@ -0,0 +1,85 @@ +# Evidence and results + +## Dual observation + +Interactive journeys observe two channels from the same process: + +- The normal PTY proves that a suggestion or prompt was visible and accepted real user input. +- A harness-controlled JSONL side channel reports stable Symposium decisions and state transitions. + +The side channel attaches an additional reporting sink. It must not enable quiet mode, bypass confirmation, change command decisions, or write into a hook's protocol stdout. Events needed by the initial scenarios include discovery, confirmation requested and answered, enablement, installation, and hook dispatch. Later scenarios may add predicate and cache events without changing the product-facing stream. + +Terminal assertions use only the stable text necessary to prove that a user could understand and answer the prompt. Colors, wrapping, and complete screens are not ordinary snapshots. Detailed behavioral assertions use structured events and final state. + +## Canonical event journal + +The runner merges harness events, Symposium side-channel events, agent protocol events, and observed state changes into one coarse journal while retaining sanitized source artifacts. + +Each event envelope contains: + +- schema version; +- run, scenario, and attempt identifiers; +- source and source-local sequence; +- correlated operation identifier; +- event kind; +- real monotonic offset; and +- normalized payload. + +Provider-operation events record requested limits and reported input, cache-read, cache-write, and output tokens. Aggregate token counts and derived cost are evidence, not estimates substituted for missing accounting. + +Sequence is strict within one source. Receipt order is diagnostic and does not imply causal order across processes. Assertions express partial order within a source or correlated operation, such as discovery before confirmation and confirmation before installation. Unrelated sources remain unordered unless explicitly correlated. + +Dynamic paths, process IDs, ports, and timestamps are normalized before comparison. Unknown additive event kinds are retained and ignored unless required. Breaking envelope changes increment the schema version. Event payloads use stable identifiers and exclude secrets. + +## Assertions + +Authoritative assertions prefer: + +- exact configuration and allowed filesystem state; +- process exit status and normalized system events; +- protocol completion and tool activity; +- hook stdout containing only the selected agent's protocol output; +- discovery, consent, predicate, and cache decisions; and +- hook, skill, MCP, and subcommand capability witnesses. + +Model prose is checked only through a narrow fixture-defined nonce or fact when that is the available witness. Full responses are diagnostic, not gating. + +## Results and failure ownership + +A run has four results: + +- `Passed`: the requested journey and assertions completed. +- `Failed`: the environment ran, but Symposium or the interaction violated the contract. +- `InfrastructureError`: credentials, provider, runtime, environment, or harness failed. +- `Unavailable`: preflight found that the selected adapter or environment lacks a required capability. + +A result may also carry modifiers that preserve important qualifications without creating another base result: + +- `non-authoritative(contaminated-auth-context)` means local credentials could not be separated from user or agent configuration. +- `stability-warning(recovered-infrastructure-error)` means a recognized infrastructure failure occurred before the complete fresh-state retry passed. + +Modifiers are recorded in the summary, journal, and aggregate reports. They never turn `Failed` into `Passed` or make a non-authoritative run satisfy a conformance or release requirement. + +Explicitly requesting an unavailable combination exits unsuccessfully; ordinary `cargo test` remains unaffected. There is no expected-failure scenario result. A known product-gap reproducer still returns `Failed` when run directly. + +Suite aggregation consults the coverage table separately. A `Gap(issue)` reproducer runs on its reporting schedule but is excluded from the release gate and listed as a known gap. Adding a gap, removing its reproducer, or increasing the gap count requires review. Covered scenarios retain their ordinary gating behavior. + +Failures name an owning phase such as `environment.prepare`, `symposium.cli`, `symposium.state`, `agent.start`, `agent.turn`, `fixture.mcp`, `assertion`, or `cleanup`. A Symposium crash, missing prompt, wrong state, or completed agent turn without its required witness is `Failed`. + +Exceeding a scenario-owned token or operation limit is `Failed` because the witness did not fit its contract. Harness-controlled context that already exceeds the declared limit, exhaustion of a daily or monthly token ledger, or an operator ceiling stopping the run is `InfrastructureError` owned by `runner.budget`. Missing trustworthy provider accounting makes scheduled paid execution `Unavailable`. + +## Retries and cleanup + +Assertion and Symposium failures are never retried automatically. A recognized transient infrastructure error may retry the complete scenario once with fresh state. Individual steps are never replayed inside an existing container or conversation. + +Both attempts are preserved. A recovered run remains `Passed` with the `stability-warning(recovered-infrastructure-error)` modifier and attempt metadata, so infrastructure reliability still counts the transient. + +On a deadline, the runner captures current evidence, attempts graceful termination, kills the complete process tree after a bounded cleanup deadline, and verifies that no process or container remains. + +## Artifact safety + +Artifacts live under `target/agent-tests//`. Every run keeps a compact summary. Failures keep sanitized journals, terminal output, selected logs, workspace diffs, agent events, explicitly allowed Symposium state, and a redaction report. Successful runs keep rich artifacts only with `--keep-artifacts`. + +Capture is allowlist-based. The runner never archives a complete container, home, authentication directory, or process environment. Credentials are secret handles supplied only to the process that needs them. Known values, provider headers, credential-bearing URLs, command arguments, and environment fields are redacted in memory before persistence. + +Every run injects harmless secret canaries and verifies that none survive. If sanitization cannot complete, rich artifacts are withheld and the summary reports the redaction failure. Upload-time filtering is not considered sufficient. diff --git a/md/rfds/agent-interaction-testing/proposed-agent-interaction-testing.md b/md/rfds/agent-interaction-testing/proposed-agent-interaction-testing.md deleted file mode 100644 index 386af031..00000000 --- a/md/rfds/agent-interaction-testing/proposed-agent-interaction-testing.md +++ /dev/null @@ -1,80 +0,0 @@ -# Agent interaction tests - -Agent interaction tests exercise Symposium together with a real coding agent in a controlled project. They complement the ordinary test suite: use ordinary tests for exhaustive CLI and registry logic, and use these journeys when the interaction among the user, Symposium, and the agent is the behavior under test. - -The feature is experimental. Real-agent runs consume provider capacity and may expose provider variability, so they are opt-in and separate from `cargo test`. - -## Discovering scenarios - -List the scenarios and their required capabilities: - -```console -cargo xtask agent-test --list -``` - -The output identifies whether the selected agent and environment can run each scenario. A missing capability is reported as unavailable rather than being mistaken for a product failure. - -## Running locally - -For a fast development run, use the host environment: - -```console -cargo xtask agent-test --agent claude --environment host --scenario trusted-dependency-guidance -``` - -The host runner creates isolated project, home, configuration, cache, and temporary directories. It may use the agent authentication already present on the machine. Host results are useful for debugging but are not authoritative because installed tools and the operating system can still influence the run. - -## Running a production-conformance journey - -Use the container environment for the authoritative Linux rehearsal: - -```console -$env:ANTHROPIC_API_KEY = "..." -cargo xtask agent-test --agent claude --environment container --scenario trusted-dependency-guidance -``` - -The runner prepares a cached base image and a thin layer containing the Symposium binary for the current revision. It then creates a fresh, restricted container for the scenario. All turns and deliberate restarts in that scenario share its state. - -To test an already-built compatible Linux artifact, select it explicitly: - -```console -cargo xtask agent-test --agent claude --environment container --symposium-bin ./artifacts/symposium-linux-x86_64 --scenario trusted-dependency-guidance -``` - -The runner never silently substitutes a released package or falls back from a requested container to the host. - -## Reading a result - -Each run ends as one of: - -* `Passed` — the journey and its graders succeeded; -* `Failed` — the environment worked, but observed behavior violated the scenario; -* `InfrastructureError` — setup, authentication, provider access, or the runner failed; -* `Unavailable` — a requested driver or environment lacks a required capability. - -The console summary names the failed step and points to `target/agent-tests//`. Failure artifacts include a sanitized event journal, relevant logs, the conversation transcript, file diffs, and final inspected state. Use `--keep-artifacts` to retain the same detail after a successful run: - -```console -cargo xtask agent-test --agent claude --environment container --scenario consent-enable --keep-artifacts -``` - -## Writing a scenario - -Scenarios are initially written with typed Rust builders, but contain portable data rather than arbitrary closures. A typical scenario describes this sequence: - -1. Create a Rust fixture whose dependency has a trusted Symposium extension. -2. Start a persistent agent session in the fixture. -3. Ask the agent to implement a small, controlled Rust task. -4. Wait for protocol completion rather than sleeping for a guessed duration. -5. Assert that Symposium selected and delivered the extension. -6. Inspect the resulting files and run targeted Rust checks. - -Check exact state at deterministic boundaries. For example, check that consent was recorded, a hook event occurred, an MCP tool was invoked, or `cargo test` passed. Do not snapshot an entire model answer or require incidental wording. - -A scenario declares capabilities such as `persistent-conversation`, `pty-input`, `hooks`, and `mcp`. It does not contain Claude-specific branching. Agent-specific behavior belongs in the adapter. - -## CI operation - -Container journeys run only in a trusted scheduled or manually dispatched workflow. The workflow uses a restricted spending credential and does not expose it to fork pull requests. While the suite is experimental, failures are visible but do not block ordinary pull requests. - -When investigating runtime, compare the recorded phases separately: image preparation, environment startup, agent execution, and grading. A slow provider turn should not be diagnosed as slow container startup. diff --git a/md/rfds/agent-interaction-testing/proposed-guide/README.md b/md/rfds/agent-interaction-testing/proposed-guide/README.md new file mode 100644 index 00000000..cecb4546 --- /dev/null +++ b/md/rfds/agent-interaction-testing/proposed-guide/README.md @@ -0,0 +1,117 @@ +# Agent interaction tests + +Agent interaction tests exercise Symposium with scripted users, real processes, and selected real coding agents. They complement ordinary tests: use deterministic tests for exhaustive Symposium logic and these journeys when the process, terminal, user, or agent boundary is itself under test. + +The feature is experimental. Real-agent runs consume provider capacity and are opt-in. + +## Discover scenarios + +```console +cargo xtask agent-test --list +``` + +The scenario list reports required agent, environment, operating-system, and witness capabilities. The RFD's contract table maps the tracer's Symposium promises to executable scenarios and linked product gaps. + +Running `cargo xtask agent-test` without a scenario prints an execution plan and does not start an agent. + +Repeat `--scenario` to select more than one journey: + +```console +cargo xtask agent-test --agent claude --environment container --auth api-key --scenario dependency-consent-accept --scenario dependency-consent-decline +``` + +## Run on the host + +```console +cargo xtask agent-test --agent claude --environment host --auth local --scenario dependency-consent-accept +``` + +The host runner creates fresh project, Symposium, agent, cache, and temporary directories. Local authentication is used only when explicitly requested. If the adapter cannot separate credentials from normal agent configuration, the result is marked non-authoritative. + +Host runs are useful for debugging but may still be affected by installed tools and the operating system. + +## Run Linux conformance + +```console +$env:ANTHROPIC_API_KEY = "..." +cargo xtask agent-test --agent claude --environment container --auth api-key --scenario dependency-consent-accept +``` + +The runner prepares a pinned base image and one content-addressed Linux `cargo-agents` build from the checkout. Each scenario receives a fresh restricted container. Scenario runtime is hermetic except for the selected agent provider. + +To test an existing compatible Linux artifact: + +```console +cargo xtask agent-test --agent claude --environment container --auth api-key --symposium-bin ./artifacts/cargo-agents-linux-x86_64 --scenario dependency-consent-accept +``` + +The override is checked for operating system, architecture, executable format, and available version metadata. The runner never substitutes a released package, PATH binary, host environment, or different execution backend silently. + +## Read the execution plan + +Before a paid run, the plan reports information such as: + +```text +Selected scenarios: 2 +CLI-only scenarios: 1 +Real-agent scenarios: 1 +Maximum agent turns: 1 +Maximum provider requests: 4 +Maximum tool calls: 3 +Input-side token guard: 25,000 +Output-token guard: 1,000 +Daily tokens remaining: 30,000 +Monthly tokens remaining: 750,000 +Per-run cost allowance: $0.20 +Monthly provider cap: $5.00 +Environment: Linux container +Agent/runtime: Claude, pinned +``` + +Real-agent scenarios enforce cumulative input, cache-read, cache-write, and output tokens as well as provider-request, turn, tool-call, deadline, and run-wide limits. Cached tokens still count even when they cost less. + +The runner reserves the complete scenario ceiling from its daily and monthly ledgers before contacting the provider. If there is not enough capacity, the run does not start. A retry needs a separate reservation. The initial tracer permits at most one paid run per day and 25 per month. Its base-token estimate is approximately $0.09 per run, its allowance including cache-price differences is $0.20, and its dedicated provider key has a $5 monthly cap. Measured calibration usage lowers the scheduled token limit; it never rises automatically. + +## Read a result + +Each run ends as: + +- `Passed`: the journey and assertions succeeded. +- `Failed`: the environment worked, but the behavior violated the contract. +- `InfrastructureError`: setup, authentication, provider, runtime, harness, or an operator-imposed budget stopped the run. +- `Unavailable`: the selected combination lacks a required capability. + +Results may carry modifiers. `non-authoritative(contaminated-auth-context)` means local authentication could not be isolated from agent configuration. `stability-warning(recovered-infrastructure-error)` means a complete fresh-state retry recovered from a recognized infrastructure failure. A modifier cannot turn a product failure into a pass or satisfy a conformance requirement with non-authoritative evidence. + +A scenario that cannot produce its witness within its own token budget is `Failed`. Oversized harness context, exhausted daily or monthly capacity, or a lower operator limit is `InfrastructureError` owned by `runner.budget`. Scheduled paid execution is `Unavailable` when the adapter cannot report trustworthy usage. + +The summary also names the owning phase. Only a recognized transient infrastructure error may retry the entire scenario once with fresh state. Product failures and individual steps are never retried. A known-gap reproducer still returns `Failed` when run directly; scheduled reports identify it separately from release-gating covered scenarios. + +Artifacts are under `target/agent-tests//`. Failure artifacts contain only allowlisted, sanitized evidence and a redaction report. Complete homes, authentication directories, and process environments are never archived. Use `--keep-artifacts` to retain rich evidence for a passing run. + +## Write a scenario + +Scenarios use typed Rust builders but contain portable data rather than arbitrary closures. Every behavioral branch is a separate linear scenario with fresh state. + +A typical consent journey: + +1. Compose a Rust fixture whose dependency embeds a plugin awaiting consent. +2. Start with empty Symposium and agent configuration. +3. Run real `cargo agents init --add-agent ` and assert setup. +4. Run real `cargo agents sync` under a parsed PTY. +5. Select the intended prompt option with explicit keys. +6. Assert terminal anchors, structured events, exit status, and final state. +7. Start a persistent agent session when delivery is under test. +8. Assert a narrow capability witness such as a fixture nonce, hook trace, or MCP server log. + +Scenarios declare contract IDs, required capabilities, permissions, scenario-owned token and operation budgets, and external endpoints. They do not contain Claude-specific paths or judge general response quality. An operator-supplied lower budget is shown separately and cannot manufacture a Symposium failure. + +Time-dependent scenarios mutate controlled persisted inputs instead of sleeping or changing the production clock. They may set a cache expiry into the past, write a fixture `state.toml`, set a file mtime, or disable the sync debounce. Process and agent deadlines always use real monotonic time. + +## CI operation + +Deterministic tests block pull requests. Stable agent-free PTY and small Linux-container scenarios may graduate after meeting runtime and reliability criteria. Real-agent tracer journeys run in trusted scheduled or manual jobs and begin as non-gating observations. + +### Future release gating + +A real-agent journey can become release-gating after at least 19 of its latest 20 eligible scheduled runs pass for the same pinned manifest. Assertion failures are not retried. Falling below the target requires a reviewed quarantine decision with a linked issue; quarantined journeys continue to run and report. diff --git a/md/rfds/agent-interaction-testing/scenario-model/README.md b/md/rfds/agent-interaction-testing/scenario-model/README.md new file mode 100644 index 00000000..69e67f55 --- /dev/null +++ b/md/rfds/agent-interaction-testing/scenario-model/README.md @@ -0,0 +1,62 @@ +# Scenario model + +## Shared engine + +Agent interaction tests extend `symposium-testlib`; they do not create a second fixture or assertion system. Ordinary tests and `cargo xtask agent-test` use the same fixture composition, scenario model, event vocabulary, and assertions. + +`cargo test` remains the frontend for deterministic and selected host scenarios. Xtask is a thin orchestration frontend for environment selection, credentials, containers, filtering, real-agent execution, and artifact retention. + +## Scenario data + +Scenarios are authored with typed Rust builders. The underlying model contains portable data rather than arbitrary closures, so it can be serialized and may later gain a TOML or YAML frontend. + +A scenario declares: + +- fixture layers and controlled services; +- required environment and agent capabilities; +- ordered CLI, user, agent, mutation, restart, and checkpoint steps; +- a capability witness for every real-agent interaction; +- deadlines, resource limits, and scenario-owned agent budgets; +- a least-privilege permission policy; +- contract rule identifiers; and +- assertions and artifact-retention policy. + +Scenarios select capabilities, not agent brands. Agent-specific paths, authentication fields, event types, and witness mechanisms remain in adapters. + +Agent token budgets are cumulative across every provider request made for the journey, not merely the number of user-visible turns. They bound input, cache-read, cache-write, and output tokens separately, plus provider requests and tool calls. Cached tokens still count toward the token budget even when their dollar price is lower. + +## State and branching + +Scenarios are linear. Accept, decline, ask-later, and Escape are separate scenarios that may reuse fixture descriptions but never writable state. + +Every scenario and retry begins with a fresh workspace, user configuration, agent configuration, cache, services, and conversation. Steps within one scenario share state deliberately, including across declared process or agent restarts. Persistence and cache scenarios express repeated commands in that one journey because preserved state is what they test. + +Scenario logic does not branch around unexpected output. A missing or different checkpoint fails at that step. + +## Production process boundary + +Authoritative user journeys invoke the compiled `cargo-agents` executable through the production-facing `cargo agents` command. Interactive commands run under a PTY. The runner captures the rendered terminal, sanitized raw bytes, exit status, structured events, and resulting state. + +The runner must not silently replace a requested process or PTY step with an in-process call. Existing deterministic tests may continue calling Symposium's Rust entry points directly, but that path does not prove Cargo dispatch, PATH setup, terminal interaction, hook subprocesses, or process exit behavior. + +## PTY scripting + +The PTY driver parses ANSI output into a rendered screen instead of treating the stream as plain stdout. Waits query narrow anchors in that screen. Input steps represent lines and explicit keys such as Enter, Escape, arrows, EOF, and interrupt. + +Ordinary scenarios use a fixed terminal size, UTF-8 locale, declared TERM and color mode, and the native PTY backend for the operating system, including ConPTY or equivalent on Windows. The terminal profile and backend are recorded with the result. + +Screen normalization handles cursor movement, redraws, color, and newline differences. Raw sanitized bytes remain diagnostic evidence. A small rendering suite separately tests color and resizing; ordinary journeys do not snapshot the complete screen. + +This adopts the useful interaction model from `cli-testing-library`: wait for what a user can see, then send user input, without adopting its Node implementation. + +## Time-dependent scenarios + +Tests never synchronize with fixed sleeps. Every wait targets an observable condition and has a real monotonic deadline. + +This RFD does not add a production clock seam. Time-dependent tests mutate controlled persisted inputs before starting the command that observes them. Predicate-cache tests set the persisted expiry into the past, update-throttle tests set `state.toml`, filesystem tests set the relevant mtime, and sync tests may use `sync-debounce-secs = 0`. + +These mutations test the production comparison against the real wall clock without waiting for time to pass. If a later contract cannot be tested this way, its clock abstraction requires a separate design. Container, agent, TLS, provider, and process deadlines always use real time. + +## Why typed Rust instead of a scenario DSL? + +Typed builders provide compiler-assisted refactoring, IDE discovery, and direct reuse of test helpers while the vocabulary is evolving. The tradeoff is recompilation and a higher contribution barrier for non-Rust authors. Keeping the model serializable and closure-free preserves the option to add a data-file frontend after the vocabulary stabilizes. From 4d9f2f65edffc1a50fcb30de0be3c7d13030cf09 Mon Sep 17 00:00:00 2001 From: Awesome Rustacean Date: Sat, 22 Aug 2026 01:57:10 +0300 Subject: [PATCH 3/4] docs(rfd): refine agent interaction testing design --- md/rfds/agent-interaction-testing/README.md | 16 ++--- .../agent-adapters/README.md | 11 ++- .../coverage-and-ci/README.md | 67 +++++++++---------- .../environments/README.md | 8 +-- .../evidence/README.md | 10 +-- .../proposed-guide/README.md | 30 ++++----- .../scenario-model/README.md | 35 +++++++--- 7 files changed, 91 insertions(+), 86 deletions(-) diff --git a/md/rfds/agent-interaction-testing/README.md b/md/rfds/agent-interaction-testing/README.md index 32c051bd..0e894a16 100644 --- a/md/rfds/agent-interaction-testing/README.md +++ b/md/rfds/agent-interaction-testing/README.md @@ -39,21 +39,21 @@ run cargo agents sync under a PTY wait for the dependency suggestion choose Enable assert the visible prompt, structured events, exit status, config, and files -start a persistent Claude session +run one bounded query through the Claude adapter assert a fixture capability witness ``` A separate decline scenario begins from fresh state, selects “No, don't ask again,” restarts the CLI, and proves that the decision persists and the prompt does not return. Ask-later and Escape variants record nothing. -The same typed scenario model runs through deterministic, native process, Linux-container, and selected real-agent layers. Unsupported combinations are reported explicitly rather than silently weakened. +The same registered scenario runs through in-process, native-process, Linux-container, and selected real-agent layers. Unsupported combinations are reported explicitly rather than silently weakened. ## Design chapters -- [Scenario model](./scenario-model/README.md) defines the shared engine, typed linear scenarios, production process boundary, PTY scripting, and explicit time-state fixtures. -- [Agent adapters](./agent-adapters/README.md) defines Claude, ACP, fake adapters, persistent sessions, capability witnesses, permissions, and runtime pinning. +- [Scenario model](./scenario-model/README.md) defines declarative registration metadata, imperative Rust bodies, the production process boundary, PTY scripting, and explicit time-state fixtures. +- [Agent adapters](./agent-adapters/README.md) defines Claude, ACP and fake follow-ups, bounded queries, capability witnesses, permissions, and runtime pinning. - [Execution environments](./environments/README.md) defines host and container isolation, native OS coverage, binary provenance, initialization, networking, trust, and authentication. - [Evidence and results](./evidence/README.md) defines dual observation, canonical events, assertions, results, retries, cleanup, and artifact safety. -- [Coverage and CI](./coverage-and-ci/README.md) defines the contract table, coverage obligations, tracer journeys, command interface, budgets, reliability policy, and implementation steps. +- [Coverage and CI](./coverage-and-ci/README.md) defines the contract table, coverage obligations, tracer journeys, command interface, cost controls, and implementation steps. - [Proposed guide](./proposed-guide/README.md) shows how developers would list, run, inspect, and author scenarios. ## Key boundaries @@ -68,7 +68,7 @@ Every real-agent journey has a bounded capability witness. General prose and cod ## Scope and milestones -This RFD is implemented when the tracer is proven: the consent journey works through real native processes, a parsed PTY, structured evidence, a fresh Linux container, and a persistent-Claude witness, with useful failure artifacts and measured runtime. +This RFD is implemented when the tracer is proven: the consent journey works through real native processes, a parsed PTY, structured evidence, a fresh Linux container, and one bounded Claude capability witness, with useful failure artifacts and measured runtime. Broader catalog automation, consent branches, cross-platform process lanes, fake and ACP conformance, hook and MCP witnesses, and trusted release CI are follow-on direction rather than acceptance criteria for this RFD. They require tracked issues or follow-on RFDs after the tracer informs the interfaces. See [Coverage and CI](./coverage-and-ci/README.md#milestones-and-follow-on-direction). @@ -76,11 +76,11 @@ Broader catalog automation, consent branches, cross-platform process lanes, fake ### Does this replace the current integration tests? -No. It reuses them and adds missing process, PTY, isolation, observation, and persistent-agent seams. A real-agent call is added only when activation inside the agent is the behavior under test. +No. It reuses them and adds missing process, PTY, isolation, observation, and agent-delivery seams. A real-agent query is added only when activation inside the agent is the behavior under test. ### Why not adopt one of the linked CLI testing projects? -`cli-testing-library` provides a useful screen-query and user-event model, which this design borrows. Its Node implementation and platform constraints are not a good foundation for this Rust, cross-platform harness. `cli-testing-specialist` targets generic generated CLI validation and does not provide Symposium-specific state, consent, hooks, MCP, or persistent-agent behavior. +`cli-testing-library` provides a useful screen-query and user-event model, which this design borrows. Its Node implementation and platform constraints are not a good foundation for this Rust, cross-platform harness. `cli-testing-specialist` targets generic generated CLI validation and does not provide Symposium-specific state, consent, hooks, MCP, or agent-delivery behavior. ## Implementation status diff --git a/md/rfds/agent-interaction-testing/agent-adapters/README.md b/md/rfds/agent-interaction-testing/agent-adapters/README.md index c6ec245d..a277bbf2 100644 --- a/md/rfds/agent-interaction-testing/agent-adapters/README.md +++ b/md/rfds/agent-interaction-testing/agent-adapters/README.md @@ -6,24 +6,23 @@ An `AgentDriver`: - reports its capabilities and supported witness forms; - prepares only agent-specific runtime and authentication state; -- starts and stops a persistent session; -- sends turns and waits for protocol completion; +- runs a bounded query and waits for protocol completion; - reports input, cache-read, cache-write, and output usage for every provider request; - enforces supported output and operation limits and responds to runner cancellation; - applies scenario-declared permission policy; and - returns normalized events plus sanitized raw provider artifacts. -The driver interface is capability-based. A scenario requiring an unsupported capability or witness is `Unavailable` for that adapter rather than weakened to a filesystem-only check. An adapter without trustworthy usage accounting cannot run in scheduled paid CI. +The driver interface is capability-based. A scenario requiring an unsupported capability or witness is `Unavailable` for that adapter rather than weakened to a filesystem-only check. An adapter without trustworthy usage accounting cannot run a paid tracer query. ## Initial adapters -Claude is the first production adapter because Symposium developers already use it. Main journeys use its structured SDK so completion and tool activity are observable. One narrow PTY smoke test covers the interactive Claude entry point. +Claude is the first production adapter because Symposium developers already use it. The tracer uses its structured SDK so completion, tool activity, and provider usage are observable without paying for a second interactive-entry-point smoke test. -The existing `AgentSession::ClaudeSdk` path starts a fresh query for each prompt. Its replacement maintains one conversation across turns so that confirmation, restarts, and follow-up behavior are genuine interactions. +The tracer needs one fresh, bounded Claude query to prove delivery of the fixture capability. The provisional driver therefore does not introduce persistent-conversation machinery. A later scenario that genuinely depends on multiple turns or an agent restart must add that capability deliberately and test it before the driver contract grows to include it. The tracer implements Claude behind a provisional capability-based interface. It does not claim cross-agent behavioral consistency from one production adapter. -Before the interface is declared stable, a follow-up adds deterministic fake adapters for success, failure, timeout, malformed-event, and missing-capability paths. The existing persistent ACP path then becomes a second adapter and is contract-tested against a fixture ACP agent. Only Claude belongs to this RFD's scheduled real-agent tracer. +Before the interface is declared stable, a follow-up adds deterministic fake adapters for success, failure, timeout, malformed-event, and missing-capability paths. The existing persistent ACP path can then inform a separately tested session capability and a second adapter. Only the bounded Claude query belongs to this RFD's real-agent tracer. ## Capability witnesses diff --git a/md/rfds/agent-interaction-testing/coverage-and-ci/README.md b/md/rfds/agent-interaction-testing/coverage-and-ci/README.md index 1a2fa6b0..917dcaeb 100644 --- a/md/rfds/agent-interaction-testing/coverage-and-ci/README.md +++ b/md/rfds/agent-interaction-testing/coverage-and-ci/README.md @@ -6,27 +6,30 @@ The tracer begins with a reviewed Markdown table of the Symposium promises it ex - `Committed(step)`: this RFD commits to implementing the row in the named tracer step. It becomes `Covered` after the required scenarios pass. - `Covered`: every required tracer scenario exists and passes. -- `Gap(issue)`: the implementation is known to violate the specification, a linked issue owns the discrepancy, and an executable reproducer returns `Failed` when run directly. The reproducer reports on its schedule but is excluded from the release gate. +- `Gap(issue)`: the implementation is known to violate the specification, a linked issue owns the discrepancy, and an executable reproducer returns `Failed` when run directly. - `Direction(follow-up)`: the rule is outside this RFD's tracer commitment and must be carried into a closing follow-up issue or RFD. It is not counted as tracer coverage. -Accepted RFDs and current reference documentation remain authoritative. The table must not copy an implementation bug into the expected result. Adding a gap, removing its reproducer, or increasing the gap count requires review. +Accepted RFDs and current reference documentation remain authoritative. The table must not copy an implementation bug into the expected result. Typed Rust scenarios name the rules they prove. After enough journeys exist to expose stable catalog requirements, a follow-up may make the table machine-readable, validate layer and operating-system obligations, and generate a coverage report. This RFD does not build that meta-tool before the first journey. ## Coverage layers -Every `Covered` contract rule has deterministic coverage. Covered user-visible branches have real-process scenarios. A PTY is required only when the production command is interactive; hooks and other noninteractive subprocesses use piped stdin, stdout, and stderr. Covered agent-delivery mechanisms have selected real-agent witnesses. The matrix below records intended obligations, including follow-on direction; it does not claim those rows are implemented by the tracer. Linux containers rehearse representative production paths rather than duplicating the deterministic matrix. +The matrix separates the boundary being exercised from the command used to run the test. In-process tests call Symposium's Rust entry points. Real-process tests spawn the compiled binary, using a PTY only for interactive commands and pipes for hooks and other noninteractive subprocesses. Real-agent tests are reserved for delivery that can fail only inside the agent. A deterministic black-box process regression may still run under `cargo test`; it does not become an agent test merely because it spawns a binary. -| Rule ID | Contract | State | Deterministic | Real process | Real agent | +The matrix records intended obligations, including follow-on direction; it does not claim those rows are implemented by the tracer. Linux containers rehearse representative production paths rather than duplicating every in-process test. + +| Rule ID | Contract | State | In-process | Real process | Real agent | |---|---|---|---:|---:|---:| -| `consent.accept` | Undecided candidate is accepted | `Committed(steps 1, 4)` | required | required, PTY | one delivery smoke | -| `consent.decline` | Undecided candidate is declined | `Committed(step 1)` | required | required, PTY | not required | +| `consent.accept` | Undecided candidate is accepted | `Committed(steps 1, 3)` | required | required, PTY | not required | +| `consent.decline` | Undecided candidate is declined | `Committed(steps 1, 3)` | required | required, PTY | not required | | `consent.defer` | Ask later records nothing | `Direction(follow-up)` | required | required, PTY | not required | | `cli.noninteractive` | Noninteractive execution never prompts | `Direction(follow-up)` | required | required, pipes | not required | | `enablement.disable-precedence` | Disable overrides other enablement | `Direction(follow-up)` | required | representative, pipes | not required | | `cache.expiration` | Cache expiration reevaluates its input | `Direction(follow-up)` | required | required, pipes | not required | -| `hook.stdout-protocol` | Hook stdout contains only protocol output | `Committed(step 1)` | required | required, pipes | not required | -| `isolation.skill-inventory` | Isolated custom-skill inventory exactly matches the fixture | `Committed(steps 1, 3)` | required | required, pipes | required | +| `hook.stdout-protocol` | Hook stdout contains only protocol output | `Committed(step 1)` | not sufficient | required, pipes | not required | +| `isolation.skill-inventory` | Isolated custom-skill inventory exactly matches the fixture | `Committed(steps 1, 3, 4)` | not required | required, pipes | required | +| `delivery.skill` | An enabled fixture skill reaches the selected agent | `Committed(step 4)` | not sufficient | required, pipes | one delivery smoke | | `use.search-endpoint` | Non-workspace `use` search uses only its declared fixture endpoint | `Direction(follow-up)` | required | required, pipes | not required | | `delivery.hook` | Hook delivery reaches an agent | `Direction(follow-up)` | required | representative, pipes | required | | `delivery.mcp` | MCP registration reaches an agent | `Direction(follow-up)` | required | representative, pipes | required | @@ -37,9 +40,9 @@ The first audit must include conflicting enablement entries. The accepted regist ## Tracer journeys -This RFD commits to two fresh-state consent journeys. The accepted branch runs real `init` and `sync`, answers the prompt, verifies installation, and obtains a capability witness from a persistent Claude session. The declined branch verifies that nothing is installed, the decision persists, and a later sync does not ask again. +This RFD commits to two fresh-state consent journeys. The accepted branch runs real `init` and `sync`, answers the prompt, and verifies installation. Step 4 extends that branch with one bounded Claude query that proves the nonce-bearing fixture skill reached the agent. The declined branch verifies that nothing is installed, the decision persists, and a later sync does not ask again. -The tracer also adds the deterministic hook-stdout regression because the structured side channel must not contaminate an agent protocol. It plants an isolation canary and verifies the exact custom-skill inventory owned by the fixture. +The tracer also fixes the hook stdout contamination bug and adds a deterministic black-box process regression proving that stdout contains only the hook protocol. It plants an isolation canary and verifies the exact custom-skill inventory owned by the fixture. Ask-later and Escape, custom predicates, cache reuse and expiration, malformed registries, enablement precedence, non-workspace `use`, hook delivery, MCP delivery, and registry resynchronization remain stated follow-on families. They use deterministic or real-process coverage by default. A real agent is added only where delivery into the agent could fail. @@ -61,58 +64,50 @@ Initial options are: --symposium-bin --auth --max-agent-turns +--max-provider-requests --max-input-tokens --max-output-tokens --max-tool-calls +--confirm-paid-run --keep-artifacts ``` -`--scenario` is repeatable. No scenario means “print the execution plan,” not “start an agent.” A selection containing a real-agent journey requires an explicit agent. Missing runtime, credentials, or capability yields `Unavailable`; a requested container never silently falls back to the host. +`--scenario` is repeatable. No scenario means “print the execution plan,” not “start an agent.” A selection containing a real-agent journey requires an explicit agent and `--confirm-paid-run`. Missing runtime, credentials, or capability yields `Unavailable`; a requested container never silently falls back to the host. -The plan reports CLI-only and real-agent scenarios, scenario and operator token limits, maximum turns and tool calls, remaining daily and monthly capacity, environment, binary provenance, and pinned runtime before execution. +The plan reports CLI-only and real-agent scenarios, scenario and operator token limits, maximum turns, provider requests, and tool calls, the provider-side spending cap, environment, binary provenance, and pinned runtime before execution. ## Cost and runtime controls -Each real-agent scenario declares maximum cumulative input, cache-read, cache-write, and output tokens; provider requests; user turns; tool calls; real-time deadline; and usage class. Scheduled paid execution requires trustworthy provider accounting. Cached tokens remain visible and count toward token limits even when their billable price is lower. +Each real-agent scenario declares maximum cumulative input, cache-read, cache-write, and output tokens; provider requests; user turns; tool calls; real-time deadline; and usage class. Paid execution requires trustworthy provider accounting. Cached tokens remain visible and count toward token limits even when their billable price is lower. Scenario-owned limits define the product contract. Exceeding one is `Failed`. Operator flags and run-wide limits are protective ceilings. If a lower `--max-agent-turns` or run-wide cap stops an otherwise valid scenario, the result is `InfrastructureError` owned by `runner.budget`, never a Symposium failure. The execution plan shows both limits and their effective minimum before paid work begins. -Before any provider request, the runner estimates harness-controlled prompt, fixture, skill, and tool context and rejects an oversized request as `InfrastructureError` owned by `runner.budget`. The adapter accounts for agent-owned context that can only be measured by the provider. A changed agent or model pin returns to manual calibration rather than inheriting the previous allowance. - -The runner reserves the complete scenario ceiling from daily and monthly token ledgers before starting paid work. If either ledger lacks capacity, no provider request is made. Actual usage is charged after the run and unused capacity is released. A retry requires a second complete reservation; a billable first attempt is never retried when the remaining ledger cannot cover it. +Before any provider request, the runner checks the declared fixture and prompt inputs it controls. The adapter reports actual provider usage for agent-owned context. A scenario that cannot fit its declared limit is reduced before the limit is raised. -CI uses a dedicated restricted provider key with a $5 monthly provider-side spending limit as the final backstop. Real-agent concurrency begins at one. Any follow-up latest-agent canary receives a smaller budget than pinned conformance. Credentials alone never enable paid tests; ordinary `cargo test` retains its explicit agent-testing gate. +Authoritative paid tracer runs use a dedicated restricted provider key with a $5 monthly provider-side spending limit as the aggregate backstop. A host run using `--auth local` is non-authoritative, cannot claim that provider cap, and still obeys the scenario's hard token and operation limits. Real-agent concurrency is one. Credentials alone never enable paid tests: the user must select a real-agent scenario, name the agent, and pass `--confirm-paid-run`. Ordinary `cargo test` retains its explicit agent-testing gate. Runtime reporting separates checkout build or image preparation, warm environment startup, agent execution, and assertion/evidence processing. This makes container overhead distinguishable from provider latency. -The tracer begins with five manually triggered calibration runs. Its provisional guard permits one user turn, at most four provider requests, three tool calls, 25,000 total input-side tokens across base input, cache reads, and cache writes, and 1,000 cumulative output tokens. It reserves 30,000 total tokens per day and 750,000 per month, allowing at most one paid run per day and 25 per month. If the pinned runtime cannot produce the nonce witness within that guard, the prompt, tools, fixture, and context are reduced before any limit is raised. +The tracer's provisional guard permits one user turn, at most four provider requests, three tool calls, 25,000 total input-side tokens across base input, cache reads, and cache writes, and 1,000 cumulative output tokens. Initial manual runs record actual usage so these limits can be reduced. If the pinned runtime cannot produce the nonce witness within the guard, the prompt, tools, fixture, and context are reduced before any limit is raised. -The initial scheduled limit is the maximum observed calibration usage plus 20 percent, never more than the provisional guard. After 20 eligible runs, it may be reviewed against P95 usage plus 20 percent. Limits never rise automatically after an agent upgrade or unusual run. - -At the standard Sonnet base price of $3 per million input tokens and $15 per million output tokens, the provisional base-token estimate is approximately $0.09 per run. The runner allows up to $0.20 per run for cache-price differences while the provider key caps the month at $5. Actual cost should be lower after calibration. The estimate is recalculated when the model pin or [provider pricing](https://www.anthropic.com/news/claude-sonnet-5) changes; tokens remain the primary limit and dollars are derived reporting. +At the standard Sonnet base price of $3 per million input tokens and $15 per million output tokens, the provisional base-token estimate is approximately $0.09 per run. The runner reports a conservative allowance of $0.20 per run for cache-price differences while the provider key caps the month at $5. The estimate is recalculated when the model pin or [provider pricing](https://www.anthropic.com/news/claude-sonnet-5) changes; tokens remain the primary limit and dollars are derived reporting. ## CI lanes in this RFD - Fast deterministic tests block every pull request. - Stable native agent-free process and PTY scenarios may become PR-blocking. - A small agent-free Linux-container suite may graduate if its measured runtime is acceptable. -- Real-agent tracer journeys run only in trusted scheduled or manual jobs during this RFD. - -The nightly cadence produces the first 20 eligible samples in about 20 days when infrastructure is available. - -## Post-tracer release policy - -Assertion and Symposium failures are never retried to improve a product result. Each scheduled execution is an independent sample from fresh state. A real-agent journey becomes release-gating only after at least 19 of its latest 20 eligible scheduled executions pass for the same scenario contract and pinned adapter/runtime manifest. At least one eligible execution in that window must exercise the release-candidate revision. `InfrastructureError`, `Unavailable`, and `non-authoritative` runs do not enter the product pass-rate denominator; their rates are reported separately and they cannot satisfy the release-candidate requirement. - -Falling below the target triggers a quarantine decision. Until reviewed, the release gate remains blocked. An approved quarantine requires a linked issue, continues running and reporting the journey, and excludes it from the gate. Adding or extending a quarantine requires review. A release gate evaluates the rolling target, not whether someone manually reran the latest failure. +- Real-agent tracer journeys are explicitly selected, manually run, and non-gating during this RFD. Agent-free tests graduate after an observation period with no unexplained flakes, acceptable runtime, actionable failure artifacts, reliable cleanup, and consistently successful secret-canary validation. Provider credentials are never exposed to fork pull requests. +This RFD does not define scheduled ownership, quarantine, pass-rate, or release-gating policy for real-agent tests. A follow-up may propose those mechanisms using measured tracer reliability, runtime, and cost rather than assumptions. + ## Milestones and follow-on direction ### Tracer proven -The dependency-consent journey reuses the current fixture infrastructure and runs real `init` and `sync` processes. Accept and decline work through a parsed PTY, structured events agree with final state, the scenario runs in a fresh Linux container, and the accepted branch produces a persistent-Claude capability witness. Failure artifacts, cleanup, cost, and phase timing are demonstrated. +The dependency-consent journey reuses the current fixture infrastructure and runs real `init` and `sync` processes. Accept and decline work through a parsed PTY, structured events agree with final state, the scenario runs in a fresh Linux container, and the accepted branch produces a bounded Claude capability witness. Failure artifacts, cleanup, cost, and phase timing are demonstrated. This validates the architecture and completes this RFD. @@ -128,7 +123,7 @@ Catalog automation, full release reporting, a latest-agent canary, and broader C From an empty user configuration, run compiled `cargo agents init --add-agent ` and `cargo agents sync` processes under a PTY for one dependency candidate. Add only the scenario steps, side-channel events, terminal anchors, assertions, and artifacts required for initialization plus accept and decline. -Verify both branches against terminal output, structured events, exit status, configuration, filesystem state, the host-state canary, and the exact fixture-controlled custom-skill inventory. Add the deterministic assertion that invoking a hook through pipes writes only valid protocol output to stdout. +Verify both branches against terminal output, structured events, exit status, configuration, filesystem state, the host-state canary, and the exact fixture-controlled custom-skill inventory. Fix the hook stdout contamination bug and add a black-box regression that invokes the compiled hook through pipes and proves stdout is valid protocol output only. ### Step 2: Record the tracer contracts @@ -138,17 +133,17 @@ Verify manually that every `Covered` row names an executable scenario and every ### Step 3: Isolate the journey in Linux -Add Docker execution, the content-addressed Symposium binary, fixture services, least-privilege rules, and infrastructure diagnostics. Run the existing consent scenario unchanged. +Add Docker execution, the content-addressed Symposium binary, least-privilege rules, networking disabled, and infrastructure diagnostics. Run the existing consent scenario unchanged. Do not add provider egress or general fixture-service infrastructure in this step. Verify cold preparation, warm startup, the same host-state canary and custom-skill inventory assertions used by the host backend, and parity with the remaining host assertions. ### Step 4: Add the first real-agent witness -Add the persistent Claude adapter and extend the accepted branch with a fixture-skill witness. Pin its runtime and retain one interactive Claude smoke test. +Add the bounded Claude adapter and extend the container-backed accepted branch with one fixture-skill query. Pin its runtime and add only the allowlisted provider egress and restricted API-key handling this query requires. -Verify the capability nonce, persistent session, installation and hook-registration evidence, redaction, and error classification. +Verify the capability nonce, exact pre-query custom-skill inventory, installation and hook-registration evidence, usage limits, redaction, and error classification. -Record scheduled outcomes without automatic assertion retries. The tracer remains non-gating while it accumulates reliability evidence. +Record the manually invoked result without automatic paid retries. The tracer remains non-gating. Before closing the RFD, correct `md/design/running-tests.md` so it documents the `SYMPOSIUM_ENABLE_AGENT_TESTING` gate. Keep `TestMode::AgentOnly`, `test-agents.toml`, and `tests/agent_harness/run_scenario.py` temporarily for existing Claude and ACP coverage, but mark that path as superseded and add no new scenarios to it. File its removal with the ACP follow-up, after remaining scenarios migrate. diff --git a/md/rfds/agent-interaction-testing/environments/README.md b/md/rfds/agent-interaction-testing/environments/README.md index 6825ccf5..a6256842 100644 --- a/md/rfds/agent-interaction-testing/environments/README.md +++ b/md/rfds/agent-interaction-testing/environments/README.md @@ -18,7 +18,7 @@ Scenarios never mutate the runner's process-global environment. Concurrent scena Every host and container run tests host-state exclusion. The runner places a harmless synthetic capability canary in a harness-owned decoy host configuration that is outside the fresh scenario home and, for local-auth tests, exercises the adapter's filtered credential bridge. It never writes a canary into the developer's real home. The runner asserts that the canary is absent from copied homes, agent-visible custom capabilities, and persisted artifacts. -The runner also asserts that the custom-skill inventory in every harness-controlled user and project scope exactly matches the fixture, including before every real-agent session. Agent-provided built-in capabilities are recorded separately and are not claimed as host state. +The runner also asserts that the custom-skill inventory in every harness-controlled user and project scope exactly matches the fixture, including before every real-agent query. Agent-provided built-in capabilities are recorded separately and are not claimed as host state. Agent concurrency is capped separately from CLI-only concurrency. A scenario may request an exclusive resource only for an external tool that genuinely cannot be isolated. @@ -47,9 +47,9 @@ Symposium initialization is product behavior. Fresh-user scenarios run the real Tracer fixture content is repository-owned and reviewed. An extension marked untrusted is awaiting Symposium consent; it is not arbitrary hostile code. Testing malicious hooks, MCP servers, agents, or fixtures requires a separate security-testing design. -Scenario runtime is hermetic except for the selected real-agent provider. Rust projects use path dependencies or a controlled local registry. Registry, git-source, MCP, hook, predicate, and failure scenarios use local fixture processes or pinned image tools. A `cargo agents use` scenario that searches for a non-workspace crate must route `CargoPm::search` to a declared local fixture endpoint; it never queries crates.io. Fixture services can return exact versions, delays, disconnects, malformed data, and cache validators. +The agent-free tracer container runs with networking disabled. Rust projects use path dependencies or a controlled local registry prepared before scenario execution. Later registry, git-source, MCP, hook, predicate, and failure scenarios may add declared local fixture processes or pinned image tools. A `cargo agents use` scenario that searches for a non-workspace crate must route `CargoPm::search` to a declared local fixture endpoint; it never queries crates.io. Fixture services can return exact versions, delays, disconnects, malformed data, and cache validators. -The scenario container has no direct external egress. HTTPS provider traffic crosses a CONNECT proxy with an allowlist of destination hosts and ports. The proxy does not terminate TLS and the harness installs no interception certificate authority. Local registries, MCP servers, and other fixture services remain on the internal network. Any additional external endpoint must be declared in the scenario and recorded in the execution plan and manifest. +Step 4 adds only the provider egress needed by the real-agent query. The scenario container has no direct external egress. HTTPS provider traffic crosses a CONNECT proxy with an allowlist of destination hosts and ports. The proxy does not terminate TLS and the harness installs no interception certificate authority. Any fixture service remains on the internal network. Any additional external endpoint must be declared in the scenario and recorded in the execution plan and manifest. Image construction and dependency acquisition finish before scenario execution and are identified by lockfiles and digests. Network failures are simulated through fixture services rather than public outages. @@ -57,6 +57,6 @@ Image construction and dependency acquisition finish before scenario execution a Container conformance uses a restricted API key available only to trusted jobs and never mounts local agent state. -Host authentication is explicit: `--auth api-key` or `--auth local`. API-key mode uses a fresh agent home. Local mode bridges only the minimum adapter-supported credential material, read-only, while agent settings, Symposium configuration, skills, hooks, MCP configuration, caches, and conversations remain fresh. +Host authentication is explicit: `--auth api-key` or `--auth local`. API-key mode uses a fresh agent home. Local mode bridges only the minimum adapter-supported credential material, read-only, while agent settings, Symposium configuration, skills, hooks, MCP configuration, caches, and query history remain fresh. If an agent cannot separate credentials from user configuration, the base result carries the `non-authoritative(contaminated-auth-context)` modifier. The manifest records the mode and inherited credential paths without their contents. diff --git a/md/rfds/agent-interaction-testing/evidence/README.md b/md/rfds/agent-interaction-testing/evidence/README.md index e6f93881..7c9b9f07 100644 --- a/md/rfds/agent-interaction-testing/evidence/README.md +++ b/md/rfds/agent-interaction-testing/evidence/README.md @@ -58,19 +58,19 @@ A result may also carry modifiers that preserve important qualifications without - `non-authoritative(contaminated-auth-context)` means local credentials could not be separated from user or agent configuration. - `stability-warning(recovered-infrastructure-error)` means a recognized infrastructure failure occurred before the complete fresh-state retry passed. -Modifiers are recorded in the summary, journal, and aggregate reports. They never turn `Failed` into `Passed` or make a non-authoritative run satisfy a conformance or release requirement. +Modifiers are recorded in the summary, journal, and aggregate reports. They never turn `Failed` into `Passed` or make a non-authoritative run satisfy a conformance requirement. Explicitly requesting an unavailable combination exits unsuccessfully; ordinary `cargo test` remains unaffected. There is no expected-failure scenario result. A known product-gap reproducer still returns `Failed` when run directly. -Suite aggregation consults the coverage table separately. A `Gap(issue)` reproducer runs on its reporting schedule but is excluded from the release gate and listed as a known gap. Adding a gap, removing its reproducer, or increasing the gap count requires review. Covered scenarios retain their ordinary gating behavior. +The coverage table records a `Gap(issue)` separately from completed tracer coverage. Its executable reproducer still returns `Failed`; this RFD does not add expected-failure results or release-gate policy. -Failures name an owning phase such as `environment.prepare`, `symposium.cli`, `symposium.state`, `agent.start`, `agent.turn`, `fixture.mcp`, `assertion`, or `cleanup`. A Symposium crash, missing prompt, wrong state, or completed agent turn without its required witness is `Failed`. +Failures name an owning phase such as `environment.prepare`, `symposium.cli`, `symposium.state`, `agent.start`, `agent.query`, `fixture.mcp`, `assertion`, or `cleanup`. A Symposium crash, missing prompt, wrong state, or completed agent query without its required witness is `Failed`. -Exceeding a scenario-owned token or operation limit is `Failed` because the witness did not fit its contract. Harness-controlled context that already exceeds the declared limit, exhaustion of a daily or monthly token ledger, or an operator ceiling stopping the run is `InfrastructureError` owned by `runner.budget`. Missing trustworthy provider accounting makes scheduled paid execution `Unavailable`. +Exceeding a scenario-owned token or operation limit is `Failed` because the witness did not fit its contract. Harness-controlled context that already exceeds the declared limit or an operator ceiling stopping the run is `InfrastructureError` owned by `runner.budget`. Missing trustworthy provider accounting makes a paid query `Unavailable`. ## Retries and cleanup -Assertion and Symposium failures are never retried automatically. A recognized transient infrastructure error may retry the complete scenario once with fresh state. Individual steps are never replayed inside an existing container or conversation. +Assertion and Symposium failures are never retried automatically. An agent-free scenario with a recognized transient infrastructure error may retry once from fresh state. A scenario that contacted a paid provider is never retried automatically; another attempt requires a new explicit invocation. Individual steps are never replayed inside an existing container or query context. Both attempts are preserved. A recovered run remains `Passed` with the `stability-warning(recovered-infrastructure-error)` modifier and attempt metadata, so infrastructure reliability still counts the transient. diff --git a/md/rfds/agent-interaction-testing/proposed-guide/README.md b/md/rfds/agent-interaction-testing/proposed-guide/README.md index cecb4546..756d2b75 100644 --- a/md/rfds/agent-interaction-testing/proposed-guide/README.md +++ b/md/rfds/agent-interaction-testing/proposed-guide/README.md @@ -17,16 +17,16 @@ Running `cargo xtask agent-test` without a scenario prints an execution plan and Repeat `--scenario` to select more than one journey: ```console -cargo xtask agent-test --agent claude --environment container --auth api-key --scenario dependency-consent-accept --scenario dependency-consent-decline +cargo xtask agent-test --agent claude --environment container --auth api-key --confirm-paid-run --scenario dependency-consent-accept --scenario dependency-consent-decline ``` ## Run on the host ```console -cargo xtask agent-test --agent claude --environment host --auth local --scenario dependency-consent-accept +cargo xtask agent-test --agent claude --environment host --auth local --confirm-paid-run --scenario dependency-consent-accept ``` -The host runner creates fresh project, Symposium, agent, cache, and temporary directories. Local authentication is used only when explicitly requested. If the adapter cannot separate credentials from normal agent configuration, the result is marked non-authoritative. +The host runner creates fresh project, Symposium, agent, cache, and temporary directories. Local authentication is used only when explicitly requested. If the adapter cannot separate credentials from normal agent configuration, the result is marked non-authoritative. Local authentication also cannot claim the tracer key's $5 provider cap; the execution plan reports that limitation while retaining the scenario's hard token and operation limits. Host runs are useful for debugging but may still be affected by installed tools and the operating system. @@ -34,7 +34,7 @@ Host runs are useful for debugging but may still be affected by installed tools ```console $env:ANTHROPIC_API_KEY = "..." -cargo xtask agent-test --agent claude --environment container --auth api-key --scenario dependency-consent-accept +cargo xtask agent-test --agent claude --environment container --auth api-key --confirm-paid-run --scenario dependency-consent-accept ``` The runner prepares a pinned base image and one content-addressed Linux `cargo-agents` build from the checkout. Each scenario receives a fresh restricted container. Scenario runtime is hermetic except for the selected agent provider. @@ -42,7 +42,7 @@ The runner prepares a pinned base image and one content-addressed Linux `cargo-a To test an existing compatible Linux artifact: ```console -cargo xtask agent-test --agent claude --environment container --auth api-key --symposium-bin ./artifacts/cargo-agents-linux-x86_64 --scenario dependency-consent-accept +cargo xtask agent-test --agent claude --environment container --auth api-key --confirm-paid-run --symposium-bin ./artifacts/cargo-agents-linux-x86_64 --scenario dependency-consent-accept ``` The override is checked for operating system, architecture, executable format, and available version metadata. The runner never substitutes a released package, PATH binary, host environment, or different execution backend silently. @@ -60,17 +60,15 @@ Maximum provider requests: 4 Maximum tool calls: 3 Input-side token guard: 25,000 Output-token guard: 1,000 -Daily tokens remaining: 30,000 -Monthly tokens remaining: 750,000 Per-run cost allowance: $0.20 Monthly provider cap: $5.00 Environment: Linux container Agent/runtime: Claude, pinned ``` -Real-agent scenarios enforce cumulative input, cache-read, cache-write, and output tokens as well as provider-request, turn, tool-call, deadline, and run-wide limits. Cached tokens still count even when they cost less. +Real-agent scenarios enforce cumulative input, cache-read, cache-write, and output tokens as well as provider-request, turn, tool-call, deadline, and run-wide limits. Cached tokens still count even when they cost less. A paid run requires explicit selection, an agent name, and `--confirm-paid-run`. -The runner reserves the complete scenario ceiling from its daily and monthly ledgers before contacting the provider. If there is not enough capacity, the run does not start. A retry needs a separate reservation. The initial tracer permits at most one paid run per day and 25 per month. Its base-token estimate is approximately $0.09 per run, its allowance including cache-price differences is $0.20, and its dedicated provider key has a $5 monthly cap. Measured calibration usage lowers the scheduled token limit; it never rises automatically. +The initial tracer permits one user turn, at most four provider requests, three tool calls, 25,000 total input-side tokens, and 1,000 output tokens. Its base-token estimate is approximately $0.09 per run, its conservative allowance including cache-price differences is $0.20, and its dedicated provider key has a $5 monthly cap. Initial manual runs record usage so the limits can be reduced. The prompt and fixture are reduced before a limit is raised. ## Read a result @@ -83,15 +81,15 @@ Each run ends as: Results may carry modifiers. `non-authoritative(contaminated-auth-context)` means local authentication could not be isolated from agent configuration. `stability-warning(recovered-infrastructure-error)` means a complete fresh-state retry recovered from a recognized infrastructure failure. A modifier cannot turn a product failure into a pass or satisfy a conformance requirement with non-authoritative evidence. -A scenario that cannot produce its witness within its own token budget is `Failed`. Oversized harness context, exhausted daily or monthly capacity, or a lower operator limit is `InfrastructureError` owned by `runner.budget`. Scheduled paid execution is `Unavailable` when the adapter cannot report trustworthy usage. +A scenario that cannot produce its witness within its own token budget is `Failed`. Oversized harness context or a lower operator limit is `InfrastructureError` owned by `runner.budget`. Paid execution is `Unavailable` when the adapter cannot report trustworthy usage. -The summary also names the owning phase. Only a recognized transient infrastructure error may retry the entire scenario once with fresh state. Product failures and individual steps are never retried. A known-gap reproducer still returns `Failed` when run directly; scheduled reports identify it separately from release-gating covered scenarios. +The summary also names the owning phase. An agent-free scenario may retry once from fresh state after a recognized transient infrastructure error. A scenario that contacted a paid provider is never retried automatically. Product failures and individual steps are never retried. A known-gap reproducer still returns `Failed` when run directly. Artifacts are under `target/agent-tests//`. Failure artifacts contain only allowlisted, sanitized evidence and a redaction report. Complete homes, authentication directories, and process environments are never archived. Use `--keep-artifacts` to retain rich evidence for a passing run. ## Write a scenario -Scenarios use typed Rust builders but contain portable data rather than arbitrary closures. Every behavioral branch is a separate linear scenario with fresh state. +Each scenario registers declarative metadata and an asynchronous Rust body returning `Result`. The metadata names fixtures, requirements, contract IDs, permissions, budgets, and external endpoints so the runner can preflight without executing the body. The body uses a constrained `ScenarioContext`; it cannot reach undeclared host state, credentials, or agent-specific APIs. Every behavioral branch is a separate fresh-state scenario. A typical consent journey: @@ -101,7 +99,7 @@ A typical consent journey: 4. Run real `cargo agents sync` under a parsed PTY. 5. Select the intended prompt option with explicit keys. 6. Assert terminal anchors, structured events, exit status, and final state. -7. Start a persistent agent session when delivery is under test. +7. Run one bounded agent query when delivery is under test. 8. Assert a narrow capability witness such as a fixture nonce, hook trace, or MCP server log. Scenarios declare contract IDs, required capabilities, permissions, scenario-owned token and operation budgets, and external endpoints. They do not contain Claude-specific paths or judge general response quality. An operator-supplied lower budget is shown separately and cannot manufacture a Symposium failure. @@ -110,8 +108,6 @@ Time-dependent scenarios mutate controlled persisted inputs instead of sleeping ## CI operation -Deterministic tests block pull requests. Stable agent-free PTY and small Linux-container scenarios may graduate after meeting runtime and reliability criteria. Real-agent tracer journeys run in trusted scheduled or manual jobs and begin as non-gating observations. +Fast ordinary tests block pull requests. Stable agent-free process, PTY, and small Linux-container scenarios may graduate after meeting runtime and reliability criteria. Real-agent tracer journeys are manually selected and non-gating during this RFD. -### Future release gating - -A real-agent journey can become release-gating after at least 19 of its latest 20 eligible scheduled runs pass for the same pinned manifest. Assertion failures are not retried. Falling below the target requires a reviewed quarantine decision with a linked issue; quarantined journeys continue to run and report. +Scheduling, triage ownership, quarantine, pass-rate targets, and release gating require a follow-up informed by measured tracer reliability, runtime, and cost. diff --git a/md/rfds/agent-interaction-testing/scenario-model/README.md b/md/rfds/agent-interaction-testing/scenario-model/README.md index 69e67f55..9d936e94 100644 --- a/md/rfds/agent-interaction-testing/scenario-model/README.md +++ b/md/rfds/agent-interaction-testing/scenario-model/README.md @@ -6,20 +6,33 @@ Agent interaction tests extend `symposium-testlib`; they do not create a second `cargo test` remains the frontend for deterministic and selected host scenarios. Xtask is a thin orchestration frontend for environment selection, credentials, containers, filtering, real-agent execution, and artifact retention. -## Scenario data +## Scenario registration and body -Scenarios are authored with typed Rust builders. The underlying model contains portable data rather than arbitrary closures, so it can be serialized and may later gain a TOML or YAML frontend. - -A scenario declares: +Each scenario has declarative registration metadata and an imperative Rust body. The runner can list and preflight the metadata without executing the body. The metadata declares: +- a stable name and short description; - fixture layers and controlled services; - required environment and agent capabilities; -- ordered CLI, user, agent, mutation, restart, and checkpoint steps; - a capability witness for every real-agent interaction; - deadlines, resource limits, and scenario-owned agent budgets; - a least-privilege permission policy; - contract rule identifiers; and -- assertions and artifact-retention policy. +- artifact-retention policy. + +The body is an ordinary asynchronous Rust function returning `Result`. It receives a constrained `ScenarioContext` for running commands, driving the terminal, querying an agent, mutating controlled fixture state, and making assertions. Using Rust control flow and `?` keeps a failure at the operation that caused it instead of reporting one interpreter failure for the whole journey. + +```rust,ignore +async fn dependency_consent_accept(cx: &mut ScenarioContext) -> Result<()> { + cx.run_init().await?; + let mut sync = cx.spawn_sync_pty().await?; + sync.wait_for("Enable this dependency?").await?; + sync.press_enter().await?; + cx.assert_skill_installed("fixture-skill")?; + Ok(()) +} +``` + +The body cannot access undeclared host paths, process-global environment, credentials, or agent-specific APIs. Those remain behind `ScenarioContext`, environment backends, and agent adapters. A paid query, external endpoint, fixture service, or privileged operation must be declared in metadata so preflight cannot be bypassed by imperative code. Scenarios select capabilities, not agent brands. Agent-specific paths, authentication fields, event types, and witness mechanisms remain in adapters. @@ -27,9 +40,9 @@ Agent token budgets are cumulative across every provider request made for the jo ## State and branching -Scenarios are linear. Accept, decline, ask-later, and Escape are separate scenarios that may reuse fixture descriptions but never writable state. +Each scenario follows one expected behavioral path. Accept, decline, ask-later, and Escape are separate scenarios that may reuse fixture descriptions but never writable state. -Every scenario and retry begins with a fresh workspace, user configuration, agent configuration, cache, services, and conversation. Steps within one scenario share state deliberately, including across declared process or agent restarts. Persistence and cache scenarios express repeated commands in that one journey because preserved state is what they test. +Every scenario and retry begins with a fresh workspace, user configuration, agent configuration, cache, services, and agent query context. Steps within one scenario share state deliberately, including across declared process restarts. Persistence and cache scenarios express repeated commands in that one journey because preserved state is what they test. Scenario logic does not branch around unexpected output. A missing or different checkpoint fails at that step. @@ -57,6 +70,8 @@ This RFD does not add a production clock seam. Time-dependent tests mutate contr These mutations test the production comparison against the real wall clock without waiting for time to pass. If a later contract cannot be tested this way, its clock abstraction requires a separate design. Container, agent, TLS, provider, and process deadlines always use real time. -## Why typed Rust instead of a scenario DSL? +## Why metadata plus an imperative Rust body? + +Preflight needs declarative metadata before fixtures, containers, credentials, or paid agents are started. Journey execution benefits from ordinary Rust: compiler-assisted refactoring, direct reuse of test helpers, native asynchronous control flow, and line-local errors through `?`. -Typed builders provide compiler-assisted refactoring, IDE discovery, and direct reuse of test helpers while the vocabulary is evolving. The tradeoff is recompilation and a higher contribution barrier for non-Rust authors. Keeping the model serializable and closure-free preserves the option to add a data-file frontend after the vocabulary stabilizes. +A fully data-driven scenario would require the harness to grow an interpreter for every new interaction and would concentrate failures at that interpreter boundary. The constrained context preserves backend and adapter neutrality without creating a second programming language. Only registration metadata and the resulting execution plan need to be serializable; scenario bodies do not. From d3e13c8b544d6d3f639997313d3b9eb733d3ec7f Mon Sep 17 00:00:00 2001 From: Awesome Rustacean Date: Sun, 23 Aug 2026 01:07:57 +0300 Subject: [PATCH 4/4] docs(rfd): standardize agent interaction testing --- md/rfds/agent-interaction-testing/README.md | 229 ++++++++++++++---- .../agent-adapters/README.md | 18 +- .../coverage-and-ci/README.md | 62 +---- .../environments/README.md | 11 +- .../evidence/README.md | 33 ++- .../proposed-guide/README.md | 10 +- .../scenario-model/README.md | 12 +- 7 files changed, 254 insertions(+), 121 deletions(-) diff --git a/md/rfds/agent-interaction-testing/README.md b/md/rfds/agent-interaction-testing/README.md index 0e894a16..9f212641 100644 --- a/md/rfds/agent-interaction-testing/README.md +++ b/md/rfds/agent-interaction-testing/README.md @@ -2,36 +2,26 @@ ## TL;DR -- Extend the existing test infrastructure with scripted user, CLI, and real-agent journeys. -- Start every journey from controlled fixtures and isolated user and agent state. -- Run production-facing `cargo agents` processes under a real PTY. -- Keep exhaustive registry logic deterministic; use real agents only where delivery into an agent can fail. -- Use Claude first without placing Claude-specific concepts in scenarios. -- Use a Linux container for isolated Linux conformance and native lanes for Windows and macOS behavior. +- Add an experimental `cargo xtask agent-test` command for scripted journeys through real Symposium processes, terminals, isolated environments, and selected coding agents. +- Reuse the existing fixture infrastructure. Ordinary `cargo test` remains the fast, exhaustive layer for Symposium logic. +- Prove one tracer: accept or decline a dependency suggestion, repeat it in a fresh Linux container, and use one bounded Claude query to prove skill delivery. +- Keep scenarios agent-neutral, treat paid runs as explicit and non-gating, and defer broader registry coverage and release policy until the tracer provides implementation evidence. ## Motivation -Symposium's value comes from the interaction between discovery, consent, configuration, skills, hooks, MCP servers, caches, the user, and the coding agent. Existing integration tests cover much of Symposium's logic, but they do not consistently exercise the complete production boundary. For example, report events currently can reach a hook's stdout before its protocol payload, while an in-process dispatch assertion can still observe the correct event. The accepted `disable` precedence also disagrees with one current enablement path. Neither boundary failure is made obvious by the current integration suite. +The current integration suite has strong fixture and in-process coverage, but it cannot exercise every production boundary. `symposium-testlib::with_fixture` runs agent tests only when `SYMPOSIUM_ENABLE_AGENT_TESTING` is set, and the current agent path does not provide a fresh agent home, a real terminal conversation, or container isolation. -We want to begin with fixture directories and files, run real commands as a user would, answer interactive prompts, start real agents, and inspect what happened. The harness must make failures reproducible and distinguish a Symposium contract violation from an unavailable runtime or provider failure. +The discovery prompt is a concrete unreachable branch. `Output::is_interactive` requires terminal stdin and stdout, and `discovery::prompt_for_consent` returns without asking when that condition is false. Existing tests can verify discovery and noninteractive behavior, but they cannot select Enable or No through the interface a user sees. -This is integration testing, not agent evaluation. Measuring whether Symposium makes agents write better Rust requires baselines and statistical analysis and is outside this RFD. A future evaluation system may reuse these fixtures, adapters, and environments. +The compiled process also has behavior that an in-process assertion cannot observe. `src/bin/cargo-agents.rs` installs the normal report layer before identifying a hook command, so report output can precede the hook protocol payload on stdout even though the hook later uses `Output::quiet()`. A black-box process test is required to expose that failure. -## Behavioral contract +The missing evidence is therefore not more unit coverage. We need to start from controlled directories and configuration, execute the production command, provide user input, and inspect the visible output, structured decisions, and resulting state. A selected real agent is needed only to prove that Symposium-delivered capability crosses into the agent. -The tests verify that Symposium: +This RFD does not evaluate whether an agent writes better Rust. Effectiveness evaluation requires a baseline, repeated samples, and statistical analysis. The tracer proves integration behavior, not causal improvement. -- discovers extensions relevant to the current project; -- respects trust and explicit user choices; -- delivers enabled skills, hooks, MCP servers, and subcommands; -- expresses journeys through an adapter-neutral contract; and -- makes failures visible and contains their effects. - -Accepted RFDs and reference documentation define expected behavior, even when the implementation currently disagrees. For example, the accepted registry contract says `disable` overrides `use` and `auto-enable`; one current code path does not yet enforce that rule consistently. The coverage table marks this as follow-on direction instead of copying the bug into the expected result or claiming coverage. It becomes `Gap(issue)` only when a linked issue and executable reproducer exist. +## Change in a nutshell -## First journey - -The tracer journey starts with an empty Symposium home, empty agent configuration, and a Rust project whose dependency embeds a plugin awaiting consent. +The first accepted journey starts with an empty Symposium home, an empty agent configuration, and a Rust project whose dependency embeds a plugin awaiting consent: ```text run cargo agents init --add-agent claude @@ -39,49 +29,198 @@ run cargo agents sync under a PTY wait for the dependency suggestion choose Enable assert the visible prompt, structured events, exit status, config, and files +repeat the journey in a fresh Linux container run one bounded query through the Claude adapter -assert a fixture capability witness +assert the scenario nonce from the installed fixture skill ``` -A separate decline scenario begins from fresh state, selects “No, don't ask again,” restarts the CLI, and proves that the decision persists and the prompt does not return. Ask-later and Escape variants record nothing. +A separate decline scenario starts from fresh state, selects "No, don't ask again," restarts the CLI, and proves that the decision persists and the prompt does not return. + +The design follows these invariants: + +- Authoritative CLI assertions execute the compiled `cargo-agents` binary. Interactive commands use a PTY; hooks use pipes. +- Scenario registration metadata declares fixtures, capabilities, permissions, contracts, and budgets before execution. An asynchronous Rust body drives the journey through a constrained context. +- Host and container backends run the same scenario body. A requested environment never silently falls back to another. +- A real agent is used only when delivery into that agent is the behavior under test. Every paid query has a narrow capability witness and hard resource limits. +- Visible terminal output, structured events, and final state must agree. No one observation channel substitutes for the others. + +## Detailed plans + +### Behavioral contract + +The harness is intended to prove that Symposium: + +- discovers extensions relevant to the current project; +- respects trust and explicit user choices; +- installs and removes the expected configuration and files; +- delivers selected skills, hooks, MCP servers, and subcommands across the agent boundary; and +- reports failures without leaking host state or credentials. + +Accepted RFDs and reference documentation define the expected behavior when the implementation disagrees. For example, the [accepted discovery contract](../registry-centric-plugins/discovery-sync/README.md#enablement-configuration) gives `disable` precedence over `use` and `auto-enable`. The [coverage table](./coverage-and-ci/README.md#contract-table) records an implementation discrepancy as a gap or follow-up instead of treating current behavior as the contract. + +### Reference-level design + +The new engine is additive. Existing fixtures, `TestContext`, simulations, and deterministic tests remain. `cargo test` owns fast coverage, including deterministic black-box process regressions. `cargo xtask agent-test` orchestrates explicit environments, credentials, containers, filtering, artifacts, and real-agent execution. + +The supporting chapters are the authoritative homes for the detailed contracts: + +- [Scenario model](./scenario-model/README.md) defines registration metadata, imperative Rust bodies, the production process boundary, PTY scripting, and controlled time-dependent state. +- [Agent adapters](./agent-adapters/README.md) defines the provisional driver, Claude, later ACP and fake adapters, capability witnesses, permissions, and runtime pinning. +- [Execution environments](./environments/README.md) defines host and container isolation, native operating-system coverage, binary provenance, networking, fixture trust, and authentication. +- [Evidence and results](./evidence/README.md) defines observation channels, the event journal, assertions, result classification, retries, cleanup, and artifact safety. +- [Coverage and CI](./coverage-and-ci/README.md) defines the contract table, coverage layers, tracer obligations, command interface, cost controls, and CI boundaries. +- [Proposed guide](./proposed-guide/README.md) shows the intended developer-facing workflow. + +### Scope and compatibility + +This RFD is complete when: + +- the accept and decline journeys execute as real native processes through a parsed PTY; +- visible output, structured events, exit status, configuration, and filesystem state agree; +- the same journeys pass in a fresh Linux container; +- the accepted branch produces one bounded Claude capability witness containing its scenario nonce; and +- failures produce sanitized, useful artifacts with measured phase timing and provider usage. + +The tracer also fixes the known hook stdout contamination and adds its process-level regression. This provides an immediately useful result before the larger harness is complete. + +This RFD does not commit to every consent branch, exhaustive registry scenarios, persistent agent conversations, fake or ACP conformance, native Windows and macOS agent runs, hook and MCP delivery witnesses, scheduled real-agent execution, or release gating. Those remain [post-tracer direction](./coverage-and-ci/README.md#post-tracer-direction). + +The new command does not replace or reinterpret existing test results. Existing `cargo test` fixtures and assertions remain valid, and deterministic regressions continue to belong there when they can observe the production boundary. The agent-test runner adds a second, explicitly selected frontend for journeys that require a real terminal, controlled home, container, or agent. + +### Safety and interpretation boundaries + +Scenario fixtures are repository-owned and reviewed. An extension awaiting user consent is not treated as hostile code. The container improves reproducibility and least privilege; it is not a sandbox for testing malicious plugins, hooks, MCP servers, or agents. + +The isolation canary proves that a named decoy capability did not enter the controlled agent state. The scenario nonce proves that the selected fixture capability did enter the agent. Neither witness proves that the agent followed general instructions or produced high-quality code. + +Claude is the first production adapter because it is available to current developers. The scenario and driver contracts use agent-neutral concepts, but one adapter does not prove behavioral consistency across agents. The driver remains provisional until later fake and ACP implementations test the boundary. + +### Drawbacks + +The runner creates a second test frontend with its own scenario registration, preflight, artifact, and result code. Even though it reuses fixtures and assertions, maintainers must keep its behavior aligned with `cargo test` and the production CLI. + +Docker and provider credentials raise the contribution barrier. Most contributors can run the host, agent-free journeys, but reproducing Linux isolation or the Claude witness requires additional software, credentials, and provider access. + +Real-agent execution is nondeterministic, slower, and paid. Narrow witnesses and hard budgets limit those risks; they do not eliminate provider outages, model changes, or occasional inconclusive runs. + +The tracer covers only consent and skill delivery. A passing tracer could create false confidence if it is presented as broad registry or agent conformance. Reports must identify the exact contracts and evidence each journey proves. + +PTY behavior differs across operating systems, so Linux container success cannot satisfy native Windows or macOS requirements. The tracer proves Linux container behavior and the host platform used during development; native expansion remains separate work. + +Container preparation adds runtime beyond the current in-process suite. The runner reports checkout build or image preparation separately from warm startup, agent execution, and evidence processing so the team can decide which agent-free scenarios are suitable for pull-request CI. + +### Rationale and alternatives + +The selected design keeps exhaustive deterministic coverage in `cargo test` and adds an opt-in orchestrator only for evidence that the existing frontend cannot obtain. This separates inexpensive product logic from process, terminal, isolation, and provider costs. -The same registered scenario runs through in-process, native-process, Linux-container, and selected real-agent layers. Unsupported combinations are reported explicitly rather than silently weakened. +#### Extend only the existing test harness -## Design chapters +One test command and one fixture API would be simpler. The existing harness should continue to gain deterministic process regressions where practical, including the hook stdout test. It cannot make provider credentials, Docker, PTY interaction, and paid execution safe defaults for ordinary `cargo test`, however. Keeping those concerns in an explicit command preserves the current suite's speed and accessibility. -- [Scenario model](./scenario-model/README.md) defines declarative registration metadata, imperative Rust bodies, the production process boundary, PTY scripting, and explicit time-state fixtures. -- [Agent adapters](./agent-adapters/README.md) defines Claude, ACP and fake follow-ups, bounded queries, capability witnesses, permissions, and runtime pinning. -- [Execution environments](./environments/README.md) defines host and container isolation, native OS coverage, binary provenance, initialization, networking, trust, and authentication. -- [Evidence and results](./evidence/README.md) defines dual observation, canonical events, assertions, results, retries, cleanup, and artifact safety. -- [Coverage and CI](./coverage-and-ci/README.md) defines the contract table, coverage obligations, tracer journeys, command interface, cost controls, and implementation steps. -- [Proposed guide](./proposed-guide/README.md) shows how developers would list, run, inspect, and author scenarios. +#### Stop at black-box process and PTY tests -## Key boundaries +This would prove discovery, consent, hook output, and persisted state without provider cost. It would not prove that a capability installed by Symposium is visible inside a supported agent. One bounded nonce query is retained because crossing that final boundary is a central claim of the integration. -The new engine is additive. Existing fixtures, `TestContext`, simulations, and deterministic tests remain. `cargo test` handles fast coverage; `cargo xtask agent-test` is a thin orchestration frontend for selecting expensive environments and agents. +#### Describe complete journeys as data -Authoritative user journeys execute the compiled binary through `cargo agents`; they do not substitute an in-process call. PTY output proves that a user can see and answer a prompt, while a structured side channel and final state prove the underlying decision. +A fully declarative format could be serialized and generated by external tools. It would also require a scenario interpreter and concentrate errors from an entire journey at the interpreter boundary. Declarative registration metadata is retained for preflight and discovery, while an imperative asynchronous Rust body provides normal control flow and local `?` failure sites. -Scenario fixtures are reviewed repository content. “Untrusted” means awaiting user consent, not hostile code. The container improves reproducibility and least privilege but is not claimed as a sandbox for malicious extensions. +#### Add persistent agent sessions immediately -Every real-agent journey has a bounded capability witness. General prose and code quality are not graded. Claude is the first production adapter. Fake and ACP conformance are required follow-ups before the provisional driver interface can be called stable. +Persistent sessions will be useful for confirmation and restart journeys that span multiple agent turns. The tracer uses one query, so a session abstraction would be designed without an exercising scenario. The adapter begins with a bounded single-query capability and grows only when a committed journey requires persistence. -## Scope and milestones +#### Make the container conditional on host leakage -This RFD is implemented when the tracer is proven: the consent journey works through real native processes, a parsed PTY, structured evidence, a fresh Linux container, and one bounded Claude capability witness, with useful failure artifacts and measured runtime. +The host canary can show whether a named decoy entered the controlled agent home. It cannot control installed tools, networking, system libraries, or operating-system behavior. The Linux container is therefore retained as a reproducibility boundary even when the host canary passes. -Broader catalog automation, consent branches, cross-platform process lanes, fake and ACP conformance, hook and MCP witnesses, and trusted release CI are follow-on direction rather than acceptance criteria for this RFD. They require tracked issues or follow-on RFDs after the tracer informs the interfaces. See [Coverage and CI](./coverage-and-ci/README.md#milestones-and-follow-on-direction). +#### Do nothing + +The current suite would remain unable to select both consent outcomes through the terminal, detect some compiled-process output failures, or prove that an installed capability reaches a real agent. Those are the specific blind spots this RFD exists to close. + +### Prior art + +[`cli-testing-library`](https://github.com/crutchcorn/cli-testing-library) provides a useful interaction vocabulary based on querying visible screen state and sending user events. This RFD adopts that model for parsed PTY interaction, but not its Node implementation or platform limitations. + +[`cli-testing-specialist`](https://github.com/sanae-abe/cli-testing-specialist) demonstrates generated tests for general CLI behavior. Symposium journeys need product-specific fixtures, discovery contracts, persisted consent, hook protocols, MCP evidence, and agent delivery, so generic command validation is not the central abstraction here. + +The existing Symposium integration harness supplies the fixture composition and deterministic assertions that the new runner reuses. Its strengths argue for an additive frontend rather than replacement; its inability to provide controlled interactive and agent boundaries identifies where the addition begins. + +Rust's experimental [libtest JSON output RFC](https://rust-lang.github.io/rfcs/3558-libtest-json.html) separates structured test events from presentation and validates a new harness interface before stabilization. This RFD follows the same lessons through a structured evidence channel and an experimental runner, without adopting libtest's event protocol. + +### Unresolved questions + +No design question currently blocks acceptance. The tracer contract, scope, evidence layers, and ownership boundaries are defined. + +Implementation must still establish: + +- which PTY implementation satisfies the parsed-terminal contract on the initial host platform; +- whether the Claude adapter can report trustworthy usage and expose the controlled custom-skill inventory without agent-specific behavior leaking into scenarios; and +- the measured cold preparation, warm startup, provider usage, and total cost of the container-backed witness. + +These measurements may refine internal interfaces and limits. If an implementation result makes a required witness or isolation guarantee infeasible, the contract returns to discussion rather than being weakened silently. + +Scheduling, release gating, persistent sessions, additional adapters, and native operating-system coverage are deliberately deferred. They are future design questions, not acceptance blockers for the tracer. + +### Future possibilities + +Additional registry journeys can register new metadata and bodies against the same `ScenarioContext`. New agents can implement the adapter contract without adding agent-specific paths to scenarios. Persistent conversations can become an adapter capability when the first multi-turn journey supplies a concrete test. CI and release policy can be designed from measured reliability, runtime, and cost instead of estimates. + +These extensions build on the process, scenario, environment, adapter, and evidence boundaries established here; none requires replacing the tracer architecture. They are not commitments of this RFD and are not independent reasons to accept it. + +### Proposed documentation + +The [agent interaction test guide](./proposed-guide/README.md) is written as the developer documentation should read once the experimental command exists. It explains scenario discovery, host and container execution, paid-run confirmation, budgets, results, artifacts, scenario authoring, and the initial CI boundary. ## Frequently asked questions -### Does this replace the current integration tests? +### What does a passing real-agent journey prove? + +It proves the contracts named by that journey and only those contracts. For the tracer, matching terminal, event, exit, and state evidence proves the consent behavior; an exact controlled inventory plus the scenario nonce proves that the fixture skill crossed into the selected agent. It does not prove general response quality or all registry behavior. + +### Is this an agent-effectiveness evaluation? + +No. Effectiveness evaluation compares outcomes, needs a baseline and repeated samples, and may judge code quality. This harness verifies that specified Symposium interactions and delivery boundaries work. Effectiveness studies may later use the harness as execution infrastructure, but their claims and methodology remain separate. + +## Implementation plan and status + +Implementation has not begun. Each step leaves the repository with an independently useful, passing result. + +### Step 1: Fix hook stdout at the process boundary + +Change hook execution so stdout contains only the selected agent's protocol payload. Add a deterministic black-box regression that spawns the compiled hook command with piped stdin, stdout, and stderr. + +The new agent-test runner, PTY support, containers, and agent adapters remain absent. + +- [ ] Verify that stdout parses as the expected hook protocol and that human report output is absent. +- [ ] Run the existing hook and integration tests. + +### Step 2: Run the host consent journeys + +Add scenario registration metadata, the constrained asynchronous `ScenarioContext`, the `cargo xtask agent-test` frontend, the structured side channel, and the minimum PTY driver needed for dependency-consent accept and decline. Reuse the current fixture composition and assertion helpers. + +Containers and real-agent execution remain absent. Reconcile the contract table with the executable scenario names after the journeys pass; do not add catalog code generation. + +- [ ] Verify accept and decline against terminal anchors, structured events, exit status, configuration, filesystem state, the host-state canary, and exact fixture-controlled custom-skill inventory. +- [ ] Verify every `Covered` contract row names an executable scenario and every `Gap(issue)` row names an issue and failing reproducer. + +### Step 3: Run the same journeys in Linux + +Add Docker execution, a content-addressed Linux Symposium binary, least-privilege container rules, disabled scenario networking, cleanup, and infrastructure diagnostics. Run the Step 2 scenario bodies unchanged. + +Provider egress, credentials, and real-agent execution remain absent. + +- [ ] Verify cold preparation and warm startup separately. +- [ ] Verify the host-state canary, custom-skill inventory, cleanup, and parity with the remaining host assertions. + +### Step 4: Add one real-agent delivery witness -No. It reuses them and adds missing process, PTY, isolation, observation, and agent-delivery seams. A real-agent query is added only when activation inside the agent is the behavior under test. +Add the bounded Claude adapter and extend the container-backed accepted branch with one fixture-skill query. Pin the runtime and add only the allowlisted provider egress and restricted API-key handling required by that query. -### Why not adopt one of the linked CLI testing projects? +Persistent conversations, other agents, scheduled execution, and release gating remain absent. -`cli-testing-library` provides a useful screen-query and user-event model, which this design borrows. Its Node implementation and platform constraints are not a good foundation for this Rust, cross-platform harness. `cli-testing-specialist` targets generic generated CLI validation and does not provide Symposium-specific state, consent, hooks, MCP, or agent-delivery behavior. +- [ ] Verify the capability nonce, exact pre-query custom-skill inventory, installation and hook-registration evidence, usage limits, redaction, cleanup, and error classification. +- [ ] Record phase timing, provider usage, and conservative cost from an explicitly confirmed manual run without automatic paid retries. -## Implementation status +Before closing the RFD, correct `md/design/running-tests.md` so it documents the `SYMPOSIUM_ENABLE_AGENT_TESTING` gate. Keep `TestMode::AgentOnly`, `test-agents.toml`, and `tests/agent_harness/run_scenario.py` temporarily for existing Claude and ACP coverage, but mark that path as superseded and add no new scenarios to it. File its removal with the ACP follow-up after the remaining scenarios migrate. -This RFD describes proposed experimental infrastructure. Implementation has not begun, and the tracer milestone has not been reached. +Closing the RFD also requires follow-up issues or RFDs for catalog automation, fake and ACP conformance, remaining scenario families, native operating-system expansion, and release CI graduation. diff --git a/md/rfds/agent-interaction-testing/agent-adapters/README.md b/md/rfds/agent-interaction-testing/agent-adapters/README.md index a277bbf2..0f2b8adc 100644 --- a/md/rfds/agent-interaction-testing/agent-adapters/README.md +++ b/md/rfds/agent-interaction-testing/agent-adapters/README.md @@ -1,6 +1,8 @@ # Agent adapters -## Driver contract +## Adapter contract + +An agent adapter implements the `AgentDriver` boundary. Scenarios request capabilities through `ScenarioContext`; they do not call an adapter or agent SDK directly. An `AgentDriver`: @@ -12,17 +14,23 @@ An `AgentDriver`: - applies scenario-declared permission policy; and - returns normalized events plus sanitized raw provider artifacts. -The driver interface is capability-based. A scenario requiring an unsupported capability or witness is `Unavailable` for that adapter rather than weakened to a filesystem-only check. An adapter without trustworthy usage accounting cannot run a paid tracer query. +The driver interface is capability-based. A scenario requiring an unsupported capability or witness is `Unavailable` for that adapter rather than weakened to a filesystem-only check. An adapter without trustworthy usage accounting cannot run a paid tracer query. The [result contract](../evidence/README.md#results-and-failure-ownership) owns this classification. + +## Adapter scope -## Initial adapters +| Adapter | Role | Scope | +|---|---|---| +| Claude | First production adapter | One fresh, bounded skill-delivery query | +| Fake | Deterministic contract adapter | Follow-up before stabilizing `AgentDriver` | +| ACP fixture | Second protocol adapter and future session input | Follow-up | -Claude is the first production adapter because Symposium developers already use it. The tracer uses its structured SDK so completion, tool activity, and provider usage are observable without paying for a second interactive-entry-point smoke test. +The Claude adapter uses its structured SDK so completion, tool activity, and provider usage are observable. The tracer does not add a separate interactive-entry-point smoke test. The tracer needs one fresh, bounded Claude query to prove delivery of the fixture capability. The provisional driver therefore does not introduce persistent-conversation machinery. A later scenario that genuinely depends on multiple turns or an agent restart must add that capability deliberately and test it before the driver contract grows to include it. The tracer implements Claude behind a provisional capability-based interface. It does not claim cross-agent behavioral consistency from one production adapter. -Before the interface is declared stable, a follow-up adds deterministic fake adapters for success, failure, timeout, malformed-event, and missing-capability paths. The existing persistent ACP path can then inform a separately tested session capability and a second adapter. Only the bounded Claude query belongs to this RFD's real-agent tracer. +Before the interface is declared stable, a follow-up adds deterministic fake adapters for success, failure, timeout, malformed-event, and missing-capability paths. The existing persistent ACP path can then inform a separately tested session capability and a second adapter. Only the bounded Claude query belongs to the tracer. ## Capability witnesses diff --git a/md/rfds/agent-interaction-testing/coverage-and-ci/README.md b/md/rfds/agent-interaction-testing/coverage-and-ci/README.md index 917dcaeb..5a82d3c7 100644 --- a/md/rfds/agent-interaction-testing/coverage-and-ci/README.md +++ b/md/rfds/agent-interaction-testing/coverage-and-ci/README.md @@ -2,14 +2,14 @@ ## Contract table -The tracer begins with a reviewed Markdown table of the Symposium promises it exercises. Each row has a stable rule identifier, a behavioral statement, a link to the accepted specification, its required test layers, and one state: +The tracer begins with a reviewed Markdown table of the Symposium promises it exercises. Each row has a stable rule identifier, a behavioral statement, its required test layers, and one state: - `Committed(step)`: this RFD commits to implementing the row in the named tracer step. It becomes `Covered` after the required scenarios pass. - `Covered`: every required tracer scenario exists and passes. - `Gap(issue)`: the implementation is known to violate the specification, a linked issue owns the discrepancy, and an executable reproducer returns `Failed` when run directly. - `Direction(follow-up)`: the rule is outside this RFD's tracer commitment and must be carried into a closing follow-up issue or RFD. It is not counted as tracer coverage. -Accepted RFDs and current reference documentation remain authoritative. The table must not copy an implementation bug into the expected result. +Accepted RFDs and current reference documentation remain authoritative; this table is a coverage index rather than a replacement specification. It must not copy an implementation bug into the expected result. Typed Rust scenarios name the rules they prove. After enough journeys exist to expose stable catalog requirements, a follow-up may make the table machine-readable, validate layer and operating-system obligations, and generate a coverage report. This RFD does not build that meta-tool before the first journey. @@ -21,14 +21,14 @@ The matrix records intended obligations, including follow-on direction; it does | Rule ID | Contract | State | In-process | Real process | Real agent | |---|---|---|---:|---:|---:| -| `consent.accept` | Undecided candidate is accepted | `Committed(steps 1, 3)` | required | required, PTY | not required | -| `consent.decline` | Undecided candidate is declined | `Committed(steps 1, 3)` | required | required, PTY | not required | +| `consent.accept` | Undecided candidate is accepted | `Committed(steps 2, 3)` | required | required, PTY | not required | +| `consent.decline` | Undecided candidate is declined | `Committed(steps 2, 3)` | required | required, PTY | not required | | `consent.defer` | Ask later records nothing | `Direction(follow-up)` | required | required, PTY | not required | | `cli.noninteractive` | Noninteractive execution never prompts | `Direction(follow-up)` | required | required, pipes | not required | | `enablement.disable-precedence` | Disable overrides other enablement | `Direction(follow-up)` | required | representative, pipes | not required | | `cache.expiration` | Cache expiration reevaluates its input | `Direction(follow-up)` | required | required, pipes | not required | | `hook.stdout-protocol` | Hook stdout contains only protocol output | `Committed(step 1)` | not sufficient | required, pipes | not required | -| `isolation.skill-inventory` | Isolated custom-skill inventory exactly matches the fixture | `Committed(steps 1, 3, 4)` | not required | required, pipes | required | +| `isolation.skill-inventory` | Isolated custom-skill inventory exactly matches the fixture | `Committed(steps 2, 3, 4)` | not required | required, pipes | required | | `delivery.skill` | An enabled fixture skill reaches the selected agent | `Committed(step 4)` | not sufficient | required, pipes | one delivery smoke | | `use.search-endpoint` | Non-workspace `use` search uses only its declared fixture endpoint | `Direction(follow-up)` | required | required, pipes | not required | | `delivery.hook` | Hook delivery reaches an agent | `Direction(follow-up)` | required | representative, pipes | required | @@ -72,7 +72,7 @@ Initial options are: --keep-artifacts ``` -`--scenario` is repeatable. No scenario means “print the execution plan,” not “start an agent.” A selection containing a real-agent journey requires an explicit agent and `--confirm-paid-run`. Missing runtime, credentials, or capability yields `Unavailable`; a requested container never silently falls back to the host. +`--scenario` is repeatable. No scenario means "print the execution plan," not "start an agent." A selection containing a real-agent journey requires an explicit agent and `--confirm-paid-run`. Missing runtime, credentials, or capability yields `Unavailable`; a requested container never silently falls back to the host. The plan reports CLI-only and real-agent scenarios, scenario and operator token limits, maximum turns, provider requests, and tool calls, the provider-side spending cap, environment, binary provenance, and pinned runtime before execution. @@ -90,61 +90,21 @@ Runtime reporting separates checkout build or image preparation, warm environmen The tracer's provisional guard permits one user turn, at most four provider requests, three tool calls, 25,000 total input-side tokens across base input, cache reads, and cache writes, and 1,000 cumulative output tokens. Initial manual runs record actual usage so these limits can be reduced. If the pinned runtime cannot produce the nonce witness within the guard, the prompt, tools, fixture, and context are reduced before any limit is raised. -At the standard Sonnet base price of $3 per million input tokens and $15 per million output tokens, the provisional base-token estimate is approximately $0.09 per run. The runner reports a conservative allowance of $0.20 per run for cache-price differences while the provider key caps the month at $5. The estimate is recalculated when the model pin or [provider pricing](https://www.anthropic.com/news/claude-sonnet-5) changes; tokens remain the primary limit and dollars are derived reporting. +At Sonnet 5's standard price after its introductory period, $3 per million input tokens and $15 per million output tokens, the provisional base-token ceiling is approximately $0.09 per run. The runner reports a conservative allowance of $0.20 per run for cache-price differences while the provider key caps the month at $5. The estimate is recalculated when the model pin or [provider pricing](https://www.anthropic.com/research/claude-sonnet-5) changes; tokens remain the primary limit and dollars are derived reporting. -## CI lanes in this RFD +## Tracer CI boundary - Fast deterministic tests block every pull request. - Stable native agent-free process and PTY scenarios may become PR-blocking. - A small agent-free Linux-container suite may graduate if its measured runtime is acceptable. -- Real-agent tracer journeys are explicitly selected, manually run, and non-gating during this RFD. +- Real-agent tracer journeys are explicitly selected, manually run, and non-gating while the runner is experimental. Agent-free tests graduate after an observation period with no unexplained flakes, acceptable runtime, actionable failure artifacts, reliable cleanup, and consistently successful secret-canary validation. Provider credentials are never exposed to fork pull requests. -This RFD does not define scheduled ownership, quarantine, pass-rate, or release-gating policy for real-agent tests. A follow-up may propose those mechanisms using measured tracer reliability, runtime, and cost rather than assumptions. +Scheduled ownership, quarantine, pass-rate, and release-gating policy for real-agent tests are outside the tracer contract. A follow-up may propose those mechanisms using measured reliability, runtime, and cost rather than assumptions. -## Milestones and follow-on direction - -### Tracer proven - -The dependency-consent journey reuses the current fixture infrastructure and runs real `init` and `sync` processes. Accept and decline work through a parsed PTY, structured events agree with final state, the scenario runs in a fresh Linux container, and the accepted branch produces a bounded Claude capability witness. Failure artifacts, cleanup, cost, and phase timing are demonstrated. - -This validates the architecture and completes this RFD. - -### Post-tracer direction +## Post-tracer direction After the tracer, tracked follow-ups can expand the contract table across registry, discovery, predicate, cache, and delivery behavior. They can add every consent branch, representative Linux-container and native Windows/macOS lanes, fake and fixture-ACP adapter contracts, and selected hook and MCP witnesses. Catalog automation, full release reporting, a latest-agent canary, and broader CI graduation are separate commitments informed by tracer evidence. They are not acceptance criteria for this RFD. - -## Implementation plan - -### Step 1: Run the first black-box host journey - -From an empty user configuration, run compiled `cargo agents init --add-agent ` and `cargo agents sync` processes under a PTY for one dependency candidate. Add only the scenario steps, side-channel events, terminal anchors, assertions, and artifacts required for initialization plus accept and decline. - -Verify both branches against terminal output, structured events, exit status, configuration, filesystem state, the host-state canary, and the exact fixture-controlled custom-skill inventory. Fix the hook stdout contamination bug and add a black-box regression that invokes the compiled hook through pipes and proves stdout is valid protocol output only. - -### Step 2: Record the tracer contracts - -Write the initial Markdown contract table from the behavior exercised by step 1. A discrepancy becomes `Gap(issue)` only when it has a product issue and executable reproducer; otherwise it remains follow-on direction. Do not add catalog code generation or validation. - -Verify manually that every `Covered` row names an executable scenario and every `Gap(issue)` row names both an issue and a reproducer. - -### Step 3: Isolate the journey in Linux - -Add Docker execution, the content-addressed Symposium binary, least-privilege rules, networking disabled, and infrastructure diagnostics. Run the existing consent scenario unchanged. Do not add provider egress or general fixture-service infrastructure in this step. - -Verify cold preparation, warm startup, the same host-state canary and custom-skill inventory assertions used by the host backend, and parity with the remaining host assertions. - -### Step 4: Add the first real-agent witness - -Add the bounded Claude adapter and extend the container-backed accepted branch with one fixture-skill query. Pin its runtime and add only the allowlisted provider egress and restricted API-key handling this query requires. - -Verify the capability nonce, exact pre-query custom-skill inventory, installation and hook-registration evidence, usage limits, redaction, and error classification. - -Record the manually invoked result without automatic paid retries. The tracer remains non-gating. - -Before closing the RFD, correct `md/design/running-tests.md` so it documents the `SYMPOSIUM_ENABLE_AGENT_TESTING` gate. Keep `TestMode::AgentOnly`, `test-agents.toml`, and `tests/agent_harness/run_scenario.py` temporarily for existing Claude and ACP coverage, but mark that path as superseded and add no new scenarios to it. File its removal with the ACP follow-up, after remaining scenarios migrate. - -Also file follow-up issues or RFDs for catalog automation, fake and ACP conformance, remaining scenario families, native operating-system expansion, and release CI graduation. diff --git a/md/rfds/agent-interaction-testing/environments/README.md b/md/rfds/agent-interaction-testing/environments/README.md index a6256842..d2d9be5f 100644 --- a/md/rfds/agent-interaction-testing/environments/README.md +++ b/md/rfds/agent-interaction-testing/environments/README.md @@ -2,11 +2,16 @@ ## Environment backends -The host backend is for fast local iteration. It creates fresh project, home, configuration, cache, and temporary directories and passes an explicit filtered environment to every child process. Host results remain non-authoritative because installed tools and the operating system can influence them. +| Backend | Purpose | Isolation claim | +|---|---|---| +| Host | Fast local iteration and native process behavior | Fresh test state, but installed tools and the host operating system remain observable | +| Linux container | Reproducible Linux conformance | Fresh restricted container with controlled tools, filesystem, and networking | + +The host backend creates fresh project, home, configuration, cache, and temporary directories and passes an explicit filtered environment to every child process. Host results remain non-authoritative because installed tools and the operating system can influence them. The Linux container backend is the isolated conformance environment. The tracer uses Docker behind an interface that can later support another container runtime, a VM, or a remote worker. Linux-container results are authoritative only for Linux. -Windows and macOS use native deterministic and real-process/PTY CI lanes with fresh test directories. These lanes cover platform-specific paths, command dispatch, shell behavior, PTYs, permissions, and process handling without claiming container-strength isolation. Native real-agent smokes may be added when trusted runners and credentials are available. +Windows and macOS coverage is follow-up work. Native deterministic and real-process/PTY lanes use fresh test directories and cover platform-specific paths, command dispatch, shell behavior, PTYs, permissions, and process handling without claiming container-strength isolation. Native real-agent smokes require trusted runners and credentials. ## Scenario isolation @@ -59,4 +64,4 @@ Container conformance uses a restricted API key available only to trusted jobs a Host authentication is explicit: `--auth api-key` or `--auth local`. API-key mode uses a fresh agent home. Local mode bridges only the minimum adapter-supported credential material, read-only, while agent settings, Symposium configuration, skills, hooks, MCP configuration, caches, and query history remain fresh. -If an agent cannot separate credentials from user configuration, the base result carries the `non-authoritative(contaminated-auth-context)` modifier. The manifest records the mode and inherited credential paths without their contents. +If an agent cannot separate credentials from user configuration, the base result carries the [`non-authoritative(contaminated-auth-context)` modifier](../evidence/README.md#results-and-failure-ownership). The manifest records the mode and inherited credential paths without their contents. diff --git a/md/rfds/agent-interaction-testing/evidence/README.md b/md/rfds/agent-interaction-testing/evidence/README.md index 7c9b9f07..a8bfbeb7 100644 --- a/md/rfds/agent-interaction-testing/evidence/README.md +++ b/md/rfds/agent-interaction-testing/evidence/README.md @@ -25,6 +25,25 @@ Each event envelope contains: - real monotonic offset; and - normalized payload. +A confirmation event has this shape: + +```json +{ + "schema_version": 1, + "run_id": "run-fixture-01", + "scenario_id": "dependency-consent-accept", + "attempt": 1, + "source": "symposium", + "source_sequence": 4, + "operation_id": "sync-01", + "kind": "confirmation.answered", + "monotonic_offset_ms": 184, + "payload": { + "decision": "enable" + } +} +``` + Provider-operation events record requested limits and reported input, cache-read, cache-write, and output tokens. Aggregate token counts and derived cost are evidence, not estimates substituted for missing accounting. Sequence is strict within one source. Receipt order is diagnostic and does not imply causal order across processes. Assertions express partial order within a source or correlated operation, such as discovery before confirmation and confirmation before installation. Unrelated sources remain unordered unless explicitly correlated. @@ -46,12 +65,14 @@ Model prose is checked only through a narrow fixture-defined nonce or fact when ## Results and failure ownership -A run has four results: +A run has four base results: -- `Passed`: the requested journey and assertions completed. -- `Failed`: the environment ran, but Symposium or the interaction violated the contract. -- `InfrastructureError`: credentials, provider, runtime, environment, or harness failed. -- `Unavailable`: preflight found that the selected adapter or environment lacks a required capability. +| Result | Meaning | +|---|---| +| `Passed` | The requested journey and assertions completed. | +| `Failed` | The environment ran, but Symposium or the interaction violated the contract. | +| `InfrastructureError` | Credentials, provider, runtime, environment, runner budget, or harness failed. | +| `Unavailable` | Preflight found that the selected adapter or environment lacks a required capability. | A result may also carry modifiers that preserve important qualifications without creating another base result: @@ -62,7 +83,7 @@ Modifiers are recorded in the summary, journal, and aggregate reports. They neve Explicitly requesting an unavailable combination exits unsuccessfully; ordinary `cargo test` remains unaffected. There is no expected-failure scenario result. A known product-gap reproducer still returns `Failed` when run directly. -The coverage table records a `Gap(issue)` separately from completed tracer coverage. Its executable reproducer still returns `Failed`; this RFD does not add expected-failure results or release-gate policy. +The [coverage table](../coverage-and-ci/README.md#contract-table) records a `Gap(issue)` separately from completed tracer coverage. Its executable reproducer still returns `Failed`; the result vocabulary has no expected-failure state. Failures name an owning phase such as `environment.prepare`, `symposium.cli`, `symposium.state`, `agent.start`, `agent.query`, `fixture.mcp`, `assertion`, or `cleanup`. A Symposium crash, missing prompt, wrong state, or completed agent query without its required witness is `Failed`. diff --git a/md/rfds/agent-interaction-testing/proposed-guide/README.md b/md/rfds/agent-interaction-testing/proposed-guide/README.md index 756d2b75..62fac6be 100644 --- a/md/rfds/agent-interaction-testing/proposed-guide/README.md +++ b/md/rfds/agent-interaction-testing/proposed-guide/README.md @@ -10,7 +10,7 @@ The feature is experimental. Real-agent runs consume provider capacity and are o cargo xtask agent-test --list ``` -The scenario list reports required agent, environment, operating-system, and witness capabilities. The RFD's contract table maps the tracer's Symposium promises to executable scenarios and linked product gaps. +The scenario list reports required agent, environment, operating-system, and witness capabilities. The contract table maps the tracer's Symposium promises to executable scenarios and linked product gaps. Running `cargo xtask agent-test` without a scenario prints an execution plan and does not start an agent. @@ -66,9 +66,11 @@ Environment: Linux container Agent/runtime: Claude, pinned ``` +The [cost and runtime controls](../coverage-and-ci/README.md#cost-and-runtime-controls) are authoritative. The values above are the initial experimental tracer defaults. + Real-agent scenarios enforce cumulative input, cache-read, cache-write, and output tokens as well as provider-request, turn, tool-call, deadline, and run-wide limits. Cached tokens still count even when they cost less. A paid run requires explicit selection, an agent name, and `--confirm-paid-run`. -The initial tracer permits one user turn, at most four provider requests, three tool calls, 25,000 total input-side tokens, and 1,000 output tokens. Its base-token estimate is approximately $0.09 per run, its conservative allowance including cache-price differences is $0.20, and its dedicated provider key has a $5 monthly cap. Initial manual runs record usage so the limits can be reduced. The prompt and fixture are reduced before a limit is raised. +The initial tracer permits one user turn, at most four provider requests, three tool calls, 25,000 total input-side tokens, and 1,000 output tokens. At the standard post-introductory Sonnet 5 price, its base-token ceiling is approximately $0.09 per run. Its conservative allowance including cache-price differences is $0.20, and its dedicated provider key has a $5 monthly cap. Initial manual runs record usage so the limits can be reduced. The prompt and fixture are reduced before a limit is raised. ## Read a result @@ -108,6 +110,6 @@ Time-dependent scenarios mutate controlled persisted inputs instead of sleeping ## CI operation -Fast ordinary tests block pull requests. Stable agent-free process, PTY, and small Linux-container scenarios may graduate after meeting runtime and reliability criteria. Real-agent tracer journeys are manually selected and non-gating during this RFD. +Fast ordinary tests block pull requests. Stable agent-free process, PTY, and small Linux-container scenarios may graduate after meeting runtime and reliability criteria. Real-agent tracer journeys are manually selected and non-gating while the command is experimental. -Scheduling, triage ownership, quarantine, pass-rate targets, and release gating require a follow-up informed by measured tracer reliability, runtime, and cost. +Scheduled execution, triage ownership, quarantine, pass-rate targets, and release gating are not part of the experimental command. They require a separate policy informed by measured reliability, runtime, and cost. diff --git a/md/rfds/agent-interaction-testing/scenario-model/README.md b/md/rfds/agent-interaction-testing/scenario-model/README.md index 9d936e94..12eead1b 100644 --- a/md/rfds/agent-interaction-testing/scenario-model/README.md +++ b/md/rfds/agent-interaction-testing/scenario-model/README.md @@ -1,10 +1,10 @@ # Scenario model -## Shared engine +## Test frontends and shared engine Agent interaction tests extend `symposium-testlib`; they do not create a second fixture or assertion system. Ordinary tests and `cargo xtask agent-test` use the same fixture composition, scenario model, event vocabulary, and assertions. -`cargo test` remains the frontend for deterministic and selected host scenarios. Xtask is a thin orchestration frontend for environment selection, credentials, containers, filtering, real-agent execution, and artifact retention. +`cargo test` remains the frontend for deterministic and selected host scenarios. `cargo xtask agent-test` orchestrates environment selection, credentials, containers, filtering, real-agent execution, and artifact retention. A scenario does not change meaning when selected through a different frontend. ## Scenario registration and body @@ -32,7 +32,7 @@ async fn dependency_consent_accept(cx: &mut ScenarioContext) -> Result<()> { } ``` -The body cannot access undeclared host paths, process-global environment, credentials, or agent-specific APIs. Those remain behind `ScenarioContext`, environment backends, and agent adapters. A paid query, external endpoint, fixture service, or privileged operation must be declared in metadata so preflight cannot be bypassed by imperative code. +The body cannot access undeclared host paths, process-global environment, credentials, or agent-specific APIs. Those remain behind `ScenarioContext`, environment backends, and agent adapters. A paid query, external endpoint, fixture service, or privileged operation must be declared in metadata. The context rejects an operation that was not authorized by the registration metadata. Scenarios select capabilities, not agent brands. Agent-specific paths, authentication fields, event types, and witness mechanisms remain in adapters. @@ -60,8 +60,6 @@ Ordinary scenarios use a fixed terminal size, UTF-8 locale, declared TERM and co Screen normalization handles cursor movement, redraws, color, and newline differences. Raw sanitized bytes remain diagnostic evidence. A small rendering suite separately tests color and resizing; ordinary journeys do not snapshot the complete screen. -This adopts the useful interaction model from `cli-testing-library`: wait for what a user can see, then send user input, without adopting its Node implementation. - ## Time-dependent scenarios Tests never synchronize with fixed sleeps. Every wait targets an observable condition and has a real monotonic deadline. @@ -70,8 +68,8 @@ This RFD does not add a production clock seam. Time-dependent tests mutate contr These mutations test the production comparison against the real wall clock without waiting for time to pass. If a later contract cannot be tested this way, its clock abstraction requires a separate design. Container, agent, TLS, provider, and process deadlines always use real time. -## Why metadata plus an imperative Rust body? +## Metadata and body boundary Preflight needs declarative metadata before fixtures, containers, credentials, or paid agents are started. Journey execution benefits from ordinary Rust: compiler-assisted refactoring, direct reuse of test helpers, native asynchronous control flow, and line-local errors through `?`. -A fully data-driven scenario would require the harness to grow an interpreter for every new interaction and would concentrate failures at that interpreter boundary. The constrained context preserves backend and adapter neutrality without creating a second programming language. Only registration metadata and the resulting execution plan need to be serializable; scenario bodies do not. +Only registration metadata and the resulting execution plan are serializable. Scenario bodies are compiled Rust and are not loaded from TOML, YAML, or another scenario language. The constrained context preserves backend and adapter neutrality without exposing backend objects to the body. The [root rationale](../README.md#describe-complete-journeys-as-data) records why the design does not use a fully data-driven scenario format.