diff --git a/md/SUMMARY.md b/md/SUMMARY.md index 0ed860b4..ef43a934 100644 --- a/md/SUMMARY.md +++ b/md/SUMMARY.md @@ -96,6 +96,13 @@ - [Discovery & sync](./rfds/registry-centric-plugins/discovery-sync/README.md) - [User-managed plugins](./rfds/registry-centric-plugins/user-managed-plugins/README.md) - [Predicate caching](./rfds/predicate-caching/README.md) + - [Agent interaction testing](./rfds/agent-interaction-testing/README.md) + - [Scenario model](./rfds/agent-interaction-testing/scenario-model/README.md) + - [Agent adapters](./rfds/agent-interaction-testing/agent-adapters/README.md) + - [Execution environments](./rfds/agent-interaction-testing/environments/README.md) + - [Evidence and results](./rfds/agent-interaction-testing/evidence/README.md) + - [Coverage and CI](./rfds/agent-interaction-testing/coverage-and-ci/README.md) + - [Proposed: Agent interaction tests](./rfds/agent-interaction-testing/proposed-guide/README.md) - [Completed](./rfds/completed.md) - [Configuration parsing and normalization](./rfds/config-normalization/README.md) - [RFD Process](./rfds/rfd-process/README.md) diff --git a/md/rfds/agent-interaction-testing/README.md b/md/rfds/agent-interaction-testing/README.md new file mode 100644 index 00000000..9f212641 --- /dev/null +++ b/md/rfds/agent-interaction-testing/README.md @@ -0,0 +1,226 @@ +# Agent interaction testing + +## TL;DR + +- Add an experimental `cargo xtask agent-test` command for scripted journeys through real Symposium processes, terminals, isolated environments, and selected coding agents. +- Reuse the existing fixture infrastructure. Ordinary `cargo test` remains the fast, exhaustive layer for Symposium logic. +- Prove one tracer: accept or decline a dependency suggestion, repeat it in a fresh Linux container, and use one bounded Claude query to prove skill delivery. +- Keep scenarios agent-neutral, treat paid runs as explicit and non-gating, and defer broader registry coverage and release policy until the tracer provides implementation evidence. + +## Motivation + +The current integration suite has strong fixture and in-process coverage, but it cannot exercise every production boundary. `symposium-testlib::with_fixture` runs agent tests only when `SYMPOSIUM_ENABLE_AGENT_TESTING` is set, and the current agent path does not provide a fresh agent home, a real terminal conversation, or container isolation. + +The discovery prompt is a concrete unreachable branch. `Output::is_interactive` requires terminal stdin and stdout, and `discovery::prompt_for_consent` returns without asking when that condition is false. Existing tests can verify discovery and noninteractive behavior, but they cannot select Enable or No through the interface a user sees. + +The compiled process also has behavior that an in-process assertion cannot observe. `src/bin/cargo-agents.rs` installs the normal report layer before identifying a hook command, so report output can precede the hook protocol payload on stdout even though the hook later uses `Output::quiet()`. A black-box process test is required to expose that failure. + +The missing evidence is therefore not more unit coverage. We need to start from controlled directories and configuration, execute the production command, provide user input, and inspect the visible output, structured decisions, and resulting state. A selected real agent is needed only to prove that Symposium-delivered capability crosses into the agent. + +This RFD does not evaluate whether an agent writes better Rust. Effectiveness evaluation requires a baseline, repeated samples, and statistical analysis. The tracer proves integration behavior, not causal improvement. + +## Change in a nutshell + +The first accepted journey starts with an empty Symposium home, an empty agent configuration, and a Rust project whose dependency embeds a plugin awaiting consent: + +```text +run cargo agents init --add-agent claude +run cargo agents sync under a PTY +wait for the dependency suggestion +choose Enable +assert the visible prompt, structured events, exit status, config, and files +repeat the journey in a fresh Linux container +run one bounded query through the Claude adapter +assert the scenario nonce from the installed fixture skill +``` + +A separate decline scenario starts from fresh state, selects "No, don't ask again," restarts the CLI, and proves that the decision persists and the prompt does not return. + +The design follows these invariants: + +- Authoritative CLI assertions execute the compiled `cargo-agents` binary. Interactive commands use a PTY; hooks use pipes. +- Scenario registration metadata declares fixtures, capabilities, permissions, contracts, and budgets before execution. An asynchronous Rust body drives the journey through a constrained context. +- Host and container backends run the same scenario body. A requested environment never silently falls back to another. +- A real agent is used only when delivery into that agent is the behavior under test. Every paid query has a narrow capability witness and hard resource limits. +- Visible terminal output, structured events, and final state must agree. No one observation channel substitutes for the others. + +## Detailed plans + +### Behavioral contract + +The harness is intended to prove that Symposium: + +- discovers extensions relevant to the current project; +- respects trust and explicit user choices; +- installs and removes the expected configuration and files; +- delivers selected skills, hooks, MCP servers, and subcommands across the agent boundary; and +- reports failures without leaking host state or credentials. + +Accepted RFDs and reference documentation define the expected behavior when the implementation disagrees. For example, the [accepted discovery contract](../registry-centric-plugins/discovery-sync/README.md#enablement-configuration) gives `disable` precedence over `use` and `auto-enable`. The [coverage table](./coverage-and-ci/README.md#contract-table) records an implementation discrepancy as a gap or follow-up instead of treating current behavior as the contract. + +### Reference-level design + +The new engine is additive. Existing fixtures, `TestContext`, simulations, and deterministic tests remain. `cargo test` owns fast coverage, including deterministic black-box process regressions. `cargo xtask agent-test` orchestrates explicit environments, credentials, containers, filtering, artifacts, and real-agent execution. + +The supporting chapters are the authoritative homes for the detailed contracts: + +- [Scenario model](./scenario-model/README.md) defines registration metadata, imperative Rust bodies, the production process boundary, PTY scripting, and controlled time-dependent state. +- [Agent adapters](./agent-adapters/README.md) defines the provisional driver, Claude, later ACP and fake adapters, capability witnesses, permissions, and runtime pinning. +- [Execution environments](./environments/README.md) defines host and container isolation, native operating-system coverage, binary provenance, networking, fixture trust, and authentication. +- [Evidence and results](./evidence/README.md) defines observation channels, the event journal, assertions, result classification, retries, cleanup, and artifact safety. +- [Coverage and CI](./coverage-and-ci/README.md) defines the contract table, coverage layers, tracer obligations, command interface, cost controls, and CI boundaries. +- [Proposed guide](./proposed-guide/README.md) shows the intended developer-facing workflow. + +### Scope and compatibility + +This RFD is complete when: + +- the accept and decline journeys execute as real native processes through a parsed PTY; +- visible output, structured events, exit status, configuration, and filesystem state agree; +- the same journeys pass in a fresh Linux container; +- the accepted branch produces one bounded Claude capability witness containing its scenario nonce; and +- failures produce sanitized, useful artifacts with measured phase timing and provider usage. + +The tracer also fixes the known hook stdout contamination and adds its process-level regression. This provides an immediately useful result before the larger harness is complete. + +This RFD does not commit to every consent branch, exhaustive registry scenarios, persistent agent conversations, fake or ACP conformance, native Windows and macOS agent runs, hook and MCP delivery witnesses, scheduled real-agent execution, or release gating. Those remain [post-tracer direction](./coverage-and-ci/README.md#post-tracer-direction). + +The new command does not replace or reinterpret existing test results. Existing `cargo test` fixtures and assertions remain valid, and deterministic regressions continue to belong there when they can observe the production boundary. The agent-test runner adds a second, explicitly selected frontend for journeys that require a real terminal, controlled home, container, or agent. + +### Safety and interpretation boundaries + +Scenario fixtures are repository-owned and reviewed. An extension awaiting user consent is not treated as hostile code. The container improves reproducibility and least privilege; it is not a sandbox for testing malicious plugins, hooks, MCP servers, or agents. + +The isolation canary proves that a named decoy capability did not enter the controlled agent state. The scenario nonce proves that the selected fixture capability did enter the agent. Neither witness proves that the agent followed general instructions or produced high-quality code. + +Claude is the first production adapter because it is available to current developers. The scenario and driver contracts use agent-neutral concepts, but one adapter does not prove behavioral consistency across agents. The driver remains provisional until later fake and ACP implementations test the boundary. + +### Drawbacks + +The runner creates a second test frontend with its own scenario registration, preflight, artifact, and result code. Even though it reuses fixtures and assertions, maintainers must keep its behavior aligned with `cargo test` and the production CLI. + +Docker and provider credentials raise the contribution barrier. Most contributors can run the host, agent-free journeys, but reproducing Linux isolation or the Claude witness requires additional software, credentials, and provider access. + +Real-agent execution is nondeterministic, slower, and paid. Narrow witnesses and hard budgets limit those risks; they do not eliminate provider outages, model changes, or occasional inconclusive runs. + +The tracer covers only consent and skill delivery. A passing tracer could create false confidence if it is presented as broad registry or agent conformance. Reports must identify the exact contracts and evidence each journey proves. + +PTY behavior differs across operating systems, so Linux container success cannot satisfy native Windows or macOS requirements. The tracer proves Linux container behavior and the host platform used during development; native expansion remains separate work. + +Container preparation adds runtime beyond the current in-process suite. The runner reports checkout build or image preparation separately from warm startup, agent execution, and evidence processing so the team can decide which agent-free scenarios are suitable for pull-request CI. + +### Rationale and alternatives + +The selected design keeps exhaustive deterministic coverage in `cargo test` and adds an opt-in orchestrator only for evidence that the existing frontend cannot obtain. This separates inexpensive product logic from process, terminal, isolation, and provider costs. + +#### Extend only the existing test harness + +One test command and one fixture API would be simpler. The existing harness should continue to gain deterministic process regressions where practical, including the hook stdout test. It cannot make provider credentials, Docker, PTY interaction, and paid execution safe defaults for ordinary `cargo test`, however. Keeping those concerns in an explicit command preserves the current suite's speed and accessibility. + +#### Stop at black-box process and PTY tests + +This would prove discovery, consent, hook output, and persisted state without provider cost. It would not prove that a capability installed by Symposium is visible inside a supported agent. One bounded nonce query is retained because crossing that final boundary is a central claim of the integration. + +#### Describe complete journeys as data + +A fully declarative format could be serialized and generated by external tools. It would also require a scenario interpreter and concentrate errors from an entire journey at the interpreter boundary. Declarative registration metadata is retained for preflight and discovery, while an imperative asynchronous Rust body provides normal control flow and local `?` failure sites. + +#### Add persistent agent sessions immediately + +Persistent sessions will be useful for confirmation and restart journeys that span multiple agent turns. The tracer uses one query, so a session abstraction would be designed without an exercising scenario. The adapter begins with a bounded single-query capability and grows only when a committed journey requires persistence. + +#### Make the container conditional on host leakage + +The host canary can show whether a named decoy entered the controlled agent home. It cannot control installed tools, networking, system libraries, or operating-system behavior. The Linux container is therefore retained as a reproducibility boundary even when the host canary passes. + +#### Do nothing + +The current suite would remain unable to select both consent outcomes through the terminal, detect some compiled-process output failures, or prove that an installed capability reaches a real agent. Those are the specific blind spots this RFD exists to close. + +### Prior art + +[`cli-testing-library`](https://github.com/crutchcorn/cli-testing-library) provides a useful interaction vocabulary based on querying visible screen state and sending user events. This RFD adopts that model for parsed PTY interaction, but not its Node implementation or platform limitations. + +[`cli-testing-specialist`](https://github.com/sanae-abe/cli-testing-specialist) demonstrates generated tests for general CLI behavior. Symposium journeys need product-specific fixtures, discovery contracts, persisted consent, hook protocols, MCP evidence, and agent delivery, so generic command validation is not the central abstraction here. + +The existing Symposium integration harness supplies the fixture composition and deterministic assertions that the new runner reuses. Its strengths argue for an additive frontend rather than replacement; its inability to provide controlled interactive and agent boundaries identifies where the addition begins. + +Rust's experimental [libtest JSON output RFC](https://rust-lang.github.io/rfcs/3558-libtest-json.html) separates structured test events from presentation and validates a new harness interface before stabilization. This RFD follows the same lessons through a structured evidence channel and an experimental runner, without adopting libtest's event protocol. + +### Unresolved questions + +No design question currently blocks acceptance. The tracer contract, scope, evidence layers, and ownership boundaries are defined. + +Implementation must still establish: + +- which PTY implementation satisfies the parsed-terminal contract on the initial host platform; +- whether the Claude adapter can report trustworthy usage and expose the controlled custom-skill inventory without agent-specific behavior leaking into scenarios; and +- the measured cold preparation, warm startup, provider usage, and total cost of the container-backed witness. + +These measurements may refine internal interfaces and limits. If an implementation result makes a required witness or isolation guarantee infeasible, the contract returns to discussion rather than being weakened silently. + +Scheduling, release gating, persistent sessions, additional adapters, and native operating-system coverage are deliberately deferred. They are future design questions, not acceptance blockers for the tracer. + +### Future possibilities + +Additional registry journeys can register new metadata and bodies against the same `ScenarioContext`. New agents can implement the adapter contract without adding agent-specific paths to scenarios. Persistent conversations can become an adapter capability when the first multi-turn journey supplies a concrete test. CI and release policy can be designed from measured reliability, runtime, and cost instead of estimates. + +These extensions build on the process, scenario, environment, adapter, and evidence boundaries established here; none requires replacing the tracer architecture. They are not commitments of this RFD and are not independent reasons to accept it. + +### Proposed documentation + +The [agent interaction test guide](./proposed-guide/README.md) is written as the developer documentation should read once the experimental command exists. It explains scenario discovery, host and container execution, paid-run confirmation, budgets, results, artifacts, scenario authoring, and the initial CI boundary. + +## Frequently asked questions + +### What does a passing real-agent journey prove? + +It proves the contracts named by that journey and only those contracts. For the tracer, matching terminal, event, exit, and state evidence proves the consent behavior; an exact controlled inventory plus the scenario nonce proves that the fixture skill crossed into the selected agent. It does not prove general response quality or all registry behavior. + +### Is this an agent-effectiveness evaluation? + +No. Effectiveness evaluation compares outcomes, needs a baseline and repeated samples, and may judge code quality. This harness verifies that specified Symposium interactions and delivery boundaries work. Effectiveness studies may later use the harness as execution infrastructure, but their claims and methodology remain separate. + +## Implementation plan and status + +Implementation has not begun. Each step leaves the repository with an independently useful, passing result. + +### Step 1: Fix hook stdout at the process boundary + +Change hook execution so stdout contains only the selected agent's protocol payload. Add a deterministic black-box regression that spawns the compiled hook command with piped stdin, stdout, and stderr. + +The new agent-test runner, PTY support, containers, and agent adapters remain absent. + +- [ ] Verify that stdout parses as the expected hook protocol and that human report output is absent. +- [ ] Run the existing hook and integration tests. + +### Step 2: Run the host consent journeys + +Add scenario registration metadata, the constrained asynchronous `ScenarioContext`, the `cargo xtask agent-test` frontend, the structured side channel, and the minimum PTY driver needed for dependency-consent accept and decline. Reuse the current fixture composition and assertion helpers. + +Containers and real-agent execution remain absent. Reconcile the contract table with the executable scenario names after the journeys pass; do not add catalog code generation. + +- [ ] Verify accept and decline against terminal anchors, structured events, exit status, configuration, filesystem state, the host-state canary, and exact fixture-controlled custom-skill inventory. +- [ ] Verify every `Covered` contract row names an executable scenario and every `Gap(issue)` row names an issue and failing reproducer. + +### Step 3: Run the same journeys in Linux + +Add Docker execution, a content-addressed Linux Symposium binary, least-privilege container rules, disabled scenario networking, cleanup, and infrastructure diagnostics. Run the Step 2 scenario bodies unchanged. + +Provider egress, credentials, and real-agent execution remain absent. + +- [ ] Verify cold preparation and warm startup separately. +- [ ] Verify the host-state canary, custom-skill inventory, cleanup, and parity with the remaining host assertions. + +### Step 4: Add one real-agent delivery witness + +Add the bounded Claude adapter and extend the container-backed accepted branch with one fixture-skill query. Pin the runtime and add only the allowlisted provider egress and restricted API-key handling required by that query. + +Persistent conversations, other agents, scheduled execution, and release gating remain absent. + +- [ ] Verify the capability nonce, exact pre-query custom-skill inventory, installation and hook-registration evidence, usage limits, redaction, cleanup, and error classification. +- [ ] Record phase timing, provider usage, and conservative cost from an explicitly confirmed manual run without automatic paid retries. + +Before closing the RFD, correct `md/design/running-tests.md` so it documents the `SYMPOSIUM_ENABLE_AGENT_TESTING` gate. Keep `TestMode::AgentOnly`, `test-agents.toml`, and `tests/agent_harness/run_scenario.py` temporarily for existing Claude and ACP coverage, but mark that path as superseded and add no new scenarios to it. File its removal with the ACP follow-up after the remaining scenarios migrate. + +Closing the RFD also requires follow-up issues or RFDs for catalog automation, fake and ACP conformance, remaining scenario families, native operating-system expansion, and release CI graduation. diff --git a/md/rfds/agent-interaction-testing/agent-adapters/README.md b/md/rfds/agent-interaction-testing/agent-adapters/README.md new file mode 100644 index 00000000..0f2b8adc --- /dev/null +++ b/md/rfds/agent-interaction-testing/agent-adapters/README.md @@ -0,0 +1,58 @@ +# Agent adapters + +## Adapter contract + +An agent adapter implements the `AgentDriver` boundary. Scenarios request capabilities through `ScenarioContext`; they do not call an adapter or agent SDK directly. + +An `AgentDriver`: + +- reports its capabilities and supported witness forms; +- prepares only agent-specific runtime and authentication state; +- runs a bounded query and waits for protocol completion; +- reports input, cache-read, cache-write, and output usage for every provider request; +- enforces supported output and operation limits and responds to runner cancellation; +- applies scenario-declared permission policy; and +- returns normalized events plus sanitized raw provider artifacts. + +The driver interface is capability-based. A scenario requiring an unsupported capability or witness is `Unavailable` for that adapter rather than weakened to a filesystem-only check. An adapter without trustworthy usage accounting cannot run a paid tracer query. The [result contract](../evidence/README.md#results-and-failure-ownership) owns this classification. + +## Adapter scope + +| Adapter | Role | Scope | +|---|---|---| +| Claude | First production adapter | One fresh, bounded skill-delivery query | +| Fake | Deterministic contract adapter | Follow-up before stabilizing `AgentDriver` | +| ACP fixture | Second protocol adapter and future session input | Follow-up | + +The Claude adapter uses its structured SDK so completion, tool activity, and provider usage are observable. The tracer does not add a separate interactive-entry-point smoke test. + +The tracer needs one fresh, bounded Claude query to prove delivery of the fixture capability. The provisional driver therefore does not introduce persistent-conversation machinery. A later scenario that genuinely depends on multiple turns or an agent restart must add that capability deliberately and test it before the driver contract grows to include it. + +The tracer implements Claude behind a provisional capability-based interface. It does not claim cross-agent behavioral consistency from one production adapter. + +Before the interface is declared stable, a follow-up adds deterministic fake adapters for success, failure, timeout, malformed-event, and missing-capability paths. The existing persistent ACP path can then inform a separately tested session capability and a second adapter. Only the bounded Claude query belongs to the tracer. + +## Capability witnesses + +Every real-agent journey defines bounded evidence that an installed capability crossed into the running agent: + +- A skill witness exposes a scenario nonce through a structured load or tool event, with a narrow exact response as fallback. +- A hook witness is the corresponding hook trace. +- An MCP witness is the fixture server's initialization, tool-list, or explicitly requested tool-call log. +- A subcommand witness is the observed subprocess invocation and exit status. + +Capability witnesses do not grade general prose, code quality, or whether Symposium makes an agent write better Rust. Those are effectiveness-evaluation questions outside this RFD. + +## Permissions + +Every real-agent scenario declares allowed read and write roots, executable commands, MCP tools, and non-provider network access. Unexpected requests are denied and fail the scenario. Adapters normalize permission requests and outcomes into the event journal and may not automatically select an arbitrary approval option. + +The runner also verifies that the agent did not write outside allowed roots or leave undeclared processes. A denial scenario names the forbidden operation and expected result explicitly. Witnesses use the least powerful operation available. + +## Runtime reproducibility + +Pinned conformance runs fix the agent CLI or SDK version, dependency lock, base-image digest, and model identifier where the provider supports one. Results record requested and provider-returned model metadata, agent version, Symposium revision, and scenario version. + +A follow-up may add a smaller latest-agent canary for current supported releases. Canary failures indicate upstream compatibility work and do not rewrite pinned conformance results. Advancing a pin is a reviewed compatibility change. + +Provider behavior cannot always be frozen completely. Assertions therefore remain limited to stable protocol boundaries and capability witnesses. diff --git a/md/rfds/agent-interaction-testing/coverage-and-ci/README.md b/md/rfds/agent-interaction-testing/coverage-and-ci/README.md new file mode 100644 index 00000000..5a82d3c7 --- /dev/null +++ b/md/rfds/agent-interaction-testing/coverage-and-ci/README.md @@ -0,0 +1,110 @@ +# Coverage and CI + +## Contract table + +The tracer begins with a reviewed Markdown table of the Symposium promises it exercises. Each row has a stable rule identifier, a behavioral statement, its required test layers, and one state: + +- `Committed(step)`: this RFD commits to implementing the row in the named tracer step. It becomes `Covered` after the required scenarios pass. +- `Covered`: every required tracer scenario exists and passes. +- `Gap(issue)`: the implementation is known to violate the specification, a linked issue owns the discrepancy, and an executable reproducer returns `Failed` when run directly. +- `Direction(follow-up)`: the rule is outside this RFD's tracer commitment and must be carried into a closing follow-up issue or RFD. It is not counted as tracer coverage. + +Accepted RFDs and current reference documentation remain authoritative; this table is a coverage index rather than a replacement specification. It must not copy an implementation bug into the expected result. + +Typed Rust scenarios name the rules they prove. After enough journeys exist to expose stable catalog requirements, a follow-up may make the table machine-readable, validate layer and operating-system obligations, and generate a coverage report. This RFD does not build that meta-tool before the first journey. + +## Coverage layers + +The matrix separates the boundary being exercised from the command used to run the test. In-process tests call Symposium's Rust entry points. Real-process tests spawn the compiled binary, using a PTY only for interactive commands and pipes for hooks and other noninteractive subprocesses. Real-agent tests are reserved for delivery that can fail only inside the agent. A deterministic black-box process regression may still run under `cargo test`; it does not become an agent test merely because it spawns a binary. + +The matrix records intended obligations, including follow-on direction; it does not claim those rows are implemented by the tracer. Linux containers rehearse representative production paths rather than duplicating every in-process test. + +| Rule ID | Contract | State | In-process | Real process | Real agent | +|---|---|---|---:|---:|---:| +| `consent.accept` | Undecided candidate is accepted | `Committed(steps 2, 3)` | required | required, PTY | not required | +| `consent.decline` | Undecided candidate is declined | `Committed(steps 2, 3)` | required | required, PTY | not required | +| `consent.defer` | Ask later records nothing | `Direction(follow-up)` | required | required, PTY | not required | +| `cli.noninteractive` | Noninteractive execution never prompts | `Direction(follow-up)` | required | required, pipes | not required | +| `enablement.disable-precedence` | Disable overrides other enablement | `Direction(follow-up)` | required | representative, pipes | not required | +| `cache.expiration` | Cache expiration reevaluates its input | `Direction(follow-up)` | required | required, pipes | not required | +| `hook.stdout-protocol` | Hook stdout contains only protocol output | `Committed(step 1)` | not sufficient | required, pipes | not required | +| `isolation.skill-inventory` | Isolated custom-skill inventory exactly matches the fixture | `Committed(steps 2, 3, 4)` | not required | required, pipes | required | +| `delivery.skill` | An enabled fixture skill reaches the selected agent | `Committed(step 4)` | not sufficient | required, pipes | one delivery smoke | +| `use.search-endpoint` | Non-workspace `use` search uses only its declared fixture endpoint | `Direction(follow-up)` | required | required, pipes | not required | +| `delivery.hook` | Hook delivery reaches an agent | `Direction(follow-up)` | required | representative, pipes | required | +| `delivery.mcp` | MCP registration reaches an agent | `Direction(follow-up)` | required | representative, pipes | required | + +Operating-system applicability is recorded separately. A Linux-container pass cannot satisfy a Windows-native or macOS-native requirement. + +The first audit must include conflicting enablement entries. The accepted registry contract says `disable` wins over `use` and `auto-enable`, even though one current implementation path checks `use` first. + +## Tracer journeys + +This RFD commits to two fresh-state consent journeys. The accepted branch runs real `init` and `sync`, answers the prompt, and verifies installation. Step 4 extends that branch with one bounded Claude query that proves the nonce-bearing fixture skill reached the agent. The declined branch verifies that nothing is installed, the decision persists, and a later sync does not ask again. + +The tracer also fixes the hook stdout contamination bug and adds a deterministic black-box process regression proving that stdout contains only the hook protocol. It plants an isolation canary and verifies the exact custom-skill inventory owned by the fixture. + +Ask-later and Escape, custom predicates, cache reuse and expiration, malformed registries, enablement precedence, non-workspace `use`, hook delivery, MCP delivery, and registry resynchronization remain stated follow-on families. They use deterministic or real-process coverage by default. A real agent is added only where delivery into the agent could fail. + +## Command interface + +The orchestration entry point is: + +```console +cargo xtask agent-test [OPTIONS] +``` + +Initial options are: + +```text +--list +--agent +--environment +--scenario ... +--symposium-bin +--auth +--max-agent-turns +--max-provider-requests +--max-input-tokens +--max-output-tokens +--max-tool-calls +--confirm-paid-run +--keep-artifacts +``` + +`--scenario` is repeatable. No scenario means "print the execution plan," not "start an agent." A selection containing a real-agent journey requires an explicit agent and `--confirm-paid-run`. Missing runtime, credentials, or capability yields `Unavailable`; a requested container never silently falls back to the host. + +The plan reports CLI-only and real-agent scenarios, scenario and operator token limits, maximum turns, provider requests, and tool calls, the provider-side spending cap, environment, binary provenance, and pinned runtime before execution. + +## Cost and runtime controls + +Each real-agent scenario declares maximum cumulative input, cache-read, cache-write, and output tokens; provider requests; user turns; tool calls; real-time deadline; and usage class. Paid execution requires trustworthy provider accounting. Cached tokens remain visible and count toward token limits even when their billable price is lower. + +Scenario-owned limits define the product contract. Exceeding one is `Failed`. Operator flags and run-wide limits are protective ceilings. If a lower `--max-agent-turns` or run-wide cap stops an otherwise valid scenario, the result is `InfrastructureError` owned by `runner.budget`, never a Symposium failure. The execution plan shows both limits and their effective minimum before paid work begins. + +Before any provider request, the runner checks the declared fixture and prompt inputs it controls. The adapter reports actual provider usage for agent-owned context. A scenario that cannot fit its declared limit is reduced before the limit is raised. + +Authoritative paid tracer runs use a dedicated restricted provider key with a $5 monthly provider-side spending limit as the aggregate backstop. A host run using `--auth local` is non-authoritative, cannot claim that provider cap, and still obeys the scenario's hard token and operation limits. Real-agent concurrency is one. Credentials alone never enable paid tests: the user must select a real-agent scenario, name the agent, and pass `--confirm-paid-run`. Ordinary `cargo test` retains its explicit agent-testing gate. + +Runtime reporting separates checkout build or image preparation, warm environment startup, agent execution, and assertion/evidence processing. This makes container overhead distinguishable from provider latency. + +The tracer's provisional guard permits one user turn, at most four provider requests, three tool calls, 25,000 total input-side tokens across base input, cache reads, and cache writes, and 1,000 cumulative output tokens. Initial manual runs record actual usage so these limits can be reduced. If the pinned runtime cannot produce the nonce witness within the guard, the prompt, tools, fixture, and context are reduced before any limit is raised. + +At Sonnet 5's standard price after its introductory period, $3 per million input tokens and $15 per million output tokens, the provisional base-token ceiling is approximately $0.09 per run. The runner reports a conservative allowance of $0.20 per run for cache-price differences while the provider key caps the month at $5. The estimate is recalculated when the model pin or [provider pricing](https://www.anthropic.com/research/claude-sonnet-5) changes; tokens remain the primary limit and dollars are derived reporting. + +## Tracer CI boundary + +- Fast deterministic tests block every pull request. +- Stable native agent-free process and PTY scenarios may become PR-blocking. +- A small agent-free Linux-container suite may graduate if its measured runtime is acceptable. +- Real-agent tracer journeys are explicitly selected, manually run, and non-gating while the runner is experimental. + +Agent-free tests graduate after an observation period with no unexplained flakes, acceptable runtime, actionable failure artifacts, reliable cleanup, and consistently successful secret-canary validation. Provider credentials are never exposed to fork pull requests. + +Scheduled ownership, quarantine, pass-rate, and release-gating policy for real-agent tests are outside the tracer contract. A follow-up may propose those mechanisms using measured reliability, runtime, and cost rather than assumptions. + +## Post-tracer direction + +After the tracer, tracked follow-ups can expand the contract table across registry, discovery, predicate, cache, and delivery behavior. They can add every consent branch, representative Linux-container and native Windows/macOS lanes, fake and fixture-ACP adapter contracts, and selected hook and MCP witnesses. + +Catalog automation, full release reporting, a latest-agent canary, and broader CI graduation are separate commitments informed by tracer evidence. They are not acceptance criteria for this RFD. diff --git a/md/rfds/agent-interaction-testing/environments/README.md b/md/rfds/agent-interaction-testing/environments/README.md new file mode 100644 index 00000000..d2d9be5f --- /dev/null +++ b/md/rfds/agent-interaction-testing/environments/README.md @@ -0,0 +1,67 @@ +# Execution environments + +## Environment backends + +| Backend | Purpose | Isolation claim | +|---|---|---| +| Host | Fast local iteration and native process behavior | Fresh test state, but installed tools and the host operating system remain observable | +| Linux container | Reproducible Linux conformance | Fresh restricted container with controlled tools, filesystem, and networking | + +The host backend creates fresh project, home, configuration, cache, and temporary directories and passes an explicit filtered environment to every child process. Host results remain non-authoritative because installed tools and the operating system can influence them. + +The Linux container backend is the isolated conformance environment. The tracer uses Docker behind an interface that can later support another container runtime, a VM, or a remote worker. Linux-container results are authoritative only for Linux. + +Windows and macOS coverage is follow-up work. Native deterministic and real-process/PTY lanes use fresh test directories and cover platform-specific paths, command dispatch, shell behavior, PTYs, permissions, and process handling without claiming container-strength isolation. Native real-agent smokes require trusted runners and credentials. + +## Scenario isolation + +Every scenario and retry attempt receives unique workspace, home, cache, network, container, service, trace, and artifact identifiers. A fresh container is created per scenario, while deliberate CLI and agent restarts within that scenario retain its writable state. + +Fixtures are copied into the environment rather than mounting the repository. Containers run as non-root with a read-only root filesystem, dropped capabilities, no Docker socket, explicit writable directories, and CPU, memory, process, and time limits. + +Scenarios never mutate the runner's process-global environment. Concurrent scenarios share only immutable or content-addressed build assets. Image and binary preparation use an interprocess lock and publish a completed read-only artifact. Services use isolated networks and dynamic host ports. Resources are labelled by run and scenario, and cleanup is idempotent. + +Every host and container run tests host-state exclusion. The runner places a harmless synthetic capability canary in a harness-owned decoy host configuration that is outside the fresh scenario home and, for local-auth tests, exercises the adapter's filtered credential bridge. It never writes a canary into the developer's real home. The runner asserts that the canary is absent from copied homes, agent-visible custom capabilities, and persisted artifacts. + +The runner also asserts that the custom-skill inventory in every harness-controlled user and project scope exactly matches the fixture, including before every real-agent query. Agent-provided built-in capabilities are recorded separately and are not claimed as host state. + +Agent concurrency is capped separately from CLI-only concurrency. A scenario may request an exclusive resource only for an external tool that genuinely cannot be isolated. + +## Symposium binary provenance + +By default, the runner builds one Linux `cargo-agents` artifact from the current checkout and reuses it across selected container scenarios. Its content-addressed key includes: + +- source revision or dirty-source digest; +- `Cargo.lock` digest; +- Rust toolchain; +- target triple; +- profile and feature set; and +- container base-image digest. + +Dependency and intermediate build layers are reused, so preparation is incremental rather than a per-scenario copy pause. + +`--symposium-bin` is an explicit override, never an automatic PATH lookup. The runner checks its executable format, operating system, architecture, and available version metadata. Results record the binary digest and provenance as a checkout build or explicit override. Host runs apply the same checks to local artifacts. + +## Provisioning and initialization + +Placing the checkout binary on the isolated PATH is harness preparation, not an installer assertion. The tracer does not test `cargo install`, `cargo binstall`, release archives, or a future package manager. + +Symposium initialization is product behavior. Fresh-user scenarios run the real `cargo agents init` process and assert configuration creation, agent setup, hook installation, optional choices, repeated-init idempotency, and later hook activation. Distribution paths can later feed the same post-provisioning scenarios. + +## Network and fixture trust + +Tracer fixture content is repository-owned and reviewed. An extension marked untrusted is awaiting Symposium consent; it is not arbitrary hostile code. Testing malicious hooks, MCP servers, agents, or fixtures requires a separate security-testing design. + +The agent-free tracer container runs with networking disabled. Rust projects use path dependencies or a controlled local registry prepared before scenario execution. Later registry, git-source, MCP, hook, predicate, and failure scenarios may add declared local fixture processes or pinned image tools. A `cargo agents use` scenario that searches for a non-workspace crate must route `CargoPm::search` to a declared local fixture endpoint; it never queries crates.io. Fixture services can return exact versions, delays, disconnects, malformed data, and cache validators. + +Step 4 adds only the provider egress needed by the real-agent query. The scenario container has no direct external egress. HTTPS provider traffic crosses a CONNECT proxy with an allowlist of destination hosts and ports. The proxy does not terminate TLS and the harness installs no interception certificate authority. Any fixture service remains on the internal network. Any additional external endpoint must be declared in the scenario and recorded in the execution plan and manifest. + +Image construction and dependency acquisition finish before scenario execution and are identified by lockfiles and digests. Network failures are simulated through fixture services rather than public outages. + +## Authentication + +Container conformance uses a restricted API key available only to trusted jobs and never mounts local agent state. + +Host authentication is explicit: `--auth api-key` or `--auth local`. API-key mode uses a fresh agent home. Local mode bridges only the minimum adapter-supported credential material, read-only, while agent settings, Symposium configuration, skills, hooks, MCP configuration, caches, and query history remain fresh. + +If an agent cannot separate credentials from user configuration, the base result carries the [`non-authoritative(contaminated-auth-context)` modifier](../evidence/README.md#results-and-failure-ownership). The manifest records the mode and inherited credential paths without their contents. diff --git a/md/rfds/agent-interaction-testing/evidence/README.md b/md/rfds/agent-interaction-testing/evidence/README.md new file mode 100644 index 00000000..a8bfbeb7 --- /dev/null +++ b/md/rfds/agent-interaction-testing/evidence/README.md @@ -0,0 +1,106 @@ +# Evidence and results + +## Dual observation + +Interactive journeys observe two channels from the same process: + +- The normal PTY proves that a suggestion or prompt was visible and accepted real user input. +- A harness-controlled JSONL side channel reports stable Symposium decisions and state transitions. + +The side channel attaches an additional reporting sink. It must not enable quiet mode, bypass confirmation, change command decisions, or write into a hook's protocol stdout. Events needed by the initial scenarios include discovery, confirmation requested and answered, enablement, installation, and hook dispatch. Later scenarios may add predicate and cache events without changing the product-facing stream. + +Terminal assertions use only the stable text necessary to prove that a user could understand and answer the prompt. Colors, wrapping, and complete screens are not ordinary snapshots. Detailed behavioral assertions use structured events and final state. + +## Canonical event journal + +The runner merges harness events, Symposium side-channel events, agent protocol events, and observed state changes into one coarse journal while retaining sanitized source artifacts. + +Each event envelope contains: + +- schema version; +- run, scenario, and attempt identifiers; +- source and source-local sequence; +- correlated operation identifier; +- event kind; +- real monotonic offset; and +- normalized payload. + +A confirmation event has this shape: + +```json +{ + "schema_version": 1, + "run_id": "run-fixture-01", + "scenario_id": "dependency-consent-accept", + "attempt": 1, + "source": "symposium", + "source_sequence": 4, + "operation_id": "sync-01", + "kind": "confirmation.answered", + "monotonic_offset_ms": 184, + "payload": { + "decision": "enable" + } +} +``` + +Provider-operation events record requested limits and reported input, cache-read, cache-write, and output tokens. Aggregate token counts and derived cost are evidence, not estimates substituted for missing accounting. + +Sequence is strict within one source. Receipt order is diagnostic and does not imply causal order across processes. Assertions express partial order within a source or correlated operation, such as discovery before confirmation and confirmation before installation. Unrelated sources remain unordered unless explicitly correlated. + +Dynamic paths, process IDs, ports, and timestamps are normalized before comparison. Unknown additive event kinds are retained and ignored unless required. Breaking envelope changes increment the schema version. Event payloads use stable identifiers and exclude secrets. + +## Assertions + +Authoritative assertions prefer: + +- exact configuration and allowed filesystem state; +- process exit status and normalized system events; +- protocol completion and tool activity; +- hook stdout containing only the selected agent's protocol output; +- discovery, consent, predicate, and cache decisions; and +- hook, skill, MCP, and subcommand capability witnesses. + +Model prose is checked only through a narrow fixture-defined nonce or fact when that is the available witness. Full responses are diagnostic, not gating. + +## Results and failure ownership + +A run has four base results: + +| Result | Meaning | +|---|---| +| `Passed` | The requested journey and assertions completed. | +| `Failed` | The environment ran, but Symposium or the interaction violated the contract. | +| `InfrastructureError` | Credentials, provider, runtime, environment, runner budget, or harness failed. | +| `Unavailable` | Preflight found that the selected adapter or environment lacks a required capability. | + +A result may also carry modifiers that preserve important qualifications without creating another base result: + +- `non-authoritative(contaminated-auth-context)` means local credentials could not be separated from user or agent configuration. +- `stability-warning(recovered-infrastructure-error)` means a recognized infrastructure failure occurred before the complete fresh-state retry passed. + +Modifiers are recorded in the summary, journal, and aggregate reports. They never turn `Failed` into `Passed` or make a non-authoritative run satisfy a conformance requirement. + +Explicitly requesting an unavailable combination exits unsuccessfully; ordinary `cargo test` remains unaffected. There is no expected-failure scenario result. A known product-gap reproducer still returns `Failed` when run directly. + +The [coverage table](../coverage-and-ci/README.md#contract-table) records a `Gap(issue)` separately from completed tracer coverage. Its executable reproducer still returns `Failed`; the result vocabulary has no expected-failure state. + +Failures name an owning phase such as `environment.prepare`, `symposium.cli`, `symposium.state`, `agent.start`, `agent.query`, `fixture.mcp`, `assertion`, or `cleanup`. A Symposium crash, missing prompt, wrong state, or completed agent query without its required witness is `Failed`. + +Exceeding a scenario-owned token or operation limit is `Failed` because the witness did not fit its contract. Harness-controlled context that already exceeds the declared limit or an operator ceiling stopping the run is `InfrastructureError` owned by `runner.budget`. Missing trustworthy provider accounting makes a paid query `Unavailable`. + +## Retries and cleanup + +Assertion and Symposium failures are never retried automatically. An agent-free scenario with a recognized transient infrastructure error may retry once from fresh state. A scenario that contacted a paid provider is never retried automatically; another attempt requires a new explicit invocation. Individual steps are never replayed inside an existing container or query context. + +Both attempts are preserved. A recovered run remains `Passed` with the `stability-warning(recovered-infrastructure-error)` modifier and attempt metadata, so infrastructure reliability still counts the transient. + +On a deadline, the runner captures current evidence, attempts graceful termination, kills the complete process tree after a bounded cleanup deadline, and verifies that no process or container remains. + +## Artifact safety + +Artifacts live under `target/agent-tests//`. Every run keeps a compact summary. Failures keep sanitized journals, terminal output, selected logs, workspace diffs, agent events, explicitly allowed Symposium state, and a redaction report. Successful runs keep rich artifacts only with `--keep-artifacts`. + +Capture is allowlist-based. The runner never archives a complete container, home, authentication directory, or process environment. Credentials are secret handles supplied only to the process that needs them. Known values, provider headers, credential-bearing URLs, command arguments, and environment fields are redacted in memory before persistence. + +Every run injects harmless secret canaries and verifies that none survive. If sanitization cannot complete, rich artifacts are withheld and the summary reports the redaction failure. Upload-time filtering is not considered sufficient. diff --git a/md/rfds/agent-interaction-testing/proposed-guide/README.md b/md/rfds/agent-interaction-testing/proposed-guide/README.md new file mode 100644 index 00000000..62fac6be --- /dev/null +++ b/md/rfds/agent-interaction-testing/proposed-guide/README.md @@ -0,0 +1,115 @@ +# Agent interaction tests + +Agent interaction tests exercise Symposium with scripted users, real processes, and selected real coding agents. They complement ordinary tests: use deterministic tests for exhaustive Symposium logic and these journeys when the process, terminal, user, or agent boundary is itself under test. + +The feature is experimental. Real-agent runs consume provider capacity and are opt-in. + +## Discover scenarios + +```console +cargo xtask agent-test --list +``` + +The scenario list reports required agent, environment, operating-system, and witness capabilities. The contract table maps the tracer's Symposium promises to executable scenarios and linked product gaps. + +Running `cargo xtask agent-test` without a scenario prints an execution plan and does not start an agent. + +Repeat `--scenario` to select more than one journey: + +```console +cargo xtask agent-test --agent claude --environment container --auth api-key --confirm-paid-run --scenario dependency-consent-accept --scenario dependency-consent-decline +``` + +## Run on the host + +```console +cargo xtask agent-test --agent claude --environment host --auth local --confirm-paid-run --scenario dependency-consent-accept +``` + +The host runner creates fresh project, Symposium, agent, cache, and temporary directories. Local authentication is used only when explicitly requested. If the adapter cannot separate credentials from normal agent configuration, the result is marked non-authoritative. Local authentication also cannot claim the tracer key's $5 provider cap; the execution plan reports that limitation while retaining the scenario's hard token and operation limits. + +Host runs are useful for debugging but may still be affected by installed tools and the operating system. + +## Run Linux conformance + +```console +$env:ANTHROPIC_API_KEY = "..." +cargo xtask agent-test --agent claude --environment container --auth api-key --confirm-paid-run --scenario dependency-consent-accept +``` + +The runner prepares a pinned base image and one content-addressed Linux `cargo-agents` build from the checkout. Each scenario receives a fresh restricted container. Scenario runtime is hermetic except for the selected agent provider. + +To test an existing compatible Linux artifact: + +```console +cargo xtask agent-test --agent claude --environment container --auth api-key --confirm-paid-run --symposium-bin ./artifacts/cargo-agents-linux-x86_64 --scenario dependency-consent-accept +``` + +The override is checked for operating system, architecture, executable format, and available version metadata. The runner never substitutes a released package, PATH binary, host environment, or different execution backend silently. + +## Read the execution plan + +Before a paid run, the plan reports information such as: + +```text +Selected scenarios: 2 +CLI-only scenarios: 1 +Real-agent scenarios: 1 +Maximum agent turns: 1 +Maximum provider requests: 4 +Maximum tool calls: 3 +Input-side token guard: 25,000 +Output-token guard: 1,000 +Per-run cost allowance: $0.20 +Monthly provider cap: $5.00 +Environment: Linux container +Agent/runtime: Claude, pinned +``` + +The [cost and runtime controls](../coverage-and-ci/README.md#cost-and-runtime-controls) are authoritative. The values above are the initial experimental tracer defaults. + +Real-agent scenarios enforce cumulative input, cache-read, cache-write, and output tokens as well as provider-request, turn, tool-call, deadline, and run-wide limits. Cached tokens still count even when they cost less. A paid run requires explicit selection, an agent name, and `--confirm-paid-run`. + +The initial tracer permits one user turn, at most four provider requests, three tool calls, 25,000 total input-side tokens, and 1,000 output tokens. At the standard post-introductory Sonnet 5 price, its base-token ceiling is approximately $0.09 per run. Its conservative allowance including cache-price differences is $0.20, and its dedicated provider key has a $5 monthly cap. Initial manual runs record usage so the limits can be reduced. The prompt and fixture are reduced before a limit is raised. + +## Read a result + +Each run ends as: + +- `Passed`: the journey and assertions succeeded. +- `Failed`: the environment worked, but the behavior violated the contract. +- `InfrastructureError`: setup, authentication, provider, runtime, harness, or an operator-imposed budget stopped the run. +- `Unavailable`: the selected combination lacks a required capability. + +Results may carry modifiers. `non-authoritative(contaminated-auth-context)` means local authentication could not be isolated from agent configuration. `stability-warning(recovered-infrastructure-error)` means a complete fresh-state retry recovered from a recognized infrastructure failure. A modifier cannot turn a product failure into a pass or satisfy a conformance requirement with non-authoritative evidence. + +A scenario that cannot produce its witness within its own token budget is `Failed`. Oversized harness context or a lower operator limit is `InfrastructureError` owned by `runner.budget`. Paid execution is `Unavailable` when the adapter cannot report trustworthy usage. + +The summary also names the owning phase. An agent-free scenario may retry once from fresh state after a recognized transient infrastructure error. A scenario that contacted a paid provider is never retried automatically. Product failures and individual steps are never retried. A known-gap reproducer still returns `Failed` when run directly. + +Artifacts are under `target/agent-tests//`. Failure artifacts contain only allowlisted, sanitized evidence and a redaction report. Complete homes, authentication directories, and process environments are never archived. Use `--keep-artifacts` to retain rich evidence for a passing run. + +## Write a scenario + +Each scenario registers declarative metadata and an asynchronous Rust body returning `Result`. The metadata names fixtures, requirements, contract IDs, permissions, budgets, and external endpoints so the runner can preflight without executing the body. The body uses a constrained `ScenarioContext`; it cannot reach undeclared host state, credentials, or agent-specific APIs. Every behavioral branch is a separate fresh-state scenario. + +A typical consent journey: + +1. Compose a Rust fixture whose dependency embeds a plugin awaiting consent. +2. Start with empty Symposium and agent configuration. +3. Run real `cargo agents init --add-agent ` and assert setup. +4. Run real `cargo agents sync` under a parsed PTY. +5. Select the intended prompt option with explicit keys. +6. Assert terminal anchors, structured events, exit status, and final state. +7. Run one bounded agent query when delivery is under test. +8. Assert a narrow capability witness such as a fixture nonce, hook trace, or MCP server log. + +Scenarios declare contract IDs, required capabilities, permissions, scenario-owned token and operation budgets, and external endpoints. They do not contain Claude-specific paths or judge general response quality. An operator-supplied lower budget is shown separately and cannot manufacture a Symposium failure. + +Time-dependent scenarios mutate controlled persisted inputs instead of sleeping or changing the production clock. They may set a cache expiry into the past, write a fixture `state.toml`, set a file mtime, or disable the sync debounce. Process and agent deadlines always use real monotonic time. + +## CI operation + +Fast ordinary tests block pull requests. Stable agent-free process, PTY, and small Linux-container scenarios may graduate after meeting runtime and reliability criteria. Real-agent tracer journeys are manually selected and non-gating while the command is experimental. + +Scheduled execution, triage ownership, quarantine, pass-rate targets, and release gating are not part of the experimental command. They require a separate policy informed by measured reliability, runtime, and cost. diff --git a/md/rfds/agent-interaction-testing/scenario-model/README.md b/md/rfds/agent-interaction-testing/scenario-model/README.md new file mode 100644 index 00000000..12eead1b --- /dev/null +++ b/md/rfds/agent-interaction-testing/scenario-model/README.md @@ -0,0 +1,75 @@ +# Scenario model + +## Test frontends and shared engine + +Agent interaction tests extend `symposium-testlib`; they do not create a second fixture or assertion system. Ordinary tests and `cargo xtask agent-test` use the same fixture composition, scenario model, event vocabulary, and assertions. + +`cargo test` remains the frontend for deterministic and selected host scenarios. `cargo xtask agent-test` orchestrates environment selection, credentials, containers, filtering, real-agent execution, and artifact retention. A scenario does not change meaning when selected through a different frontend. + +## Scenario registration and body + +Each scenario has declarative registration metadata and an imperative Rust body. The runner can list and preflight the metadata without executing the body. The metadata declares: + +- a stable name and short description; +- fixture layers and controlled services; +- required environment and agent capabilities; +- a capability witness for every real-agent interaction; +- deadlines, resource limits, and scenario-owned agent budgets; +- a least-privilege permission policy; +- contract rule identifiers; and +- artifact-retention policy. + +The body is an ordinary asynchronous Rust function returning `Result`. It receives a constrained `ScenarioContext` for running commands, driving the terminal, querying an agent, mutating controlled fixture state, and making assertions. Using Rust control flow and `?` keeps a failure at the operation that caused it instead of reporting one interpreter failure for the whole journey. + +```rust,ignore +async fn dependency_consent_accept(cx: &mut ScenarioContext) -> Result<()> { + cx.run_init().await?; + let mut sync = cx.spawn_sync_pty().await?; + sync.wait_for("Enable this dependency?").await?; + sync.press_enter().await?; + cx.assert_skill_installed("fixture-skill")?; + Ok(()) +} +``` + +The body cannot access undeclared host paths, process-global environment, credentials, or agent-specific APIs. Those remain behind `ScenarioContext`, environment backends, and agent adapters. A paid query, external endpoint, fixture service, or privileged operation must be declared in metadata. The context rejects an operation that was not authorized by the registration metadata. + +Scenarios select capabilities, not agent brands. Agent-specific paths, authentication fields, event types, and witness mechanisms remain in adapters. + +Agent token budgets are cumulative across every provider request made for the journey, not merely the number of user-visible turns. They bound input, cache-read, cache-write, and output tokens separately, plus provider requests and tool calls. Cached tokens still count toward the token budget even when their dollar price is lower. + +## State and branching + +Each scenario follows one expected behavioral path. Accept, decline, ask-later, and Escape are separate scenarios that may reuse fixture descriptions but never writable state. + +Every scenario and retry begins with a fresh workspace, user configuration, agent configuration, cache, services, and agent query context. Steps within one scenario share state deliberately, including across declared process restarts. Persistence and cache scenarios express repeated commands in that one journey because preserved state is what they test. + +Scenario logic does not branch around unexpected output. A missing or different checkpoint fails at that step. + +## Production process boundary + +Authoritative user journeys invoke the compiled `cargo-agents` executable through the production-facing `cargo agents` command. Interactive commands run under a PTY. The runner captures the rendered terminal, sanitized raw bytes, exit status, structured events, and resulting state. + +The runner must not silently replace a requested process or PTY step with an in-process call. Existing deterministic tests may continue calling Symposium's Rust entry points directly, but that path does not prove Cargo dispatch, PATH setup, terminal interaction, hook subprocesses, or process exit behavior. + +## PTY scripting + +The PTY driver parses ANSI output into a rendered screen instead of treating the stream as plain stdout. Waits query narrow anchors in that screen. Input steps represent lines and explicit keys such as Enter, Escape, arrows, EOF, and interrupt. + +Ordinary scenarios use a fixed terminal size, UTF-8 locale, declared TERM and color mode, and the native PTY backend for the operating system, including ConPTY or equivalent on Windows. The terminal profile and backend are recorded with the result. + +Screen normalization handles cursor movement, redraws, color, and newline differences. Raw sanitized bytes remain diagnostic evidence. A small rendering suite separately tests color and resizing; ordinary journeys do not snapshot the complete screen. + +## Time-dependent scenarios + +Tests never synchronize with fixed sleeps. Every wait targets an observable condition and has a real monotonic deadline. + +This RFD does not add a production clock seam. Time-dependent tests mutate controlled persisted inputs before starting the command that observes them. Predicate-cache tests set the persisted expiry into the past, update-throttle tests set `state.toml`, filesystem tests set the relevant mtime, and sync tests may use `sync-debounce-secs = 0`. + +These mutations test the production comparison against the real wall clock without waiting for time to pass. If a later contract cannot be tested this way, its clock abstraction requires a separate design. Container, agent, TLS, provider, and process deadlines always use real time. + +## Metadata and body boundary + +Preflight needs declarative metadata before fixtures, containers, credentials, or paid agents are started. Journey execution benefits from ordinary Rust: compiler-assisted refactoring, direct reuse of test helpers, native asynchronous control flow, and line-local errors through `?`. + +Only registration metadata and the resulting execution plan are serializable. Scenario bodies are compiled Rust and are not loaded from TOML, YAML, or another scenario language. The constrained context preserves backend and adapter neutrality without exposing backend objects to the body. The [root rationale](../README.md#describe-complete-journeys-as-data) records why the design does not use a fully data-driven scenario format.