test(orka): add harness v2 E2E coverage for AgentKit runtimes

[sozercan]

Add automated tests that run AgentKit's real Python ACP child inside Orka's production harness v2 supervisor. Current CI can pass without exercising this composition, leaving the image entrypoint, runtime identity, provider proxy, MCP, and session lifecycle integration unverified.

Verified baseline

• Harness v2 support landed in feat(runtime): add Orka harness v2 ACP mode #22 at AgentKit commit 9a6de8f. The v2 integration runs agentkit-serve --protocol acp; --protocol orka selects the older harness v1 HTTP/SSE adapter.
• The focused ACP/CLI suite passed 57 cases. Coverage measured for acp.py was 574/660 executable lines, 87.0%, and 175/236 branches, 74.2%. These numbers describe that focused run, not repository-wide coverage or an existing CI coverage gate. The merged common-package CI suite also passed 676 tests.
• The passing live test on the merged commit used Vekil v0.14.3 and received DONE42. It tests one MAF OpenAI HTTP completion, without the Orka v2 supervisor. See the live script.
• The container smoke job checks startup for all three adapters. Its Orka-specific exercise uses harness v1, verifies authentication, and submits an already-expired turn.
• Orka has AgentKit-profile v2 authentication, lifecycle, and provider-proxy tests. Its test configuration uses a helper ACP process, so these tests do not establish compatibility with AgentKit's Python child.

Deliverable 1: deterministic offline v2 E2E

Create one reusable test entrypoint and fixture directory, for example scripts/orka-harness-v2-e2e.sh and test/orka-harness-v2/. Exact names can follow repository conventions.

1. Build the current AgentKit frontend and agent images using the existing Makefile/build flow. Exercise pydantic-ai, microsoft-agent-framework, and langgraph.
2. Layer Orka's production supervisor onto each built agent image using the existing docker-build-acp-agentkit-runtime target and official AgentKit composition Dockerfile. Resolve the AgentKit source image to an immutable digest; set AGENTKIT_RUNTIME_IMAGE to that reference and AGENTKIT_ADAPTER_DIGEST to the same digest. A run-scoped local registry is acceptable when needed for composition.
3. Fetch Orka at an explicit full commit SHA. The inspected compatible source baseline is 55cb3d5232b4a9b697e72471e346c0a6493d4c21; verify it during implementation and record the chosen pin. CI must not depend on an unpinned branch or a pre-existing sibling checkout.
4. Start the composed image on Linux with the production supervisor entrypoint, child process, identity separation, and session cleanup behavior. Use deterministic local provider and controller/broker fixtures. The supervisor's provider/MCP proxies and the framework's clients must remain real.
5. Drive the native v2 API with Orka's client/conformance implementation where practical. Orka's internal/harness/v2 packages cannot be imported directly from AgentKit's Go module; compile/run a thin probe within the pinned Orka module boundary if using those packages. Reuse canonical types rather than inventing a parallel wire contract.
6. Share the exerciser between offline and live modes. Keep the new coverage separate from the existing v1 and OpenAI HTTP smoke tests.

The exercised path must be:

v2 test driver
  -> production Orka supervisor
  -> actual AgentKit ACP stdio child / framework adapter
  -> supervisor provider proxy and prompt-scoped MCP server
  -> deterministic provider and broker fixtures

An offline echo runtime or Orka's helper ACP child does not satisfy this test. Offline means no external model credentials or model calls; initial image/dependency downloads are acceptable.

Required offline scenarios

Scenario	Observable proof
Startup and identity	v2 health/capabilities, expected adapter identity and profile, successful creation of the real ACP child, and initialization/session establishment.
Prompt completion	A prompt traverses the real provider proxy, the fixture observes the expected model/request, output carries a unique test marker, and the native terminal settlement indicates success.
Brokered MCP tool	The deterministic provider requests one allowed tool; a real MCP call reaches the broker fixture with the expected name/arguments/correlation; the result returns to the provider and affects the final answer. Verify tool event ordering and terminal outcomes.
Session continuation	Two successful prompts share one RuntimeSession. The provider observes the previous successful user/assistant history exactly once.
Failure and cancellation	Inject a provider/tool failure and cancel a genuinely running, blocked prompt. Assert native failure/cancellation outcomes and no conflicting success or duplicate terminal events. Check history rollback where the canonical session lifecycle permits continuation.
Deadline and cleanup	Expire a prompt after execution has started, then verify bounded settlement, expected child/session cleanup, and release of run-owned resources. Follow Orka's canonical rules for whether the session remains reusable.
Admission failures	Wrong authentication/fences and mismatched model/config identity are rejected at the appropriate boundary, with no unauthorized provider/tool work.

Use explicit synchronization in fixtures to prove a prompt/tool is running before cancelling or expiring it. Avoid timing-only sleeps or relying on a live model to remain busy.

Preserve the v2 contract

• providerKind is agentkit; adapter identity is agentkit-serve-acp.
• The registered model matches the baked model.name. agentConfigurationDigest is SHA-256 over the exact /agent/agent.yaml bytes.
• The baked agent config has no direct tools, baked brokered tools, or context providers. No per-Task AgentConfiguration overrides are introduced.
• Provider access uses the supervisor-injected loopback URL/token. The one prompt-scoped MCP server carries the supervisor's scoped authorization.
• supportsAgentSessionConfiguration remains false; approvalRequiredTools stays empty. Permitted brokered tools must match the registered policy and submitted allowedTools.
• Preserve the production Linux process/identity controls. Do not weaken authentication, fences, digest checks, or cleanup to make CI pass.

Deliverable 2: live v2 E2E

Add a live variant of the same composed-runtime test, initially using MAF and the existing Copilot-backed Vekil setup. The offline matrix supplies coverage for all three adapters; a three-adapter live matrix is not required for this issue.

• Reuse COPILOT_GITHUB_TOKEN from repository secrets and the existing digest-pinned Vekil image: ghcr.io/sozercan/vekil:v0.14.3@sha256:996b628fbe8c7a35d33e9d6bb855f2613228fc5c9b09498dae6ea6b208a0071b.
• Prove a real model response through harness v2, one brokered tool round-trip, and a second successful prompt in the same session.
• Exercise cancellation/deadline cleanup through a controlled blocking operation where practical; the deterministic offline suite remains the authoritative timing/lifecycle check.
• Use bounded waits and explicit assertions that each requested scenario executed. Model readiness or a green job alone is insufficient evidence.
• A missing secret on an untrusted/fork run may produce an explicit skip. With a configured secret, authentication/readiness/inference failures must fail the live check. Do not copy the older script's credential-rejection success/skip path into the new lane.
• Logs and artifacts must omit bearer tokens, session credentials, and raw authorization material. Record source/image digests, adapter, scenario outcomes, and sanitized diagnostics.

Deliverable 3: protocol rejection tests and CI wiring

Add behavior tests through the ACP public input/output interface for currently uncovered rejection paths, including:

• malformed JSON and invalid JSON-RPC envelopes/methods/IDs;
• duplicate active request IDs without disturbing the original request;
• unsupported protocol versions and session creation before initialization;
• invalid working directory/additional directories and malformed, excessive, or unsupported MCP server configuration.

Where applicable, send a valid request after rejection to prove the reader/server recovers and that rejected input did not start provider/tool work. Prefer these behavior checks over tests that merely mirror branches.

Wire the offline matrix into normal PR/push CI without a secret gate. Add the live lane with the repository's existing secret-access rules. Include syntax/lint checks for new scripts, bounded job timeouts, and run-scoped cleanup on success, failure, and interruption. Preserve the existing test suites and smoke checks.

Document exact local commands, Linux/container prerequisites, the Orka revision pin, offline versus live behavior, and how to read failures in docs/orka.md / docs/development.md. If adding coverage reporting, keep ACP results distinguishable from whole-package totals; an arbitrary percentage target is not the primary acceptance criterion.

Acceptance criteria

• CI builds and exercises the actual composed Orka v2 + AgentKit image from the commit under test for all three adapters.
• The offline suite needs no external model credentials and proves the scenarios above using production ACP/provider/MCP paths.
• The live MAF lane records an actual v2 model response, tool round-trip, and session continuation; a skip cannot be presented as executed live coverage.
• The identified ACP rejection behaviors have focused regression tests.
• Failure output identifies the adapter and scenario, includes safe diagnostics, and leaves no run-owned containers/processes/networks behind.
• Local reproduction commands and the tested source/image pins are documented.
• Existing Go checks, all four Python suites, container checks, and the new tests pass.

Starting points and validation

AgentKit:

• ACP implementation, ACP tests, and CLI tests.
• Orka deployment contract, existing live script, .github/workflows/ci.yml, and each adapter's tests/test_tool_events.py and tests/test_remote_mcp.py.

Orka at the inspected pin:

• Native v2 protocol and conformance package.
• AgentKit composition Dockerfile and provider profile/environment projection.
• MCP proxy tests and AgentKit provider-proxy tests.

Start with one composed MAF image and deterministic prompt completion, then add MCP/continuation/lifecycle scenarios, expand the offline adapter matrix, and reuse the runner for live coverage.

Existing validation commands:

uv run --directory runtimes/common --extra dev pytest -q tests/test_acp_protocol.py tests/test_cli_protocol.py
uv run --directory runtimes/common --extra dev pytest -q
go test ./...
make lint
go run github.com/rhysd/actionlint/cmd/actionlint@v1.7.12 .github/workflows/*.yml

Run all adapter Python suites using the installation/test pattern in the CI workflow, plus the new offline and live entrypoints. Validate new shell scripts with bash -n.

Scope is the composed AgentKit runtime boundary. A full Kubernetes controller/recovery/publication suite, framework upgrades, harness v1 refactoring, and new runtime capabilities are not required. Reuse existing Orka components and add only the test support needed to exercise this path.