diff --git a/.dockerignore b/.dockerignore index c8ebabb7e..af6607012 100644 --- a/.dockerignore +++ b/.dockerignore @@ -20,6 +20,8 @@ !examples/harness/ !examples/harness/echo/ !examples/harness/echo/*.go +!examples/harness/foundry-responses/ +!examples/harness/foundry-responses/*.go !*.go **/*_test.go diff --git a/config/samples/core_v1alpha1_agentruntime_foundry.yaml b/config/samples/core_v1alpha1_agentruntime_foundry.yaml index 07fcc0296..fdd8cd181 100644 --- a/config/samples/core_v1alpha1_agentruntime_foundry.yaml +++ b/config/samples/core_v1alpha1_agentruntime_foundry.yaml @@ -4,26 +4,29 @@ metadata: labels: app.kubernetes.io/name: orka app.kubernetes.io/managed-by: kustomize - name: sample-foundry-runtime + name: sample-foundry-responses-runtime spec: - # Namespace-local facade for an operator-deployed Azure AI Foundry hosted-agent adapter. - # Foundry credentials belong to the adapter Secret. Orka Tool credentials stay in - # Orka-governed credential sources and are used only by brokered execution. + # Namespace-local facade for an operator-deployed Foundry hosted Responses + # adapter (examples/harness/foundry-responses). Foundry credentials belong to + # the adapter Secret. Orka Tool credentials stay in Orka-governed credential + # sources and are used only by brokered execution. contractVersion: orka.harness.v1 deployment: mode: external-endpoint - endpoint: http://sample-foundry-runtime.default.svc.cluster.local:8080 + endpoint: http://sample-foundry-responses-runtime.default.svc.cluster.local:8080 clientAuth: bearerTokenSecretRef: - name: sample-foundry-runtime-token - key: token + name: sample-foundry-responses-runtime-token + key: harness-bearer capabilities: + # Mirror only classes whose hosted AgentKit deployment statically includes + # the matching probe-only conformance_read/conformance_write schema and has + # passed live AgentRuntime conformance. Add write only after that gate. toolExecutionModes: - observed - brokered brokeredToolClasses: - read - - write supportsCancel: true supportsRuntimeSessions: true supportsContinuation: true diff --git a/examples/bring-your-own-agent-runtime-demo/README.md b/examples/bring-your-own-agent-runtime-demo/README.md index 67240b48e..acaddf86c 100644 --- a/examples/bring-your-own-agent-runtime-demo/README.md +++ b/examples/bring-your-own-agent-runtime-demo/README.md @@ -90,7 +90,7 @@ examples/fibey-custom-agent-demo/switch-backend.sh http examples/fibey-custom-agent-demo/switch-backend.sh foundry ``` -The workflow, Tool CRDs, approval UX, and task/result APIs remain Orka-owned. Remote backends receive safe tool schemas and scoped turn metadata only; they do not receive downstream Tool credentials. +The workflow, Tool CRDs, approval UX, and task/result APIs remain Orka-owned. Remote backends receive safe tool schemas and scoped turn metadata only; they do not receive downstream Tool credentials. For Foundry hosted AgentKit Responses, the adapter does not send request-level `tools`; AgentKit schemas must be configured statically and the facade capabilities must match the conformance-tested classes. ## Troubleshooting diff --git a/examples/fibey-custom-agent-demo/README.md b/examples/fibey-custom-agent-demo/README.md index 66111f349..11be51647 100644 --- a/examples/fibey-custom-agent-demo/README.md +++ b/examples/fibey-custom-agent-demo/README.md @@ -2,7 +2,7 @@ This demo exercises the first bring-your-own agent runtime slice: Orka registers a namespace-local `AgentRuntime` facade for a remote execution backend, then an `Agent` routes `type: agent` work to it with `spec.runtime.runtimeRef`. -The checked-in backend is a deterministic generic HTTP harness fixture. It advertises `runtimeName: fibey-http-runtime`, supports `orka.harness.v1`, and runs in `observed` tool mode by default. AgentKit Serve and Foundry should plug in by swapping only the backend Service/adapter endpoint and `AgentRuntime` facade, not the Orka workflow. +The checked-in backend is a deterministic generic HTTP harness fixture. It advertises `runtimeName: fibey-http-runtime`, supports `orka.harness.v1`, and runs in `observed` tool mode by default. AgentKit Serve, Foundry Assistants, and Foundry hosted AgentKit Responses should plug in by swapping only the backend Service/adapter endpoint and `AgentRuntime` facade, not the Orka workflow. ## Backend facades @@ -10,7 +10,7 @@ The checked-in backend is a deterministic generic HTTP harness fixture. It adver | --- | --- | --- | | `fibey-http-runtime` | Generic mock/self-hosted HTTP runtime | Harness bearer token only | | `fibey-agentkit-runtime` | AgentKit Serve adapter | Adapter/runtime config only | -| `fibey-foundry-runtime` | Foundry adapter | Adapter Secret; no Orka Tool production credentials | +| `fibey-agentkit-foundry-responses` | Foundry hosted AgentKit Responses adapter | Adapter Secret; no Orka Tool production credentials | `fibey-agentkit-runtime` is intentionally observed-only in the checked-in demo: it should show `toolExecutionModes: [observed]`, `supportsCancel: true`, and `supportsRuntimeSessions: true`, with no `brokeredToolClasses` or `supportsContinuation`. AgentKit brokered read/write/coordination exist only for deployments that explicitly enable those conformance-gated profiles. @@ -106,12 +106,21 @@ kubectl apply -f examples/fibey-custom-agent-demo/agentruntime-agentkit.yaml kubectl apply -f examples/fibey-custom-agent-demo/agent-agentkit.yaml kubectl wait --for=condition=Ready agentruntime/fibey-agentkit-runtime --timeout=60s -# Foundry adapter facade; requires a Service named fibey-foundry-runtime. -# Build/deploy examples/harness/foundry with ORKA_FOUNDRY_* credentials first. +# Foundry hosted AgentKit Responses facade; requires a Service named fibey-agentkit-foundry-responses. +# Build/deploy examples/harness/foundry-responses with ORKA_FOUNDRY_RESPONSES_* credentials first. kubectl apply -f examples/fibey-custom-agent-demo/secret-foundry.yaml kubectl apply -f examples/fibey-custom-agent-demo/agentruntime-foundry.yaml kubectl apply -f examples/fibey-custom-agent-demo/agent-foundry.yaml -kubectl wait --for=condition=Ready agentruntime/fibey-foundry-runtime --timeout=60s +kubectl wait --for=condition=Ready agentruntime/fibey-agentkit-foundry-responses --timeout=60s + +# Optional literal brokered Fibey read/write scenario once downstream services exist. +kubectl apply -f examples/fibey-custom-agent-demo/tools-foundry-responses.yaml +kubectl apply -f examples/fibey-custom-agent-demo/task-foundry-responses.yaml + +# After the task runs through read, write approval, and completion, verify evidence. +examples/fibey-custom-agent-demo/verify-foundry-responses.sh \ + --task fibey-foundry-responses-quincy-north-alert \ + --namespace default ``` Run the same task against another backend by changing only `spec.agentRef.name`, for example: @@ -125,4 +134,4 @@ examples/fibey-custom-agent-demo/switch-backend.sh http The script validates the selected `AgentRuntime` and `Agent`, then patches only the Task's `spec.agentRef.name`. -Brokered mode is used only when the selected runtime advertises brokered capabilities and the task/agent exposes allowed tools. Current AgentKit Serve facades do not advertise brokered mode, so AgentKit-owned tools remain internal to AgentKit and Orka observes only lifecycle/output frames. Orka-owned side-effect tools stay behind Orka brokered governance; production tool credentials are not handed to the remote backend. +Brokered mode is used only when the selected runtime advertises brokered capabilities and the task/agent exposes allowed tools. Current AgentKit Serve facades do not advertise brokered mode, so AgentKit-owned tools remain internal to AgentKit and Orka observes only lifecycle/output frames. The Foundry hosted Responses facade must advertise only the brokered classes statically configured in AgentKit and verified by conformance. Orka-owned side-effect tools stay behind Orka brokered governance; production tool credentials are not handed to the remote backend, and hosted Responses requests do not include request-level `tools`. diff --git a/examples/fibey-custom-agent-demo/agent-foundry.yaml b/examples/fibey-custom-agent-demo/agent-foundry.yaml index fcb8a28b0..ef1058cb0 100644 --- a/examples/fibey-custom-agent-demo/agent-foundry.yaml +++ b/examples/fibey-custom-agent-demo/agent-foundry.yaml @@ -1,4 +1,4 @@ -# Optional Agent that selects the Foundry hosted-agent backend facade. +# Optional Agent that selects the Foundry hosted AgentKit Responses backend facade. apiVersion: core.orka.ai/v1alpha1 kind: Agent metadata: @@ -6,7 +6,7 @@ metadata: spec: runtime: runtimeRef: - name: fibey-foundry-runtime + name: fibey-agentkit-foundry-responses systemPrompt: inline: | - You are Fibey's Foundry-backed incident scout. Produce a concise dossier and request Orka-brokered tools for evidence or consequential actions. + You are Fibey's Foundry-hosted AgentKit incident scout. Use Orka-brokered read tools for evidence and Orka-brokered write tools only when approval is required. diff --git a/examples/fibey-custom-agent-demo/agentruntime-foundry.yaml b/examples/fibey-custom-agent-demo/agentruntime-foundry.yaml index ded01edb3..1b42bb41c 100644 --- a/examples/fibey-custom-agent-demo/agentruntime-foundry.yaml +++ b/examples/fibey-custom-agent-demo/agentruntime-foundry.yaml @@ -1,20 +1,25 @@ -# Optional namespace-local facade for a Foundry hosted-agent adapter implementing orka.harness.v1. -# Apply this with secret-foundry.yaml and point an Agent runtimeRef at fibey-foundry-runtime -# after deploying the adapter Service named fibey-foundry-runtime. +# Optional namespace-local facade for a Foundry hosted AgentKit Responses adapter +# implementing orka.harness.v1. Apply this with secret-foundry.yaml and point an +# Agent runtimeRef at fibey-agentkit-foundry-responses after deploying the +# examples/harness/foundry-responses adapter Service with the same name. apiVersion: core.orka.ai/v1alpha1 kind: AgentRuntime metadata: - name: fibey-foundry-runtime + name: fibey-agentkit-foundry-responses spec: contractVersion: orka.harness.v1 deployment: mode: external-endpoint - endpoint: http://fibey-foundry-runtime.default.svc.cluster.local:8080 + endpoint: http://fibey-agentkit-foundry-responses.default.svc.cluster.local:8080 clientAuth: bearerTokenSecretRef: - name: fibey-foundry-runtime-token + name: fibey-agentkit-foundry-responses-token key: token capabilities: + # This sample assumes the hosted AgentKit deployment has static Fibey tool + # schemas plus probe-only conformance_read/conformance_write schemas, and + # has passed live AgentRuntime read/write conformance. Fake-server adapter + # tests alone are not sufficient. Narrow this list until each gate passes. toolExecutionModes: - observed - brokered diff --git a/examples/fibey-custom-agent-demo/secret-foundry.yaml b/examples/fibey-custom-agent-demo/secret-foundry.yaml index 3c328d638..c2e25ba33 100644 --- a/examples/fibey-custom-agent-demo/secret-foundry.yaml +++ b/examples/fibey-custom-agent-demo/secret-foundry.yaml @@ -1,13 +1,14 @@ -# Optional harness bearer token for the Foundry adapter facade. -# This authenticates Orka to the adapter endpoint; Foundry credentials stay in the adapter Secret. +# Optional harness bearer token for the Foundry hosted Responses adapter facade. +# This authenticates Orka to the adapter endpoint; Foundry credentials stay in +# the adapter Deployment Secret/env and are never stored in this AgentRuntime. apiVersion: v1 kind: Secret metadata: - name: fibey-foundry-runtime-token + name: fibey-agentkit-foundry-responses-token annotations: - orka.ai/agent-runtime-endpoint: http://fibey-foundry-runtime.default.svc.cluster.local:8080 + orka.ai/agent-runtime-endpoint: http://fibey-agentkit-foundry-responses.default.svc.cluster.local:8080 labels: orka.ai/agent-runtime-auth: "true" - orka.ai/agent-runtime-name: fibey-foundry-runtime + orka.ai/agent-runtime-name: fibey-agentkit-foundry-responses stringData: - token: mock-token + token: REDACTED diff --git a/examples/fibey-custom-agent-demo/switch-backend.sh b/examples/fibey-custom-agent-demo/switch-backend.sh index fcfb41345..e0812e715 100755 --- a/examples/fibey-custom-agent-demo/switch-backend.sh +++ b/examples/fibey-custom-agent-demo/switch-backend.sh @@ -27,7 +27,7 @@ case "${backend}" in ;; foundry) agent="fibey-remote-foundry" - runtime="fibey-foundry-runtime" + runtime="fibey-agentkit-foundry-responses" ;; -h|--help|help|"") usage diff --git a/examples/fibey-custom-agent-demo/task-foundry-responses.yaml b/examples/fibey-custom-agent-demo/task-foundry-responses.yaml new file mode 100644 index 000000000..1bf07439d --- /dev/null +++ b/examples/fibey-custom-agent-demo/task-foundry-responses.yaml @@ -0,0 +1,20 @@ +# Optional Foundry hosted AgentKit Responses task that exposes the Fibey brokered +# read/write tools. Apply with tools-foundry-responses.yaml and the Foundry +# Responses facade manifests after deploying the downstream mock/real services. +apiVersion: core.orka.ai/v1alpha1 +kind: Task +metadata: + name: fibey-foundry-responses-quincy-north-alert +spec: + type: agent + agentRef: + name: fibey-remote-foundry + agentRuntime: + allowedTools: + - check-network-telemetry + - get-active-incidents + - dispatch-work-order + - escalate-incident + prompt: | + Quincy North alert: pump telemetry is anomalous after overnight maintenance. + Investigate likely cause with brokered read tools, summarize evidence, and request approval before dispatching any work order. diff --git a/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-completion-after-terminal.json b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-completion-after-terminal.json new file mode 100644 index 000000000..b3f79d2e7 --- /dev/null +++ b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-completion-after-terminal.json @@ -0,0 +1,86 @@ +{ + "events": [ + { + "seq": 1, + "eventType": "ToolCallStarted", + "toolName": "check-network-telemetry", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "read-call-1" + }, + { + "seq": 2, + "eventType": "ToolCallCompleted", + "toolName": "check-network-telemetry", + "toolCallID": "read-call-1", + "content": { + "approved": true + } + }, + { + "seq": 3, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "write-call-1" + }, + { + "seq": 4, + "eventType": "ApprovalRequested", + "toolName": "dispatch-work-order", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "targetTool": "dispatch-work-order", + "toolCallID": "write-call-1" + } + }, + { + "seq": 5, + "eventType": "ApprovalApproved", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "decision": "approve" + } + }, + { + "seq": 6, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "approvalID": "approval-1", + "brokeredClass": "write", + "executionState": "started", + "idempotencyKey": "approval-1", + "executionIdempotencyKey": "approval-1" + }, + "toolCallID": "write-call-1" + }, + { + "seq": 7, + "eventType": "AgentRuntimeCompleted" + }, + { + "seq": 8, + "eventType": "ToolCallCompleted", + "toolName": "dispatch-work-order", + "toolCallID": "write-call-1", + "content": { + "approved": true, + "executionIdempotencyKey": "approval-1" + } + }, + { + "seq": 9, + "eventType": "TaskSucceeded" + } + ] +} diff --git a/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-decision-before-request.json b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-decision-before-request.json new file mode 100644 index 000000000..2be354064 --- /dev/null +++ b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-decision-before-request.json @@ -0,0 +1,66 @@ +{ + "events": [ + { + "seq": 1, + "eventType": "ToolCallStarted", + "toolName": "check-network-telemetry", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "read-call-1" + }, + { + "seq": 2, + "eventType": "ApprovalApproved", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "decision": "approve" + } + }, + { + "seq": 3, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "write-call-1" + }, + { + "seq": 4, + "eventType": "ApprovalRequested", + "toolName": "dispatch-work-order", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "targetTool": "dispatch-work-order", + "toolCallID": "write-call-1" + } + }, + { + "seq": 5, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "approvalID": "approval-1", + "brokeredClass": "write", + "executionState": "started", + "idempotencyKey": "approval-1" + }, + "toolCallID": "write-call-1" + }, + { + "seq": 6, + "eventType": "AgentRuntimeCompleted" + }, + { + "seq": 7, + "eventType": "TaskSucceeded" + } + ] +} diff --git a/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-declined-write.json b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-declined-write.json new file mode 100644 index 000000000..cbd68639d --- /dev/null +++ b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-declined-write.json @@ -0,0 +1,66 @@ +{ + "events": [ + { + "seq": 1, + "eventType": "ToolCallStarted", + "toolName": "check-network-telemetry", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "read-call-1" + }, + { + "seq": 2, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "write-call-1" + }, + { + "seq": 3, + "eventType": "ApprovalRequested", + "toolName": "dispatch-work-order", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "targetTool": "dispatch-work-order", + "toolCallID": "write-call-1" + } + }, + { + "seq": 4, + "eventType": "ApprovalDeclined", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "decision": "decline" + } + }, + { + "seq": 5, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "approvalID": "approval-1", + "brokeredClass": "write", + "executionState": "started", + "idempotencyKey": "approval-1" + }, + "toolCallID": "write-call-1" + }, + { + "seq": 6, + "eventType": "AgentRuntimeCompleted" + }, + { + "seq": 7, + "eventType": "TaskSucceeded" + } + ] +} diff --git a/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-duplicate-write.json b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-duplicate-write.json new file mode 100644 index 000000000..e3b643210 --- /dev/null +++ b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-duplicate-write.json @@ -0,0 +1,78 @@ +{ + "events": [ + { + "seq": 1, + "eventType": "ToolCallStarted", + "toolName": "check-network-telemetry", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "read-call-1" + }, + { + "seq": 2, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "write-call-1" + }, + { + "seq": 3, + "eventType": "ApprovalRequested", + "toolName": "dispatch-work-order", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "targetTool": "dispatch-work-order", + "toolCallID": "write-call-1" + } + }, + { + "seq": 4, + "eventType": "ApprovalApproved", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "decision": "approve" + } + }, + { + "seq": 5, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "approvalID": "approval-1", + "brokeredClass": "write", + "executionState": "started", + "idempotencyKey": "approval-1" + }, + "toolCallID": "write-call-1" + }, + { + "seq": 6, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "approvalID": "approval-1", + "brokeredClass": "write", + "executionState": "started", + "idempotencyKey": "approval-1" + }, + "toolCallID": "write-call-1" + }, + { + "seq": 7, + "eventType": "AgentRuntimeCompleted" + }, + { + "seq": 8, + "eventType": "TaskSucceeded" + } + ] +} diff --git a/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-failure-after-start.json b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-failure-after-start.json new file mode 100644 index 000000000..40c146fcf --- /dev/null +++ b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-failure-after-start.json @@ -0,0 +1,88 @@ +{ + "events": [ + { + "seq": 1, + "eventType": "ToolCallStarted", + "toolName": "check-network-telemetry", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "read-call-1" + }, + { + "seq": 2, + "eventType": "ToolCallCompleted", + "toolName": "check-network-telemetry", + "toolCallID": "read-call-1", + "content": { + "approved": true + } + }, + { + "seq": 3, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "write-call-1" + }, + { + "seq": 4, + "eventType": "ApprovalRequested", + "toolName": "dispatch-work-order", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "targetTool": "dispatch-work-order", + "toolCallID": "write-call-1" + } + }, + { + "seq": 5, + "eventType": "ApprovalApproved", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "decision": "approve" + } + }, + { + "seq": 6, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "approvalID": "approval-1", + "brokeredClass": "write", + "executionState": "started", + "idempotencyKey": "approval-1", + "executionIdempotencyKey": "approval-1" + }, + "toolCallID": "write-call-1" + }, + { + "seq": 7, + "eventType": "ToolCallFailed", + "toolName": "dispatch-work-order", + "toolCallID": "write-call-1", + "content": { + "approved": false, + "toolError": { + "code": "tool_execution_failed" + } + } + }, + { + "seq": 8, + "eventType": "AgentRuntimeCompleted" + }, + { + "seq": 9, + "eventType": "TaskSucceeded" + } + ] +} diff --git a/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-generic-idempotency-only.json b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-generic-idempotency-only.json new file mode 100644 index 000000000..684dd3c3d --- /dev/null +++ b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-generic-idempotency-only.json @@ -0,0 +1,84 @@ +{ + "events": [ + { + "seq": 1, + "eventType": "ToolCallStarted", + "toolName": "check-network-telemetry", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "read-call-1" + }, + { + "seq": 2, + "eventType": "ToolCallCompleted", + "toolName": "check-network-telemetry", + "toolCallID": "read-call-1", + "content": { + "approved": true + } + }, + { + "seq": 3, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "write-call-1" + }, + { + "seq": 4, + "eventType": "ApprovalRequested", + "toolName": "dispatch-work-order", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "targetTool": "dispatch-work-order", + "toolCallID": "write-call-1" + } + }, + { + "seq": 5, + "eventType": "ApprovalApproved", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "decision": "approve" + } + }, + { + "seq": 6, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "approvalID": "approval-1", + "brokeredClass": "write", + "executionState": "started", + "idempotencyKey": "approval-1" + }, + "toolCallID": "write-call-1" + }, + { + "seq": 7, + "eventType": "ToolCallCompleted", + "toolName": "dispatch-work-order", + "toolCallID": "write-call-1", + "content": { + "approved": true + } + }, + { + "seq": 8, + "eventType": "AgentRuntimeCompleted" + }, + { + "seq": 9, + "eventType": "TaskSucceeded" + } + ] +} diff --git a/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-mismatched-write-request.json b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-mismatched-write-request.json new file mode 100644 index 000000000..a7df08693 --- /dev/null +++ b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-mismatched-write-request.json @@ -0,0 +1,66 @@ +{ + "events": [ + { + "seq": 1, + "eventType": "ToolCallStarted", + "toolName": "check-network-telemetry", + "toolCallID": "read-call-1", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + } + }, + { + "seq": 2, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "toolCallID": "dispatch-call-1", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + } + }, + { + "seq": 3, + "eventType": "ApprovalRequested", + "toolName": "escalate-incident", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "targetTool": "escalate-incident", + "toolCallID": "escalate-call-1" + } + }, + { + "seq": 4, + "eventType": "ApprovalApproved", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "decision": "approve" + } + }, + { + "seq": 5, + "eventType": "ToolCallStarted", + "toolName": "escalate-incident", + "toolCallID": "escalate-call-1", + "content": { + "approvalID": "approval-1", + "brokeredClass": "write", + "executionState": "started", + "idempotencyKey": "approval-1" + } + }, + { + "seq": 6, + "eventType": "AgentRuntimeCompleted" + }, + { + "seq": 7, + "eventType": "TaskSucceeded" + } + ] +} diff --git a/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-missing-approval-decision.json b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-missing-approval-decision.json new file mode 100644 index 000000000..f10bd3a4f --- /dev/null +++ b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-missing-approval-decision.json @@ -0,0 +1,57 @@ +{ + "events": [ + { + "seq": 1, + "eventType": "ToolCallStarted", + "toolName": "check-network-telemetry", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "read-call-1" + }, + { + "seq": 2, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "write-call-1" + }, + { + "seq": 3, + "eventType": "ApprovalRequested", + "toolName": "dispatch-work-order", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "targetTool": "dispatch-work-order", + "toolCallID": "write-call-1" + } + }, + { + "seq": 4, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "approvalID": "approval-1", + "brokeredClass": "write", + "executionState": "started", + "idempotencyKey": "approval-1" + }, + "toolCallID": "write-call-1" + }, + { + "seq": 5, + "eventType": "AgentRuntimeCompleted" + }, + { + "seq": 6, + "eventType": "TaskSucceeded" + } + ] +} diff --git a/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-missing-approval-id.json b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-missing-approval-id.json new file mode 100644 index 000000000..09b8d0bc3 --- /dev/null +++ b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-missing-approval-id.json @@ -0,0 +1,59 @@ +{ + "events": [ + { + "seq": 1, + "eventType": "ToolCallStarted", + "toolName": "check-network-telemetry", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "read-call-1" + }, + { + "seq": 2, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "shared-id" + }, + { + "seq": 3, + "eventType": "ApprovalRequested", + "toolName": "dispatch-work-order", + "toolCallID": "shared-id", + "content": { + "toolCallID": "shared-id" + } + }, + { + "seq": 4, + "eventType": "ApprovalApproved", + "toolCallID": "shared-id" + }, + { + "seq": 5, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "toolCallID": "shared-id", + "content": { + "brokeredClass": "write", + "executionState": "started", + "idempotencyKey": "shared-id" + } + }, + { + "seq": 6, + "eventType": "AgentRuntimeCompleted" + }, + { + "seq": 7, + "eventType": "TaskSucceeded" + } + ] +} diff --git a/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-missing-write-exec.json b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-missing-write-exec.json new file mode 100644 index 000000000..b7def3b42 --- /dev/null +++ b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-missing-write-exec.json @@ -0,0 +1,54 @@ +{ + "events": [ + { + "seq": 1, + "eventType": "ToolCallStarted", + "toolName": "check-network-telemetry", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "read-call-1" + }, + { + "seq": 2, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "write-call-1" + }, + { + "seq": 3, + "eventType": "ApprovalRequested", + "toolName": "dispatch-work-order", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "targetTool": "dispatch-work-order", + "toolCallID": "write-call-1" + } + }, + { + "seq": 4, + "eventType": "ApprovalApproved", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "decision": "approve" + } + }, + { + "seq": 5, + "eventType": "AgentRuntimeCompleted" + }, + { + "seq": 6, + "eventType": "TaskSucceeded" + } + ] +} diff --git a/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-overlapping-write-marker.json b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-overlapping-write-marker.json new file mode 100644 index 000000000..497d5114e --- /dev/null +++ b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-overlapping-write-marker.json @@ -0,0 +1,56 @@ +{ + "events": [ + { + "seq": 1, + "eventType": "ToolCallStarted", + "toolName": "check-network-telemetry", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "read-call-1" + }, + { + "seq": 2, + "eventType": "ApprovalRequested", + "toolName": "dispatch-work-order", + "content": { + "approvalID": "approval-1", + "targetTool": "dispatch-work-order", + "toolCallID": "write-call-1" + } + }, + { + "seq": 3, + "eventType": "ApprovalApproved", + "content": { + "approvalID": "approval-1", + "decision": "approve" + } + }, + { + "seq": 4, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "harness": { + "frameType": "ToolCallRequested" + }, + "approvalID": "approval-1", + "brokeredClass": "write", + "executionState": "started", + "idempotencyKey": "approval-1" + }, + "toolCallID": "write-call-1" + }, + { + "seq": 5, + "eventType": "AgentRuntimeCompleted" + }, + { + "seq": 6, + "eventType": "TaskSucceeded" + } + ] +} diff --git a/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-partial-idempotency.json b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-partial-idempotency.json new file mode 100644 index 000000000..bc431667b --- /dev/null +++ b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-partial-idempotency.json @@ -0,0 +1,108 @@ +{ + "events": [ + { + "seq": 1, + "eventType": "ToolCallStarted", + "toolName": "check-network-telemetry", + "toolCallID": "read-call-1", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + } + }, + { + "seq": 2, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "toolCallID": "dispatch-call-1", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + } + }, + { + "seq": 3, + "eventType": "ApprovalRequested", + "toolName": "dispatch-work-order", + "toolCallID": "approval-dispatch", + "content": { + "approvalID": "approval-dispatch", + "targetTool": "dispatch-work-order", + "toolCallID": "dispatch-call-1" + } + }, + { + "seq": 4, + "eventType": "ApprovalApproved", + "toolCallID": "approval-dispatch", + "content": { + "approvalID": "approval-dispatch", + "decision": "approve" + } + }, + { + "seq": 5, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "toolCallID": "dispatch-call-1", + "content": { + "approvalID": "approval-dispatch", + "brokeredClass": "write", + "executionState": "started", + "idempotencyKey": "approval-dispatch" + } + }, + { + "seq": 6, + "eventType": "ToolCallStarted", + "toolName": "escalate-incident", + "toolCallID": "escalate-call-1", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + } + }, + { + "seq": 7, + "eventType": "ApprovalRequested", + "toolName": "escalate-incident", + "toolCallID": "approval-escalate", + "content": { + "approvalID": "approval-escalate", + "targetTool": "escalate-incident", + "toolCallID": "escalate-call-1" + } + }, + { + "seq": 8, + "eventType": "ApprovalApproved", + "toolCallID": "approval-escalate", + "content": { + "approvalID": "approval-escalate", + "decision": "approve" + } + }, + { + "seq": 9, + "eventType": "ToolCallStarted", + "toolName": "escalate-incident", + "toolCallID": "escalate-call-1", + "content": { + "approvalID": "approval-escalate", + "brokeredClass": "write", + "executionState": "started" + } + }, + { + "seq": 10, + "eventType": "AgentRuntimeCompleted" + }, + { + "seq": 11, + "eventType": "TaskSucceeded" + } + ] +} diff --git a/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-pass.json b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-pass.json new file mode 100644 index 000000000..e49fcce4d --- /dev/null +++ b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-pass.json @@ -0,0 +1,86 @@ +{ + "events": [ + { + "seq": 1, + "eventType": "ToolCallStarted", + "toolName": "check-network-telemetry", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "read-call-1" + }, + { + "seq": 2, + "eventType": "ToolCallCompleted", + "toolName": "check-network-telemetry", + "toolCallID": "read-call-1", + "content": { + "approved": true + } + }, + { + "seq": 3, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "write-call-1" + }, + { + "seq": 4, + "eventType": "ApprovalRequested", + "toolName": "dispatch-work-order", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "targetTool": "dispatch-work-order", + "toolCallID": "write-call-1" + } + }, + { + "seq": 5, + "eventType": "ApprovalApproved", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "decision": "approve" + } + }, + { + "seq": 6, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "approvalID": "approval-1", + "brokeredClass": "write", + "executionState": "started", + "idempotencyKey": "approval-1", + "executionIdempotencyKey": "approval-1" + }, + "toolCallID": "write-call-1" + }, + { + "seq": 7, + "eventType": "ToolCallCompleted", + "toolName": "dispatch-work-order", + "toolCallID": "write-call-1", + "content": { + "approved": true, + "executionIdempotencyKey": "approval-1" + } + }, + { + "seq": 8, + "eventType": "AgentRuntimeCompleted" + }, + { + "seq": 9, + "eventType": "TaskSucceeded" + } + ] +} diff --git a/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-success-before-runtime-completion.json b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-success-before-runtime-completion.json new file mode 100644 index 000000000..fecf9adbf --- /dev/null +++ b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-success-before-runtime-completion.json @@ -0,0 +1,67 @@ +{ + "events": [ + { + "seq": 1, + "eventType": "ToolCallStarted", + "toolName": "check-network-telemetry", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "read-call-1" + }, + { + "seq": 2, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "write-call-1" + }, + { + "seq": 3, + "eventType": "ApprovalRequested", + "toolName": "dispatch-work-order", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "targetTool": "dispatch-work-order", + "toolCallID": "write-call-1" + } + }, + { + "seq": 4, + "eventType": "ApprovalApproved", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "decision": "approve" + } + }, + { + "seq": 5, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "approvalID": "approval-1", + "brokeredClass": "write", + "executionState": "started", + "idempotencyKey": "approval-1", + "executionIdempotencyKey": "approval-1" + }, + "toolCallID": "write-call-1" + }, + { + "seq": 6, + "eventType": "TaskSucceeded" + }, + { + "seq": 7, + "eventType": "AgentRuntimeCompleted" + } + ] +} diff --git a/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-tail-page.json b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-tail-page.json new file mode 100644 index 000000000..3b3515f3e --- /dev/null +++ b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-tail-page.json @@ -0,0 +1,69 @@ +{ + "events": [ + { + "seq": 101, + "eventType": "ToolCallStarted", + "toolName": "check-network-telemetry", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "read-call-1" + }, + { + "seq": 102, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "write-call-1" + }, + { + "seq": 103, + "eventType": "ApprovalRequested", + "toolName": "dispatch-work-order", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "targetTool": "dispatch-work-order", + "toolCallID": "write-call-1" + } + }, + { + "seq": 104, + "eventType": "ApprovalApproved", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "decision": "approve" + } + }, + { + "seq": 105, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "approvalID": "approval-1", + "brokeredClass": "write", + "executionState": "started", + "idempotencyKey": "approval-1", + "executionIdempotencyKey": "approval-1" + }, + "toolCallID": "write-call-1" + }, + { + "seq": 106, + "eventType": "AgentRuntimeCompleted" + }, + { + "seq": 107, + "eventType": "TaskSucceeded" + } + ], + "afterSeq": 100, + "latestSeq": 107 +} diff --git a/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-task-failed.json b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-task-failed.json new file mode 100644 index 000000000..7dbf70433 --- /dev/null +++ b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-task-failed.json @@ -0,0 +1,67 @@ +{ + "events": [ + { + "seq": 1, + "eventType": "ToolCallStarted", + "toolName": "check-network-telemetry", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "read-call-1" + }, + { + "seq": 2, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "write-call-1" + }, + { + "seq": 3, + "eventType": "ApprovalRequested", + "toolName": "dispatch-work-order", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "targetTool": "dispatch-work-order", + "toolCallID": "write-call-1" + } + }, + { + "seq": 4, + "eventType": "ApprovalApproved", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "decision": "approve" + } + }, + { + "seq": 5, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "approvalID": "approval-1", + "brokeredClass": "write", + "executionState": "started", + "idempotencyKey": "approval-1", + "executionIdempotencyKey": "approval-1" + }, + "toolCallID": "write-call-1" + }, + { + "seq": 6, + "eventType": "AgentRuntimeCompleted" + }, + { + "seq": 7, + "eventType": "TaskFailed" + } + ] +} diff --git a/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-terminal-write-terminal.json b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-terminal-write-terminal.json new file mode 100644 index 000000000..82ef64da9 --- /dev/null +++ b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-terminal-write-terminal.json @@ -0,0 +1,71 @@ +{ + "events": [ + { + "seq": 1, + "eventType": "ToolCallStarted", + "toolName": "check-network-telemetry", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "read-call-1" + }, + { + "seq": 2, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "write-call-1" + }, + { + "seq": 3, + "eventType": "ApprovalRequested", + "toolName": "dispatch-work-order", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "targetTool": "dispatch-work-order", + "toolCallID": "write-call-1" + } + }, + { + "seq": 4, + "eventType": "ApprovalApproved", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "decision": "approve" + } + }, + { + "seq": 5, + "eventType": "AgentRuntimeCompleted" + }, + { + "seq": 6, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "approvalID": "approval-1", + "brokeredClass": "write", + "executionState": "started", + "idempotencyKey": "approval-1", + "executionIdempotencyKey": "approval-1" + }, + "toolCallID": "write-call-1" + }, + { + "seq": 7, + "eventType": "AgentRuntimeCompleted" + }, + { + "seq": 8, + "eventType": "TaskSucceeded" + } + ] +} diff --git a/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-truncated-page.json b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-truncated-page.json new file mode 100644 index 000000000..5bc553c05 --- /dev/null +++ b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-truncated-page.json @@ -0,0 +1,69 @@ +{ + "events": [ + { + "seq": 1, + "eventType": "ToolCallStarted", + "toolName": "check-network-telemetry", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "read-call-1" + }, + { + "seq": 2, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "write-call-1" + }, + { + "seq": 3, + "eventType": "ApprovalRequested", + "toolName": "dispatch-work-order", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "targetTool": "dispatch-work-order", + "toolCallID": "write-call-1" + } + }, + { + "seq": 4, + "eventType": "ApprovalApproved", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "decision": "approve" + } + }, + { + "seq": 5, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "approvalID": "approval-1", + "brokeredClass": "write", + "executionState": "started", + "idempotencyKey": "approval-1", + "executionIdempotencyKey": "approval-1" + }, + "toolCallID": "write-call-1" + }, + { + "seq": 6, + "eventType": "AgentRuntimeCompleted" + }, + { + "seq": 7, + "eventType": "TaskSucceeded" + } + ], + "afterSeq": 0, + "latestSeq": 8 +} diff --git a/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-write-after-terminal.json b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-write-after-terminal.json new file mode 100644 index 000000000..fe491ddea --- /dev/null +++ b/examples/fibey-custom-agent-demo/testdata/foundry-responses-events-write-after-terminal.json @@ -0,0 +1,67 @@ +{ + "events": [ + { + "seq": 1, + "eventType": "ToolCallStarted", + "toolName": "check-network-telemetry", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "read-call-1" + }, + { + "seq": 2, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "harness": { + "frameType": "ToolCallRequested" + } + }, + "toolCallID": "write-call-1" + }, + { + "seq": 3, + "eventType": "ApprovalRequested", + "toolName": "dispatch-work-order", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "targetTool": "dispatch-work-order", + "toolCallID": "write-call-1" + } + }, + { + "seq": 4, + "eventType": "ApprovalApproved", + "toolCallID": "approval-1", + "content": { + "approvalID": "approval-1", + "decision": "approve" + } + }, + { + "seq": 5, + "eventType": "AgentRuntimeCompleted" + }, + { + "seq": 6, + "eventType": "ToolCallStarted", + "toolName": "dispatch-work-order", + "content": { + "approvalID": "approval-1", + "brokeredClass": "write", + "executionState": "started", + "idempotencyKey": "approval-1", + "executionIdempotencyKey": "approval-1" + }, + "toolCallID": "write-call-1" + }, + { + "seq": 7, + "eventType": "TaskSucceeded" + } + ] +} diff --git a/examples/fibey-custom-agent-demo/tools-foundry-responses.yaml b/examples/fibey-custom-agent-demo/tools-foundry-responses.yaml new file mode 100644 index 000000000..ac115b231 --- /dev/null +++ b/examples/fibey-custom-agent-demo/tools-foundry-responses.yaml @@ -0,0 +1,85 @@ +# Optional Fibey brokered tools for a Foundry hosted AgentKit Responses demo. +# These are not included in the default kustomization because the downstream +# services are deployment-specific. Remote hosted AgentKit receives only safe +# schemas; Orka owns execution, credentials, approvals, idempotency, and audit. +apiVersion: core.orka.ai/v1alpha1 +kind: Tool +metadata: + name: check-network-telemetry +spec: + description: Read sanitized Fibey network telemetry for a site or asset. + brokeredToolClass: read + parameters: + type: object + properties: + site: + type: string + asset: + type: string + required: + - site + http: + url: http://fibey-telemetry.default.svc.cluster.local:8080/check + method: POST +--- +apiVersion: core.orka.ai/v1alpha1 +kind: Tool +metadata: + name: get-active-incidents +spec: + description: Read active Fibey incident records for a site. + brokeredToolClass: read + parameters: + type: object + properties: + site: + type: string + required: + - site + http: + url: http://fibey-incidents.default.svc.cluster.local:8080/active + method: POST +--- +apiVersion: core.orka.ai/v1alpha1 +kind: Tool +metadata: + name: dispatch-work-order +spec: + description: Dispatch an approved Fibey work order. Requires Orka approval and idempotency. + brokeredToolClass: write + parameters: + type: object + properties: + site: + type: string + action: + type: string + severity: + type: string + required: + - site + - action + http: + url: http://fibey-dispatch.default.svc.cluster.local:8080/work-orders + method: POST +--- +apiVersion: core.orka.ai/v1alpha1 +kind: Tool +metadata: + name: escalate-incident +spec: + description: Escalate an approved Fibey incident to an on-call owner. Requires Orka approval and idempotency. + brokeredToolClass: write + parameters: + type: object + properties: + incident: + type: string + reason: + type: string + required: + - incident + - reason + http: + url: http://fibey-dispatch.default.svc.cluster.local:8080/escalations + method: POST diff --git a/examples/fibey-custom-agent-demo/verify-foundry-responses.sh b/examples/fibey-custom-agent-demo/verify-foundry-responses.sh new file mode 100755 index 000000000..e2d5e4d72 --- /dev/null +++ b/examples/fibey-custom-agent-demo/verify-foundry-responses.sh @@ -0,0 +1,419 @@ +#!/usr/bin/env bash +set -euo pipefail + +usage() { + cat >&2 <<'USAGE' +Usage: examples/fibey-custom-agent-demo/verify-foundry-responses.sh [--task NAME] [--namespace NAME] [--json EVENTS.json] + +Checks the live Fibey Foundry hosted AgentKit Responses scenario evidence from +Orka task events. By default it paginates `orka task events --output json` +through `latestSeq`. Use --json to verify a previously captured event payload +without contacting Orka; payloads with `latestSeq` must be complete. + +Expected evidence: + - read brokered tool request for check-network-telemetry or get-active-incidents + - write brokered tool request for dispatch-work-order or escalate-incident + - matching ApprovalRequested and ApprovalApproved events precede write execution + - an executionIdempotencyKey is present in the write execution ledger event + - matching ToolCallCompleted events exist for each brokered read and write execution + - terminal TaskSucceeded/AgentRuntimeCompleted/TurnCompleted-style event exists + +This verifier does not approve tasks and never reads Foundry credentials. +USAGE +} + +task="fibey-foundry-responses-quincy-north-alert" +namespace="default" +json_file="" + +while [[ $# -gt 0 ]]; do + case "$1" in + --task) + [[ $# -ge 2 ]] || { echo "--task requires a value" >&2; exit 2; } + task="$2" + shift 2 + ;; + --namespace) + [[ $# -ge 2 ]] || { echo "--namespace requires a value" >&2; exit 2; } + namespace="$2" + shift 2 + ;; + --json) + [[ $# -ge 2 ]] || { echo "--json requires a path" >&2; exit 2; } + json_file="$2" + shift 2 + ;; + -h|--help) + usage + exit 0 + ;; + *) + echo "unknown argument: $1" >&2 + usage + exit 2 + ;; + esac +done + +require_cmd() { + command -v "$1" >/dev/null 2>&1 || { echo "error: $1 is required" >&2; exit 2; } +} + +require_cmd python3 + +json_tmp="" +cleanup() { + [[ -z "$json_tmp" ]] || rm -f "$json_tmp" +} +trap cleanup EXIT + +if [[ -z "$json_file" ]]; then + require_cmd orka + json_tmp="$(mktemp)" + script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + event_fetcher="${script_dir}/../harness/foundry-responses/fetch_task_events.py" + python3 "$event_fetcher" --task "$task" --namespace "$namespace" --output "$json_tmp" + json_file="$json_tmp" +fi + +python3 - "$json_file" <<'PY' +import json +import sys +from pathlib import Path + +path = Path(sys.argv[1]) +try: + payload = json.loads(path.read_text()) +except Exception as exc: # noqa: BLE001 - user-facing validation script + raise SystemExit(f"error: read event JSON: {exc}") + +if isinstance(payload, dict): + if isinstance(payload.get("events"), list): + events = payload["events"] + elif isinstance(payload.get("items"), list): + events = payload["items"] + else: + events = [payload] +elif isinstance(payload, list): + events = payload +else: + raise SystemExit("error: event JSON must be an object or list") + +if isinstance(payload, dict) and payload.get("latestSeq") is not None: + try: + after_seq = int(payload.get("afterSeq", 0)) + latest_seq = int(payload["latestSeq"]) + sequences = [int(event.get("seq", 0)) for event in events if isinstance(event, dict)] + except (TypeError, ValueError) as exc: + raise SystemExit(f"error: invalid event sequence metadata: {exc}") from exc + if after_seq != 0: + raise SystemExit(f"error: event JSON is incomplete: afterSeq must be 0, got {after_seq}") + complete_sequence = len(sequences) == latest_seq and all( + seq == expected for expected, seq in enumerate(sequences, start=1) + ) + if not complete_sequence: + captured_seq = max(sequences, default=0) + raise SystemExit( + "error: event JSON is incomplete: " + f"captured sequences do not cover 1 through latestSeq {latest_seq} " + f"(highest captured sequence {captured_seq})" + ) + +READ_TOOLS = {"check-network-telemetry", "get-active-incidents"} +WRITE_TOOLS = {"dispatch-work-order", "escalate-incident"} +TERMINAL_TYPES = { + "TaskSucceeded", + "AgentRuntimeCompleted", + "TurnCompleted", + "TaskCompleted", +} +TASK_TERMINAL_TYPES = {"TaskSucceeded", "TaskFailed", "TaskCancelled"} +def field(event, name): + if not isinstance(event, dict): + return None + if name in event: + return event[name] + content = event.get("content") + if isinstance(content, dict) and name in content: + return content[name] + if isinstance(content, str): + try: + decoded = json.loads(content) + except Exception: # noqa: BLE001 + decoded = None + if isinstance(decoded, dict) and name in decoded: + return decoded[name] + return None + + +def content_field(event, name): + if not isinstance(event, dict): + return None + content = event.get("content") + if isinstance(content, dict): + return content.get(name) + if isinstance(content, str): + try: + decoded = json.loads(content) + except Exception: # noqa: BLE001 + return None + if isinstance(decoded, dict): + return decoded.get(name) + return None + + +def event_type(event): + if not isinstance(event, dict): + return "" + return str(event.get("type") or event.get("eventType") or "") + + +def tool_name(event): + for key in ("toolName", "tool", "name"): + value = field(event, key) + if isinstance(value, str) and value: + return value + return "" + + +def seq(event): + if not isinstance(event, dict): + return None + for name in ("seq", "sequence", "_verifyOrder", "id"): + value = event.get(name) + if value is None: + continue + try: + return int(value) + except Exception: # noqa: BLE001 + continue + return None + + +def execution_idempotency_value(value): + if isinstance(value, dict): + for key, nested in value.items(): + if key in {"executionIdempotencyKey", "Execution-Idempotency-Key"}: + if isinstance(nested, str) and nested.strip(): + return nested.strip() + found = execution_idempotency_value(nested) + if found: + return found + elif isinstance(value, list): + for item in value: + found = execution_idempotency_value(item) + if found: + return found + elif isinstance(value, str): + try: + decoded = json.loads(value) + except Exception: # noqa: BLE001 + return "" + return execution_idempotency_value(decoded) + return "" + + +def approval_id(event): + for key in ("approvalID", "approvalId"): + value = field(event, key) + if isinstance(value, str) and value.strip(): + return value.strip() + return "" + + +def tool_call_id(event): + for key in ("toolCallID", "toolCallId"): + value = field(event, key) + if isinstance(value, str) and value.strip(): + return value.strip() + return "" + + +def is_write_execution_start(event): + if event_type(event) != "ToolCallStarted": + return False + if is_harness_tool_request(event): + return False + return field(event, "executionState") == "started" and field(event, "brokeredClass") == "write" + + +def is_harness_tool_request(event): + if event_type(event) == "ToolCallRequested": + return True + harness_identity = field(event, "harness") + return ( + event_type(event) == "ToolCallStarted" + and isinstance(harness_identity, dict) + and harness_identity.get("frameType") == "ToolCallRequested" + ) + + +def matching_tool_events(source, candidates, before_order=None): + source_tool = tool_name(source) + source_call_id = tool_call_id(source) + source_order = seq(source) + if not source_tool or not source_call_id or source_order is None: + return [] + return [ + candidate for candidate in candidates + if tool_name(candidate) == source_tool + and tool_call_id(candidate) == source_call_id + and seq(candidate) is not None + and seq(candidate) > source_order + and (before_order is None or seq(candidate) < before_order) + ] + + +ordered_events = [] +for index, event in enumerate(events, start=1): + if isinstance(event, dict) and seq(event) is None: + copied = dict(event) + copied["_verifyOrder"] = index + ordered_events.append(copied) + else: + ordered_events.append(event) + +events = ordered_events +read_events = [e for e in events if tool_name(e) in READ_TOOLS and is_harness_tool_request(e)] +write_events = [e for e in events if tool_name(e) in WRITE_TOOLS] +write_request_events = [e for e in write_events if is_harness_tool_request(e)] +approval_request_events = [e for e in events if event_type(e) == "ApprovalRequested"] +approval_approved_events = [e for e in events if event_type(e) == "ApprovalApproved"] +approval_declined_events = [e for e in events if event_type(e) == "ApprovalDeclined"] +write_exec_events = [e for e in write_events if is_write_execution_start(e)] +write_start_events = write_exec_events +tool_completed_events = [e for e in events if event_type(e) == "ToolCallCompleted"] +tool_failed_events = [e for e in events if event_type(e) == "ToolCallFailed"] +terminal_events = [e for e in events if event_type(e) in TERMINAL_TYPES] +task_terminal_events = [e for e in events if event_type(e) in TASK_TERMINAL_TYPES] +first_terminal_order = min((seq(event) for event in terminal_events), default=None) +execution_idempotency_events = [e for e in write_exec_events if execution_idempotency_value(e)] + +failures = [] +if not read_events: + failures.append("missing read brokered tool event for check-network-telemetry/get-active-incidents") +if not write_request_events: + failures.append("missing write brokered tool event for dispatch-work-order/escalate-incident") +if not approval_request_events: + failures.append("missing ApprovalRequested event") +if not approval_approved_events: + failures.append("missing ApprovalApproved event") +if not write_exec_events: + failures.append("missing write ToolCallStarted event after approval") +for event in read_events: + read_tool = tool_name(event) + if matching_tool_events(event, tool_failed_events, first_terminal_order): + failures.append(f"read tool call for {read_tool} has matching ToolCallFailed") + elif not matching_tool_events(event, tool_completed_events, first_terminal_order): + failures.append(f"read tool call for {read_tool} is missing ToolCallCompleted") +if write_exec_events: + for event in write_exec_events: + write_tool = tool_name(event) + write_order = seq(event) + if not execution_idempotency_value(event): + failures.append(f"write execution for {write_tool} is missing execution idempotency key evidence") + if matching_tool_events(event, tool_failed_events, first_terminal_order): + failures.append(f"write execution for {write_tool} has matching ToolCallFailed") + elif not matching_tool_events(event, tool_completed_events, first_terminal_order): + failures.append(f"write execution for {write_tool} is missing ToolCallCompleted") + write_tool_call_id = tool_call_id(event) + if not write_tool_call_id: + failures.append(f"write execution for {write_tool} is missing toolCallID") + continue + matching_write_requests = [ + request for request in write_request_events + if tool_name(request) == write_tool + and tool_call_id(request) == write_tool_call_id + and seq(request) < write_order + ] + if not matching_write_requests: + failures.append(f"write execution for {write_tool} has no matching preceding mapped request") + continue + write_approval_id = approval_id(event) + if not write_approval_id: + failures.append(f"write execution for {write_tool} is missing approvalID") + continue + matching_approval_requests = [ + approval for approval in approval_request_events + if approval_id(approval) == write_approval_id + and tool_name(approval) == write_tool + and content_field(approval, "toolCallID") == write_tool_call_id + and seq(approval) < write_order + and any(seq(request) < seq(approval) for request in matching_write_requests) + ] + if not matching_approval_requests: + failures.append(f"write execution for {write_tool} has no matching preceding ApprovalRequested") + matching_approved = [ + approval for approval in approval_approved_events + if approval_id(approval) == write_approval_id + and seq(approval) < write_order + and any(seq(request) < seq(approval) for request in matching_approval_requests) + ] + if not matching_approved: + failures.append(f"write execution for {write_tool} has no matching preceding ApprovalApproved") + matching_declined = [ + approval for approval in approval_declined_events + if approval_id(approval) == write_approval_id and seq(approval) < write_order + ] + if matching_declined: + failures.append(f"write execution for {write_tool} follows ApprovalDeclined") +if not execution_idempotency_events: + failures.append("missing write ToolCallStarted execution idempotency key evidence") + +missing_idempotency_tools = sorted( + tool for tool in {tool_name(event) for event in write_exec_events} + if tool not in {tool_name(event) for event in execution_idempotency_events} +) +if missing_idempotency_tools: + failures.append( + "missing write execution idempotency key evidence for: " + ", ".join(missing_idempotency_tools) + ) + +starts_by_tool = {} +for event in write_start_events: + starts_by_tool.setdefault(tool_name(event), 0) + starts_by_tool[tool_name(event)] += 1 +for write_tool, count in starts_by_tool.items(): + if count > 1: + failures.append(f"duplicate write execution starts for {write_tool}") + +execution_idempotency_by_tool = {} +for event in execution_idempotency_events: + execution_idempotency_by_tool.setdefault(tool_name(event), set()).add(execution_idempotency_value(event)) +for write_tool, keys in execution_idempotency_by_tool.items(): + if len(keys) > 1: + failures.append(f"multiple write execution idempotency keys for {write_tool}") +if not terminal_events: + failures.append("missing terminal completion event") +elif write_exec_events: + latest_write_seq = max(seq(event) for event in write_exec_events) + earliest_terminal_seq = min(seq(event) for event in terminal_events) + if earliest_terminal_seq <= latest_write_seq: + failures.append("terminal completion event does not follow all write executions") +if not task_terminal_events: + failures.append("missing final TaskSucceeded lifecycle event") +else: + final_task_terminal = max(task_terminal_events, key=seq) + final_task_type = event_type(final_task_terminal) + if final_task_type != "TaskSucceeded": + failures.append(f"final Task lifecycle outcome is {final_task_type}, want TaskSucceeded") + final_event = max(events, key=seq) if events else None + final_event_type = event_type(final_event) if final_event else "" + if final_event_type != "TaskSucceeded": + failures.append(f"final execution event is {final_event_type}, want TaskSucceeded") + +if failures: + print("Fibey Foundry Responses verification failed:", file=sys.stderr) + for failure in failures: + print(f"- {failure}", file=sys.stderr) + raise SystemExit(1) + +print("Fibey Foundry Responses verification passed:") +print(f"- read events: {len(read_events)}") +print(f"- write requests: {len(write_request_events)}") +print(f"- approval requests: {len(approval_request_events)}") +print(f"- approval decisions: {len(approval_approved_events)}") +print(f"- execution idempotency evidence events: {len(execution_idempotency_events)}") +print(f"- completed tool calls: {len(tool_completed_events)}") +print(f"- terminal events: {len(terminal_events)}") +PY diff --git a/examples/harness/foundry-responses/Dockerfile b/examples/harness/foundry-responses/Dockerfile new file mode 100644 index 000000000..00920fef9 --- /dev/null +++ b/examples/harness/foundry-responses/Dockerfile @@ -0,0 +1,13 @@ +# syntax=docker/dockerfile:1 +FROM golang:1.26-alpine AS build +WORKDIR /src +COPY go.mod go.sum ./ +RUN go mod download +COPY . . +RUN mkdir -p /out && CGO_ENABLED=0 GOOS=linux go build -o /out/orka-foundry-responses-harness-adapter ./examples/harness/foundry-responses + +FROM gcr.io/distroless/static:nonroot +COPY --from=build /out/orka-foundry-responses-harness-adapter /orka-foundry-responses-harness-adapter +USER 65532:65532 +EXPOSE 8090 +ENTRYPOINT ["/orka-foundry-responses-harness-adapter"] diff --git a/examples/harness/foundry-responses/README.md b/examples/harness/foundry-responses/README.md new file mode 100644 index 000000000..e17f68282 --- /dev/null +++ b/examples/harness/foundry-responses/README.md @@ -0,0 +1,163 @@ +# Foundry hosted Responses AgentRuntime adapter + +This adapter presents an Azure AI Foundry **hosted AgentKit agent** endpoint as an `orka.harness.v1` runtime. It targets endpoint-scoped hosted Responses: + +```text +POST /agents//endpoint/protocols/openai/responses?api-version=... +``` + +Use this adapter for AgentKit agents deployed as Foundry hosted agents. Use `examples/harness/foundry` only for the older Assistants/threads/run protocol. + +## Security model + +- The adapter never sends request-level `tools` to hosted `/responses`; hosted AgentKit must be statically configured with the safe function schemas it is allowed to request. +- Orka remains authoritative. Hosted AgentKit can request a function by name, but Orka still validates the request against the Task policy and Tool CRDs, performs approval checks, injects idempotency keys, executes/brokers the tool, and audits the result. +- Orka Tool URLs, auth refs, headers, and production tool credentials are not sent to Foundry. +- Adapter bearer tokens and Foundry credentials must live in Kubernetes Secrets or environment variables. Do not put them in `AgentRuntime` specs or logs. + +## Configuration + +| Env var | Purpose | +| --- | --- | +| `ORKA_FOUNDRY_RESPONSES_ADAPTER_ADDR` | HTTP listen address, default `:8090`. | +| `ORKA_FOUNDRY_RESPONSES_RUNTIME_NAME` | Runtime name advertised in `/v1/capabilities`, default `foundry-agentkit-responses`. | +| `ORKA_FOUNDRY_RESPONSES_ADAPTER_BEARER_TOKEN` | Bearer token Orka uses for mutating/streaming harness endpoints. | +| `ORKA_FOUNDRY_RESPONSES_ENDPOINT` | Preferred full hosted Responses endpoint URL, including `/agents//endpoint/protocols/openai/responses`. The adapter appends `api-version` from `ORKA_FOUNDRY_RESPONSES_API_VERSION` when missing. | +| `ORKA_FOUNDRY_RESPONSES_PROJECT_ENDPOINT` + `ORKA_FOUNDRY_RESPONSES_AGENT_NAME` | Optional alternative to build the hosted Responses endpoint from a project endpoint and agent name. | +| `ORKA_FOUNDRY_RESPONSES_API_VERSION` | API-version query value, default `v1`, matching the current `AIProjectClient.get_openai_client(agent_name=...)` default query. Override it for deployments pinned to another preview version. | +| `ORKA_FOUNDRY_RESPONSES_API_KEY` | Static API-key auth mode. Tests/demo only unless your deployment standard permits it. | +| `ORKA_FOUNDRY_RESPONSES_AUTH_BEARER` | Static bearer auth mode. Tests/demo only unless supplied by a production token refresher sidecar. | +| `ORKA_FOUNDRY_RESPONSES_TOKEN_AUDIENCE` | Reserved for future workload-identity token refresh support; currently not used. | +| `ORKA_FOUNDRY_RESPONSES_BROKERED_CONTINUATION_PROOF` | Required whenever brokered tool classes are enabled. The Orka-only proof is sent on hosted Responses continuations in both the `X-AgentKit-Brokered-Continuation-Proof` header and `brokered_continuation_proof` request-body field, so gateways that strip custom headers can still forward it. Set it to match AgentKit's `AGENTKIT_FOUNDRY_BROKERED_CONTINUATION_PROOF`. | +| `ORKA_FOUNDRY_RESPONSES_BROKERED_TOOL_CLASSES` | Comma-separated static classes the hosted AgentKit deployment has been configured and conformance-tested to request, e.g. `read` or `read,write`. Empty means observed-only. | +| `ORKA_FOUNDRY_RESPONSES_POLL_TIMEOUT` | Per-request timeout for hosted Responses calls, default `20s`. | +| `ORKA_FOUNDRY_RESPONSES_STATE_RETENTION` | How long terminal in-memory turn/session state is retained, default `10m`. | + +Exactly one Foundry auth mode (`API_KEY` or `AUTH_BEARER`) must be set. + +## Endpoint safety rules + +The hosted Responses endpoint must: + +- use HTTPS in production; +- use HTTP only for loopback tests (`localhost`, `127.0.0.1`, or `::1`); +- end in `/responses`; +- not include username/password, fragments, or query parameters other than `api-version`. + +The adapter returns degraded health and rejects starts when the endpoint is unsafe. `GET /v1/health` always returns a harness health body; `GET /v1/ready` is a Kubernetes readiness helper that returns HTTP 503 until the same configuration is ready. + +## Capability discipline + +Capabilities must reflect the **static schemas actually deployed in AgentKit**: + +- If `ORKA_FOUNDRY_RESPONSES_BROKERED_TOOL_CLASSES` is empty, the adapter advertises only `observed` and `supportsContinuation=false`. +- If it is `read`, the adapter advertises brokered read only. +- Advertise `write` only after the hosted AgentKit deployment has a static write schema and passes write conformance. Orka will still gate the write with approval/idempotency, but the hosted model must not be told it can request writes unless that path is intentionally enabled. + +AgentRuntime readiness deliberately deep-probes every advertised brokered class. Because this adapter never sends request-level `tools`, the hosted AgentKit deployment must statically expose the probe-only `conformance_read` and/or `conformance_write` schemas in addition to its real tools. These schemas take an empty object and must be safe to call: Orka's conformance client supplies the synthetic result and no production tool credential is sent to Foundry. Fake-server tests alone are not sufficient; if the hosted deployment cannot request the matching probe tool, leave that brokered class unadvertised and readiness will fail closed. + +## Protocol mapping + +Initial turn: + +```json +{"input":"Investigate incident"} +``` + +No `tools` field is sent. + +Hosted AgentKit function call: + +```json +{"type":"function_call","call_id":"call_1","name":"check-network-telemetry","arguments":"{\"site\":\"quincy-north\"}"} +``` + +Adapter emits `ToolCallRequested` with the exact `call_id`, function name, and compact JSON object arguments. If a hosted response returns multiple `function_call` items, the adapter emits one `ToolCallRequested` frame per call and waits until Orka returns every pending result before continuing. + +Orka continuation: + +```json +{"type":"function_call_output","call_id":"call_1","output":"{\"approved\":true,\"output\":{\"success\":true}}","status":"completed"} +``` + +The hosted continuation request includes `previous_response_id`, `agent_session_id` when a Foundry session is known, and one or more `function_call_output` items. Raw REST calls include the hosted-agent feature header required by Foundry hosted-agent endpoints. + +## Output and error encoding + +`function_call_output.output` is always a compact JSON string: + +- successful object result: `{"approved":true,"output":}` +- successful array or scalar result: `{"approved":true,"output":{"result":}}` +- declined approval or policy/execution error: `{"approved":false,"error":}` + +Object outputs remain unchanged. Arrays and scalars are wrapped under `output.result` so the AgentKit continuation always receives an object-shaped `output`. Error codes are allowlisted and messages are replaced with stable generic text before crossing the hosted-provider boundary. Approval decline, tool policy rejection, and tool execution failure fixtures live under `testdata/golden/`. + +## State, restart, and sessions + +This MVP stores turn state in memory. That is intentionally fail-safe: + +- duplicate identical `/continue` calls for a submitted call are accepted without a second hosted continuation; +- conflicting duplicate `/continue` calls are rejected; +- if the adapter restarts while a tool approval is pending, `/continue` returns `turn not found` and does not call Foundry, so the adapter itself does not duplicate a side effect; +- Orka's broker/idempotency ledger remains the source of truth for actual write execution. + +The adapter captures Foundry session identifiers from the hosted response body (`agent_session_id`) and compatibility headers such as `x-agent-session-id`, then reuses the session in the `agent_session_id` request field for later calls with the same Orka `runtimeSessionID`. Session/auth values are stored only in memory and are not logged. + +## Local build + +```bash +docker build -t foundry-responses-harness-adapter:local -f examples/harness/foundry-responses/Dockerfile . +``` + + +## Kubernetes smoke skeleton + +`kubernetes.example.yaml` contains a credentials-free Deployment, Service, Secret placeholders, and matching `AgentRuntime` facade for a read-profile hosted Responses smoke. It references the local image tag built above; load that image into your local cluster or replace it with an explicitly published image. Replace the `REDACTED` values through your secret-management flow, set the hosted Responses endpoint or project/agent-name pair, ensure the hosted agent statically exposes the probe-only `conformance_read` schema, and keep `ORKA_FOUNDRY_RESPONSES_BROKERED_TOOL_CLASSES` narrowed to classes whose live AgentRuntime conformance passed. + +```bash +kubectl apply -f examples/harness/foundry-responses/kubernetes.example.yaml +kubectl wait --for=condition=Ready agentruntime/sample-foundry-responses-runtime --timeout=60s +``` + +For write-profile smoke, first prove the hosted AgentKit deployment has a static write schema, then add `write` to both the adapter env and the `AgentRuntime.spec.capabilities.brokeredToolClasses`. Orka still performs approval and idempotent write execution; the hosted endpoint receives only `function_call_output` continuations. + +## Tests + +```bash +go test ./examples/harness/foundry-responses +``` + +For the full deterministic local validation bundle, including the focused Orka harness/controller suites and explicit AgentKit fixture tests when --agentkit is provided: + +```bash +examples/harness/foundry-responses/validate.sh +``` + +For the live Foundry hosted AgentKit smoke gate, first run the credentials-safe preflight and then apply only when your current Kubernetes context is the intended Orka cluster: + +```bash +examples/harness/foundry-responses/live-smoke.sh +ORKA_FOUNDRY_RESPONSES_ADAPTER_IMAGE= \ + examples/harness/foundry-responses/live-smoke.sh --apply --wait + +# After the live task completes, capture redacted evidence. +examples/harness/foundry-responses/live-evidence.sh \ + --namespace \ + --runtime fibey-agentkit-foundry-responses \ + --task fibey-foundry-responses-quincy-north-alert +``` + +The tests use a fake hosted Responses server and golden fixtures for initial requests, function calls, `ToolCallRequested`, continuations, final messages, error encoding, and buffered multiple-call behavior. + +See [`VALIDATION.md`](VALIDATION.md) for the brokered-plan evidence matrix, local commands, and remaining live Foundry/Fibey gates. + +## Troubleshooting + +| Symptom | Likely cause | Fix | +| --- | --- | --- | +| Health is degraded | Missing adapter bearer, missing/unsafe endpoint, bad brokered class config, or missing Foundry auth | Check env vars and endpoint safety. | +| `tools` rejected by hosted endpoint | You are using the wrong adapter or adding request-level tools | This adapter intentionally never sends `tools`; configure AgentKit static schemas instead. | +| Brokered start rejected | Adapter does not advertise the requested tool class | Set `ORKA_FOUNDRY_RESPONSES_BROKERED_TOOL_CLASSES` only after AgentKit static schema/conformance is ready. | +| Turn fails with unknown tool | Hosted AgentKit requested a function name not present in Orka's safe `StartTurnRequest.input.tools` | Fix the AgentKit static schema or Task `allowedTools`. | +| Turn fails with malformed arguments | Hosted AgentKit emitted non-object or invalid JSON arguments | Fix the hosted schema/prompting; Orka rejects before tool execution. | +| Continue returns `turn not found` after restart | In-memory state was lost while waiting for approval | Re-run/fail the Task safely; no hosted continuation was sent by the restarted adapter. | diff --git a/examples/harness/foundry-responses/VALIDATION.md b/examples/harness/foundry-responses/VALIDATION.md new file mode 100644 index 000000000..2a16e9d59 --- /dev/null +++ b/examples/harness/foundry-responses/VALIDATION.md @@ -0,0 +1,134 @@ +# Foundry hosted Responses validation matrix + +This document maps the brokered Foundry hosted Responses plan to deterministic +local evidence and the remaining live gates. It intentionally contains no live +endpoints, credentials, tokens, or Foundry project identifiers. + +## Local deterministic validation + +Run the bundled deterministic validator from the Orka repository root: + +```bash +examples/harness/foundry-responses/validate.sh +``` + +Or run the component commands manually: + +```bash +go test ./examples/harness/foundry-responses -count=1 + +go test \ + ./examples/harness/foundry \ + ./examples/harness/foundry-responses \ + ./examples/harness/echo \ + ./internal/harness \ + ./internal/harness/conformance + +go test ./internal/controller -run 'Test.*(AgentRuntime|Harness|Brokered|Runtime)' + +find examples -type f -name '*.sh' -print0 | sort -z | \ + xargs -0 -n1 bash -n +``` + +Full non-e2e validation: + +```bash +make test +``` + +When a clean AgentKit checkout is available, run the shared deterministic +Foundry brokered protocol suite explicitly: + +```bash +examples/harness/foundry-responses/validate.sh --agentkit ../agentkit +``` + +Equivalent manual command from the AgentKit checkout's `runtimes/common` directory: + +```bash +uv run --extra dev pytest -q \ + tests/test_foundry_brokered_protocol.py \ + tests/test_brokered_schema.py \ + tests/test_foundry_protocol.py +``` + +## Requirement evidence + +| Plan requirement | Local evidence | +| --- | --- | +| Separate hosted Responses adapter path | `examples/harness/foundry-responses/{main.go,main_test.go,README.md,Dockerfile}` | +| Existing Assistants adapter remains Assistants-only | `examples/harness/foundry/README.md` distinguishes Assistants/threads from hosted Responses. | +| Hosted `/responses` initial request does not send request-level `tools` | `TestResponsesAdapterObservedTurnCompletes`, `TestResponsesAdapterBrokeredReadContinuationAndGoldenFixtures`, and conformance tests assert no `tools` field reaches the fake hosted endpoint. | +| Endpoint URL safety | `TestResponsesEndpointSafety`, `TestResponsesAdapterDoesNotFollowCredentialedRedirects`, and `TestSanitizeEndpointDoesNotReturnRawMalformedURL`. | +| Mutating/streaming harness endpoints require bearer auth | `TestResponsesAdapterPassesObservedConformanceByDefault`, `TestResponsesAdapterPassesBrokeredReadConformance`, and `TestResponsesAdapterPassesBrokeredWriteConformance` run harness conformance with `RequireAuth`. | +| Observed hosted response maps to `TurnCompleted` | `TestResponsesAdapterObservedTurnCompletes` and observed conformance. | +| Responses `function_call` maps to `ToolCallRequested` | `TestResponsesAdapterBrokeredReadContinuationAndGoldenFixtures`, `TestResponsesConsumesAgentKitBrokeredFixtures`, and brokered read/write conformance. | +| Orka `ToolCallResult` maps to `function_call_output` | `TestResponsesAdapterBrokeredReadContinuationAndGoldenFixtures`, `TestResponsesAdapterWriteParksUntilDeclinedApprovalContinue`, `TestResponsesAdapterContinuesToolExecutionFailurePayload`, golden fixtures, and AgentKit fixture tests. | +| Canonical error/decline encoding is stable | `TestCanonicalErrorAndDeclineOutputFixtures` and `TestResponsesAgentKitErrorPayloadFixturesMatchCanonicalEncoding`. | +| Unknown tool and malformed arguments are rejected before Orka execution | `TestResponsesAdapterRejectsUnknownToolBeforeOrkaExecution` and `TestResponsesAdapterRejectsMalformedArguments`. | +| Multiple hosted calls are buffered until all results arrive | `TestResponsesAdapterMultipleFunctionCallsBufferedAndContinued`. | +| Duplicate/replayed function calls fail closed | `TestResponsesRepeatedSubmittedFunctionCallFailsTurn` and `TestResponsesMixedRepeatedFunctionCallFailsTurn`. | +| Duplicate identical `/continue` is idempotent and conflicting duplicates reject | `TestResponsesAdapterDuplicateContinueIsIdempotentAndConflictsReject`, `TestResponsesAdapterAlreadySubmittedContinueDoesNotResubmit`, and `TestResponsesAdapterAlreadySubmittedPendingResultIsNoop`. | +| Continuation failures fail closed rather than duplicating hosted progress | `TestResponsesAdapterContinuationFailureFailsClosedWithoutDuplicatePost`. | +| Brokered approval waits remain governed by Orka | The adapter does not expire pending brokered calls after Orka may have executed an approved tool, and `TestResponsesAdapterBrokeredMaxTurnIsUnknown` verifies that it advertises no misleading fixed turn ceiling. | +| Restart/state loss fails safely without hosted continuation | `TestResponsesAdapterStateLossContinueFailsSafely`. | +| Runtime session continuity | `TestResponsesAdapterRuntimeSessionHeaderReuse` and `TestResponsesAdapterContinuationUsesTurnSessionAfterSessionMapCleanup`. | +| Hosted response status handling is fail-closed | `TestResponsesFailureStatusDoesNotCompleteWithPartialText`, `TestResponsesFailureStatusWithFunctionCallFailsBeforeToolRequest`, `TestResponsesNonTerminalStatusDoesNotCompleteWithPartialText`, and `TestResponsesMissingStatusDoesNotCompleteWithPartialText`. | +| Large hosted output and uncertain initial platform failures are safe | `TestResponsesLargeOutputFails` and `TestResponsesInitialPlatformErrorRetainsFailedTurn`. | +| No secrets or endpoint URLs in golden fixtures | `TestResponsesGoldenFixturesDoNotContainEndpointsOrSecrets`. | +| Existing Orka broker approval/idempotency/write ledger behavior | `go test ./internal/controller -run 'Test.*(AgentRuntime|Harness|Brokered|Runtime)'`, especially brokered write approval, decline, replay, and unresolved-ledger tests in `internal/controller/harness_wrapper_test.go`. | +| Turn admission, credential retention, cancellation, and endpoint safety | Focused adapter tests cover bounded consumed-turn tombstones, discarding unused resolved turn environment values, cancel/initial-response races, malformed/mismatched cancel rejection before mutation, and empty-hostname endpoint rejection; `validate.sh` also checks Docker-context inclusion and the live-smoke shell preflight. | +| Fibey live evidence verifier behavior | `examples/harness/foundry-responses/validate.sh` tests paginated event aggregation and runs `verify-foundry-responses.sh` against pass/fail fixtures, including truncated pages and write execution after a terminal event. | +| Kubernetes smoke skeleton is credentials-free | `examples/harness/foundry-responses/kubernetes.example.yaml` uses `REDACTED` placeholders and a read-only advertised class by default. | + +## Live gates that cannot be satisfied by local fixtures + +Use the credentials-safe live smoke helper as the first live preflight/deploy step: + +```bash +examples/harness/foundry-responses/live-smoke.sh +ORKA_FOUNDRY_RESPONSES_ADAPTER_IMAGE= \ + examples/harness/foundry-responses/live-smoke.sh --apply --wait +``` + +The helper validates required environment without printing secret values and can +deploy the adapter `Deployment`, `Service`, `Secret` placeholders, and matching +`AgentRuntime` facade into the selected namespace. It does not replace the +required real hosted AgentKit deployment, downstream tools, human approvals, or +Fibey scenario verification. + +The following remain required before declaring the full hosted Foundry/Fibey plan +complete: + +1. Deploy an AgentKit prototype as a real Foundry hosted agent with static safe + brokered schemas, the probe-only `conformance_read`/`conformance_write` + schemas for every advertised class, and a configured brokered continuation + proof. +2. Deploy the Orka `foundry-responses` adapter with real Foundry auth and a real + hosted `/responses` endpoint. +3. Verify `AgentRuntime` readiness for the read profile. +4. Run a read task and verify `ToolCallRequested` then successful completion. +5. Enable write only after the hosted AgentKit deployment has a static write + schema and passes write conformance. +6. Run a write task and verify `ApprovalRequested`, no downstream execution + before approval, approved continuation, idempotency key delivery, and no + credential/tool URL leakage in logs. +7. Run the Fibey read/write scenario with `check-network-telemetry`, + `get-active-incidents`, `dispatch-work-order`, and optional + `escalate-incident`, then verify replay produces no second dispatch. Capture + Orka task events and run: + + ```bash + examples/fibey-custom-agent-demo/verify-foundry-responses.sh \ + --task fibey-foundry-responses-quincy-north-alert \ + --namespace + ``` + + To collect a credentials-safe evidence bundle after the run: + + ```bash + examples/harness/foundry-responses/live-evidence.sh \ + --namespace \ + --runtime fibey-agentkit-foundry-responses \ + --task fibey-foundry-responses-quincy-north-alert + ``` diff --git a/examples/harness/foundry-responses/fetch_task_events.py b/examples/harness/foundry-responses/fetch_task_events.py new file mode 100755 index 000000000..8da0d3290 --- /dev/null +++ b/examples/harness/foundry-responses/fetch_task_events.py @@ -0,0 +1,129 @@ +#!/usr/bin/env python3 +"""Fetch a complete Orka task event stream through the paginated CLI.""" + +from __future__ import annotations + +import argparse +import json +import subprocess +import sys +from pathlib import Path +from typing import Any, Callable + +PAGE_LIMIT = 1000 +MAX_PAGES = 10000 + +Runner = Callable[..., subprocess.CompletedProcess[str]] + + +def _int_field(payload: dict[str, Any], name: str) -> int: + value = payload.get(name, 0) + if isinstance(value, bool): + raise ValueError(f"{name} must be an integer") + try: + parsed = int(value) + except (TypeError, ValueError) as exc: + raise ValueError(f"{name} must be an integer") from exc + if parsed < 0: + raise ValueError(f"{name} must be non-negative") + return parsed + + +def fetch_task_events( + task: str, + namespace: str, + *, + runner: Runner = subprocess.run, +) -> dict[str, Any]: + after = 0 + all_events: list[dict[str, Any]] = [] + first_payload: dict[str, Any] | None = None + + for _ in range(MAX_PAGES): + command = [ + "orka", + "task", + "events", + task, + "--namespace", + namespace, + "--after", + str(after), + "--limit", + str(PAGE_LIMIT), + "--output", + "json", + ] + result = runner(command, capture_output=True, text=True, check=False) + if result.returncode != 0: + detail = result.stderr.strip() or result.stdout.strip() or "unknown error" + raise RuntimeError(f"orka task events failed: {detail}") + + try: + payload = json.loads(result.stdout) + except json.JSONDecodeError as exc: + raise ValueError(f"orka task events returned invalid JSON: {exc}") from exc + if not isinstance(payload, dict): + raise ValueError("orka task events response must be a JSON object") + if first_payload is None: + first_payload = payload + + response_after = _int_field(payload, "afterSeq") + latest = _int_field(payload, "latestSeq") + if response_after != after: + raise ValueError( + f"orka task events returned afterSeq {response_after}, expected {after}" + ) + events = payload.get("events") + if not isinstance(events, list): + raise ValueError("orka task events response is missing an events array") + + page_events: list[dict[str, Any]] = [] + last_seq = after + expected_seq = after + 1 + for event in events: + if not isinstance(event, dict): + raise ValueError("orka task events returned a non-object event") + seq = _int_field(event, "seq") + if seq != expected_seq: + raise ValueError( + f"orka task events returned sequence {seq}, expected {expected_seq}" + ) + last_seq = seq + expected_seq += 1 + page_events.append(event) + all_events.extend(page_events) + + if last_seq >= latest: + base = dict(first_payload) + base["afterSeq"] = 0 + base["latestSeq"] = latest + base["events"] = all_events + return base + if not page_events: + raise RuntimeError( + f"orka task events stopped at sequence {after} before latestSeq {latest}" + ) + after = last_seq + + raise RuntimeError(f"orka task events exceeded {MAX_PAGES} pages") + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--task", required=True) + parser.add_argument("--namespace", required=True) + parser.add_argument("--output", required=True, type=Path) + args = parser.parse_args() + + try: + payload = fetch_task_events(args.task, args.namespace) + args.output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n") + except (OSError, RuntimeError, ValueError) as exc: + print(f"error: {exc}", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/examples/harness/foundry-responses/kubernetes.example.yaml b/examples/harness/foundry-responses/kubernetes.example.yaml new file mode 100644 index 000000000..04659452f --- /dev/null +++ b/examples/harness/foundry-responses/kubernetes.example.yaml @@ -0,0 +1,118 @@ +# Example Kubernetes deployment for the Foundry hosted Responses adapter. +# Replace REDACTED placeholders through your secret-management flow before use. +# Do not commit real Foundry credentials or adapter bearer values. +apiVersion: v1 +kind: Secret +metadata: + name: sample-foundry-responses-runtime-token + annotations: + orka.ai/agent-runtime-endpoint: http://sample-foundry-responses-runtime.default.svc.cluster.local:8080 + labels: + orka.ai/agent-runtime-auth: "true" + orka.ai/agent-runtime-name: sample-foundry-responses-runtime +stringData: + harness-bearer: REDACTED +--- +apiVersion: v1 +kind: Secret +metadata: + name: sample-foundry-responses-adapter-config +stringData: + foundry-auth: REDACTED + continuation-proof: REDACTED +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: sample-foundry-responses-runtime + labels: + app.kubernetes.io/name: sample-foundry-responses-runtime +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: sample-foundry-responses-runtime + template: + metadata: + labels: + app.kubernetes.io/name: sample-foundry-responses-runtime + spec: + containers: + - name: adapter + # Build and load/publish examples/harness/foundry-responses/Dockerfile first. + image: foundry-responses-harness-adapter:local + imagePullPolicy: IfNotPresent + env: + - name: ORKA_FOUNDRY_RESPONSES_ADAPTER_ADDR + value: :8090 + - name: ORKA_FOUNDRY_RESPONSES_RUNTIME_NAME + value: sample-foundry-responses-runtime + - name: ORKA_FOUNDRY_RESPONSES_ADAPTER_BEARER_TOKEN + valueFrom: + secretKeyRef: + name: sample-foundry-responses-runtime-token + key: harness-bearer + # Use ORKA_FOUNDRY_RESPONSES_PROJECT_ENDPOINT plus AGENT_NAME instead of + # the full endpoint when you want the adapter to compose the hosted URL. + - name: ORKA_FOUNDRY_RESPONSES_ENDPOINT + value: https://example.services.ai.azure.com/agents/sample-agent/endpoint/protocols/openai/responses + - name: ORKA_FOUNDRY_RESPONSES_API_VERSION + value: v1 + - name: ORKA_FOUNDRY_RESPONSES_API_KEY + valueFrom: + secretKeyRef: + name: sample-foundry-responses-adapter-config + key: foundry-auth + - name: ORKA_FOUNDRY_RESPONSES_BROKERED_CONTINUATION_PROOF + valueFrom: + secretKeyRef: + name: sample-foundry-responses-adapter-config + key: continuation-proof + # The hosted agent must statically expose probe-only conformance_read + # before read can pass AgentRuntime readiness. Advertise only classes + # whose live brokered conformance passed. + - name: ORKA_FOUNDRY_RESPONSES_BROKERED_TOOL_CLASSES + value: read + ports: + - name: http + containerPort: 8090 + readinessProbe: + httpGet: + path: /v1/ready + port: http + periodSeconds: 5 +--- +apiVersion: v1 +kind: Service +metadata: + name: sample-foundry-responses-runtime +spec: + selector: + app.kubernetes.io/name: sample-foundry-responses-runtime + ports: + - name: http + port: 8080 + targetPort: http +--- +apiVersion: core.orka.ai/v1alpha1 +kind: AgentRuntime +metadata: + name: sample-foundry-responses-runtime +spec: + contractVersion: orka.harness.v1 + deployment: + mode: external-endpoint + endpoint: http://sample-foundry-responses-runtime.default.svc.cluster.local:8080 + clientAuth: + bearerTokenSecretRef: + name: sample-foundry-responses-runtime-token + key: harness-bearer + capabilities: + toolExecutionModes: + - observed + - brokered + brokeredToolClasses: + - read + supportsCancel: true + supportsRuntimeSessions: true + supportsContinuation: true diff --git a/examples/harness/foundry-responses/live-evidence.sh b/examples/harness/foundry-responses/live-evidence.sh new file mode 100755 index 000000000..aa25d0697 --- /dev/null +++ b/examples/harness/foundry-responses/live-evidence.sh @@ -0,0 +1,505 @@ +#!/usr/bin/env bash +set -euo pipefail + +usage() { + cat >&2 <<'USAGE' +Usage: examples/harness/foundry-responses/live-evidence.sh [--namespace NAME] [--runtime NAME] [--task NAME] [--out DIR] [--logs-since DURATION] + +Capture a credentials-safe evidence bundle for the live Foundry hosted +Responses/Fibey gate after the task has run. The bundle stores Kubernetes +AgentRuntime metadata, the complete paginated Orka task event stream, approvals, +verifier output, and a summary-only adapter log scan. It intentionally does not +store raw adapter logs. + +Defaults: + --namespace default + --runtime fibey-agentkit-foundry-responses + --task fibey-foundry-responses-quincy-north-alert + --out ./foundry-responses-live-evidence- + --logs-since empty, meaning scan all available deployment logs. Set a kubectl + duration such as 30m only when the run start time is known. + +Required commands: kubectl, orka, python3. +USAGE +} + +namespace="default" +runtime="fibey-agentkit-foundry-responses" +task="fibey-foundry-responses-quincy-north-alert" +out_dir="" +logs_since="" + +while [[ $# -gt 0 ]]; do + case "$1" in + --namespace) + [[ $# -ge 2 ]] || { echo "--namespace requires a value" >&2; exit 2; } + namespace="$2" + shift 2 + ;; + --runtime) + [[ $# -ge 2 ]] || { echo "--runtime requires a value" >&2; exit 2; } + runtime="$2" + shift 2 + ;; + --task) + [[ $# -ge 2 ]] || { echo "--task requires a value" >&2; exit 2; } + task="$2" + shift 2 + ;; + --out) + [[ $# -ge 2 ]] || { echo "--out requires a value" >&2; exit 2; } + out_dir="$2" + shift 2 + ;; + --logs-since) + [[ $# -ge 2 ]] || { echo "--logs-since requires a value" >&2; exit 2; } + logs_since="$2" + shift 2 + ;; + -h|--help) + usage + exit 0 + ;; + *) + echo "unknown argument: $1" >&2 + usage + exit 2 + ;; + esac +done + +require_cmd() { + command -v "$1" >/dev/null 2>&1 || { echo "error: $1 is required" >&2; exit 2; } +} + +require_cmd awk +require_cmd kubectl +require_cmd orka +require_cmd python3 + +if [[ -z "$out_dir" ]]; then + out_dir="foundry-responses-live-evidence-$(date -u +%Y%m%dT%H%M%SZ)" +fi + +umask 077 +mkdir -p "$out_dir" +runtime_tmp="" +events_tmp="" +approvals_tmp="" +pods_tmp="" +log_scan_tmp="" +log_pipe_dir="" +cleanup() { + [[ -z "${runtime_tmp:-}" ]] || rm -f "$runtime_tmp" + [[ -z "${events_tmp:-}" ]] || rm -f "$events_tmp" + [[ -z "${approvals_tmp:-}" ]] || rm -f "$approvals_tmp" + [[ -z "${pods_tmp:-}" ]] || rm -f "$pods_tmp" + [[ -z "${log_scan_tmp:-}" ]] || rm -f "$log_scan_tmp" + [[ -z "${log_pipe_dir:-}" ]] || rm -rf "$log_pipe_dir" +} +trap cleanup EXIT + +report="$out_dir/README.md" +agentruntime_json="$out_dir/agentruntime.json" +events_json="$out_dir/task-events.json" +approvals_json="$out_dir/task-approvals.json" +verifier_out="$out_dir/fibey-verifier.txt" +log_scan="$out_dir/adapter-log-scan.txt" +artifact_forbidden_pattern="(api[-_]?key|authorization|bearer|secret|password|credential|txn-token|ORKA_FOUNDRY_RESPONSES_|https?://[^[:space:]\"<>]*(fibey-telemetry|fibey-incidents|fibey-dispatch|support-tool))" +log_forbidden_pattern="(api[-_]?key|authorization|bearer|secret|password|credential|txn-token|ORKA_FOUNDRY_RESPONSES_|https?://[^[:space:]\"<>]*(fibey-telemetry|fibey-incidents|fibey-dispatch|support-tool))" + +{ + echo "# Foundry Responses live evidence" + echo + echo "- Captured at: $(date -u +%Y-%m-%dT%H:%M:%SZ)" + echo "- Namespace: ${namespace}" + echo "- AgentRuntime: ${runtime}" + echo "- Task: ${task}" + echo + echo "Raw adapter logs are not stored by this script." +} >"$report" + +scan_saved_artifact() { + local file="$1" + local label="$2" + if [[ ! -s "$file" ]]; then + return + fi + if grep -Eiq "$artifact_forbidden_pattern" "$file"; then + rm -f "$file" + echo "error: forbidden credential/tool-url pattern detected in ${label}; removed ${file}" >&2 + exit 1 + fi +} + +runtime_tmp="$(mktemp)" +kubectl -n "$namespace" get agentruntime "$runtime" -o json >"$runtime_tmp" +python3 - "$runtime_tmp" "$agentruntime_json" <<'PY' +import json +import sys +from pathlib import Path + +runtime = json.loads(Path(sys.argv[1]).read_text()) +status = runtime.get("status") or {} +metadata = runtime.get("metadata") or {} +metadata_generation = metadata.get("generation") +observed_generation = status.get("observedGeneration") +if metadata_generation is not None and observed_generation != metadata_generation: + raise SystemExit("AgentRuntime status.observedGeneration does not match metadata.generation") +conditions = status.get("conditions") or [] +ready = [c for c in conditions if c.get("type") == "Ready"] +if not ready or str(ready[-1].get("status", "")).lower() != "true": + raise SystemExit("AgentRuntime Ready=True was not observed") +condition_generation = ready[-1].get("observedGeneration") +if condition_generation is not None and metadata_generation is not None and condition_generation != metadata_generation: + raise SystemExit("AgentRuntime Ready condition observedGeneration does not match metadata.generation") +observed = status.get("observedCapabilities") or status.get("observedCapabilitiesRaw") or {} +if isinstance(observed, dict): + modes = observed.get("toolExecutionModes") or [] + classes = observed.get("brokeredToolClasses") or [] + continuation = observed.get("supportsContinuation") +else: + modes, classes, continuation = [], [], None +if "brokered" not in modes: + raise SystemExit("AgentRuntime observed capabilities do not include brokered mode") +if not {"read", "write"}.issubset(set(classes)): + raise SystemExit("AgentRuntime observed capabilities do not include both read and write brokered classes") +if continuation is not True: + raise SystemExit("AgentRuntime observed capabilities do not include supportsContinuation=true") +summary = { + "kind": runtime.get("kind", "AgentRuntime"), + "metadata": { + "name": (runtime.get("metadata") or {}).get("name"), + "namespace": (runtime.get("metadata") or {}).get("namespace"), + "generation": (runtime.get("metadata") or {}).get("generation"), + }, + "status": { + "ready": ready[-1], + "observedCapabilities": observed, + "observedGeneration": status.get("observedGeneration"), + }, +} +Path(sys.argv[2]).write_text(json.dumps(summary, indent=2, sort_keys=True) + "\n") +PY +rm -f "$runtime_tmp" +scan_saved_artifact "$agentruntime_json" "AgentRuntime summary" + +events_tmp="$(mktemp)" +approvals_tmp="$(mktemp)" +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +python3 "${script_dir}/fetch_task_events.py" \ + --task "$task" \ + --namespace "$namespace" \ + --output "$events_tmp" +orka task approvals "$task" --namespace "$namespace" --output json >"$approvals_tmp" +python3 - "$events_tmp" "$events_json" <<'PY' +import json +import sys +from pathlib import Path + +payload = json.loads(Path(sys.argv[1]).read_text()) +if not isinstance(payload, dict) or not isinstance(payload.get("events"), list): + raise SystemExit("task event capture must be a paginated event object") +events = payload["events"] +latest_seq = int(payload.get("latestSeq", 0)) +after_seq = int(payload.get("afterSeq", 0)) +sequence_values = [ + int(event.get("seq", 0)) + for event in events + if isinstance(event, dict) and event.get("seq") is not None +] +if after_seq != 0: + raise SystemExit(f"task event capture is incomplete: afterSeq must be 0, got {after_seq}") +complete_sequence = len(sequence_values) == latest_seq and all( + seq == expected for expected, seq in enumerate(sequence_values, start=1) +) +if not complete_sequence: + raise SystemExit( + "task event capture is incomplete: " + f"sequences do not cover 1 through latestSeq {latest_seq}" + ) + +safe_top_level = { + "seq", + "type", + "eventType", + "severity", + "toolName", + "tool", + "name", + "toolCallID", + "toolCallId", + "approvalID", + "approvalId", + "targetTool", + "brokeredClass", + "executionState", + "idempotencyKey", + "executionIdempotencyKey", + "Idempotency-Key", + "createdAt", +} +safe_scalar_content_keys = { + "approvalID", + "approvalId", + "targetTool", + "toolCallID", + "toolCallId", + "toolName", + "tool", + "name", + "brokeredClass", + "executionState", + "decision", +} + + +def scalar(value): + return value if isinstance(value, (str, int, float, bool)) and not isinstance(value, type(None)) else None + + +def error_marker_is_set(value): + return value not in (None, "", False, 0, {}, []) + + +def has_direct_error(value): + if not isinstance(value, dict): + return False + return ( + error_marker_is_set(value.get("error")) + or error_marker_is_set(value.get("errorCode")) + or error_marker_is_set(value.get("toolError")) + ) + + +def safe_content(value): + if not isinstance(value, dict): + return {} + safe = {} + for key in safe_scalar_content_keys: + found = scalar(value.get(key)) + if found not in (None, ""): + safe[key] = found + for key in ("executionIdempotencyKey", "Execution-Idempotency-Key"): + idempotency = scalar(value.get(key)) + if isinstance(idempotency, str) and idempotency.strip(): + safe["executionIdempotencyKey"] = idempotency.strip() + break + for key in ("idempotencyKey", "Idempotency-Key"): + idempotency = scalar(value.get(key)) + if isinstance(idempotency, str) and idempotency.strip(): + safe["idempotencyKey"] = idempotency.strip() + break + harness = value.get("harness") + if isinstance(harness, dict): + frame_type = scalar(harness.get("frameType")) + if isinstance(frame_type, str) and frame_type: + safe["harness"] = {"frameType": frame_type} + return safe + + +safe_events = [] +for event in events: + if not isinstance(event, dict): + raise SystemExit("task event capture contains a non-object event") + safe_event = {} + for key in safe_top_level: + if key in event: + value = scalar(event[key]) + if value not in (None, ""): + safe_event[key] = value + content = event.get("content") + if isinstance(content, str): + try: + content = json.loads(content) + except Exception: # noqa: BLE001 - unsafe content is omitted, not stored + content = {} + redacted_content = safe_content(content) + if redacted_content: + safe_event["content"] = redacted_content + safe_event["hasError"] = has_direct_error(event) or has_direct_error(content) + safe_events.append(safe_event) + +safe_payload = { + "namespace": scalar(payload.get("namespace")), + "streamType": scalar(payload.get("streamType")), + "streamID": scalar(payload.get("streamID")), + "afterSeq": 0, + "latestSeq": latest_seq, + "events": safe_events, +} +Path(sys.argv[2]).write_text(json.dumps(safe_payload, indent=2, sort_keys=True) + "\n") +PY +scan_saved_artifact "$events_json" "redacted task events" +python3 - "$approvals_tmp" "$approvals_json" <<'PY' +import json +import sys +from pathlib import Path + +payload = json.loads(Path(sys.argv[1]).read_text()) +if isinstance(payload, dict): + items = payload.get("approvals") or payload.get("items") or payload.get("events") or [] + if not isinstance(items, list): + items = [payload] +elif isinstance(payload, list): + items = payload +else: + items = [] +summary = [] +for index, item in enumerate(items, start=1): + item = item if isinstance(item, dict) else {} + summary.append({ + "index": index, + "id": item.get("id"), + "status": item.get("status"), + "targetTool": item.get("targetTool"), + "toolCallID": item.get("toolCallID"), + "decisionTime": item.get("decisionTime"), + }) +Path(sys.argv[2]).write_text(json.dumps({"approvalCount": len(items), "approvals": summary}, indent=2, sort_keys=True) + "\n") +PY +scan_saved_artifact "$approvals_json" "task approvals summary" + +fibey_verifier="${script_dir}/../fibey-custom-agent-demo/verify-foundry-responses.sh" +if [[ ! -x "$fibey_verifier" ]]; then + fibey_verifier="${script_dir}/../../fibey-custom-agent-demo/verify-foundry-responses.sh" +fi +"$fibey_verifier" --json "$events_tmp" >/dev/null +"$fibey_verifier" --json "$events_json" >"$verifier_out" +scan_saved_artifact "$verifier_out" "Fibey verifier output" + +pods_tmp="$(mktemp)" +kubectl -n "$namespace" get pods -l "app.kubernetes.io/name=${runtime}" -o json >"$pods_tmp" +python3 - "$pods_tmp" <<'PY' +import json +import sys +from pathlib import Path + +pods = json.loads(Path(sys.argv[1]).read_text()).get("items") or [] +if not pods: + raise SystemExit("no adapter pods found for log evidence") +restarted = [] +for pod in pods: + pod_name = (pod.get("metadata") or {}).get("name", "") + statuses = (pod.get("status") or {}).get("containerStatuses") or [] + for status in statuses: + if int(status.get("restartCount") or 0) > 0: + restarted.append(f"{pod_name}/{status.get('name', '')}") +if restarted: + raise SystemExit("adapter pod restarts observed; previous logs must be inspected before evidence can pass: " + ", ".join(restarted)) +PY +rm -f "$pods_tmp" + +log_err="$out_dir/adapter-log-error.txt" +pods="$(kubectl -n "$namespace" get pods \ + -l "app.kubernetes.io/name=${runtime}" \ + -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>"$log_err")" +if [[ -z "$pods" ]]; then + { + echo "adapter log scan: FAILED" + echo "No adapter pods were found for app.kubernetes.io/name=${runtime}." + echo "kubectl error: $(tr '\n' ' ' <"$log_err")" + } >"$log_scan" + rm -f "$log_err" + cat "$log_scan" >&2 + exit 1 +fi +log_scan_tmp="$(mktemp)" +log_pipe_dir="$(mktemp -d)" +log_match_fifo="$log_pipe_dir/match" +log_count_fifo="$log_pipe_dir/count" +mkfifo "$log_match_fifo" "$log_count_fifo" +log_pods=0 +log_lines=0 +log_nonblank_lines=0 +while IFS= read -r pod; do + [[ -n "$pod" ]] || continue + log_pods=$((log_pods + 1)) + log_args=(logs "pod/${pod}" --all-containers) + if [[ -n "$logs_since" ]]; then + log_args+=(--since "$logs_since") + fi + : >"$log_scan_tmp" + grep -Ei "$log_forbidden_pattern" <"$log_match_fifo" >/dev/null & + match_pid=$! + awk ' + { count++; if ($0 ~ /[^[:space:]]/) nonblank++ } + END { printf "%d %d\n", count, nonblank } + ' <"$log_count_fifo" >"$log_scan_tmp" & + count_pid=$! + set +e + kubectl -n "$namespace" "${log_args[@]}" 2>>"$log_err" | tee "$log_match_fifo" >"$log_count_fifo" + pipeline_status=("${PIPESTATUS[@]}") + wait "$match_pid" + match_status=$? + wait "$count_pid" + count_status=$? + set -e + if (( pipeline_status[0] != 0 )); then + { + echo "adapter log scan: FAILED" + echo "Could not retrieve adapter logs from pod/${pod}. Raw logs were not stored." + echo "kubectl error: $(tr '\n' ' ' <"$log_err")" + } >"$log_scan" + rm -f "$log_err" + cat "$log_scan" >&2 + exit 1 + fi + if (( pipeline_status[1] != 0 || count_status != 0 || (match_status != 0 && match_status != 1) )); then + { + echo "adapter log scan: FAILED" + echo "Could not scan adapter logs from pod/${pod}. Raw logs were not stored." + } >"$log_scan" + rm -f "$log_err" + cat "$log_scan" >&2 + exit 1 + fi + if (( match_status == 0 )); then + { + echo "adapter log scan: FAILED" + echo "A forbidden credential/tool-url pattern was detected in adapter logs. Raw logs were not stored." + } >"$log_scan" + rm -f "$log_err" + cat "$log_scan" >&2 + exit 1 + fi + read -r pod_lines pod_nonblank_lines <"$log_scan_tmp" + log_lines=$((log_lines + pod_lines)) + log_nonblank_lines=$((log_nonblank_lines + pod_nonblank_lines)) +done <<<"$pods" +rm -f "$log_err" +if (( log_nonblank_lines == 0 )); then + { + echo "adapter log scan: FAILED" + echo "No adapter logs were returned from pods for deployment/${runtime}; evidence is indeterminate." + } >"$log_scan" + cat "$log_scan" >&2 + exit 1 +fi +{ + echo "adapter log scan: passed" + echo "scanned pods: ${log_pods}" + echo "scanned lines: ${log_lines}" + echo "scanned nonblank lines: ${log_nonblank_lines}" +} >"$log_scan" + +{ + echo + echo "## Evidence files" + echo + echo "- AgentRuntime JSON: $(basename "$agentruntime_json")" + echo "- Task events JSON: $(basename "$events_json")" + echo "- Task approvals JSON: $(basename "$approvals_json")" + echo "- Fibey verifier output: $(basename "$verifier_out")" + echo "- Adapter log scan summary: $(basename "$log_scan")" + echo + echo "## Verifier output" + echo + sed 's/^/> /' "$verifier_out" + echo + echo "## Adapter log scan" + echo + sed 's/^/> /' "$log_scan" +} >>"$report" + +echo "Foundry Responses live evidence captured in: $out_dir" >&2 diff --git a/examples/harness/foundry-responses/live-smoke.sh b/examples/harness/foundry-responses/live-smoke.sh new file mode 100755 index 000000000..f9c501ceb --- /dev/null +++ b/examples/harness/foundry-responses/live-smoke.sh @@ -0,0 +1,434 @@ +#!/usr/bin/env bash +set -euo pipefail + +usage() { + cat >&2 <<'USAGE' +Usage: examples/harness/foundry-responses/live-smoke.sh [--apply] [--wait] [--namespace NAME] + +Credentials-safe preflight/deploy helper for the live Foundry hosted AgentKit +Responses smoke gate. By default this performs preflight only. With --apply it +creates/updates the namespace-local adapter Deployment, Service, Secrets, and +AgentRuntime facade, then optionally waits for readiness with --wait. + +Required environment for preflight/apply: + ORKA_FOUNDRY_RESPONSES_ENDPOINT + Full hosted Responses URL, OR set both: + ORKA_FOUNDRY_RESPONSES_PROJECT_ENDPOINT + ORKA_FOUNDRY_RESPONSES_AGENT_NAME + Exactly one Foundry auth value: + ORKA_FOUNDRY_RESPONSES_API_KEY + ORKA_FOUNDRY_RESPONSES_AUTH_BEARER + +Optional environment: + ORKA_FOUNDRY_RESPONSES_NAMESPACE default: foundry-responses-smoke + ORKA_FOUNDRY_RESPONSES_RUNTIME_NAME default: sample-foundry-responses-runtime + ORKA_FOUNDRY_RESPONSES_ADAPTER_IMAGE required with --apply; build/publish this PR's Dockerfile first + ORKA_FOUNDRY_RESPONSES_ADAPTER_BEARER_TOKEN generated if absent for this run + ORKA_FOUNDRY_RESPONSES_API_VERSION default: v1 + ORKA_FOUNDRY_RESPONSES_BROKERED_TOOL_CLASSES default: read + Set explicitly to an empty string for observed-only mode. Every advertised + class requires the hosted agent to statically expose the matching probe-only + conformance_read/conformance_write schema. + ORKA_FOUNDRY_RESPONSES_BROKERED_CONTINUATION_PROOF required when brokered classes are enabled + +The script never prints secret values. Do not run with shell tracing (set -x). +USAGE +} + +apply=0 +wait_ready=0 +namespace="${ORKA_FOUNDRY_RESPONSES_NAMESPACE:-foundry-responses-smoke}" + +while [[ $# -gt 0 ]]; do + case "$1" in + --apply) + apply=1 + shift + ;; + --wait) + wait_ready=1 + shift + ;; + --namespace) + [[ $# -ge 2 ]] || { echo "--namespace requires a value" >&2; exit 2; } + namespace="$2" + shift 2 + ;; + -h|--help) + usage + exit 0 + ;; + *) + echo "unknown argument: $1" >&2 + usage + exit 2 + ;; + esac +done + +if [[ "$wait_ready" == "1" && "$apply" != "1" ]]; then + echo "error: --wait requires --apply" >&2 + exit 2 +fi + +runtime_name="${ORKA_FOUNDRY_RESPONSES_RUNTIME_NAME:-sample-foundry-responses-runtime}" +service_url="http://${runtime_name}.${namespace}.svc.cluster.local:8080" +adapter_image="${ORKA_FOUNDRY_RESPONSES_ADAPTER_IMAGE:-}" +api_version="${ORKA_FOUNDRY_RESPONSES_API_VERSION:-v1}" +brokered_classes="${ORKA_FOUNDRY_RESPONSES_BROKERED_TOOL_CLASSES-read}" +endpoint="${ORKA_FOUNDRY_RESPONSES_ENDPOINT:-}" +project_endpoint="${ORKA_FOUNDRY_RESPONSES_PROJECT_ENDPOINT:-}" +agent_name="${ORKA_FOUNDRY_RESPONSES_AGENT_NAME:-}" +api_key="${ORKA_FOUNDRY_RESPONSES_API_KEY:-}" +auth_bearer="${ORKA_FOUNDRY_RESPONSES_AUTH_BEARER:-}" +adapter_bearer="${ORKA_FOUNDRY_RESPONSES_ADAPTER_BEARER_TOKEN:-}" +continuation_proof="${ORKA_FOUNDRY_RESPONSES_BROKERED_CONTINUATION_PROOF:-}" +rollout_nonce="$(date -u +%Y%m%dT%H%M%SZ)" + +fail() { + echo "error: $*" >&2 + exit 2 +} + +require_cmd() { + command -v "$1" >/dev/null 2>&1 || fail "$1 is required" +} + +authority_has_hostname() { + local authority="$1" + if [[ "$authority" == \[* ]]; then + [[ "$authority" =~ ^\[[^]]+\](:[0-9]+)?$ ]] + return + fi + [[ -n "${authority%%:*}" ]] +} + +is_https_or_loopback_http() { + local value="$1" + local rest authority + [[ -z "$value" ]] && return 0 + [[ "$value" != *$'\n'* && "$value" != *$'\r'* && "$value" != *$'\t'* ]] || return 1 + if [[ "$value" == https://* ]]; then + rest="${value#https://}" + authority="${rest%%/*}" + authority="${authority%%\?*}" + authority="${authority%%#*}" + [[ -n "$authority" && "$authority" != *@* && "$authority" != *[[:space:]]* ]] || return 1 + authority_has_hostname "$authority" + return + fi + [[ "$value" == http://* ]] || return 1 + rest="${value#http://}" + authority="${rest%%/*}" + authority="${authority%%\?*}" + authority="${authority%%#*}" + [[ -n "$authority" && "$authority" != *@* ]] || return 1 + if [[ "$authority" == \[* ]]; then + [[ "$authority" =~ ^\[::1\](:[0-9]+)?$ ]] + else + [[ "$authority" =~ ^(localhost|127\.0\.0\.1)(:[0-9]+)?$ ]] + fi +} + +require_nonempty_name() { + local label="$1" + local value="$2" + [[ "$value" =~ ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ ]] || \ + fail "$label must be a Kubernetes DNS label, got '$value'" +} + +url_authority() { + local value="$1" + local rest="${value#*://}" + rest="${rest%%#*}" + rest="${rest%%\?*}" + printf '%s' "${rest%%/*}" +} + +responses_endpoint_is_safe() { + local value="$1" + local rest authority after path query pair key val + [[ -z "$value" ]] && return 0 + is_https_or_loopback_http "$value" || return 1 + [[ "$value" != *#* ]] || return 1 + authority="$(url_authority "$value")" + [[ -n "$authority" && "$authority" != *@* ]] || return 1 + rest="${value#*://}" + after="${rest#"$authority"}" + path="${after%%\?*}" + [[ "${path%/}" == */responses ]] || return 1 + if [[ "$after" == *\?* ]]; then + query="${after#*\?}" + [[ -n "$query" ]] || return 1 + IFS='&' read -r -a pairs <<<"$query" + for pair in "${pairs[@]}"; do + [[ "$pair" == *=* ]] || return 1 + key="${pair%%=*}" + val="${pair#*=}" + [[ "$key" == "api-version" && -n "$val" ]] || return 1 + done + fi +} + +project_endpoint_is_safe() { + local value="$1" + local authority + [[ -z "$value" ]] && return 0 + is_https_or_loopback_http "$value" || return 1 + [[ "$value" != *#* && "$value" != *\?* ]] || return 1 + authority="$(url_authority "$value")" + [[ -n "$authority" && "$authority" != *@* ]] +} + +indent_block() { + sed 's/^/ /' +} + +preflight() { + require_nonempty_name namespace "$namespace" + require_nonempty_name "runtime name" "$runtime_name" + + if [[ -z "$endpoint" ]]; then + [[ -n "$project_endpoint" && -n "$agent_name" ]] || \ + fail "set ORKA_FOUNDRY_RESPONSES_ENDPOINT or PROJECT_ENDPOINT plus AGENT_NAME" + fi + responses_endpoint_is_safe "$endpoint" || fail "ORKA_FOUNDRY_RESPONSES_ENDPOINT must be a safe /responses URL (https, or loopback http for tests; only api-version query allowed)" + project_endpoint_is_safe "$project_endpoint" || fail "ORKA_FOUNDRY_RESPONSES_PROJECT_ENDPOINT must be https (or loopback http for tests) without userinfo, query, or fragment" + + if [[ -n "$api_key" && -n "$auth_bearer" ]]; then + fail "set exactly one of ORKA_FOUNDRY_RESPONSES_API_KEY or ORKA_FOUNDRY_RESPONSES_AUTH_BEARER" + fi + if [[ -z "$api_key" && -z "$auth_bearer" ]]; then + fail "set one Foundry auth value: ORKA_FOUNDRY_RESPONSES_API_KEY or ORKA_FOUNDRY_RESPONSES_AUTH_BEARER" + fi + + if [[ -n "${brokered_classes//[[:space:],]/}" ]]; then + IFS=',' read -r -a classes <<<"$brokered_classes" + for class in "${classes[@]}"; do + class="$(printf '%s' "$class" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')" + [[ -z "$class" ]] && continue + [[ "$class" == "read" || "$class" == "write" ]] || fail "unsupported brokered class '$class' (expected read/write)" + done + fi + if [[ -n "${brokered_classes//[[:space:],]/}" && -z "${continuation_proof//[[:space:]]/}" ]]; then + fail "ORKA_FOUNDRY_RESPONSES_BROKERED_CONTINUATION_PROOF is required when brokered classes are enabled" + fi + + if [[ "$apply" == "1" && -z "$adapter_image" ]]; then + fail "ORKA_FOUNDRY_RESPONSES_ADAPTER_IMAGE is required with --apply" + fi + if [[ "$apply" == "1" || "$wait_ready" == "1" ]]; then + require_cmd kubectl + fi +} + +random_bearer() { + if command -v openssl >/dev/null 2>&1; then + openssl rand -hex 32 + else + od -An -N32 -tx1 /dev/urandom | tr -d ' \n' + printf '\n' + fi +} + + +decode_base64() { + if printf '' | base64 --decode >/dev/null 2>&1; then + base64 --decode + else + base64 -D + fi +} + +encode_base64() { + printf '%s' "$1" | base64 | tr -d '[:space:]' +} + +existing_adapter_bearer() { + local encoded + encoded="$(kubectl -n "$namespace" get secret "${runtime_name}-token" -o jsonpath='{.data.harness-bearer}' 2>/dev/null || true)" + [[ -n "$encoded" ]] || return 1 + printf '%s' "$encoded" | decode_base64 +} + +emit_secret_yaml() { + local foundry_key_name="foundry-auth" + local foundry_value="$api_key" + if [[ -n "$auth_bearer" ]]; then + foundry_key_name="foundry-bearer" + foundry_value="$auth_bearer" + fi + + cat <&2 + exit 0 +fi + +kubectl get namespace "$namespace" >/dev/null 2>&1 || kubectl create namespace "$namespace" >/dev/null +# Validate CRDs/admission for non-secret resources before writing live credentials. +emit_runtime_yaml | kubectl -n "$namespace" apply --server-side --field-manager=orka-foundry-responses-live-smoke --dry-run=server -f - >/dev/null +emit_secret_yaml | kubectl -n "$namespace" apply --server-side --field-manager=orka-foundry-responses-live-smoke -f - >/dev/null +kubectl -n "$namespace" annotate secret "${runtime_name}-token" kubectl.kubernetes.io/last-applied-configuration- --overwrite >/dev/null 2>&1 || true +kubectl -n "$namespace" annotate secret "${runtime_name}-adapter-config" kubectl.kubernetes.io/last-applied-configuration- --overwrite >/dev/null 2>&1 || true +emit_runtime_yaml | kubectl -n "$namespace" apply --server-side --field-manager=orka-foundry-responses-live-smoke -f - +if kubectl -n "$namespace" get "deployment/${runtime_name}" >/dev/null 2>&1; then + kubectl -n "$namespace" rollout restart "deployment/${runtime_name}" >/dev/null +fi + +echo "Applied Foundry Responses adapter smoke resources in namespace '$namespace'." >&2 + +if [[ "$wait_ready" == "1" ]]; then + kubectl -n "$namespace" rollout status "deployment/${runtime_name}" --timeout=120s + kubectl -n "$namespace" wait --for=condition=Ready "agentruntime/${runtime_name}" --timeout=120s +fi diff --git a/examples/harness/foundry-responses/main.go b/examples/harness/foundry-responses/main.go new file mode 100644 index 000000000..3898e4b6e --- /dev/null +++ b/examples/harness/foundry-responses/main.go @@ -0,0 +1,1925 @@ +package main + +import ( + "bytes" + "context" + "crypto/sha256" + "crypto/subtle" + "encoding/json" + "errors" + "fmt" + "io" + "log" + "maps" + "net/http" + "net/url" + "os" + "reflect" + "slices" + "sort" + "strconv" + "strings" + "sync" + "time" + + "github.com/orka-agents/orka/internal/harness" +) + +const ( + defaultAddr = ":8090" + defaultAPIVersion = "v1" + defaultRequestTimeout = 20 * time.Second + defaultStateRetention = 10 * time.Minute + defaultReadHeaderTimeout = 5 * time.Second + defaultReadTimeout = 30 * time.Second + defaultIdleTimeout = 60 * time.Second + maxFoundryOutputBytes = 1 << 20 + maxFoundryBodyBytes = 4 << 20 + maxConsumedTurnIDs = 1024 + maxQuarantinedRuntimeSessions = 1024 + maxBrokeredToolCalls = 32 + readinessPath = "/v1/ready" + foundryInitialUnknown = "foundry_initial_unknown" + + envAddr = "ORKA_FOUNDRY_RESPONSES_ADAPTER_ADDR" + envRuntimeName = "ORKA_FOUNDRY_RESPONSES_RUNTIME_NAME" + envAdapterBearer = "ORKA_FOUNDRY_RESPONSES_ADAPTER_BEARER_" + "TOKEN" + envEndpoint = "ORKA_FOUNDRY_RESPONSES_ENDPOINT" + envProjectEndpoint = "ORKA_FOUNDRY_RESPONSES_PROJECT_ENDPOINT" + envAgentName = "ORKA_FOUNDRY_RESPONSES_AGENT_NAME" + envAuthBearer = "ORKA_FOUNDRY_RESPONSES_AUTH_BEARER" + envFoundryAuth = "ORKA_FOUNDRY_RESPONSES_API_" + "KEY" + envAPIVersion = "ORKA_FOUNDRY_RESPONSES_API_VERSION" + envRequestTimeout = "ORKA_FOUNDRY_RESPONSES_POLL_TIMEOUT" + envStateRetention = "ORKA_FOUNDRY_RESPONSES_STATE_RETENTION" + envContinuationProof = "ORKA_FOUNDRY_RESPONSES_BROKERED_CONTINUATION_PROOF" + envBrokeredToolClasses = "ORKA_FOUNDRY_RESPONSES_BROKERED_TOOL_CLASSES" + envAudience = "ORKA_FOUNDRY_RESPONSES_TOKEN_AUDIENCE" +) + +type config struct { + addr string + runtimeName string + adapterBearer string + endpoint string + projectEndpoint string + agentName string + authBearer string + foundryAuth string + apiVersion string + requestTimeout time.Duration + stateRetention time.Duration + continuationProof string + brokeredToolClasses []harness.BrokeredToolClass + configError string +} + +type server struct { + cfg config + client *http.Client + + mu sync.Mutex + turns map[harness.HarnessTurnID]*turnState + consumedTurns map[harness.HarnessTurnID]struct{} + consumedOrder []harness.HarnessTurnID + runtimeSessions map[harness.RuntimeSessionID]foundrySession + quarantinedSessions map[harness.RuntimeSessionID]struct{} + quarantineSaturated bool +} + +type toolResultDigest [sha256.Size]byte + +type responseDisposition int + +const ( + responseRejected responseDisposition = iota + responsePending + responseCompleted +) + +type foundrySession struct { + ID string + LastSeen time.Time +} + +type turnState struct { + request harness.StartTurnRequest + initializing bool + responseID string + foundrySessionID string + pendingTools map[string]string + suppressedToolCalls map[string]struct{} + requestedToolCalls int + bufferedResults map[string]harness.ToolCallResult + bufferedDigests map[string]toolResultDigest + submittedDigests map[string]toolResultDigest + frames []harness.HarnessEventFrame + completed bool + continuationInFlight bool + hostedRequestSequence uint64 + activeHostedRequestID uint64 + activeHostedRequestCancel context.CancelFunc + frameUpdates chan struct{} + continueMu sync.Mutex +} + +type responsesRequest struct { + Input any `json:"input"` + PreviousResponseID string `json:"previous_response_id,omitempty"` + AgentSessionID string `json:"agent_session_id,omitempty"` + // BrokeredContinuationProof carries the continuation proof in the request + // BODY in addition to the X-AgentKit-Brokered-Continuation-Proof header. + // Some hosted-agent gateways (e.g. Microsoft Foundry) strip custom request + // headers before forwarding to the container, which would reject the + // function_call_output continuation; the body survives, so a + // gateway-tolerant runtime can recover the proof from here. Only set on + // continuation requests (PreviousResponseID present). + BrokeredContinuationProof string `json:"brokered_continuation_proof,omitempty"` +} + +type responsesFunctionCallOutput struct { + Type string `json:"type"` + CallID string `json:"call_id"` + Output string `json:"output"` + Status string `json:"status,omitempty"` +} + +type responsesResponse struct { + ID string `json:"id"` + AgentSessionID string `json:"agent_session_id,omitempty"` + Status string `json:"status,omitempty"` + Output []responsesOutput `json:"output,omitempty"` + Error *responsesError `json:"error,omitempty"` +} + +type responsesOutput struct { + Type string `json:"type"` + Role string `json:"role,omitempty"` + Name string `json:"name,omitempty"` + CallID string `json:"call_id,omitempty"` + Arguments json.RawMessage `json:"arguments,omitempty"` + Content any `json:"content,omitempty"` + Text string `json:"text,omitempty"` + Status string `json:"status,omitempty"` +} + +type responsesError struct { + Code string `json:"code,omitempty"` + Message string `json:"message,omitempty"` +} + +type pendingFunctionCall struct { + callID string + name string + args json.RawMessage +} + +var sseSizeProbeTime = time.Date(2000, time.January, 1, 0, 0, 0, 123456789, time.UTC) + +const responsesEndpointRequirement = "foundry hosted Responses endpoint must use https " + + "(http allowed only for loopback), target /responses, and must not include credentials, " + + "fragments, or query parameters other than api-version" + +func main() { + cfg := loadConfig() + s := newServer(cfg, &http.Client{Timeout: cfg.requestTimeout}) + logEndpoint, err := s.responsesEndpoint() + if err != nil { + logEndpoint = cfg.endpoint + } + log.Printf( + "Foundry hosted Responses AgentRuntime adapter listening on %s runtime=%s endpoint=%s", + cfg.addr, + cfg.runtimeName, + sanitizeEndpoint(logEndpoint), + ) + if err := newAdapterHTTPServer(cfg.addr, s.handler()).ListenAndServe(); err != nil { + log.Fatal(err) + } +} + +func newAdapterHTTPServer(addr string, handler http.Handler) *http.Server { + return &http.Server{ + Addr: addr, + Handler: handler, + ReadHeaderTimeout: defaultReadHeaderTimeout, + ReadTimeout: defaultReadTimeout, + IdleTimeout: defaultIdleTimeout, + } +} + +func loadConfig() config { + classes, classErr := parseBrokeredToolClasses(os.Getenv(envBrokeredToolClasses)) + cfg := config{ + addr: firstNonBlank(os.Getenv(envAddr), defaultAddr), + runtimeName: firstNonBlank(os.Getenv(envRuntimeName), "foundry-agentkit-responses"), + adapterBearer: strings.TrimSpace(os.Getenv(envAdapterBearer)), + endpoint: strings.TrimSpace(os.Getenv(envEndpoint)), + projectEndpoint: strings.TrimRight(strings.TrimSpace(os.Getenv(envProjectEndpoint)), "/"), + agentName: strings.TrimSpace(os.Getenv(envAgentName)), + authBearer: strings.TrimSpace(os.Getenv(envAuthBearer)), + foundryAuth: strings.TrimSpace(os.Getenv(envFoundryAuth)), + apiVersion: firstNonBlank(os.Getenv(envAPIVersion), defaultAPIVersion), + requestTimeout: parseDurationEnv(envRequestTimeout, defaultRequestTimeout), + stateRetention: parseDurationEnv(envStateRetention, defaultStateRetention), + continuationProof: strings.TrimSpace(os.Getenv(envContinuationProof)), + brokeredToolClasses: classes, + } + _ = os.Getenv(envAudience) // Reserved for a future workload-identity token provider; never logged. + if classErr != nil { + cfg.configError = classErr.Error() + } + return cfg +} + +func (c config) validationError() error { + if c.configError != "" { + return errors.New(c.configError) + } + if len(c.brokeredToolClasses) > 0 && strings.TrimSpace(c.continuationProof) == "" { + return fmt.Errorf("%s is required when brokered tool classes are enabled", envContinuationProof) + } + return nil +} + +func newServer(cfg config, client *http.Client) *server { + if client == nil { + client = &http.Client{Timeout: cfg.requestTimeout} + } + clientCopy := *client + if clientCopy.CheckRedirect == nil { + clientCopy.CheckRedirect = func(*http.Request, []*http.Request) error { + return http.ErrUseLastResponse + } + } + return &server{ + cfg: cfg, + client: &clientCopy, + turns: map[harness.HarnessTurnID]*turnState{}, + consumedTurns: map[harness.HarnessTurnID]struct{}{}, + runtimeSessions: map[harness.RuntimeSessionID]foundrySession{}, + quarantinedSessions: map[harness.RuntimeSessionID]struct{}{}, + } +} + +func (s *server) handler() http.Handler { + mux := http.NewServeMux() + mux.HandleFunc(harness.HealthPath, s.health) + mux.HandleFunc(readinessPath, s.ready) + mux.HandleFunc(harness.CapabilitiesPath, s.capabilities) + mux.HandleFunc(harness.TurnsPath, s.startTurn) + mux.HandleFunc(harness.TurnsPath+"/", s.turn) + return mux +} + +func (s *server) health(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodGet { + harness.WriteError(w, http.StatusMethodNotAllowed, "method not allowed") + return + } + _, endpointErr := s.responsesEndpoint() + configErr := s.cfg.validationError() + s.mu.Lock() + quarantineSaturated := s.quarantineSaturated + s.mu.Unlock() + ready := configErr == nil && s.cfg.adapterBearer != "" && endpointErr == nil && + exactlyOneFoundryAuth(s.cfg) && !quarantineSaturated + status := harness.HealthStatusOK + msg := "ready" + if !ready { + status = harness.HealthStatusDegraded + parts := []string{ + "adapter bearer, safe Foundry hosted Responses endpoint, and exactly one Foundry auth mode are required", + } + if configErr != nil { + parts = append(parts, configErr.Error()) + } + if endpointErr != nil { + parts = append(parts, endpointErr.Error()) + } + if quarantineSaturated { + parts = append(parts, "runtime session quarantine capacity exhausted; restart required") + } + msg = strings.Join(parts, "; ") + } + harness.WriteJSON(w, http.StatusOK, harness.HealthResponse{ + Version: harness.ProtocolVersion, + Status: status, + Ready: ready, + CheckedAt: time.Now().UTC(), + Message: msg, + Metadata: map[string]string{"backend": "foundry-responses"}, + }) +} + +func (s *server) ready(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodGet { + harness.WriteError(w, http.StatusMethodNotAllowed, "method not allowed") + return + } + _, endpointErr := s.responsesEndpoint() + s.mu.Lock() + quarantineSaturated := s.quarantineSaturated + s.mu.Unlock() + ready := s.cfg.validationError() == nil && s.cfg.adapterBearer != "" && + endpointErr == nil && exactlyOneFoundryAuth(s.cfg) && !quarantineSaturated + if !ready { + harness.WriteError(w, http.StatusServiceUnavailable, "adapter is not ready") + return + } + harness.WriteJSON(w, http.StatusOK, map[string]string{"status": "ready"}) +} + +func (s *server) capabilities(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodGet { + harness.WriteError(w, http.StatusMethodNotAllowed, "method not allowed") + return + } + modes := []harness.ToolExecutionMode{harness.ToolExecutionModeObserved} + maxTurnSeconds := int(s.cfg.requestTimeout.Seconds()) + brokeredEnabled := len(s.cfg.brokeredToolClasses) > 0 && s.cfg.validationError() == nil + if brokeredEnabled { + modes = append(modes, harness.ToolExecutionModeBrokered) + // A brokered turn can contain multiple hosted request/tool-result rounds, + // each with its own request timeout and approval wait. The harness contract + // has only one runtime-wide ceiling, so advertise unknown rather than an + // understated duration when brokered mode is available. + maxTurnSeconds = 0 + } + advertisedClasses := []harness.BrokeredToolClass(nil) + if brokeredEnabled { + advertisedClasses = append(advertisedClasses, s.cfg.brokeredToolClasses...) + } + harness.WriteJSON(w, http.StatusOK, harness.CapabilitiesResponse{ + Version: harness.ProtocolVersion, + ProtocolVersion: harness.ProtocolVersion, + Transport: harness.HTTPTransport, + RuntimeName: s.cfg.runtimeName, + RuntimeVersion: "foundry-responses-adapter", + ProviderKind: harness.ProviderKindRemote, + ToolExecutionModes: modes, + BrokeredToolClasses: advertisedClasses, + SupportsCancel: true, + SupportsRuntimeSessions: true, + SupportsContinuation: brokeredEnabled, + SupportsArtifacts: false, + MaxConcurrentTurns: 1, + MaxTurnSeconds: maxTurnSeconds, + MaxOutputBytes: maxFoundryOutputBytes, + Metadata: map[string]string{"backend": "foundry-responses"}, + }) +} + +func (s *server) startTurn(w http.ResponseWriter, r *http.Request) { + if !s.authorized(w, r) { + return + } + if r.Method != http.MethodPost { + harness.WriteError(w, http.StatusMethodNotAllowed, "method not allowed") + return + } + var req harness.StartTurnRequest + if err := json.NewDecoder(r.Body).Decode(&req); err != nil { + harness.WriteError(w, http.StatusBadRequest, "invalid JSON request") + return + } + if err := req.Validate(); err != nil { + harness.WriteError(w, http.StatusBadRequest, err.Error()) + return + } + if err := s.validateStartRequest(req); err != nil { + harness.WriteError(w, http.StatusBadRequest, err.Error()) + return + } + // Resolved turn environment values can contain credentials. This adapter does + // not consume them, so discard them before duplicate comparison or retention. + req.Input.Env = nil + eventsPath, err := harness.EventStreamPath(req.TurnID) + if err != nil { + harness.WriteError(w, http.StatusBadRequest, err.Error()) + return + } + s.mu.Lock() + if existing := s.turns[req.TurnID]; existing != nil { + response := startTurnResponse(existing.request, eventsPath) + if !sameStartTurnRequest(existing.request, req) { + s.mu.Unlock() + harness.WriteError(w, http.StatusConflict, "turn already exists") + return + } + if existing.initializing { + s.mu.Unlock() + harness.WriteError(w, http.StatusConflict, "turn initialization in progress") + return + } + s.mu.Unlock() + harness.WriteJSON(w, http.StatusAccepted, response) + return + } + if _, consumed := s.consumedTurns[req.TurnID]; consumed { + s.mu.Unlock() + harness.WriteError(w, http.StatusConflict, "turn already completed") + return + } + if s.quarantineSaturated { + s.mu.Unlock() + harness.WriteError( + w, + http.StatusServiceUnavailable, + "adapter unavailable after runtime session quarantine capacity exhausted", + ) + return + } + if _, quarantined := s.quarantinedSessions[req.RuntimeSessionID]; quarantined { + s.mu.Unlock() + harness.WriteError( + w, + http.StatusGone, + "runtime session unavailable after unconfirmed hosted cancellation", + ) + return + } + if s.activeTurnCountLocked() >= 1 { + s.mu.Unlock() + harness.WriteError(w, http.StatusConflict, "maximum concurrent turns reached") + return + } + turn := &turnState{ + request: req, + initializing: true, + pendingTools: map[string]string{}, + suppressedToolCalls: map[string]struct{}{}, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{}, + frameUpdates: make(chan struct{}), + } + s.appendFrameLocked(turn, harness.FrameTurnStarted, "foundry hosted response started") + s.turns[req.TurnID] = turn + s.mu.Unlock() + + ctx, cancel := s.foundryRequestContext(r.Context(), req.Deadline) + s.mu.Lock() + if turn.completed { + turn.initializing = false + s.mu.Unlock() + cancel() + harness.WriteJSON(w, http.StatusAccepted, startTurnResponse(req, eventsPath)) + return + } + existingFoundrySessionID := s.runtimeSessions[req.RuntimeSessionID].ID + hostedRequestID := s.activateHostedRequestLocked(turn, cancel) + s.mu.Unlock() + defer cancel() + var response responsesResponse + initialRequest := responsesRequest{ + Input: req.Input.Prompt, + AgentSessionID: existingFoundrySessionID, + } + foundrySessionID, err := s.postResponses(ctx, initialRequest, &response) + s.mu.Lock() + s.clearHostedRequestLocked(turn, hostedRequestID) + if err != nil { + turn.initializing = false + s.quarantineRuntimeSessionLocked(turn) + completed := turn.completed + if !completed { + s.appendFailedLocked( + turn, + foundryInitialUnknown, + "initial hosted request failed after submission was attempted; "+ + "failing closed to avoid a duplicate hosted turn", + ) + } + s.mu.Unlock() + if completed { + harness.WriteJSON(w, http.StatusAccepted, startTurnResponse(req, eventsPath)) + return + } + log.Printf( + "Foundry hosted Responses initial request failed after submission for turn %q (error type %T)", + req.TurnID, + err, + ) + // Accept the turn so Orka consumes and persists the terminal failure frame. + // Returning a control-plane error here would cause the caller to retry a + // submission whose outcome is unknown at Foundry. + harness.WriteJSON(w, http.StatusAccepted, startTurnResponse(req, eventsPath)) + return + } + if !turn.completed { + disposition := s.handleResponsesResponseLocked(turn, response) + if disposition == responseRejected { + if _, reused := s.runtimeSessions[turn.request.RuntimeSessionID]; reused { + s.quarantineRuntimeSessionLocked(turn) + } + } else { + s.setTurnSessionLocked(turn, foundrySessionID) + } + if disposition == responseCompleted { + s.publishRuntimeSessionLocked(turn) + } + } + turn.initializing = false + s.mu.Unlock() + harness.WriteJSON(w, http.StatusAccepted, startTurnResponse(req, eventsPath)) +} + +func (s *server) turn(w http.ResponseWriter, r *http.Request) { + if !s.authorized(w, r) { + return + } + turnID, resource, err := harness.ParseTurnResourcePath(r.URL.EscapedPath()) + if err != nil { + harness.WriteError(w, http.StatusNotFound, "not found") + return + } + s.mu.Lock() + turn := s.turns[turnID] + _, consumed := s.consumedTurns[turnID] + s.mu.Unlock() + if turn == nil { + message := "turn state unavailable after runtime restart" + if consumed { + message = "terminal turn expired from runtime retention" + } + harness.WriteError(w, http.StatusGone, message) + return + } + switch resource { + case harness.TurnResourceEvents: + if r.Method != http.MethodGet { + harness.WriteError(w, http.StatusMethodNotAllowed, "method not allowed") + return + } + s.streamEvents(w, r, turn) + case harness.TurnResourceContinue: + if r.Method != http.MethodPost { + harness.WriteError(w, http.StatusMethodNotAllowed, "method not allowed") + return + } + s.continueTurn(w, r, turn) + case harness.TurnResourceCancel: + s.cancelTurn(w, r, turn) + default: + harness.WriteError(w, http.StatusNotFound, "not found") + } +} + +func (s *server) streamEvents(w http.ResponseWriter, r *http.Request, turn *turnState) { + afterSeq := parseAfterSeq(r.URL.Query().Get("afterSeq")) + w.Header().Set("Content-Type", "text/event-stream") + w.WriteHeader(http.StatusOK) + if flusher, ok := w.(http.Flusher); ok { + flusher.Flush() + } + nextSeq := afterSeq + for { + s.mu.Lock() + frames := append([]harness.HarnessEventFrame(nil), turn.frames...) + completed := turn.completed + updates := turn.frameUpdates + if updates == nil { + updates = make(chan struct{}) + turn.frameUpdates = updates + } + s.mu.Unlock() + for _, frame := range frames { + if frame.Seq <= nextSeq { + continue + } + if frame.Type == harness.FrameToolCallRequested { + s.mu.Lock() + _, suppressed := turn.suppressedToolCalls[frame.ToolCallID] + terminal := turn.completed + s.mu.Unlock() + if suppressed || terminal { + continue + } + } + if err := harness.WriteSSEFrame(w, frame); err != nil { + return + } + nextSeq = frame.Seq + if frame.Type == harness.FrameToolCallRequested { + // Pace brokered calls one at a time. A partial /continue result, + // cancellation, or terminal transition wakes the stream before the + // next request is considered, so suppression is observed live. + break + } + } + if completed { + _ = harness.WriteSSEDone(w) + return + } + select { + case <-r.Context().Done(): + return + case <-updates: + } + } +} + +func (s *server) continueTurn(w http.ResponseWriter, r *http.Request, turn *turnState) { + var req harness.ContinueTurnRequest + if err := json.NewDecoder(r.Body).Decode(&req); err != nil { + harness.WriteError(w, http.StatusBadRequest, "invalid JSON request") + return + } + if err := req.Validate(); err != nil { + harness.WriteError(w, http.StatusBadRequest, err.Error()) + return + } + if !sameContinueIdentity(turn.request, req) { + harness.WriteError(w, http.StatusBadRequest, "continue request does not match started turn") + return + } + turn.continueMu.Lock() + defer turn.continueMu.Unlock() + + if err := s.ensureTerminalContinueIsDuplicate(turn, req.ToolResults); err != nil { + harness.WriteError(w, http.StatusConflict, err.Error()) + return + } + s.mu.Lock() + completed := turn.completed + s.mu.Unlock() + if completed { + harness.WriteJSON( + w, + http.StatusAccepted, + continueResponse(req, "duplicate continue accepted for terminal turn"), + ) + return + } + s.mu.Lock() + previousResponseID := turn.responseID + foundrySessionID := turn.foundrySessionID + s.mu.Unlock() + if strings.TrimSpace(previousResponseID) == "" { + harness.WriteError(w, http.StatusConflict, "cannot continue before Foundry response id is known") + return + } + + resultsToSubmit, err := s.recordContinueResults(turn, req.ToolResults) + if err != nil { + harness.WriteError(w, http.StatusConflict, err.Error()) + return + } + if len(resultsToSubmit) == 0 { + harness.WriteJSON(w, http.StatusAccepted, continueResponse(req, "continue accepted")) + return + } + outputs, err := functionCallOutputs(resultsToSubmit) + if err != nil { + harness.WriteError(w, http.StatusBadRequest, err.Error()) + return + } + ctx, cancel := s.foundryRequestContext(r.Context(), turn.request.Deadline) + s.mu.Lock() + if turn.completed { + s.mu.Unlock() + cancel() + harness.WriteError(w, http.StatusConflict, "turn completed before hosted continuation submission") + return + } + turn.continuationInFlight = true + hostedRequestID := s.activateHostedRequestLocked(turn, cancel) + s.mu.Unlock() + defer func() { + cancel() + s.mu.Lock() + turn.continuationInFlight = false + s.mu.Unlock() + }() + var response responsesResponse + continuation := responsesRequest{ + PreviousResponseID: previousResponseID, + AgentSessionID: foundrySessionID, + Input: outputs, + } + updatedSessionID, err := s.postResponses(ctx, continuation, &response) + s.mu.Lock() + s.clearHostedRequestLocked(turn, hostedRequestID) + if err != nil { + s.quarantineRuntimeSessionLocked(turn) + completed := turn.completed + if !completed { + s.appendFailedLocked( + turn, + "foundry_continuation_unknown", + "hosted continuation failed after submission was attempted; "+ + "failing closed to avoid duplicate continuation", + ) + } + s.mu.Unlock() + if completed { + harness.WriteError(w, http.StatusConflict, "turn completed while hosted continuation was in flight") + return + } + log.Printf( + "Foundry hosted Responses continuation failed after submission for turn %q (error type %T)", + req.TurnID, + err, + ) + harness.WriteError(w, http.StatusBadGateway, "hosted continuation failed after submission was attempted") + return + } + if turn.completed { + s.mu.Unlock() + harness.WriteError(w, http.StatusConflict, "turn completed while hosted continuation was in flight") + return + } + for _, result := range resultsToSubmit { + toolName := firstNonBlank(turn.pendingTools[result.ToolCallID], result.ToolCallID) + frame := s.newToolResultFrame(turn, int64(len(turn.frames)+1), toolName, result) + if !harnessFrameFitsSSE(frame) { + s.appendFailedLocked( + turn, + "brokered_tool_result_frame_too_large", + "brokered tool result exceeded the harness SSE frame limit", + ) + s.mu.Unlock() + harness.WriteError(w, http.StatusBadGateway, "brokered tool result exceeds harness SSE frame limit") + return + } + s.appendPreparedFrameLocked(turn, frame) + delete(turn.pendingTools, result.ToolCallID) + delete(turn.bufferedResults, result.ToolCallID) + delete(turn.bufferedDigests, result.ToolCallID) + } + disposition := s.handleResponsesResponseLocked(turn, response) + if disposition == responseRejected { + if _, reused := s.runtimeSessions[turn.request.RuntimeSessionID]; reused { + s.quarantineRuntimeSessionLocked(turn) + } + } else { + s.setTurnSessionLocked(turn, updatedSessionID) + } + if disposition == responseCompleted { + s.publishRuntimeSessionLocked(turn) + } + s.mu.Unlock() + harness.WriteJSON(w, http.StatusAccepted, continueResponse(req, "continue accepted")) +} + +func (s *server) cancelTurn(w http.ResponseWriter, r *http.Request, turn *turnState) { + if r.Method != http.MethodPost { + harness.WriteError(w, http.StatusMethodNotAllowed, "method not allowed") + return + } + var req harness.CancelTurnRequest + if err := json.NewDecoder(r.Body).Decode(&req); err != nil { + harness.WriteError(w, http.StatusBadRequest, "invalid JSON request") + return + } + if err := req.Validate(); err != nil { + harness.WriteError(w, http.StatusBadRequest, err.Error()) + return + } + if !sameCancelIdentity(turn.request, req) { + harness.WriteError(w, http.StatusBadRequest, "cancel request does not match started turn") + return + } + var cancelHostedRequest context.CancelFunc + s.mu.Lock() + if !turn.completed { + cancelHostedRequest = turn.activeHostedRequestCancel + _, reusedRuntimeSession := s.runtimeSessions[turn.request.RuntimeSessionID] + if cancelHostedRequest != nil || reusedRuntimeSession { + s.quarantineRuntimeSessionLocked(turn) + } + s.suppressPendingToolCallsLocked(turn) + s.clearBufferedToolResultsLocked(turn) + s.appendFrameLocked(turn, harness.FrameTurnCancelled, "turn cancelled") + turn.completed = true + s.scheduleTurnCleanupLocked(turn) + } + s.mu.Unlock() + if cancelHostedRequest != nil { + cancelHostedRequest() + } + harness.WriteJSON( + w, + http.StatusAccepted, + harness.CancelTurnResponse{ + Version: harness.ProtocolVersion, + Accepted: true, + RuntimeSessionID: req.RuntimeSessionID, + TurnID: req.TurnID, + CorrelationID: req.CorrelationID, + }, + ) +} + +func (s *server) handleResponsesResponse(turn *turnState, response responsesResponse) { + s.mu.Lock() + defer s.mu.Unlock() + _ = s.handleResponsesResponseLocked(turn, response) +} + +func (s *server) handleResponsesResponseLocked(turn *turnState, response responsesResponse) responseDisposition { + if turn.completed { + return responseRejected + } + responseIDPresent := strings.TrimSpace(response.ID) != "" + if responseIDPresent { + turn.responseID = response.ID + } + if response.Error != nil { + s.appendFailedLocked( + turn, + "foundry_response_error", + "Foundry hosted Responses returned an error", + ) + return responseRejected + } + if isFailureStatus(response.Status) { + s.appendFailedLocked(turn, "foundry_"+response.Status, "Foundry hosted Responses status "+response.Status) + return responseRejected + } + if strings.TrimSpace(response.Status) == "" { + s.appendFailedLocked(turn, "foundry_status_missing", "Foundry hosted Responses status is missing") + return responseRejected + } + if !isCompletionStatus(response.Status) { + s.appendFailedLocked(turn, "foundry_"+response.Status, "Foundry hosted Responses status "+response.Status) + return responseRejected + } + calls, err := s.extractFunctionCalls(turn.request, response.Output) + if err != nil { + s.appendFailedLocked(turn, "foundry_function_call_invalid", err.Error()) + return responseRejected + } + if len(calls) > 0 { + if len(calls) > maxBrokeredToolCalls-turn.requestedToolCalls { + s.appendFailedLocked( + turn, + "foundry_tool_call_limit_exceeded", + "hosted response exceeded the brokered tool-call limit", + ) + return responseRejected + } + if !responseIDPresent { + s.appendFailedLocked( + turn, + "foundry_response_id_missing", + "hosted response returned a function_call without an id needed for continuation", + ) + return responseRejected + } + seenInResponse := map[string]struct{}{} + for _, call := range calls { + if _, seen := seenInResponse[call.callID]; seen { + s.appendFailedLocked( + turn, + "foundry_repeated_function_call", + "hosted response repeated a function call id", + ) + return responseRejected + } + seenInResponse[call.callID] = struct{}{} + if _, submitted := turn.submittedDigests[call.callID]; submitted { + s.appendFailedLocked( + turn, + "foundry_repeated_function_call", + "hosted response repeated an already-submitted function call", + ) + return responseRejected + } + if _, pending := turn.pendingTools[call.callID]; pending { + s.appendFailedLocked( + turn, + "foundry_repeated_function_call", + "hosted response repeated an already-pending function call", + ) + return responseRejected + } + } + frames := make([]harness.HarnessEventFrame, 0, len(calls)) + baseSeq := int64(len(turn.frames) + 1) + for index, call := range calls { + frame := s.newFrame( + turn, + baseSeq+int64(index), + harness.FrameToolCallRequested, + "foundry hosted tool call requested", + func(f *harness.HarnessEventFrame) { + f.ToolName = call.name + f.ToolCallID = call.callID + f.Content = call.args + }, + ) + if !harnessFrameFitsSSE(frame) { + s.appendFailedLocked( + turn, + "foundry_tool_call_frame_too_large", + "hosted function call exceeded the harness SSE frame limit", + ) + return responseRejected + } + frames = append(frames, frame) + } + turn.requestedToolCalls += len(calls) + for index, call := range calls { + turn.pendingTools[call.callID] = call.name + s.appendPreparedFrameLocked(turn, frames[index]) + } + return responsePending + } + result := responsesMessageText(response.Output) + if len([]byte(result)) > maxFoundryOutputBytes { + s.appendFailedLocked(turn, "foundry_output_too_large", "foundry completion exceeded advertised output limit") + return responseRejected + } + completedFrame := s.newFrame( + turn, + int64(len(turn.frames)+1), + harness.FrameTurnCompleted, + "foundry hosted response completed", + func(f *harness.HarnessEventFrame) { + f.Completed = &harness.TurnCompleted{Result: result, FinalEventSeq: f.Seq} + }, + ) + if !harnessFrameFitsSSE(completedFrame) { + s.appendFailedLocked( + turn, + "foundry_output_frame_too_large", + "foundry completion exceeded the harness SSE frame limit", + ) + return responseRejected + } + s.appendPreparedFrameLocked(turn, completedFrame) + s.clearBufferedToolResultsLocked(turn) + turn.completed = true + s.scheduleTurnCleanupLocked(turn) + return responseCompleted +} + +func (s *server) extractFunctionCalls( + request harness.StartTurnRequest, + output []responsesOutput, +) ([]pendingFunctionCall, error) { + calls := []responsesOutput{} + for _, item := range output { + if strings.EqualFold(strings.TrimSpace(item.Type), "function_call") { + calls = append(calls, item) + } + } + if len(calls) == 0 { + return nil, nil + } + if request.ToolExecutionMode != harness.ToolExecutionModeBrokered { + return nil, fmt.Errorf( + "hosted response requested a function_call while Orka turn is not in brokered mode", + ) + } + pending := make([]pendingFunctionCall, 0, len(calls)) + seenCallIDs := map[string]struct{}{} + for _, call := range calls { + callID := strings.TrimSpace(call.CallID) + name := strings.TrimSpace(call.Name) + if callID == "" { + return nil, fmt.Errorf("hosted response function_call missing call_id") + } + if _, exists := seenCallIDs[callID]; exists { + return nil, fmt.Errorf("hosted response repeated function_call call_id %q", callID) + } + seenCallIDs[callID] = struct{}{} + if name == "" { + return nil, fmt.Errorf("hosted response function_call %q missing name", callID) + } + definition, ok := findToolDefinition(request.Input.Tools, name) + if !ok { + return nil, fmt.Errorf( + "hosted response requested tool %q that Orka did not expose for this turn", + name, + ) + } + if !s.supportsBrokeredClass(definition.BrokeredClass) { + return nil, fmt.Errorf( + "hosted response requested tool %q with unsupported brokered class %q", + name, + definition.BrokeredClass, + ) + } + args, err := normalizeResponsesToolArguments(call.Arguments) + if err != nil { + return nil, err + } + pending = append(pending, pendingFunctionCall{callID: callID, name: name, args: args}) + } + return pending, nil +} + +func (s *server) recordContinueResults( + turn *turnState, + results []harness.ToolCallResult, +) ([]harness.ToolCallResult, error) { + s.mu.Lock() + defer s.mu.Unlock() + if turn.completed { + return nil, fmt.Errorf("turn is already terminal") + } + if len(turn.pendingTools) == 0 { + return nil, fmt.Errorf("no tool calls are pending for this turn") + } + newResults := map[string]harness.ToolCallResult{} + newDigests := map[string]toolResultDigest{} + for _, result := range results { + payload, err := canonicalToolResultOutput(result) + if err != nil { + return nil, err + } + digest := digestToolResultPayload(payload) + if submitted, done := turn.submittedDigests[result.ToolCallID]; done { + if submitted == digest { + continue + } + return nil, fmt.Errorf("conflicting duplicate result for tool call %q", result.ToolCallID) + } + if _, pending := turn.pendingTools[result.ToolCallID]; !pending { + return nil, fmt.Errorf("tool result %q is not pending for this turn", result.ToolCallID) + } + if buffered, exists := turn.bufferedDigests[result.ToolCallID]; exists { + if buffered == digest { + continue + } + return nil, fmt.Errorf("conflicting duplicate result for tool call %q", result.ToolCallID) + } + if buffered, exists := newDigests[result.ToolCallID]; exists { + if buffered == digest { + continue + } + return nil, fmt.Errorf("conflicting duplicate result for tool call %q", result.ToolCallID) + } + newResults[result.ToolCallID] = result + newDigests[result.ToolCallID] = digest + } + + unsubmittedIDs := make([]string, 0, len(turn.pendingTools)) + for id := range turn.pendingTools { + if _, submitted := turn.submittedDigests[id]; !submitted { + unsubmittedIDs = append(unsubmittedIDs, id) + } + } + sort.Strings(unsubmittedIDs) + baseSeq := int64(len(turn.frames) + 1) + for index, id := range unsubmittedIDs { + result, isNew := newResults[id] + if !isNew { + continue + } + toolName := firstNonBlank(turn.pendingTools[id], id) + frame := s.newToolResultFrame(turn, baseSeq+int64(index), toolName, result) + if !toolResultFrameFitsSSE(frame) { + maps.Copy(turn.submittedDigests, turn.bufferedDigests) + maps.Copy(turn.submittedDigests, newDigests) + if _, reused := s.runtimeSessions[turn.request.RuntimeSessionID]; reused { + s.quarantineRuntimeSessionLocked(turn) + } + s.appendFailedLocked( + turn, + "brokered_tool_result_frame_too_large", + "brokered tool result exceeded the harness SSE frame limit", + ) + return nil, fmt.Errorf("brokered tool result %q exceeds harness SSE frame limit", id) + } + } + + for id, result := range newResults { + turn.bufferedResults[id] = result + turn.bufferedDigests[id] = newDigests[id] + } + readyCount := 0 + for id := range turn.pendingTools { + if _, submitted := turn.submittedDigests[id]; submitted { + readyCount++ + continue + } + if _, buffered := turn.bufferedResults[id]; buffered { + readyCount++ + } + } + if readyCount < len(turn.pendingTools) { + if len(newResults) > 0 { + s.notifyTurnUpdatedLocked(turn) + } + return nil, nil + } + toSubmit := make([]harness.ToolCallResult, 0, len(unsubmittedIDs)) + for _, id := range unsubmittedIDs { + toSubmit = append(toSubmit, turn.bufferedResults[id]) + } + if len(toSubmit) == 0 { + return nil, nil + } + for _, result := range toSubmit { + turn.submittedDigests[result.ToolCallID] = turn.bufferedDigests[result.ToolCallID] + } + return toSubmit, nil +} + +func (s *server) ensureTerminalContinueIsDuplicate(turn *turnState, results []harness.ToolCallResult) error { + s.mu.Lock() + defer s.mu.Unlock() + if !turn.completed { + return nil + } + for _, result := range results { + payload, err := canonicalToolResultOutput(result) + if err != nil { + return err + } + submitted, done := turn.submittedDigests[result.ToolCallID] + if !done { + return fmt.Errorf("terminal turn cannot accept new tool result %q", result.ToolCallID) + } + if submitted != digestToolResultPayload(payload) { + return fmt.Errorf("conflicting duplicate result for tool call %q", result.ToolCallID) + } + } + return nil +} + +func functionCallOutputs(results []harness.ToolCallResult) ([]responsesFunctionCallOutput, error) { + outputs := make([]responsesFunctionCallOutput, 0, len(results)) + for _, result := range results { + payload, err := canonicalToolResultOutput(result) + if err != nil { + return nil, err + } + outputs = append( + outputs, + responsesFunctionCallOutput{ + Type: "function_call_output", + CallID: result.ToolCallID, + Output: payload, + Status: "completed", + }, + ) + } + return outputs, nil +} + +func canonicalToolResultOutput(result harness.ToolCallResult) (string, error) { + if result.Error != nil { + return compactJSON(struct { + Approved bool `json:"approved"` + Error *harness.ErrorInfo `json:"error"` + }{Approved: false, Error: hostedToolError(result.Error)}) + } + if !result.Approved { + return compactJSON(struct { + Approved bool `json:"approved"` + Error *harness.ErrorInfo `json:"error"` + }{Approved: false, Error: &harness.ErrorInfo{Code: "approval_declined", Message: "tool call was not approved"}}) + } + output := json.RawMessage(`{}`) + if len(result.Output) > 0 { + compacted, err := compactRawJSON(result.Output) + if err != nil { + return "", fmt.Errorf("tool result %q output must be valid JSON: %w", result.ToolCallID, err) + } + if trimmed := bytes.TrimSpace(compacted); len(trimmed) > 0 && trimmed[0] != '{' { + wrapped, err := json.Marshal(struct { + Result json.RawMessage `json:"result"` + }{Result: compacted}) + if err != nil { + return "", fmt.Errorf("wrap tool result %q output: %w", result.ToolCallID, err) + } + output = wrapped + } else { + output = compacted + } + } + return compactJSON(struct { + Approved bool `json:"approved"` + Output json.RawMessage `json:"output"` + }{Approved: true, Output: output}) +} + +func hostedToolError(err *harness.ErrorInfo) *harness.ErrorInfo { + if err == nil { + return nil + } + switch strings.TrimSpace(err.Code) { + case "approval_declined": + return &harness.ErrorInfo{Code: "approval_declined", Message: "tool call was not approved"} + case "tool_policy_rejected": + return &harness.ErrorInfo{Code: "tool_policy_rejected", Message: "tool call was rejected by policy"} + case "tool_execution_failed": + return &harness.ErrorInfo{Code: "tool_execution_failed", Message: "tool execution failed"} + default: + return &harness.ErrorInfo{Code: "tool_execution_failed", Message: "tool execution failed"} + } +} + +func compactRawJSON(raw json.RawMessage) (json.RawMessage, error) { + var compacted bytes.Buffer + if err := json.Compact(&compacted, raw); err != nil { + return nil, err + } + return json.RawMessage(compacted.Bytes()), nil +} + +func compactJSON(value any) (string, error) { + encoded, err := json.Marshal(value) + if err != nil { + return "", err + } + return string(encoded), nil +} + +func (s *server) postResponses( + ctx context.Context, + body responsesRequest, + out *responsesResponse, +) (string, error) { + if !exactlyOneFoundryAuth(s.cfg) { + return "", fmt.Errorf("exactly one Foundry auth mode is required") + } + endpoint, err := s.responsesEndpoint() + if err != nil { + return "", err + } + sessionID := strings.TrimSpace(body.AgentSessionID) + // Carry the continuation proof in the body as well as the header, so it + // survives hosted-agent gateways (e.g. Foundry) that strip custom request + // headers before forwarding to the runtime container. Only on continuations. + if body.PreviousResponseID != "" && s.cfg.continuationProof != "" { + body.BrokeredContinuationProof = s.cfg.continuationProof + } + payload, err := json.Marshal(body) + if err != nil { + return "", err + } + req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(payload)) + if err != nil { + return "", err + } + req.Header.Set("Content-Type", "application/json") + req.Header.Set("Accept", "application/json") + req.Header.Set("Foundry-Features", "HostedAgents=V1Preview") + if s.cfg.foundryAuth != "" { + req.Header.Set("api-key", s.cfg.foundryAuth) + } + if s.cfg.authBearer != "" { + req.Header.Set("Authorization", "Bearer "+s.cfg.authBearer) + } + if body.PreviousResponseID != "" && s.cfg.continuationProof != "" { + req.Header.Set("X-AgentKit-Brokered-Continuation-Proof", s.cfg.continuationProof) + } + if sessionID != "" { + req.Header.Set("x-agent-session-id", sessionID) + } + resp, err := s.client.Do(req) + if err != nil { + return "", err + } + defer resp.Body.Close() //nolint:errcheck + sessionID = firstNonBlank( + resp.Header.Get("x-agent-session-id"), + resp.Header.Get("x-ms-agent-session-id"), + sessionID, + ) + if resp.StatusCode < 200 || resp.StatusCode >= 300 { + data, _ := io.ReadAll(io.LimitReader(resp.Body, 4096)) + return "", fmt.Errorf( + "foundry hosted Responses request failed: HTTP %d: %s", + resp.StatusCode, + strings.TrimSpace(string(data)), + ) + } + responseBody, err := io.ReadAll(io.LimitReader(resp.Body, maxFoundryBodyBytes+1)) + if err != nil { + return "", fmt.Errorf("read Foundry hosted Responses response: %w", err) + } + if len(responseBody) > maxFoundryBodyBytes { + return "", fmt.Errorf("foundry hosted Responses response exceeded %d bytes", maxFoundryBodyBytes) + } + if out != nil { + if err := json.Unmarshal(responseBody, out); err != nil { + return "", fmt.Errorf("decode Foundry hosted Responses response: %w", err) + } + sessionID = firstNonBlank(out.AgentSessionID, sessionID) + } + return sessionID, nil +} + +func (s *server) responsesEndpoint() (string, error) { + if strings.TrimSpace(s.cfg.endpoint) != "" { + return responsesEndpointWithVersion(s.cfg.endpoint, s.cfg.apiVersion) + } + if strings.TrimSpace(s.cfg.projectEndpoint) == "" || strings.TrimSpace(s.cfg.agentName) == "" { + return "", fmt.Errorf( + "%s; set %s or %s plus %s", + responsesEndpointRequirement, + envEndpoint, + envProjectEndpoint, + envAgentName, + ) + } + if !projectEndpointIsSafe(s.cfg.projectEndpoint) { + return "", fmt.Errorf("%s; project endpoint is unsafe", responsesEndpointRequirement) + } + base := strings.TrimRight( + s.cfg.projectEndpoint, + "/", + ) + "/agents/" + url.PathEscape( + s.cfg.agentName, + ) + "/endpoint/protocols/openai/responses" + return responsesEndpointWithVersion(base, s.cfg.apiVersion) +} + +func responsesEndpointWithVersion(raw, apiVersion string) (string, error) { + if !responsesEndpointIsSafe(raw) { + return "", errors.New(responsesEndpointRequirement) + } + u, err := url.Parse(strings.TrimSpace(raw)) + if err != nil { + return "", errors.New(responsesEndpointRequirement) + } + if strings.TrimSpace(apiVersion) != "" { + q := u.Query() + if q.Get("api-version") == "" { + q.Set("api-version", strings.TrimSpace(apiVersion)) + u.RawQuery = q.Encode() + } + } + return u.String(), nil +} + +func responsesEndpointIsSafe(raw string) bool { + trimmed := strings.TrimSpace(raw) + parsed, err := url.Parse(trimmed) + if err != nil || parsed.Scheme == "" || parsed.Host == "" || strings.TrimSpace(parsed.Hostname()) == "" || + strings.TrimSpace(parsed.Path) == "" { + return false + } + if parsed.User != nil || parsed.ForceQuery || parsed.Fragment != "" || strings.Contains(trimmed, "#") { + return false + } + if !strings.HasSuffix(strings.TrimRight(parsed.Path, "/"), "/responses") { + return false + } + if parsed.RawQuery != "" { + values, err := url.ParseQuery(parsed.RawQuery) + if err != nil { + return false + } + for key, vals := range values { + if key != "api-version" || len(vals) == 0 { + return false + } + for _, val := range vals { + if strings.TrimSpace(val) == "" { + return false + } + } + } + } + if strings.EqualFold(parsed.Scheme, "https") { + return true + } + if !strings.EqualFold(parsed.Scheme, "http") { + return false + } + host := strings.Trim(strings.ToLower(parsed.Hostname()), "[]") + return host == "localhost" || host == "127.0.0.1" || host == "::1" +} + +func projectEndpointIsSafe(raw string) bool { + trimmed := strings.TrimSpace(raw) + parsed, err := url.Parse(trimmed) + if err != nil || parsed.Scheme == "" || parsed.Host == "" || strings.TrimSpace(parsed.Hostname()) == "" { + return false + } + if parsed.User != nil || parsed.ForceQuery || parsed.RawQuery != "" || parsed.Fragment != "" || + strings.Contains(trimmed, "#") { + return false + } + if strings.EqualFold(parsed.Scheme, "https") { + return true + } + if !strings.EqualFold(parsed.Scheme, "http") { + return false + } + host := strings.Trim(strings.ToLower(parsed.Hostname()), "[]") + return host == "localhost" || host == "127.0.0.1" || host == "::1" +} + +func exactlyOneFoundryAuth(cfg config) bool { + hasKey := strings.TrimSpace(cfg.foundryAuth) != "" + hasBearer := strings.TrimSpace(cfg.authBearer) != "" + return hasKey != hasBearer +} + +func (s *server) quarantineRuntimeSessionLocked(turn *turnState) { + // A local request failure or cancellation does not prove that Foundry + // stopped an already accepted operation. Invalidate and quarantine this + // logical runtime session for the adapter process lifetime so no later turn + // can overlap with or inherit state from an uncertain hosted outcome. + sessionID := turn.request.RuntimeSessionID + delete(s.runtimeSessions, sessionID) + if s.quarantineSaturated { + return + } + if _, exists := s.quarantinedSessions[sessionID]; exists { + return + } + if len(s.quarantinedSessions) >= maxQuarantinedRuntimeSessions { + // Keep memory bounded and fail closed globally until process restart. + s.quarantineSaturated = true + return + } + s.quarantinedSessions[sessionID] = struct{}{} +} + +func (s *server) activateHostedRequestLocked( + turn *turnState, + cancel context.CancelFunc, +) uint64 { + turn.hostedRequestSequence++ + turn.activeHostedRequestID = turn.hostedRequestSequence + turn.activeHostedRequestCancel = cancel + return turn.activeHostedRequestID +} + +func (s *server) clearHostedRequestLocked(turn *turnState, requestID uint64) { + if turn.activeHostedRequestID != requestID { + return + } + turn.activeHostedRequestID = 0 + turn.activeHostedRequestCancel = nil +} + +func (s *server) foundryRequestContext( + parent context.Context, + turnDeadline time.Time, +) (context.Context, context.CancelFunc) { + timeout := s.cfg.requestTimeout + if timeout <= 0 { + timeout = defaultRequestTimeout + } + deadline := time.Now().Add(timeout) + if !turnDeadline.IsZero() && turnDeadline.Before(deadline) { + deadline = turnDeadline + } + return context.WithDeadline(context.WithoutCancel(parent), deadline) +} + +func (s *server) validateStartRequest(req harness.StartTurnRequest) error { + if err := s.cfg.validationError(); err != nil { + return err + } + if _, err := s.responsesEndpoint(); err != nil { + return err + } + if !exactlyOneFoundryAuth(s.cfg) { + return fmt.Errorf("exactly one Foundry auth mode is required") + } + if req.ToolExecutionMode == harness.ToolExecutionModeBrokered { + if len(s.cfg.brokeredToolClasses) == 0 { + return fmt.Errorf("brokered mode is not enabled for this hosted Responses adapter") + } + for _, tool := range req.Input.Tools { + if !s.supportsBrokeredClass(tool.BrokeredClass) { + return fmt.Errorf( + "brokered tool %q class %q is not advertised by this adapter", + tool.Name, + tool.BrokeredClass, + ) + } + } + } + return nil +} + +func (s *server) supportsBrokeredClass(class harness.BrokeredToolClass) bool { + return slices.Contains(s.cfg.brokeredToolClasses, class) +} + +func findToolDefinition(definitions []harness.ToolDefinition, name string) (harness.ToolDefinition, bool) { + name = strings.TrimSpace(name) + for _, definition := range definitions { + if strings.TrimSpace(definition.Name) == name { + return definition, true + } + } + return harness.ToolDefinition{}, false +} + +func normalizeResponsesToolArguments(raw json.RawMessage) (json.RawMessage, error) { + if len(raw) == 0 || strings.TrimSpace(string(raw)) == "null" { + return json.RawMessage(`{}`), nil + } + var encoded string + if err := json.Unmarshal(raw, &encoded); err == nil { + encoded = strings.TrimSpace(encoded) + if encoded == "" { + return json.RawMessage(`{}`), nil + } + return normalizeResponsesToolArguments(json.RawMessage(encoded)) + } + compacted, err := compactRawJSON(raw) + if err != nil { + return nil, fmt.Errorf("hosted response function_call arguments must be a valid JSON object") + } + trimmed := bytes.TrimSpace(compacted) + if len(trimmed) < 2 || trimmed[0] != '{' || trimmed[len(trimmed)-1] != '}' { + return nil, fmt.Errorf("hosted response function_call arguments must be a valid JSON object") + } + return json.RawMessage(append([]byte(nil), trimmed...)), nil +} + +func responsesMessageText(output []responsesOutput) string { + parts := []string{} + for _, item := range output { + typeName := strings.TrimSpace(item.Type) + if typeName == "message" || typeName == "output_text" || typeName == "text" || typeName == "" { + if text := outputItemText(item); text != "" { + parts = append(parts, text) + } + } + } + return strings.Join(parts, "\n") +} + +func outputItemText(item responsesOutput) string { + if strings.TrimSpace(item.Text) != "" { + return strings.TrimSpace(item.Text) + } + switch content := item.Content.(type) { + case string: + return strings.TrimSpace(content) + case []any: + parts := []string{} + for _, entry := range content { + if m, ok := entry.(map[string]any); ok { + if text, ok := m["text"].(string); ok && strings.TrimSpace(text) != "" { + parts = append(parts, strings.TrimSpace(text)) + } + if refusal, ok := m["refusal"].(string); ok && strings.TrimSpace(refusal) != "" { + parts = append(parts, strings.TrimSpace(refusal)) + } + if textMap, ok := m["text"].(map[string]any); ok { + if value, ok := textMap["value"].(string); ok && strings.TrimSpace(value) != "" { + parts = append(parts, strings.TrimSpace(value)) + } + } + } + } + return strings.Join(parts, "\n") + } + return "" +} + +func digestToolResultPayload(payload string) toolResultDigest { + return sha256.Sum256([]byte(payload)) +} + +func (s *server) newToolResultFrame( + turn *turnState, + seq int64, + toolName string, + result harness.ToolCallResult, +) harness.HarnessEventFrame { + return s.newFrame( + turn, + seq, + harness.FrameToolResultReceived, + "brokered tool result received", + func(f *harness.HarnessEventFrame) { + f.ToolName = toolName + f.ToolCallID = result.ToolCallID + f.Content = result.Output + f.Error = result.Error + }, + ) +} + +func (s *server) suppressPendingToolCallsLocked(turn *turnState) { + if turn.suppressedToolCalls == nil { + turn.suppressedToolCalls = map[string]struct{}{} + } + for toolCallID := range turn.pendingTools { + turn.suppressedToolCalls[toolCallID] = struct{}{} + } +} + +func (s *server) clearBufferedToolResultsLocked(turn *turnState) { + clear(turn.bufferedResults) + clear(turn.bufferedDigests) +} + +func isCompletionStatus(status string) bool { + return strings.EqualFold(strings.TrimSpace(status), "completed") +} + +func isFailureStatus(status string) bool { + switch strings.ToLower(strings.TrimSpace(status)) { + case "failed", "cancelled", "expired", "incomplete": + return true + default: + return false + } +} + +func (s *server) setTurnSessionLocked(turn *turnState, sessionID string) { + sessionID = strings.TrimSpace(sessionID) + if sessionID == "" { + return + } + turn.foundrySessionID = sessionID +} + +func (s *server) publishRuntimeSessionLocked(turn *turnState) { + if strings.TrimSpace(turn.foundrySessionID) == "" { + return + } + s.runtimeSessions[turn.request.RuntimeSessionID] = foundrySession{ + ID: turn.foundrySessionID, + LastSeen: time.Now().UTC(), + } +} + +func (s *server) appendFailedLocked(turn *turnState, reason, msg string) { + if turn.completed { + return + } + s.suppressPendingToolCallsLocked(turn) + s.clearBufferedToolResultsLocked(turn) + failedFrame := s.newFrame( + turn, + int64(len(turn.frames)+1), + harness.FrameTurnFailed, + "foundry hosted response failed", + func(f *harness.HarnessEventFrame) { + f.Failed = &harness.TurnFailed{Reason: reason, Message: msg} + f.Error = &harness.ErrorInfo{Code: reason, Message: msg} + }, + ) + if !harnessFrameFitsSSE(failedFrame) { + failedFrame = s.newFrame( + turn, + int64(len(turn.frames)+1), + harness.FrameTurnFailed, + "foundry hosted response failed", + func(f *harness.HarnessEventFrame) { + fallbackReason := "foundry_failure_frame_too_large" + message := "failure detail exceeded the harness SSE frame limit" + f.Failed = &harness.TurnFailed{Reason: fallbackReason, Message: message} + f.Error = &harness.ErrorInfo{Code: fallbackReason, Message: message} + }, + ) + } + s.appendPreparedFrameLocked(turn, failedFrame) + turn.completed = true + s.scheduleTurnCleanupLocked(turn) +} + +func (s *server) scheduleTurnCleanupLocked(turn *turnState) { + turnID := turn.request.TurnID + retention := s.cfg.stateRetention + if retention <= 0 { + retention = defaultStateRetention + } + time.AfterFunc(retention, func() { + s.mu.Lock() + defer s.mu.Unlock() + if current := s.turns[turnID]; current == turn { + delete(s.turns, turnID) + s.markTurnConsumedLocked(turnID) + } + activeSessions := s.activeRuntimeSessionsLocked() + cutoff := time.Now().UTC().Add(-retention) + for sessionID, session := range s.runtimeSessions { + if activeSessions[sessionID] { + continue + } + if session.LastSeen.Before(cutoff) { + delete(s.runtimeSessions, sessionID) + } + } + }) +} + +func (s *server) markTurnConsumedLocked(turnID harness.HarnessTurnID) { + if _, consumed := s.consumedTurns[turnID]; consumed { + return + } + s.consumedTurns[turnID] = struct{}{} + s.consumedOrder = append(s.consumedOrder, turnID) + for len(s.consumedOrder) > maxConsumedTurnIDs { + oldest := s.consumedOrder[0] + s.consumedOrder = s.consumedOrder[1:] + delete(s.consumedTurns, oldest) + } +} + +func (s *server) activeTurnCountLocked() int { + active := 0 + for _, turn := range s.turns { + if turn != nil && (turn.initializing || turn.continuationInFlight || !turn.completed) { + active++ + } + } + return active +} + +func (s *server) activeRuntimeSessionsLocked() map[harness.RuntimeSessionID]bool { + active := map[harness.RuntimeSessionID]bool{} + for _, turn := range s.turns { + if turn != nil && !turn.completed { + active[turn.request.RuntimeSessionID] = true + } + } + return active +} + +func (s *server) appendFrameLocked(turn *turnState, typ harness.FrameType, summary string) { + frame := s.newFrame(turn, int64(len(turn.frames)+1), typ, summary, nil) + s.appendPreparedFrameLocked(turn, frame) +} + +func (s *server) newFrame( + turn *turnState, + seq int64, + typ harness.FrameType, + summary string, + mutate func(*harness.HarnessEventFrame), +) harness.HarnessEventFrame { + frame := harness.HarnessEventFrame{ + Version: harness.ProtocolVersion, + Type: typ, + RuntimeSessionID: turn.request.RuntimeSessionID, + TurnID: turn.request.TurnID, + CorrelationID: turn.request.CorrelationID, + Seq: seq, + CreatedAt: time.Now().UTC(), + Summary: summary, + Metadata: map[string]string{"backend": "foundry-responses"}, + } + if mutate != nil { + mutate(&frame) + } + return frame +} + +func toolResultFrameFitsSSE(frame harness.HarnessEventFrame) bool { + // RFC3339Nano omits trailing zeros. Probe with full nanosecond precision so + // the preflight is conservative for the exact sequence the frame will use. + frame.CreatedAt = sseSizeProbeTime + return harnessFrameFitsSSE(frame) +} + +func harnessFrameFitsSSE(frame harness.HarnessEventFrame) bool { + payload, err := json.Marshal(frame) + return err == nil && len("data: ")+len(payload) < harness.MaxSSEFrameBytes +} + +func (s *server) appendPreparedFrameLocked(turn *turnState, frame harness.HarnessEventFrame) { + turn.frames = append(turn.frames, frame) + s.notifyTurnUpdatedLocked(turn) +} + +func (s *server) notifyTurnUpdatedLocked(turn *turnState) { + if turn.frameUpdates != nil { + close(turn.frameUpdates) + } + turn.frameUpdates = make(chan struct{}) +} + +func (s *server) authorized(w http.ResponseWriter, r *http.Request) bool { + if s.cfg.adapterBearer == "" { + harness.WriteError(w, http.StatusUnauthorized, "adapter bearer token is required") + return false + } + got := strings.TrimSpace(strings.TrimPrefix(r.Header.Get("Authorization"), "Bearer ")) + if got == "" || subtle.ConstantTimeCompare([]byte(got), []byte(s.cfg.adapterBearer)) != 1 { + harness.WriteError(w, http.StatusUnauthorized, "unauthorized") + return false + } + return true +} + +func sameStartTurnRequest(existing, retry harness.StartTurnRequest) bool { + return reflect.DeepEqual(existing, retry) +} + +func sameCancelIdentity(start harness.StartTurnRequest, cancel harness.CancelTurnRequest) bool { + return start.Namespace == cancel.Namespace && + start.TaskName == cancel.TaskName && + start.SessionName == cancel.SessionName && + start.RuntimeSessionID == cancel.RuntimeSessionID && + start.TurnID == cancel.TurnID && + start.CorrelationID == cancel.CorrelationID +} + +func sameContinueIdentity(start harness.StartTurnRequest, cont harness.ContinueTurnRequest) bool { + return start.Namespace == cont.Namespace && + start.TaskName == cont.TaskName && + start.SessionName == cont.SessionName && + start.RuntimeSessionID == cont.RuntimeSessionID && + start.TurnID == cont.TurnID && + start.CorrelationID == cont.CorrelationID +} + +func startTurnResponse(req harness.StartTurnRequest, eventsPath string) harness.StartTurnResponse { + return harness.StartTurnResponse{ + Version: harness.ProtocolVersion, + Accepted: true, + RuntimeSessionID: req.RuntimeSessionID, + TurnID: req.TurnID, + CorrelationID: req.CorrelationID, + EventStreamPath: eventsPath, + } +} + +func continueResponse(req harness.ContinueTurnRequest, msg string) harness.ContinueTurnResponse { + return harness.ContinueTurnResponse{ + Version: harness.ProtocolVersion, + Accepted: true, + RuntimeSessionID: req.RuntimeSessionID, + TurnID: req.TurnID, + CorrelationID: req.CorrelationID, + Message: msg, + } +} + +func parseAfterSeq(value string) int64 { + parsed, err := strconv.ParseInt(strings.TrimSpace(value), 10, 64) + if err != nil || parsed < 0 { + return 0 + } + return parsed +} + +func parseBrokeredToolClasses(raw string) ([]harness.BrokeredToolClass, error) { + raw = strings.TrimSpace(raw) + if raw == "" { + return nil, nil + } + classes := []harness.BrokeredToolClass{} + seen := map[harness.BrokeredToolClass]struct{}{} + for part := range strings.SplitSeq(raw, ",") { + class := harness.BrokeredToolClass(strings.TrimSpace(part)) + if class == "" { + continue + } + switch class { + case harness.BrokeredToolClassRead, harness.BrokeredToolClassWrite: + default: + return nil, fmt.Errorf( + "unsupported %s value %q; supported values are read,write", + envBrokeredToolClasses, + class, + ) + } + if _, ok := seen[class]; ok { + continue + } + seen[class] = struct{}{} + classes = append(classes, class) + } + return classes, nil +} + +func firstNonBlank(values ...string) string { + for _, value := range values { + if strings.TrimSpace(value) != "" { + return strings.TrimSpace(value) + } + } + return "" +} + +func parseDurationEnv(name string, fallback time.Duration) time.Duration { + if value := strings.TrimSpace(os.Getenv(name)); value != "" { + if parsed, err := time.ParseDuration(value); err == nil && parsed > 0 { + return parsed + } + } + return fallback +} + +func sanitizeEndpoint(raw string) string { + if raw == "" { + return "" + } + u, err := url.Parse(raw) + if err != nil { + return "[invalid endpoint]" + } + u.User = nil + q := u.Query() + for key := range q { + if key != "api-version" { + q.Del(key) + } + } + u.RawQuery = q.Encode() + u.Fragment = "" + return u.String() +} diff --git a/examples/harness/foundry-responses/main_test.go b/examples/harness/foundry-responses/main_test.go new file mode 100644 index 000000000..26ce9660c --- /dev/null +++ b/examples/harness/foundry-responses/main_test.go @@ -0,0 +1,3538 @@ +package main + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "net/http/httptest" + "os" + "reflect" + "slices" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/orka-agents/orka/internal/harness" + "github.com/orka-agents/orka/internal/harness/conformance" +) + +const ( + fakeSessionID = "session-1" + existingSessionID = "existing-session" + foundryFailedReason = "foundry_failed" + testContinuationProof = "proof-for-test" +) + +func TestResponsesAdapterObservedTurnCompletes(t *testing.T) { + foundry := newFakeResponses(t, fakeResponsesConfig{scenario: "observed"}) + adapter := newTestResponsesAdapter(t, foundry.endpoint(), nil) + client := newHarnessClient(t, adapter) + + request := responsesStartTurnRequest("foundry-observed") + if _, err := client.StartTurn(context.Background(), request); err != nil { + t.Fatalf("StartTurn: %v", err) + } + var frames []harness.HarnessEventFrame + if err := client.StreamFrames(context.Background(), request.TurnID, 0, func(frame harness.HarnessEventFrame) error { + frames = append(frames, frame) + return nil + }); err != nil { + t.Fatalf("StreamFrames: %v", err) + } + if !hasFrameType(frames, harness.FrameTurnStarted) || !hasFrameType(frames, harness.FrameTurnCompleted) { + t.Fatalf("frames = %#v, want started and completed", frames) + } + if got := frames[len(frames)-1].Completed.Result; got != "foundry final answer" { + t.Fatalf("result = %q", got) + } + if foundry.sawRequestLevelTools.Load() { + t.Fatalf("hosted Responses request included request-level tools") + } + if got := foundry.requestHeader(0).Get("Foundry-Features"); got != "HostedAgents=V1Preview" { + t.Fatalf("Foundry-Features = %q, want HostedAgents=V1Preview", got) + } + assertJSONFileEqual(t, "testdata/golden/01_initial_hosted_request.json", foundry.requestBody(0)) +} + +func TestResponsesAdapterDoesNotFollowCredentialedRedirects(t *testing.T) { + redirectTargetHit := atomic.Bool{} + redirectTarget := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) { + redirectTargetHit.Store(true) + })) + t.Cleanup(redirectTarget.Close) + + redirector := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + http.Redirect(w, r, redirectTarget.URL+"/capture", http.StatusTemporaryRedirect) + })) + t.Cleanup(redirector.Close) + + endpoint := redirector.URL + + "/agents/test-agent/endpoint/protocols/openai/responses?api-version=v1" + adapter := newTestResponsesAdapter(t, endpoint, nil) + client := newHarnessClient(t, adapter) + request := responsesStartTurnRequest("foundry-redirect") + if _, err := client.StartTurn(context.Background(), request); err != nil { + t.Fatalf("StartTurn should accept terminal redirect failure: %v", err) + } + frames := streamCurrentFrames(t, client, request.TurnID) + if failed := findFrame(frames, harness.FrameTurnFailed); failed == nil || + failed.Failed.Reason != foundryInitialUnknown { + t.Fatalf("failed frame = %#v, want foundry_initial_unknown", failed) + } + if redirectTargetHit.Load() { + t.Fatal("redirect target was called; credentialed Foundry request followed an unvalidated redirect") + } +} + +func TestResponsesAdapterBrokeredReadContinuationAndGoldenFixtures(t *testing.T) { + foundry := newFakeResponses(t, fakeResponsesConfig{scenario: "function_call", toolName: "support-ticket-lookup"}) + adapter := newTestResponsesAdapter( + t, + foundry.endpoint(), + []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + ) + client := newHarnessClient(t, adapter) + + request := responsesStartTurnRequest("foundry-brokered") + request.ToolExecutionMode = harness.ToolExecutionModeBrokered + request.Input.Tools = []harness.ToolDefinition{{ + Name: "support-ticket-lookup", + Description: "Look up support ticket", + BrokeredClass: harness.BrokeredToolClassRead, + Parameters: json.RawMessage(`{"type":"object","properties":{"incident":{"type":"string"}}}`), + }} + if _, err := client.StartTurn(context.Background(), request); err != nil { + t.Fatalf("StartTurn: %v", err) + } + if foundry.sawRequestLevelTools.Load() { + t.Fatalf("hosted Responses request included request-level tools") + } + assertJSONFileEqual(t, "testdata/golden/01_initial_hosted_request.json", foundry.requestBody(0)) + + var frames []harness.HarnessEventFrame + var continueRequest harness.ContinueTurnRequest + streamCtx, streamCancel := context.WithTimeout(context.Background(), 2*time.Second) + defer streamCancel() + if err := client.StreamFrames(streamCtx, request.TurnID, 0, func(frame harness.HarnessEventFrame) error { + frames = append(frames, frame) + if frame.Type != harness.FrameToolCallRequested { + return nil + } + continueRequest = goldenContinueRequest(request, frame.ToolCallID, json.RawMessage(`{"success":true}`)) + _, err := client.ContinueTurn(context.Background(), continueRequest) + return err + }); err != nil { + t.Fatalf("StreamFrames through continue: %v", err) + } + requested := findFrame(frames, harness.FrameToolCallRequested) + if requested == nil { + t.Fatalf("frames = %#v, want tool request", frames) + } + if requested.ToolName != "support-ticket-lookup" || requested.ToolCallID != "call-1" { + t.Fatalf("tool request = %#v", requested) + } + assertJSONFileEqual(t, "testdata/golden/03_tool_call_requested_frame.json", scrubFrameForGolden(*requested)) + + assertJSONFileEqual(t, "testdata/golden/04_orka_continue_request.json", continueRequest) + assertJSONFileEqual(t, "testdata/golden/05_hosted_continuation_request.json", foundry.requestBody(1)) + if got := foundry.requestHeader(1).Get("x-agent-session-id"); got != fakeSessionID { + t.Fatalf("continuation x-agent-session-id = %q, want session-1", got) + } + if !hasFrameType(frames, harness.FrameToolResultReceived) || !hasFrameType(frames, harness.FrameTurnCompleted) { + t.Fatalf("frames = %#v, want tool result and completion", frames) + } + if foundry.postCount.Load() != 2 { + t.Fatalf("hosted post count = %d, want 2", foundry.postCount.Load()) + } +} + +func TestResponsesAdapterRuntimeSessionHeaderReuse(t *testing.T) { + foundry := newFakeResponses(t, fakeResponsesConfig{scenario: "observed"}) + adapter := newTestResponsesAdapter(t, foundry.endpoint(), nil) + client := newHarnessClient(t, adapter) + + first := responsesStartTurnRequest("foundry-session-one") + if _, err := client.StartTurn(context.Background(), first); err != nil { + t.Fatalf("StartTurn first: %v", err) + } + second := responsesStartTurnRequest("foundry-session-two") + second.RuntimeSessionID = first.RuntimeSessionID + if _, err := client.StartTurn(context.Background(), second); err != nil { + t.Fatalf("StartTurn second: %v", err) + } + if got := foundry.requestHeader(1).Get("x-agent-session-id"); got != fakeSessionID { + t.Fatalf("same runtimeSessionID header = %q, want session-1", got) + } + if got := requestMap(t, foundry.requestBody(1))["agent_session_id"]; got != fakeSessionID { + t.Fatalf("same runtimeSessionID body agent_session_id = %#v, want %q", got, fakeSessionID) + } + + third := responsesStartTurnRequest("foundry-session-three") + if _, err := client.StartTurn(context.Background(), third); err != nil { + t.Fatalf("StartTurn third: %v", err) + } + if got := foundry.requestHeader(2).Get("x-agent-session-id"); got != "" { + t.Fatalf("new runtimeSessionID header = %q, want empty", got) + } +} + +func TestResponsesAdapterInterleavedResponsesRetainResponseSpecificSession(t *testing.T) { + foundry := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + body, _ := ioReadAll(r.Body) + var request responsesRequest + if err := json.Unmarshal(body, &request); err != nil { + http.Error(w, "invalid request", http.StatusBadRequest) + return + } + prompt, _ := request.Input.(string) + writeJSON(w, map[string]any{ + "id": "response-" + prompt, + "agent_session_id": "session-" + prompt, + "status": "completed", + "output": []any{map[string]any{ + "type": "message", + "role": "assistant", + "content": []any{map[string]any{"type": "output_text", "text": "done"}}, + }}, + }) + })) + t.Cleanup(foundry.Close) + server := newServer(config{ + runtimeName: "foundry-responses-test", + adapterBearer: "adapter-auth-value", + endpoint: foundry.URL + "/agents/test-agent/endpoint/protocols/openai/responses?api-version=v1", + foundryAuth: "foundry-auth-value", + requestTimeout: time.Second, + stateRetention: time.Minute, + }, &http.Client{Timeout: time.Second}) + firstRequest := responsesStartTurnRequest("foundry-interleaved-session-a") + secondRequest := responsesStartTurnRequest("foundry-interleaved-session-b") + secondRequest.RuntimeSessionID = firstRequest.RuntimeSessionID + + var firstResponse responsesResponse + firstSession, err := server.postResponses( + context.Background(), + responsesRequest{Input: "a"}, + &firstResponse, + ) + if err != nil { + t.Fatalf("first postResponses: %v", err) + } + var secondResponse responsesResponse + secondSession, err := server.postResponses( + context.Background(), + responsesRequest{Input: "b"}, + &secondResponse, + ) + if err != nil { + t.Fatalf("second postResponses: %v", err) + } + + firstTurn := &turnState{request: firstRequest} + secondTurn := &turnState{request: secondRequest} + server.mu.Lock() + server.setTurnSessionLocked(secondTurn, secondSession) + server.setTurnSessionLocked(firstTurn, firstSession) + server.mu.Unlock() + if firstTurn.foundrySessionID != "session-a" { + t.Fatalf("first turn session = %q, want session-a", firstTurn.foundrySessionID) + } + if secondTurn.foundrySessionID != "session-b" { + t.Fatalf("second turn session = %q, want session-b", secondTurn.foundrySessionID) + } +} + +func TestResponsesAdapterContinuationUsesTurnSessionAfterSessionMapCleanup(t *testing.T) { + foundry := newFakeResponses(t, fakeResponsesConfig{scenario: "function_call", toolName: "support-ticket-lookup"}) + adapter, server := newTestResponsesAdapterWithServer( + t, + foundry.endpoint(), + []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + ) + client := newHarnessClient(t, adapter) + request := brokeredReadRequest("foundry-session-cleanup") + + if _, err := client.StartTurn(context.Background(), request); err != nil { + t.Fatalf("StartTurn: %v", err) + } + frames := streamCurrentFrames(t, client, request.TurnID) + requested := findFrame(frames, harness.FrameToolCallRequested) + if requested == nil { + t.Fatalf("frames = %#v, want tool request", frames) + } + + server.mu.Lock() + delete(server.runtimeSessions, request.RuntimeSessionID) + server.mu.Unlock() + + continueRequest := goldenContinueRequest(request, requested.ToolCallID, json.RawMessage(`{"success":true}`)) + if _, err := client.ContinueTurn(context.Background(), continueRequest); err != nil { + t.Fatalf("ContinueTurn after runtime session cleanup: %v", err) + } + if got := requestMap(t, foundry.requestBody(1))["agent_session_id"]; got != fakeSessionID { + t.Fatalf("agent_session_id after runtime session cleanup = %#v, want %q", got, fakeSessionID) + } +} + +func TestResponsesAdapterDuplicateStartDuringInitializationRejected(t *testing.T) { + received := make(chan struct{}) + release := make(chan struct{}) + foundry := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodPost { + http.NotFound(w, r) + return + } + select { + case <-received: + default: + close(received) + } + <-release + writeJSON(w, finalResponsesMessage()) + })) + t.Cleanup(foundry.Close) + endpoint := foundry.URL + "/agents/test-agent/endpoint/protocols/openai/responses?api-version=v1" + adapter := newTestResponsesAdapter(t, endpoint, nil) + client := newHarnessClient(t, adapter) + request := responsesStartTurnRequest("foundry-duplicate-start") + + firstErr := make(chan error, 1) + go func() { + _, err := client.StartTurn(context.Background(), request) + firstErr <- err + }() + <-received + if _, err := client.StartTurn(context.Background(), request); err == nil || + !strings.Contains(err.Error(), "initialization in progress") { + t.Fatalf("duplicate StartTurn error = %v, want initialization conflict", err) + } + close(release) + if err := <-firstErr; err != nil { + t.Fatalf("initial StartTurn: %v", err) + } +} + +func TestResponsesAdapterEnforcesAdvertisedConcurrentTurnLimit(t *testing.T) { + received := make(chan struct{}) + release := make(chan struct{}) + var releaseOnce sync.Once + releaseFoundry := func() { releaseOnce.Do(func() { close(release) }) } + foundry := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + select { + case <-received: + default: + close(received) + } + <-release + writeJSON(w, finalResponsesMessage()) + })) + t.Cleanup(foundry.Close) + endpoint := foundry.URL + "/agents/test-agent/endpoint/protocols/openai/responses?api-version=v1" + adapter := newTestResponsesAdapter(t, endpoint, nil) + t.Cleanup(releaseFoundry) + client := newHarnessClient(t, adapter) + first := responsesStartTurnRequest("foundry-concurrent-first") + second := responsesStartTurnRequest("foundry-concurrent-second") + second.RuntimeSessionID = first.RuntimeSessionID + + firstErr := make(chan error, 1) + go func() { + _, err := client.StartTurn(context.Background(), first) + firstErr <- err + }() + select { + case <-received: + case <-time.After(time.Second): + t.Fatal("first Foundry request did not arrive") + } + if _, err := client.StartTurn(context.Background(), second); err == nil || + !strings.Contains(err.Error(), "maximum concurrent turns reached") { + t.Fatalf("second concurrent StartTurn error = %v, want admission rejection", err) + } + + releaseFoundry() + select { + case err := <-firstErr: + if err != nil { + t.Fatalf("first StartTurn: %v", err) + } + case <-time.After(time.Second): + t.Fatal("first StartTurn did not finish") + } + if _, err := client.StartTurn(context.Background(), second); err != nil { + t.Fatalf("second StartTurn after first completed: %v", err) + } +} + +func TestResponsesAdapterDiscardsUnusedTurnEnvironment(t *testing.T) { + foundry := newFakeResponses(t, fakeResponsesConfig{scenario: "observed"}) + adapter, server := newTestResponsesAdapterWithServer(t, foundry.endpoint(), nil) + client := newHarnessClient(t, adapter) + request := responsesStartTurnRequest("foundry-env-redaction") + request.Input.Env = []harness.TurnEnvVar{{Name: "SENSITIVE_VALUE", Value: "do-not-retain"}} + + if _, err := client.StartTurn(context.Background(), request); err != nil { + t.Fatalf("StartTurn: %v", err) + } + server.mu.Lock() + turn := server.turns[request.TurnID] + var retainedRequest harness.StartTurnRequest + if turn != nil { + retainedRequest = turn.request + } + server.mu.Unlock() + if turn == nil { + t.Fatal("turn was not retained for duplicate handling") + } + if len(retainedRequest.Input.Env) != 0 { + t.Fatalf("retained turn env = %#v, want discarded credentials", retainedRequest.Input.Env) + } + encoded, err := json.Marshal(retainedRequest) + if err != nil { + t.Fatalf("marshal retained request: %v", err) + } + if bytes.Contains(encoded, []byte("do-not-retain")) { + t.Fatalf("retained request contains discarded environment value: %s", encoded) + } + if _, err := client.StartTurn(context.Background(), request); err != nil { + t.Fatalf("duplicate StartTurn with original environment: %v", err) + } +} + +func TestResponsesAdapterRejectedResponseDoesNotRetainSession(t *testing.T) { + foundry := newFakeResponses(t, fakeResponsesConfig{scenario: "failed_with_session"}) + adapter, server := newTestResponsesAdapterWithServer(t, foundry.endpoint(), nil) + client := newHarnessClient(t, adapter) + first := responsesStartTurnRequest("foundry-rejected-session-first") + + if _, err := client.StartTurn(context.Background(), first); err != nil { + t.Fatalf("first StartTurn: %v", err) + } + frames := streamCurrentFrames(t, client, first.TurnID) + if failed := findFrame(frames, harness.FrameTurnFailed); failed == nil || failed.Failed.Reason != foundryFailedReason { + t.Fatalf("failed frame = %#v, want foundry_failed", failed) + } + server.mu.Lock() + _, retained := server.runtimeSessions[first.RuntimeSessionID] + server.mu.Unlock() + if retained { + t.Fatal("rejected hosted response retained runtime session") + } + + second := responsesStartTurnRequest("foundry-rejected-session-second") + second.RuntimeSessionID = first.RuntimeSessionID + if _, err := client.StartTurn(context.Background(), second); err != nil { + t.Fatalf("second StartTurn: %v", err) + } + if got := requestMap(t, foundry.requestBody(1))["agent_session_id"]; got != nil { + t.Fatalf("second request agent_session_id = %#v, want no rejected session reuse", got) + } +} + +func TestResponsesAdapterRejectedReusedSessionIsQuarantined(t *testing.T) { + foundry := newFakeResponses(t, fakeResponsesConfig{scenario: "failed_with_session"}) + adapter, server := newTestResponsesAdapterWithServer(t, foundry.endpoint(), nil) + client := newHarnessClient(t, adapter) + request := responsesStartTurnRequest("foundry-rejected-reused-session") + server.mu.Lock() + server.runtimeSessions[request.RuntimeSessionID] = foundrySession{ + ID: existingSessionID, + LastSeen: time.Now().UTC(), + } + server.mu.Unlock() + + if _, err := client.StartTurn(context.Background(), request); err != nil { + t.Fatalf("StartTurn: %v", err) + } + if got := requestMap(t, foundry.requestBody(0))["agent_session_id"]; got != existingSessionID { + t.Fatalf("initial agent_session_id = %#v, want existing-session", got) + } + frames := streamCurrentFrames(t, client, request.TurnID) + if failed := findFrame(frames, harness.FrameTurnFailed); failed == nil || failed.Failed.Reason != foundryFailedReason { + t.Fatalf("failed frame = %#v, want foundry_failed", failed) + } + server.mu.Lock() + _, retained := server.runtimeSessions[request.RuntimeSessionID] + _, quarantined := server.quarantinedSessions[request.RuntimeSessionID] + server.mu.Unlock() + if retained || !quarantined { + t.Fatalf("rejected reused session retained=%v quarantined=%v", retained, quarantined) + } + retry := responsesStartTurnRequest("foundry-rejected-reused-session-retry") + retry.RuntimeSessionID = request.RuntimeSessionID + if _, err := client.StartTurn(context.Background(), retry); err == nil || + !strings.Contains(err.Error(), "runtime session unavailable after unconfirmed hosted cancellation") { + t.Fatalf("StartTurn with rejected reused session error = %v", err) + } + if got := foundry.postCount.Load(); got != 1 { + t.Fatalf("hosted post count = %d, want no retry for quarantined session", got) + } +} + +func TestResponsesAdapterCancelPendingReusedSessionQuarantinesIt(t *testing.T) { + foundry := newFakeResponses(t, fakeResponsesConfig{ + scenario: "function_call", + toolName: "support-ticket-lookup", + }) + adapter, server := newTestResponsesAdapterWithServer( + t, + foundry.endpoint(), + []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + ) + client := newHarnessClient(t, adapter) + request := brokeredReadRequest("foundry-cancel-pending-reused-session") + server.mu.Lock() + server.runtimeSessions[request.RuntimeSessionID] = foundrySession{ + ID: existingSessionID, + LastSeen: time.Now().UTC(), + } + server.mu.Unlock() + + if _, err := client.StartTurn(context.Background(), request); err != nil { + t.Fatalf("StartTurn: %v", err) + } + frames := streamCurrentFrames(t, client, request.TurnID) + if findFrame(frames, harness.FrameToolCallRequested) == nil { + t.Fatalf("frames = %#v, want pending tool call", frames) + } + if _, err := client.CancelTurn(context.Background(), cancelRequestForStart(request)); err != nil { + t.Fatalf("CancelTurn: %v", err) + } + server.mu.Lock() + _, retained := server.runtimeSessions[request.RuntimeSessionID] + _, quarantined := server.quarantinedSessions[request.RuntimeSessionID] + server.mu.Unlock() + if retained || !quarantined { + t.Fatalf("cancelled pending reused session retained=%v quarantined=%v", retained, quarantined) + } + retry := brokeredReadRequest("foundry-cancel-pending-reused-session-retry") + retry.RuntimeSessionID = request.RuntimeSessionID + if _, err := client.StartTurn(context.Background(), retry); err == nil || + !strings.Contains(err.Error(), "runtime session unavailable after unconfirmed hosted cancellation") { + t.Fatalf("StartTurn with cancelled pending reused session error = %v", err) + } + if got := foundry.postCount.Load(); got != 1 { + t.Fatalf("hosted post count = %d, want no retry for quarantined session", got) + } +} + +func TestResponsesAdapterCancelDuringInitialPostCancelsHostedRequest(t *testing.T) { + received := make(chan struct{}) + hostedRequestCancelled := make(chan struct{}) + releaseHostedResponse := make(chan struct{}) + var releaseOnce sync.Once + releaseResponse := func() { releaseOnce.Do(func() { close(releaseHostedResponse) }) } + var postCount atomic.Int32 + backendClient := &http.Client{Transport: roundTripFunc(func(r *http.Request) (*http.Response, error) { + if postCount.Add(1) > 1 { + return jsonHTTPResponse(r, finalResponsesMessage()) + } + body, err := ioReadAll(r.Body) + if err != nil { + return nil, err + } + var decoded map[string]any + if err := json.Unmarshal(body, &decoded); err != nil { + return nil, err + } + if got := decoded["agent_session_id"]; got != existingSessionID { + return nil, fmt.Errorf("agent_session_id = %#v, want existing-session", got) + } + close(received) + <-r.Context().Done() + close(hostedRequestCancelled) + // Model a hosted service that accepted the request before client-side + // cancellation and later returns success despite the canceled context. + <-releaseHostedResponse + return jsonHTTPResponseWithSession(r, finalResponsesMessage(), "cancelled-session") + })} + endpoint := "https://foundry.example/agents/test-agent/endpoint/protocols/openai/responses?api-version=v1" + adapter, server := newTestResponsesAdapterWithHTTPClient(t, endpoint, nil, backendClient) + t.Cleanup(releaseResponse) + client := newHarnessClient(t, adapter) + request := responsesStartTurnRequest("foundry-cancel-initial") + server.mu.Lock() + server.runtimeSessions[request.RuntimeSessionID] = foundrySession{ + ID: existingSessionID, + LastSeen: time.Now().UTC(), + } + server.mu.Unlock() + + startErr := make(chan error, 1) + go func() { + _, err := client.StartTurn(context.Background(), request) + startErr <- err + }() + select { + case <-received: + case <-time.After(time.Second): + t.Fatal("initial Foundry request did not arrive") + } + if _, err := client.CancelTurn(context.Background(), cancelRequestForStart(request)); err != nil { + t.Fatalf("CancelTurn: %v", err) + } + select { + case <-hostedRequestCancelled: + case <-time.After(time.Second): + t.Fatal("cancellation did not cancel the initial hosted request context") + } + releaseResponse() + select { + case err := <-startErr: + if err != nil { + t.Fatalf("StartTurn after cancellation: %v", err) + } + case <-time.After(time.Second): + t.Fatal("StartTurn did not finish after the late hosted response") + } + + server.mu.Lock() + turn := server.turns[request.TurnID] + _, sessionRetained := server.runtimeSessions[request.RuntimeSessionID] + _, sessionQuarantined := server.quarantinedSessions[request.RuntimeSessionID] + foundrySessionID := "" + if turn != nil { + foundrySessionID = turn.foundrySessionID + } + server.mu.Unlock() + if turn == nil || !turn.completed || !hasFrameType(turn.frames, harness.FrameTurnCancelled) || + hasFrameType(turn.frames, harness.FrameTurnCompleted) { + t.Fatalf("turn = %#v, want retained terminal cancellation", turn) + } + if sessionRetained || foundrySessionID != "" || !sessionQuarantined { + t.Fatalf( + "cancelled initial post retained session map=%v turnSession=%q quarantined=%v", + sessionRetained, + foundrySessionID, + sessionQuarantined, + ) + } + + quarantined := responsesStartTurnRequest("foundry-quarantined-after-cancel-initial") + quarantined.RuntimeSessionID = request.RuntimeSessionID + if _, err := client.StartTurn(context.Background(), quarantined); err == nil || + !strings.Contains(err.Error(), "runtime session unavailable after unconfirmed hosted cancellation") { + t.Fatalf("StartTurn with quarantined runtime session error = %v", err) + } + second := responsesStartTurnRequest("foundry-after-cancel-initial") + if _, err := client.StartTurn(context.Background(), second); err != nil { + t.Fatalf("StartTurn with a different runtime session after cancellation: %v", err) + } +} + +func TestResponsesAdapterInitialPostSurvivesControlDisconnect(t *testing.T) { + received := make(chan struct{}) + release := make(chan struct{}) + var postCount atomic.Int32 + foundry := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + postCount.Add(1) + select { + case <-received: + default: + close(received) + } + select { + case <-release: + writeJSON(w, finalResponsesMessage()) + case <-r.Context().Done(): + return + } + })) + t.Cleanup(foundry.Close) + endpoint := foundry.URL + "/agents/test-agent/endpoint/protocols/openai/responses?api-version=v1" + adapter, adapterServer := newTestResponsesAdapterWithServer(t, endpoint, nil) + client := newHarnessClient(t, adapter) + request := responsesStartTurnRequest("foundry-control-disconnect") + + ctx, cancel := context.WithCancel(context.Background()) + startErr := make(chan error, 1) + go func() { + _, err := client.StartTurn(ctx, request) + startErr <- err + }() + <-received + cancel() + if err := <-startErr; err == nil { + t.Fatal("StartTurn after control disconnect error = nil, want client cancellation") + } + close(release) + + deadline := time.Now().Add(time.Second) + for { + adapterServer.mu.Lock() + turn := adapterServer.turns[request.TurnID] + completed := turn != nil && turn.completed && !turn.initializing + adapterServer.mu.Unlock() + if completed { + break + } + if time.Now().After(deadline) { + t.Fatal("adapter did not retain and complete the initial hosted response after control disconnect") + } + time.Sleep(10 * time.Millisecond) + } + if _, err := client.StartTurn(context.Background(), request); err != nil { + t.Fatalf("idempotent StartTurn retry: %v", err) + } + if got := postCount.Load(); got != 1 { + t.Fatalf("hosted post count = %d, want 1 after control disconnect and retry", got) + } +} + +func TestResponsesAdapterQuarantineCapacityFailsClosed(t *testing.T) { + server := newServer(config{ + runtimeName: "foundry-responses-test", + adapterBearer: "adapter-auth-value", + endpoint: "https://foundry.example/agents/test-agent/endpoint/protocols/openai/responses?api-version=v1", + foundryAuth: "foundry-auth-value", + requestTimeout: time.Second, + stateRetention: time.Minute, + }, &http.Client{Transport: roundTripFunc(func(*http.Request) (*http.Response, error) { + return nil, errors.New("hosted request should not be attempted after quarantine saturation") + })}) + server.mu.Lock() + for i := range maxQuarantinedRuntimeSessions { + server.quarantineRuntimeSessionLocked(&turnState{request: responsesStartTurnRequest(fmt.Sprintf("quarantine-%d", i))}) + } + if got := len(server.quarantinedSessions); got != maxQuarantinedRuntimeSessions { + server.mu.Unlock() + t.Fatalf("quarantine count = %d, want %d", got, maxQuarantinedRuntimeSessions) + } + if server.quarantineSaturated { + server.mu.Unlock() + t.Fatal("quarantine saturated before exceeding the bounded tombstone capacity") + } + server.quarantineRuntimeSessionLocked(&turnState{request: responsesStartTurnRequest("quarantine-overflow")}) + gotCount := len(server.quarantinedSessions) + saturated := server.quarantineSaturated + server.mu.Unlock() + if gotCount != maxQuarantinedRuntimeSessions || !saturated { + t.Fatalf("quarantine count=%d saturated=%v, want bounded saturated state", gotCount, saturated) + } + + adapter := httptest.NewServer(server.handler()) + t.Cleanup(adapter.Close) + client := newHarnessClient(t, adapter) + afterSaturation := responsesStartTurnRequest("after-quarantine-saturation") + if _, err := client.StartTurn(context.Background(), afterSaturation); err == nil || + !strings.Contains(err.Error(), "quarantine capacity exhausted") { + t.Fatalf("StartTurn after quarantine saturation error = %v", err) + } + response, err := http.Get(adapter.URL + readinessPath) //nolint:gosec // Test-only loopback server. + if err != nil { + t.Fatalf("GET readiness: %v", err) + } + defer response.Body.Close() //nolint:errcheck + if response.StatusCode != http.StatusServiceUnavailable { + t.Fatalf("readiness status = %d, want 503 after quarantine saturation", response.StatusCode) + } +} + +func TestResponsesAdapterFoundryRequestContextDetachesAndUsesEarlierTurnDeadline(t *testing.T) { + server := newServer(config{requestTimeout: time.Second}, &http.Client{Timeout: time.Second}) + parent, cancelParent := context.WithCancel(context.Background()) + cancelParent() + turnDeadline := time.Now().Add(100 * time.Millisecond) + ctx, cancel := server.foundryRequestContext(parent, turnDeadline) + defer cancel() + if err := ctx.Err(); err != nil { + t.Fatalf("detached context inherited control cancellation: %v", err) + } + deadline, ok := ctx.Deadline() + if !ok { + t.Fatal("detached context has no deadline") + } + if delta := deadline.Sub(turnDeadline); delta < -10*time.Millisecond || delta > 10*time.Millisecond { + t.Fatalf("deadline = %v, want turn deadline %v", deadline, turnDeadline) + } + select { + case <-ctx.Done(): + if !errors.Is(ctx.Err(), context.DeadlineExceeded) { + t.Fatalf("detached context error = %v, want deadline exceeded", ctx.Err()) + } + case <-time.After(time.Second): + t.Fatal("detached context did not stop at turn deadline") + } +} + +func TestResponsesAdapterInitialPostHonorsTurnDeadline(t *testing.T) { + foundry := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + time.Sleep(900 * time.Millisecond) + writeJSON(w, finalResponsesMessage()) + })) + t.Cleanup(foundry.Close) + endpoint := foundry.URL + "/agents/test-agent/endpoint/protocols/openai/responses?api-version=v1" + adapter, server := newTestResponsesAdapterWithServer(t, endpoint, nil) + client := newHarnessClient(t, adapter) + request := responsesStartTurnRequest("foundry-initial-deadline") + request.Deadline = time.Now().Add(150 * time.Millisecond) + + started := time.Now() + if _, err := client.StartTurn(context.Background(), request); err != nil { + t.Fatalf("StartTurn should accept terminal deadline failure: %v", err) + } + if elapsed := time.Since(started); elapsed > 800*time.Millisecond { + t.Fatalf("StartTurn elapsed = %v, want turn deadline to beat request timeout", elapsed) + } + frames := streamCurrentFrames(t, client, request.TurnID) + if failed := findFrame(frames, harness.FrameTurnFailed); failed == nil || + failed.Failed.Reason != foundryInitialUnknown { + t.Fatalf("failed frame = %#v, want foundry_initial_unknown", failed) + } + server.mu.Lock() + _, quarantined := server.quarantinedSessions[request.RuntimeSessionID] + server.mu.Unlock() + if !quarantined { + t.Fatal("deadline-uncertain initial request did not quarantine its runtime session") + } + retry := responsesStartTurnRequest("foundry-initial-deadline-retry") + retry.RuntimeSessionID = request.RuntimeSessionID + if _, err := client.StartTurn(context.Background(), retry); err == nil || + !strings.Contains(err.Error(), "runtime session unavailable after unconfirmed hosted cancellation") { + t.Fatalf("StartTurn with deadline-quarantined runtime session error = %v", err) + } +} + +func TestResponsesAdapterRequiresProofForBrokeredReadinessAndStart(t *testing.T) { + foundry := newFakeResponses(t, fakeResponsesConfig{scenario: "observed"}) + server := newServer(config{ + runtimeName: "foundry-responses-test", + adapterBearer: "adapter-auth-value", + endpoint: foundry.endpoint(), + foundryAuth: "foundry-auth-value", + requestTimeout: time.Second, + stateRetention: time.Minute, + brokeredToolClasses: []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + }, &http.Client{Timeout: time.Second}) + adapter := httptest.NewServer(server.handler()) + t.Cleanup(adapter.Close) + + response, err := http.Get(adapter.URL + readinessPath) + if err != nil { + t.Fatalf("GET ready: %v", err) + } + _ = response.Body.Close() + if response.StatusCode != http.StatusServiceUnavailable { + t.Fatalf("ready status = %d, want 503", response.StatusCode) + } + client := newHarnessClient(t, adapter) + capabilities, err := client.Capabilities(context.Background()) + if err != nil { + t.Fatalf("Capabilities: %v", err) + } + if slices.Contains(capabilities.ToolExecutionModes, harness.ToolExecutionModeBrokered) || + capabilities.SupportsContinuation || len(capabilities.BrokeredToolClasses) != 0 { + t.Fatalf("capabilities = %#v, missing proof must disable brokered advertisement", capabilities) + } + request := brokeredReadRequest("foundry-missing-proof") + if _, err := client.StartTurn(context.Background(), request); err == nil || + !strings.Contains(err.Error(), envContinuationProof) { + t.Fatalf("StartTurn missing proof error = %v", err) + } + if got := foundry.postCount.Load(); got != 0 { + t.Fatalf("hosted post count = %d, want no request without continuation proof", got) + } +} + +func TestResponsesAdapterReadyEndpointReflectsConfigReadiness(t *testing.T) { + unready := httptest.NewServer(newServer(config{ + runtimeName: "foundry-responses-test", + adapterBearer: "adapter-auth-value", + endpoint: "https://example.com/agents/a/endpoint/protocols/openai/responses?api-version=v1", + requestTimeout: time.Second, + }, &http.Client{Timeout: time.Second}).handler()) + t.Cleanup(unready.Close) + resp, err := http.Get(unready.URL + readinessPath) //nolint:gosec,noctx // local test server + if err != nil { + t.Fatalf("GET unready: %v", err) + } + defer resp.Body.Close() //nolint:errcheck + if resp.StatusCode != http.StatusServiceUnavailable { + t.Fatalf("unready status = %d, want %d", resp.StatusCode, http.StatusServiceUnavailable) + } + + foundry := newFakeResponses(t, fakeResponsesConfig{scenario: "observed"}) + ready := newTestResponsesAdapter(t, foundry.endpoint(), nil) + readyResp, err := http.Get(ready.URL + readinessPath) //nolint:gosec,noctx // local test server + if err != nil { + t.Fatalf("GET ready: %v", err) + } + defer readyResp.Body.Close() //nolint:errcheck + if readyResp.StatusCode != http.StatusOK { + t.Fatalf("ready status = %d, want %d", readyResp.StatusCode, http.StatusOK) + } +} + +func TestResponsesAdapterHTTPServerHasBoundedReadTimeouts(t *testing.T) { + httpServer := newAdapterHTTPServer(":0", http.NewServeMux()) + if got := httpServer.ReadHeaderTimeout; got != defaultReadHeaderTimeout { + t.Fatalf("ReadHeaderTimeout = %v, want %v", got, defaultReadHeaderTimeout) + } + if got := httpServer.ReadTimeout; got != defaultReadTimeout { + t.Fatalf("ReadTimeout = %v, want %v", got, defaultReadTimeout) + } + if got := httpServer.IdleTimeout; got != defaultIdleTimeout { + t.Fatalf("IdleTimeout = %v, want %v", got, defaultIdleTimeout) + } +} + +func TestResponsesAdapterPassesObservedConformanceByDefault(t *testing.T) { + foundry := newFakeResponses(t, fakeResponsesConfig{scenario: "observed"}) + adapter := newTestResponsesAdapter(t, foundry.endpoint(), nil) + defer adapter.Close() + + result := conformance.Check(context.Background(), conformance.Target{ + BaseURL: adapter.URL, + BearerToken: "adapter-auth-value", + ControlTimeout: 2 * time.Second, + ProbeTurn: true, + RequireAuth: true, + }) + if !result.Passed { + t.Fatalf("observed conformance failed: %s failures=%v", result.Message, result.Failures) + } + caps := result.ObservedCapabilities + if caps == nil { + t.Fatal("ObservedCapabilities = nil") + } + if !reflect.DeepEqual(caps.ToolExecutionModes, []harness.ToolExecutionMode{harness.ToolExecutionModeObserved}) { + t.Fatalf("ToolExecutionModes = %#v, want observed only", caps.ToolExecutionModes) + } + if len(caps.BrokeredToolClasses) != 0 { + t.Fatalf("BrokeredToolClasses = %#v, want none", caps.BrokeredToolClasses) + } + if caps.SupportsContinuation { + t.Fatal("SupportsContinuation = true, want false when no brokered classes are configured") + } +} + +func TestResponsesAdapterPassesBrokeredReadConformance(t *testing.T) { + foundry := newFakeResponses(t, fakeResponsesConfig{scenario: "function_call", toolName: "conformance_read"}) + adapter := newTestResponsesAdapter( + t, + foundry.endpoint(), + []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + ) + defer adapter.Close() + + result := conformance.Check(context.Background(), conformance.Target{ + BaseURL: adapter.URL, + BearerToken: "adapter-auth-value", + ControlTimeout: 2 * time.Second, + ProbeBrokeredRead: true, + RequireAuth: true, + }) + if !result.Passed { + t.Fatalf("brokered read conformance failed: %s failures=%v", result.Message, result.Failures) + } + if foundry.sawRequestLevelTools.Load() { + t.Fatalf("hosted Responses request included request-level tools") + } +} + +func TestResponsesAdapterPassesBrokeredWriteConformance(t *testing.T) { + foundry := newFakeResponses(t, fakeResponsesConfig{scenario: "function_call", toolName: "conformance_write"}) + adapter := newTestResponsesAdapter( + t, + foundry.endpoint(), + []harness.BrokeredToolClass{harness.BrokeredToolClassWrite}, + ) + defer adapter.Close() + + result := conformance.Check(context.Background(), conformance.Target{ + BaseURL: adapter.URL, + BearerToken: "adapter-auth-value", + ControlTimeout: 2 * time.Second, + ProbeBrokeredWrite: true, + RequireAuth: true, + }) + if !result.Passed { + t.Fatalf("brokered write conformance failed: %s failures=%v", result.Message, result.Failures) + } +} + +func TestResponsesAdapterWriteParksUntilDeclinedApprovalContinue(t *testing.T) { + foundry := newFakeResponses(t, fakeResponsesConfig{scenario: "function_call", toolName: "dispatch-work-order"}) + adapter := newTestResponsesAdapter( + t, + foundry.endpoint(), + []harness.BrokeredToolClass{harness.BrokeredToolClassWrite}, + ) + client := newHarnessClient(t, adapter) + request := brokeredWriteRequest("foundry-write-declined") + + if _, err := client.StartTurn(context.Background(), request); err != nil { + t.Fatalf("StartTurn: %v", err) + } + frames := streamCurrentFrames(t, client, request.TurnID) + requested := findFrame(frames, harness.FrameToolCallRequested) + if requested == nil { + t.Fatalf("frames = %#v, want write tool request", frames) + } + if requested.ToolName != "dispatch-work-order" { + t.Fatalf("ToolName = %q, want dispatch-work-order", requested.ToolName) + } + if hasFrameType(frames, harness.FrameToolResultReceived) || hasFrameType(frames, harness.FrameTurnCompleted) { + t.Fatalf("frames = %#v, write should park until Orka continuation", frames) + } + if foundry.postCount.Load() != 1 { + t.Fatalf("hosted post count before approval = %d, want 1", foundry.postCount.Load()) + } + + declined := harness.ContinueTurnRequest{ + Version: harness.ProtocolVersion, + Namespace: request.Namespace, + TaskName: request.TaskName, + SessionName: request.SessionName, + RuntimeSessionID: request.RuntimeSessionID, + TurnID: request.TurnID, + CorrelationID: request.CorrelationID, + ToolResults: []harness.ToolCallResult{toolResultForRequest( + request, + requested.ToolCallID, + false, + nil, + &harness.ErrorInfo{Code: "approval_declined", Message: "human declined"}, + )}, + } + if _, err := client.ContinueTurn(context.Background(), declined); err != nil { + t.Fatalf("ContinueTurn declined approval: %v", err) + } + continuation := requestMap(t, foundry.requestBody(1)) + items, ok := continuation["input"].([]any) + if !ok || len(items) != 1 { + t.Fatalf("continuation input = %#v, want one item", continuation["input"]) + } + item, ok := items[0].(map[string]any) + if !ok { + t.Fatalf("continuation item = %#v, want object", items[0]) + } + wantOutput := `{"approved":false,"error":{"code":"approval_declined","message":"tool call was not approved"}}` + if got := item["output"]; got != wantOutput { + t.Fatalf("declined output = %#v, want %s", got, wantOutput) + } + frames = streamCurrentFrames(t, client, request.TurnID) + toolResult := findFrame(frames, harness.FrameToolResultReceived) + if toolResult == nil || toolResult.Error == nil || toolResult.Error.Code != "approval_declined" { + t.Fatalf("tool result frame = %#v, want approval_declined", toolResult) + } + if !hasFrameType(frames, harness.FrameTurnCompleted) { + t.Fatalf("frames = %#v, want final completion after declined continuation", frames) + } +} + +func TestResponsesAdapterRejectsUnknownToolBeforeOrkaExecution(t *testing.T) { + foundry := newFakeResponses(t, fakeResponsesConfig{scenario: "function_call", toolName: "unknown-tool"}) + adapter := newTestResponsesAdapter( + t, + foundry.endpoint(), + []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + ) + client := newHarnessClient(t, adapter) + request := brokeredReadRequest("foundry-unknown-tool") + + if _, err := client.StartTurn(context.Background(), request); err != nil { + t.Fatalf("StartTurn: %v", err) + } + frames := streamCurrentFrames(t, client, request.TurnID) + if hasFrameType(frames, harness.FrameToolCallRequested) { + t.Fatalf("frames = %#v, should not request Orka execution for an unknown tool", frames) + } + failed := findFrame(frames, harness.FrameTurnFailed) + if failed == nil || !strings.Contains(failed.Failed.Message, "did not expose") { + t.Fatalf("failed frame = %#v, want unknown-tool rejection", failed) + } +} + +func TestResponsesAdapterRejectsMalformedArguments(t *testing.T) { + foundry := newFakeResponses( + t, + fakeResponsesConfig{scenario: "malformed_arguments", toolName: "support-ticket-lookup"}, + ) + adapter := newTestResponsesAdapter( + t, + foundry.endpoint(), + []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + ) + client := newHarnessClient(t, adapter) + request := brokeredReadRequest("foundry-malformed-args") + + if _, err := client.StartTurn(context.Background(), request); err != nil { + t.Fatalf("StartTurn: %v", err) + } + frames := streamCurrentFrames(t, client, request.TurnID) + if hasFrameType(frames, harness.FrameToolCallRequested) { + t.Fatalf("frames = %#v, should not request Orka execution for malformed arguments", frames) + } + failed := findFrame(frames, harness.FrameTurnFailed) + if failed == nil || !strings.Contains(failed.Failed.Message, "arguments") { + t.Fatalf("failed frame = %#v, want malformed arguments rejection", failed) + } +} + +func TestResponsesAdapterMultipleFunctionCallsBufferedAndContinued(t *testing.T) { + foundry := newFakeResponses(t, fakeResponsesConfig{scenario: "multiple_calls", toolName: "support-ticket-lookup"}) + adapter := newTestResponsesAdapter( + t, + foundry.endpoint(), + []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + ) + client := newHarnessClient(t, adapter) + request := brokeredReadRequest("foundry-multiple") + + if _, err := client.StartTurn(context.Background(), request); err != nil { + t.Fatalf("StartTurn: %v", err) + } + var frames []harness.HarnessEventFrame + var requested []string + err := client.StreamFrames(context.Background(), request.TurnID, 0, func(frame harness.HarnessEventFrame) error { + frames = append(frames, frame) + if frame.Type != harness.FrameToolCallRequested { + return nil + } + requested = append(requested, frame.ToolCallID) + result := toolResultForRequest( + request, + frame.ToolCallID, + true, + json.RawMessage(fmt.Sprintf(`{"success":true,"call":%d}`, len(requested))), + nil, + ) + _, continueErr := client.ContinueTurn(context.Background(), harness.ContinueTurnRequest{ + Version: harness.ProtocolVersion, + Namespace: request.Namespace, + TaskName: request.TaskName, + SessionName: request.SessionName, + RuntimeSessionID: request.RuntimeSessionID, + TurnID: request.TurnID, + CorrelationID: request.CorrelationID, + ToolResults: []harness.ToolCallResult{result}, + }) + return continueErr + }) + if err != nil { + t.Fatalf("StreamFrames: %v", err) + } + if !reflect.DeepEqual(requested, []string{"call-1", "call-2"}) { + t.Fatalf("requested tool calls = %#v, want call-1 then call-2", requested) + } + continuation := requestMap(t, foundry.requestBody(1)) + items, ok := continuation["input"].([]any) + if !ok || len(items) != 2 { + t.Fatalf("continuation input = %#v, want two function_call_output items", continuation["input"]) + } + if got := continuation["agent_session_id"]; got != fakeSessionID { + t.Fatalf("agent_session_id = %#v, want %q", got, fakeSessionID) + } + if !hasFrameType(frames, harness.FrameToolResultReceived) || !hasFrameType(frames, harness.FrameTurnCompleted) { + t.Fatalf("frames = %#v, want tool results and completion", frames) + } +} + +func TestResponsesAdapterDuplicateContinueIsIdempotentAndConflictsReject(t *testing.T) { + foundry := newFakeResponses(t, fakeResponsesConfig{scenario: "function_call", toolName: "support-ticket-lookup"}) + adapter := newTestResponsesAdapter( + t, + foundry.endpoint(), + []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + ) + client := newHarnessClient(t, adapter) + request := brokeredReadRequest("foundry-duplicate") + + if _, err := client.StartTurn(context.Background(), request); err != nil { + t.Fatalf("StartTurn: %v", err) + } + frames := streamCurrentFrames(t, client, request.TurnID) + requested := findFrame(frames, harness.FrameToolCallRequested) + if requested == nil { + t.Fatalf("frames = %#v, want tool request", frames) + } + continueRequest := goldenContinueRequest(request, requested.ToolCallID, json.RawMessage(`{"success":true}`)) + if _, err := client.ContinueTurn(context.Background(), continueRequest); err != nil { + t.Fatalf("ContinueTurn first: %v", err) + } + if _, err := client.ContinueTurn(context.Background(), continueRequest); err != nil { + t.Fatalf("ContinueTurn duplicate: %v", err) + } + if foundry.postCount.Load() != 2 { + t.Fatalf("hosted post count after duplicate = %d, want 2", foundry.postCount.Load()) + } + conflicting := goldenContinueRequest(request, requested.ToolCallID, json.RawMessage(`{"success":false}`)) + if _, err := client.ContinueTurn(context.Background(), conflicting); err == nil { + t.Fatalf("conflicting duplicate continue succeeded, want conflict") + } +} + +func TestResponsesAdapterSendsBrokeredContinuationProof(t *testing.T) { + foundry := newFakeResponses(t, fakeResponsesConfig{ + scenario: "function_call", + toolName: "support-ticket-lookup", + requiredProof: testContinuationProof, + }) + s := newServer(config{ + runtimeName: "foundry-responses-test", + adapterBearer: "adapter-auth-value", + endpoint: foundry.endpoint(), + foundryAuth: "foundry-auth-value", + requestTimeout: time.Second, + stateRetention: time.Minute, + continuationProof: testContinuationProof, + brokeredToolClasses: []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + }, &http.Client{Timeout: time.Second}) + adapter := httptest.NewServer(s.handler()) + t.Cleanup(adapter.Close) + client := newHarnessClient(t, adapter) + request := brokeredReadRequest("foundry-continuation-proof") + + if _, err := client.StartTurn(context.Background(), request); err != nil { + t.Fatalf("StartTurn: %v", err) + } + frames := streamCurrentFrames(t, client, request.TurnID) + requested := findFrame(frames, harness.FrameToolCallRequested) + if requested == nil { + t.Fatalf("frames = %#v, want tool request", frames) + } + continueRequest := goldenContinueRequest(request, requested.ToolCallID, json.RawMessage(`{"success":true}`)) + if _, err := client.ContinueTurn(context.Background(), continueRequest); err != nil { + t.Fatalf("ContinueTurn: %v", err) + } + if got := foundry.requestHeader(1).Get("X-AgentKit-Brokered-Continuation-Proof"); got != testContinuationProof { + t.Fatalf("continuation proof header = %q, want proof-for-test", got) + } + if got := requestMap(t, foundry.requestBody(1))["brokered_continuation_proof"]; got != testContinuationProof { + t.Fatalf("continuation proof body = %#v, want proof-for-test", got) + } + if got := foundry.requestHeader(0).Get("X-AgentKit-Brokered-Continuation-Proof"); got != "" { + t.Fatalf("initial proof header = %q, want empty", got) + } + if _, ok := requestMap(t, foundry.requestBody(0))["brokered_continuation_proof"]; ok { + t.Fatal("initial request included brokered continuation proof body") + } +} + +func TestResponsesAdapterCancelDuringHostedContinuationCancelsRequest(t *testing.T) { + continuationReceived := make(chan struct{}) + hostedRequestCancelled := make(chan struct{}) + releaseHostedResponse := make(chan struct{}) + var releaseOnce sync.Once + releaseResponse := func() { releaseOnce.Do(func() { close(releaseHostedResponse) }) } + backendClient := &http.Client{Transport: roundTripFunc(func(r *http.Request) (*http.Response, error) { + body, err := ioReadAll(r.Body) + if err != nil { + return nil, err + } + var decoded map[string]any + if err := json.Unmarshal(body, &decoded); err != nil { + return nil, err + } + if _, continuing := decoded["previous_response_id"]; !continuing { + return jsonHTTPResponseWithSession( + r, + functionCallResponse("support-ticket-lookup"), + existingSessionID, + ) + } + close(continuationReceived) + <-r.Context().Done() + close(hostedRequestCancelled) + // Model a hosted continuation that ignores client cancellation and + // eventually reports success with a replacement session identifier. + <-releaseHostedResponse + return jsonHTTPResponseWithSession(r, finalResponsesMessage(), "session-after-cancel") + })} + endpoint := "https://foundry.example/agents/test-agent/endpoint/protocols/openai/responses?api-version=v1" + adapter, server := newTestResponsesAdapterWithHTTPClient( + t, + endpoint, + []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + backendClient, + ) + t.Cleanup(releaseResponse) + client := newHarnessClient(t, adapter) + request := brokeredReadRequest("foundry-cancel-continuation") + server.mu.Lock() + server.runtimeSessions[request.RuntimeSessionID] = foundrySession{ + ID: existingSessionID, + LastSeen: time.Now().UTC(), + } + server.mu.Unlock() + if _, err := client.StartTurn(context.Background(), request); err != nil { + t.Fatalf("StartTurn: %v", err) + } + frames := streamCurrentFrames(t, client, request.TurnID) + requested := findFrame(frames, harness.FrameToolCallRequested) + if requested == nil { + t.Fatalf("frames = %#v, want tool request", frames) + } + server.mu.Lock() + publishedBeforeCompletion := server.runtimeSessions[request.RuntimeSessionID] + server.mu.Unlock() + if publishedBeforeCompletion.ID != existingSessionID { + t.Fatalf("runtime session before continuation = %#v, want existing session", publishedBeforeCompletion) + } + + continueRequest := goldenContinueRequest(request, requested.ToolCallID, json.RawMessage(`{"success":true}`)) + continueErr := make(chan error, 1) + go func() { + _, err := client.ContinueTurn(context.Background(), continueRequest) + continueErr <- err + }() + select { + case <-continuationReceived: + case <-time.After(time.Second): + t.Fatal("hosted continuation did not arrive") + } + if _, err := client.CancelTurn(context.Background(), cancelRequestForStart(request)); err != nil { + t.Fatalf("CancelTurn during continuation: %v", err) + } + select { + case <-hostedRequestCancelled: + case <-time.After(time.Second): + t.Fatal("cancellation did not cancel the hosted continuation request context") + } + server.mu.Lock() + _, publishedAfterCancellation := server.runtimeSessions[request.RuntimeSessionID] + _, sessionQuarantined := server.quarantinedSessions[request.RuntimeSessionID] + server.mu.Unlock() + if publishedAfterCancellation || !sessionQuarantined { + t.Fatalf( + "runtime session after cancellation published=%v quarantined=%v", + publishedAfterCancellation, + sessionQuarantined, + ) + } + releaseResponse() + select { + case err := <-continueErr: + if err == nil || !strings.Contains(err.Error(), "turn completed while hosted continuation was in flight") { + t.Fatalf("ContinueTurn after cancellation error = %v", err) + } + case <-time.After(time.Second): + t.Fatal("ContinueTurn did not finish after the late hosted response") + } + + quarantined := responsesStartTurnRequest("foundry-quarantined-after-cancel-continuation") + quarantined.RuntimeSessionID = request.RuntimeSessionID + if _, err := client.StartTurn(context.Background(), quarantined); err == nil || + !strings.Contains(err.Error(), "runtime session unavailable after unconfirmed hosted cancellation") { + t.Fatalf("StartTurn with quarantined runtime session error = %v", err) + } + second := responsesStartTurnRequest("foundry-after-cancel-continuation") + if _, err := client.StartTurn(context.Background(), second); err != nil { + t.Fatalf("StartTurn with a different runtime session after cancellation: %v", err) + } + frames = streamCurrentFrames(t, client, request.TurnID) + if !hasFrameType(frames, harness.FrameTurnCancelled) || hasFrameType(frames, harness.FrameTurnCompleted) { + t.Fatalf("frames = %#v, want cancellation to win continuation race", frames) + } + server.mu.Lock() + _, publishedAfterLateResponse := server.runtimeSessions[request.RuntimeSessionID] + server.mu.Unlock() + if publishedAfterLateResponse { + t.Fatal("late continuation response restored the cancelled runtime session") + } +} + +func TestResponsesAdapterContinuationHonorsOriginalTurnDeadline(t *testing.T) { + foundry := newFakeResponses(t, fakeResponsesConfig{scenario: "function_call", toolName: "support-ticket-lookup"}) + adapter := newTestResponsesAdapter( + t, + foundry.endpoint(), + []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + ) + client := newHarnessClient(t, adapter) + request := brokeredReadRequest("foundry-continuation-deadline") + request.Deadline = time.Now().Add(300 * time.Millisecond) + + if _, err := client.StartTurn(context.Background(), request); err != nil { + t.Fatalf("StartTurn: %v", err) + } + frames := streamCurrentFrames(t, client, request.TurnID) + requested := findFrame(frames, harness.FrameToolCallRequested) + if requested == nil { + t.Fatalf("frames = %#v, want tool request", frames) + } + if wait := time.Until(request.Deadline); wait > 0 { + time.Sleep(wait + 10*time.Millisecond) + } + continueRequest := goldenContinueRequest(request, requested.ToolCallID, json.RawMessage(`{"success":true}`)) + if _, err := client.ContinueTurn(context.Background(), continueRequest); err == nil { + t.Fatal("ContinueTurn after original deadline succeeded, want failure") + } + if got := foundry.postCount.Load(); got != 1 { + t.Fatalf("hosted post count = %d, want no continuation after turn deadline", got) + } + frames = streamCurrentFrames(t, client, request.TurnID) + failed := findFrame(frames, harness.FrameTurnFailed) + if failed == nil || failed.Failed == nil || failed.Failed.Reason != "foundry_continuation_unknown" { + t.Fatalf("failed frame = %#v, want foundry_continuation_unknown", failed) + } +} + +func TestResponsesAdapterRejectedContinueDoesNotBufferPartialResults(t *testing.T) { + server := newServer(config{ + runtimeName: "test", + adapterBearer: "adapter-auth-value", + endpoint: "http://127.0.0.1/agents/test-agent/endpoint/protocols/openai/responses?api-version=v1", + foundryAuth: "foundry-auth-value", + requestTimeout: time.Second, + stateRetention: time.Minute, + brokeredToolClasses: []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + }, &http.Client{Timeout: time.Second}) + request := brokeredReadRequest("foundry-partial-reject") + valid := toolResultForRequest(request, "call-1", true, json.RawMessage(`{"success":true}`), nil) + unknown := toolResultForRequest(request, "call-missing", true, json.RawMessage(`{"success":true}`), nil) + turn := &turnState{ + request: request, + pendingTools: map[string]string{"call-1": "support-ticket-lookup"}, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{}, + } + if _, err := server.recordContinueResults(turn, []harness.ToolCallResult{valid, unknown}); err == nil { + t.Fatalf("recordContinueResults succeeded, want unknown tool result error") + } + if len(turn.bufferedResults) != 0 || len(turn.bufferedDigests) != 0 { + t.Fatalf("buffered state = %#v/%#v, want no partial buffering", turn.bufferedResults, turn.bufferedDigests) + } +} + +func TestResponsesAdapterAlreadySubmittedContinueDoesNotResubmit(t *testing.T) { + server := newServer(config{ + runtimeName: "test", + adapterBearer: "adapter-auth-value", + endpoint: "http://127.0.0.1/agents/test-agent/endpoint/protocols/openai/responses?api-version=v1", + foundryAuth: "foundry-auth-value", + requestTimeout: time.Second, + stateRetention: time.Minute, + brokeredToolClasses: []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + }, &http.Client{Timeout: time.Second}) + request := brokeredReadRequest("foundry-already-submitted") + result := toolResultForRequest(request, "call-1", true, json.RawMessage(`{"success":true}`), nil) + payload, err := canonicalToolResultOutput(result) + if err != nil { + t.Fatalf("canonicalToolResultOutput: %v", err) + } + turn := &turnState{ + request: request, + pendingTools: map[string]string{"call-1": "support-ticket-lookup"}, + bufferedResults: map[string]harness.ToolCallResult{"call-1": result}, + bufferedDigests: map[string]toolResultDigest{"call-1": digestToolResultPayload(payload)}, + submittedDigests: map[string]toolResultDigest{"call-1": digestToolResultPayload(payload)}, + } + toSubmit, err := server.recordContinueResults(turn, []harness.ToolCallResult{result}) + if err != nil { + t.Fatalf("recordContinueResults: %v", err) + } + if toSubmit != nil { + t.Fatalf("toSubmit = %#v, want nil for already submitted duplicate", toSubmit) + } +} + +func TestResponsesAdapterContinuesToolExecutionFailurePayload(t *testing.T) { + foundry := newFakeResponses(t, fakeResponsesConfig{scenario: "function_call", toolName: "support-ticket-lookup"}) + adapter := newTestResponsesAdapter( + t, + foundry.endpoint(), + []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + ) + client := newHarnessClient(t, adapter) + request := brokeredReadRequest("foundry-tool-failure") + + if _, err := client.StartTurn(context.Background(), request); err != nil { + t.Fatalf("StartTurn: %v", err) + } + frames := streamCurrentFrames(t, client, request.TurnID) + requested := findFrame(frames, harness.FrameToolCallRequested) + if requested == nil { + t.Fatalf("frames = %#v, want tool request", frames) + } + failure := harness.ContinueTurnRequest{ + Version: harness.ProtocolVersion, + Namespace: request.Namespace, + TaskName: request.TaskName, + SessionName: request.SessionName, + RuntimeSessionID: request.RuntimeSessionID, + TurnID: request.TurnID, + CorrelationID: request.CorrelationID, + ToolResults: []harness.ToolCallResult{toolResultForRequest( + request, + requested.ToolCallID, + true, + nil, + &harness.ErrorInfo{Code: "tool_execution_failed", Message: "downstream failed"}, + )}, + } + if _, err := client.ContinueTurn(context.Background(), failure); err != nil { + t.Fatalf("ContinueTurn tool failure: %v", err) + } + continuation := requestMap(t, foundry.requestBody(1)) + items, ok := continuation["input"].([]any) + if !ok || len(items) != 1 { + t.Fatalf("continuation input = %#v, want one item", continuation["input"]) + } + item, ok := items[0].(map[string]any) + if !ok { + t.Fatalf("continuation item = %#v, want object", items[0]) + } + wantOutput := `{"approved":false,"error":{"code":"tool_execution_failed","message":"tool execution failed"}}` + if got := item["output"]; got != wantOutput { + t.Fatalf("failure output = %#v, want %s", got, wantOutput) + } + frames = streamCurrentFrames(t, client, request.TurnID) + toolResult := findFrame(frames, harness.FrameToolResultReceived) + if toolResult == nil || toolResult.Error == nil || toolResult.Error.Code != "tool_execution_failed" { + t.Fatalf("tool result frame = %#v, want tool_execution_failed", toolResult) + } + if !hasFrameType(frames, harness.FrameTurnCompleted) { + t.Fatalf("frames = %#v, want final completion after tool failure continuation", frames) + } +} + +func TestResponsesAdapterContinuationFailureFailsClosedWithoutDuplicatePost(t *testing.T) { + foundry := newFakeResponses(t, fakeResponsesConfig{ + scenario: "function_call", + toolName: "support-ticket-lookup", + continuationStatus: http.StatusInternalServerError, + }) + adapter, server := newTestResponsesAdapterWithServer( + t, + foundry.endpoint(), + []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + ) + client := newHarnessClient(t, adapter) + request := brokeredReadRequest("foundry-continuation-failure") + + if _, err := client.StartTurn(context.Background(), request); err != nil { + t.Fatalf("StartTurn: %v", err) + } + frames := streamCurrentFrames(t, client, request.TurnID) + requested := findFrame(frames, harness.FrameToolCallRequested) + if requested == nil { + t.Fatalf("frames = %#v, want tool request", frames) + } + continueRequest := goldenContinueRequest(request, requested.ToolCallID, json.RawMessage(`{"success":true}`)) + _, continueErr := client.ContinueTurn(context.Background(), continueRequest) + if continueErr == nil { + t.Fatalf("ContinueTurn succeeded, want hosted continuation failure") + } + if strings.Contains(continueErr.Error(), foundry.URL) || strings.Contains(continueErr.Error(), "HTTP 500") { + t.Fatalf("ContinueTurn error leaked upstream detail: %v", continueErr) + } + if foundry.postCount.Load() != 2 { + t.Fatalf("hosted post count after failed continue = %d, want 2", foundry.postCount.Load()) + } + if _, err := client.ContinueTurn(context.Background(), continueRequest); err != nil { + t.Fatalf("duplicate ContinueTurn after fail-closed terminal state: %v", err) + } + if foundry.postCount.Load() != 2 { + t.Fatalf("hosted post count after duplicate = %d, want no second continuation", foundry.postCount.Load()) + } + frames = streamCurrentFrames(t, client, request.TurnID) + failed := findFrame(frames, harness.FrameTurnFailed) + if failed == nil || failed.Failed.Reason != "foundry_continuation_unknown" { + t.Fatalf("failed frame = %#v, want fail-closed continuation failure", failed) + } + if strings.Contains(failed.Failed.Message, foundry.URL) || + strings.Contains(failed.Failed.Message, "HTTP 500") { + t.Fatalf("failed frame leaked upstream detail: %#v", failed.Failed) + } + server.mu.Lock() + _, quarantined := server.quarantinedSessions[request.RuntimeSessionID] + server.mu.Unlock() + if !quarantined { + t.Fatal("uncertain continuation failure did not quarantine its runtime session") + } + retry := responsesStartTurnRequest("foundry-continuation-failure-retry") + retry.RuntimeSessionID = request.RuntimeSessionID + if _, err := client.StartTurn(context.Background(), retry); err == nil || + !strings.Contains(err.Error(), "runtime session unavailable after unconfirmed hosted cancellation") { + t.Fatalf("StartTurn with continuation-quarantined runtime session error = %v", err) + } +} + +func TestResponsesAdapterHostedErrorDoesNotExposeUpstreamDiagnostics(t *testing.T) { + server := newServer(config{}, &http.Client{Timeout: time.Second}) + request := responsesStartTurnRequest("foundry-safe-hosted-error") + turn := &turnState{request: request, frameUpdates: make(chan struct{})} + server.appendFrameLocked(turn, harness.FrameTurnStarted, "foundry hosted response started") + secret := "proof-that-must-not-reach-task-events" + server.handleResponsesResponse(turn, responsesResponse{ + ID: "resp-error", + Status: "failed", + Error: &responsesError{ + Code: secret, + Message: "upstream echoed brokered_continuation_proof=" + secret, + }, + }) + failed := findFrame(turn.frames, harness.FrameTurnFailed) + if failed == nil || failed.Failed == nil || failed.Error == nil { + t.Fatalf("failed frame = %#v, want safe terminal failure", failed) + } + if failed.Failed.Reason != "foundry_response_error" || + failed.Failed.Message != "Foundry hosted Responses returned an error" { + t.Fatalf("failed frame = %#v, want fixed safe upstream error", failed) + } + payload, err := json.Marshal(failed) + if err != nil { + t.Fatalf("marshal failed frame: %v", err) + } + if strings.Contains(string(payload), secret) { + t.Fatalf("failed frame leaked upstream diagnostics: %s", payload) + } +} + +func TestResponsesRepeatedSubmittedFunctionCallFailsTurn(t *testing.T) { + server := newServer(config{ + runtimeName: "test", + adapterBearer: "adapter-auth-value", + endpoint: "http://127.0.0.1/agents/test-agent/endpoint/protocols/openai/responses?api-version=v1", + foundryAuth: "foundry-auth-value", + requestTimeout: time.Second, + stateRetention: time.Minute, + brokeredToolClasses: []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + }, &http.Client{Timeout: time.Second}) + request := brokeredReadRequest("foundry-repeated-call") + turn := &turnState{ + request: request, + pendingTools: map[string]string{}, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{"call-1": digestToolResultPayload(`{"approved":true}`)}, + } + server.appendFrameLocked(turn, harness.FrameTurnStarted, "foundry hosted response started") + server.handleResponsesResponse(turn, responsesResponse{ + ID: "resp-repeat", + Status: "completed", + Output: []responsesOutput{{ + Type: "function_call", + CallID: "call-1", + Name: "support-ticket-lookup", + Arguments: json.RawMessage(`{"incident":"inc-1"}`), + }}, + }) + failed := findFrame(turn.frames, harness.FrameTurnFailed) + if failed == nil || failed.Failed.Reason != "foundry_repeated_function_call" { + t.Fatalf("failed frame = %#v, want foundry_repeated_function_call", failed) + } +} + +func TestResponsesMixedRepeatedFunctionCallFailsTurn(t *testing.T) { + server := newServer(config{ + runtimeName: "test", + adapterBearer: "adapter-auth-value", + endpoint: "http://127.0.0.1/agents/test-agent/endpoint/protocols/openai/responses?api-version=v1", + foundryAuth: "foundry-auth-value", + requestTimeout: time.Second, + stateRetention: time.Minute, + brokeredToolClasses: []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + }, &http.Client{Timeout: time.Second}) + request := brokeredReadRequest("foundry-mixed-repeated-call") + turn := &turnState{ + request: request, + pendingTools: map[string]string{"call-1": "support-ticket-lookup"}, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{}, + } + server.appendFrameLocked(turn, harness.FrameTurnStarted, "foundry hosted response started") + server.handleResponsesResponse(turn, responsesResponse{ + ID: "resp-repeat", + Status: "completed", + Output: []responsesOutput{ + { + Type: "function_call", + CallID: "call-2", + Name: "support-ticket-lookup", + Arguments: json.RawMessage(`{"incident":"inc-2"}`), + }, + { + Type: "function_call", + CallID: "call-1", + Name: "support-ticket-lookup", + Arguments: json.RawMessage(`{"incident":"inc-1"}`), + }, + }, + }) + failed := findFrame(turn.frames, harness.FrameTurnFailed) + if failed == nil || failed.Failed.Reason != "foundry_repeated_function_call" { + t.Fatalf("failed frame = %#v, want foundry_repeated_function_call", failed) + } + if _, exists := turn.pendingTools["call-2"]; exists { + t.Fatal("new call was accepted after repeated pending call") + } +} + +func TestResponsesRecordContinueResultsRejectsTerminalTurn(t *testing.T) { + server := newServer(config{}, &http.Client{Timeout: time.Second}) + request := brokeredReadRequest("foundry-terminal-record") + turn := &turnState{ + request: request, + pendingTools: map[string]string{"call-1": "support-ticket-lookup"}, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{}, + completed: true, + } + result := toolResultForRequest(request, "call-1", true, json.RawMessage(`{"success":true}`), nil) + if _, err := server.recordContinueResults(turn, []harness.ToolCallResult{result}); err == nil || + !strings.Contains(err.Error(), "already terminal") { + t.Fatalf("recordContinueResults terminal error = %v", err) + } + if len(turn.submittedDigests) != 0 || len(turn.bufferedResults) != 0 { + t.Fatalf("terminal result mutation = %#v/%#v", turn.submittedDigests, turn.bufferedResults) + } +} + +func TestResponsesAdapterAlreadySubmittedPendingResultIsNoop(t *testing.T) { + server := newServer(config{}, &http.Client{Timeout: time.Second}) + request := brokeredReadRequest("foundry-submitted-noop") + turn := &turnState{ + request: request, + pendingTools: map[string]string{"call-1": "support-ticket-lookup"}, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{}, + } + result := toolResultForRequest(request, "call-1", true, json.RawMessage(`{"success":true}`), nil) + payload, err := canonicalToolResultOutput(result) + if err != nil { + t.Fatalf("canonicalToolResultOutput: %v", err) + } + turn.submittedDigests["call-1"] = digestToolResultPayload(payload) + + toSubmit, err := server.recordContinueResults(turn, []harness.ToolCallResult{result}) + if err != nil { + t.Fatalf("recordContinueResults: %v", err) + } + if len(toSubmit) != 0 { + t.Fatalf("toSubmit = %#v, want duplicate submitted result to be a no-op", toSubmit) + } +} + +func TestResponsesAdapterBrokeredMaxTurnIsUnknown(t *testing.T) { + foundry := newFakeResponses(t, fakeResponsesConfig{scenario: "observed"}) + s := newServer(config{ + runtimeName: "foundry-responses-test", + adapterBearer: "adapter-auth-value", + endpoint: foundry.endpoint(), + foundryAuth: "foundry-auth-value", + requestTimeout: 2 * time.Second, + stateRetention: time.Minute, + continuationProof: testContinuationProof, + brokeredToolClasses: []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + }, &http.Client{Timeout: time.Second}) + adapter := httptest.NewServer(s.handler()) + t.Cleanup(adapter.Close) + client := newHarnessClient(t, adapter) + + caps, err := client.Capabilities(context.Background()) + if err != nil { + t.Fatalf("Capabilities: %v", err) + } + if caps.MaxTurnSeconds != 0 { + t.Fatalf("MaxTurnSeconds = %d, want unknown ceiling for brokered turns", caps.MaxTurnSeconds) + } +} + +func TestResponsesAdapterStateLossContinueFailsSafely(t *testing.T) { + foundry := newFakeResponses(t, fakeResponsesConfig{scenario: "function_call", toolName: "support-ticket-lookup"}) + adapter := newTestResponsesAdapter( + t, + foundry.endpoint(), + []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + ) + client := newHarnessClient(t, adapter) + request := brokeredReadRequest("foundry-state-loss") + if _, err := client.StartTurn(context.Background(), request); err != nil { + t.Fatalf("StartTurn: %v", err) + } + frames := streamCurrentFrames(t, client, request.TurnID) + requested := findFrame(frames, harness.FrameToolCallRequested) + if requested == nil { + t.Fatalf("frames = %#v, want tool request", frames) + } + + restarted := newTestResponsesAdapter( + t, + foundry.endpoint(), + []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + ) + restartedClient, err := harness.NewClient( + restarted.URL, + harness.WithBearerToken("adapter-auth-value"), + harness.WithControlTimeout(2*time.Second), + ) + if err != nil { + t.Fatalf("NewClient restarted: %v", err) + } + _, err = restartedClient.ContinueTurn( + context.Background(), + goldenContinueRequest(request, requested.ToolCallID, json.RawMessage(`{"success":true}`)), + ) + if err == nil || !strings.Contains(err.Error(), "410") { + t.Fatalf("restart continue error = %v, want non-retryable gone state", err) + } + if foundry.postCount.Load() != 1 { + t.Fatalf( + "hosted post count after state-loss continue = %d, want no duplicate continuation", + foundry.postCount.Load(), + ) + } +} + +func TestPostResponsesRejectsOversizedAndTrailingBodies(t *testing.T) { + valid, err := json.Marshal(finalResponsesMessage()) + if err != nil { + t.Fatalf("marshal valid response: %v", err) + } + tests := []struct { + name string + body []byte + wantErr string + }{ + { + name: "oversized", + body: append(append([]byte(nil), valid...), bytes.Repeat([]byte(" "), maxFoundryBodyBytes+1-len(valid))...), + wantErr: "exceeded", + }, + { + name: "trailing data", + body: append(append([]byte(nil), valid...), []byte("trailing")...), + wantErr: "decode", + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write(tt.body) + })) + t.Cleanup(upstream.Close) + server := newServer(config{ + endpoint: upstream.URL + "/agents/test/endpoint/protocols/openai/responses?api-version=v1", + foundryAuth: "foundry-auth-value", + requestTimeout: time.Second, + }, &http.Client{Timeout: time.Second}) + var response responsesResponse + _, err := server.postResponses( + context.Background(), + responsesRequest{Input: "test"}, + &response, + ) + if err == nil || !strings.Contains(err.Error(), tt.wantErr) { + t.Fatalf("postResponses error = %v, want %q", err, tt.wantErr) + } + }) + } +} + +func TestResponsesMessageTextPreservesRefusal(t *testing.T) { + got := responsesMessageText([]responsesOutput{{ + Type: "message", + Content: []any{map[string]any{ + "type": "refusal", + "refusal": "I cannot perform that request.", + }}, + }}) + if got != "I cannot perform that request." { + t.Fatalf("responsesMessageText refusal = %q", got) + } +} + +func TestResponsesAPIVersionDefaultsToSDKValue(t *testing.T) { + t.Setenv(envAPIVersion, "") + if got := loadConfig().apiVersion; got != defaultAPIVersion { + t.Fatalf("loadConfig apiVersion = %q, want %q", got, defaultAPIVersion) + } + t.Setenv(envAPIVersion, "2025-11-15-preview") + if got := loadConfig().apiVersion; got != "2025-11-15-preview" { + t.Fatalf("loadConfig apiVersion override = %q", got) + } +} + +func TestResponsesAdapterRejectsConsumedTurnAfterCleanup(t *testing.T) { + foundry := newFakeResponses(t, fakeResponsesConfig{scenario: "platform_error"}) + server := newServer(config{ + runtimeName: "foundry-responses-test", + adapterBearer: "adapter-auth-value", + endpoint: foundry.endpoint(), + foundryAuth: "foundry-auth-value", + requestTimeout: time.Second, + stateRetention: 50 * time.Millisecond, + }, &http.Client{Timeout: time.Second}) + adapter := httptest.NewServer(server.handler()) + t.Cleanup(adapter.Close) + client := newHarnessClient(t, adapter) + request := responsesStartTurnRequest("foundry-consumed-turn") + + if _, err := client.StartTurn(context.Background(), request); err != nil { + t.Fatalf("StartTurn: %v", err) + } + server.mu.Lock() + _, retainedBeforeCleanup := server.turns[request.TurnID] + _, consumedBeforeCleanup := server.consumedTurns[request.TurnID] + server.mu.Unlock() + if !retainedBeforeCleanup || consumedBeforeCleanup { + t.Fatalf( + "pre-cleanup state retained=%v consumed=%v, want retained state to own admission", + retainedBeforeCleanup, + consumedBeforeCleanup, + ) + } + deadline := time.Now().Add(time.Second) + for { + server.mu.Lock() + _, retained := server.turns[request.TurnID] + _, consumed := server.consumedTurns[request.TurnID] + server.mu.Unlock() + if !retained && consumed { + break + } + if time.Now().After(deadline) { + t.Fatal("terminal turn was not evicted into the consumed-turn tombstone") + } + time.Sleep(5 * time.Millisecond) + } + if err := client.StreamFrames(context.Background(), request.TurnID, 0, func(harness.HarnessEventFrame) error { + return nil + }); err == nil || !strings.Contains(err.Error(), "410") { + t.Fatalf("expired terminal stream error = %v, want non-retryable 410", err) + } + if _, err := client.StartTurn(context.Background(), request); err == nil || + !strings.Contains(err.Error(), "turn already completed") { + t.Fatalf("retry after cleanup error = %v, want consumed-turn conflict", err) + } + if got := foundry.postCount.Load(); got != 1 { + t.Fatalf("hosted post count after consumed retry = %d, want 1", got) + } +} + +func TestResponsesConsumedTurnTombstonesAreBounded(t *testing.T) { + server := newServer(config{}, nil) + server.mu.Lock() + for i := 0; i <= maxConsumedTurnIDs; i++ { + server.markTurnConsumedLocked(harness.HarnessTurnID(fmt.Sprintf("turn-%d", i))) + } + _, oldestRetained := server.consumedTurns["turn-0"] + _, newestRetained := server.consumedTurns[harness.HarnessTurnID(fmt.Sprintf("turn-%d", maxConsumedTurnIDs))] + count := len(server.consumedTurns) + server.mu.Unlock() + if count != maxConsumedTurnIDs || oldestRetained || !newestRetained { + t.Fatalf( + "consumed tombstone count=%d oldest=%v newest=%v, want bounded FIFO", + count, + oldestRetained, + newestRetained, + ) + } +} + +func TestResponsesAdapterValidatesCancelIdentityBeforeMutation(t *testing.T) { + foundry := newFakeResponses(t, fakeResponsesConfig{scenario: "function_call", toolName: "support-ticket-lookup"}) + adapter, server := newTestResponsesAdapterWithServer( + t, + foundry.endpoint(), + []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + ) + client := newHarnessClient(t, adapter) + request := brokeredReadRequest("foundry-cancel-identity") + if _, err := client.StartTurn(context.Background(), request); err != nil { + t.Fatalf("StartTurn: %v", err) + } + + cancelPath, err := harness.CancelTurnPath(request.TurnID) + if err != nil { + t.Fatalf("CancelTurnPath: %v", err) + } + malformed, err := http.NewRequest(http.MethodPost, adapter.URL+cancelPath, strings.NewReader("{")) + if err != nil { + t.Fatalf("NewRequest: %v", err) + } + malformed.Header.Set("Authorization", "Bearer adapter-auth-value") + response, err := http.DefaultClient.Do(malformed) + if err != nil { + t.Fatalf("malformed cancel request: %v", err) + } + _ = response.Body.Close() + if response.StatusCode != http.StatusBadRequest { + t.Fatalf("malformed cancel status = %d, want 400", response.StatusCode) + } + + mismatched := cancelRequestForStart(request) + mismatched.CorrelationID = "other-correlation" + if _, err := client.CancelTurn(context.Background(), mismatched); err == nil || + !strings.Contains(err.Error(), "does not match started turn") { + t.Fatalf("mismatched cancel error = %v, want identity rejection", err) + } + server.mu.Lock() + turn := server.turns[request.TurnID] + completedAfterReject := turn == nil || turn.completed + server.mu.Unlock() + if completedAfterReject { + t.Fatal("malformed or mismatched cancel mutated the active turn") + } + + if _, err := client.CancelTurn(context.Background(), cancelRequestForStart(request)); err != nil { + t.Fatalf("valid CancelTurn: %v", err) + } + frames := streamCurrentFrames(t, client, request.TurnID) + if !hasFrameType(frames, harness.FrameTurnCancelled) { + t.Fatalf("frames = %#v, want TurnCancelled", frames) + } +} + +func TestResponsesEndpointComposesProjectAndAgent(t *testing.T) { + server := newServer(config{ + projectEndpoint: "https://example.services.ai.azure.com/api/projects/project-a/", + agentName: "agent/name", + apiVersion: "v1", + }, nil) + endpoint, err := server.responsesEndpoint() + if err != nil { + t.Fatalf("responsesEndpoint: %v", err) + } + want := "https://example.services.ai.azure.com/api/projects/project-a/agents/" + + "agent%2Fname/endpoint/protocols/openai/responses?api-version=v1" + if endpoint != want { + t.Fatalf("responsesEndpoint = %q, want %q", endpoint, want) + } +} + +func TestResponsesEndpointSafety(t *testing.T) { + tests := []struct { + name string + endpoint string + want bool + }{ + { + name: "https responses without version", + endpoint: "https://example.openai.azure.com/agents/a/endpoint/protocols/openai/responses", + want: true, + }, + { + name: "https responses with explicit version", + endpoint: "https://example.openai.azure.com/agents/a/endpoint/protocols/openai/responses?api-version=v1", + want: true, + }, + { + name: "loopback http", + endpoint: "http://127.0.0.1:8080/agents/a/endpoint/protocols/openai/responses?api-version=v1", + want: true, + }, + { + name: "empty hostname", + endpoint: "https://:443/agents/a/endpoint/protocols/openai/responses?api-version=v1", + want: false, + }, + { + name: "http non-loopback", + endpoint: "http://example.com/agents/a/endpoint/protocols/openai/responses?api-version=v1", + want: false, + }, + { + name: "userinfo", + endpoint: "https://user:pass@example.com/agents/a/endpoint/protocols/openai/responses?api-version=v1", + want: false, + }, + { + name: "fragment", + endpoint: "https://example.com/agents/a/endpoint/protocols/openai/responses?api-version=v1#fragment", + want: false, + }, + { + name: "secret query", + endpoint: "https://example.com/agents/a/endpoint/protocols/openai/responses?api-version=v1&unsafe=x", + want: false, + }, + {name: "not responses", endpoint: "https://example.com/threads", want: false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := responsesEndpointIsSafe(tt.endpoint); got != tt.want { + t.Fatalf("responsesEndpointIsSafe(%q) = %v, want %v", tt.endpoint, got, tt.want) + } + }) + } +} + +func TestProjectEndpointSafety(t *testing.T) { + tests := []struct { + name string + endpoint string + want bool + }{ + {name: "https", endpoint: "https://example.services.ai.azure.com", want: true}, + {name: "empty hostname", endpoint: "https://:443", want: false}, + {name: "query", endpoint: "https://example.services.ai.azure.com?unsafe=x", want: false}, + {name: "loopback", endpoint: "http://127.0.0.1:8080", want: true}, + {name: "non-loopback http", endpoint: "http://example.com", want: false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := projectEndpointIsSafe(tt.endpoint); got != tt.want { + t.Fatalf("projectEndpointIsSafe(%q) = %v, want %v", tt.endpoint, got, tt.want) + } + }) + } +} + +func TestResponsesRejectsOversizedHostedFunctionCallBatch(t *testing.T) { + server := newServer(config{ + brokeredToolClasses: []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + }, nil) + turn := &turnState{ + request: brokeredReadRequest("foundry-tool-call-cap"), + pendingTools: map[string]string{}, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{}, + } + server.appendFrameLocked(turn, harness.FrameTurnStarted, "foundry hosted response started") + output := make([]responsesOutput, 0, maxBrokeredToolCalls+1) + for i := 0; i <= maxBrokeredToolCalls; i++ { + output = append(output, responsesOutput{ + Type: "function_call", + CallID: fmt.Sprintf("call-%d", i), + Name: "support-ticket-lookup", + Arguments: json.RawMessage(`{"incident":"inc-1"}`), + }) + } + server.handleResponsesResponse(turn, responsesResponse{ID: "resp-cap", Status: "completed", Output: output}) + failed := findFrame(turn.frames, harness.FrameTurnFailed) + if failed == nil || failed.Failed.Reason != "foundry_tool_call_limit_exceeded" { + t.Fatalf("failed frame = %#v, want foundry_tool_call_limit_exceeded", failed) + } + if hasFrameType(turn.frames, harness.FrameToolCallRequested) || turn.requestedToolCalls != 0 { + t.Fatalf("oversized hosted batch was partially accepted: frames=%#v count=%d", turn.frames, turn.requestedToolCalls) + } +} + +func TestResponsesParserConsumesGoldenFixtures(t *testing.T) { + server := newServer( + config{ + runtimeName: "test", + adapterBearer: "adapter-auth-value", + endpoint: "http://127.0.0.1/agents/test-agent/endpoint/protocols/openai/responses?api-version=v1", + foundryAuth: "foundry-auth-value", + requestTimeout: time.Second, + stateRetention: time.Minute, + brokeredToolClasses: []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + }, + &http.Client{Timeout: time.Second}, + ) + request := brokeredReadRequest("foundry-brokered") + turn := &turnState{ + request: request, + pendingTools: map[string]string{}, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{}, + } + server.appendFrameLocked(turn, harness.FrameTurnStarted, "foundry hosted response started") + var functionCall responsesResponse + decodeFixtureInto(t, "testdata/golden/02_function_call_response.json", &functionCall) + server.handleResponsesResponse(turn, functionCall) + requested := findFrame(turn.frames, harness.FrameToolCallRequested) + if requested == nil { + t.Fatalf("frames = %#v, want tool request from fixture", turn.frames) + } + assertJSONFileEqual(t, "testdata/golden/03_tool_call_requested_frame.json", scrubFrameForGolden(*requested)) + + finalTurn := &turnState{ + request: responsesStartTurnRequest("foundry-final"), + pendingTools: map[string]string{}, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{}, + } + server.appendFrameLocked(finalTurn, harness.FrameTurnStarted, "foundry hosted response started") + var finalMessage responsesResponse + decodeFixtureInto(t, "testdata/golden/06_final_message_response.json", &finalMessage) + server.handleResponsesResponse(finalTurn, finalMessage) + completed := findFrame(finalTurn.frames, harness.FrameTurnCompleted) + if completed == nil || completed.Completed.Result != "foundry final answer" { + t.Fatalf("completed frame = %#v, want final answer", completed) + } + + multipleTurn := &turnState{ + request: request, + pendingTools: map[string]string{}, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{}, + } + server.appendFrameLocked(multipleTurn, harness.FrameTurnStarted, "foundry hosted response started") + var multiple responsesResponse + decodeFixtureInto(t, "testdata/golden/10_multiple_calls_response.json", &multiple) + server.handleResponsesResponse(multipleTurn, multiple) + requests := findFrames(multipleTurn.frames, harness.FrameToolCallRequested) + if len(requests) != 2 { + t.Fatalf("frames = %#v, want two tool requests from multiple-call fixture", multipleTurn.frames) + } +} + +func TestResponsesConsumesAgentKitBrokeredFixtures(t *testing.T) { + server := newServer( + config{ + runtimeName: "test", + adapterBearer: "adapter-auth-value", + endpoint: "http://127.0.0.1/agents/test-agent/endpoint/protocols/openai/responses?api-version=v1", + foundryAuth: "foundry-auth-value", + requestTimeout: time.Second, + stateRetention: time.Minute, + brokeredToolClasses: []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + }, + &http.Client{Timeout: time.Second}, + ) + request := responsesStartTurnRequest("agentkit-brokered-fixture") + request.ToolExecutionMode = harness.ToolExecutionModeBrokered + request.Input.Tools = []harness.ToolDefinition{{ + Name: "conformance_read", + Description: "Synthetic AgentKit brokered fixture tool", + BrokeredClass: harness.BrokeredToolClassRead, + Parameters: json.RawMessage(`{"type":"object","properties":{"probe":{"type":"boolean"}}}`), + }} + turn := &turnState{ + request: request, + pendingTools: map[string]string{}, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{}, + } + server.appendFrameLocked(turn, harness.FrameTurnStarted, "foundry hosted response started") + var functionCall responsesResponse + decodeFixtureInto( + t, + "testdata/agentkit-foundry-brokered/function_call_response.json", + &functionCall, + ) + server.handleResponsesResponse(turn, functionCall) + requested := findFrame(turn.frames, harness.FrameToolCallRequested) + if requested == nil { + t.Fatalf("frames = %#v, want tool request from AgentKit fixture", turn.frames) + } + if requested.ToolName != "conformance_read" || requested.ToolCallID != "call_caresp_test_1" { + t.Fatalf("tool request = %#v, want AgentKit fixture call", requested) + } + assertJSONFileEqual(t, "testdata/agentkit-foundry-brokered/initial_request.json", responsesRequest{ + Input: "please read telemetry", + }) + + outputs, err := functionCallOutputs([]harness.ToolCallResult{{ + Version: harness.ProtocolVersion, + RuntimeSessionID: request.RuntimeSessionID, + TurnID: request.TurnID, + ToolCallID: "call_caresp_test_1", + IdempotencyKey: harness.ToolRequestIdempotencyKey(request.RuntimeSessionID, request.TurnID, "call_caresp_test_1"), + Approved: true, + Output: json.RawMessage(`{"success":true}`), + }}) + if err != nil { + t.Fatalf("functionCallOutputs: %v", err) + } + assertJSONFileEqual(t, "testdata/agentkit-foundry-brokered/continuation_request.json", responsesRequest{ + PreviousResponseID: "caresp_test", + Input: outputs, + }) + + finalTurn := &turnState{ + request: responsesStartTurnRequest("agentkit-final-fixture"), + pendingTools: map[string]string{}, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{}, + } + server.appendFrameLocked(finalTurn, harness.FrameTurnStarted, "foundry hosted response started") + var finalMessage responsesResponse + decodeFixtureInto(t, "testdata/agentkit-foundry-brokered/final_message_response.json", &finalMessage) + server.handleResponsesResponse(finalTurn, finalMessage) + completed := findFrame(finalTurn.frames, harness.FrameTurnCompleted) + want := `Brokered tool conformance_read completed with output: {"success":true}` + if completed == nil || completed.Completed.Result != want { + t.Fatalf("completed frame = %#v, want %q", completed, want) + } +} + +func TestResponsesAgentKitErrorPayloadFixturesMatchCanonicalEncoding(t *testing.T) { + tests := []struct { + name string + result harness.ToolCallResult + fixture string + }{ + { + name: "approval declined", + result: baseToolResult("call-agentkit-1", false, nil, &harness.ErrorInfo{ + Code: "approval_declined", Message: "Human declined dispatch-work-order", + }), + fixture: "testdata/agentkit-foundry-brokered/approval_declined_payload.json", + }, + { + name: "policy rejection", + result: baseToolResult("call-agentkit-2", false, nil, &harness.ErrorInfo{ + Code: "tool_policy_rejected", Message: "writes are disabled", + }), + fixture: "testdata/agentkit-foundry-brokered/tool_policy_rejection_payload.json", + }, + { + name: "execution failure", + result: baseToolResult("call-agentkit-3", true, nil, &harness.ErrorInfo{ + Code: "tool_execution_failed", Message: "downstream timed out", + }), + fixture: "testdata/agentkit-foundry-brokered/tool_execution_failure_payload.json", + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + payload, err := canonicalToolResultOutput(tt.result) + if err != nil { + t.Fatalf("canonicalToolResultOutput: %v", err) + } + assertJSONFileEqual(t, tt.fixture, json.RawMessage(payload)) + }) + } +} + +func TestResponsesGoldenFixturesDoNotContainEndpointsOrSecrets(t *testing.T) { + for _, root := range []string{"testdata/golden", "testdata/agentkit-foundry-brokered"} { + entries, err := os.ReadDir(root) + if err != nil { + t.Fatalf("ReadDir(%s): %v", root, err) + } + for _, entry := range entries { + if entry.IsDir() || !strings.HasSuffix(entry.Name(), ".json") { + continue + } + path := root + "/" + entry.Name() + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("ReadFile(%s): %v", path, err) + } + text := strings.ToLower(string(data)) + for _, forbidden := range []string{"http://", "https://", "authorization", "api_key", "api-key", "bearer "} { + if strings.Contains(text, forbidden) { + t.Fatalf("fixture %s contains forbidden %q", path, forbidden) + } + } + } + } +} + +func TestResponsesPreservesNumericJSONTokens(t *testing.T) { + args, err := normalizeResponsesToolArguments(json.RawMessage(`{"id":9007199254740993}`)) + if err != nil { + t.Fatalf("normalizeResponsesToolArguments: %v", err) + } + if got := string(args); got != `{"id":9007199254740993}` { + t.Fatalf("arguments = %s, want numeric token preserved", got) + } + result := baseToolResult("call-2", true, json.RawMessage(`{"id":9007199254740993}`), nil) + payload, err := canonicalToolResultOutput(result) + if err != nil { + t.Fatalf("canonicalToolResultOutput: %v", err) + } + want := `{"approved":true,"output":{"id":9007199254740993}}` + if payload != want { + t.Fatalf("payload = %s, want %s", payload, want) + } +} + +func TestResponsesLargeOutputFails(t *testing.T) { + server := newServer(config{ + runtimeName: "test", + adapterBearer: "adapter-auth-value", + endpoint: "http://127.0.0.1/agents/test-agent/endpoint/protocols/openai/responses?api-version=v1", + foundryAuth: "foundry-auth-value", + requestTimeout: time.Second, + stateRetention: time.Minute, + }, &http.Client{Timeout: time.Second}) + turn := &turnState{ + request: responsesStartTurnRequest("foundry-large-output"), + pendingTools: map[string]string{}, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{}, + } + server.appendFrameLocked(turn, harness.FrameTurnStarted, "foundry hosted response started") + server.handleResponsesResponse(turn, responsesResponse{ + ID: "resp-large", + Status: "completed", + Output: []responsesOutput{{ + Type: "message", + Content: strings.Repeat("x", maxFoundryOutputBytes+1), + }}, + }) + if hasFrameType(turn.frames, harness.FrameTurnCompleted) { + t.Fatalf("frames = %#v, oversized response should not complete", turn.frames) + } + failed := findFrame(turn.frames, harness.FrameTurnFailed) + if failed == nil || failed.Failed.Reason != "foundry_output_too_large" { + t.Fatalf("failed frame = %#v, want foundry_output_too_large", failed) + } +} + +func TestResponsesOutputThatExceedsSSEFrameLimitFails(t *testing.T) { + server := newServer(config{ + runtimeName: "test", + adapterBearer: "adapter-auth-value", + endpoint: "http://127.0.0.1/agents/test-agent/endpoint/protocols/openai/responses?api-version=v1", + foundryAuth: "foundry-auth-value", + requestTimeout: time.Second, + stateRetention: time.Minute, + }, &http.Client{Timeout: time.Second}) + turn := &turnState{ + request: responsesStartTurnRequest("foundry-large-frame-output"), + pendingTools: map[string]string{}, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{}, + } + server.appendFrameLocked(turn, harness.FrameTurnStarted, "foundry hosted response started") + server.handleResponsesResponse(turn, responsesResponse{ + ID: "resp-large-frame", + Status: "completed", + Output: []responsesOutput{{ + Type: "message", + Content: strings.Repeat("\"", maxFoundryOutputBytes), + }}, + }) + if hasFrameType(turn.frames, harness.FrameTurnCompleted) { + t.Fatalf("frames = %#v, oversized SSE frame should not complete", turn.frames) + } + failed := findFrame(turn.frames, harness.FrameTurnFailed) + if failed == nil || failed.Failed.Reason != "foundry_output_frame_too_large" { + t.Fatalf("failed frame = %#v, want foundry_output_frame_too_large", failed) + } +} + +func TestResponsesOversizedToolCallFrameFailsBeforeRequestingTool(t *testing.T) { + server := newServer(config{ + runtimeName: "test", + adapterBearer: "adapter-auth-value", + endpoint: "http://127.0.0.1/agents/test-agent/endpoint/protocols/openai/responses?api-version=v1", + foundryAuth: "foundry-auth-value", + requestTimeout: time.Second, + stateRetention: time.Minute, + brokeredToolClasses: []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + }, &http.Client{Timeout: time.Second}) + turn := &turnState{ + request: brokeredReadRequest("foundry-large-tool-call-frame"), + pendingTools: map[string]string{}, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{}, + } + server.appendFrameLocked(turn, harness.FrameTurnStarted, "foundry hosted response started") + arguments, err := json.Marshal(map[string]any{"payload": strings.Repeat("x", harness.MaxSSEFrameBytes)}) + if err != nil { + t.Fatalf("marshal arguments: %v", err) + } + server.handleResponsesResponse(turn, responsesResponse{ + ID: "resp-large-tool-call", + Status: "completed", + Output: []responsesOutput{{ + Type: "function_call", + CallID: "call-1", + Name: "support-ticket-lookup", + Arguments: arguments, + }}, + }) + if hasFrameType(turn.frames, harness.FrameToolCallRequested) { + t.Fatalf("frames = %#v, oversized tool call should not be requested", turn.frames) + } + failed := findFrame(turn.frames, harness.FrameTurnFailed) + if failed == nil || failed.Failed.Reason != "foundry_tool_call_frame_too_large" { + t.Fatalf("failed frame = %#v, want foundry_tool_call_frame_too_large", failed) + } +} + +func TestResponsesToolResultPreflightUsesExactEventualSequence(t *testing.T) { + server := newServer(config{}, &http.Client{Timeout: time.Second}) + request := brokeredReadRequest("foundry-tool-result-boundary") + turn := &turnState{ + request: request, + responseID: "resp-1", + pendingTools: map[string]string{ + "call-1": "support-ticket-lookup", + "call-2": "support-ticket-lookup", + }, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{}, + } + for range 8 { + server.appendFrameLocked(turn, harness.FrameTurnStarted, "foundry hosted response started") + } + emptyResult := toolResultForRequest(request, "call-1", true, json.RawMessage(`{"payload":""}`), nil) + seqNineFrame := server.newToolResultFrame(turn, 9, "support-ticket-lookup", emptyResult) + seqNineFrame.CreatedAt = sseSizeProbeTime + seqTenFrame := server.newToolResultFrame(turn, 10, "support-ticket-lookup", emptyResult) + seqTenFrame.CreatedAt = sseSizeProbeTime + seqNineJSON, err := json.Marshal(seqNineFrame) + if err != nil { + t.Fatalf("marshal sequence-nine frame: %v", err) + } + seqTenJSON, err := json.Marshal(seqTenFrame) + if err != nil { + t.Fatalf("marshal sequence-ten frame: %v", err) + } + if len(seqTenJSON) <= len(seqNineJSON) { + t.Fatalf("frame sizes seq10=%d seq9=%d, want decimal-boundary overhead", len(seqTenJSON), len(seqNineJSON)) + } + payloadSize := harness.MaxSSEFrameBytes - len("data: ") - len(seqNineJSON) - 1 + if payloadSize <= 0 { + t.Fatalf("payload boundary = %d, want positive", payloadSize) + } + output := json.RawMessage(`{"payload":"` + strings.Repeat("x", payloadSize) + `"}`) + result := toolResultForRequest(request, "call-1", true, output, nil) + if !toolResultFrameFitsSSE(server.newToolResultFrame(turn, 9, "support-ticket-lookup", result)) { + t.Fatal("eventual sequence-nine frame should fit") + } + if toolResultFrameFitsSSE(server.newToolResultFrame(turn, 10, "support-ticket-lookup", result)) { + t.Fatal("sequence-ten probe unexpectedly fits boundary frame") + } + toSubmit, err := server.recordContinueResults(turn, []harness.ToolCallResult{result}) + if err != nil { + t.Fatalf("recordContinueResults rejected exact eventual sequence: %v", err) + } + if len(toSubmit) != 0 || len(turn.bufferedResults) != 1 { + t.Fatalf("toSubmit=%#v buffered=%#v, want one accepted partial result", toSubmit, turn.bufferedResults) + } +} + +func TestResponsesOversizedBatchValidatesAllResultsBeforeFailure(t *testing.T) { + server := newServer(config{}, &http.Client{Timeout: time.Second}) + request := brokeredReadRequest("foundry-oversized-batch") + turn := &turnState{ + request: request, + pendingTools: map[string]string{ + "call-1": "support-ticket-lookup", + }, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{}, + } + server.appendFrameLocked(turn, harness.FrameTurnStarted, "foundry hosted response started") + oversizedOutput, err := json.Marshal(map[string]any{"payload": strings.Repeat("x", harness.MaxSSEFrameBytes)}) + if err != nil { + t.Fatalf("marshal oversized output: %v", err) + } + oversized := toolResultForRequest(request, "call-1", true, oversizedOutput, nil) + unknown := toolResultForRequest(request, "call-missing", true, json.RawMessage(`{"success":true}`), nil) + + if _, err := server.recordContinueResults(turn, []harness.ToolCallResult{oversized, unknown}); err == nil || + !strings.Contains(err.Error(), "not pending") { + t.Fatalf("recordContinueResults error = %v, want structural rejection", err) + } + if turn.completed || len(turn.bufferedResults) != 0 || len(turn.bufferedDigests) != 0 { + t.Fatalf( + "invalid batch mutated turn: completed=%v buffers=%#v/%#v", + turn.completed, + turn.bufferedResults, + turn.bufferedDigests, + ) + } +} + +func TestResponsesStreamRechecksSuppressionBetweenToolFrames(t *testing.T) { + server := newServer(config{ + adapterBearer: "adapter-auth-value", + brokeredToolClasses: []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + }, &http.Client{Timeout: time.Second}) + request := brokeredReadRequest("foundry-live-suppression") + turn := &turnState{ + request: request, + pendingTools: map[string]string{}, + suppressedToolCalls: map[string]struct{}{}, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{}, + } + server.turns[request.TurnID] = turn + server.appendFrameLocked(turn, harness.FrameTurnStarted, "foundry hosted response started") + server.handleResponsesResponse(turn, responsesResponse{ + ID: "resp-live-suppression", + Status: "completed", + Output: []responsesOutput{ + { + Type: "function_call", + CallID: "call-1", + Name: "support-ticket-lookup", + Arguments: json.RawMessage(`{"incident":"inc-1"}`), + }, + { + Type: "function_call", + CallID: "call-2", + Name: "support-ticket-lookup", + Arguments: json.RawMessage(`{"incident":"inc-2"}`), + }, + }, + }) + + adapter := httptest.NewServer(server.handler()) + t.Cleanup(adapter.Close) + client := newHarnessClient(t, adapter) + var requested []string + var frames []harness.HarnessEventFrame + err := client.StreamFrames(context.Background(), request.TurnID, 0, func(frame harness.HarnessEventFrame) error { + frames = append(frames, frame) + if frame.Type != harness.FrameToolCallRequested { + return nil + } + requested = append(requested, frame.ToolCallID) + if len(requested) == 1 { + _, cancelErr := client.CancelTurn(context.Background(), cancelRequestForStart(request)) + return cancelErr + } + return nil + }) + if err != nil { + t.Fatalf("StreamFrames: %v", err) + } + if !reflect.DeepEqual(requested, []string{"call-1"}) { + t.Fatalf("requested tool calls = %#v, want only first call before cancellation", requested) + } + if !hasFrameType(frames, harness.FrameTurnCancelled) { + t.Fatalf("frames = %#v, want terminal cancellation", frames) + } +} + +func TestResponsesTerminalFailureSuppressesPendingToolFrames(t *testing.T) { + server := newServer(config{ + adapterBearer: "adapter-auth-value", + brokeredToolClasses: []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + }, &http.Client{Timeout: time.Second}) + request := brokeredReadRequest("foundry-suppress-pending") + turn := &turnState{ + request: request, + pendingTools: map[string]string{}, + suppressedToolCalls: map[string]struct{}{}, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{}, + } + server.turns[request.TurnID] = turn + server.runtimeSessions[request.RuntimeSessionID] = foundrySession{ + ID: existingSessionID, + LastSeen: time.Now().UTC(), + } + server.appendFrameLocked(turn, harness.FrameTurnStarted, "foundry hosted response started") + server.handleResponsesResponse(turn, responsesResponse{ + ID: "resp-1", + Status: "completed", + Output: []responsesOutput{ + { + Type: "function_call", CallID: "call-1", Name: "support-ticket-lookup", + Arguments: json.RawMessage(`{"incident":"inc-1"}`), + }, + { + Type: "function_call", CallID: "call-2", Name: "support-ticket-lookup", + Arguments: json.RawMessage(`{"incident":"inc-2"}`), + }, + }, + }) + first := toolResultForRequest(request, "call-1", true, json.RawMessage(`{"success":true}`), nil) + toSubmit, err := server.recordContinueResults(turn, []harness.ToolCallResult{first}) + if err != nil || len(toSubmit) != 0 { + t.Fatalf("buffer first result = %#v, %v", toSubmit, err) + } + oversizedOutput, err := json.Marshal(map[string]any{"payload": strings.Repeat("x", harness.MaxSSEFrameBytes)}) + if err != nil { + t.Fatalf("marshal oversized output: %v", err) + } + second := toolResultForRequest(request, "call-2", true, oversizedOutput, nil) + if _, err := server.recordContinueResults(turn, []harness.ToolCallResult{second}); err == nil { + t.Fatal("oversized second result error = nil") + } + if _, retained := server.runtimeSessions[request.RuntimeSessionID]; retained { + t.Fatal("oversized result failure retained reused runtime session") + } + if _, quarantined := server.quarantinedSessions[request.RuntimeSessionID]; !quarantined { + t.Fatal("oversized result failure did not quarantine reused runtime session") + } + + adapter := httptest.NewServer(server.handler()) + t.Cleanup(adapter.Close) + client := newHarnessClient(t, adapter) + frames := streamCurrentFrames(t, client, request.TurnID) + if hasFrameType(frames, harness.FrameToolCallRequested) || !hasFrameType(frames, harness.FrameTurnFailed) { + t.Fatalf("replayed frames = %#v, want pending tool requests suppressed before terminal failure", frames) + } +} + +func TestResponsesOversizedResultTombstonesPreviouslyBufferedResults(t *testing.T) { + server := newServer(config{}, &http.Client{Timeout: time.Second}) + request := brokeredReadRequest("foundry-partial-then-oversized") + turn := &turnState{ + request: request, + pendingTools: map[string]string{ + "call-1": "support-ticket-lookup", + "call-2": "support-ticket-lookup", + }, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{}, + } + server.appendFrameLocked(turn, harness.FrameTurnStarted, "foundry hosted response started") + first := toolResultForRequest(request, "call-1", true, json.RawMessage(`{"success":true}`), nil) + toSubmit, err := server.recordContinueResults(turn, []harness.ToolCallResult{first}) + if err != nil || len(toSubmit) != 0 { + t.Fatalf("buffer first result = %#v, %v", toSubmit, err) + } + oversizedOutput, err := json.Marshal(map[string]any{"payload": strings.Repeat("x", harness.MaxSSEFrameBytes)}) + if err != nil { + t.Fatalf("marshal oversized output: %v", err) + } + second := toolResultForRequest(request, "call-2", true, oversizedOutput, nil) + if _, err := server.recordContinueResults(turn, []harness.ToolCallResult{second}); err == nil { + t.Fatal("oversized second result error = nil") + } + if !turn.completed || len(turn.bufferedResults) != 0 || len(turn.bufferedDigests) != 0 { + t.Fatalf( + "terminal oversized state = completed:%v buffers:%#v/%#v", + turn.completed, + turn.bufferedResults, + turn.bufferedDigests, + ) + } + for _, result := range []harness.ToolCallResult{first, second} { + if err := server.ensureTerminalContinueIsDuplicate(turn, []harness.ToolCallResult{result}); err != nil { + t.Fatalf("terminal duplicate %s rejected: %v", result.ToolCallID, err) + } + } +} + +func TestResponsesOversizedToolResultFrameFailsBeforeContinuation(t *testing.T) { + server := newServer(config{ + runtimeName: "test", + adapterBearer: "adapter-auth-value", + endpoint: "http://127.0.0.1/agents/test-agent/endpoint/protocols/openai/responses?api-version=v1", + foundryAuth: "foundry-auth-value", + requestTimeout: time.Second, + stateRetention: time.Minute, + brokeredToolClasses: []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + }, &http.Client{Timeout: time.Second}) + request := brokeredReadRequest("foundry-large-tool-result-frame") + turn := &turnState{ + request: request, + responseID: "resp-1", + pendingTools: map[string]string{"call-1": "support-ticket-lookup"}, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{}, + } + server.appendFrameLocked(turn, harness.FrameTurnStarted, "foundry hosted response started") + output, err := json.Marshal(map[string]any{"payload": strings.Repeat("x", harness.MaxSSEFrameBytes)}) + if err != nil { + t.Fatalf("marshal output: %v", err) + } + result := toolResultForRequest(request, "call-1", true, output, nil) + if _, err := server.recordContinueResults(turn, []harness.ToolCallResult{result}); err == nil { + t.Fatal("recordContinueResults oversized output error = nil") + } + if hasFrameType(turn.frames, harness.FrameToolResultReceived) { + t.Fatalf("frames = %#v, oversized tool result should not be streamed", turn.frames) + } + failed := findFrame(turn.frames, harness.FrameTurnFailed) + if failed == nil || failed.Failed.Reason != "brokered_tool_result_frame_too_large" { + t.Fatalf("failed frame = %#v, want brokered_tool_result_frame_too_large", failed) + } + if len(turn.bufferedResults) != 0 || len(turn.bufferedDigests) != 0 { + t.Fatalf("oversized result retained buffered state: %#v/%#v", turn.bufferedResults, turn.bufferedDigests) + } + if len(turn.submittedDigests) != 1 { + t.Fatalf("oversized result submitted digests = %#v, want duplicate tombstone", turn.submittedDigests) + } + if err := server.ensureTerminalContinueIsDuplicate(turn, []harness.ToolCallResult{result}); err != nil { + t.Fatalf("identical oversized result retry was not accepted as duplicate: %v", err) + } +} + +func TestResponsesInitialPlatformErrorRetainsFailedTurn(t *testing.T) { + foundry := newFakeResponses(t, fakeResponsesConfig{scenario: "platform_error"}) + adapter, server := newTestResponsesAdapterWithServer(t, foundry.endpoint(), nil) + client := newHarnessClient(t, adapter) + request := responsesStartTurnRequest("foundry-platform-error") + + if _, err := client.StartTurn(context.Background(), request); err != nil { + t.Fatalf("StartTurn should accept the terminal failed turn: %v", err) + } + server.mu.Lock() + turn := server.turns[request.TurnID] + if turn == nil { + server.mu.Unlock() + t.Fatalf("turn %q was discarded after an uncertain initial hosted response", request.TurnID) + } + if turn.initializing || !turn.completed { + server.mu.Unlock() + t.Fatalf("turn state = initializing:%v completed:%v, want terminal failed turn", turn.initializing, turn.completed) + } + failed := findFrame(turn.frames, harness.FrameTurnFailed) + server.mu.Unlock() + if failed == nil || failed.Failed.Reason != foundryInitialUnknown { + t.Fatalf("failed frame = %#v, want foundry_initial_unknown", failed) + } + if strings.Contains(failed.Failed.Message, foundry.URL) || + strings.Contains(failed.Failed.Message, "unknown scenario") { + t.Fatalf("failed frame leaked upstream detail: %#v", failed.Failed) + } + frames := streamCurrentFrames(t, client, request.TurnID) + if streamed := findFrame(frames, harness.FrameTurnFailed); streamed == nil || + streamed.Failed.Reason != foundryInitialUnknown { + t.Fatalf("streamed failed frame = %#v, want foundry_initial_unknown", streamed) + } + if _, err := client.StartTurn(context.Background(), request); err != nil { + t.Fatalf("retrying retained TurnID: %v", err) + } + if foundry.postCount.Load() != 1 { + t.Fatalf("hosted post count after retry = %d, want 1", foundry.postCount.Load()) + } +} + +//nolint:dupl // Mirrors failure-status regression with a distinct non-terminal status. +func TestResponsesNonTerminalStatusDoesNotCompleteWithPartialText(t *testing.T) { + server := newServer(config{ + runtimeName: "test", + adapterBearer: "adapter-auth-value", + endpoint: "http://127.0.0.1/agents/test-agent/endpoint/protocols/openai/responses?api-version=v1", + foundryAuth: "foundry-auth-value", + requestTimeout: time.Second, + stateRetention: time.Minute, + }, &http.Client{Timeout: time.Second}) + turn := &turnState{ + request: responsesStartTurnRequest("foundry-in-progress"), + pendingTools: map[string]string{}, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{}, + } + server.appendFrameLocked(turn, harness.FrameTurnStarted, "foundry hosted response started") + server.handleResponsesResponse(turn, responsesResponse{ + ID: "resp-in-progress", + Status: "in_progress", + Output: []responsesOutput{{Type: "message", Content: "partial text"}}, + }) + if hasFrameType(turn.frames, harness.FrameTurnCompleted) { + t.Fatalf("frames = %#v, in-progress response should not complete", turn.frames) + } + failed := findFrame(turn.frames, harness.FrameTurnFailed) + if failed == nil || failed.Failed.Reason != "foundry_in_progress" { + t.Fatalf("failed frame = %#v, want foundry_in_progress", failed) + } +} + +//nolint:dupl // Mirrors non-terminal-status regression with a distinct failed status. +func TestResponsesFailureStatusDoesNotCompleteWithPartialText(t *testing.T) { + server := newServer(config{ + runtimeName: "test", + adapterBearer: "adapter-auth-value", + endpoint: "http://127.0.0.1/agents/test-agent/endpoint/protocols/openai/responses?api-version=v1", + foundryAuth: "foundry-auth-value", + requestTimeout: time.Second, + stateRetention: time.Minute, + }, &http.Client{Timeout: time.Second}) + turn := &turnState{ + request: responsesStartTurnRequest("foundry-failed-status"), + pendingTools: map[string]string{}, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{}, + } + server.appendFrameLocked(turn, harness.FrameTurnStarted, "foundry hosted response started") + server.handleResponsesResponse(turn, responsesResponse{ + ID: "resp-failed", + Status: "failed", + Output: []responsesOutput{{ + Type: "message", + Content: "partial text should not be completed", + }}, + }) + if hasFrameType(turn.frames, harness.FrameTurnCompleted) { + t.Fatalf("frames = %#v, failed response should not complete", turn.frames) + } + failed := findFrame(turn.frames, harness.FrameTurnFailed) + if failed == nil || failed.Failed.Reason != foundryFailedReason { + t.Fatalf("failed frame = %#v, want foundry_failed", failed) + } +} + +func TestResponsesFailureStatusWithFunctionCallFailsBeforeToolRequest(t *testing.T) { + server := newServer(config{ + runtimeName: "test", + adapterBearer: "adapter-auth-value", + endpoint: "http://127.0.0.1/agents/test-agent/endpoint/protocols/openai/responses?api-version=v1", + foundryAuth: "foundry-auth-value", + requestTimeout: time.Second, + stateRetention: time.Minute, + brokeredToolClasses: []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + }, &http.Client{Timeout: time.Second}) + turn := &turnState{ + request: brokeredReadRequest("foundry-failed-function-call"), + pendingTools: map[string]string{}, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{}, + } + server.appendFrameLocked(turn, harness.FrameTurnStarted, "foundry hosted response started") + server.handleResponsesResponse(turn, responsesResponse{ + ID: "resp-failed", + Status: "incomplete", + Output: []responsesOutput{{ + Type: "function_call", + CallID: "call-1", + Name: "support-ticket-lookup", + Arguments: json.RawMessage(`{"incident":"inc-1"}`), + }}, + }) + if hasFrameType(turn.frames, harness.FrameToolCallRequested) { + t.Fatalf("frames = %#v, failed response should not request a tool", turn.frames) + } + failed := findFrame(turn.frames, harness.FrameTurnFailed) + if failed == nil || failed.Failed.Reason != "foundry_incomplete" { + t.Fatalf("failed frame = %#v, want foundry_incomplete", failed) + } +} + +func TestSanitizeEndpointDoesNotReturnRawMalformedURL(t *testing.T) { + raw := "http://[::1" + "?unsafe=do-not-log" + if got := sanitizeEndpoint(raw); got == raw || strings.Contains(got, "do-not-log") { + t.Fatalf("sanitizeEndpoint(%q) = %q, want redacted placeholder", raw, got) + } +} + +func TestResponsesMissingStatusDoesNotCompleteWithPartialText(t *testing.T) { + server := newServer(config{ + runtimeName: "test", + adapterBearer: "adapter-auth-value", + endpoint: "http://127.0.0.1/agents/test-agent/endpoint/protocols/openai/responses?api-version=v1", + foundryAuth: "foundry-auth-value", + requestTimeout: time.Second, + stateRetention: time.Minute, + }, &http.Client{Timeout: time.Second}) + turn := &turnState{ + request: responsesStartTurnRequest("foundry-missing-status"), + pendingTools: map[string]string{}, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{}, + } + server.appendFrameLocked(turn, harness.FrameTurnStarted, "foundry hosted response started") + server.handleResponsesResponse(turn, responsesResponse{ + ID: "resp-missing-status", + Output: []responsesOutput{{Type: "message", Content: "partial text"}}, + }) + if hasFrameType(turn.frames, harness.FrameTurnCompleted) { + t.Fatalf("frames = %#v, missing status response should not complete", turn.frames) + } + failed := findFrame(turn.frames, harness.FrameTurnFailed) + if failed == nil || failed.Failed.Reason != "foundry_status_missing" { + t.Fatalf("failed frame = %#v, want foundry_status_missing", failed) + } +} + +func TestResponsesFunctionCallWithoutResponseIDFailsBeforeToolRequest(t *testing.T) { + server := newServer(config{ + runtimeName: "test", + adapterBearer: "adapter-auth-value", + endpoint: "http://127.0.0.1/agents/test-agent/endpoint/protocols/openai/responses?api-version=v1", + foundryAuth: "foundry-auth-value", + requestTimeout: time.Second, + stateRetention: time.Minute, + brokeredToolClasses: []harness.BrokeredToolClass{harness.BrokeredToolClassRead}, + }, &http.Client{Timeout: time.Second}) + turn := &turnState{ + request: brokeredReadRequest("foundry-missing-id"), + pendingTools: map[string]string{}, + bufferedResults: map[string]harness.ToolCallResult{}, + bufferedDigests: map[string]toolResultDigest{}, + submittedDigests: map[string]toolResultDigest{}, + } + server.appendFrameLocked(turn, harness.FrameTurnStarted, "foundry hosted response started") + server.handleResponsesResponse(turn, responsesResponse{ + Status: "completed", + Output: []responsesOutput{{ + Type: "function_call", + CallID: "call-1", + Name: "support-ticket-lookup", + Arguments: json.RawMessage(`{"incident":"inc-1"}`), + }}, + }) + if hasFrameType(turn.frames, harness.FrameToolCallRequested) { + t.Fatalf("frames = %#v, should not request Orka execution without response id", turn.frames) + } + failed := findFrame(turn.frames, harness.FrameTurnFailed) + if failed == nil || failed.Failed.Reason != "foundry_response_id_missing" { + t.Fatalf("failed frame = %#v, want missing response id failure", failed) + } +} + +func TestCanonicalToolResultOutputWrapsNonObjectJSON(t *testing.T) { + tests := []struct { + name string + output json.RawMessage + want string + }{ + {name: "object", output: json.RawMessage(`{"value":1}`), want: `{"approved":true,"output":{"value":1}}`}, + {name: "array", output: json.RawMessage(`[1,2]`), want: `{"approved":true,"output":{"result":[1,2]}}`}, + {name: "string", output: json.RawMessage(`"ok"`), want: `{"approved":true,"output":{"result":"ok"}}`}, + {name: "boolean", output: json.RawMessage(`true`), want: `{"approved":true,"output":{"result":true}}`}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + result := baseToolResult("call-1", true, tt.output, nil) + got, err := canonicalToolResultOutput(result) + if err != nil { + t.Fatalf("canonicalToolResultOutput: %v", err) + } + if got != tt.want { + t.Fatalf("canonical output = %s, want %s", got, tt.want) + } + }) + } +} + +func TestCanonicalToolResultErrorUsesSafeHostedEnvelope(t *testing.T) { + result := baseToolResult("call-1", true, nil, &harness.ErrorInfo{ + Code: "transport_failed", + Message: "POST https://tools.internal.example/v1/run: connection refused", + }) + got, err := canonicalToolResultOutput(result) + if err != nil { + t.Fatalf("canonicalToolResultOutput: %v", err) + } + want := `{"approved":false,"error":{"code":"tool_execution_failed","message":"tool execution failed"}}` + if got != want { + t.Fatalf("canonical error output = %s, want %s", got, want) + } + if strings.Contains(got, "tools.internal.example") || strings.Contains(got, "https://") { + t.Fatalf("canonical error output leaked tool URL: %s", got) + } +} + +func TestCanonicalErrorAndDeclineOutputFixtures(t *testing.T) { + tests := []struct { + name string + result harness.ToolCallResult + fixture string + }{ + { + name: "approval declined", + result: baseToolResult("call-1", false, nil, &harness.ErrorInfo{ + Code: "approval_declined", Message: "human declined", + }), + fixture: "testdata/golden/07_approval_declined_output.json", + }, + { + name: "policy rejection", + result: baseToolResult("call-1", false, nil, &harness.ErrorInfo{ + Code: "tool_policy_rejected", Message: "tool is not allowed", + }), + fixture: "testdata/golden/08_tool_policy_rejection_output.json", + }, + { + name: "execution failure", + result: baseToolResult("call-1", true, nil, &harness.ErrorInfo{ + Code: "tool_execution_failed", Message: "downstream failed", + }), + fixture: "testdata/golden/09_tool_execution_failure_output.json", + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + outputs, err := functionCallOutputs([]harness.ToolCallResult{tt.result}) + if err != nil { + t.Fatalf("functionCallOutputs: %v", err) + } + assertJSONFileEqual(t, tt.fixture, outputs[0]) + }) + } +} + +type fakeResponsesConfig struct { + scenario string + toolName string + continuationStatus int + requiredProof string +} + +type fakeResponses struct { + *httptest.Server + cfg fakeResponsesConfig + mu sync.Mutex + requests []json.RawMessage + headers []http.Header + postCount atomic.Int32 + sawRequestLevelTools atomic.Bool +} + +func newFakeResponses(t *testing.T, cfg fakeResponsesConfig) *fakeResponses { + t.Helper() + if cfg.scenario == "" { + cfg.scenario = "observed" + } + if cfg.toolName == "" { + cfg.toolName = "support-ticket-lookup" + } + f := &fakeResponses{cfg: cfg} + mux := http.NewServeMux() + mux.HandleFunc( + "/agents/test-agent/endpoint/protocols/openai/responses", + func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodPost { + http.NotFound(w, r) + return + } + if got := r.URL.Query().Get("api-version"); got != "v1" { + http.Error(w, "missing api-version", http.StatusBadRequest) + return + } + body, _ := ioReadAll(r.Body) + var decoded map[string]any + _ = json.Unmarshal(body, &decoded) + if _, ok := decoded["tools"]; ok { + f.sawRequestLevelTools.Store(true) + } + f.mu.Lock() + f.requests = append(f.requests, append(json.RawMessage(nil), body...)) + f.headers = append(f.headers, r.Header.Clone()) + f.mu.Unlock() + f.postCount.Add(1) + if _, continuing := decoded["previous_response_id"]; continuing { + if got := r.Header.Get("x-agent-session-id"); got != fakeSessionID { + http.Error(w, "missing session header", http.StatusBadRequest) + return + } + if got := decoded["agent_session_id"]; got != fakeSessionID { + http.Error(w, "missing body session", http.StatusBadRequest) + return + } + if f.cfg.continuationStatus != 0 { + http.Error(w, "continuation failed", f.cfg.continuationStatus) + return + } + writeJSON(w, finalResponsesMessage()) + return + } + w.Header().Set("x-agent-session-id", fakeSessionID) + switch f.cfg.scenario { + case "observed": + writeJSON(w, finalResponsesMessage()) + case "function_call": + writeJSON(w, functionCallResponse(f.cfg.toolName)) + case "failed_with_session": + writeJSON(w, map[string]any{ + "id": "resp-failed", + "agent_session_id": fakeSessionID, + "status": "failed", + }) + case "malformed_arguments": + writeJSON( + w, + map[string]any{ + "id": "resp-1", + "status": "completed", + "output": []any{ + map[string]any{ + "type": "function_call", + "call_id": "call-1", + "name": f.cfg.toolName, + "arguments": "not-json", + }, + }, + }, + ) + case "multiple_calls": + writeJSON(w, multipleCallsResponse()) + default: + http.Error(w, "unknown scenario", http.StatusInternalServerError) + } + }, + ) + f.Server = httptest.NewServer(mux) + t.Cleanup(f.Close) + return f +} + +func (f *fakeResponses) endpoint() string { + return f.URL + "/agents/test-agent/endpoint/protocols/openai/responses?api-version=v1" +} + +func (f *fakeResponses) requestBody(i int) json.RawMessage { + f.mu.Lock() + defer f.mu.Unlock() + return append(json.RawMessage(nil), f.requests[i]...) +} + +func (f *fakeResponses) requestHeader(i int) http.Header { + f.mu.Lock() + defer f.mu.Unlock() + return f.headers[i].Clone() +} + +func functionCallResponse(toolName string) map[string]any { + return map[string]any{ + "id": "resp-1", + "agent_session_id": fakeSessionID, + "status": "completed", + "output": []any{ + map[string]any{ + "type": "function_call", + "call_id": "call-1", + "name": toolName, + "arguments": `{"incident":"inc-1"}`, + }, + }, + } +} + +func multipleCallsResponse() map[string]any { + return map[string]any{"id": "resp-1", "agent_session_id": fakeSessionID, "status": "completed", "output": []any{ + map[string]any{ + "type": "function_call", + "call_id": "call-1", + "name": "support-ticket-lookup", + "arguments": `{"incident":"inc-1"}`, + }, + map[string]any{ + "type": "function_call", + "call_id": "call-2", + "name": "support-ticket-lookup", + "arguments": `{"incident":"inc-2"}`, + }, + }} +} + +func finalResponsesMessage() map[string]any { + return map[string]any{ + "id": "resp-2", + "agent_session_id": fakeSessionID, + "status": "completed", + "output": []any{ + map[string]any{ + "type": "message", + "role": "assistant", + "content": []any{map[string]any{"type": "output_text", "text": "foundry final answer"}}, + }, + }, + } +} + +func newTestResponsesAdapter(t *testing.T, endpoint string, classes []harness.BrokeredToolClass) *httptest.Server { + t.Helper() + adapter, _ := newTestResponsesAdapterWithServer(t, endpoint, classes) + return adapter +} + +func newTestResponsesAdapterWithServer( + t *testing.T, + endpoint string, + classes []harness.BrokeredToolClass, +) (*httptest.Server, *server) { + t.Helper() + return newTestResponsesAdapterWithHTTPClient( + t, + endpoint, + classes, + &http.Client{Timeout: time.Second}, + ) +} + +func newTestResponsesAdapterWithHTTPClient( + t *testing.T, + endpoint string, + classes []harness.BrokeredToolClass, + backendClient *http.Client, +) (*httptest.Server, *server) { + t.Helper() + continuationProof := "" + if len(classes) > 0 { + continuationProof = testContinuationProof + } + s := newServer(config{ + addr: ":0", + runtimeName: "foundry-responses-test", + adapterBearer: "adapter-auth-value", + endpoint: endpoint, + foundryAuth: "foundry-auth-value", + apiVersion: "v1", + requestTimeout: time.Second, + stateRetention: time.Minute, + continuationProof: continuationProof, + brokeredToolClasses: append([]harness.BrokeredToolClass(nil), classes...), + }, backendClient) + adapter := httptest.NewServer(s.handler()) + t.Cleanup(adapter.Close) + return adapter, s +} + +type roundTripFunc func(*http.Request) (*http.Response, error) + +func (f roundTripFunc) RoundTrip(r *http.Request) (*http.Response, error) { + return f(r) +} + +func jsonHTTPResponse(r *http.Request, value any) (*http.Response, error) { + payload, err := json.Marshal(value) + if err != nil { + return nil, err + } + return &http.Response{ + StatusCode: http.StatusOK, + Header: make(http.Header), + Body: io.NopCloser(bytes.NewReader(payload)), + Request: r, + }, nil +} + +func jsonHTTPResponseWithSession( + r *http.Request, + value any, + sessionID string, +) (*http.Response, error) { + response, err := jsonHTTPResponse(r, value) + if err != nil { + return nil, err + } + response.Header.Set("x-agent-session-id", sessionID) + return response, nil +} + +func newHarnessClient(t *testing.T, adapter *httptest.Server) *harness.Client { + t.Helper() + client, err := harness.NewClient( + adapter.URL, + harness.WithBearerToken("adapter-auth-value"), + harness.WithControlTimeout(2*time.Second), + ) + if err != nil { + t.Fatalf("NewClient: %v", err) + } + return client +} + +func responsesStartTurnRequest(name string) harness.StartTurnRequest { + return harness.StartTurnRequest{ + Version: harness.ProtocolVersion, + Namespace: "default", + TaskName: name, + SessionName: name, + RuntimeSessionID: harness.RuntimeSessionID(name + "-runtime"), + TurnID: harness.HarnessTurnID(name + "-turn"), + CorrelationID: name + "-corr", + Deadline: time.Now().UTC().Add(time.Minute), + AuthIdentity: harness.AuthIdentity{Subject: "task:default/" + name}, + ToolExecutionMode: harness.ToolExecutionModeObserved, + Input: harness.TurnInput{Prompt: "Investigate incident"}, + } +} + +func cancelRequestForStart(start harness.StartTurnRequest) harness.CancelTurnRequest { + return harness.CancelTurnRequest{ + Version: harness.ProtocolVersion, + Namespace: start.Namespace, + TaskName: start.TaskName, + SessionName: start.SessionName, + RuntimeSessionID: start.RuntimeSessionID, + TurnID: start.TurnID, + CorrelationID: start.CorrelationID, + Reason: "test cancellation", + } +} + +func brokeredReadRequest(name string) harness.StartTurnRequest { + request := responsesStartTurnRequest(name) + request.ToolExecutionMode = harness.ToolExecutionModeBrokered + request.Input.Tools = []harness.ToolDefinition{{ + Name: "support-ticket-lookup", + Description: "Look up support ticket", + BrokeredClass: harness.BrokeredToolClassRead, + Parameters: json.RawMessage(`{"type":"object","properties":{"incident":{"type":"string"}}}`), + }} + return request +} + +func brokeredWriteRequest(name string) harness.StartTurnRequest { + request := responsesStartTurnRequest(name) + request.ToolExecutionMode = harness.ToolExecutionModeBrokered + request.Input.Tools = []harness.ToolDefinition{{ + Name: "dispatch-work-order", + Description: "Dispatch a work order", + BrokeredClass: harness.BrokeredToolClassWrite, + Parameters: json.RawMessage( + `{"type":"object","properties":{"incident":{"type":"string"}},"required":["incident"]}`, + ), + }} + return request +} + +func goldenContinueRequest( + request harness.StartTurnRequest, + callID string, + output json.RawMessage, +) harness.ContinueTurnRequest { + return harness.ContinueTurnRequest{ + Version: harness.ProtocolVersion, + Namespace: request.Namespace, + TaskName: request.TaskName, + SessionName: request.SessionName, + RuntimeSessionID: request.RuntimeSessionID, + TurnID: request.TurnID, + CorrelationID: request.CorrelationID, + ToolResults: []harness.ToolCallResult{toolResultForRequest(request, callID, true, output, nil)}, + } +} + +func toolResultForRequest( + request harness.StartTurnRequest, + callID string, + approved bool, + output json.RawMessage, + errInfo *harness.ErrorInfo, +) harness.ToolCallResult { + return harness.ToolCallResult{ + Version: harness.ProtocolVersion, + RuntimeSessionID: request.RuntimeSessionID, + TurnID: request.TurnID, + ToolCallID: callID, + IdempotencyKey: harness.ToolRequestIdempotencyKey(request.RuntimeSessionID, request.TurnID, callID), + Approved: approved, + Output: output, + Error: errInfo, + } +} + +func baseToolResult( + callID string, + approved bool, + output json.RawMessage, + errInfo *harness.ErrorInfo, +) harness.ToolCallResult { + return harness.ToolCallResult{ + Version: harness.ProtocolVersion, + RuntimeSessionID: "foundry-brokered-runtime", + TurnID: "foundry-brokered-turn", + ToolCallID: callID, + IdempotencyKey: harness.ToolRequestIdempotencyKey( + "foundry-brokered-runtime", + "foundry-brokered-turn", + callID, + ), + Approved: approved, + Output: output, + Error: errInfo, + } +} + +func streamCurrentFrames( + t *testing.T, + client *harness.Client, + turnID harness.HarnessTurnID, +) []harness.HarnessEventFrame { + t.Helper() + var frames []harness.HarnessEventFrame + ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond) + defer cancel() + err := client.StreamFrames(ctx, turnID, 0, func(frame harness.HarnessEventFrame) error { + frames = append(frames, frame) + return nil + }) + if err != nil && ctx.Err() == nil { + t.Fatalf("StreamFrames: %v", err) + } + return frames +} + +func hasFrameType(frames []harness.HarnessEventFrame, typ harness.FrameType) bool { + return findFrame(frames, typ) != nil +} + +func findFrame(frames []harness.HarnessEventFrame, typ harness.FrameType) *harness.HarnessEventFrame { + for i := range frames { + if frames[i].Type == typ { + return &frames[i] + } + } + return nil +} + +func findFrames(frames []harness.HarnessEventFrame, typ harness.FrameType) []harness.HarnessEventFrame { + out := []harness.HarnessEventFrame{} + for _, frame := range frames { + if frame.Type == typ { + out = append(out, frame) + } + } + return out +} + +func scrubFrameForGolden(frame harness.HarnessEventFrame) map[string]any { + encoded, _ := json.Marshal(frame) + var decoded map[string]any + _ = json.Unmarshal(encoded, &decoded) + delete(decoded, "createdAt") + return decoded +} + +func assertJSONFileEqual(t *testing.T, path string, actual any) { + t.Helper() + expectedBytes, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read fixture %s: %v", path, err) + } + expected := decodeJSONForCompare(t, expectedBytes) + actualBytes, err := json.Marshal(actual) + if err != nil { + t.Fatalf("marshal actual: %v", err) + } + actualValue := decodeJSONForCompare(t, actualBytes) + if !reflect.DeepEqual(expected, actualValue) { + expectedPretty, _ := json.MarshalIndent(expected, "", " ") + actualPretty, _ := json.MarshalIndent(actualValue, "", " ") + t.Fatalf("JSON mismatch for %s\nexpected: %s\nactual: %s", path, expectedPretty, actualPretty) + } +} + +func requestMap(t *testing.T, data json.RawMessage) map[string]any { + t.Helper() + decoded, ok := decodeJSONForCompare(t, data).(map[string]any) + if !ok { + t.Fatalf("request body is not a JSON object: %s", string(data)) + } + return decoded +} + +func decodeJSONForCompare(t *testing.T, data []byte) any { + t.Helper() + decoder := json.NewDecoder(bytes.NewReader(data)) + decoder.UseNumber() + var value any + if err := decoder.Decode(&value); err != nil { + t.Fatalf("decode JSON: %v\n%s", err, string(data)) + } + return value +} + +func decodeFixtureInto(t *testing.T, path string, out any) { + t.Helper() + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read fixture %s: %v", path, err) + } + if err := json.Unmarshal(data, out); err != nil { + t.Fatalf("decode fixture %s: %v", path, err) + } +} + +func writeJSON(w http.ResponseWriter, value any) { + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(value) +} + +func ioReadAll(r io.Reader) ([]byte, error) { + return io.ReadAll(r) +} diff --git a/examples/harness/foundry-responses/test_fetch_task_events.py b/examples/harness/foundry-responses/test_fetch_task_events.py new file mode 100644 index 000000000..7234485f4 --- /dev/null +++ b/examples/harness/foundry-responses/test_fetch_task_events.py @@ -0,0 +1,76 @@ +import importlib.util +import json +import subprocess +import unittest +from pathlib import Path + +MODULE_PATH = Path(__file__).with_name("fetch_task_events.py") +SPEC = importlib.util.spec_from_file_location("fetch_task_events", MODULE_PATH) +assert SPEC and SPEC.loader +MODULE = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(MODULE) + + +class FakeRunner: + def __init__(self, pages): + self.pages = pages + self.commands = [] + + def __call__(self, command, **kwargs): + self.commands.append((command, kwargs)) + after = int(command[command.index("--after") + 1]) + payload = self.pages[after] + return subprocess.CompletedProcess(command, 0, json.dumps(payload), "") + + +class FetchTaskEventsTests(unittest.TestCase): + def test_fetches_until_latest_sequence(self): + runner = FakeRunner( + { + 0: { + "namespace": "ns", + "streamType": "task", + "streamID": "task-a", + "afterSeq": 0, + "latestSeq": 3, + "events": [{"seq": 1}, {"seq": 2}], + }, + 2: { + "namespace": "ns", + "streamType": "task", + "streamID": "task-a", + "afterSeq": 2, + "latestSeq": 3, + "events": [{"seq": 3}], + }, + } + ) + + payload = MODULE.fetch_task_events("task-a", "ns", runner=runner) + + self.assertEqual([event["seq"] for event in payload["events"]], [1, 2, 3]) + self.assertEqual(payload["latestSeq"], 3) + self.assertEqual(payload["afterSeq"], 0) + self.assertEqual(len(runner.commands), 2) + self.assertIn("1000", runner.commands[0][0]) + + def test_accepts_empty_stream(self): + runner = FakeRunner({0: {"afterSeq": 0, "latestSeq": 0, "events": []}}) + payload = MODULE.fetch_task_events("task-a", "ns", runner=runner) + self.assertEqual(payload["events"], []) + + def test_rejects_empty_truncated_page(self): + runner = FakeRunner({0: {"afterSeq": 0, "latestSeq": 2, "events": []}}) + with self.assertRaisesRegex(RuntimeError, "stopped at sequence 0"): + MODULE.fetch_task_events("task-a", "ns", runner=runner) + + def test_rejects_sequence_gaps(self): + runner = FakeRunner( + {0: {"afterSeq": 0, "latestSeq": 2, "events": [{"seq": 2}]}} + ) + with self.assertRaisesRegex(ValueError, "sequence 2, expected 1"): + MODULE.fetch_task_events("task-a", "ns", runner=runner) + + +if __name__ == "__main__": + unittest.main() diff --git a/examples/harness/foundry-responses/testdata/agentkit-foundry-brokered/approval_declined_payload.json b/examples/harness/foundry-responses/testdata/agentkit-foundry-brokered/approval_declined_payload.json new file mode 100644 index 000000000..07aef6b58 --- /dev/null +++ b/examples/harness/foundry-responses/testdata/agentkit-foundry-brokered/approval_declined_payload.json @@ -0,0 +1,7 @@ +{ + "approved": false, + "error": { + "code": "approval_declined", + "message": "tool call was not approved" + } +} diff --git a/examples/harness/foundry-responses/testdata/agentkit-foundry-brokered/continuation_request.json b/examples/harness/foundry-responses/testdata/agentkit-foundry-brokered/continuation_request.json new file mode 100644 index 000000000..e5a8e9128 --- /dev/null +++ b/examples/harness/foundry-responses/testdata/agentkit-foundry-brokered/continuation_request.json @@ -0,0 +1,11 @@ +{ + "previous_response_id": "caresp_test", + "input": [ + { + "type": "function_call_output", + "call_id": "call_caresp_test_1", + "output": "{\"approved\":true,\"output\":{\"success\":true}}", + "status": "completed" + } + ] +} diff --git a/examples/harness/foundry-responses/testdata/agentkit-foundry-brokered/final_message_response.json b/examples/harness/foundry-responses/testdata/agentkit-foundry-brokered/final_message_response.json new file mode 100644 index 000000000..6d4786fef --- /dev/null +++ b/examples/harness/foundry-responses/testdata/agentkit-foundry-brokered/final_message_response.json @@ -0,0 +1,25 @@ +{ + "id": "caresp_final", + "object": "response", + "created_at": 0, + "status": "completed", + "model": "gpt-4o-mini", + "output": [ + { + "id": "msg_final", + "type": "message", + "status": "completed", + "role": "assistant", + "content": [ + { + "type": "output_text", + "text": "Brokered tool conformance_read completed with output: {\"success\":true}", + "annotations": [] + } + ], + "response_id": "caresp_final" + } + ], + "usage": {"input_tokens": 0, "output_tokens": 0, "total_tokens": 0}, + "previous_response_id": "caresp_test" +} diff --git a/examples/harness/foundry-responses/testdata/agentkit-foundry-brokered/function_call_response.json b/examples/harness/foundry-responses/testdata/agentkit-foundry-brokered/function_call_response.json new file mode 100644 index 000000000..adc0d9dc4 --- /dev/null +++ b/examples/harness/foundry-responses/testdata/agentkit-foundry-brokered/function_call_response.json @@ -0,0 +1,19 @@ +{ + "id": "caresp_test", + "object": "response", + "created_at": 0, + "status": "completed", + "model": "gpt-4o-mini", + "output": [ + { + "id": "fc_test", + "type": "function_call", + "call_id": "call_caresp_test_1", + "name": "conformance_read", + "arguments": "{\"probe\":true}", + "status": "completed", + "response_id": "caresp_test" + } + ], + "usage": {"input_tokens": 0, "output_tokens": 0, "total_tokens": 0} +} diff --git a/examples/harness/foundry-responses/testdata/agentkit-foundry-brokered/initial_request.json b/examples/harness/foundry-responses/testdata/agentkit-foundry-brokered/initial_request.json new file mode 100644 index 000000000..02c472731 --- /dev/null +++ b/examples/harness/foundry-responses/testdata/agentkit-foundry-brokered/initial_request.json @@ -0,0 +1,3 @@ +{ + "input": "please read telemetry" +} diff --git a/examples/harness/foundry-responses/testdata/agentkit-foundry-brokered/tool_execution_failure_payload.json b/examples/harness/foundry-responses/testdata/agentkit-foundry-brokered/tool_execution_failure_payload.json new file mode 100644 index 000000000..dc529a152 --- /dev/null +++ b/examples/harness/foundry-responses/testdata/agentkit-foundry-brokered/tool_execution_failure_payload.json @@ -0,0 +1,7 @@ +{ + "approved": false, + "error": { + "code": "tool_execution_failed", + "message": "tool execution failed" + } +} diff --git a/examples/harness/foundry-responses/testdata/agentkit-foundry-brokered/tool_policy_rejection_payload.json b/examples/harness/foundry-responses/testdata/agentkit-foundry-brokered/tool_policy_rejection_payload.json new file mode 100644 index 000000000..6ce46ae57 --- /dev/null +++ b/examples/harness/foundry-responses/testdata/agentkit-foundry-brokered/tool_policy_rejection_payload.json @@ -0,0 +1,7 @@ +{ + "approved": false, + "error": { + "code": "tool_policy_rejected", + "message": "tool call was rejected by policy" + } +} diff --git a/examples/harness/foundry-responses/testdata/golden/01_initial_hosted_request.json b/examples/harness/foundry-responses/testdata/golden/01_initial_hosted_request.json new file mode 100644 index 000000000..e717d54e2 --- /dev/null +++ b/examples/harness/foundry-responses/testdata/golden/01_initial_hosted_request.json @@ -0,0 +1,3 @@ +{ + "input": "Investigate incident" +} diff --git a/examples/harness/foundry-responses/testdata/golden/02_function_call_response.json b/examples/harness/foundry-responses/testdata/golden/02_function_call_response.json new file mode 100644 index 000000000..dd61e135f --- /dev/null +++ b/examples/harness/foundry-responses/testdata/golden/02_function_call_response.json @@ -0,0 +1,13 @@ +{ + "id": "resp-1", + "status": "completed", + "agent_session_id": "session-1", + "output": [ + { + "type": "function_call", + "call_id": "call-1", + "name": "support-ticket-lookup", + "arguments": "{\"incident\":\"inc-1\"}" + } + ] +} diff --git a/examples/harness/foundry-responses/testdata/golden/03_tool_call_requested_frame.json b/examples/harness/foundry-responses/testdata/golden/03_tool_call_requested_frame.json new file mode 100644 index 000000000..b2bc39125 --- /dev/null +++ b/examples/harness/foundry-responses/testdata/golden/03_tool_call_requested_frame.json @@ -0,0 +1,13 @@ +{ + "version": "orka.harness.v1", + "type": "ToolCallRequested", + "runtimeSessionID": "foundry-brokered-runtime", + "turnID": "foundry-brokered-turn", + "correlationID": "foundry-brokered-corr", + "seq": 2, + "summary": "foundry hosted tool call requested", + "content": {"incident":"inc-1"}, + "toolName": "support-ticket-lookup", + "toolCallID": "call-1", + "metadata": {"backend":"foundry-responses"} +} diff --git a/examples/harness/foundry-responses/testdata/golden/04_orka_continue_request.json b/examples/harness/foundry-responses/testdata/golden/04_orka_continue_request.json new file mode 100644 index 000000000..8ba6d6990 --- /dev/null +++ b/examples/harness/foundry-responses/testdata/golden/04_orka_continue_request.json @@ -0,0 +1,20 @@ +{ + "version": "orka.harness.v1", + "namespace": "default", + "taskName": "foundry-brokered", + "sessionName": "foundry-brokered", + "runtimeSessionID": "foundry-brokered-runtime", + "turnID": "foundry-brokered-turn", + "correlationID": "foundry-brokered-corr", + "toolResults": [ + { + "version": "orka.harness.v1", + "runtimeSessionID": "foundry-brokered-runtime", + "turnID": "foundry-brokered-turn", + "toolCallID": "call-1", + "idempotencyKey": "foundry-brokered-runtime:foundry-brokered-turn:call-1", + "approved": true, + "output": {"success":true} + } + ] +} diff --git a/examples/harness/foundry-responses/testdata/golden/05_hosted_continuation_request.json b/examples/harness/foundry-responses/testdata/golden/05_hosted_continuation_request.json new file mode 100644 index 000000000..1c67dca1f --- /dev/null +++ b/examples/harness/foundry-responses/testdata/golden/05_hosted_continuation_request.json @@ -0,0 +1,13 @@ +{ + "agent_session_id": "session-1", + "brokered_continuation_proof": "proof-for-test", + "input": [ + { + "call_id": "call-1", + "output": "{\"approved\":true,\"output\":{\"success\":true}}", + "status": "completed", + "type": "function_call_output" + } + ], + "previous_response_id": "resp-1" +} diff --git a/examples/harness/foundry-responses/testdata/golden/06_final_message_response.json b/examples/harness/foundry-responses/testdata/golden/06_final_message_response.json new file mode 100644 index 000000000..8086699ac --- /dev/null +++ b/examples/harness/foundry-responses/testdata/golden/06_final_message_response.json @@ -0,0 +1,14 @@ +{ + "id": "resp-2", + "status": "completed", + "agent_session_id": "session-1", + "output": [ + { + "type": "message", + "role": "assistant", + "content": [ + {"type":"output_text","text":"foundry final answer"} + ] + } + ] +} diff --git a/examples/harness/foundry-responses/testdata/golden/07_approval_declined_output.json b/examples/harness/foundry-responses/testdata/golden/07_approval_declined_output.json new file mode 100644 index 000000000..b0cadb210 --- /dev/null +++ b/examples/harness/foundry-responses/testdata/golden/07_approval_declined_output.json @@ -0,0 +1,6 @@ +{ + "type": "function_call_output", + "call_id": "call-1", + "output": "{\"approved\":false,\"error\":{\"code\":\"approval_declined\",\"message\":\"tool call was not approved\"}}", + "status": "completed" +} diff --git a/examples/harness/foundry-responses/testdata/golden/08_tool_policy_rejection_output.json b/examples/harness/foundry-responses/testdata/golden/08_tool_policy_rejection_output.json new file mode 100644 index 000000000..acf356a26 --- /dev/null +++ b/examples/harness/foundry-responses/testdata/golden/08_tool_policy_rejection_output.json @@ -0,0 +1,6 @@ +{ + "type": "function_call_output", + "call_id": "call-1", + "output": "{\"approved\":false,\"error\":{\"code\":\"tool_policy_rejected\",\"message\":\"tool call was rejected by policy\"}}", + "status": "completed" +} diff --git a/examples/harness/foundry-responses/testdata/golden/09_tool_execution_failure_output.json b/examples/harness/foundry-responses/testdata/golden/09_tool_execution_failure_output.json new file mode 100644 index 000000000..e0ce3703d --- /dev/null +++ b/examples/harness/foundry-responses/testdata/golden/09_tool_execution_failure_output.json @@ -0,0 +1,6 @@ +{ + "type": "function_call_output", + "call_id": "call-1", + "output": "{\"approved\":false,\"error\":{\"code\":\"tool_execution_failed\",\"message\":\"tool execution failed\"}}", + "status": "completed" +} diff --git a/examples/harness/foundry-responses/testdata/golden/10_multiple_calls_response.json b/examples/harness/foundry-responses/testdata/golden/10_multiple_calls_response.json new file mode 100644 index 000000000..cf2ba79c3 --- /dev/null +++ b/examples/harness/foundry-responses/testdata/golden/10_multiple_calls_response.json @@ -0,0 +1,9 @@ +{ + "id": "resp-1", + "status": "completed", + "agent_session_id": "session-1", + "output": [ + {"type":"function_call","call_id":"call-1","name":"support-ticket-lookup","arguments":"{\"incident\":\"inc-1\"}"}, + {"type":"function_call","call_id":"call-2","name":"support-ticket-lookup","arguments":"{\"incident\":\"inc-2\"}"} + ] +} diff --git a/examples/harness/foundry-responses/validate.sh b/examples/harness/foundry-responses/validate.sh new file mode 100755 index 000000000..5a4cf42c8 --- /dev/null +++ b/examples/harness/foundry-responses/validate.sh @@ -0,0 +1,323 @@ +#!/usr/bin/env bash +set -euo pipefail + +usage() { + cat >&2 <<'USAGE' +Usage: examples/harness/foundry-responses/validate.sh [--agentkit PATH] [--full] + +Runs deterministic local validation for the Orka Foundry hosted Responses adapter. +No live Foundry endpoint, token, API key, or Kubernetes cluster is required. + +Options: + --agentkit PATH Also run AgentKit's deterministic Foundry brokered protocol + tests from PATH/runtimes/common. This is explicit because a + sibling AgentKit checkout may contain unrelated local changes. + --full Also run `make test` for the full non-e2e Orka suite. + -h, --help Show this help. +USAGE +} + +repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)" +agentkit_root="" +run_full=0 + +while [[ $# -gt 0 ]]; do + case "$1" in + --agentkit) + [[ $# -ge 2 ]] || { echo "--agentkit requires a path" >&2; exit 2; } + agentkit_root="$2" + shift 2 + ;; + --full) + run_full=1 + shift + ;; + -h|--help) + usage + exit 0 + ;; + *) + echo "unknown argument: $1" >&2 + usage + exit 2 + ;; + esac +done + +run() { + printf '\n==> %s\n' "$*" >&2 + "$@" +} + +cd "$repo_root" + +grep -qx '!examples/harness/foundry-responses/\*.go' .dockerignore || { + echo "Foundry Responses Go sources are not included in the Docker build context" >&2 + exit 1 +} + +run go test ./examples/harness/foundry-responses -count=1 +run go test \ + ./examples/harness/foundry \ + ./examples/harness/foundry-responses \ + ./examples/harness/echo \ + ./internal/harness \ + ./internal/harness/conformance +run go test ./internal/controller -run 'Test.*(AgentRuntime|Harness|Brokered|Runtime)' +run python3 -m unittest examples/harness/foundry-responses/test_fetch_task_events.py +while IFS= read -r -d '' script; do + run bash -n "$script" +done < <(find examples -type f -name '*.sh' -print0 | sort -z) + +expect_verifier_failure() { + local fixture="$1" + local expected="$2" + local label="$3" + local out_file err_file code + out_file="$(mktemp)" + err_file="$(mktemp)" + set +e + examples/fibey-custom-agent-demo/verify-foundry-responses.sh --json "$fixture" >"$out_file" 2>"$err_file" + code=$? + set -e + if [[ "$code" == "0" ]]; then + cat "$out_file" >&2 + rm -f "$out_file" "$err_file" + echo "expected ${label} verifier fixture to fail" >&2 + exit 1 + fi + if ! grep -q "$expected" "$err_file"; then + cat "$err_file" >&2 + rm -f "$out_file" "$err_file" + echo "${label} fixture failed for the wrong reason" >&2 + exit 1 + fi + rm -f "$out_file" "$err_file" +} + +live_smoke_err="$(mktemp)" +set +e +ORKA_FOUNDRY_RESPONSES_ENDPOINT="https://:443/agents/test/endpoint/protocols/openai/responses" \ + ORKA_FOUNDRY_RESPONSES_PROJECT_ENDPOINT="" \ + ORKA_FOUNDRY_RESPONSES_AGENT_NAME="" \ + ORKA_FOUNDRY_RESPONSES_API_KEY="placeholder" \ + ORKA_FOUNDRY_RESPONSES_AUTH_BEARER="" \ + examples/harness/foundry-responses/live-smoke.sh >/dev/null 2>"$live_smoke_err" +live_smoke_code=$? +set -e +if [[ "$live_smoke_code" == "0" ]] || ! grep -q "safe /responses URL" "$live_smoke_err"; then + cat "$live_smoke_err" >&2 + rm -f "$live_smoke_err" + echo "expected empty-hostname live smoke preflight to fail" >&2 + exit 1 +fi +rm -f "$live_smoke_err" + +missing_image_err="$(mktemp)" +set +e +ORKA_FOUNDRY_RESPONSES_ENDPOINT="http://127.0.0.1/agents/test/endpoint/protocols/openai/responses" \ + ORKA_FOUNDRY_RESPONSES_API_KEY="placeholder" \ + ORKA_FOUNDRY_RESPONSES_AUTH_BEARER="" \ + ORKA_FOUNDRY_RESPONSES_ADAPTER_IMAGE="" \ + ORKA_FOUNDRY_RESPONSES_BROKERED_TOOL_CLASSES="" \ + examples/harness/foundry-responses/live-smoke.sh --apply >/dev/null 2>"$missing_image_err" +missing_image_code=$? +set -e +if [[ "$missing_image_code" == "0" ]] || ! grep -q "ADAPTER_IMAGE is required" "$missing_image_err"; then + cat "$missing_image_err" >&2 + rm -f "$missing_image_err" + echo "expected live smoke apply without an explicit image to fail" >&2 + exit 1 +fi +rm -f "$missing_image_err" + +missing_proof_err="$(mktemp)" +set +e +ORKA_FOUNDRY_RESPONSES_ENDPOINT="http://127.0.0.1/agents/test/endpoint/protocols/openai/responses" \ + ORKA_FOUNDRY_RESPONSES_API_KEY="placeholder" \ + ORKA_FOUNDRY_RESPONSES_AUTH_BEARER="" \ + ORKA_FOUNDRY_RESPONSES_BROKERED_TOOL_CLASSES="read" \ + ORKA_FOUNDRY_RESPONSES_BROKERED_CONTINUATION_PROOF="" \ + examples/harness/foundry-responses/live-smoke.sh >/dev/null 2>"$missing_proof_err" +missing_proof_code=$? +set -e +if [[ "$missing_proof_code" == "0" ]] || ! grep -q "CONTINUATION_PROOF is required" "$missing_proof_err"; then + cat "$missing_proof_err" >&2 + rm -f "$missing_proof_err" + echo "expected brokered live smoke without continuation proof to fail" >&2 + exit 1 +fi +rm -f "$missing_proof_err" + +whitespace_proof_err="$(mktemp)" +set +e +ORKA_FOUNDRY_RESPONSES_ENDPOINT="http://127.0.0.1/agents/test/endpoint/protocols/openai/responses" \ + ORKA_FOUNDRY_RESPONSES_API_KEY="placeholder" \ + ORKA_FOUNDRY_RESPONSES_AUTH_BEARER="" \ + ORKA_FOUNDRY_RESPONSES_BROKERED_TOOL_CLASSES="read" \ + ORKA_FOUNDRY_RESPONSES_BROKERED_CONTINUATION_PROOF=" " \ + examples/harness/foundry-responses/live-smoke.sh >/dev/null 2>"$whitespace_proof_err" +whitespace_proof_code=$? +set -e +if [[ "$whitespace_proof_code" == "0" ]] || ! grep -q "CONTINUATION_PROOF is required" "$whitespace_proof_err"; then + cat "$whitespace_proof_err" >&2 + rm -f "$whitespace_proof_err" + echo "expected whitespace-only brokered continuation proof to fail" >&2 + exit 1 +fi +rm -f "$whitespace_proof_err" + +invalid_class_err="$(mktemp)" +set +e +ORKA_FOUNDRY_RESPONSES_ENDPOINT="http://127.0.0.1/agents/test/endpoint/protocols/openai/responses" \ + ORKA_FOUNDRY_RESPONSES_API_KEY="placeholder" \ + ORKA_FOUNDRY_RESPONSES_AUTH_BEARER="" \ + ORKA_FOUNDRY_RESPONSES_BROKERED_TOOL_CLASSES="r ead" \ + examples/harness/foundry-responses/live-smoke.sh >/dev/null 2>"$invalid_class_err" +invalid_class_code=$? +set -e +if [[ "$invalid_class_code" == "0" ]] || ! grep -q "unsupported brokered class 'r ead'" "$invalid_class_err"; then + cat "$invalid_class_err" >&2 + rm -f "$invalid_class_err" + echo "expected internally spaced brokered class to fail preflight" >&2 + exit 1 +fi +rm -f "$invalid_class_err" + +smoke_tmp="$(mktemp -d)" +smoke_capture="$smoke_tmp/rendered.yaml" +cat >"$smoke_tmp/kubectl" <<'SH' +#!/usr/bin/env bash +set -euo pipefail +if [[ " $* " == *" apply "* ]]; then + cat >>"$CAPTURE_FILE" + printf '\n---\n' >>"$CAPTURE_FILE" + exit 0 +fi +if [[ "${1:-}" == "get" && "${2:-}" == "namespace" ]]; then + exit 0 +fi +if [[ " $* " == *" get deployment/"* ]]; then + exit 1 +fi +exit 0 +SH +chmod +x "$smoke_tmp/kubectl" +PATH="$smoke_tmp:$PATH" \ + CAPTURE_FILE="$smoke_capture" \ + ORKA_FOUNDRY_RESPONSES_ENDPOINT="http://127.0.0.1/agents/test/endpoint/protocols/openai/responses" \ + ORKA_FOUNDRY_RESPONSES_API_KEY="placeholder" \ + ORKA_FOUNDRY_RESPONSES_AUTH_BEARER="" \ + ORKA_FOUNDRY_RESPONSES_ADAPTER_BEARER_TOKEN="adapter-placeholder" \ + ORKA_FOUNDRY_RESPONSES_ADAPTER_IMAGE="example.invalid/foundry-adapter:test" \ + ORKA_FOUNDRY_RESPONSES_BROKERED_TOOL_CLASSES="" \ + examples/harness/foundry-responses/live-smoke.sh --apply >/dev/null 2>"$smoke_tmp/stderr" +if grep -q '^ - brokered$' "$smoke_capture" || \ + grep -q '^ brokeredToolClasses:' "$smoke_capture" || \ + ! grep -q '^ supportsContinuation: false$' "$smoke_capture"; then + cat "$smoke_capture" >&2 + rm -rf "$smoke_tmp" + echo "observed-only live smoke rendered incompatible brokered capabilities" >&2 + exit 1 +fi +rm -rf "$smoke_tmp" + +run examples/fibey-custom-agent-demo/verify-foundry-responses.sh \ + --json examples/fibey-custom-agent-demo/testdata/foundry-responses-events-pass.json +expect_verifier_failure \ + examples/fibey-custom-agent-demo/testdata/foundry-responses-events-missing-write-exec.json \ + "missing write ToolCallStarted event after approval" \ + "missing-write" +expect_verifier_failure \ + examples/fibey-custom-agent-demo/testdata/foundry-responses-events-duplicate-write.json \ + "duplicate write execution starts for dispatch-work-order" \ + "duplicate-write" +expect_verifier_failure \ + examples/fibey-custom-agent-demo/testdata/foundry-responses-events-missing-approval-decision.json \ + "missing ApprovalApproved event" \ + "missing-approval-decision" +expect_verifier_failure \ + examples/fibey-custom-agent-demo/testdata/foundry-responses-events-declined-write.json \ + "follows ApprovalDeclined" \ + "declined-write" +expect_verifier_failure \ + examples/fibey-custom-agent-demo/testdata/foundry-responses-events-missing-approval-id.json \ + "is missing approvalID" \ + "missing-approval-id" +expect_verifier_failure \ + examples/fibey-custom-agent-demo/testdata/foundry-responses-events-decision-before-request.json \ + "has no matching preceding ApprovalApproved" \ + "decision-before-request" +expect_verifier_failure \ + examples/fibey-custom-agent-demo/testdata/foundry-responses-events-overlapping-write-marker.json \ + "missing write ToolCallStarted event after approval" \ + "overlapping-write-marker" +expect_verifier_failure \ + examples/fibey-custom-agent-demo/testdata/foundry-responses-events-mismatched-write-request.json \ + "has no matching preceding mapped request" \ + "mismatched-write-request" +expect_verifier_failure \ + examples/fibey-custom-agent-demo/testdata/foundry-responses-events-partial-idempotency.json \ + "write execution for escalate-incident is missing execution idempotency key evidence" \ + "partial-idempotency" +expect_verifier_failure \ + examples/fibey-custom-agent-demo/testdata/foundry-responses-events-generic-idempotency-only.json \ + "write execution for dispatch-work-order is missing execution idempotency key evidence" \ + "generic-idempotency-only" +expect_verifier_failure \ + examples/fibey-custom-agent-demo/testdata/foundry-responses-events-failure-after-start.json \ + "write execution for dispatch-work-order has matching ToolCallFailed" \ + "failure-after-start" +expect_verifier_failure \ + examples/fibey-custom-agent-demo/testdata/foundry-responses-events-completion-after-terminal.json \ + "write execution for dispatch-work-order is missing ToolCallCompleted" \ + "completion-after-terminal" +expect_verifier_failure \ + examples/fibey-custom-agent-demo/testdata/foundry-responses-events-truncated-page.json \ + "event JSON is incomplete" \ + "truncated-event-page" +expect_verifier_failure \ + examples/fibey-custom-agent-demo/testdata/foundry-responses-events-tail-page.json \ + "afterSeq must be 0" \ + "tail-event-page" +expect_verifier_failure \ + examples/fibey-custom-agent-demo/testdata/foundry-responses-events-write-after-terminal.json \ + "terminal completion event does not follow all write executions" \ + "write-after-terminal" +expect_verifier_failure \ + examples/fibey-custom-agent-demo/testdata/foundry-responses-events-terminal-write-terminal.json \ + "terminal completion event does not follow all write executions" \ + "terminal-write-terminal" +expect_verifier_failure \ + examples/fibey-custom-agent-demo/testdata/foundry-responses-events-task-failed.json \ + "final Task lifecycle outcome is TaskFailed" \ + "task-failed-after-runtime-completion" +expect_verifier_failure \ + examples/fibey-custom-agent-demo/testdata/foundry-responses-events-success-before-runtime-completion.json \ + "final execution event is AgentRuntimeCompleted" \ + "task-success-before-runtime-completion" + +if [[ "$run_full" == "1" ]]; then + run make test +fi + +if [[ -n "$agentkit_root" ]]; then + common_dir="$agentkit_root/runtimes/common" + if [[ ! -d "$common_dir" ]]; then + echo "AgentKit common runtime directory not found: $common_dir" >&2 + exit 2 + fi + if ! command -v uv >/dev/null 2>&1; then + echo "uv is required for AgentKit validation but was not found in PATH" >&2 + exit 2 + fi + ( + cd "$common_dir" + run uv run --extra dev pytest -q \ + tests/test_foundry_brokered_protocol.py \ + tests/test_brokered_schema.py \ + tests/test_foundry_protocol.py + ) +fi + +printf '\nFoundry hosted Responses local validation passed.\n' >&2 diff --git a/examples/harness/foundry/README.md b/examples/harness/foundry/README.md index 92204a099..f30512562 100644 --- a/examples/harness/foundry/README.md +++ b/examples/harness/foundry/README.md @@ -1,6 +1,28 @@ -# Foundry AgentRuntime adapter +# Foundry Assistants/threads AgentRuntime adapter -This example adapter presents Azure AI Foundry hosted agents as an `orka.harness.v1` AgentRuntime endpoint. It keeps Foundry-specific IDs and credentials inside the adapter deployment while Orka keeps owning task lifecycle, brokered tool policy, approvals, idempotency, and result storage. +This example adapter presents the Azure AI Foundry/OpenAI **Assistants threads/runs** protocol as an `orka.harness.v1` AgentRuntime endpoint. + +It is intentionally **not** the adapter for AgentKit agents deployed as Foundry hosted agents. Hosted AgentKit uses endpoint-scoped Responses and must be driven by `examples/harness/foundry-responses` instead. + +## Protocol target + +This adapter uses the Assistants-style API shape: + +```text +POST /threads +POST /threads/{threadID}/runs with assistant_id +GET /threads/{threadID}/runs/{runID} +POST /threads/{threadID}/runs/{runID}/submit_tool_outputs +``` + +Because it targets Assistants/runs, it may send safe Orka tool schemas in the run request and it requires a real Assistants `assistant_id`. Do not set `ORKA_FOUNDRY_AGENT_ID` to a hosted AgentKit agent name; hosted agents are not `asst_*` Assistants resources. + +For Foundry hosted AgentKit over Responses: + +- use `examples/harness/foundry-responses`; +- do not send request-level `tools`; +- statically configure safe brokered schemas in AgentKit; +- resume with `function_call_output` and `previous_response_id`. ## Configuration @@ -9,20 +31,20 @@ This example adapter presents Azure AI Foundry hosted agents as an `orka.harness | `ORKA_FOUNDRY_ADAPTER_ADDR` | HTTP listen address, default `:8090`. | | `ORKA_FOUNDRY_RUNTIME_NAME` | Runtime name advertised in `/v1/capabilities`. | | `ORKA_FOUNDRY_ADAPTER_BEARER_TOKEN` | Bearer token Orka uses for mutating harness endpoints. | -| `ORKA_FOUNDRY_ENDPOINT` | Foundry agents endpoint base URL. Must be HTTPS in production; plain HTTP is accepted only for loopback/local tests. Do not include userinfo, query strings, or fragments. Put API versions in `ORKA_FOUNDRY_API_VERSION`. | -| `ORKA_FOUNDRY_AGENT_ID` | Foundry hosted-agent ID. | +| `ORKA_FOUNDRY_ENDPOINT` | Foundry Assistants endpoint base URL. Must be HTTPS in production; plain HTTP is accepted only for loopback/local tests. Do not include userinfo, query strings, or fragments. Put API versions in `ORKA_FOUNDRY_API_VERSION`. | +| `ORKA_FOUNDRY_AGENT_ID` | Assistants `assistant_id` for the thread run. | | `ORKA_FOUNDRY_API_KEY` | Foundry API key, sent as `api-key`. | | `ORKA_FOUNDRY_AUTH_BEARER` | Optional bearer auth alternative. | | `ORKA_FOUNDRY_API_VERSION` | API version query parameter, default `v1`. | Use Kubernetes Secrets for every token/key value. Do not put Foundry credentials in `AgentRuntime` CRDs. -For `AgentRuntime` readiness/conformance, the hosted Foundry agent must follow Orka brokered-tool probe prompts by calling exactly one provided function tool (for example `conformance_read` or `conformance_write`) and completing after Orka returns the brokered result. The adapter enforces that Foundry can only request tools supplied in the current `StartTurnRequest.input.tools` payload. +For `AgentRuntime` readiness/conformance, the Assistants runtime must follow Orka brokered-tool probe prompts by calling exactly one provided function tool (for example `conformance_read` or `conformance_write`) and completing after Orka returns the brokered result. The adapter enforces that Foundry can only request tools supplied in the current `StartTurnRequest.input.tools` payload. ## Protocol mapping - `StartTurnRequest` creates a Foundry thread and run. -- Orka safe `input.tools` schemas are passed to Foundry as function definitions only; Orka Tool URLs and credentials are never sent. +- In brokered mode, Orka safe `input.tools` schemas are passed to the Assistants run as function definitions only; Orka Tool URLs and credentials are never sent. - Foundry `requires_action.submit_tool_outputs.tool_calls[]` is mapped to `ToolCallRequested` frames. - `/v1/turns/{turnID}/continue` submits Orka-brokered tool outputs back to Foundry. - Foundry completion messages are mapped to `TurnCompleted`. diff --git a/go.mod b/go.mod index d4eeedcf7..f9f6099b9 100644 --- a/go.mod +++ b/go.mod @@ -144,7 +144,7 @@ require ( github.com/valyala/fastjson v1.6.10 // indirect github.com/x448/float16 v0.8.4 // indirect github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect - github.com/yuin/goldmark v1.7.13 // indirect + github.com/yuin/goldmark v1.7.17 // indirect github.com/yuin/goldmark-emoji v1.0.6 // indirect go.opentelemetry.io/auto/sdk v1.2.1 // indirect go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.65.0 // indirect diff --git a/go.sum b/go.sum index fdde1bef0..a37a79a16 100644 --- a/go.sum +++ b/go.sum @@ -310,8 +310,8 @@ github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e h1:JVG44RsyaB9T2KIHavM github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e/go.mod h1:RbqR21r5mrJuqunuUZ/Dhy/avygyECGrLceyNeo4LiM= github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU= github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E= -github.com/yuin/goldmark v1.7.13 h1:GPddIs617DnBLFFVJFgpo1aBfe/4xcvMc3SB5t/D0pA= -github.com/yuin/goldmark v1.7.13/go.mod h1:ip/1k0VRfGynBgxOz0yCqHrbZXhcjxyuS66Brc7iBKg= +github.com/yuin/goldmark v1.7.17 h1:p36OVWwRb246iHxA/U4p8OPEpOTESm4n+g+8t0EE5uA= +github.com/yuin/goldmark v1.7.17/go.mod h1:ip/1k0VRfGynBgxOz0yCqHrbZXhcjxyuS66Brc7iBKg= github.com/yuin/goldmark-emoji v1.0.6 h1:QWfF2FYaXwL74tfGOW5izeiZepUDroDJfWubQI9HTHs= github.com/yuin/goldmark-emoji v1.0.6/go.mod h1:ukxJDKFpdFb5x0a5HqbdlcKtebh086iJpI31LTKmWuA= go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64= diff --git a/internal/controller/harness_wrapper.go b/internal/controller/harness_wrapper.go index e4e064510..4577f6ca4 100644 --- a/internal/controller/harness_wrapper.go +++ b/internal/controller/harness_wrapper.go @@ -57,6 +57,7 @@ const ( harnessWrapperCorrelationIDAnno = "orka.ai/harness-wrapper-correlation-id" harnessWrapperLastFrameSeqAnno = "orka.ai/harness-wrapper-last-frame-seq" harnessWrapperStartedAnno = "orka.ai/harness-wrapper-started" + harnessWrapperSubmissionAttemptedAnno = "orka.ai/harness-wrapper-submission-attempted" harnessWrapperPlannedAtAnno = "orka.ai/harness-wrapper-planned-at" harnessWrapperMetadataAnno = "orka.ai/harness-wrapper-metadata" harnessWrapperRuntimeRefAnno = "orka.ai/harness-wrapper-runtime-ref" @@ -81,6 +82,16 @@ func taskHasHarnessWrapperTurn(task *corev1alpha1.Task) bool { taskHasPlannedHarnessWrapperTurn(task) } +func taskHasHarnessWrapperSubmissionAttempt(task *corev1alpha1.Task) bool { + if task == nil || task.Annotations == nil { + return false + } + return strings.EqualFold( + strings.TrimSpace(task.Annotations[harnessWrapperSubmissionAttemptedAnno]), + scheduledRunLabelValue, + ) +} + func taskHasPlannedHarnessWrapperTurn(task *corev1alpha1.Task) bool { if task == nil || task.Annotations == nil { return false @@ -418,7 +429,7 @@ func (r *TaskReconciler) runHarnessWrapperTask(ctx context.Context, task *corev1 var err error startedPlannedTurn := false if taskHasPlannedHarnessWrapperTurn(task) { - if taskHasHarnessWrapperTurn(task) { + if taskHasHarnessWrapperTurn(task) || taskHasHarnessWrapperSubmissionAttempt(task) { startedPlannedTurn = true } request, err = r.plannedHarnessWrapperStartTurnRequest(ctx, task, agent, now.Time) @@ -509,7 +520,7 @@ func (r *TaskReconciler) runHarnessWrapperTask(ctx context.Context, task *corev1 if !hasFrames { if err := r.validateHarnessWrapperCapabilities(ctx, client, request); err != nil { if target.RuntimeRefName != "" && harnessWrapperAuthError(err) { - if shouldWait, waitErr := r.waitForHarnessWrapperAuthRetry(ctx, task); waitErr != nil { + if shouldWait, waitErr := r.waitForHarnessWrapperAuthRetry(ctx, task, false); waitErr != nil { return ctrl.Result{}, waitErr } else if shouldWait { return ctrl.Result{RequeueAfter: time.Second}, nil @@ -520,6 +531,9 @@ func (r *TaskReconciler) runHarnessWrapperTask(ctx context.Context, task *corev1 } return r.failTask(ctx, task, err.Error()) } + if err := r.patchHarnessWrapperSubmissionAttempted(ctx, task); err != nil { + return ctrl.Result{}, err + } if _, err := client.StartTurn(ctx, request); err != nil { message := err.Error() switch { @@ -535,7 +549,7 @@ func (r *TaskReconciler) runHarnessWrapperTask(ctx context.Context, task *corev1 } return ctrl.Result{RequeueAfter: time.Second}, nil case target.RuntimeRefName != "" && harnessWrapperAuthError(err): - if wait, waitErr := r.waitForHarnessWrapperAuthRetry(ctx, task); waitErr != nil { + if wait, waitErr := r.waitForHarnessWrapperAuthRetry(ctx, task, true); waitErr != nil { return ctrl.Result{}, waitErr } else if wait { return ctrl.Result{RequeueAfter: time.Second}, nil @@ -549,7 +563,14 @@ func (r *TaskReconciler) runHarnessWrapperTask(ctx context.Context, task *corev1 } } turnAccepted = true - if err := r.patchHarnessWrapperStarted(ctx, task); err != nil { + if !taskHasHarnessWrapperTurn(task) { + if err := r.patchHarnessWrapperStarted(ctx, task, false); err != nil { + return ctrl.Result{}, err + } + } + } + if startedPlannedTurn && !taskHasHarnessWrapperTurn(task) { + if err := r.patchHarnessWrapperStarted(ctx, task, taskHasHarnessWrapperSubmissionAttempt(task)); err != nil { return ctrl.Result{}, err } } @@ -708,12 +729,20 @@ func (r *TaskReconciler) finishHarnessWrapperTask(ctx context.Context, task *cor } if err != nil && result.Completed == nil && result.Failed == nil && !result.Cancelled { if target.RuntimeRefName != "" && harnessWrapperAuthError(err) { - if wait, waitErr := r.waitForHarnessWrapperAuthRetry(ctx, task); waitErr != nil { + if wait, waitErr := r.waitForHarnessWrapperAuthRetry(ctx, task, false); waitErr != nil { return ctrl.Result{}, waitErr } else if wait { return ctrl.Result{RequeueAfter: time.Second}, nil } } + if harnessWrapperStreamErrorIsMissingTurn(err) && taskHasHarnessWrapperSubmissionAttempt(task) { + return r.completeTask( + ctx, + task, + corev1alpha1.TaskPhaseFailed, + "harness runtime lost a turn after StartTurn submission outcome became unknown", + ) + } if harnessWrapperStreamErrorIsMissingTurn(err) && r.shouldRetry(task) { if clearErr := r.clearHarnessWrapperTurnState(ctx, task); clearErr != nil { return ctrl.Result{}, clearErr @@ -926,7 +955,11 @@ func harnessWrapperAuthRetries(task *corev1alpha1.Task) int { return retries } -func (r *TaskReconciler) waitForHarnessWrapperAuthRetry(ctx context.Context, task *corev1alpha1.Task) (bool, error) { +func (r *TaskReconciler) waitForHarnessWrapperAuthRetry( + ctx context.Context, + task *corev1alpha1.Task, + clearSubmissionAttempt bool, +) (bool, error) { retries := harnessWrapperAuthRetries(task) if retries >= harnessWrapperMaxAuthRetries { return false, nil @@ -936,6 +969,9 @@ func (r *TaskReconciler) waitForHarnessWrapperAuthRetry(ctx context.Context, tas task.Annotations = map[string]string{} } task.Annotations[harnessWrapperAuthRetriesAnno] = strconv.Itoa(retries + 1) + if clearSubmissionAttempt { + delete(task.Annotations, harnessWrapperSubmissionAttemptedAnno) + } if err := r.Patch(ctx, task, patch); err != nil { return false, err } @@ -1015,6 +1051,7 @@ func (r *TaskReconciler) patchHarnessWrapperPlannedTurn( task.Annotations[harnessWrapperCorrelationIDAnno] = request.CorrelationID task.Annotations[harnessWrapperLastFrameSeqAnno] = "0" task.Annotations[harnessWrapperStartedAnno] = "false" + delete(task.Annotations, harnessWrapperSubmissionAttemptedAnno) task.Annotations[harnessWrapperPlannedAtAnno] = time.Now().UTC().Format(time.RFC3339Nano) if runtimeRefName := strings.TrimSpace(request.Metadata["runtimeRef"]); runtimeRefName != "" { task.Annotations[harnessWrapperRuntimeRefAnno] = runtimeRefName @@ -1043,7 +1080,28 @@ func (r *TaskReconciler) patchHarnessWrapperPlannedTurn( return r.Patch(ctx, task, patch) } -func (r *TaskReconciler) patchHarnessWrapperStarted(ctx context.Context, task *corev1alpha1.Task) error { +func (r *TaskReconciler) patchHarnessWrapperSubmissionAttempted(ctx context.Context, task *corev1alpha1.Task) error { + latest := &corev1alpha1.Task{} + if err := r.Get(ctx, ctrlclient.ObjectKey{Name: task.Name, Namespace: task.Namespace}, latest); err != nil { + return err + } + patch := ctrlclient.MergeFrom(latest.DeepCopy()) + if latest.Annotations == nil { + latest.Annotations = map[string]string{} + } + latest.Annotations[harnessWrapperSubmissionAttemptedAnno] = scheduledRunLabelValue + if err := r.Patch(ctx, latest, patch); err != nil { + return err + } + latest.DeepCopyInto(task) + return nil +} + +func (r *TaskReconciler) patchHarnessWrapperStarted( + ctx context.Context, + task *corev1alpha1.Task, + preserveSubmissionAttempt bool, +) error { latest := &corev1alpha1.Task{} if err := r.Get(ctx, ctrlclient.ObjectKey{Name: task.Name, Namespace: task.Namespace}, latest); err != nil { return err @@ -1067,6 +1125,9 @@ func (r *TaskReconciler) patchHarnessWrapperStarted(ctx context.Context, task *c } } latest.Annotations[harnessWrapperStartedAnno] = scheduledRunLabelValue + if !preserveSubmissionAttempt { + delete(latest.Annotations, harnessWrapperSubmissionAttemptedAnno) + } if err := r.Patch(ctx, latest, patch); err != nil { return err } @@ -1402,7 +1463,9 @@ func harnessWrapperStartTurnErrorIsRetryable(err error) bool { return false } message := err.Error() - for _, marker := range []string{"(400)", "(401)", "(403)", "unsupported version", "harness did not accept"} { + for _, marker := range []string{ + "(400)", "(401)", "(403)", "(410)", "unsupported version", "harness did not accept", + } { if strings.Contains(message, marker) { return false } @@ -1434,6 +1497,7 @@ func (r *TaskReconciler) clearHarnessWrapperTurnState(ctx context.Context, task delete(task.Annotations, harnessWrapperCorrelationIDAnno) delete(task.Annotations, harnessWrapperLastFrameSeqAnno) delete(task.Annotations, harnessWrapperStartedAnno) + delete(task.Annotations, harnessWrapperSubmissionAttemptedAnno) delete(task.Annotations, harnessWrapperPlannedAtAnno) delete(task.Annotations, harnessWrapperMetadataAnno) delete(task.Annotations, harnessWrapperRuntimeRefAnno) @@ -1452,6 +1516,24 @@ func (r *TaskReconciler) clearHarnessWrapperTurnState(ctx context.Context, task } func harnessWrapperStreamErrorIsMissingTurn(err error) bool { + if err == nil { + return false + } + var clientErr harness.ClientError + if errors.As(err, &clientErr) && clientErr.StatusCode > 0 { + return clientErr.StatusCode == 404 + } + message := err.Error() + if strings.Contains(message, "(404)") { + return true + } + if strings.Contains(message, "(410)") { + return false + } + return strings.Contains(message, "turn not found") +} + +func harnessWrapperCancelErrorIsMissingTurn(err error) bool { if err == nil { return false } @@ -1482,7 +1564,7 @@ func harnessWrapperStreamErrorIsTerminal(err error) bool { } func harnessWrapperStreamErrorIsBrokeredPause(err error) bool { - if err == nil { + if err == nil || harnessWrapperStreamErrorIsTerminal(err) { return false } return strings.Contains(err.Error(), "continue brokered tool call") || errors.Is(err, errHarnessBrokeredApprovalPending) @@ -1513,7 +1595,7 @@ func (r *TaskReconciler) cancelHarnessWrapperTurn(ctx context.Context, task *cor CorrelationID: strings.TrimSpace(task.Annotations[harnessWrapperCorrelationIDAnno]), Reason: reason, }) - if err != nil && harnessWrapperStreamErrorIsMissingTurn(err) { + if err != nil && harnessWrapperCancelErrorIsMissingTurn(err) { return nil } return err diff --git a/internal/controller/harness_wrapper_test.go b/internal/controller/harness_wrapper_test.go index e2e00c7ed..eef56efe9 100644 --- a/internal/controller/harness_wrapper_test.go +++ b/internal/controller/harness_wrapper_test.go @@ -107,7 +107,7 @@ func TestPatchHarnessWrapperStartedPreservesPlannedTurnAnnotationsFromLocalTask( local.Annotations[harnessWrapperOutputFetchRetriesAnno] = "1" r := newUnitReconciler(newTestScheme(), task) - if err := r.patchHarnessWrapperStarted(context.Background(), local); err != nil { + if err := r.patchHarnessWrapperStarted(context.Background(), local, false); err != nil { t.Fatalf("patchHarnessWrapperStarted: %v", err) } @@ -1615,6 +1615,119 @@ func TestHarnessWrapperPendingFirstOnlyPlansTurn(t *testing.T) { } } +func TestHarnessWrapperStartTurnAttemptIsPersistedBeforeSubmission(t *testing.T) { + startCalls := 0 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch { + case r.Method == http.MethodGet && r.URL.Path == harness.CapabilitiesPath: + harness.WriteJSON(w, http.StatusOK, harness.CapabilitiesResponse{ + Version: harness.ProtocolVersion, + ProtocolVersion: harness.ProtocolVersion, + Transport: harness.HTTPTransport, + RuntimeName: "codex", + ProviderKind: harness.ProviderKindKubernetesService, + ToolExecutionModes: []harness.ToolExecutionMode{harness.ToolExecutionModeObserved}, + SupportsCancel: true, + SupportsRuntimeSessions: true, + }) + case r.Method == http.MethodPost && r.URL.Path == harness.TurnsPath: + startCalls++ + harness.WriteError(w, http.StatusInternalServerError, "submission outcome unknown") + default: + http.NotFound(w, r) + } + })) + defer srv.Close() + t.Setenv(harnessWrapperEndpointEnv, srv.URL) + + task, agent := harnessWrapperTaskAndAgent() + secret := attachHarnessWrapperRuntimeSecret(task, agent) + r := newUnitReconciler(newTestScheme(), task, agent, secret) + if _, err := r.handlePending(context.Background(), task); err != nil { + t.Fatalf("planning handlePending: %v", err) + } + var planned corev1alpha1.Task + key := types.NamespacedName{Name: task.Name, Namespace: task.Namespace} + if err := r.Get(context.Background(), key, &planned); err != nil { + t.Fatalf("get planned task: %v", err) + } + + result, err := r.handlePending(context.Background(), &planned) + if err != nil { + t.Fatalf("submission handlePending: %v", err) + } + if result.RequeueAfter <= 0 { + t.Fatalf("submission requeue = %s, want positive delay", result.RequeueAfter) + } + if startCalls != 1 { + t.Fatalf("StartTurn calls after uncertain submission = %d, want 1", startCalls) + } + var attempted corev1alpha1.Task + if err := r.Get(context.Background(), key, &attempted); err != nil { + t.Fatalf("get attempted task: %v", err) + } + if !taskHasHarnessWrapperSubmissionAttempt(&attempted) || taskHasHarnessWrapperTurn(&attempted) { + t.Fatalf("submission annotations = %#v, want attempted but not started", attempted.Annotations) + } + + if _, err := r.handlePending(context.Background(), &attempted); err != nil { + t.Fatalf("recovery handlePending: %v", err) + } + if startCalls != 1 { + t.Fatalf("StartTurn calls after recovery = %d, want no duplicate", startCalls) + } + var running corev1alpha1.Task + if err := r.Get(context.Background(), key, &running); err != nil { + t.Fatalf("get running task: %v", err) + } + if running.Status.Phase != corev1alpha1.TaskPhaseRunning || !taskHasHarnessWrapperTurn(&running) { + t.Fatalf("recovered task phase=%s annotations=%#v, want Running started turn", running.Status.Phase, running.Annotations) + } + if !taskHasHarnessWrapperSubmissionAttempt(&running) { + t.Fatal("unknown submission marker was not preserved through Running recovery") + } + if _, err := r.handleRunning(context.Background(), &running); err != nil { + t.Fatalf("handleRunning after lost submission: %v", err) + } + if startCalls != 1 { + t.Fatalf("StartTurn calls after missing-turn failure = %d, want no duplicate", startCalls) + } + var failed corev1alpha1.Task + if err := r.Get(context.Background(), key, &failed); err != nil { + t.Fatalf("get failed task: %v", err) + } + if failed.Status.Phase != corev1alpha1.TaskPhaseFailed || + !strings.Contains(failed.Status.Message, "submission outcome became unknown") { + t.Fatalf("failed task phase=%s message=%q", failed.Status.Phase, failed.Status.Message) + } +} + +func TestHarnessWrapperAuthRetryClearsSubmissionAttemptAtomically(t *testing.T) { + task, _ := harnessWrapperTaskAndAgent() + task.Annotations = map[string]string{ + harnessWrapperSubmissionAttemptedAnno: scheduledRunLabelValue, + } + r := newUnitReconciler(newTestScheme(), task) + wait, err := r.waitForHarnessWrapperAuthRetry(context.Background(), task, true) + if err != nil { + t.Fatalf("waitForHarnessWrapperAuthRetry: %v", err) + } + if !wait { + t.Fatal("auth retry did not request a wait") + } + var updated corev1alpha1.Task + key := types.NamespacedName{Name: task.Name, Namespace: task.Namespace} + if err := r.Get(context.Background(), key, &updated); err != nil { + t.Fatalf("get updated task: %v", err) + } + if got := harnessWrapperAuthRetries(&updated); got != 1 { + t.Fatalf("auth retries = %d, want 1", got) + } + if taskHasHarnessWrapperSubmissionAttempt(&updated) { + t.Fatal("submission-attempt marker remained after definite auth rejection") + } +} + func TestHarnessRuntimeRunningTaskFinishesAfterStart(t *testing.T) { cfg := cliwrapper.DefaultConfig() cfg.AllowUnauthenticated = true @@ -2574,18 +2687,51 @@ func TestHarnessWrapperCapabilitiesReadErrorRetryable(t *testing.T) { } func TestHarnessWrapperStreamMissingTurnErrorClassification(t *testing.T) { - for _, message := range []string{"stream_frames failed (404): turn not found", "stream_frames failed (410): gone"} { - if !harnessWrapperStreamErrorIsMissingTurn(fmt.Errorf("%s", message)) { - t.Fatalf("%q should be classified as missing turn", message) + if !harnessWrapperStreamErrorIsMissingTurn(fmt.Errorf("stream_frames failed (404): turn not found")) { + t.Fatal("404 turn-not-found stream error should be classified as retryable missing turn") + } + if harnessWrapperStreamErrorIsMissingTurn(harness.ClientError{ + Op: "stream_frames", + StatusCode: http.StatusGone, + Message: "turn not found", + }) { + t.Fatal("typed 410 turn-not-found stream error should remain terminal") + } + for _, message := range []string{ + "stream_frames failed (410): terminal turn expired from runtime retention", + "stream_frames failed (410): turn not found", + "stream_frames failed (401): unauthorized", + } { + if harnessWrapperStreamErrorIsMissingTurn(fmt.Errorf("%s", message)) { + t.Fatalf("%q should not be classified as retryable missing turn", message) } } - if harnessWrapperStreamErrorIsMissingTurn(fmt.Errorf("stream_frames failed (401): unauthorized")) { - t.Fatal("unauthorized stream error should not be classified as missing turn") +} + +func TestHarnessWrapperCancelMissingTurnErrorClassification(t *testing.T) { + for _, message := range []string{ + "cancel_turn failed (404): turn not found", + "cancel_turn failed (410): terminal turn expired from runtime retention", + } { + if !harnessWrapperCancelErrorIsMissingTurn(fmt.Errorf("%s", message)) { + t.Fatalf("%q should be ignored during cancellation", message) + } + } +} + +func TestHarnessWrapperBrokeredPauseExcludesTerminalGone(t *testing.T) { + err := fmt.Errorf("continue brokered tool call \"call-1\": stream_frames failed (410): gone") + if harnessWrapperStreamErrorIsBrokeredPause(err) { + t.Fatal("terminal 410 continuation error should not be classified as a brokered pause") + } + if !harnessWrapperStreamErrorIsBrokeredPause(fmt.Errorf("continue brokered tool call \"call-1\": approval pending")) { + t.Fatal("ordinary continuation pause should remain classified as brokered pause") } } func TestHarnessWrapperStreamTerminalErrorClassification(t *testing.T) { for _, message := range []string{ + "stream_frames failed (410): terminal turn expired from runtime retention", "harness frame identity does not match running turn", "invalid harness frame: turn completed payload is required", "stream_frames failed: decode harness frame: invalid character", @@ -2699,6 +2845,11 @@ func TestHarnessWrapperStartTurnErrorClassification(t *testing.T) { if harnessWrapperStartTurnErrorIsRetryable(fmt.Errorf("start_turn failed (401): unauthorized")) { t.Fatal("expected auth start error to remain terminal") } + if harnessWrapperStartTurnErrorIsRetryable( + fmt.Errorf("start_turn failed (410): runtime session unavailable after unconfirmed hosted cancellation"), + ) { + t.Fatal("expected quarantined runtime session error to remain terminal") + } } func TestHarnessWrapperTurnMetadataDefaultsMaxTurns(t *testing.T) { diff --git a/internal/harness/client.go b/internal/harness/client.go index e9c4698b3..b782ba0e0 100644 --- a/internal/harness/client.go +++ b/internal/harness/client.go @@ -27,7 +27,8 @@ type Client struct { authBearerValue string } -const maxHarnessSSEFrameBytes = 1 << 20 +// MaxSSEFrameBytes is the largest single SSE data line the harness client accepts. +const MaxSSEFrameBytes = 1 << 20 var errSSEDone = errors.New("harness SSE stream done") @@ -327,7 +328,7 @@ func (c *Client) resolve(rel string) *url.URL { func readSSEFrames(r io.Reader, emit func(HarnessEventFrame) error) error { scanner := bufio.NewScanner(r) - scanner.Buffer(make([]byte, 0, 64*1024), maxHarnessSSEFrameBytes) + scanner.Buffer(make([]byte, 0, 64*1024), MaxSSEFrameBytes) var data strings.Builder for scanner.Scan() { line := scanner.Text() diff --git a/website/docs/guides/bring-your-own-agent-runtime.md b/website/docs/guides/bring-your-own-agent-runtime.md index 740cd8300..8006e6259 100644 --- a/website/docs/guides/bring-your-own-agent-runtime.md +++ b/website/docs/guides/bring-your-own-agent-runtime.md @@ -84,6 +84,38 @@ metadata: orka.ai/agent-runtime-endpoint: http://support-http-runtime.default.svc.cluster.local:8080 ``` + +## Foundry hosted AgentKit over Responses + +For AgentKit agents deployed as Foundry hosted agents, use the `examples/harness/foundry-responses` adapter rather than the Assistants/threads adapter. Hosted Responses requests are endpoint-scoped and must not include request-level `tools`; AgentKit must be statically configured with the safe function schemas it may request. Orka still validates every `function_call` against Task policy and Tool CRDs, performs approval/idempotency, executes the tool, and resumes the hosted response with `function_call_output` plus `previous_response_id`. + +Advertise only the brokered classes that the hosted AgentKit deployment is statically configured and conformance-tested to request: + +Readiness deep-probes each advertised class. Since hosted Responses requests do not carry request-level tools, the AgentKit deployment must also statically expose the probe-only `conformance_read` and/or `conformance_write` schema for those classes. Leave a class unadvertised until that live probe succeeds; local fake-server conformance alone does not satisfy the readiness gate. + +```yaml +apiVersion: core.orka.ai/v1alpha1 +kind: AgentRuntime +metadata: + name: foundry-agentkit-responses +spec: + contractVersion: orka.harness.v1 + deployment: + mode: external-endpoint + endpoint: http://foundry-agentkit-responses.default.svc.cluster.local:8080 + clientAuth: + bearerTokenSecretRef: + name: foundry-agentkit-responses-token + key: token + capabilities: + toolExecutionModes: [observed, brokered] + brokeredToolClasses: [read] + supportsRuntimeSessions: true + supportsContinuation: true +``` + +Add `write` only after the hosted AgentKit static write schema and brokered-write conformance pass. The adapter's MVP state is in-memory and fail-safe: duplicate identical continuations are no-ops, conflicting duplicates are rejected, and a restart while waiting for approval returns `turn not found` without sending a hosted continuation. + ## Expose a brokered tool ```yaml