Validate AgentKit tool calls that wait for human approval in Orka v2

[sozercan]

Problem

An agent may propose an action, such as creating a work order, that requires a person's approval. Orka should ask the person and decide whether the tool runs. AgentKit must wait safely and then continue with the real outcome.

That complete flow is not supported or validated for Orka v2 yet. Orka's main implementation belongs in orka-agents/orka#582. This issue covers AgentKit's compatibility, tests, setup guidance, and any runtime changes those tests show are necessary.

Cover both ways AgentKit is used in the Fibey integration: running directly under Orka and running as a hosted agent in Microsoft Foundry.

Why this matters

Human reviews can last longer than an ordinary tool call. A timeout, expired pending request, or incorrect interpretation of a result could end the conversation early or cause work to repeat.

AgentKit already has useful waiting and continuation behavior. A new approval system inside AgentKit is not required. The work is to prove that the existing behavior follows Orka's approval rules and fix specific mismatches.

What already works, and what is missing

• Direct AgentKit waits for tool calls and supports cancellation. The Microsoft Agent Framework path used by the demo disables automatic tool-call retries after connection loss, because the tool may already have run.
• Hosted AgentKit can return a proposed tool call and later continue from a matching, authenticated result. It checks that the result belongs to the correct conversation and pending call.
• Hosted success, denial, duplicate delivery, restart, and expiry already have tests. The existing denial test supplies a simulated approval_declined result directly; it does not wait for an actual review in Orka.
• Direct Microsoft Agent Framework tool requests default to a 120-second timeout, controlled by AGENTKIT_MCP_TIMEOUT.
• Hosted pending state expires after 900 seconds by default. The allowed review window must fit the configured state lifetime as well as the task's limits.

In the hosted result format, approved describes a final result: true carries tool output, while false carries an error such as a rejection or tool failure. Neither value represents a pending review. Sending false merely because a person has not decided would finish the pending call incorrectly.

How to approach it

1. Agree on the waiting and decision behavior in Add human approval for tools used by Orka v2 agents orka-agents/orka#582 first. Document what AgentKit must do in direct and hosted execution.
2. Add tests through Orka's real approval path using a harmless tool that records how many times it runs.
3. Check existing tool-request timeouts and hosted pending-state settings against the supported review window. Do not simply make every timeout unlimited.
4. Verify cancellation and expiry through Orka and the Foundry bridge, where permission to continue the task is controlled.
5. Change production code only where a test demonstrates that the agreed flow cannot work. Add a new permission callback only if Orka's design actually requires one.
6. Document the supported setup and outcomes in plain language, including what happens when a review takes too long.

Keep approval policy, reviewer permissions, stored human decisions, and the review interface in Orka. AgentKit's local tools[].approval setting is separate. Its auto and always modes are currently unsupported and do not enable Orka-managed approval; omission and never remain valid.

Acceptance checks

• Before a reviewer decides, the tool has run zero times and AgentKit has not reported that the action succeeded.
• Approval allows one tool execution, and AgentKit continues with its actual output without repeating the original prompt or completed tool steps.
• Decline and approval expiry produce Orka's agreed outcome without executing the tool. A tool error after approval is not presented as a human decline.
• The documented direct configuration supports the agreed wait duration, including beyond the current two-minute default if Orka allows it.
• Hosted continuation remains valid throughout the supported review window. A missing or expired pending request fails clearly and cannot restart the old action.
• Hosted AgentKit process restart with supported saved state preserves matching pending calls. Missing state does not become permission to repeat work; this check does not imply recovery of an entire lost Foundry runtime session.
• Cancellation while waiting prevents a later decision from reviving the cancelled task.
• Duplicate or conflicting result delivery does not repeat completed work. Wrong conversation IDs, wrong call IDs, and unauthenticated continuations remain rejected.
• With available runtime capacity, a separate conversation using its own agent process or hosted session can progress while one waits for review.
• Existing tools that need no human approval continue to work.
• End-to-end evidence covers direct AgentKit and AgentKit hosted in Foundry through Orka's approval API. Use simulated actions, and keep unrun live checks explicit.
• Runtime changes are limited to demonstrated compatibility gaps. Passing tests may establish that existing code is sufficient.

Dependencies and scope

• Main implementation and shared behavior: Add human approval for tools used by Orka v2 agents orka-agents/orka#582.
• Foundry bridge integration: Let Foundry-hosted agents wait for Orka tool approvals orka-agents/agent-runtime-foundry#4.
• feat(runtime): support skills and governed tool workflows #25 supplies the current skills and repeated tool-workflow integration.
• feat(foundry): add generated protocol wrapper target for AgentKit runtimes #6 is broader Foundry packaging work; this issue specifically addresses Orka-managed approval waiting.

This issue does not introduce AgentKit approval policy or a separate review interface. It qualifies the supported runtime paths and documents combinations that remain unsupported.

Code starting points

The existing waiting behavior and limits are present on main at 9a6de8fe:

• Current Orka v2 restrictions.
• Direct tool timeout settings.
• Direct tool calls without automatic replay.
• Hosted continuation validation.
• Hosted state lifetime and recovery.
• Existing simulated denial test.