feat: add agent loop resilience (§9.8, §9.9, §9.10, §12.5)#343
Merged
Conversation
Add three new spec sections and update the agent loop algorithm to be resilient to real-world failures: §9.8 Resilient Argument Parsing (SHOULD) - 4-strategy fallback chain for malformed LLM JSON in tool call arguments - Strip markdown fences, extract JSON blocks, strip trailing commas - MUST NOT silently substitute empty objects on failure §9.9 Tool Execution Error Safety (MUST) - Catch exceptions/panics from user-provided tool handlers - Convert to error result string, emit error event, continue loop - Agent loop MUST NOT terminate due to tool handler failure §9.10 LLM Call Retry (SHOULD) - Retry execute_llm up to MAX_LLM_RETRIES (default 3) with exp backoff - On exhaustion, error MUST carry conversation state (messages) for resume - Independent of executor-level HTTP retry (§12.5) - Respects cancellation token during backoff waits §12.5 Executor Retry (SHOULD) - Executors SHOULD retry 429/5xx internally with exponential backoff - MUST NOT retry 4xx (except 429) by default Also updates: - §9.1 constants table (adds MAX_LLM_RETRIES) - §9.2 algorithm pseudocode (inline retry, try/catch, resilient parse) - §12.4 error conditions table (new ExecuteError, tool handler note) - §13.7 unified turn() signature (adds max_llm_retries parameter) - §13.7 execution order summary (reflects new steps) Motivated by real consumer integration feedback from a production Tauri desktop app migrating to the prompty Rust runtime. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- §9.8: Enhanced ParseArguments with 4-strategy JSON fallback chain (direct parse, strip markdown fences, extract balanced JSON block, strip trailing commas). Added ExtractFirstJsonBlock helper. - §9.9: Wrapped ToolDispatch.DispatchAsync calls in try-catch in both sequential and parallel agent loop paths. Tool errors are now caught and fed back to the LLM as error strings instead of crashing the loop. - §9.10: Added InvokeWithRetryAsync with exponential backoff + jitter for LLM calls in the agent loop. Added ExecuteError exception type that preserves conversation state. Added maxLlmRetries parameter to all TurnAsync overloads (default: 3). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Resilient JSON parsing: 4-strategy fallback - Tool execution panic safety: catch_unwind in execute_user_handler - LLM call retry: exponential backoff with jitter - Add ExecuteError and InvokerError::ExecuteRetryExhausted - Add max_llm_retries to TurnOptions (default: 3) - 14 new tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- §9.8 Resilient JSON parsing: _resilient_json_parse() with 4 fallback strategies (direct, strip markdown fences, extract JSON block, strip trailing commas). Used in both dispatch_tool() and dispatch_tool_async(). - §9.9 Tool execution error safety: wrap dispatch_tool calls in _dispatch_tools_with_extensions with try/except safety net. - §9.10 LLM call retry: _invoke_with_retry/_invoke_with_retry_async with exponential backoff and jitter in the agent loop. New max_llm_retries parameter on turn()/turn_async(). ExecuteError carries conversation state. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The backoff formula used millisecond-scale values (2^n * 100 + rand * 100), giving ~200ms/400ms/800ms delays. The spec §9.10 defines: backoff = min(2^attempts + jitter(), 60) where values are in seconds, giving ~2s/4s/8s delays capped at 60s. - Fix backoff calculation to use seconds, convert to ms for setTimeout - Add fake timers to resilience tests so they don't wait real seconds - Add maxLlmRetries: 1 to spec-vectors tests (mock executors shouldn't retry) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Use tokio::select! to race backoff sleep against cancellation polling, so cancellation during a long backoff (up to 60s) is detected within 100ms. - Fix jitter range from 0-500ms to 0-1000ms to match Python/C# runtimes. - Add tokio 'macros' feature to Cargo.toml (required for tokio::select!). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Update specification, core concepts, and all 4 implementation pages with: - §9.8 Resilient Argument Parsing (4-strategy fallback chain) - §9.9 Tool Execution Error Safety (catch + feed back to LLM) - §9.10 LLM Call Retry (exponential backoff, ExecuteError with messages) Also fixes: - Rust backoff formula: jitter now applied inside cap (was outside) - All code samples show max_llm_retries parameter - Spec doc algorithm updated to match actual spec/spec.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
sethjuarez
force-pushed
the
sethjuarez/spec-agent-loop-resilience
branch
from
April 13, 2026 16:06
1afb0a9 to
71c0f5d
Compare
…t Foundry Add tests that verify the FoundryConnection/Entra ID auth path in the Foundry executor, complementing the existing API-key-based integration tests. Three tests: 1. Token acquisition — verifies DefaultAzureCredential can obtain a cognitive services token 2. Chat completion — full pipeline via Entra ID auth (no API key) 3. Streaming chat — streaming variant of the above Tests auto-skip when AZURE_OPENAI_ENDPOINT or AZURE_OPENAI_CHAT_DEPLOYMENT env vars are missing, and gracefully handle auth failures with helpful messages (run az login, check RBAC role). Temporarily clears AZURE_OPENAI_API_KEY during test execution so the AzureOpenAI SDK uses only DefaultAzureCredential (the two auth mechanisms are mutually exclusive in the SDK). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…dry executor Add tests/entra_id.rs to prompty-foundry with 5 integration tests: - test_entra_id_token_acquisition: verifies DefaultAzureCredential can acquire a token for the Cognitive Services scope - test_entra_id_chat_completion: chat via kind:foundry + Entra ID auth - test_entra_id_chat_completion_streaming: streaming chat via Entra ID - test_entra_id_structured_output: structured output via outputSchema - test_entra_id_agent_tool_calling: agent loop with tool calls Tests are gated behind #[cfg(feature = "entra_id")] and #[ignore], so they only run when explicitly requested: cargo test -p prompty-foundry --features entra_id --test entra_id -- --ignored --test-threads=1 The tests temporarily suppress AZURE_OPENAI_API_KEY to force the Entra ID code path, restoring it afterward. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… integration tests - Re-enable FoundryConnection (kind: foundry) in the Foundry executor's _resolve_client() and _resolve_client_async() methods - Add _build_client_from_entra() and _build_async_client_from_entra() methods that use DefaultAzureCredential + get_bearer_token_provider with scope https://cognitiveservices.azure.com/.default - Add make_entra_agent() helper to integration test conftest.py - Add skip_entra / has_entra skip logic for when endpoint env vars missing - Create test_entra_id.py with 3 integration tests: * test_entra_id_token_acquisition * test_entra_id_chat_completion * test_entra_id_chat_completion_async - Tests gracefully skip on missing credentials or auth failures Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ests - Fix: invoke() was not calling unwrap_structured() before returning, causing structured output to leak the __prompty_structured envelope. run() and turn() were correct; only invoke() was affected. - Add mock-based structured output pipeline tests to all 4 runtimes: Rust (4 tests), Python (5 tests), TypeScript (14 tests), C# (9 tests). These exercise invoke()/run() with mocked providers to catch pipeline bugs without requiring real API keys. - Update .env.example files with AZURE_TENANT_ID for Entra ID testing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
sethjuarez
force-pushed
the
sethjuarez/spec-agent-loop-resilience
branch
from
April 13, 2026 18:35
774ae71 to
eceaeaa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds three new resilience features to the agent loop spec and implements them across all four runtimes (Rust, Python, TypeScript, C#), based on consumer integration feedback from a Tauri desktop application.
Spec Changes (
spec/spec.md){...}block → strip trailing commas. Error string on total failure (no silent{}).errorevent, continue loop.execute_llmon transient errors (429, 5xx) with exponential backoffmin(2^n + jitter, 60).ExecuteErrorcarries conversationmessagesfor resume.Also updated: §9.1 constants (
MAX_LLM_RETRIES), §9.2 algorithm pseudocode, §12.4 error table, §13.7 unified signature.Runtime Implementations
Test Results
Bug Fixes Applied During Review
Documentation Updates
specification/agent-loop.mdx— Added §9.8, §9.9, §9.10 sections; updated §9.1 and §9.2core-concepts/agent-mode.mdx— Rewrote "Error Recovery & Resilience" with examples in all 4 languagesimplementation/{python,typescript,csharp,rust}.mdx— Added resilience bullet points andmax_llm_retriesto code samplesError Types
All runtimes expose an
ExecuteErrorthat carries the conversationmessages, enabling callers to resume failed agent loops: