Skip to content

feat: add agent loop resilience (§9.8, §9.9, §9.10, §12.5)#343

Merged
sethjuarez merged 12 commits into
mainfrom
sethjuarez/spec-agent-loop-resilience
Apr 13, 2026
Merged

feat: add agent loop resilience (§9.8, §9.9, §9.10, §12.5)#343
sethjuarez merged 12 commits into
mainfrom
sethjuarez/spec-agent-loop-resilience

Conversation

@sethjuarez

Copy link
Copy Markdown
Member

Summary

Adds three new resilience features to the agent loop spec and implements them across all four runtimes (Rust, Python, TypeScript, C#), based on consumer integration feedback from a Tauri desktop application.

Spec Changes (spec/spec.md)

Section Feature Level Description
§9.8 Resilient Argument Parsing SHOULD 4-strategy JSON fallback: direct → strip markdown fences → extract {...} block → strip trailing commas. Error string on total failure (no silent {}).
§9.9 Tool Execution Error Safety MUST Catch exceptions/panics from tool handlers, convert to error string, emit error event, continue loop.
§9.10 LLM Call Retry SHOULD Retry execute_llm on transient errors (429, 5xx) with exponential backoff min(2^n + jitter, 60). ExecuteError carries conversation messages for resume.
§12.5 Executor Retry SHOULD Executors retry HTTP internally (transparent to loop).

Also updated: §9.1 constants (MAX_LLM_RETRIES), §9.2 algorithm pseudocode, §12.4 error table, §13.7 unified signature.

Runtime Implementations

Feature Rust Python TypeScript C#
§9.8 Resilient Parsing
§9.9 Tool Error Safety
§9.10 LLM Retry

Test Results

Runtime Unit Tests Integration Tests
Rust 237 ✅ 9/10 ✅ (1 pre-existing structured output test issue)
Python 910 ✅ 48/48 ✅
TypeScript 611 ✅ 21/21 ✅
C# 735 ✅ 273/275 ✅ (2 skipped — Entra ID perms)

Bug Fixes Applied During Review

  • TypeScript: Backoff formula used milliseconds instead of seconds
  • Rust: Backoff sleep wasn't cancellable; jitter applied outside cap (max 61s → should be 60s)

Documentation Updates

  • specification/agent-loop.mdx — Added §9.8, §9.9, §9.10 sections; updated §9.1 and §9.2
  • core-concepts/agent-mode.mdx — Rewrote "Error Recovery & Resilience" with examples in all 4 languages
  • implementation/{python,typescript,csharp,rust}.mdx — Added resilience bullet points and max_llm_retries to code samples

Error Types

All runtimes expose an ExecuteError that carries the conversation messages, enabling callers to resume failed agent loops:

# Python
except ExecuteError as e:
    resume_messages = e.messages

# TypeScript
catch (e) { if (e instanceof ExecuteError) resume = e.messages; }

# C#
catch (ExecuteError e) { var resume = e.Messages; }

// Rust
Err(InvokerError::ExecuteRetryExhausted { messages, .. }) => { /* resume */ }

sethjuarez and others added 8 commits April 13, 2026 08:15
Add three new spec sections and update the agent loop algorithm to be
resilient to real-world failures:

§9.8 Resilient Argument Parsing (SHOULD)
- 4-strategy fallback chain for malformed LLM JSON in tool call arguments
- Strip markdown fences, extract JSON blocks, strip trailing commas
- MUST NOT silently substitute empty objects on failure

§9.9 Tool Execution Error Safety (MUST)
- Catch exceptions/panics from user-provided tool handlers
- Convert to error result string, emit error event, continue loop
- Agent loop MUST NOT terminate due to tool handler failure

§9.10 LLM Call Retry (SHOULD)
- Retry execute_llm up to MAX_LLM_RETRIES (default 3) with exp backoff
- On exhaustion, error MUST carry conversation state (messages) for resume
- Independent of executor-level HTTP retry (§12.5)
- Respects cancellation token during backoff waits

§12.5 Executor Retry (SHOULD)
- Executors SHOULD retry 429/5xx internally with exponential backoff
- MUST NOT retry 4xx (except 429) by default

Also updates:
- §9.1 constants table (adds MAX_LLM_RETRIES)
- §9.2 algorithm pseudocode (inline retry, try/catch, resilient parse)
- §12.4 error conditions table (new ExecuteError, tool handler note)
- §13.7 unified turn() signature (adds max_llm_retries parameter)
- §13.7 execution order summary (reflects new steps)

Motivated by real consumer integration feedback from a production Tauri
desktop app migrating to the prompty Rust runtime.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- §9.8: Enhanced ParseArguments with 4-strategy JSON fallback chain
  (direct parse, strip markdown fences, extract balanced JSON block,
  strip trailing commas). Added ExtractFirstJsonBlock helper.

- §9.9: Wrapped ToolDispatch.DispatchAsync calls in try-catch in both
  sequential and parallel agent loop paths. Tool errors are now caught
  and fed back to the LLM as error strings instead of crashing the loop.

- §9.10: Added InvokeWithRetryAsync with exponential backoff + jitter
  for LLM calls in the agent loop. Added ExecuteError exception type
  that preserves conversation state. Added maxLlmRetries parameter
  to all TurnAsync overloads (default: 3).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Resilient JSON parsing: 4-strategy fallback

- Tool execution panic safety: catch_unwind in execute_user_handler

- LLM call retry: exponential backoff with jitter

- Add ExecuteError and InvokerError::ExecuteRetryExhausted

- Add max_llm_retries to TurnOptions (default: 3)

- 14 new tests

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- §9.8 Resilient JSON parsing: _resilient_json_parse() with 4 fallback
  strategies (direct, strip markdown fences, extract JSON block, strip
  trailing commas). Used in both dispatch_tool() and dispatch_tool_async().
- §9.9 Tool execution error safety: wrap dispatch_tool calls in
  _dispatch_tools_with_extensions with try/except safety net.
- §9.10 LLM call retry: _invoke_with_retry/_invoke_with_retry_async with
  exponential backoff and jitter in the agent loop. New max_llm_retries
  parameter on turn()/turn_async(). ExecuteError carries conversation state.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The backoff formula used millisecond-scale values (2^n * 100 + rand * 100),
giving ~200ms/400ms/800ms delays. The spec §9.10 defines:
  backoff = min(2^attempts + jitter(), 60)
where values are in seconds, giving ~2s/4s/8s delays capped at 60s.

- Fix backoff calculation to use seconds, convert to ms for setTimeout
- Add fake timers to resilience tests so they don't wait real seconds
- Add maxLlmRetries: 1 to spec-vectors tests (mock executors shouldn't retry)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Use tokio::select! to race backoff sleep against cancellation polling,
  so cancellation during a long backoff (up to 60s) is detected within 100ms.
- Fix jitter range from 0-500ms to 0-1000ms to match Python/C# runtimes.
- Add tokio 'macros' feature to Cargo.toml (required for tokio::select!).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Update specification, core concepts, and all 4 implementation pages with:
- §9.8 Resilient Argument Parsing (4-strategy fallback chain)
- §9.9 Tool Execution Error Safety (catch + feed back to LLM)
- §9.10 LLM Call Retry (exponential backoff, ExecuteError with messages)

Also fixes:
- Rust backoff formula: jitter now applied inside cap (was outside)
- All code samples show max_llm_retries parameter
- Spec doc algorithm updated to match actual spec/spec.md

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sethjuarez
sethjuarez force-pushed the sethjuarez/spec-agent-loop-resilience branch from 1afb0a9 to 71c0f5d Compare April 13, 2026 16:06
sethjuarez and others added 4 commits April 13, 2026 09:39
…t Foundry

Add tests that verify the FoundryConnection/Entra ID auth path in the
Foundry executor, complementing the existing API-key-based integration
tests. Three tests:

1. Token acquisition — verifies DefaultAzureCredential can obtain a
   cognitive services token
2. Chat completion — full pipeline via Entra ID auth (no API key)
3. Streaming chat — streaming variant of the above

Tests auto-skip when AZURE_OPENAI_ENDPOINT or AZURE_OPENAI_CHAT_DEPLOYMENT
env vars are missing, and gracefully handle auth failures with helpful
messages (run az login, check RBAC role).

Temporarily clears AZURE_OPENAI_API_KEY during test execution so the
AzureOpenAI SDK uses only DefaultAzureCredential (the two auth mechanisms
are mutually exclusive in the SDK).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…dry executor

Add tests/entra_id.rs to prompty-foundry with 5 integration tests:
- test_entra_id_token_acquisition: verifies DefaultAzureCredential can
  acquire a token for the Cognitive Services scope
- test_entra_id_chat_completion: chat via kind:foundry + Entra ID auth
- test_entra_id_chat_completion_streaming: streaming chat via Entra ID
- test_entra_id_structured_output: structured output via outputSchema
- test_entra_id_agent_tool_calling: agent loop with tool calls

Tests are gated behind #[cfg(feature = "entra_id")] and #[ignore], so
they only run when explicitly requested:
  cargo test -p prompty-foundry --features entra_id --test entra_id -- --ignored --test-threads=1

The tests temporarily suppress AZURE_OPENAI_API_KEY to force the Entra ID
code path, restoring it afterward.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… integration tests

- Re-enable FoundryConnection (kind: foundry) in the Foundry executor's
  _resolve_client() and _resolve_client_async() methods
- Add _build_client_from_entra() and _build_async_client_from_entra()
  methods that use DefaultAzureCredential + get_bearer_token_provider
  with scope https://cognitiveservices.azure.com/.default
- Add make_entra_agent() helper to integration test conftest.py
- Add skip_entra / has_entra skip logic for when endpoint env vars missing
- Create test_entra_id.py with 3 integration tests:
  * test_entra_id_token_acquisition
  * test_entra_id_chat_completion
  * test_entra_id_chat_completion_async
- Tests gracefully skip on missing credentials or auth failures

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ests

- Fix: invoke() was not calling unwrap_structured() before returning,
  causing structured output to leak the __prompty_structured envelope.
  run() and turn() were correct; only invoke() was affected.

- Add mock-based structured output pipeline tests to all 4 runtimes:
  Rust (4 tests), Python (5 tests), TypeScript (14 tests), C# (9 tests).
  These exercise invoke()/run() with mocked providers to catch pipeline
  bugs without requiring real API keys.

- Update .env.example files with AZURE_TENANT_ID for Entra ID testing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sethjuarez
sethjuarez force-pushed the sethjuarez/spec-agent-loop-resilience branch from 774ae71 to eceaeaa Compare April 13, 2026 18:35
@sethjuarez
sethjuarez merged commit 6bfdfee into main Apr 13, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant