Skip to content

feat: add context compaction across all runtimes + Rust DX improvements#345

Merged
sethjuarez merged 19 commits into
mainfrom
sethjuarez/spec-agent-loop-resilience
Apr 14, 2026
Merged

feat: add context compaction across all runtimes + Rust DX improvements#345
sethjuarez merged 19 commits into
mainfrom
sethjuarez/spec-agent-loop-resilience

Conversation

@sethjuarez

@sethjuarez sethjuarez commented Apr 13, 2026

Copy link
Copy Markdown
Member

Context Compaction (§13.3)

Adds LLM-powered context compaction to all 4 runtimes. When trim_to_context_window() drops messages, compaction replaces the low-signal default summary with a higher-quality one.

Dual-mode API

  1. .prompty file — declarative: point to a prompt that summarizes dropped messages
  2. Function/callable — programmatic: custom summarization logic

Both are opt-in. Default behavior unchanged (zero breaking changes). Failures fall back silently to the existing summarize_dropped() result.

Changes by runtime

Runtime Files New Tests
Spec spec/spec.md — §13.3 rewritten, events table updated, unified signature updated —
Python core/context.py, core/pipeline.py, exports +10
TypeScript context.ts, pipeline.ts, agent-events.ts, exports +11
Rust context.rs, pipeline.rs, lib.rs +11
C# ContextWindow.cs, Pipeline.cs, CompactionStrategy.cs, AgentEvents.cs +14
Docs agent-extensions.mdx — Context Compaction section with all 4 languages —

Rust DX Improvements

  • TurnOptions::builder() — fluent builder for the 13-field TurnOptions struct (+6 tests)
  • prompty::prelude::* — convenience re-exports of common types

Consumer feedback addressed

This closes the last remaining core gap from consumer integration feedback (alpha.9):

  • ✅ Robust JSON parsing (alpha.9)
  • ✅ Panic-safe tool execution (alpha.9)
  • ✅ Retry with exponential backoff (alpha.9)
  • ✅ Tracing sanitization (alpha.9)
  • ✅ LLM-powered context compaction (this PR)

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

sethjuarez and others added 19 commits April 13, 2026 08:15
@sethjuarez
spec: add agent loop resilience (§9.8, §9.9, §9.10, §12.5)
6020ac9 
Add three new spec sections and update the agent loop algorithm to be
resilient to real-world failures:

§9.8 Resilient Argument Parsing (SHOULD)
- 4-strategy fallback chain for malformed LLM JSON in tool call arguments
- Strip markdown fences, extract JSON blocks, strip trailing commas
- MUST NOT silently substitute empty objects on failure

§9.9 Tool Execution Error Safety (MUST)
- Catch exceptions/panics from user-provided tool handlers
- Convert to error result string, emit error event, continue loop
- Agent loop MUST NOT terminate due to tool handler failure

§9.10 LLM Call Retry (SHOULD)
- Retry execute_llm up to MAX_LLM_RETRIES (default 3) with exp backoff
- On exhaustion, error MUST carry conversation state (messages) for resume
- Independent of executor-level HTTP retry (§12.5)
- Respects cancellation token during backoff waits

§12.5 Executor Retry (SHOULD)
- Executors SHOULD retry 429/5xx internally with exponential backoff
- MUST NOT retry 4xx (except 429) by default

Also updates:
- §9.1 constants table (adds MAX_LLM_RETRIES)
- §9.2 algorithm pseudocode (inline retry, try/catch, resilient parse)
- §12.4 error conditions table (new ExecuteError, tool handler note)
- §13.7 unified turn() signature (adds max_llm_retries parameter)
- §13.7 execution order summary (reflects new steps)

Motivated by real consumer integration feedback from a production Tauri
desktop app migrating to the prompty Rust runtime.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sethjuarez
feat(ts): add agent loop resilience features
b1ab87b 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sethjuarez
feat(csharp): add agent loop resilience features (§9.8, §9.9, §9.10)
eefec5c 
- §9.8: Enhanced ParseArguments with 4-strategy JSON fallback chain
  (direct parse, strip markdown fences, extract balanced JSON block,
  strip trailing commas). Added ExtractFirstJsonBlock helper.

- §9.9: Wrapped ToolDispatch.DispatchAsync calls in try-catch in both
  sequential and parallel agent loop paths. Tool errors are now caught
  and fed back to the LLM as error strings instead of crashing the loop.

- §9.10: Added InvokeWithRetryAsync with exponential backoff + jitter
  for LLM calls in the agent loop. Added ExecuteError exception type
  that preserves conversation state. Added maxLlmRetries parameter
  to all TurnAsync overloads (default: 3).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sethjuarez
feat(rust): add agent loop resilience features
273105e 
- Resilient JSON parsing: 4-strategy fallback

- Tool execution panic safety: catch_unwind in execute_user_handler

- LLM call retry: exponential backoff with jitter

- Add ExecuteError and InvokerError::ExecuteRetryExhausted

- Add max_llm_retries to TurnOptions (default: 3)

- 14 new tests

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sethjuarez
feat: add agent loop resilience (§9.8, §9.9, §9.10)
d6df11b 
- §9.8 Resilient JSON parsing: _resilient_json_parse() with 4 fallback
  strategies (direct, strip markdown fences, extract JSON block, strip
  trailing commas). Used in both dispatch_tool() and dispatch_tool_async().
- §9.9 Tool execution error safety: wrap dispatch_tool calls in
  _dispatch_tools_with_extensions with try/except safety net.
- §9.10 LLM call retry: _invoke_with_retry/_invoke_with_retry_async with
  exponential backoff and jitter in the agent loop. New max_llm_retries
  parameter on turn()/turn_async(). ExecuteError carries conversation state.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sethjuarez
fix(ts): correct retry backoff to use seconds per spec §9.10
63920f8 
The backoff formula used millisecond-scale values (2^n * 100 + rand * 100),
giving ~200ms/400ms/800ms delays. The spec §9.10 defines:
  backoff = min(2^attempts + jitter(), 60)
where values are in seconds, giving ~2s/4s/8s delays capped at 60s.

- Fix backoff calculation to use seconds, convert to ms for setTimeout
- Add fake timers to resilience tests so they don't wait real seconds
- Add maxLlmRetries: 1 to spec-vectors tests (mock executors shouldn't retry)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sethjuarez
fix(rust): respect cancellation during retry backoff per spec §9.10
0897666 
- Use tokio::select! to race backoff sleep against cancellation polling,
  so cancellation during a long backoff (up to 60s) is detected within 100ms.
- Fix jitter range from 0-500ms to 0-1000ms to match Python/C# runtimes.
- Add tokio 'macros' feature to Cargo.toml (required for tokio::select!).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sethjuarez
docs: add agent loop resilience (§9.8, §9.9, §9.10) to user-facing docs
71c0f5d 
Update specification, core concepts, and all 4 implementation pages with:
- §9.8 Resilient Argument Parsing (4-strategy fallback chain)
- §9.9 Tool Execution Error Safety (catch + feed back to LLM)
- §9.10 LLM Call Retry (exponential backoff, ExecuteError with messages)

Also fixes:
- Rust backoff formula: jitter now applied inside cap (was outside)
- All code samples show max_llm_retries parameter
- Spec doc algorithm updated to match actual spec/spec.md

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sethjuarez
Add Entra ID (DefaultAzureCredential) integration tests for TypeScrip…
2988526 
…t Foundry

Add tests that verify the FoundryConnection/Entra ID auth path in the
Foundry executor, complementing the existing API-key-based integration
tests. Three tests:

1. Token acquisition — verifies DefaultAzureCredential can obtain a
   cognitive services token
2. Chat completion — full pipeline via Entra ID auth (no API key)
3. Streaming chat — streaming variant of the above

Tests auto-skip when AZURE_OPENAI_ENDPOINT or AZURE_OPENAI_CHAT_DEPLOYMENT
env vars are missing, and gracefully handle auth failures with helpful
messages (run az login, check RBAC role).

Temporarily clears AZURE_OPENAI_API_KEY during test execution so the
AzureOpenAI SDK uses only DefaultAzureCredential (the two auth mechanisms
are mutually exclusive in the SDK).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sethjuarez
Add Entra ID (DefaultAzureCredential) integration tests for Rust Foun…
54cdcb7 
…dry executor

Add tests/entra_id.rs to prompty-foundry with 5 integration tests:
- test_entra_id_token_acquisition: verifies DefaultAzureCredential can
  acquire a token for the Cognitive Services scope
- test_entra_id_chat_completion: chat via kind:foundry + Entra ID auth
- test_entra_id_chat_completion_streaming: streaming chat via Entra ID
- test_entra_id_structured_output: structured output via outputSchema
- test_entra_id_agent_tool_calling: agent loop with tool calls

Tests are gated behind #[cfg(feature = "entra_id")] and #[ignore], so
they only run when explicitly requested:
  cargo test -p prompty-foundry --features entra_id --test entra_id -- --ignored --test-threads=1

The tests temporarily suppress AZURE_OPENAI_API_KEY to force the Entra ID
code path, restoring it afterward.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sethjuarez
Add Entra ID (DefaultAzureCredential) support to Foundry executor and…
4ad2f72 
… integration tests

- Re-enable FoundryConnection (kind: foundry) in the Foundry executor's
  _resolve_client() and _resolve_client_async() methods
- Add _build_client_from_entra() and _build_async_client_from_entra()
  methods that use DefaultAzureCredential + get_bearer_token_provider
  with scope https://cognitiveservices.azure.com/.default
- Add make_entra_agent() helper to integration test conftest.py
- Add skip_entra / has_entra skip logic for when endpoint env vars missing
- Create test_entra_id.py with 3 integration tests:
  * test_entra_id_token_acquisition
  * test_entra_id_chat_completion
  * test_entra_id_chat_completion_async
- Tests gracefully skip on missing credentials or auth failures

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sethjuarez
fix(rust): unwrap structured output in invoke() + add mock pipeline t…
eceaeaa 
…ests

- Fix: invoke() was not calling unwrap_structured() before returning,
  causing structured output to leak the __prompty_structured envelope.
  run() and turn() were correct; only invoke() was affected.

- Add mock-based structured output pipeline tests to all 4 runtimes:
  Rust (4 tests), Python (5 tests), TypeScript (14 tests), C# (9 tests).
  These exercise invoke()/run() with mocked providers to catch pipeline
  bugs without requiring real API keys.

- Update .env.example files with AZURE_TENANT_ID for Entra ID testing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sethjuarez
feat(csharp): add context compaction to C# pipeline
7395ad4 
Add CompactionStrategy abstraction for LLM-powered or function-based
context summarization when messages are trimmed in TurnAsync.

Changes:
- Add CompactionStrategy abstract class with FromPrompty() and
  FromFunction() factory methods (CompactionStrategy.cs)
- Add FormatDroppedMessages() to ContextWindow for readable message
  formatting with tool call support (ContextWindow.cs)
- Add compaction parameter to all TurnAsync overloads (Pipeline.cs)
- Add ApplyCompactionAsync() and ReplaceSummaryMessage() helpers
- Add CompactionStart/Complete/Failed event types (AgentEvents.cs)
- On failure, fall back to existing SummarizeDropped() summary
- Add 14 tests covering formatting, strategy creation, compaction
  application, event emission, and failure handling

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sethjuarez
feat(typescript): add context compaction to turn()
977674b 
Add a compaction option to TurnOptions that enables LLM-powered or
function-based context summarization when messages are trimmed by
trimToContextWindow.

Two modes:
- String path: invokes a .prompty file with formatted dropped messages
- Function: calls a sync/async function that returns a summary string

On failure, falls back to the existing summarizeDropped() result.

Changes:
- agent-events.ts: add compaction_start/complete/failed event types
- context.ts: add formatDroppedMessages() utility
- pipeline.ts: add compaction to TurnOptions, applyCompaction and
  replaceSummaryMessage helpers, wire into both trim sites
- core/index.ts, src/index.ts: export formatDroppedMessages
- compaction.test.ts: 11 tests covering function, async, failure,
  events, empty result, and default behavior

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sethjuarez
Add context compaction to turn() and turn_async()
4abd20b 
Add a compaction parameter that enables LLM-powered or function-based
context summarization when messages are trimmed by context_budget.

Three modes:
- String/Path: invoke a .prompty file with dropped messages as input
- Callable: call function(dropped_messages) -> summary string
- None (default): unchanged behavior using built-in summarize_dropped()

On failure, falls back to the existing default summary already in messages.

Changes:
- core/context.py: add format_dropped_messages() for readable message formatting
- core/pipeline.py: add compaction param to turn()/turn_async(), add
  _apply_compaction/_apply_compaction_async/_replace_summary_message helpers
- core/__init__.py, __init__.py: export format_dropped_messages
- tests/test_agent_extensions.py: 10 new tests for compaction behavior

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sethjuarez
feat(rust): add context compaction to TurnOptions
e1c4399 
Add Compaction enum (Prompty file or async function) to TurnOptions
that replaces the low-signal default summary with a higher-quality
one when messages are trimmed by trim_to_context_window().

Changes:
- context.rs: Change trim_to_context_window return type from
  (usize, Vec<Message>) to (Vec<Message>, Vec<Message>) to expose
  dropped messages. Add format_dropped_messages() helper.
- pipeline.rs: Add Compaction enum, CompactionFn type alias,
  apply_compaction() and replace_summary_message() helpers.
  Wire compaction into both trim callsites in turn().
- lib.rs: Export Compaction, CompactionFn, format_dropped_messages.
- tracing/mod.rs: Re-export SpanEmitter for internal use.
- tests/compaction_tests.rs: 11 tests covering format, trim,
  function-based compaction, failure fallback, and defaults.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sethjuarez
Add Context Compaction section to agent extensions docs
8fc07d8 
Add documentation for the context compaction feature that replaces
simple trim summaries with higher-quality summaries using either a
.prompty file or a custom function. Includes code examples for all
four languages (Python, TypeScript, C#, Rust), a sample compaction
.prompty file, and usage tips.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sethjuarez
Add TurnOptions builder pattern and prelude module to Rust runtime
2ab9dd3 
- Add TurnOptionsBuilder with fluent API for all 13 TurnOptions fields
- Add individual .tool() method for ergonomic single-tool registration
- Export TurnOptionsBuilder from lib.rs
- Create prelude module with common re-exports for use prompty::prelude::*
- Add 6 unit tests: defaults, chaining, single/multiple tools, compaction, cancellation

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sethjuarez
Merge origin/main into sethjuarez/spec-agent-loop-resilience
1256bee 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sethjuarez
sethjuarez merged commit 3490cbb into main Apr 14, 2026
25 checks passed
@sethjuarez
sethjuarez deleted the sethjuarez/spec-agent-loop-resilience branch June 9, 2026 19:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@sethjuarez