Add prepared prompt session API to avoid re-rendering static templates each chat turn

[sethjuarez]

Summary

Today turn(agent, inputs) calls prepare() once per external user turn. That is correct and predictable, but it means chat-style runtimes re-render and re-parse the .prompty template every time the user sends a message, even when the prompt's static structure has not changed.

For extension chat sessions and long-running applications, we could add a runtime-level prepared-session API that renders/parses the stable prompt skeleton once, preserves the template-defined thread insertion point, and then efficiently expands new conversation/thread messages on each turn.

Current behavior

A normal chat session effectively does this for each user message:

turn N
  prepare(agent, inputs)
    render template
    parse role markers
    expand thread input into Message[]
  execute(agent, messages)
  process(response)

This is intentionally stateless and correct. The template may contain conditionals, non-thread inputs, rich inputs, strict-mode parser handling, or a thread variable placed anywhere in the prompt body. Re-running prepare() guarantees the returned Message[] matches the latest inputs.

Inside a single tool-calling agent loop, however, we do not re-render/re-parse the prompt. The loop mutates/extends the prepared message list with assistant/tool messages and continues execution.

Motivation

For interactive chat sessions, especially in the VS Code extension, the static prompt structure is often stable across turns:

• system/developer instructions
• static few-shot messages
• template-defined thread insertion point
• trailing prompt messages, if any

The only thing changing each turn is usually the conversation thread. Re-rendering and parsing the entire prompt every turn is unnecessary work and also makes traces noisier because every user turn contains a full prepare -> render -> parse subtree.

Proposed direction

Add a runtime concept for a prepared prompt session / prepared template skeleton. Names are illustrative:

const session = await prepareSession(agent, initialInputs);

const result1 = await session.turn({ conversation });
const result2 = await session.turn({ conversation });

or lower-level:

const prepared = await prepareTemplate(agent, inputsWithoutThread);
const messages = expandPreparedThread(prepared, { conversation });

The key difference from prepare() is that the runtime would preserve an intermediate representation instead of only returning final Message[]:

PreparedPrompt
  prefix messages
  thread/rich-input marker(s)
  suffix messages
  parse/render metadata needed for strict mode
  cache key / invalidation metadata

Then each turn can splice or expand thread messages into the preserved marker location without re-rendering static content.

Correctness constraints

This should be treated as an optimization API, not a replacement for prepare().

The implementation needs to handle or explicitly reject cases where caching could be wrong:

• non-thread inputs changed between turns
• prompt file or agent.instructions changed
• template conditionals depend on values that change per turn
• template conditionals depend on the conversation/thread value itself
• strict-mode parser pre-render context and nonce handling must remain valid
• rich inputs (thread, image, file, audio) need explicit marker preservation semantics
• guardrails, steering, context trimming, and compaction can mutate runtime messages and must not mutate the cached skeleton
• multi-thread prompts may contain more than one rich/thread insertion point

A safe first version could require callers to declare which inputs are stable and invalidate/fall back to full prepare() when other inputs change.

Acceptance criteria

• Add a runtime API for preparing a reusable prompt/session skeleton.
• Preserve template-defined thread insertion location instead of assuming append-only conversation history.
• Support at least the common chat case: one thread input plus static prompt instructions.
• Include tests proving the main template renders/parses once while subsequent turns only expand/merge thread messages.
• Include invalidation/fallback behavior for changed non-thread inputs.
• Keep existing prepare(), turn(), and invoke() behavior unchanged.
• Implement consistently across runtimes or document rollout order if staged.

Notes

This issue came from observing trace output in the VS Code extension: seeing prepare -> render -> parse once per user turn is expected with today's turn() API, but it points to a useful runtime optimization for long-lived chat sessions.