fix(foundry): prove durable memory provider parity

[sozercan]

This was generated by AI during triage.

Summary

Follow up on AgentKit ↔ Microsoft Agent Framework / Foundry hosted-agent parity for durable external memory.

The generic AgentKit memory wiring has been added behind provider-neutral config:

context:
  providers:
    - name: validation-memory
      type: memory
      endpointEnv: MEMORY_ENDPOINT
      storeNameEnv: MEMORY_STORE_NAME
      auth:
        type: workload-identity-token
        audience: https://ai.azure.com/.default

However, live validation did not prove true durable Memory Store parity. Some same-session recall worked, but that can come from conversation/session history rather than the external Memory Store.

Current evidence

What works

• AgentKit MAF image builds with generic context.providers[].type: memory.
• Standalone and Foundry hosted memory agents start successfully.
• Hosted /invocations and /responses return HTTP 200.
• Session propagation now works generically:
  • standalone: X-AgentKit-Session-Id
  • Foundry hosted: Foundry session query/header forwarded to runtime
• Same-session hosted recall can return a remembered value.

What did not pass

A true durable-memory test was not proven.

Observed failures/suspicious behavior:

• Standalone second-turn recall failed in one run:

  You have not asked me to remember any validation codename yet.
  

• Hosted /responses recalled a stale/wrong value in one run:

  MEMORY_OK_2291
  

  instead of the new value requested for that session:

  MEMORY_RESP_OK_2291
  

• Earlier hosted logs showed Foundry Memory Store search failing internally against the embedding deployment:

  Memory search failed ...
  Authentication to the Azure OpenAI resource failed.
  deployment: text-embedding-3-small
  

After additional RBAC changes, that specific log did not appear in the latest run, but durable memory recall was still not clearly proven.

Likely causes to investigate

1. Memory Store backing auth/RBAC

   • Ensure the Foundry project identity and hosted-agent identity can call the AI account and embedding deployment used by the Memory Store.
   • If RBAC was granted after Memory Store creation, the Memory Store may need to be recreated or refreshed.

2. Scope/session mismatch

   • Validate how scope maps to the MAF FoundryMemoryProvider and hosted sessions.
   • Use unique memory scopes per test run to avoid stale cross-test recall.

3. False positives from session history

   • Same-session recall is not enough to prove external memory.
   • Need a fresh-session/fresh-container test where only the Memory Store can provide the answer.

4. Memory extraction/commit semantics

   • Asking the model to “remember this” may not synchronously create a durable memory item.
   • Use the Memory Store API directly to seed/list/search memory before invoking the AgentKit image.

Proposed validation flow

Create a deterministic durable-memory validation script under deploy/foundry or test/foundry-hosted-agent:

1. Generate a unique scope, e.g. agentkit-memory-test-<timestamp>.

2. Seed memory directly through the Memory Store API with a canary value, e.g. MEMORY_DURABLE_OK_<nonce>.

3. Verify the seeded memory can be listed/searched through the Memory Store API.

4. Start a fresh standalone AgentKit container with:

   MEMORY_ENDPOINT=...
   MEMORY_STORE_NAME=...
   AGENTKIT_MEMORY_SCOPE=<unique-scope>

5. Ask for the canary in a fresh AgentKit session and assert the response contains it.

6. Deploy the same AgentKit-built image as a Foundry hosted agent.

7. Ask for the same canary through both:

   • /invocations
   • /responses

8. Assert all three paths retrieve the same durable memory canary:

   • standalone /v1/chat/completions
   • hosted /invocations
   • hosted /responses

Acceptance criteria

• No Foundry-specific keys are added to AgentKit core schema.
• Durable memory uses only generic AgentKit config: context.providers[].type: memory plus env declarations.
• Foundry-specific provisioning/seeding lives in deployment/test tooling.
• Memory validation seeds memory through the Memory Store API, not only through model conversation.
• Validation uses a unique scope per run.
• Validation starts a fresh session/container before recall so chat history cannot satisfy the assertion.
• Standalone AgentKit image recalls the seeded memory canary.
• Foundry hosted AgentKit image recalls the seeded memory canary through /invocations.
• Foundry hosted AgentKit image recalls the seeded memory canary through /responses.
• Logs show no Memory Store embedding auth errors during validation.

Non-goals

• Do not add core schema such as foundry.memoryStore.
• Do not depend on session-history recall as proof of Memory Store parity.
• Do not delete shared validation resources as part of the fix unless explicitly approved.

[sozercan]

This was generated by AI during triage.

Additional investigation update

Follow-up validation after creating this issue confirmed the main risk: same-session recall is not sufficient proof of durable Memory Store parity.

Observed behavior:

• Standalone AgentKit memory image can start and answer memory prompts, but a fresh second-turn check did not reliably recall the newly provided canary.
• Hosted /invocations and /responses both returned HTTP 200 after the session propagation and stream: true tolerance fixes.
• Hosted same-session recall can succeed, but that can be satisfied by conversation/session history.
• Hosted /responses returned a previously used canary in one run instead of the newly seeded prompt value, which suggests stale scope/session state or provider behavior rather than a clean durable-memory proof.
• Earlier logs showed Memory Store search failing against the text-embedding-3-small deployment with an Azure OpenAI authentication error. Later runs did not always show that log, but durable memory retrieval still was not proven.

Recommendation remains: seed Memory Store directly through the Memory Store API with a unique scope, verify the memory can be listed/searched through the API, then query it from a fresh AgentKit session/container and fresh hosted agent session. Do not rely on "remember this" chat prompts as the seed step.