MCP-1E: Add multimodal MCP input handling for local paths and inline payloads

[Atharva-Kanherkar]

MCP-1E: Add multimodal MCP input handling for local paths and inline payloads

Summary

Add a practical MCP input layer for screenshots, audio, video, and PDFs so multimodal memory works without depending on FastAPI upload endpoints.

Scope

• local file-path input support
• base64 input fallback for non-local clients
• temporary-file materialization for inline payloads
• file-size limits and validation errors
• validation that local paths are readable from the server filesystem

Acceptance Criteria

• screenshot, audio, video, and PDF-backed memories can be stored via MCP
• text recall can find media-backed memories
• remote deployments fail clearly when callers provide local paths that are not readable from the MCP server host
• v1 returns metadata and durable paths, not raw binary blobs

Dependencies

• Parent: MCP-1: Add an MCP server layer for agentic-memory #55
• Depends on: MCP-1A: Extract a shared runtime factory and remove global config mutation #58, MCP-1C: Define MCP tool schemas, routing, and bounded response contracts #60
• Blocks: MCP-1F: Package the server and document setup, ops, and migration guidance #63

[Atharva-Kanherkar]

@claude if this is skipped and we implement 5, directly, will it cause any problems other than not able to use multi modal things?

[claude]

Claude finished @Atharva-Kanherkar's task in 1m 33s —— View job

---

Answer

• Gather context
• Analyze dependency chain and codebase impact
• Answer question

---

Short answer: Yes, skipping #62 and jumping to #63 will cause real functional problems beyond just missing multimodal input — but the severity depends on what #63 needs to do.

What #62 actually provides (already implemented in this branch):

Looking at the current code, #62's implementation is already complete in this branch:

• MediaInputModel + InlineContentModel schemas (mcp_server/schemas.py:67-77) — the data structures all of remember_fact, remember_episode, remember_procedure use for media input
• PreparedMedia dataclass + _prepare_media() (mcp_server/server.py:118-465) — materializes temp files from base64, validates local paths exist and are readable, resolves MIME types
• _resolve_media_contract() (server.py:477-506) — validates that modality is consistent with the attached media type
• _cleanup_prepared_media() — cleans up temp files
• MediaTooLargeError handling and clear remote-path error messages

Why skipping it causes more than just "no multimodal":

If #63 is implemented without these pieces, you'll hit concrete breakage:

1. Missing schema types — if MCP-1F: Package the server and document setup, ops, and migration guidance #63 adds new tools or extends existing ones that accept a media parameter, there's no MediaInputModel to validate against. That's a hard import/validation error.
2. No temp-file lifecycle — inline base64 payloads need to be written to disk before the store layer touches them. Without _prepare_media, any media-backed memory store call would either receive None or an unresolved base64 string as a file path.
3. No modality↔media_type contract — _resolve_media_contract guards against storing a modality="audio" record with an image attached. Without it, bad data silently enters the stores.
4. Remote-path safety — the check that raises FileNotFoundError for unreadable local paths (line 426–430) is part of MCP-1E: Add multimodal MCP input handling for local paths and inline payloads #62. Without it, callers get an opaque internal error instead of a clear message.

Practical recommendation:

Since the code for #62 is already implemented in this branch, the cleanest path is to merge #62 first (review → PR → merge), then build #63 on top of it. Implementing #63 without merging #62 means you'd need to duplicate or re-implement the media infrastructure inline in #63, which creates divergence and makes #62's PR moot.

If you're asking whether #63's non-media functionality could be built independently — yes, parts of it likely could. But any path through #63 that involves storing or retrieving media-backed memories would be broken or require re-implementing what #62 already provides.