Image input (vision): let vision-capable models see images from chat, files, and tools

[acoliver]

Problem

The agent is text-only end to end today:

• Message (src/llm/client.rs:797) carries role, text, thinking, tool uses. There is no attachment field.
• build_model_requests (src/llm/client.rs:248) emits only text and tool parts.
• parse_response (src/llm/client.rs:752-754) explicitly drops image parts: "Ignore other parts (tool returns, images, etc.)".
• UserEvent::SendMessage { text, conversation_id } (src/events/types.rs) is text-only, so the UI has no way to hand an image to a turn.
• The ReadFile tool (src/agent/tools/read_file.rs) treats images as binary and never returns their content.
• MCP tool results that contain images render as the literal string "[image]" (src/llm/client_agent.rs:449).

Meanwhile the model registry already knows which models accept images: Modalities { input: Vec<String> } parses models.dev modalities.input including "image" (src/registry/types.rs:196-219). And the pinned serdesAI core (rev aa0eb871) already models binary media: BinaryContent { data, media_type }, FilePart::from_bytes, ToolReturnContent::Image, and response output_files (serdes-ai-core/src/messages/parts.rs). So most of the vocabulary exists; the app layer just never uses it.

Reference implementation on disk

llxprt-code (~/projects/llxprt/branch-1/llxprt-code):

• Capability gate: LlxprtModelCapabilitiesSchema with a vision: boolean derived from the models.dev registry (packages/core/src/models/schema.ts:108).
• Media block contract: ClassifiedMediaBlock { mimeType, data, encoding?, filename?, mediaType } shared from providers into core (packages/core/src/runtime/contracts/MediaBlockContracts.ts).
• Admission control: packages/core/src/storage/media-admission-service.ts decides which media types may enter a session.

Prospective design

1. Domain model: add Attachment { mime_type: String, data: AttachmentData, filename: Option<String> } to Message, where AttachmentData holds bytes (not base64 strings) plus a source reference (clipboard, file path, tool result). Keep text as the primary content; attachments are additive.
2. Request build: build_model_requests maps each attachment to a serdesAI FilePart::from_bytes part on the user request. Verify the pinned serdes-ai-providers rev actually serializes FilePart into Anthropic image blocks and OpenAI image_url blocks; if it does not, that work lands upstream in serdesAI first (we already fork it at acoliver/serdesAI).
3. Capability gate (fail fast): helper supports_vision(provider_id, model_id) reading modalities.input. Sending an attachment to a model without image input returns a clear error naming the model and suggesting a vision-capable profile, instead of a provider 400. The composer disables or greys the attach action when the active profile lacks vision.
4. Entry points:
   • Paste: the chat input already handles Cmd+V/Ctrl+V for text (PR Accept Ctrl+V paste on macOS and load MCP edit drafts from config.json #247). Extend paste handling to detect ClipboardItem::image and stage it as an attachment chip under the composer.
   • File: /image <path> composer command plus drag-and-drop on the chat view.
   • Tools: ReadFile returns a ToolReturnContent::Image part (not "[image]") when the path is an image and the active model has vision; MCP image returns flow through unchanged.
5. Events: SendMessage grows attachment_paths: Vec<PathBuf> (paths and mime metadata only, never bytes) so the event bus stays lean; the service loads bytes at request-build time, mirroring the llxprt principle that capability boundaries carry bounded metadata, not base64 payloads.
6. Persistence: conversations store attachment references (filename, mime, size, hash) with bytes under a per-conversation media dir (sibling of the conversations dir). Downscale large images at attach time to bound storage and context cost; keep the original only if the user opts in.
7. Response side: parse_response keeps ignoring model-emitted images for now (chat models we use do not emit them); log a warning instead of a silent drop.

Acceptance criteria

• Pasting a screenshot into the composer on a vision-capable profile (e.g. glm-4.6 via ZAI, gpt-5.6) sends it and the model accurately describes the image.
• Attaching on a non-vision profile produces a named-model error, not a raw provider failure.
• Images survive app restart via the conversation store and are re-sent on follow-up turns only while the conversation references them.
• ReadFile on a PNG returns image content to a vision model.
• No image bytes ever pass through the event bus or land in config.json.

Open questions

• Does the pinned serdesAI providers rev serialize image blocks for both Anthropic-compatible and OpenAI-compatible wire formats, or does that need an upstream PR?
• Do we want audio/PDF parts in the same pass (modalities.input already lists them) or images only first?