Skip to content

Swift SDK: support message attachments (image + file) for agent snapshots #1144

Description

@willwashburn

Summary

The Swift SDK's messaging API (Channel.post, AgentClient.post(to:message:), AgentClient.dm(to:message:)) only accepts text. There is no way to attach binary/image data or a file to a message.

Native apps that want to send agents a visual snapshot of their state can't do it cleanly. The Whiteboard macOS app, for example, needs to send the agent a board snapshot = a PNG render + a scene.json scene graph alongside the user's text.

Request

Add attachment support to the Swift SDK send paths, e.g.:

try await agent.dm(
    to: "designer",
    message: "match the logo color",
    attachments: [
        .image(data: pngData, filename: "board.png"),
        .text(sceneJSON, filename: "scene.json")
    ]
)

and surface attachments on the inbound/observer side so receiving agents can read them.

Notes / open questions

  • How should attachments be transported — inline (base64 in payload), or uploaded to a blob store with a URL/handle in the message (cf. relayfile)? A size threshold that switches between the two would be ideal.
  • Should the TS and Python SDKs get the same surface for parity?
  • For headless CLI-backed agents (codex/claude), delivering attachments as readable file paths in the workspace may be the most useful form.

Motivation

Agent perception in Whiteboard = PNG snapshot + JSON scene graph. Without attachment support we either inline base64 (bloats messages) or write files out-of-band and pass paths. First-class attachment support would make agent perception a one-call operation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions