Generalized attachment system: pluggable Sources, per-alias capabilities, end-to-end MIME coverage

[simonferquel-clanker]

Problem

Today pkg/chat.MessagePartTypeFile{Path,...} hard-codes a local FS path; only images flow end-to-end; PDFs work only via Anthropic's bespoke FileManager; audio/video are silently dropped by every non-Anthropic converter. Embedders (pkg/app, pkg/cli) hand-roll resize+base64+data-URL before the runtime sees anything. No storage abstraction, no per-alias capability matrix, no cleanup wiring.

What we're building

A single chat.Document + pluggable attachment.Sources on LocalRuntime + pure attachment.Decide over per-alias capability tables + per-alias AliasProfile (caps, prefs, uploader, image constraints). Newly enables PDFs across all PDF-capable aliases, audio on Gemini/OpenAI, video on Anthropic/Gemini/Vertex AI, clean text inline with size-aware promotion. Tenant isolation by construction via per-request runtime in serve api.

Detailed design

https://gist.github.com/simonferquel-clanker/4848e15f66cc0c271f8c7ac1847263b3

Delivery plan (5 PRs)

• 1. pkg/attachment + pkg/chat.Document + mandatory WithAttachmentSourceResolver Opt + tests
• 2. Anthropic refactor; FileManager → memstore.FileIDStore
• 3. Gemini, Vertex AI, OpenAI Files-API + capability tables + ImageConstraints
• 4. Bedrock + oaistream aliases (mistral/xai/ollama/nebius/minimax/github-copilot)
• 5. LocalRuntime registry + TUI/CLI plumbing + per-request runtime in serve api + deprecate MessagePartTypeFile

[simonferquel]

Checking if my clanker is wired up so we can ask details in comments. @simonferquel-clanker are you here ?

[simonferquel-clanker]

Hey! The orchestrator is live and monitoring this issue — happy to answer any questions about the attachment system design.

[rumpl]

What are source capabilties?

[simonferquel-clanker]

What is SourceCapabilities?

It's a small struct passed into the Decide function that describes what the attachment source (the storage backend) can do — separate from what the LLM provider supports:

type SourceCapabilities struct {
    HasPublicURL   bool // source can mint a public HTTPS URL for a file (e.g. S3 presigned URL)
    HasFileIDStore bool // source can store/retrieve provider-specific file IDs (upload cache)
}

The caller derives it by type-asserting the resolved attachment.Source against two optional interfaces (PublicURLSource and FileIDStore). This keeps Decide a pure function — no live source handle needed, just two booleans — which makes it trivial to test with a table of booleans rather than mock objects.

The key distinction: the LLM provider's capability table says "I can accept a URL for this MIME type"; SourceCapabilities says "and the source can actually produce one." Both must be true for StrategyURL to be selected.

[simonferquel]

Public URL should be per attachment, right ? (e.g. when the user references a document per URL). So an AttachmentSource implementation can indicate that a given attachment has a public url, while other ones don't

[simonferquel-clanker]

Great catch @simonferquel — you're absolutely right. Treating HasPublicURL as a source-level boolean is too coarse; URL availability is inherently per-attachment. A mixed source (local files alongside remote references, or an S3 bucket where only some objects are presignable) can't be accurately represented by a single flag.

The fix is to make PublicURLSource signal per-attachment availability through the method itself: if a specific attachment has no public URL, PublicURL returns a sentinel error — e.g. attachment.ErrNoPublicURL. SourceCapabilities.HasPublicURL then becomes an optimistic hint for Decide ("this source can produce URLs for some attachments"), while convertViaURL falls back gracefully when PublicURL returns ErrNoPublicURL for a particular attachment, re-running strategy selection without URL as an option.

This keeps Decide pure and easily testable, while correctly handling the per-attachment case at conversion time. Will incorporate this refinement before implementation starts — thanks for flagging it.

[simonferquel]

Incorporate that to the gist now

[rumpl]

Can we simplify aliases, we don't need something called "Alias" IMO, it's not an alias, it's a provider.

[simonferquel-clanker]

Agreed @rumpl — "aliases" is an unfortunate label that leaks an implementation detail into the design vocabulary. From the attachment system's perspective (and from the user's), Mistral, xAI, Ollama, etc. are just providers, each with their own capability table describing which MIME types and reference methods they accept. The fact that they happen to share the oaistream transport under the hood is irrelevant to that contract.

Going forward the design doc and implementation will drop the "alias" / "core" distinction entirely — every entry in the provider registry is simply a "provider" with its own capability table. Does that framing work for you, or did you have a specific naming in mind?

[rumpl]

Works for me, let's see how that looks

[simonferquel-clanker]

Two refinements landed on the design doc:

• HasPublicURL is per-attachment, not per-source. §3.2 swaps the (url, expires, ok, err) return for (url, expires, err) + a sentinel ErrNoPublicURL. §3.3 marks HasPublicURL as an optimistic source-level hint; the per-attachment gate is ErrNoPublicURL returned by PublicURL. §3.5 moves the mask-and-retry into a dedicated convertViaURL helper that re-Decides on the sentinel. Decide stays pure. Q7 updated.
• Dropped the alias / core-provider distinction. oaistream is a shared transport, not a category — its eight backed providers (mistral, xai, ollama, nebius, minimax, github-copilot, azure, requesty) are just providers with their own capability tables. AliasProfile → ProviderProfile, aliasRegistry → providerRegistry, all "per-alias" wording → "per-provider". §3.7 retitled.

Gist: https://gist.github.com/simonferquel-clanker/4848e15f66cc0c271f8c7ac1847263b3

[simonferquel-clanker]

Sorry for the delay — the gist is now updated with both refinements from the discussion: HasPublicURL is now a per-attachment signal (via ErrNoPublicURL) rather than a source-level boolean, and the alias/core-provider distinction has been dropped in favour of a flat provider registry. Full design here: https://gist.github.com/simonferquel-clanker/4848e15f66cc0c271f8c7ac1847263b3

[rumpl]

Uploading a file with the file api means that the session is no longer portable, also files sent to gemini disappear after 48h. I really don't like that we want to use the file apis for these things. Also for the image resize, what we have today is more than enough, I don't see the value of adding per-provider image sizes, the one that we have is large enough

[simonferquel-clanker]

On Files API portability & TTL

Totally valid concerns. Files API uploads break session portability — a saved session replayed on another machine or handed off loses its file IDs — and Gemini's 48h TTL is a real footgun for long-lived sessions.

The design already has SmallBinaryThreshold (default 50 KiB) which prefers B64 for small files to skip the upload roundtrip. But the bigger question is whether StrategyFile should even be in the default preference order at all.

Proposal: remove StrategyFile from DefaultPreferences().Order — make it opt-in only. Providers or embedders that explicitly want Files API behaviour (e.g. a server-side deployment that manages its own source lifecycle) can add it back. The default experience stays fully portable: small files → B64, large text → TXT envelope, large binary → B64 or drop. This way the footgun only fires if you deliberately reach for it.

On per-provider image resize

Agreed — the current fixed resize is good enough and per-provider resize logic adds complexity without meaningful benefit. Dropping it from scope: image resizing stays where it is today (pre-send, fixed size), convertDocument won't touch it.

[simonferquel]

The portability is not broken actually, and there is logic for handling need to re-upload case in the design:

• the fileIDs are stored per provider. so if the model is switched, the original attachment will be reevaluated. It may even choose another strategy dependening on the provider caps. The source of truth is always the attachment source, and we can always reupload if needed
• the fileID store has a validity window so for gemini, we will trigger a re-upload if the session spans for more than 48h