Open providers over closed capabilities: the provider contract and plugin architecture

[JArmandoAnaya]

Why this shape

The standing invariant — zero orphaned capabilities, none declared without a surface that
renders it — is what closes the capability vocabulary. If plugins could add capabilities, every
installed plugin would create orphans by definition. Closing capabilities is also what lets a
single generic dashboard — the Inference screen organized by what a connection enables (#499) —
render any provider without plugins shipping frontend code.

So: closed capabilities, open providers. ModelCapability
(src/visionset/kernel/domain/inference.py) grows only by a deliberate kernel change shipped
alongside the surface that consumes it. What a plugin adds is a driver for a capability the
application already defines.

The provider contract (to be designed here)

A provider must be able to:

• identify itself and the contract version it targets;
• declare the families it serves and its curated entries (pinned revisions, sizes);
• price a download for an exact snapshot;
• fetch weights into the content-addressed cache;
• run predictions over domain types for the capabilities it maps to;
• refuse what it cannot do, in prose, under the unknown-type refusal rule established when
  sam2_video was recognised as a point-prompt model and an unknown type made to refuse (fix(inference): sam2_video is a point-prompt model, and an unknown type refuses #458):
  never guess, and name what is supported.

Capability mapping is derived from declared families through the same mechanism
src/visionset/inference/families.py uses today: CAPABILITY_BY_FAMILY is built from
SEGMENTER_FAMILIES and DETECTOR_FAMILIES with dict.fromkeys, never written out a second
time, so adding a family and declaring its capability are one edit. A provider never hand-writes
its capability map; the change that made connections declare what their model can be asked for
(#486) proves that derivation with a mutation test.

Layering

ModelProvider (src/visionset/kernel/ports/model_provider.py) stays a pure kernel port.
Provider authors get a toolkit layer above it — working name visionset.inference.sdk — offering
the kernel's device and precision rules (precisions_for), cache access
(src/visionset/inference/cache.py), the digest discipline in
src/visionset/inference/integrity.py that came with the weights integrity check (#475),
integration with the download job that reports its bytes (#493), and refusal prose helpers.

The layer carries an explicit contract version. The contract is experimental until
declared stable in a later release: everything is free before first publication and breaking
after. That is the same lesson taught by the suggest route's detail field, which was accepted,
published and then ignored (#463) — applied deliberately this time.

Discovery

An entry-point group, mirroring the exporter precedent: pyproject.toml already registers nine
exporters under [project.entry-points."visionset.formats"].

The group name is an open naming decision (candidate: visionset.providers) — flagged
deferred-needs-input on this issue, to be decided before the first implementation PR that
registers one.

Degradation and trust

A connection whose provider is not installed does not break and is not deleted: it renders
with prose naming the missing provider and coherent allowed_actions — the missing-format
treatment. Documentation states plainly that installing a plugin is trusting its author with code
execution in the workers, exactly as any pip package.

0.1.0 scope vs. later

In 0.1.0: the contract exists internally; both shipped adapters —
src/visionset/inference/sam_provider.py (point-prompted, SAM 2 family) and
src/visionset/inference/transformers_provider.py (text-prompted, Grounding DINO family),
resolved by provider_for in src/visionset/inference/providers.py — are migrated to consume it;
the discovery mechanism works for in-tree providers; nothing is documented as a public API.

Later (its own release, its own decision): contract declared stable, public docs, external
plugin support announced.

The contract is not published as a public extension API in 0.1.0, but the shipped local adapters
must run on it — the contract is proven by our own drivers before any publication. Plugins are
backend driver + declarative metadata only; plugin-supplied UI is out of scope. Nothing here
touches the http connection type or precludes hosted execution.

Sub-decisions this epic must produce

Each recorded on this issue before its slice:

1. Contract version scheme.
2. Entry-point group name (deferred-needs-input).
3. Exact toolkit surface.
4. How a provider's curated entries merge into the create-form dropdown, meaning
   frontend/ui-core/src/screens/inferenceCatalog.ts and the grouped model select that shipped
   with the curated model list and download lifecycle (feat(inference): curated model list, closed device and precision fields, download lifecycle #470).
5. The conformance test suite a provider must pass; the two shipped adapters are its first
   subjects.

[JArmandoAnaya]

Decision (Armando, 2026-08-10): Closed capabilities, open providers. Plugins may add
providers (drivers) for capabilities the application already defines; a plugin can
never introduce a capability. A new capability arrives only with a VisionSet release,
together with the surface that renders it. The provider architecture is part of
milestone 0.1.0: it is not published as a public extension API yet, but the shipped
local adapters must run on it, so the contract is proven by our own drivers before any
publication. Plugins are backend driver + declarative metadata only; plugin-supplied UI
is out of scope.

[JArmandoAnaya]

Designing this epic's discovery mechanism surfaced a contradiction in the wire's forward-compatibility rules: the generated client tolerates an unknown response key but refuses an unknown enum member. That is now tracked as its own sub-decision (#504), because an open provider set is exactly the case where a vocabulary grows without a coordinated release on both sides.

[JArmandoAnaya]

Decision (Armando, 2026-08-16): This issue now owns the hosted ModelProvider adapter scope, transferred from #418 (whose port deliverable shipped). The provider contract designed here must cover the hosted/cloud execution path — local-first, cloud-compatible after. — supersedes any prior direction.

[JArmandoAnaya]

Decision (Armando, 2026-08-17): five sub-decisions settle the shape of the provider
contract, under the closed-capabilities rule this issue already carries. Discovery is by
entry point, which reverses an earlier reading; the comment that follows this one records
that reversal and the objection it had to answer.

Contract shape — a descriptor and a factory, with weights a second protocol. A driver
declares provider_id, a families mapping and a curated tuple, and builds a runner for
a connection. Where its weights come from is a separate protocol a driver either
satisfies or does not, rather than three more members whose only implementation would be a
refusal: a driver that runs a model somewhere else has nothing to price, nothing to fetch
and no config to read a family out of. Callers ask which it is.

families is a mapping and not a set, and that is the derivation guarantee rather than
a convenience. It maps a model_type onto a member of the capability vocabulary, so a
family acquires an adapter and a declared capability in the same edit. Declaring the two
separately is how a build ends up running a model that declares nothing, which is invisible
to every client filtering on what a connection can be asked.

Curated entries are served by this workspace rather than held in the frontend. A driver
ships the checkpoints it offers by name, and the application serves that list. Each entry
carries a model id, a pinned revision, its family, a one-line hint and — where a checkpoint
must be cleared before it can be fetched — a note and a URL. It carries no size: a
download's cost is read live, ahead of the confirmation, so a number frozen into a catalog
entry would be a second answer to a question already answered accurately. The frontend's
hardcoded totalBytes retires when the served catalog lands. Group headings stay in the
frontend, keyed off the capability, so a driver ships no copy that renders as a heading.

Compatibility is the plugin's own dependency pin, with an import-time backstop, and the
contract carries no version member. A plugin states which VisionSet it was built for the
way it states everything else it needs — as an ordinary dependency specifier — and the
installation refuses a mismatch before it loads the entry point, so a driver built against
a contract this build no longer speaks fails as a sentence rather than inside a forward
pass. A version member on the contract would be a second, disagreeing gate. One consequence
bites while this project is pre-1.0: a prerelease sorts before its release, so a plugin must
pin a prerelease floor to be usable against a prerelease build at all.

The entry-point group is visionset.providers, matching how exporters are already
discovered.

The hosted execution path is proven, not built, in this epic. The scope transferred here
on 2026-08-16 is satisfied by demonstrating that the contract admits a driver with no
weights at all — declared families, no weights source, a runner it can build — as a fixture
in the conformance suite. The adapter that actually speaks to an endpoint is its own piece of
work, filed as #648, and it is blocked on a second gap filed as #649: a
connection does not record which driver serves it, which is what stops an endpoint from
resolving to one and what stops a download from asking the driver that owns it. Storing an
endpoint's credentials stays where it already is, in #519.

[JArmandoAnaya]

Recorded reversal: providers are discovered through an entry-point group.

Two docstrings in this area used to state the opposite — that there is no entry-point group
for providers, and that this was deliberate. That reading is reversed, and it is worth
recording why rather than quietly deleting it, because the objection behind it was a real
one and the arrangement that replaced it has to answer it rather than ignore it.

The objection was this: a workspace must not be able to predict something merely because a
package was installed. Where a model can run, and whether it may run at all, is a property
of a connection — something somebody created, pointed at a model, and fetched weights for
— and an installation that could quietly acquire new predictive ability by way of a
dependency would put that decision somewhere nobody can see it.

Entry points do not do that, and the distinction is what makes the reversal safe. Discovery
loads nothing and fetches nothing; it reads declarations. Installing a driver makes a
family reachable, and reachable is not present: there is still no connection, still no
weights on disk, and still nothing that will answer a request. The connection remains the
registry, and every existing one goes on running exactly what it ran before. What the
installation gains is an answer to "if somebody creates a connection pointing at this kind
of model, is there something here that could run it" — which is a question about the
installation, not a capability granted to it.

The gain is that a third-party distribution registers the same way an in-tree driver does
and is indistinguishable from one at the point of use, which is the whole premise of open
providers over closed capabilities. A hardcoded table cannot offer that, and the alternative
— a registration call some import must perform — makes the set depend on import order and on
somebody remembering to import a module for its side effect.

The group is visionset.providers, and it matches the mechanism the format exporters have
used since they shipped.

[JArmandoAnaya]

Decision (Armando, 2026-08-17): The default model on the Inference screen is a
product decision and lives in the frontend, not on the contract. It is a model
id matched against the loaded served catalog, falling back to the first
point_suggest entry when that id is absent. No default flag on
ProviderOut or anywhere on the wire — a contract-level default would let a
plugin make a product decision, which the closed capability vocabulary exists
to prevent. Group headings likewise stay in TypeScript, keyed off the closed
ModelCapability, so no plugin ships copy that renders as UI chrome.