feat(inference): connections declare what their model can be asked for

docs/api.md

@@ -208,6 +208,24 @@ set. `tests/architecture/test_capability_reachability.py` is what now measures i
 of `BatchAction` is resolved against the published paths and the MCP tool listing, computed from
 the enum rather than from a list somebody maintains.
 
+**What a resource *is* is a second declaration, and not the same question.** `ConnectionOut`
+carries `capabilities` beside its `allowed_actions`: an action is something you may do **to** the
+connection and is decided by its state, a capability is what its model **answers** and is decided
+by the weights. Offering a tool needs both — a connection being `ready` says its files are here,
+not that they are the right kind of model — and either list may be empty without the other being.
+
+```
+GET /inference/connections/{id}  →  { "setup_state": "ready",
+                                      "allowed_actions": ["download_weights", …],
+                                      "capabilities": ["point_suggest"], … }
+```
+
+An empty `capabilities` is not a refusal to act on: the server judges every request on its own
+either way. It says only that nothing can yet rely on this connection for a particular tool —
+because its weights never arrived, because its config declared a model type this build has no
+adapter for, or because it is an `http` connection, which declares nothing until the remote
+contract says how an endpoint states what it can do.
+
 **A client renders these; it never computes them.** Re-deriving the rules from `state` and
 `progress` is what the browser used to do, and its copy drifted by dropping the batch-state
 dimension — which is why skipping a frame was offered on a batch the kernel refused every write

docs/architecture/backend/inference.md

@@ -48,6 +48,36 @@ A family this build does not serve is refused rather than guessed at — a fallb
 answers in the wrong adapter's vocabulary, which is a confident sentence about a
 model the user does not have.
 
+## One fact, two readings, one module
+
+`families.py` holds the family sets **and** the map from a family to what a
+connection may be asked for, because they are the same fact read twice: which
+adapter can run this model, and which prompts a caller may send it. The map is
+*derived* from the sets rather than listed beside them, so an adapter and its
+declaration are one edit — a family added to `SEGMENTER_FAMILIES` and forgotten in
+a hand-written map would run fine and declare nothing, and every client that
+filters on the declaration would stop offering it.
+
+The vocabulary itself is the kernel's (`ModelCapability`) and the mapping is not:
+what a tool can ask for is a domain word, while which `model_type` values this
+build serves is a fact about an optional runtime that the kernel has no view of.
+
+## The family is recorded, not only resolved
+
+Resolution used to happen on every provider build and be thrown away. A connection
+now stores what its config declared, written when the download finishes — the
+first moment the answer exists without reaching a network, and the reason nothing
+is read at connection *creation*.
+
+**The backfill for older rows is on the read path, and that is a layering fact.**
+A migration would be the natural home and cannot be one: migrations run inside the
+kernel, and the kernel may not import this package or address the model cache the
+answer lives in. So `with_families` fills a row in on the first read of it, from
+files already on the disk, once — a row that has an answer is never asked again,
+including when the answer is "the config declared nothing". A build without the
+optional runtime records nothing rather than recording that it found nothing,
+because a build that cannot look has not looked.
+
 ## Importing this package imports nothing heavy
 
 Every reference to torch, transformers, accelerate and huggingface_hub is inside a

docs/architecture/backend/wire.md

@@ -52,6 +52,13 @@ Leaf encoding is explicit throughout: UUIDs as strings, enums as `.value`, paths
 as strings, timestamps in pydantic's format so the parity gate compares like with
 like.
 
+Almost everything a projection reads is a domain model. The one exception is
+`capabilities_of` from `visionset.inference`, and it is here rather than spelled
+out because which model families this build serves is a fact this package has no
+way to know — a second copy of that map would be exactly the drift every other
+rule in this file prevents. The direction is the usual one: a sibling below the
+surfaces, importing nothing from here.
+
 ## Where it sits
 
 The `Kernel purity` contract forbids `visionset.kernel` importing

docs/inference.md

@@ -213,10 +213,44 @@ on a machine that has one; `precision` is `fp16` or `fp32`, and `float16`, `half
 members. What this closes is a gap rather than a freedom: `gpu` used to be accepted and then
 resolved onto the CPU, so the connection described a run that never happened.
 
+## What a connection can be asked for
+
+A connection row says where a model runs and whether its weights are here. Neither answers the
+question a caller has to settle *before* asking: does this model take the kind of prompt I am
+about to send? So a connection also declares what it can be asked for.
+
+```json
+{ "setup_state": "ready", "capabilities": ["point_suggest"], … }
+```
+
+| Capability | Means | Families |
+| --- | --- | --- |
+| `point_suggest` | Give me the thing under these points | the SAM 2 family |
+| `text_detect` | Find everything these words name | the grounding-dino family |
+
+**Read from the model, never from its name.** The value comes from the `model_type` the
+downloaded config declares — the same fact that decides which adapter runs it, so a model that
+runs and a model that declares are the same list. Matching on a model id would answer confidently
+for every model this build has never heard of, and the wrongness would only surface as a refusal
+deep in a request.
+
+**Empty means nothing is known yet**, which happens four ways: the weights were never fetched, so
+nothing has read a config; the config declared no model type; it declared one this build has no
+adapter for; or it is an `http` connection, whose model runs elsewhere and which declares nothing
+until the remote contract says how an endpoint states what it can do. Empty is not a refusal —
+the server still judges every request on its own. It says only that no tool can rely on this
+connection.
+
+**It is recorded when the weights arrive**, because that is the first moment it is knowable
+without reaching a network. Editing a connection to point at another model or revision clears it
+again: nothing has read the new one, and a stale answer reads exactly like a fresh one. A
+connection created before this shipped acquires its answer the first time something reads it,
+from files already on your disk.
+
 ## Suggesting a shape from a click
 
-A connection whose model answers *places* rather than *words* can propose a shape for whatever
-sits under a point. One call, one asset, one set of points:
+A connection whose model answers *places* rather than *words* — one declaring `point_suggest` —
+can propose a shape for whatever sits under a point. One call, one asset, one set of points:
 
 ```http
 POST /inference/suggest
@@ -278,6 +312,10 @@ problem. A connection whose weights are not here yet is `INFERENCE_CONNECTION_NO
 names `download` as the remedy; one whose model answers words rather than places is
 `UNSUPPORTED_PROMPT`.
 
+That last one is still the law and is still enforced on every call. It is simply no longer how a
+person finds out: a client with `capabilities` in hand can decline to ask, which is why the
+editor now says so once on the panel instead of collecting one refusal per click.
+
 ## What a connection is not
 
 It is **not a credential store**, yet. An HTTP connection carries no secret today, and the field
@@ -356,6 +394,11 @@ about that flow forces you out of the editor or loses work: the panel is an expl
 door, and the door is optional — a host that wires no destination gets the explanation and no
 control.
 
+*No usable connection* covers one case more than it reads: a workspace can hold a connection that
+is configured, downloaded and running, and still have nothing that can answer a click, because
+the model it holds answers words. The panel says which of the two it is — nothing set up, nothing
+downloaded, or nothing of the right kind — and each names a different thing to do.
+
 ## At a terminal
 
 ```bash

docs/persistence.md

@@ -155,8 +155,10 @@ MIGRATIONS: list[Migration] = [
     Migration(version=3, name="annotation_provenance", upgrade=_add_annotation_provenance),
     Migration(version=4, name="job_queue", upgrade=_add_job_queue),
     Migration(version=5, name="schema_provenance", upgrade=_add_schema_provenance),
+    Migration(version=6, name="inference_connections", upgrade=_add_inference_connections),
+    Migration(version=7, name="model_family", upgrade=_add_model_family),
 ]
-FORMAT_VERSION: int = MIGRATIONS[-1].version  # 5
+FORMAT_VERSION: int = MIGRATIONS[-1].version  # 7
 ```
 
 **Generation 1 is the baseline, and everything after it is an ordinary migration.** A long
@@ -172,9 +174,16 @@ force again for every entry appended after the baseline.
 `tests/kernel/test_migrations.py` that builds an old-looking file. The failure is the silent
 kind: a column left in place makes its own migration find the column already there and return
 early, so `test_a_fresh_database_and_a_migrated_one_have_the_same_schema` compares a file
-against itself and passes while proving nothing. Migration 4 is the standing exception — it
-creates a *table*, and dropping that in the helper would exercise SQLite rather than this
-module.
+against itself and passes while proving nothing. The table-creating migrations — 4 and 6 — are
+the standing exception: dropping a whole table in the helper would exercise SQLite rather than
+this module.
+
+**A migration cannot always backfill what it adds, and saying which is which is part of adding
+one.** Migration 3 could attribute an annotation because the file already recorded enough to
+answer it; migration 7 cannot fill in a connection's model family at all, because that answer
+lives in a model cache the kernel is forbidden to reach. Where the value is unknowable here, the
+column arrives NULL and something outside the kernel fills it in later — and the column's own
+docstring says which, so a reader does not mistake an honest absence for a forgotten step.
 
 **There are no downgrade paths, deliberately.** Nothing walks a file backwards and the
 tests no longer do either. A downgrade is a compatibility promise and a promise is owed

docs/ui.md

@@ -418,6 +418,22 @@ want and a segmentation model proposes its shape. It runs through a model
 connection (`docs/inference.md`), and the server side of it is
 `POST /inference/suggest`.
 
+**It runs through a connection that can answer a click**, which is a narrower set
+than "the ones that are ready": only those declaring `point_suggest`. A workspace
+whose only downloaded model answers text prompts gets a panel saying so, and no
+request is sent — the server would refuse each one truthfully, which is a correct
+answer to a question the editor should not have asked. The panel tells that case
+apart from having nothing configured and from having nothing downloaded, because
+each is a different thing to go and do.
+
+Where more than one connection can answer, the panel carries a picker naming the
+model under each, and the choice is remembered **per project** — it is a
+preference about this browser, so it survives leaving the editor and does not
+become a workspace setting that everybody annotating shares. With one candidate
+there is no control at all, only a line naming what is answering. The picker
+appears on the idle card alone: changing which model answers while a proposal is
+on screen would leave a shape nothing on the card explains.
+
 The gesture:
 
 | Press | What it does |

frontend/ui-core/src/annotator/AnnotationPage.tsx

@@ -199,6 +199,19 @@ import { AddClassDialog, runAddClass } from "./AddClassDialog";
 import { FrameGallery } from "./FrameGallery";
 import { SuggestPanel } from "./SuggestPanel";
 import { useConnections, useSuggestRegion, usableConnection } from "../data/inferenceQueries";
+import { readPref, writePref } from "../data/prefs";
+
+/**
+ * Where a project's suggest-through choice is remembered.
+ *
+ * Keyed by project rather than globally: two projects can hold different
+ * schemas and different work, and the model that suits one is not the model that
+ * suits the other. Keyed by project rather than by *job* for the opposite
+ * reason — nobody wants to re-pick a model per batch.
+ */
+function preferredConnectionKey(projectId: string): string {
+  return `suggest.connection.${projectId}`;
+}
 import { PROGRESS_LABEL, outstandingWork, progressDotClass, progressTone } from "../screens/batchState";
 import type { LabelClassBody, SchemaDiff, SchemaVersion } from "../screens/queries";
 import {
@@ -799,9 +812,34 @@ function Workspace({
    * than leaving a click to vanish into it.
    */
   const connections = useConnections(session !== null);
-  const { connection, blocker } = usableConnection(connections.data?.items);
+  /**
+   * Which model this project suggests through, when there is more than one.
+   *
+   * **A preference, per project, and never a constraint.** It survives leaving
+   * the editor and coming back — which is the whole point of remembering it —
+   * and `usableConnection` falls back to the first candidate whenever the
+   * remembered one is gone, renamed away from capability, or not downloaded any
+   * more. So a deleted connection cannot leave a project unable to suggest.
+   *
+   * `readPref`/`writePref` rather than server state: it is a view preference in
+   * `prefs.ts`'s own sense — a choice about this browser, not a fact about the
+   * workspace, and one that must not turn into a write every annotator on a
+   * shared workspace fights over.
+   */
+  const [preferredConnection, setPreferredConnection] = useState<string | null>(() =>
+    readPref(preferredConnectionKey(projectId)),
+  );
+  const { connection, candidates, blocker } = usableConnection(
+    connections.data?.items,
+    preferredConnection,
+  );
   const suggestRegion = useSuggestRegion();
 
+  function chooseConnection(connectionId: string): void {
+    setPreferredConnection(connectionId);
+    writePref(preferredConnectionKey(projectId), connectionId);
+  }
+
   /**
    * Arming and disarming — and arming activates a class, exactly as every other
    * button on the strip does.
@@ -2430,6 +2468,9 @@ function Workspace({
                 heldClass={activeClass}
                 blocker={blocker}
                 refusal={suggesting.refusal}
+                candidates={candidates}
+                connectionId={connection?.id ?? null}
+                onChooseConnection={chooseConnection}
                 onAccept={acceptSuggestion}
                 onDiscard={discardSuggestion}
                 {...(onConfigureInference === undefined