feat(mcp): real tools over the SDK — thirty-three of them, and an agent that can see (#35)

[JArmandoAnaya]

Closes #35.

The MCP server was the one surface that had never called the kernel: 140 lines of thirteen stubs
with the pre-#27 names, every one returning not_implemented, and visionset mcp printing
"server would start here". It now ships thirty-three tools and an agent can run the entire
cycle over stdio — create a project, declare a schema, ingest, look at the pixels, write model
annotations, promote, publish, verify and export.

Ship-vs-fold, tool by tool (acceptance criterion 1)

Fifty candidates were recorded across PRs #99–#102. Thirty ship, twenty fold or drop, and three
are new or merged, for 33. The parity rule means evaluated, not implemented: tool-selection
accuracy degrades with count, so a tool ships only when an agent has a reason to reach for it that
no neighbour covers.

Ship (33)

Group	Tools
Projects & schema (7)	create_project list_projects get_project delete_project get_schema preview_schema_change create_schema_version
Sources & ingest (3)	ingest list_sources backfill_thumbnails
Batches (7)	list_batches get_batch approve_batch start_batch list_batch_assets complete_batch promote_batch
Jobs & annotations (10)	get_job start_job next_pending_assets get_asset_image list_asset_annotations add_annotations update_annotations delete_annotations set_asset_progress complete_job
Datasets, releases & export (6)	dataset_stats publish_release list_releases verify_release list_formats export_release

Fold or drop (20), with the argument

Folded into a parent — the parent already reads it, so a second tool is a round trip for a
field: get_project_dataset, get_dataset (→ get_project; the dataset is 1:1 and its id is the
handle every release tool needs), list_schema_versions (→ get_schema), get_source (→
list_sources), list_batch_jobs (→ get_batch), get_job_progress (→ get_job), get_asset
(→ get_asset_image, whose structured half carries every published field anyway), get_release
(→ list_releases).

Folded into ingest: register_image_source, register_video_source, start_ingest. The
kernel splits registration because a clip needs a rate and a probe while a folder needs neither;
by the time ingest runs the source already carries the kind, the path and the rate. Dispatch is
path.is_dir() — visionset ingest's shape exactly.

Dropped, nothing to poll: get_ingest_job, list_ingest_jobs, resume_ingest. Ingest is
synchronous (below), so the finished job comes back in the answer. An interrupted run is re-run,
not resumed: registration is idempotent on (kind, path, extraction_fps) and content addressing
makes the re-run free — the same argument that gave the CLI no --resume.

Dropped, no agent caller: list_dataset_assets (the annotation loop iterates batches, not
the trunk), list_dataset_changes (an audit record a person reads), remove_dataset_asset
(curation — a judgement about what a dataset should contain, not a step in producing one),
get_release_manifest (the whole frozen document is a token bill an agent cannot afford;
verify_release answers "is it intact" and export_release writes the contents somewhere usable),
get_release_assignment (export_release puts the folds on disk in the form anything downstream
consumes), rename_project.

Never offered: token administration, per docs/auth.md and #25. That decision held when the
surface actually shipped — none of the 33 touches a token, and none needs one.

Three not on the parity list

ingest (one tool standing for three candidates); preview_schema_change, which gives
SchemaService.preview its first caller since #6 — plan-before-apply matters most for the
surface that cannot see a change's consequences until it has made one; and backfill_thumbnails,
because it is the remedy get_asset_image's refusal names, and a refusal naming an unreachable
remedy is worse than no refusal.

get_asset_image — the finding worth the entry

The pixels an agent sees are not the frame its coordinates live in. A preview is capped at
DEFAULT_THUMBNAIL_MAX_EDGE (256) on its long edge; annotation geometry is always in the
asset's native pixels and is never normalized. An agent that measures a box on a 256-pixel preview
and submits it unscaled produces annotations that are individually plausible and uniformly
wrong — wrong in a way nothing downstream can detect, because every number is in range and every
shape is well formed.

So the structured half publishes both frames and the factor between them:

{ "asset_id": "…", "width": 4032, "height": 3024, "format": "jpeg",
  "image_width": 256, "image_height": 192, "resolution": "thumbnail", "scale": 15.75 }

The preview's dimensions are measured, never derived — the port caps an edge, so the size is
a function of aspect ratio and of whether the asset was smaller to begin with, and a computed
scale would be a guess. It reads them through ImageProcessor.probe, which keeps a decoder out
of a delivery module. tests/mcp/test_agent_walk.py does the multiplication for real.

Verified against mcp 1.28.1, not assumed

• [Image(...), {...}] silently yields no structuredContent. The trap that looks right. The
  supported shape is constructing a CallToolResult yourself.
• Bare CallToolResult, not Annotated[CallToolResult, Model]. The annotated form declares an
  output schema and validates structuredContent against it — which would reject the error
  envelope, since a refusal cannot also be a valid image result. Bare passes it through untouched.
• A tool returning dict[str, Any] populates structuredContent; one returning CallToolResult
  does not, for a returned dict. So guarded wraps the envelope in a CallToolResult for
  exactly the tool that declares one — otherwise get_asset_image would be the only tool in the
  surface whose refusals had to be parsed out of a text block.
• A duplicate tool name does not raise: FastMCP logs a warning and discards the second
  registration. test_registration.py counts the listing against TOOLS to catch it.
• inspect.cleandoc is mandatory. FastMCP ships __doc__ raw into the tool listing,
  indentation and all.
• Parameter documentation only happens through Annotated[..., Field(description=...)] —
  there is no docstring-argument parser. A test asserts every parameter of every tool has one.
• from __future__ import annotations cannot coexist with a module named annotations —
  server: batch/job endpoints — approve, partition, "next N pending assets", annotation submission, progress (the third-party-app contract) #29's trap, hit again. mcp/main.py drops the future import and says why.

Conventions

Domain models go straight into tool signatures — list[LabelClass], Geometry, SplitRecipe,
AssetProgress. The opposite call from REST, and for a reason: FastAPI copies a docstring into
openapi.json and turns a PEP 695 alias into a named component, so server/models.py keeps its own
spellings; FastMCP puts the same docstrings into $defs on the input schema, where they are the
best guidance an agent gets. The domain's validators then refuse malformed input with the field
path, and nothing is restated. The one exception is mcp/annotations.py's two input models, which
omit schema_version because the service overwrites it — a required input whose value is discarded
is a lie in the schema.

The inherited wart is stated rather than hidden: a discriminated union's tag carries a default in
the domain, so type shows as optional while pydantic needs it to pick a variant.
test_a_geometry_with_no_type_cannot_pick_a_variant pins it and the tool descriptions say to spell
it out.

Two failure shapes, deliberately. A malformed request is isError with the validator's
message naming the field (the API's 422); a domain refusal is an ordinary result carrying one
envelope (the API's 404/409). The envelope has four keys, always present:

{ "error": { "message": "…", "retry_with": "allow_destructive", "hint": null, "index": null } }

No code field, deliberately. The codes live in server/errors.py, which this package may not
import, and #31 forbids deriving one from a class name. What a code was needed for is one question
— "may I retry this, and with what?" — and retry_with answers it directly:
DESTRUCTIVE_SCHEMA_CHANGE gets "allow_destructive" and SCHEMA_CHANGE_WOULD_ORPHAN gets
null, which is exactly the distinction #31 warned would put a client in a retry loop. The three
gate words stay three and a test holds them to it.

Registration is one table in main.py — the CLI's rule, for the CLI's reason (a decorator in
projects.py would make it import main.py, which imports it). It is also the single place
guarded, cleandoc and the read/write ToolAnnotations are applied, so none can be forgotten.
Hints are hints and confirm is what enforces, so a test asserts the two agree in both directions.

The workspace is opened per tool call and named only by the environment. No tool takes a
workspace parameter — threading one through 33 tools puts a path an agent cannot know into every
call. Per-call rather than held: no module-level state, every tool testable with monkeypatch.setenv
alone, and — since SQLite has one writer — a stdio server holding the file would keep visionset ui
and a second agent out of a workspace nobody is using.

visionset/wire/ — promoted, not copied

cli/_json.py becomes visionset/wire/, a surface-agnostic package the CLI and MCP both import.
A second hand-written spelling of the same twenty shapes is what that rule exists to prevent. The
direction stays one-way and machine-enforced: visionset.wire joins the kernel-purity contract's
forbidden list beside the three delivery packages. The server keeps its pydantic models, because
openapi.json is generated from them.

Seven projections the CLI never needed: schema_change, schema_diff, batch_asset, geometry,
annotation, class_count, dataset_stats. Eleven new pairs in the parity gate, plus dataset
and asset_progress, which had projections but no row. Still two spellings gated, not three.

visionset mcp

cli/ui.py's shape with a subprocess where that one has uvicorn: import-linter forbids
visionset.cli importing visionset.mcp, so the target is named as a module and the child inherits
stdin and stdout — those two streams are the transport. The full four-branch precedence is applied
and then stated in VISIONSET_WORKSPACE, so the child cannot disagree. The pre-flight open is
real, not a check: it runs the migration, so NotAWorkspace is one sentence at exit 1 rather than a
JSON envelope inside the agent's first tool call. It is the one command that prints nothing on
stdout.

Acceptance criteria

• Ship-vs-fold decision recorded per parity tool — above
• Each shipped tool tested through the MCP handler — every test drives the real protocol via
  create_connected_server_and_client_session, never FastMCP.call_tool (which skips
  validation, returns an undocumented 2-tuple and raises instead of returning isError)
• Destructive-op refusal path tested — delete_project without confirm refuses and the
  project is still there; plus allow_destructive, the un-retryable orphan case, and
  allow_lossy against an injected lossy exporter
• visionset mcp lists all tools over stdio — test_registration.py
• An MCP client retrieves an asset's image content and its dimensions match the Asset row —
  test_asset_tools.py, plus the scale factor and a Pillow decode of the returned bytes

Ledger

Migration	none — FORMAT_VERSION stays 11
VERSION	0.0.1.dev0
openapi.json	byte-identical — MCP touches no route
Generated TS client	byte-identical
New kernel service / model / error / event	none
ERROR_RULES	untouched
New runtime dependency	none (mcp>=1.2 was already declared; 1.28.1 installed)
New dev dependency	anyio>=4.5 — transitive today, and relying on that is the mistake python-multipart's comment already calls out
Tests	1896, up from 1670

Also worth knowing

IngestFailure.name for a directory ingest is the full path, not the basename — it is whatever
the run's own loop was holding. Unlike Source.path and Asset.uri, which are deliberately
unpublished, it already travels on the wire through IngestFailureOut and is the same string the
REST API and the CLI report. Pre-existing and out of scope here; noted because it is an
inconsistency somebody will eventually want to settle.

Left for #36: the published transcript. tests/mcp/test_agent_walk.py is the walk it will narrate.