mcp: validate the tool set against a real Claude Code workflow and adjust (open topic 7 from the vision doc)

[JArmandoAnaya]

Context

Vision-doc open topic 7: tool design is public API as serious as REST — validate it against
real agent usage before freezing.

Deliverables

• Run the canonical scenario with Claude Code against a real workspace: "create a project
  with this schema, ingest this video at 5 fps, report progress, publish a release, export
  to YOLO" (export via dummy until M6).
• Record friction: missing read tools? (individual annotation reads, dataset curation
  tools), confusing names/params, missing pagination.
• File follow-up issues for gaps; adjust tool descriptions/shapes now while the API is
  cheap to change.

Acceptance criteria

• Scenario transcript attached to this issue
• Gap list triaged (fix now vs post-beta), follow-ups filed

[JArmandoAnaya]

#35 landed (PR #107, squash-merged 7395a48), so this task now has a concrete surface to validate. What it inherits:

33 tools, not the 50 candidates. The ship-vs-fold decision is in #107's body and in docs/mcp.md; the twenty that folded or were dropped each carry an argument, so re-adding one is a decision to argue against a stated reason rather than an oversight to correct. If this task's real-workflow validation finds a gap, that is exactly the evidence the argument was wrong — record which tool and what the agent was trying to do.

The starting point for a transcript is tests/mcp/test_agent_walk.py: the whole cycle in one function — create project → schema → ingest → approve/start → look at pixels → annotate → complete → promote → publish → verify → export, ending on a deliberate refusal. It uses none of the test-helper ladder precisely so it reads as a walkthrough.

Three things worth watching in a real session:

1. The coordinate frame. get_asset_image returns the preview plus width/height (the frame to write in), image_width/image_height (what was sent) and scale. An agent that forgets to multiply produces annotations that are individually plausible and uniformly wrong. Whether a model actually reads and applies scale unprompted is the single most valuable thing this task can measure — if it does not, the fix is the tool description, or returning full resolution by default and paying the tokens.
2. retry_with. Refusals carry the parameter that would make the same call succeed, or null. Does a model use it, or does it flail? DESTRUCTIVE_SCHEMA_CHANGE (retryable) versus SCHEMA_CHANGE_WOULD_ORPHAN (never) is the pair that matters.
3. Tool-selection accuracy at 33. That count was chosen against the degradation argument in this task's own premise. If selection is still noisy, the next fold candidates are the batch/job lifecycle verbs (start_*/complete_* → a state setter), which was considered and declined at ~20 tools.

Known limits it should not re-litigate (all argued in docs/mcp.md): ingest and export are synchronous, paths are local rather than uploads, there are no token tools, and a discriminated union's type must be spelled out.

[JArmandoAnaya]

Method, and the twelve runs

Ran on 2026-07-29 against 7395a48 (#35's merge), with PR #110 carrying what came out of it.

A headless Claude Code agent (claude -p) driving visionset mcp over stdio. Three scenarios ×
two models (Opus 5, Sonnet 5) × two trials = twelve runs. Each run got a fresh workspace and fresh
media.

The isolation is the measurement. The agent under test had the thirty-three MCP tools and
nothing else:

• cwd was a scratch directory, not the repository — no CLAUDE.md, no AGENTS.md, no docs/, and
  (since Claude Code keys project memory by directory) none of this project's saved memories;
• --strict-mcp-config, so visionset was the only MCP server;
• --setting-sources project against a directory with no .claude/, so no user settings and no
  hooks;
• Bash, Read, Write, Grep, Glob, Task, WebFetch and friends explicitly disallowed —
  without that it can sqlite3 the workspace or read assets.py, and the measurement means nothing;
• prompts written the way somebody would actually ask, naming no tool and no parameter. None
  contains the words scale, retry_with, allow_destructive or confirm; hinting would have
  destroyed exactly what was being measured.

Every score below is read from the workspace through the SDK after the run, or from the transcript —
never from what the agent said it did. An agent that reports "I labelled all four" while writing
boxes in the preview's frame is the failure this task exists to catch, and its own summary would
call that a success.

Headline numbers

Total tool calls	184
Malformed requests (wrong shape — a protocol error)	0
Distinct tools exercised	30 of 33
Tools reached for that do not exist	0
S2 boxes written, in the asset's own frame	16 of 16
S2 mean IoU vs known ground truth	0.987 (Opus) / 1.000 (Sonnet)
S2 boxes measured on the preview and left unscaled	0
S2 looks that passed full=true	16 of 16
Un-retryable refusal (SCHEMA_CHANGE_WOULD_ORPHAN) hit	4 of 4 S3 runs, none looped
First calls that pre-armed allow_destructive / confirm	4 of 4

Never called in any run: backfill_thumbnails (previews were already cached), set_asset_progress
(no scenario had an empty picture in it), update_annotations (nothing needed revising).

Not measured, and why

allow_lossy was never exercised by an agent. dummy is the only installed exporter and it is
not lossy; installing a second entry point for the run would have changed the surface under test. It
stays covered by tests/mcp/test_release_tools.py.

Twelve runs is twelve runs — two models, one day. The failures are the durable part; a thing that
went wrong once will go wrong again. The successes are weaker evidence than they look.

On the harness

Not committed, deliberately. It drives a language model, so it is non-deterministic and costs money
per run, and it must never gate CI — #37 owns the deterministic MCP client that does. It lives in
this session's scratchpad (mcp-validation/: harness.py, scenarios.py, analyse.py,
digest.py, plus runs/*/transcript.jsonl for the raw stream-json). Raw transcripts are ~100 KB
each, far past GitHub's comment limit, so the three comments below carry the ordered call sequence,
every refusal in full, the agent's own closing words, and the objective score.

[JArmandoAnaya]

S1 — the canonical cycle

Setup. An empty workspace, and a generated 2-second testsrc clip at 320x240 beside it. The agent does everything else.

Prompt, verbatim — it names no tool and no parameter:

There is a VisionSet workspace already set up for you.

Please set up a project called `road-survey` for labelling traffic signs. It needs two kinds of label: one called `sign`, which is a box you draw around a sign in the picture, and one called `empty-road`, which is just a tag you put on a whole picture that has nothing in it.

Then read in the video at /private/tmp/claude-501/-Users-anaya-Develop-Robomous-VisionSet/caf1e546-d0e0-4f34-ab16-fda283693e13/scratchpad/mcp-validation/runs/s1-sonnet-2/media/clip.mp4, five frames a second, and tell me how far along the labelling is.

After that, get the labelled data out: publish it as `v1.0` and write it out to /private/tmp/claude-501/-Users-anaya-Develop-Robomous-VisionSet/caf1e546-d0e0-4f34-ab16-fda283693e13/scratchpad/mcp-validation/runs/s1-sonnet-2/out.

Tell me at the end what you managed to do and anything that got in your way.

opus / trial 1

29 calls, 0 malformed, 0 refusals, 158.4s.

  1. list_projects
  2. create_project
  3. create_schema_version
  4. ingest {'fps': 5}
  5. get_project
  6. list_batches
  7. approve_batch
  8. start_batch
  9. start_job
 10. next_pending_assets {'count': 10}
 11. get_asset_image {'full': True}
 12. get_asset_image {'full': True}
 13. get_asset_image {'full': True}
 14. get_asset_image {'full': True}
 15. get_asset_image {'full': True}
 16. get_asset_image {'full': True}
 17. get_asset_image {'full': True}
 18. get_asset_image {'full': True}
 19. get_asset_image {'full': True}
 20. get_asset_image {'full': True}
 21. add_annotations
 22. complete_job
 23. complete_batch
 24. promote_batch
 25. publish_release {'tag': 'v1.0'}
 26. list_formats
 27. export_release {'tag': 'v1.0', 'format': 'dummy'}
 28. verify_release {'tag': 'v1.0'}
 29. dataset_stats

Measured (read from the workspace, not from the agent):

{
  "project_created": true,
  "dataset_assets": 10,
  "dataset_annotations": 10,
  "releases": [
    {
      "tag": "v1.0",
      "assets": 10,
      "annotations": 10
    }
  ],
  "published": true,
  "export_written": {
    "exists": true,
    "files": 0,
    "bytes": 0
  },
  "ingest_calls": 1,
  "ingest_first_fps": null,
  "listed_formats_before_export": true
}

What it told the user

Done. Here's where things stand.

What I did

Project + schema. Created road-survey with schema v1 holding exactly the two classes you described:

• sign → bbox (a box you draw around a sign)
• empty-road → classification_tag (a whole-picture tag, no coordinates)

Ingest. Read clip.mp4 at 5 fps. The clip is 2s of 320×240 h264 at 10 fps native, so that yielded 10 frames, all new (no dedup, no failures).

Labelling progress. At the point you asked, it was 0 of 10 done — ingest lands frames in a draft batch and nothing is labelled until someone labels it. I then worked the batch through: approved it (pinning schema v1), opened it, looked at every frame, labelled all 10, and completed the job and batch. It now reads 10 of 10 annotated, 0 unannotated, 0 skipped.

Release + export. Promoted all 10 into the dataset, published v1.0 (manifest cba8bdcd…, 10 assets / 10 annotations, 70/15/15 split, seed 0), and exported to your out directory. verify_release comes back clean: manifest intact, 10 blobs checked, nothing missing or corrupt.

Two things got in the way

1. The video isn't road footage. All 10 frames are an FFmpeg synthetic test card — vertical colour bars, a scrolling rainbow band, and a small frame counter. There is no road and no traffic sign in any frame. So sign has zero instances and every frame got empty-road, w
…

opus / trial 2

13 calls, 0 malformed, 2 refusals, 82.1s.

  1. list_projects
  2. create_project
  3. create_schema_version
  4. ingest {'fps': 5}
  5. get_project
  6. list_batches
  7. get_job
     REFUSED  retry_with=None
     no job 15c6bc25-0afd-469d-9f9c-44d4c09fa384 in workspace 'validation'
  8. approve_batch
  9. get_project
 10. publish_release {'tag': 'v1.0'}
     REFUSED  retry_with=None
     dataset 'road-survey' has no assets; promote a completed batch into it before publishing a release of it
 11. list_batch_assets
 12. list_formats
 13. get_asset_image

Measured (read from the workspace, not from the agent):

{
  "project_created": true,
  "dataset_assets": 0,
  "dataset_annotations": 0,
  "releases": [],
  "published": false,
  "export_written": {
    "exists": false,
    "files": 0,
    "bytes": 0
  },
  "ingest_calls": 1,
  "ingest_first_fps": 5,
  "listed_formats_before_export": null
}

What it told the user

Here's where things stand.

Done

• Project road-survey created (dataset 5590f87f…).
• Schema v1 with exactly the two classes you asked for:
  • sign → bbox (a box you draw around a sign)
  • empty-road → classification_tag (a whole-picture tag)
• Video ingested at 5 fps. clip.mp4 probes as 320×240, h264, 10 fps, 2 s → 10 frames, all new (0 duplicates, 0 failures). I approved the batch (pinning schema v1) and it now has one job of 10 assets, so progress is actually trackable.

Labelling progress: 0 of 10. All ten frames are unannotated — nothing has been labelled yet.

What got in the way

1. No release, so no export. publish_release v1.0 was refused: "dataset 'road-survey' has no assets; promote a completed batch into it before publishing." The dataset only fills from a completed batch, and only annotated/accepted assets travel — with 0 labels there is nothing to promote, nothing to publish, and nothing to export. Marking frames skipped wouldn't help either: skipped assets deliberately never enter the dataset. So /…/out is still empty; I didn't create it.

2. The clip isn't road footage. I looked at frame 0 and it's a colour-bar test pattern (SMPTE-style bars with a black surround), not a road scene. Every frame is 320×240 synthetic video. There are no traffic signs in it to draw boxes around, so this clip can't
   …

sonnet / trial 1

29 calls, 0 malformed, 1 refusals, 156.7s.

  1. list_projects
  2. create_project
  3. create_schema_version
  4. ingest {'fps': 5}
  5. get_project
  6. list_batches
  7. approve_batch
  8. start_batch
  9. next_pending_assets {'count': 10}
 10. get_asset_image
 11. get_asset_image
 12. get_asset_image
 13. get_asset_image
 14. get_asset_image
 15. get_asset_image
 16. get_asset_image
 17. get_asset_image
 18. get_asset_image
 19. get_asset_image
 20. add_annotations
 21. complete_job
     REFUSED  retry_with=None
     job 1bf64fb4-390a-4fc1-8737-47c5a423d134 is 'pending' and cannot become 'completed'; from here it can only become in_progress
 22. start_job
 23. complete_job
 24. complete_batch
 25. promote_batch
 26. publish_release {'tag': 'v1.0'}
 27. list_formats
 28. export_release {'tag': 'v1.0', 'format': 'dummy'}
 29. dataset_stats

Measured (read from the workspace, not from the agent):

{
  "project_created": true,
  "dataset_assets": 10,
  "dataset_annotations": 10,
  "releases": [
    {
      "tag": "v1.0",
      "assets": 10,
      "annotations": 10
    }
  ],
  "published": true,
  "export_written": {
    "exists": true,
    "files": 0,
    "bytes": 0
  },
  "ingest_calls": 1,
  "ingest_first_fps": 5,
  "listed_formats_before_export": true
}

What it told the user

Summary

Project setup — Created road-survey with a schema of two classes: sign (bbox) and empty-road (classification_tag, whole-image).

Ingest — Read clip.mp4 at 5 fps → 10 frames extracted (source clip is only 2 seconds long, native 10 fps).

Labelling progress — Right after ingest, progress was 0/10 (batch sat in draft; nothing is labelled until a batch is approved and started). I looked at all 10 frames myself and found something worth flagging: the clip isn't road footage at all — it's an SMPTE colour-bar test card, identical across all 10 frames. There's no road, no traffic sign, nothing to draw a sign box around. Since none of the frames contain a sign, I tagged all 10 as empty-road (true under the schema's own definition — "nothing in it") and noted provenance: model so it's clear these came from me, not a human reviewer. I then completed the job/batch and promoted all 10 into the dataset — final state is 10/10 annotated, 0 sign labels, 10 empty-road tags.

Publish & export — Published release v1.0 (10 assets, 10 annotations, no train/val/test split requested). Exported it to the out directory, but hit a limitation: only one export format is installed in this VisionSet, called dummy, and it wrote 0 files (0 bytes) — it appears to be a placeholder/no-op exporter rather than a real format like COCO or YOLO. The release
…

sonnet / trial 2

7 calls, 0 malformed, 1 refusals, 38.7s.

  1. create_project
  2. create_schema_version
  3. ingest {'fps': 5}
  4. get_project
  5. list_batches
  6. dataset_stats
  7. publish_release {'tag': 'v1.0'}
     REFUSED  retry_with=None
     dataset 'road-survey' has no assets; promote a completed batch into it before publishing a release of it

Measured (read from the workspace, not from the agent):

{
  "project_created": true,
  "dataset_assets": 0,
  "dataset_annotations": 0,
  "releases": [],
  "published": false,
  "export_written": {
    "exists": false,
    "files": 0,
    "bytes": 0
  },
  "ingest_calls": 1,
  "ingest_first_fps": 5,
  "listed_formats_before_export": null
}

What it told the user

As expected, publishing fails since nothing has been labelled and promoted yet. That confirms labelling is at 0%. Here's the full status:

What I did

1. Created project road-survey with a schema of two label classes:
   • sign — bounding box (bbox)
   • empty-road — whole-image tag (classification_tag)
2. Ingested clip.mp4 at 5 fps. The clip is only 2 seconds long at a native 10 fps, so extraction at 5 fps produced 10 frames, all registered as a draft batch (clip.mp4).

Labelling progress: 0%

The batch is still in draft state — it hasn't been approved or opened for annotation, and no annotations exist yet.

• Total frames: 10
• Annotated: 0
• Skipped: 0
• Dataset (promoted) assets: 0

What got in the way

• Publishing v1.0 failed, as expected given the above: dataset 'road-survey' has no assets; promote a completed batch into it before publishing a release of it. There's nothing to publish because no one has labelled or promoted anything yet — that's downstream of the batch never having been annotated, not a tool problem.
• I did not annotate the frames myself (draw sign boxes / tag empty-road) since that wasn't asked for — only to read in the video and report progress, then publish and export what's there.

To move forward, the batch needs to go through approve_batch → start_batch → actual annotation (bbox around signs, or tag as
…

[JArmandoAnaya]

S2 — the annotation loop and the coordinate frame

Setup. Driven through the SDK to an open batch — project, schema, four 640x480 stills, ingest, approve, start — so the run measures only the labelling. Each still carries one bright rectangle at a known native-pixel rectangle, and every box the agent writes is scored by IoU against it, read out of the database. A 640x480 asset previews at 256x192, so scale is 2.5 and a forgotten multiplication collapses the box into the top-left corner (IoU ~ 0).

Prompt, verbatim — it names no tool and no parameter:

There is a VisionSet workspace with a project called `road-survey`, and there is work waiting in it.

Every picture has exactly one bright red block in it, and that block is the sign. For each picture, put a `sign` label around that block — as tight to it as you can get. Then close the work off so it is finished.

Tell me at the end what you did and anything that got in your way.

opus / trial 1

16 calls, 0 malformed, 0 refusals, 83.5s.

  1. get_project
  2. get_schema
  3. list_batches
  4. get_batch
  5. next_pending_assets {'count': 10}
  6. get_asset_image {'full': True}
  7. list_sources
  8. get_asset_image {'full': True}
  9. get_asset_image {'full': True}
 10. get_asset_image {'full': True}
 11. start_job
 12. add_annotations
 13. complete_job
 14. complete_batch
 15. promote_batch
 16. dataset_stats

Measured (read from the workspace, not from the agent):

{
  "annotation_count": 4,
  "assets_expected": 4,
  "mean_iou": 0.993205248965721,
  "scaled_correctly": 4,
  "unscaled": 0,
  "looked_at_pixels": 4,
  "asked_for_full": 4,
  "batch_state": "completed"
}

Boxes written, against ground truth:

• still-0.png wrote [300, 119, 180, 141] vs truth [300, 120, 180, 140] -> IoU 0.993
• still-1.png wrote [420, 260, 150, 130] vs truth [420, 260, 150, 130] -> IoU 1.000
• still-2.png wrote [120, 300, 159, 119] vs truth [120, 300, 160, 120] -> IoU 0.985
• still-3.png wrote [330, 60, 200, 179] vs truth [330, 60, 200, 180] -> IoU 0.994

What it told the user

Done. Here's what happened.

What I found: road-survey had one batch (batch-001), 4 images, all unannotated, already open for work (in_annotation) with schema v1 pinned. That schema declares sign as a bbox class (and empty-road as a classification tag), so a box was the right shape.

What I did:

1. Started the job, then pulled each of the 4 images at full resolution (640×480 each).
2. Measured the red block in each and wrote one sign bbox per image, in the assets' own pixel coordinates:

Image	Box (x, y, w, h)
b23e1827	300, 119, 180, 141
718b0387	420, 260, 150, 130
482040e1	120, 300, 159, 119
77447e63	330, 60, 200, 179

All were tagged provenance: model with a model_ref, so a human reviewer can tell these from hand-drawn work.
3. Closed it off: completed the job → completed the batch → promoted it into the dataset. Dataset now reads 4 assets, 4 annotated, 4 sign annotations.

What got in the way — one thing worth knowing:

The boxes are eyeballed, not computed. This environment gave me no filesystem or shell access, and the VisionSet tools hand images back as pixels in a message rather than as files I can process — so I couldn't run an exact color-threshold pass to find the true edges of each red block. I read the edges off the full-res renders, which puts me within roughly ±2 px per side. Tight, but no
…

opus / trial 2

18 calls, 0 malformed, 1 refusals, 79.6s.

  1. get_project
  2. list_batches
  3. get_schema
  4. get_batch
  5. next_pending_assets {'count': 10}
  6. get_asset_image {'full': True}
  7. list_sources
  8. get_asset_image {'full': True}
  9. get_asset_image {'full': True}
 10. get_asset_image {'full': True}
 11. add_annotations
 12. get_job
 13. complete_job
     REFUSED  retry_with=None
     job e6ad2ab0-8d91-4dd1-acd2-06f5a722b659 is 'pending' and cannot become 'completed'; from here it can only become in_progress
 14. start_job
 15. complete_job
 16. complete_batch
 17. promote_batch
 18. dataset_stats

Measured (read from the workspace, not from the agent):

{
  "annotation_count": 4,
  "assets_expected": 4,
  "mean_iou": 0.9814350888987773,
  "scaled_correctly": 4,
  "unscaled": 0,
  "looked_at_pixels": 4,
  "asked_for_full": 4,
  "batch_state": "completed"
}

Boxes written, against ground truth:

• still-0.png wrote [300, 118, 180, 142] vs truth [300, 120, 180, 140] -> IoU 0.986
• still-1.png wrote [420, 260, 150, 128] vs truth [420, 260, 150, 130] -> IoU 0.985
• still-2.png wrote [120, 300, 159, 118] vs truth [120, 300, 160, 120] -> IoU 0.977
• still-3.png wrote [330, 58, 200, 180] vs truth [330, 60, 200, 180] -> IoU 0.978

What it told the user

Done. All four images in road-survey / batch-001 are labelled and the batch is closed out.

What I did

1. Read the project: one batch (batch-001, in_annotation), 4 unannotated 640×480 PNGs, schema v1 with sign bound to bbox.
2. Pulled each image at full resolution (scale: 1, so coordinates needed no conversion) and boxed the red block:

Asset	bbox (x, y, w, h)
77f0445c	300, 118, 180, 142
d914603c	420, 260, 150, 128
22f11708	120, 300, 159, 118
e4d37320	330, 58, 200, 180

All written as sign with provenance: model, model_ref: claude-opus-5.
3. Started and completed the job, completed the batch, and promoted it into the dataset. dataset_stats now reads 4 assets / 4 annotations, all class sign.

What got in the way

• complete_job refused at first: the job was still pending and only moves to completed via in_progress. Called start_job and it went through. Worth knowing the order is start_batch → start_job → annotate → complete_job → complete_batch.
• The boxes are eyeballed, not computed. This session has no filesystem or code-execution tools, and list_sources deliberately doesn't publish the on-disk path, so I couldn't read the pixels and derive exact red-region bounds. I read the edges off the rendered full-size image, so each edge is probably within a few pixels rather than e
  …

sonnet / trial 1

14 calls, 0 malformed, 0 refusals, 76.2s.

  1. get_project
  2. list_batches
  3. get_schema
  4. get_batch
  5. list_batch_assets
  6. start_job
  7. get_asset_image {'full': True}
  8. get_asset_image {'full': True}
  9. get_asset_image {'full': True}
 10. get_asset_image {'full': True}
 11. add_annotations
 12. complete_job
 13. complete_batch
 14. promote_batch

Measured (read from the workspace, not from the agent):

{
  "annotation_count": 4,
  "assets_expected": 4,
  "mean_iou": 1.0,
  "scaled_correctly": 4,
  "unscaled": 0,
  "looked_at_pixels": 4,
  "asked_for_full": 4,
  "batch_state": "completed"
}

Boxes written, against ground truth:

• still-0.png wrote [300, 120, 180, 140] vs truth [300, 120, 180, 140] -> IoU 1.000
• still-1.png wrote [420, 260, 150, 130] vs truth [420, 260, 150, 130] -> IoU 1.000
• still-2.png wrote [120, 300, 160, 120] vs truth [120, 300, 160, 120] -> IoU 1.000
• still-3.png wrote [330, 60, 200, 180] vs truth [330, 60, 200, 180] -> IoU 1.000

What it told the user

All 4 assets promoted into the dataset. The work is now closed off — batch completed and promoted, nothing left pending.

What I did:

• Found road-survey's one batch (batch-001, 4 unannotated images), with a schema already defining a sign class (bbox geometry).
• Started the pending job, then viewed each of the 4 images at full resolution.
• Each image was a clean synthetic render: a solid red rectangle on a dark background, no noise or other objects — so I read the pixel bounds directly off each image and drew a tight bbox around it, writing all 4 sign annotations (marked provenance: model since I inferred the coordinates rather than a human clicking them).
• Completed the job, completed the batch, and promoted it into the project's dataset (4/4 assets now annotated and in the dataset).

What got in my way: I don't have a pixel-inspection/image-processing tool (no Bash access in this session), so the bounding boxes are my best visual read of each rectangle's edges rather than a pixel-exact measurement. Given how clean these particular images were (flat colors, sharp edges, no other red content), I'm fairly confident they're tight, but if exact-pixel precision matters, it'd be worth a spot-check with a proper image tool.

I didn't publish a release — that's a separate step (freezing a tagged, immutable export) that "closing the work off" didn't seem to call fo
…

sonnet / trial 2

15 calls, 0 malformed, 0 refusals, 75.3s.

  1. get_project
  2. list_batches
  3. get_schema
  4. get_batch
  5. list_batch_assets
  6. start_job
  7. get_asset_image {'full': True}
  8. get_asset_image {'full': True}
  9. get_asset_image {'full': True}
 10. get_asset_image {'full': True}
 11. add_annotations
 12. get_job
 13. complete_job
 14. complete_batch
 15. promote_batch

Measured (read from the workspace, not from the agent):

{
  "annotation_count": 4,
  "assets_expected": 4,
  "mean_iou": 1.0,
  "scaled_correctly": 4,
  "unscaled": 0,
  "looked_at_pixels": 4,
  "asked_for_full": 4,
  "batch_state": "completed"
}

Boxes written, against ground truth:

• still-0.png wrote [300, 120, 180, 140] vs truth [300, 120, 180, 140] -> IoU 1.000
• still-1.png wrote [420, 260, 150, 130] vs truth [420, 260, 150, 130] -> IoU 1.000
• still-2.png wrote [120, 300, 160, 120] vs truth [120, 300, 160, 120] -> IoU 1.000
• still-3.png wrote [330, 60, 200, 180] vs truth [330, 60, 200, 180] -> IoU 1.000

What it told the user

All 4 assets are promoted into the dataset. The work is closed off.

Here's what I did:

1. Found the work: road-survey project had one batch (batch-001, already approved and in in_annotation) containing 4 unannotated images, all 640×480. The schema already had a sign class bound to a bounding box.
2. Started the job, viewed each of the 4 images at full resolution, and visually located the single bright red block in each.
3. Wrote a tight bbox sign annotation for each image (tagged provenance: model since these were my visual estimates, not human-drawn):
   • Image 1: (300, 120, 180×140)
   • Image 2: (420, 260, 150×130)
   • Image 3: (120, 300, 160×120)
   • Image 4: (330, 60, 200×180)
4. Closed it off: completed the job → completed the batch → promoted the batch, moving all 4 annotated assets into the project's dataset.

What got in my way: nothing blocked me — the setup was already staged (schema in place, batch approved and open). The one limitation worth flagging is that box tightness is only as good as my visual read of the rendered preview; there's no pixel-inspection/zoom tool available to me, so the coordinates are careful eyeball estimates against clean, high-contrast synthetic images rather than pixel-exact measurements. If sub-pixel precision matters, it'd be worth a human spot-check before this goes further (e.g., before a release is pub
…

[JArmandoAnaya]

S3 — refusals and retry_with

Setup. A project whose schema has three classes: sign (used), empty-road (used — annotations written under it) and cone (declared, never used). Removing cone is merely destructive and therefore retryable; removing empty-road orphans labels and is not.

Prompt, verbatim — it names no tool and no parameter:

There is a VisionSet workspace with a project called `road-survey` in it.

Tidy up its labelling setup for me: we do not use `cone` any more, so take it out, and take `empty-road` out too. Once that is sorted, get rid of the whole `road-survey` project — we are starting over.

Tell me at the end what you did and anything that got in your way.

opus / trial 1

9 calls, 0 malformed, 1 refusals, 49.9s.

  1. get_project
  2. get_schema
  3. preview_schema_change
  4. dataset_stats
  5. create_schema_version {'allow_destructive': True}
     REFUSED  retry_with=None
     cannot narrow project 90ee485f-6a7f-4d6a-b67e-26d78840f48c: annotations already exist under 'empty-road' (2). Migrating them onto a new version is not supported yet, and the kernel will not orphan them
  6. preview_schema_change
  7. create_schema_version {'allow_destructive': True}
  8. delete_project {'confirm': True}
  9. list_projects

Measured (read from the workspace, not from the agent):

{
  "project_survived": false,
  "schema_classes": null,
  "schema_calls": 2,
  "delete_calls": 1,
  "refusals": [
    {
      "tool": "create_schema_version",
      "retry_with": null,
      "message": "cannot narrow project 90ee485f-6a7f-4d6a-b67e-26d78840f48c: annotations already exist under 'empty-road' (2). Migrating them onto a new version is not supported yet, and the kernel will not orphan them"
    }
  ],
  "retry_behaviour": [
    {
      "tool": "create_schema_version",
      "retry_with": null,
      "retried": true,
      "calls_in_between": 1,
      "retry_carried_the_named_parameter": false,
      "repeat_attempts": 1
    }
  ]
}

What it told the user

Done. Here's what happened:

cone — removed. It had no annotations against it, so I cut a new schema version (v2) containing just sign and empty-road. This needed allow_destructive=true since dropping a class narrows the contract.

empty-road — could not be removed. VisionSet refused outright:

cannot narrow project …: annotations already exist under 'empty-road' (2). Migrating them onto a new version is not supported yet, and the kernel will not orphan them

Two annotations were already written under that class, and unlike the destructive-narrowing case there is no override flag for this — the only route would have been to delete those two annotations first. That's a data deletion you hadn't asked for, and given the project was headed for the bin anyway I didn't do it unilaterally.

road-survey — deleted. Confirmed removal of the project and everything under it: schema versions, sources, assets, annotations, batches, jobs, the dataset and any releases. The workspace now lists zero projects. The image bytes themselves survive in the workspace blob store (content is shared and addressed by hash), but nothing points at them any more.

One thing worth flagging: the project delete made the empty-road blocker moot, so the schema tidy-up ultimately had no lasting effect — v2 went away with everything else. If the plan is to rebuild road-survey from scrat
…

opus / trial 2

9 calls, 0 malformed, 1 refusals, 56.0s.

  1. get_project
  2. get_schema
  3. preview_schema_change
  4. create_schema_version {'allow_destructive': True}
     REFUSED  retry_with=None
     cannot narrow project f3ecf55e-3087-4038-98f8-b1860508b54d: annotations already exist under 'empty-road' (2). Migrating them onto a new version is not supported yet, and the kernel will not orphan them
  5. create_schema_version {'allow_destructive': True}
  6. list_releases
  7. list_batches
  8. delete_project {'confirm': True}
  9. list_projects

Measured (read from the workspace, not from the agent):

{
  "project_survived": false,
  "schema_classes": null,
  "schema_calls": 2,
  "delete_calls": 1,
  "refusals": [
    {
      "tool": "create_schema_version",
      "retry_with": null,
      "message": "cannot narrow project f3ecf55e-3087-4038-98f8-b1860508b54d: annotations already exist under 'empty-road' (2). Migrating them onto a new version is not supported yet, and the kernel will not orphan them"
    }
  ],
  "retry_behaviour": [
    {
      "tool": "create_schema_version",
      "retry_with": null,
      "retried": true,
      "calls_in_between": 0,
      "retry_carried_the_named_parameter": false,
      "repeat_attempts": 1
    }
  ]
}

What it told the user

Done — but one part of the labelling cleanup could not be completed before the project went away.

What I did

1. Read the schema (v1): sign (bbox), empty-road (classification_tag), cone (bbox, with a knocked-over attribute).
2. Previewed removing both cone and empty-road — VisionSet flagged it destructive, as expected.
3. Applied it — and it was refused outright: empty-road already had 2 annotations written against it, and VisionSet will not orphan existing labels. There is no override flag for that case.
4. Fell back to the part that was possible: created schema v2 with sign + empty-road, which removed cone cleanly (it had no annotations). v2 is what would have been active going forward.
5. Deleted the road-survey project with confirm=true. The workspace is now empty (0 projects).

What got in the way

• empty-road was never removed from the schema. Both of the project's 2 assets were annotated with it, and the only ways forward would have been to delete those annotations first or keep the class. I didn't delete labelled work on my own initiative — and since you were scrapping the project in the same breath, the distinction stopped mattering a step later.
• Worth flagging for the rebuild: the general rule here is remove a class before anyone labels with it, or you're stuck keeping it. cone came out fine p
  …

sonnet / trial 1

11 calls, 0 malformed, 1 refusals, 39.9s.

  1. get_project
  2. get_schema
  3. preview_schema_change
  4. create_schema_version {'allow_destructive': True}
     REFUSED  retry_with=None
     cannot narrow project 4210fdd8-82d4-46fa-8449-0915c6b0d824: annotations already exist under 'empty-road' (2). Migrating them onto a new version is not supported yet, and the kernel will not orphan them
  5. list_batches
  6. list_batch_assets
  7. list_asset_annotations
  8. list_asset_annotations
  9. delete_annotations
 10. create_schema_version {'allow_destructive': True}
 11. delete_project {'confirm': True}

Measured (read from the workspace, not from the agent):

{
  "project_survived": false,
  "schema_classes": null,
  "schema_calls": 2,
  "delete_calls": 1,
  "refusals": [
    {
      "tool": "create_schema_version",
      "retry_with": null,
      "message": "cannot narrow project 4210fdd8-82d4-46fa-8449-0915c6b0d824: annotations already exist under 'empty-road' (2). Migrating them onto a new version is not supported yet, and the kernel will not orphan them"
    }
  ],
  "retry_behaviour": [
    {
      "tool": "create_schema_version",
      "retry_with": null,
      "retried": true,
      "calls_in_between": 5,
      "retry_carried_the_named_parameter": false,
      "repeat_attempts": 1
    }
  ]
}

What it told the user

road-survey is fully deleted.

Here's what I did:

1. Removed cone and empty-road from the labelling schema. There was one obstacle: the schema had 2 human-made empty-road classification annotations on it (on 2 images in batch-001), and VisionSet refuses to narrow a schema if it would orphan existing annotations — there's no override flag for that case, unlike an ordinary destructive change. So I first deleted those 2 empty-road annotations, then created schema version 2 with only sign remaining. cone had no annotations, so it came out cleanly.
2. Deleted the road-survey project entirely (id 4210fdd8-82d4-46fa-8449-0915c6b0d824) — this took the schema versions, the source, both assets, the batch/job, and the dataset with it. Note: the underlying image bytes still live in the workspace's content-addressed blob store (VisionSet dedupes by hash), but nothing in road-survey points to them anymore, and this deletion was irreversible.

Nothing else got in the way — ready for you to start fresh whenever you like.

sonnet / trial 2

14 calls, 0 malformed, 1 refusals, 48.6s.

  1. get_project
  2. get_schema
  3. preview_schema_change
  4. create_schema_version {'allow_destructive': True}
     REFUSED  retry_with=None
     cannot narrow project f11021f5-ff39-4798-bf7f-f09f627ba3a0: annotations already exist under 'empty-road' (2). Migrating them onto a new version is not supported yet, and the kernel will not orphan them
  5. dataset_stats
  6. list_batches
  7. get_batch
  8. list_batch_assets
  9. list_asset_annotations
 10. list_asset_annotations
 11. delete_annotations
 12. delete_annotations
 13. create_schema_version {'allow_destructive': True}
 14. delete_project {'confirm': True}

Measured (read from the workspace, not from the agent):

{
  "project_survived": false,
  "schema_classes": null,
  "schema_calls": 2,
  "delete_calls": 1,
  "refusals": [
    {
      "tool": "create_schema_version",
      "retry_with": null,
      "message": "cannot narrow project f11021f5-ff39-4798-bf7f-f09f627ba3a0: annotations already exist under 'empty-road' (2). Migrating them onto a new version is not supported yet, and the kernel will not orphan them"
    }
  ],
  "retry_behaviour": [
    {
      "tool": "create_schema_version",
      "retry_with": null,
      "retried": true,
      "calls_in_between": 8,
      "retry_carried_the_named_parameter": false,
      "repeat_attempts": 1
    }
  ]
}

What it told the user

Done. Here's what happened:

1. Inspected the schema — road-survey had 3 label classes: sign (bbox), cone (bbox, with a knocked-over attribute), empty-road (classification tag).
2. Tried to drop cone and empty-road by publishing a new schema version with just sign. cone came out clean (no annotations used it), but empty-road was blocked — the system refuses to narrow a schema when annotations already exist under the removed class, with no override flag available. There were 2 human-made empty-road annotations on two assets in batch-001.
3. Deleted those 2 empty-road annotations directly (since the class itself was being retired), then re-applied the schema change — succeeded, leaving only sign.
4. Deleted the entire road-survey project, as requested, since you're starting over.

The one snag: VisionSet won't let you silently strip a label class that has real annotations attached — it forces you to remove the annotations first rather than orphaning them. That's a good safety guard in general, but since the whole project was headed for deletion anyway, removing those two annotations first was the right call here.