Skip to content

fix(annotator): asset-bounded suggest clicks, confidence only in the preview, selected-only labels - #514

Merged
JArmandoAnaya merged 3 commits into
mainfrom
fix/editor-polish
Aug 10, 2026
Merged

fix(annotator): asset-bounded suggest clicks, confidence only in the preview, selected-only labels#514
JArmandoAnaya merged 3 commits into
mainfrom
fix/editor-polish

Conversation

@JArmandoAnaya

Copy link
Copy Markdown
Contributor

Closes #513.

Three editor-surface corrections. Frontend only — no kernel, no server, no wire.
openapi.json and frontend/ui-core/src/generated/ are untouched and check.sh generated
says so. Nothing in frontend/annotator/src/core/ gained a React import, a DOM global or a
DOM type in a signature; all three headless gates are green.

1. A suggest click outside the asset is a no-op

What it did before, measured rather than reasoned about

Driven through AnnotationPage with a stubbed route table, a press at (900, 700) on a
640 × 480 asset, suggest armed:

ASKS: 1
BODY: {"project_id":…,"asset_id":…,"connection_id":…,
       "positive":[{"x":900,"y":700}],"negative":[],"allowed_geometries":["bbox"]}
PREVIEW: true   PROMPT DOTS: 1

So yes: the request left carrying coordinates outside the asset in both axes, a prompt dot
was painted out in the margin, and whatever came back was drawn. Nothing client-side
refused it, and nothing server-side does either — SuggestPoint is two bare floats. A
follow-up on what the server should do with an out-of-frame prompt is drafted in the
session report and not filed; this PR does not touch the server.

What ships

withinBounds joins clampPoint in core/geometry/primitives.ts — the two questions a
frame is asked, and the difference between them is the whole design. A drag that left the
picture still means make the box this big, so it clamps. A click in the margin is not a
click on anything, and clamping it would put a prompt point on the asset's edge that nobody
placed there. Inclusive at both ends, matching clampPoint's own range, because the last
row of pixels is part of the asset.

The gate is in AnnotatorCanvas.handlePointerDown, before onSuggestPoint fires: a
point the host never hears about records no click, sends no request and moves no preview —
one guarantee instead of three. It is asked in asset pixels, so imagePoint has already
accounted for zoom and pan and there is no second transform to keep in step with the first.

Clicks inside the asset are unchanged, including on top of an existing shape: the suggest
branch still sits after the pan branch and before the machine dispatch, exactly where #451
put it.

2. Confidence renders only on the live preview

Surface Before Now
Live suggestion preview vehicle 91% unchanged (confidenceLabel, cf. #451)
Canvas label vehicle · 62% vehicle
Panel row ✨ 62%
Row tooltip model_ref unchanged
Row accessible name …, confidence 62% Model-produced by <model_ref>

The boundary, stated once: the preview shows the number because that is where the
accept-or-reject decision is made; a committed annotation does not, because by then the
decision has been made. Nothing is discarded — confidence and model_ref are stored and
travel on the wire exactly as before.

PaintedAnnotation gives back the two fields it gained in #512. They now have zero
consumers, and their docstring justified them by the label suffix that is gone. A projected
field with no reader is what a renderer starts writing next; making the absence structural
is what paint.test.tshands the renderer neither the score nor who produced it pins.

confidencePercent stays exported. One consumer today — the preview, through
confidenceLabel — and that is recorded in its docstring rather than inlined away: it is
the spelling, and whoever shows the number next imports it instead of writing a second
Math.round.

Every deleted assertion, named

The #473 precedent — a pinned test that encoded the rule this PR was asked to change is
named here so its removal is a decision rather than a disappearance.

Removed / rewritten Where Replaced by
carries both fields onto the draw list paint.test.ts hands the renderer neither the score nor who produced it
carries a person's own values too, so the label can tell them apart paint.test.ts paints a model's shape exactly as it paints a person's
writes the model's score beside the class, at the shipped spelling paint.test.ts canvasLabel.test.tsxwrites no score over a model's shape
still marks a model's work when the model reported no score paint.test.ts canvasLabel.test.tsxwrites no provenance mark either
leaves a person's label exactly as it shipped / leaves an imported label alone paint.test.ts folded into paints a model's shape exactly as it paints a person's
spells the score the way the suggestion overlay already does paint.test.ts spells it one way, so no second surface can disagree with the first
marks it, and says how sure the model was panel.test.tsx marks it — that a model drew it, and no more than that
the confidence 62% half of says in words what the glyph says in a picture panel.test.tsx same test, asserting the name without it
still marks the model's work when no score was recorded panel.test.tsx marks the model's work the same whether or not a score was recorded
the object-confidence-1 assertion in marks only the rows that are the model's panel.test.tsx object-model-1 present, row text free of 41

labelText and the object-confidence-<n> testid are gone; git grep -nwE labelText and
git grep -n object-confidence are both empty.

Null-confidence-is-absent-never-zero stays tested where confidence still renders:
paint.test.tsnames the class alone when the model reported no confidence, and
rounds the confidence to whole percent at both ends beside it.

3. The canvas label renders only while its shape is selected

Where the divergence came from: it was never introduced — the React adapter has rendered
ShapeLabel unconditionally since AnnotationShape was first written, in the commit that
brought the adapter itself. DESIGN.md has recorded the opposite the whole time, under
v1's shape metrics, kept as the reference: "the class label renders only while selected,
… anchored at the first vertex, never a pointer target." So this is the adapter starting to
obey a rule that was already written down, not a regression being backed out. Nothing in the
tracker records a reversal, and no toggle exists, so the flag-and-stop branch of the brief
does not apply.

Metrics are untouched — same size, weight, anchor, lift, and still never a pointer
target. Only the visibility condition moved. (Note: the shipped size is LABEL_PX = 12 and
no explicit weight, where DESIGN.md's reference line says 11 / 700. That gap predates this
PR and changing it is out of scope; recorded in the session report.)

The read-only mode is unaffected in the way that matters: Shapes.tsx already documented
selection there as "the stroke and the label", and a viewer paints no grips, so the label is
now doing more of that work rather than less.

Mutation verification

Eight mutations. Applied to the committed tree; anchor asserted present exactly once
before and the replacement asserted present after; each reverted by git apply -R on its
own recorded diff with dirty=0 confirmed after every one; green restored on a clean tree
at the end.

# Mutation Tests that turned red
M1 the bounds check is dropped — every press is a prompt again 5 in suggestFlow.test.tsx, led by sends nothing and shows nothing
M2 the out-of-frame press is clamped onto the edge instead of dropped the same 5 — clamping is not a near-miss that slips through
M3 the bounds test excludes the edge (> / <) 2 in primitives.test.ts + 1 in suggestFlow.test.tsxcounts the asset's own edge as inside
M4 only the x axis is checked 2 in primitives.test.ts + 1records no point, so the next real click is still the first one
M5 the label renders on every shape again 2 in canvasLabel.test.tsx, led by writes nothing over an unselected shape
M6 the canvas label writes a score again 5 in canvasLabel.test.tsx
M7 the panel row shows the score again 2 in panel.test.tsx, led by shows no confidence anywhere on the row
M8 the draw list carries provenance/confidence again 2 in paint.test.ts, led by hands the renderer neither the score nor who produced it

M1 was also run against the browser suite on its own: without the check, the new scenario
fails at a suggest click in the margin around the picture asks nothing, at any zoom
the margin press leaves suggest-idle. So the browser assertion is not vacuous either.

A note on the harness, because it nearly cost a run. The first battery chained
mutate && run && revert with the test output piped through head. head closes the pipe,
the runner takes SIGPIPE, pipefail makes the chain non-zero, and four reverts silently
never executed — so mutations stacked and the next run's red read like a broken
implementation. It was recoverable only because the work was committed before the first
mutation, which is exactly what that rule is for. The rewritten harness is unconditional,
asserts a clean tree before every mutation, and refuses an empty patch.

Test plan

New: primitives.test.ts (+5, the predicate and its edges), suggestFlow.test.tsx (+6, the
gate end to end through AnnotationPage against a stubbed route table),
canvasLabel.test.tsx (new file, 7 — the label's visibility and its content), panel.test.tsx
(+1 and 4 rewritten), paint.test.ts (4, replacing 7), annotate.spec.ts (+1 browser).

Where the browser was used, and why only there. One new browser assertion: the surround
does not exist in jsdom — every rectangle is zero, so a component test about "the margin
around the picture" would be asserting about nothing. It measures both rectangles, proves
the margin is real, presses in it, and then presses on the picture so the absence is a rule
and not a broken fixture — all at a non-default zoom, since a rule written against
screen coordinates would pass at the fitted scale and refuse half the picture at any other.
Everything else here is jsdom-complete: whether a <text> node exists and whether a request
left are not things a browser knows better.

serveApi in annotate.spec.ts gains a suggestible flag (off by default, so every other
scenario keeps the empty connection list that its panel copy is about) and a
POST /inference/suggest stub. The stub is written against the real SuggestionOut
{ model_ref, region: { geometry, confidence } }. A flatter shape is refused by the
generated runtime check and surfaces as "the server answered with something this app does
not recognise"
, which looks nothing like a stub bug; that is the test-double-fidelity rule
earning itself again.

Gate

bash scripts/check.sh run in stages under the harness's ~10-minute command ceiling,
with pytest split by the directories ls tests/ reported at run time. Machine was free
(load average 3.84 on 20 cores), so this is the full gate split and not the saturation
fallback. Every exit code verbatim.

Stage Exit
pytest tests/architecture 0 22 passed
pytest tests/cli 0 336 passed
pytest tests/examples 0 41 passed
pytest tests/formats 0 151 passed, 2 skipped
pytest tests/inference 0 182 passed, 6 skipped
pytest tests/jobs 0 30 passed
pytest tests/kernel 0 1445 passed, 4 skipped
pytest tests/mcp 0 246 passed
pytest tests/packaging 0 3 passed, 1 skipped
pytest tests/server 0 690 passed
pytest tests/test_versioning.py 0 3 passed
ruff check . 0
ruff format --check . 0
mypy src/visionset 0 153 source files
lint-imports 0 4 contracts kept, 0 broken
check.sh frontend (build, tests, lint) 0 annotator 923, ui-core 926, node gates 77/0
check.sh generated (openapi, client, mcp reference, version sync) 0 no drift
playwright test (annotator + app e2e, chromium) 0 249 passed, 1 flaky
playwright test -c playwright.cycle.config.ts 0 1 passed

tests/scripts holds the node --test gates, which the frontend group runs, so it is
absent above rather than reported as exit 5. No baseline-proof exception was used or
needed.

The flaky one is e2e/gallery.spec.tstiles never overlap, at any density, which passed
on retry — cf. #511. This diff touches no gallery code.

One suite failed first and it was mine, which is the point of it: e2e/perf.spec.ts
the benchmark scene is 220 annotations… holds an itemised SVG element budget, and a
selected-only label moves it — 660 → 440 on the 220-annotation bench scene, two elements per
annotation rather than three. Its own comment says a deliberate rendering change should move
these numbers. Updated, with the reason recorded beside it. The <text> was the most
expensive of the three (a stroke, a paint order and a translate each), so the legibility
decision pays here as well.

Information architecture

No route, tab, screen, nav entry or entry point moves. The sitemap is unchanged.

Found, not fixed

  • SuggestPoint accepts any float. The server takes prompt coordinates outside the
    asset's frame and hands them to the runtime; nothing refuses or clamps them. This PR
    closes the client half only. Drafted, not filed, per the brief.
  • LABEL_PX = 12 with no explicit weight, where DESIGN.md's reference line says
    11 / 700. Predates this change; the brief scoped section 3 to visibility.
  • The save button and the save-state indicator are untouched, as directed.

cf. #417, #425, #451, #473, #511, #512.

The pane spans the whole stage on purpose, so a press in the margin around the
picture reaches the adapter with a coordinate outside the frame. A drag wants
that — "make the box this big" survives leaving the picture. A prompt point
cannot use it: there is nothing under the margin to segment. Measured before
the fix, a click at (900, 700) on a 640x480 asset sent
positive: [{"x":900,"y":700}], painted a dot there, and drew the answer.

`withinBounds` joins `clampPoint` in core/geometry/primitives — the two
questions a frame is asked, and the difference is whether a stray coordinate is
work to be salvaged or an instruction that was never given. The adapter drops
the press before the host hears about it, so no point is recorded, no request
leaves and no preview moves: one guarantee rather than three. Asked in asset
pixels, so zoom and pan are already accounted for.

cf. #451.
… picked

Two rules on one element. The class label renders **only while its shape is
selected** — a frame of forty boxes drew forty class names over the picture at
all times, which hid the asset behind the annotations of it. The panel is the
full inventory; the canvas answers what *this* one is. That rule is DESIGN.md's
own, written under the v1 metrics kept as the reference; the React adapter has
rendered the label unconditionally since the layer was first written, so this
is the adapter starting to obey a rule that was already recorded.

And the label says the class and nothing else. A confidence tells somebody
whether to accept a proposal; once accepted the shape is a label like any
other, so the number stays on the live suggestion preview and leaves the rest
of the editor — no percentage on a canvas box, on a panel row, or in a tooltip.
The Sparkles glyph is now the only provenance signal, and its tooltip carries
the model_ref alone. Nothing is discarded: confidence and model_ref are stored
unchanged, and the number's home is the batch review loop.

`PaintedAnnotation` gives back the two fields it gained for the old label. A
projected field with no reader is what a renderer starts writing next;
absence is the rule, enforced where a component cannot reach around it.
`confidencePercent` stays exported — one consumer today, and it is the spelling
whoever shows the number next must import rather than respell.

cf. #417, #425, #451, #512.
The itemised count is what the scene test exists to hold, and a selected-only
label moves it: two elements per annotation on a frame nobody has picked a
shape on rather than three. The `<text>` was the most expensive of the three —
a stroke, a paint order and a translate each — so a legibility decision pays
here as well.
@JArmandoAnaya
JArmandoAnaya merged commit cd01b24 into main Aug 10, 2026
14 checks passed
@JArmandoAnaya
JArmandoAnaya deleted the fix/editor-polish branch August 10, 2026 14:44
JArmandoAnaya added a commit that referenced this pull request Aug 21, 2026
…preview, selected-only labels (#514)

* fix(annotator): a suggest click outside the asset is not a prompt

The pane spans the whole stage on purpose, so a press in the margin around the
picture reaches the adapter with a coordinate outside the frame. A drag wants
that — "make the box this big" survives leaving the picture. A prompt point
cannot use it: there is nothing under the margin to segment. Measured before
the fix, a click at (900, 700) on a 640x480 asset sent
positive: [{"x":900,"y":700}], painted a dot there, and drew the answer.

`withinBounds` joins `clampPoint` in core/geometry/primitives — the two
questions a frame is asked, and the difference is whether a stray coordinate is
work to be salvaged or an instruction that was never given. The adapter drops
the press before the host hears about it, so no point is recorded, no request
leaves and no preview moves: one guarantee rather than three. Asked in asset
pixels, so zoom and pan are already accounted for.

cf. #451.

* fix(annotator): the canvas says the class, and only for the shape you picked

Two rules on one element. The class label renders **only while its shape is
selected** — a frame of forty boxes drew forty class names over the picture at
all times, which hid the asset behind the annotations of it. The panel is the
full inventory; the canvas answers what *this* one is. That rule is DESIGN.md's
own, written under the v1 metrics kept as the reference; the React adapter has
rendered the label unconditionally since the layer was first written, so this
is the adapter starting to obey a rule that was already recorded.

And the label says the class and nothing else. A confidence tells somebody
whether to accept a proposal; once accepted the shape is a label like any
other, so the number stays on the live suggestion preview and leaves the rest
of the editor — no percentage on a canvas box, on a panel row, or in a tooltip.
The Sparkles glyph is now the only provenance signal, and its tooltip carries
the model_ref alone. Nothing is discarded: confidence and model_ref are stored
unchanged, and the number's home is the batch review loop.

`PaintedAnnotation` gives back the two fields it gained for the old label. A
projected field with no reader is what a renderer starts writing next;
absence is the rule, enforced where a component cannot reach around it.
`confidencePercent` stays exported — one consumer today, and it is the spelling
whoever shows the number next must import rather than respell.

cf. #417, #425, #451, #512.

* test(annotator): the committed layer's element budget is 440, not 660

The itemised count is what the scene test exists to hold, and a selected-only
label moves it: two elements per annotation on a frame nobody has picked a
shape on rather than three. The `<text>` was the most expensive of the three —
a stroke, a paint order and a translate each — so a legibility decision pays
here as well.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Editor polish: asset-bounded suggest clicks, confidence only in preview, selected-only labels

1 participant