fix(inference): a suggest prompt outside the asset is refused, not answered

docs/api.md

@@ -518,7 +518,7 @@ argument for branching on `code`.
 | **405** | `METHOD_NOT_ALLOWED` |
 | **401** | `UNAUTHORIZED` — with a `WWW-Authenticate: Bearer` challenge |
 | **409** | `PROJECT_NAME_TAKEN` · `RELEASE_TAG_TAKEN` · `WORKSPACE_ALREADY_EXISTS` · `WORKSPACE_NOT_EMPTY` · `SCHEMA_VERSION_CONFLICT` · `INVALID_TRANSITION` · `STALE_WRITE` · `BATCH_NOT_EDITABLE` · `BATCH_NOT_IN_ANNOTATION` · `JOB_FINISHED` · `BATCH_NOT_COMPLETE` · `JOB_NOT_COMPLETE` · `EMPTY_BATCH` · `EMPTY_RELEASE` · `CONFIRMATION_REQUIRED` · `DESTRUCTIVE_SCHEMA_CHANGE` · `SCHEMA_CHANGE_WOULD_ORPHAN` · `UNSERIALIZABLE_MANIFEST` · `LOSSY_EXPORT_NOT_CONSENTED` |
-| **422** | `VALIDATION_ERROR` · `INVALID_NAME` · `INVALID_SCHEMA` · `UNSUPPORTED_GEOMETRY` · `INVALID_ANNOTATION` · `LABEL_CLASS_NOT_IN_SCHEMA` · `DISALLOWED_GEOMETRY` · `MISSING_REQUIRED_ATTRIBUTE` · `UNKNOWN_ATTRIBUTE` · `INVALID_ATTRIBUTE_VALUE` · `INVALID_PARTITION` · `MEDIA_ERROR` · `UNSUPPORTED_MEDIA` · `CORRUPT_MEDIA` |
+| **422** | `VALIDATION_ERROR` · `INVALID_NAME` · `INVALID_SCHEMA` · `UNSUPPORTED_GEOMETRY` · `INVALID_ANNOTATION` · `LABEL_CLASS_NOT_IN_SCHEMA` · `DISALLOWED_GEOMETRY` · `MISSING_REQUIRED_ATTRIBUTE` · `UNKNOWN_ATTRIBUTE` · `INVALID_ATTRIBUTE_VALUE` · `INVALID_PARTITION` · `MEDIA_ERROR` · `UNSUPPORTED_MEDIA` · `CORRUPT_MEDIA` · `PROMPT_POINT_OUT_OF_BOUNDS` |
 | **503** | `WORKSPACE_BUSY` |
 | **500** | `WORKSPACE_CORRUPT` · `NOT_A_WORKSPACE` · `WORKSPACE_FORMAT_TOO_NEW` · `WORKSPACE_SCHEMA_MISMATCH` · `ENTITY_NOT_FOUND` · `ENTITY_ALREADY_EXISTS` · `CONSTRAINT_VIOLATED` · `MEDIA_TOOL_UNAVAILABLE` · `INTERNAL_ERROR` |
 

docs/inference.md

@@ -294,6 +294,13 @@ POST /inference/suggest
 carved back without starting over. Refining means sending the accumulated points again, not a
 diff — the call keeps no state about your gesture, so the same points always answer the same way.
 
+**Every point has to be on the asset**, positive and negative alike: `x` in `[0, width]` and `y`
+in `[0, height]`, both ends included. One point outside refuses the whole request with 422
+`PROMPT_POINT_OUT_OF_BOUNDS`, naming the coordinate you sent and the size of the asset. Nothing is
+clamped and nothing is dropped, because both would answer a question you did not ask — a
+segmenter handed a coordinate off the picture still returns a mask, with a confidence attached,
+and that confidence is about nowhere.
+
 **`allowed_geometries` is your schema, not a preference.** The answer comes back in one of the
 kinds you named or not at all: name `polygon` and you get the outline; name only `bbox` and you
 get that outline's extent; name a kind that holds no shape and `region` is `null`. Answering in a

frontend/ui-core/src/generated/api.ts

@@ -937,11 +937,20 @@ export interface paths {
          *     and a class that admits neither gets `region: null`. Answering in a kind the
          *     schema would refuse would produce a suggestion that cannot be accepted.
          *
+         *     **Every point must be on the asset**, positive and negative alike — `x` in
+         *     `[0, width]` and `y` in `[0, height]`, both ends included, in the asset's own
+         *     pixel frame. One point off the picture refuses the whole request with 422
+         *     `PROMPT_POINT_OUT_OF_BOUNDS` rather than being dropped, because a gesture
+         *     with a point removed is a different gesture. Nothing is clamped: a
+         *     coordinate outside the frame is not a place on the image, and answering
+         *     about the nearest edge instead would return a mask, and a confidence, for a
+         *     question nobody asked.
+         *
          *     A null `region` is a successful answer with nothing to propose. Refusals are
          *     reserved for things the caller can act on: an unknown project, asset or
          *     connection is 404; a connection whose weights are not here yet, or whose kind
          *     this build cannot run, is 409 and names what to do; a connection whose model
-         *     answers words rather than places is 422.
+         *     answers words rather than places is 422, as is a prompt point off the asset.
          */
         post: operations["suggest_region"];
         delete?: never;
@@ -3624,6 +3633,12 @@ export interface components {
          *     shape a generated client types as ``number[]`` and a reader has to guess the
          *     order of. The domain's own tuples are fine — Python has positional meaning —
          *     but the wire is read by people.
+         *
+         *     Must be on the asset: `x` in `[0, width]` and `y` in `[0, height]`, both
+         *     ends included. The bounds cannot be stated as field constraints, because
+         *     they belong to the asset the request names rather than to the point, so a
+         *     coordinate off the picture is refused by the route with
+         *     `PROMPT_POINT_OUT_OF_BOUNDS` rather than by this schema.
          */
         SuggestPoint: {
             /** X */

openapi.json

@@ -3425,7 +3425,7 @@
       },
       "SuggestPoint": {
         "additionalProperties": false,
-        "description": "One click, in the asset's own pixel coordinates.\n\nAn object rather than a two-element array because a JSON ``[x, y]`` is a\nshape a generated client types as ``number[]`` and a reader has to guess the\norder of. The domain's own tuples are fine \u2014 Python has positional meaning \u2014\nbut the wire is read by people.",
+        "description": "One click, in the asset's own pixel coordinates.\n\nAn object rather than a two-element array because a JSON ``[x, y]`` is a\nshape a generated client types as ``number[]`` and a reader has to guess the\norder of. The domain's own tuples are fine \u2014 Python has positional meaning \u2014\nbut the wire is read by people.\n\nMust be on the asset: `x` in `[0, width]` and `y` in `[0, height]`, both\nends included. The bounds cannot be stated as field constraints, because\nthey belong to the asset the request names rather than to the point, so a\ncoordinate off the picture is refused by the route with\n`PROMPT_POINT_OUT_OF_BOUNDS` rather than by this schema.",
         "properties": {
           "x": {
             "title": "X",
@@ -6797,7 +6797,7 @@
     },
     "/inference/suggest": {
       "post": {
-        "description": "Propose a shape for the thing under those points.\n\nThe server side of the editor's suggest gesture. One asset, one prompt set,\none answer \u2014 batch prediction is a separate path and is not this one.\n\n**Nothing is written and nothing is remembered.** A suggestion is a proposal:\naccepting it is a later, ordinary annotation write carrying `provenance:\nmodel`, this response's `model_ref`, and its `confidence`. Discarding it\ncosts a request that already finished. The only thing that outlives the call\nis a cached image embedding, which is an optimisation rather than a record \u2014\nso the same points sent twice answer the same way, and a restart changes\nnothing but the latency of the first click.\n\n**The first click on an asset is the slow one.** A segmenter reads the whole\nimage once and then answers any number of clicks from that reading almost for\nfree, which is what makes refining by adding points practical. Sending the\naccumulated points \u2014 rather than a diff \u2014 is what keeps this stateless.\n\n**`allowed_geometries` is the caller's schema, not a preference.** The answer\nis produced in one of the kinds named or not at all: a class that admits\npolygons gets the outline, a class that admits only boxes gets its extent,\nand a class that admits neither gets `region: null`. Answering in a kind the\nschema would refuse would produce a suggestion that cannot be accepted.\n\nA null `region` is a successful answer with nothing to propose. Refusals are\nreserved for things the caller can act on: an unknown project, asset or\nconnection is 404; a connection whose weights are not here yet, or whose kind\nthis build cannot run, is 409 and names what to do; a connection whose model\nanswers words rather than places is 422.",
+        "description": "Propose a shape for the thing under those points.\n\nThe server side of the editor's suggest gesture. One asset, one prompt set,\none answer \u2014 batch prediction is a separate path and is not this one.\n\n**Nothing is written and nothing is remembered.** A suggestion is a proposal:\naccepting it is a later, ordinary annotation write carrying `provenance:\nmodel`, this response's `model_ref`, and its `confidence`. Discarding it\ncosts a request that already finished. The only thing that outlives the call\nis a cached image embedding, which is an optimisation rather than a record \u2014\nso the same points sent twice answer the same way, and a restart changes\nnothing but the latency of the first click.\n\n**The first click on an asset is the slow one.** A segmenter reads the whole\nimage once and then answers any number of clicks from that reading almost for\nfree, which is what makes refining by adding points practical. Sending the\naccumulated points \u2014 rather than a diff \u2014 is what keeps this stateless.\n\n**`allowed_geometries` is the caller's schema, not a preference.** The answer\nis produced in one of the kinds named or not at all: a class that admits\npolygons gets the outline, a class that admits only boxes gets its extent,\nand a class that admits neither gets `region: null`. Answering in a kind the\nschema would refuse would produce a suggestion that cannot be accepted.\n\n**Every point must be on the asset**, positive and negative alike \u2014 `x` in\n`[0, width]` and `y` in `[0, height]`, both ends included, in the asset's own\npixel frame. One point off the picture refuses the whole request with 422\n`PROMPT_POINT_OUT_OF_BOUNDS` rather than being dropped, because a gesture\nwith a point removed is a different gesture. Nothing is clamped: a\ncoordinate outside the frame is not a place on the image, and answering\nabout the nearest edge instead would return a mask, and a confidence, for a\nquestion nobody asked.\n\nA null `region` is a successful answer with nothing to propose. Refusals are\nreserved for things the caller can act on: an unknown project, asset or\nconnection is 404; a connection whose weights are not here yet, or whose kind\nthis build cannot run, is 409 and names what to do; a connection whose model\nanswers words rather than places is 422, as is a prompt point off the asset.",
         "operationId": "suggest_region",
         "requestBody": {
           "content": {

src/visionset/inference/suggestions.py

@@ -32,6 +32,7 @@
     PredictionRequest,
     PredictionTarget,
     media_type_of,
+    require_points_on_asset,
 )
 from visionset.kernel.services import (
     InferenceConnectionService,
@@ -73,12 +74,17 @@ def suggest(
         UnsupportedPrompt: that connection's model answers words, not places.
         ProjectNotFound: no such project.
         AssetNotFound: no such asset in that project.
+        PromptPointOutOfBounds: a point in the gesture is not on that asset.
     """
     connection = InferenceConnectionService(workspace).get(connection_id)
     provider = (pool or resident()).get(connection, workspace_root=workspace.root)
 
     ingest = IngestService(workspace)
     asset = ingest.asset(project_id, asset_id)
+    # Before the bytes are read, because a prompt that names nowhere on this
+    # asset is refused whether or not the file opens, and reading an image to
+    # answer that would be work nobody asked for.
+    require_points_on_asset(prompt, width=asset.width, height=asset.height)
     with ingest.open_content(asset) as handle:
         content = handle.read()
 

src/visionset/kernel/__init__.py

@@ -58,6 +58,7 @@
     NotAWorkspace,
     ProjectNameTaken,
     ProjectNotFound,
+    PromptPointOutOfBounds,
     ReleaseNotFound,
     ReleaseTagTaken,
     SchemaChangeWouldOrphan,
@@ -129,6 +130,7 @@
     "NotAWorkspace",
     "ProjectNameTaken",
     "ProjectNotFound",
+    "PromptPointOutOfBounds",
     "ReleaseNotFound",
     "ReleaseTagTaken",
     "SchemaChangeWouldOrphan",

src/visionset/kernel/domain/__init__.py

@@ -135,6 +135,7 @@
     PredictionTarget,
     Prompt,
     TextPrompt,
+    require_points_on_asset,
 )
 from visionset.kernel.domain.project import Project, ProjectStats
 from visionset.kernel.domain.release import (
@@ -358,6 +359,7 @@
     "progress_after_annotating",
     "report_name",
     "require_move",
+    "require_points_on_asset",
     "require_state",
     "sha256_hex",
 ]

src/visionset/kernel/domain/prediction.py

@@ -41,6 +41,7 @@
 from pydantic import BaseModel, ConfigDict, Field
 
 from visionset.kernel.domain.geometry import Geometry
+from visionset.kernel.errors import PromptPointOutOfBounds
 
 
 class PredictionTarget(BaseModel):
@@ -96,6 +97,46 @@ class PointPrompt(BaseModel):
     negative: tuple[tuple[float, float], ...] = ()
 
 
+def require_points_on_asset(prompt: PointPrompt, *, width: int | None, height: int | None) -> None:
+    """Every point in the gesture is a place on that asset, or none of it is asked.
+
+    A rule about the prompt rather than about the model, which is why it sits
+    here beside :class:`PointPrompt` and not in any provider: a coordinate past
+    the frame is meaningless to every model there will ever be, and the point of
+    refusing it in one place is that no adapter has to remember to.
+
+    **One bad point refuses the whole gesture.** Dropping it and answering the
+    rest would answer a question the caller did not ask — a prompt with a point
+    removed is a different prompt — and negatives are checked exactly like
+    positives, because a *not that* pointing at nothing steers the answer just
+    as wrongly as a *this* would.
+
+    **The frame is inclusive at both ends.** The last row of pixels is part of
+    the asset, and an exclusive rule would make the far edge a place where a
+    press silently stopped working. The editor's own hit test draws the boundary
+    the same way, and the two must agree or there is a coordinate one accepts
+    and the other refuses. A non-finite coordinate falls out of the comparisons
+    rather than being tested for, and is refused.
+
+    An asset whose dimensions were never recorded is not checked. There is
+    nothing to check against, and refusing every prompt on it would punish the
+    caller for a gap in the asset's own metadata.
+
+    Raises:
+        PromptPointOutOfBounds: some point is not on an asset that size.
+    """
+    if width is None or height is None:
+        return
+    for which, points in (("positive", prompt.positive), ("negative", prompt.negative)):
+        for x, y in points:
+            if not (0.0 <= x <= width and 0.0 <= y <= height):
+                raise PromptPointOutOfBounds(
+                    f"the {which} point at ({x:g}, {y:g}) is not on this asset, which is "
+                    f"{width} by {height} pixels; send coordinates with x in [0, {width}] "
+                    f"and y in [0, {height}]"
+                )
+
+
 Prompt = Annotated[TextPrompt | PointPrompt, Field(discriminator="kind")]
 """How a caller says what it is looking for.
 

src/visionset/kernel/errors.py

@@ -941,6 +941,27 @@ class UnsupportedPrompt(VisionSetError):
     """
 
 
+class PromptPointOutOfBounds(VisionSetError):
+    """A prompt point names a place that is not on the asset.
+
+    Prompt coordinates are in the asset's own pixel frame, so a point past its
+    width or height is a question about nothing. A segmenter handed one does not
+    fail — it returns a mask with a confidence attached — and that confidence is
+    about a place nobody asked about, which is worse than an error because it
+    looks like an answer.
+
+    Not clamped onto the nearest edge, and the difference from a drag is the
+    whole reason: a drag that left the picture still means "make the box this
+    big", while a point off the picture is not a point on anything, and moving
+    it would place a prompt somebody never gave.
+
+    ``UnsupportedPrompt``'s sibling in status and its opposite in remedy — that
+    one wants a different kind of prompt or a different connection, this one
+    wants a different coordinate with everything else unchanged — which is why
+    it carries its own code rather than folding into it.
+    """
+
+
 class InferenceConnectionNotDownloadable(VisionSetError):
     """This connection cannot be asked to fetch weights.
 

src/visionset/server/errors.py

@@ -95,6 +95,7 @@
     NotAWorkspace,
     ProjectNameTaken,
     ProjectNotFound,
+    PromptPointOutOfBounds,
     ReleaseNotFound,
     ReleaseTagTaken,
     SchemaChangeWouldOrphan,
@@ -351,6 +352,13 @@ class ErrorRule:
     # mapped anyway, because the exact-correspondence test is what keeps this
     # table honest and an unmapped kernel error answers 500 the day one appears.
     UnsupportedPrompt: ErrorRule(422, "UNSUPPORTED_PROMPT"),
+    # A click past the edge of the picture. 422 beside UNSUPPORTED_PROMPT and
+    # not 404 with the asset's own code: the asset is real and was found, and
+    # what is wrong is a coordinate in the body. Its own code rather than that
+    # one because the remedies are opposites — UNSUPPORTED_PROMPT wants another
+    # kind of prompt or another connection, this wants the same request with a
+    # different point — and a client cannot tell them apart from a shared 422.
+    PromptPointOutOfBounds: ErrorRule(422, "PROMPT_POINT_OUT_OF_BOUNDS"),
     # --- 503: transient, and a wait genuinely helps ------------------------
     WorkspaceBusy: ErrorRule(
         503, "WORKSPACE_BUSY", retry_after=RETRY_AFTER_SECONDS, expose_message=True

src/visionset/server/models.py

@@ -1881,6 +1881,12 @@ class SuggestPoint(BaseModel):
     shape a generated client types as ``number[]`` and a reader has to guess the
     order of. The domain's own tuples are fine — Python has positional meaning —
     but the wire is read by people.
+
+    Must be on the asset: `x` in `[0, width]` and `y` in `[0, height]`, both
+    ends included. The bounds cannot be stated as field constraints, because
+    they belong to the asset the request names rather than to the point, so a
+    coordinate off the picture is refused by the route with
+    `PROMPT_POINT_OUT_OF_BOUNDS` rather than by this schema.
     """
 
     model_config = ConfigDict(extra="forbid")

src/visionset/server/routes/inference.py

@@ -321,11 +321,20 @@ def suggest_region(workspace: WorkspaceDep, body: SuggestRequest) -> SuggestionO
     and a class that admits neither gets `region: null`. Answering in a kind the
     schema would refuse would produce a suggestion that cannot be accepted.
 
+    **Every point must be on the asset**, positive and negative alike — `x` in
+    `[0, width]` and `y` in `[0, height]`, both ends included, in the asset's own
+    pixel frame. One point off the picture refuses the whole request with 422
+    `PROMPT_POINT_OUT_OF_BOUNDS` rather than being dropped, because a gesture
+    with a point removed is a different gesture. Nothing is clamped: a
+    coordinate outside the frame is not a place on the image, and answering
+    about the nearest edge instead would return a mask, and a confidence, for a
+    question nobody asked.
+
     A null `region` is a successful answer with nothing to propose. Refusals are
     reserved for things the caller can act on: an unknown project, asset or
     connection is 404; a connection whose weights are not here yet, or whose kind
     this build cannot run, is 409 and names what to do; a connection whose model
-    answers words rather than places is 422.
+    answers words rather than places is 422, as is a prompt point off the asset.
     """
     prompt = PointPrompt(
         positive=tuple((point.x, point.y) for point in body.positive),