0.2.0 scope: 3D and point clouds

[JArmandoAnaya]

Draft — planning seed, not an implementation ticket. It captures partial information as of 2026-08-06; a dedicated design session will produce the real spec. Do not pick this up for implementation.

Why this is not a greenfield feature

The domain model was built for a Physical AI roadmap, and two places in the code say so in their
own words rather than in a plan.

• GeometryType (src/visionset/kernel/domain/schema.py) already declares CUBOID_3D and
  POLYLINE_3D, with the docstring "3D values exist today even though unimplemented: the domain
  never assumes 'image' anywhere — that is the Physical AI roadmap encoded as a type." The names
  are reserved; nothing implements them.
• Asset.modality (src/visionset/kernel/domain/asset.py) is Literal["image"] = "image",
  documented as "Only the 'image' modality exists today; the field is typed to extend (video,
  pointcloud, ...) without changing the wire format." It travels to the wire unchanged, as
  AssetOut.modality.

So the vocabulary is reserved in two spots and populated in neither.

What is image-locked today

• The asset model. modality is a one-member Literal, so widening it is both a wire change
  and a migration question. Spatial metadata is width and height only, and Geometry's
  docstring pins coordinates to "the asset's native reference frame — pixels for images".
• The implemented geometries. IMPLEMENTED_GEOMETRIES (kernel/domain/geometry.py) is
  derived off the Geometry union, so it holds exactly bbox, polygon, polyline and
  classification_tag. SchemaService refuses a LabelClass whose geometry falls outside that
  set, which is the mechanism by which cuboid_3d is reserved but unusable. Adding a variant
  widens the set with no second edit, and that part is cheap; everything downstream of it is not.
• Ingest. There are two SourceKind values, image directory and video, and both paths end in
  image bytes: stills go through ImageProcessor.probe, and clips are decomposed into PNG frames
  and then treated as stills. There is no third registration path and no non-image probe.
• The annotator canvas. frontend/annotator/src/adapters/viewport.ts states the model
  outright: "SVG user units are asset pixels". The stage is an <img> plus an <svg> laid
  out at the asset's native size, and the whole screen-to-image transform is 2D affine.
• Thumbnails. Cached previews come from ImageProcessor.thumbnail, and a point cloud has no
  equivalent under that port.
• Exports. Every installed exporter writes a 2D image-dataset format.

Open questions (to name, not to answer here)

• Asset representation and storage. What is one point-cloud asset — a single scan, a sensor
  sweep, a synchronized multi-sensor frame? Content hashing and the blob store are
  format-agnostic, but probing, dimensions and previews are not. Does Asset grow
  modality-specific metadata, or does a sibling model appear beside it?
• Geometries. cuboid_3d and polyline_3d are the reserved names. Are they the right ones,
  and what does a cuboid actually carry — centre, extent and rotation, in what convention? Is a 3D
  polyline the same shape as its 2D namesake with a third coordinate, or a different thing
  altogether?
• The engine. Does the headless annotator core generalize — its geometry, hit testing,
  interaction machine and tolerances are all written against a 2D affine transform — or is a
  sibling 3D engine the honest answer, sharing the document, command and wire layers and nothing
  below them?
• Format targets. The KITTI and nuScenes families are the obvious candidates, but naming them
  is a decision rather than a default.

Related open threads

Two other open issues bear directly on this one. The question of per-vertex annotation data
(#352) — OpenLane's visible/occluded marking, which PolylineGeometry cannot carry — likely
intersects 3D geometry design, because both ask what a vertex is allowed to be. And the polyline
drawing tool (#342) matters here because whatever architecture lands there for multi-vertex
drawing will constrain and inform 3D interaction.