Draft — planning seed, not an implementation ticket. It captures partial information as of 2026-08-06; a dedicated design session will produce the real spec. Do not pick this up for implementation.
What this is
Object identity across frames: the same physical thing, labeled on frame 40 and again on frame
41, understood as one track rather than as two unrelated annotations.
The deliberate boundaries this feature relaxes
Two decisions have already shipped that say "not this, not yet". They are the ones a design
session has to reopen on purpose.
1. Pasting is not predicting. The work that gave the annotator a duplicate-the-selection
command (#123) carries a decision comment from Armando dated 2026-08-06, shipped in c70dc51,
which reads in part:
Explicitly out: polyline vertex editing (stays with #342 — pasting a lane yes,
adjusting its vertices no) and any tracking/interpolation (pasting is not
predicting).
So mod+c and mod+v carry a selection across frames and deliberately stop there. The clipboard
is engine state in @visionset/annotator core, it survives a frame change, and the pasted
annotation is a new independent annotation with no link back. That is the manual half of
tracking already in a user's hands, with the inference half explicitly declined.
2. Progress is per-frame independent. AssetProgress (kernel/domain/task.py) is five
states — unannotated, annotated, skipped, review_pending, accepted — held per asset
inside a job. A frame is settled or not on its own, and nothing in the domain relates one
frame's state to the next frame's. A track spanning 200 frames has, today, 200 unconnected
progress answers.
The asset that already exists
Frames arrive from ingest already content-hashed and ordered. Asset carries frame_index,
its position in the extracted sequence, and frame_timestamp, the seconds into the clip —
described in the code as "the locator that survives a re-decomposition". Both are set on the
video path, and source_id records which clip they came out of. Sequence and provenance are
therefore not new work. What is missing is anything that spans frames.
Note the standing caveat from ingest: video-derived asset identity is reproducible within one
ffmpeg build, not across builds, and a clip registered at two extraction rates counts as two
sources.
Open questions (to name, not to answer here)
- Where identity lives. Either a
Track as a first-class domain object — an identity spanning
assets, with annotations pointing at it — or annotation-level linkage, a previous_id or
track_ref field on Annotation. The first gives the thing a name, a lifetime and a place to
hang attributes; the second is a much smaller change to a model whose annotation currently hangs
off its asset_id and nothing else.
- Progress and review. Does the five-state model stay per frame, with a track read as a
derived rollup, or does a track acquire a state of its own? What does reviewing a track mean —
accepting 200 frames as a unit, or accepting the frames and letting the track follow?
- Interpolation: assistance or data. A box interpolated between two keyframes is either a UI
convenience materialized into real annotations the moment it is accepted, or stored as derived
data that is recomputed. Those two answers differ sharply for export, for the manifest hash, and
for what provenance should say — and provenance is already human | model | import, so an
interpolated box is arguably a fourth thing.
- Correction batches. A correction batch cut over already-labeled assets opens seeded: in
BatchService.approve, an asset that already carries labels starts annotated rather than
unannotated. What happens when the thing being corrected is a track spanning assets both
inside and outside the batch?
Draft — planning seed, not an implementation ticket. It captures partial information as of 2026-08-06; a dedicated design session will produce the real spec. Do not pick this up for implementation.
What this is
Object identity across frames: the same physical thing, labeled on frame 40 and again on frame
41, understood as one track rather than as two unrelated annotations.
The deliberate boundaries this feature relaxes
Two decisions have already shipped that say "not this, not yet". They are the ones a design
session has to reopen on purpose.
1. Pasting is not predicting. The work that gave the annotator a duplicate-the-selection
command (#123) carries a decision comment from Armando dated 2026-08-06, shipped in
c70dc51,which reads in part:
So
mod+candmod+vcarry a selection across frames and deliberately stop there. The clipboardis engine state in
@visionset/annotatorcore, it survives a frame change, and the pastedannotation is a new independent annotation with no link back. That is the manual half of
tracking already in a user's hands, with the inference half explicitly declined.
2. Progress is per-frame independent.
AssetProgress(kernel/domain/task.py) is fivestates —
unannotated,annotated,skipped,review_pending,accepted— held per assetinside a job. A frame is settled or not on its own, and nothing in the domain relates one
frame's state to the next frame's. A track spanning 200 frames has, today, 200 unconnected
progress answers.
The asset that already exists
Frames arrive from ingest already content-hashed and ordered.
Assetcarriesframe_index,its position in the extracted sequence, and
frame_timestamp, the seconds into the clip —described in the code as "the locator that survives a re-decomposition". Both are set on the
video path, and
source_idrecords which clip they came out of. Sequence and provenance aretherefore not new work. What is missing is anything that spans frames.
Note the standing caveat from ingest: video-derived asset identity is reproducible within one
ffmpeg build, not across builds, and a clip registered at two extraction rates counts as two
sources.
Open questions (to name, not to answer here)
Trackas a first-class domain object — an identity spanningassets, with annotations pointing at it — or annotation-level linkage, a
previous_idortrack_reffield onAnnotation. The first gives the thing a name, a lifetime and a place tohang attributes; the second is a much smaller change to a model whose annotation currently hangs
off its
asset_idand nothing else.derived rollup, or does a track acquire a state of its own? What does reviewing a track mean —
accepting 200 frames as a unit, or accepting the frames and letting the track follow?
convenience materialized into real annotations the moment it is accepted, or stored as derived
data that is recomputed. Those two answers differ sharply for export, for the manifest hash, and
for what
provenanceshould say — andprovenanceis alreadyhuman | model | import, so aninterpolated box is arguably a fourth thing.
BatchService.approve, an asset that already carries labels startsannotatedrather thanunannotated. What happens when the thing being corrected is a track spanning assets bothinside and outside the batch?