diff --git a/docs/content/api.md b/docs/content/api.md index befede3b..bc135ce2 100644 --- a/docs/content/api.md +++ b/docs/content/api.md @@ -157,7 +157,7 @@ stills in filename order, and two calls can never disagree. **An asset with no recorded arrival sorts last.** That is every asset ingested before `Asset.ingested_at` existed, and it cannot be backfilled — the information exists nowhere, and `Source.registered_at` is not -the proxy it looks like, because registration is idempotent on `(kind, path, extraction_fps)` +the proxy it looks like, because registration is idempotent on `(kind, path, extraction_fps, ranges)` and is never rewritten. Sorting them last is the only reading that degrades quietly: treating the missing value as the epoch invents a date, and treating it as *now* would pin the oldest rows in the product to the top of a "recent" list forever. A workspace that has ingested nothing @@ -349,8 +349,9 @@ one fatal cause, `failures` for the per-item report - because by then there is n to answer. **Uploads are multipart, and the only non-JSON request shape.** Registering a source means -sending the bytes: one `files` part per image, or one `file` part plus an `extraction_fps` field -for a clip. VisionSet sets **no size limit of its own** - parts are spooled to disk past 1 MiB +sending the bytes: one `files` part per image, or one `file` part plus `extraction_fps` — and +optionally `ranges`, a JSON array of half-open `{start_seconds, end_seconds}` stretches to +extract — for a clip. VisionSet sets **no size limit of its own** - parts are spooled to disk past 1 MiB and streamed from there, so memory does not grow with the file - which means the real ceilings are your reverse proxy's (`client_max_body_size` in nginx) and free disk. Uploaded bytes are staged under `/uploads//` and, like blobs, are **never deleted**: a workspace diff --git a/docs/content/cli.md b/docs/content/cli.md index bfd09098..516276e2 100644 --- a/docs/content/cli.md +++ b/docs/content/cli.md @@ -19,7 +19,7 @@ visionset schema draft set FILE --project P [--kind K] [--note TEXT] [--revision visionset schema draft clear --project P [--kind K] visionset schema draft publish --project P [--kind K] [--revision N] [--allow-destructive] -visionset ingest PATH --project P [--fps N] [--batch-name NAME] +visionset ingest PATH --project P [--fps N] [--range S:E]... [--batch-name NAME] visionset batch list --project P visionset batch approve BATCH_ID [--jobs-of N] visionset batch pre-label BATCH_ID CONNECTION [--minimum-confidence FLOAT] [--replace-model-labels] [--geometry SHAPE]... @@ -343,13 +343,16 @@ name, no geometry, a select with no options - is refused there, named by its pos ### `visionset ingest` -`PATH --project P [--fps N] [--batch-name NAME]` - **the one command that is two SDK calls**: +`PATH --project P [--fps N] [--range S:E]... [--batch-name NAME]` - **the one command that is +two SDK calls**: `SourceService.register_images` or `register_video`, dispatched on whether the path is a directory, then `IngestService.ingest`. Registration is idempotent, so re-running the same line registers once; content addressing means it also creates no asset it created before, which is the remedy for an interrupted run. The batch id goes to stdout. -`--fps` is video-only and a usage error on a folder. The run is **synchronous**, and there is no +`--fps` and `--range` are video-only and usage errors on a folder. `--range START:END` repeats, +in seconds, and the selection is stored canonically - clamped to the clip, sorted, overlapping +and touching ranges merged. The run is **synchronous**, and there is no `--resume`: polling needs a second process, which is what `visionset server` and `GET /ingest-jobs/{id}` are for. See [ingest.md](ingest.md#at-a-terminal). diff --git a/docs/content/ingest.md b/docs/content/ingest.md index 9d6f8607..bc8cc5e5 100644 --- a/docs/content/ingest.md +++ b/docs/content/ingest.md @@ -231,10 +231,13 @@ reported.frames_produced # 8 — exact, and the assets are in the batch reported.frames_expected_estimate # 20 — an estimate, and may be None ``` -`frames_produced` is the length of what the loop kept. `frames_expected_estimate` is -`duration_seconds × extraction_fps` off the probe the source has carried since it was registered - - the same arithmetic the ingest screen shows as "Frames expected" before a run starts - so no -second pass over the clip is made for it. It is named an estimate because it is one: +`frames_produced` is the length of what the loop kept. `frames_expected_estimate` is the +domain's own count over the probe and the clip ranges the source has carried since it was +registered — per half-open range `ceil(end × fps) − ceil(start × fps)`, the whole clip +`ceil(duration × fps)`. The same arithmetic builds the extraction filter and the ingest +screen's "Frames expected", so the three cannot disagree, and no second pass over the clip is +made for it; an earlier `floor` spelling undercounted by one on every fractional product. +It is still named an estimate because here it is one: `VideoMetadata` deliberately carries no frame count (for a variable-rate stream the product is a guess), and a damaged container's own metadata is suspect besides. A partial with no denominator still states what it recovered. @@ -343,7 +346,7 @@ they have, because the filesystem already knows, and the source they end up with the path and the rate from then on. It is the only command in the CLI that is two SDK calls, and its module says so. Both are safe to -repeat: registration is idempotent on `(kind, path, extraction_fps)`, and content addressing means a +repeat: registration is idempotent on `(kind, path, extraction_fps, ranges)`, and content addressing means a second run creates nothing the first already did. The batch id goes to stdout alone, so `BATCH=$(visionset ingest …)` is the whole idiom. The per-file @@ -370,7 +373,7 @@ posture the kernel takes about a crashed process. The [API](api.md) is `enqueue` and `resume` with a worker between them. ``` -POST /projects/{id}/sources/video multipart: the clip + extraction_fps → 201 SourceOut +POST /projects/{id}/sources/video multipart: the clip + extraction_fps + ranges → 201 SourceOut POST /sources/{id}/ingest-jobs → 202 IngestJobOut GET /ingest-jobs/{id} → 200 IngestJobOut GET /batches/{id}/assets → 200 the assets @@ -413,8 +416,19 @@ for them. See [batches.md](batches.md). `@visionset/ui-core`'s ingest screen is three steps, and their order is forced by the two facts on this page rather than chosen: **`extraction_fps` belongs to the source**, so it is picked before anything is probed, and the probe only exists once -the source is registered. Registering the same clip at another rate creates a second -source, which the screen says out loud. +the source is registered. Registering the same clip at another rate — or over other +clip ranges — creates a second source, which the screen says out loud. + +A chosen clip the browser can decode also gets a preview player and a timeline: +dragging on it selects one or more **clip ranges**, half-open stretches extraction +alone will read. The selection is part of the source, like the rate. It is stored +canonically — clamped to the clip, sorted, overlapping and touching ranges merged, +a full cover collapsing to the empty selection — and the probe card echoes that +canonical form back, which is where an overlapping selection is first seen merged. +The frame count beside the rate is exact: the same +`ceil(end × fps) − ceil(start × fps)` per range that builds the extraction filter. +A clip the browser cannot decode gets no timeline and one line saying it will be +ingested whole; registration proceeds unchanged. It shows `processed` against `total` for a directory and a bare count for a clip (there is no denominator until an extraction is over), groups the per-file report by diff --git a/docs/content/mcp-tools.md b/docs/content/mcp-tools.md index b1b18213..2202d2e6 100644 --- a/docs/content/mcp-tools.md +++ b/docs/content/mcp-tools.md @@ -26,7 +26,7 @@ error envelope, and the three gate words. | `set_schema_draft` | `project`, `classes`, `kind`?, `note`?, `revision`? | Write the whole draft, creating it when there is none. | | `publish_schema_draft` | `project`, `revision`, `kind`?, `allow_destructive`? | Turn the draft into the next schema version, and clear it. | | `clear_schema_draft` | `project`, `kind`? | Throw the draft away without publishing it. | -| `ingest` | `project`, `path`, `fps`?, `batch_name`? | Register a source and read it into one batch. Blocks until the run finishes. | +| `ingest` | `project`, `path`, `fps`?, `ranges`?, `batch_name`? | Register a source and read it into one batch. Blocks until the run finishes. | | `list_sources` | `project` | List the origins registered in a project — the folders and clips it was built from. | | `backfill_thumbnails` | `project` | Render the previews that are missing for a project's assets. | | `list_batches` | `project` | List a project's batches with where each one's assets have got to. | diff --git a/docs/content/persistence.md b/docs/content/persistence.md index f56f539b..c40d8ebe 100644 --- a/docs/content/persistence.md +++ b/docs/content/persistence.md @@ -48,13 +48,14 @@ never raises `ProjectNameTaken`. But a rule with no backstop is a wish, so the store carries the constraint too: `uq_project_workspace_name` on `project (workspace_id, name COLLATE NOCASE)`, alongside `uq_schema_project_version`, `uq_schema_draft_project_kind`, `uq_member_dataset_asset`, -`uq_release_dataset_tag`, `uq_asset_project_content_hash`, `uq_source_project_kind_path_fps`, +`uq_release_dataset_tag`, `uq_asset_project_content_hash`, `uq_source_project_kind_path_fps_ranges`, `uq_annotation_asset_classification`, `uq_token_workspace_name` and `uq_inference_connection_name`. The invariant then survives a service bug, a forgotten code path, and a second process. -`uq_source_project_kind_path_fps` is one of the two whose terms are not all columns: its fourth -is `coalesce(json_extract(video, '$.extraction_fps'), 0)`. SQLite treats NULLs in a unique index as +`uq_source_project_kind_path_fps_ranges` is one of the two whose terms are not all columns: its +fourth is `coalesce(json_extract(video, '$.extraction_fps'), 0)` and its fifth +`coalesce(json_extract(video, '$.ranges'), '')`. SQLite treats NULLs in a unique index as distinct, so a nullable column would let every image directory collide with nothing at all — and an index is not a query, so no service gains a JSON path from it. That is also why neither it nor `uq_annotation_asset_classification`, which is partial, can use `checkfirst`: SQLAlchemy cannot @@ -167,11 +168,12 @@ MIGRATIONS: list[Migration] = [ Migration(version=13, name="credential_env", upgrade=_add_credential_env), Migration(version=14, name="project_created_at", upgrade=_add_project_created_at), Migration(version=15, name="connection_origin", upgrade=_add_connection_origin), + Migration(version=16, name="source_clip_ranges", upgrade=_reshape_source_origin_index), ] -FORMAT_VERSION: int = MIGRATIONS[-1].version # 15 +FORMAT_VERSION: int = MIGRATIONS[-1].version # 16 ``` -**Generation 1 is the baseline, and the fourteen entries after it are ordinary migrations.** A long +**Generation 1 is the baseline, and the fifteen entries after it are ordinary migrations.** A long chain of generations got this schema to its present shape while VisionSet was unreleased. Every database they could have upgraded was disposable test data inside this repository, so what they actually bought was an idempotency argument and an undo line per generation, plus @@ -205,6 +207,9 @@ so it backfills from the kind with the domain's own rule rather than leaving NUL something a fresh row would never mean. Where the value is unknowable here, the column arrives NULL and something outside the kernel fills it in later - and the column's own docstring says which, so a reader does not mistake an honest absence for a forgotten step. +Migration 16 adds no column at all: it reshapes the source-origin index to carry the +canonical-ranges term, and backfills nothing — a row written before ranges existed has no +`$.ranges` key, which the new index reads as the same `''` a whole-clip selection stores. **There are no downgrade paths, deliberately.** Nothing walks a file backwards and the tests no longer do either. A downgrade is a compatibility promise and a promise is owed @@ -278,8 +283,8 @@ object with `_tables` rather than repeating the DDL - `checkfirst=True` on a `Ta **SQLAlchemy cannot reflect a partial or expression-based index**, so `checkfirst` reports one absent and re-issues a `CREATE` that then fails on every fresh database. Those ask SQLite instead, via `CreateIndex(index, if_not_exists=True)`. Two indexes here are in that -category: `uq_source_project_kind_path_fps` (its fourth term is -`coalesce(json_extract(video, '$.extraction_fps'), 0)`) and +category: `uq_source_project_kind_path_fps_ranges` (its fourth and fifth terms are +`json_extract` expressions over `video`) and `uq_annotation_asset_classification` (partial, on the tag geometry). **A column arriving by `ALTER` is declared last on its row class**, because SQLite appends diff --git a/docs/content/sources.md b/docs/content/sources.md index 44355e63..ff36c715 100644 --- a/docs/content/sources.md +++ b/docs/content/sources.md @@ -51,7 +51,10 @@ name, else the path's last segment, and it is what both wire projections publish `register_images` takes it, because a clip's basename is already its filename. `VideoProvenance` is the port's own `VideoMetadata` — original fps, duration, displayed -dimensions, codec — plus the `extraction_fps` a decomposition will run at. The probe result is +dimensions, codec — plus the cut a decomposition will run at: `extraction_fps`, and the clip +`ranges` extraction reads, empty meaning the whole clip. Ranges are stored canonically — +clamped to the clip, sorted, overlaps and touches merged, a full cover collapsing to the +empty selection — so two spellings of one selection cannot fork a source. The probe result is kept whole rather than re-spelled field by field, because `metadata.fps` is the rate the file was *shot* at and `extraction_fps` is the rate we chose to *cut* it at, and re-declaring the first beside the second is how the two come to be confused. @@ -74,7 +77,7 @@ over one file**, not one source with a history. ## Registration is idempotent -The match key is `(kind, path, extraction_fps)`. Registering the same origin twice returns the +The match key is `(kind, path, extraction_fps, ranges)`. Registering the same origin twice returns the same `Source` rather than a second one, so that once ingest gives `asset.source_id` a target, "which source did this asset come from?" has one answer. @@ -98,12 +101,15 @@ because nothing referenced a source, so a duplicate was inert - and [ingest](ing that, by giving `asset.source_id` a target and letting the winner of a race decide an asset's recorded origin. -`uq_source_project_kind_path_fps` went in with it, over -`(project_id, kind, path, coalesce(json_extract(video, '$.extraction_fps'), 0))`. The fourth term -is an expression rather than a column, and it is `coalesce`d rather than left to be NULL, because -SQLite treats NULLs in a unique index as **distinct** - an image directory, whose `video` is NULL, -would otherwise never collide with itself, which is most of what the index is for. `0` cannot be -mistaken for a real rate: `extraction_fps` is `gt=0`. +`uq_source_project_kind_path_fps_ranges` went in with it — born four-term as +`uq_source_project_kind_path_fps`, reshaped by migration 16 when ranges joined the identity — over +`(project_id, kind, path, coalesce(json_extract(video, '$.extraction_fps'), 0), +coalesce(json_extract(video, '$.ranges'), ''))`. The last two terms are expressions rather than +columns, and they are `coalesce`d rather than left to be NULL, because SQLite treats NULLs in a +unique index as **distinct** - an image directory, whose `video` is NULL, would otherwise never +collide with itself, which is most of what the index is for. `0` cannot be mistaken for a real +rate (`extraction_fps` is `gt=0`), and an empty selection omits its JSON key when stored, so a +whole-clip row written in any generation lands on `''`. The two layers do what they do everywhere else in this store. The pre-check is what produces a friendly answer; the index is the guarantee. A caller that loses the race sees a raw @@ -130,8 +136,9 @@ the workspace, so both stay outside the `VisionSetError` tree - the same line ## Over HTTP, a path is an upload `SourceService` registers by path, and an HTTP client has bytes rather than a path. So the -[REST API](api.md) takes multipart - one `files` part per image, or one `file` part plus an -`extraction_fps` field for a clip - writes the parts under `/uploads/`, and registers +[REST API](api.md) takes multipart - one `files` part per image, or one `file` part plus +`extraction_fps` and an optional `ranges` field for a clip - writes the parts under +`/uploads/`, and registers what it wrote. There is **no route that accepts a server-side path**: it would hand every token holder an arbitrary-directory read, and the two surfaces that legitimately hold real paths, the CLI and MCP, call the SDK in-process and never go through HTTP. diff --git a/docs/content/ui.md b/docs/content/ui.md index 186dfaba..0a551c55 100644 --- a/docs/content/ui.md +++ b/docs/content/ui.md @@ -876,8 +876,26 @@ designing around it. assets" only means something if the parameters are part of what the source *is* - and the probe result exists only once the clip is registered. So the rate is chosen first, the clip is registered, and then its native fps, duration, codec and -resolution are shown. Registering the same clip at another rate produces a -**second source**, deliberately: idempotency is on `(kind, path, extraction_fps)`. +resolution are shown. Registering the same clip at another rate — or over other +clip ranges — produces a **second source**, deliberately: idempotency is on +`(kind, path, extraction_fps, ranges)`. + +A decodable clip gets an editor-shaped block in step 1: a compact preview +player, the cut's facts beside it — rate, frame count, selection — and a +hand-rolled multi-range timeline spanning the panel under both, its empty track +inviting the drag from inside itself. Selection is whole seconds: a drag +paints second cells, a range starts on an exact second and ends just before +its last one closes (the clip's partial final second is the one shorter +cell), and only the marker seconds are labelled under the track, as plain +numbers. The handles drag and nudge by one second (shift for ten), Delete +removes a range, and a click scrubs the player — inside a selected +range it previews, playing from that moment and stopping where the range ends. The selection rides +to registration as typed and the kernel canonicalizes; the probe card's `Ranges` +fact echoes the canonical form, which is where an overlapping selection is first +seen merged. The frame estimate is exact — the mirrored `ceil` arithmetic over +the merged selection, the same numbers the extraction filter is built from. A +clip the browser cannot decode gets no timeline and one line saying it will be +ingested whole. Three more things it inherits: diff --git a/frontend/app/e2e/gallery.spec.ts b/frontend/app/e2e/gallery.spec.ts index 1d0ddc25..403a5099 100644 --- a/frontend/app/e2e/gallery.spec.ts +++ b/frontend/app/e2e/gallery.spec.ts @@ -385,6 +385,7 @@ async function serveApi(page: Page, sent: Request[], options: Options = {}): Pro fps: 30, width: 1280, height: 720, + ranges: [], }, } satisfies Wire["SourceOut"], }); diff --git a/frontend/ui-core/src/generated/api.ts b/frontend/ui-core/src/generated/api.ts index 0858eae5..38681deb 100644 --- a/frontend/ui-core/src/generated/api.ts +++ b/frontend/ui-core/src/generated/api.ts @@ -2437,7 +2437,7 @@ export interface paths { put?: never; /** * Register Video Source - * @description Offer a project a clip, to be cut at `extraction_fps`. + * @description Offer a project a clip, to be cut at `extraction_fps` inside `ranges`. * * The clip is probed on the way in, so a file that is not a video, or one * whose bytes will not decode, is 422 here rather than a run that fails later: @@ -2445,9 +2445,11 @@ export interface paths { * `CORRUPT_MEDIA` for one that is the right kind and will not decode. The * message says what was wrong with the file and never where it was put. * - * The rate is part of what the source *is*: the same clip registered at 1 fps - * and again at 5 fps is two sources over one file, which is what makes "the - * same source yields the same assets" mean anything. + * The cut is part of what the source *is*: the same clip registered at 1 fps + * and again at 5 fps — or over different ranges — is two sources over one + * file, which is what makes "the same source yields the same assets" mean + * anything. Ranges are stored canonically (clamped, sorted, merged), and the + * response carries that canonical form. */ post: operations["register_video_source"]; delete?: never; @@ -3378,6 +3380,11 @@ export interface components { * @description The clip. */ file: string; + /** + * Ranges + * @description Which stretches of the clip to extract, as a JSON array of {"start_seconds": s, "end_seconds": e} objects, each half-open [start, end). Omitted means the whole clip. + */ + ranges?: string | null; }; /** * BySegmentsBody @@ -3471,6 +3478,16 @@ export interface components { */ type: "classification_tag"; }; + /** + * ClipRange + * @description One stretch of a clip to extract, half-open: start_seconds <= t < end_seconds. + */ + ClipRange: { + /** End Seconds */ + end_seconds: number; + /** Start Seconds */ + start_seconds: number; + }; /** * ConnectionAction * @description What can be asked of an inference connection. Order is display order. @@ -5149,7 +5166,11 @@ export interface components { }; /** * VideoProvenanceOut - * @description What a clip turned out to be, and the rate it is decomposed at. + * @description What a clip turned out to be, and the cut it is decomposed by. + * + * `ranges` is the canonical form of the selection the source was registered + * with — clamped to the clip, sorted, overlaps merged — and empty means the + * whole clip. Like `extraction_fps`, it is part of the source's identity. */ VideoProvenanceOut: { /** Codec */ @@ -5162,6 +5183,8 @@ export interface components { fps: number; /** Height */ height: number; + /** Ranges */ + ranges: components["schemas"]["ClipRange"][]; /** Width */ width: number; }; diff --git a/frontend/ui-core/src/generated/checks.ts b/frontend/ui-core/src/generated/checks.ts index 235c9c44..e1b2f4b4 100644 --- a/frontend/ui-core/src/generated/checks.ts +++ b/frontend/ui-core/src/generated/checks.ts @@ -333,8 +333,11 @@ export const checkSchemaVersionPage: Check = export const checkSourceKind: Check = /*#__PURE__*/ oneOf(["image_directory", "video"] as const); +export const checkClipRange: Check = + /*#__PURE__*/ object({ "end_seconds": [true, isNumber], "start_seconds": [true, isNumber] } as const); + export const checkVideoProvenanceOut: Check = - /*#__PURE__*/ object({ "codec": [true, isString], "duration_seconds": [true, isNumber], "extraction_fps": [true, isNumber], "fps": [true, isNumber], "height": [true, isInteger], "width": [true, isInteger] } as const); + /*#__PURE__*/ object({ "codec": [true, isString], "duration_seconds": [true, isNumber], "extraction_fps": [true, isNumber], "fps": [true, isNumber], "height": [true, isInteger], "ranges": [true, arrayOf(checkClipRange)], "width": [true, isInteger] } as const); export const checkSourceOut: Check = /*#__PURE__*/ object({ "id": [true, isString], "kind": [true, checkSourceKind], "name": [true, isString], "project_id": [true, isString], "registered_at": [true, isString], "video": [true, either([checkVideoProvenanceOut, isNull] as const)] } as const); diff --git a/frontend/ui-core/src/screens/ClipRangeTimeline.tsx b/frontend/ui-core/src/screens/ClipRangeTimeline.tsx new file mode 100644 index 00000000..183f50f8 --- /dev/null +++ b/frontend/ui-core/src/screens/ClipRangeTimeline.tsx @@ -0,0 +1,524 @@ +/** + * A multi-range selection over a clip: a mounted player, a track, and handles. + * + * Hand-rolled on purpose. The two native `` controls in + * this repo each carry one thumb; a multi-range media scrubber needs segments, + * paired handles and a playhead, which no installed primitive composes — and + * `@radix-ui/react-slider` stays out of the dependency tree. + * + * **Selection is discrete: whole seconds.** This is not an editor trimming on + * frames — a range starts on an exact second and ends on one, half-open, so a + * drag paints second cells and the handles walk boundaries. The one boundary + * that is not an integer is the clip's own end, which closes a final partial + * cell. Extraction still reads its k/fps grid inside the selection; the cells + * only decide which seconds are in. + * + * The component never merges while a pointer is down — merging a segment under + * the cursor would move what the user is holding. Overlaps live in local + * state; the kernel canonicalizes on registration, and the Selection fact + * beside the player speaks in the merged form. + */ + +import { X } from "lucide-react"; +import { + useEffect, + useRef, + useState, + type JSX, + type KeyboardEvent, + type PointerEvent, + type ReactNode, + type SyntheticEvent, +} from "react"; + +import { cn } from "../lib/cn"; +import { clock, mergedRanges, type ClipRange } from "./clipRanges"; + +function capture(target: Element, pointerId: number): void { + // jsdom implements the method but knows no pointers, so it throws where a + // browser succeeds; losing capture only degrades a drag that leaves the track. + try { + target.setPointerCapture(pointerId); + } catch { + // no pointer pipeline + } +} + +export function ClipRangeTimeline({ + src, + durationSeconds, + ranges, + onRangesChange, + aside, +}: { + /** An object URL the caller owns — the caller revokes it. Null renders no player. */ + readonly src: string | null; + readonly durationSeconds: number; + readonly ranges: readonly ClipRange[]; + readonly onRangesChange: (ranges: readonly ClipRange[]) => void; + /** The cut's facts, laid beside the player; the caller owns their content. */ + readonly aside?: ReactNode; +}): JSX.Element { + const trackRef = useRef(null); + const videoRef = useRef(null); + const [playhead, setPlayhead] = useState(0); + // Where running playback stops: the end of the clip it started inside. A + // ref, not state — timeupdate events are throttled and must read the truth + // synchronously, never a value a commit has not caught up with. + const previewEnd = useRef(null); + const [draft, setDraft] = useState<{ anchor: number; to: number } | null>(null); + const [drag, setDrag] = useState<{ index: number; side: "start" | "end" } | null>(null); + // The filmstrip: as many sampled frames as fit the bar, movie-maker style. + const [thumbs, setThumbs] = useState([]); + + useEffect(() => { + const track = trackRef.current; + if (src === null || track === null) return; + const url = src; + let cancelled = false; + let generation = 0; + let lastWidth = 0; + + function settled(video: HTMLVideoElement, name: "loadedmetadata" | "seeked"): Promise { + return new Promise((resolve, reject) => { + video.addEventListener(name, () => resolve(), { once: true }); + video.addEventListener("error", () => reject(new Error("undecodable")), { once: true }); + }); + } + + async function build(width: number): Promise { + const mine = ++generation; + lastWidth = width; + const video = document.createElement("video"); + video.muted = true; + video.preload = "auto"; + video.src = url; + try { + await settled(video, "loadedmetadata"); + const aspect = + video.videoWidth > 0 && video.videoHeight > 0 + ? video.videoWidth / video.videoHeight + : 16 / 9; + // Exactly as many slots as the bar's width holds at its own height — + // the last one is cropped by overflow rather than squeezed. + const height = 64; + const slot = Math.max(24, Math.round(height * aspect)); + const count = Math.max(1, Math.ceil(width / slot)); + const canvas = document.createElement("canvas"); + canvas.width = slot; + canvas.height = height; + const context = canvas.getContext("2d"); + if (context === null) return; + const made: string[] = []; + for (let at = 0; at < count; at += 1) { + // The center of each slot's own span of the clip, so the strip's + // distribution matches the moments the slots stand over. + video.currentTime = ((at + 0.5) / count) * durationSeconds; + await settled(video, "seeked"); + if (cancelled || mine !== generation) return; + context.drawImage(video, 0, 0, slot, height); + made.push(canvas.toDataURL("image/jpeg", 0.6)); + } + if (!cancelled && mine === generation) setThumbs(made); + } catch { + // A clip the browser cannot decode keeps the plain bar. + } finally { + video.removeAttribute("src"); + } + } + + const width = track.getBoundingClientRect().width; + if (width > 0) void build(width); + const observer = + typeof ResizeObserver === "undefined" + ? null + : new ResizeObserver((entries) => { + const seen = entries[0]?.contentRect.width ?? 0; + // Half a slot of movement before re-decoding the whole strip; the + // generation token retires a build a newer width supersedes. + if (seen > 0 && Math.abs(seen - lastWidth) >= 48) void build(seen); + }); + observer?.observe(track); + return () => { + cancelled = true; + observer?.disconnect(); + }; + }, [src, durationSeconds]); + + // The last whole-second boundary. Every boundary is an integer except the + // clip's own end, which closes a final partial cell. + const last = Math.floor(durationSeconds); + const merged = mergedRanges(ranges, durationSeconds); + + function nearestBoundary(seconds: number): number { + if (seconds <= 0) return 0; + if (seconds >= last) { + if (durationSeconds === last) return last; + return seconds - last <= durationSeconds - seconds ? last : durationSeconds; + } + return Math.round(seconds); + } + + /** The boundary strictly below an end — the latest start its range allows. */ + function boundaryBelow(seconds: number): number { + return seconds > last ? last : Math.ceil(seconds) - 1; + } + + /** The boundary strictly above a start — the earliest end its range allows. */ + function boundaryAbove(seconds: number): number { + const next = Math.floor(seconds) + 1; + return next > last ? durationSeconds : next; + } + + function toSeconds(clientX: number): number { + const rect = trackRef.current?.getBoundingClientRect(); + if (rect === undefined || rect.width === 0) return 0; + const fraction = (clientX - rect.left) / rect.width; + return Math.min(durationSeconds, Math.max(0, fraction * durationSeconds)); + } + + function pixelsToSeconds(pixels: number): number { + const rect = trackRef.current?.getBoundingClientRect(); + if (rect === undefined || rect.width === 0) return 0; + return pixels * (durationSeconds / rect.width); + } + + function percent(seconds: number): string { + return `${(seconds / durationSeconds) * 100}%`; + } + + function moveEndpoint(index: number, side: "start" | "end", seconds: number): void { + onRangesChange( + ranges.map((one, at) => { + if (at !== index) return one; + if (side === "start") { + const snapped = Math.min(nearestBoundary(seconds), boundaryBelow(one.end_seconds)); + return { ...one, start_seconds: Math.max(0, snapped) }; + } + const snapped = Math.max(nearestBoundary(seconds), boundaryAbove(one.start_seconds)); + return { ...one, end_seconds: Math.min(snapped, durationSeconds) }; + }), + ); + } + + /** + * A click is a scrub, and inside a selected range it also starts playback. + * The stop boundary is NOT armed here: `playStarted` owns it, so playback + * begun from the player's own controls obeys the clips exactly the same. + */ + function seek(at: number): void { + const video = videoRef.current; + if (video === null) return; + video.currentTime = at; + const inside = merged.find((one) => at >= one.start_seconds && at < one.end_seconds); + if (inside === undefined) return; + const played: unknown = video.play(); + // A refused autoplay only means the preview stays paused; jsdom returns no + // promise at all. + if (played instanceof Promise) void played.catch(() => undefined); + } + + /** Playback starting inside a clip arms that clip's end — whatever control started it. */ + function playStarted(event: SyntheticEvent): void { + const at = event.currentTarget.currentTime; + const inside = merged.find((one) => at >= one.start_seconds && at < one.end_seconds); + previewEnd.current = inside === undefined ? null : inside.end_seconds; + } + + function timeUpdated(event: SyntheticEvent): void { + const video = event.currentTarget; + const stop = previewEnd.current; + if (stop !== null && video.currentTime >= stop) { + video.pause(); + // timeupdate is throttled, so playback overshoots before this runs; the + // indicator must end on the clip's boundary, never past it. + video.currentTime = stop; + setPlayhead(stop); + return; + } + setPlayhead(video.currentTime); + } + + function trackPointerDown(event: PointerEvent): void { + // The browser's default for this drag is text selection, which turns a + // handle drag into a page-wide highlight; the track owns its pointer. + event.preventDefault(); + capture(event.currentTarget, event.pointerId); + const at = toSeconds(event.clientX); + setDraft({ anchor: at, to: at }); + } + + function trackPointerMove(event: PointerEvent): void { + if (drag !== null) { + moveEndpoint(drag.index, drag.side, toSeconds(event.clientX)); + return; + } + if (draft !== null) setDraft({ anchor: draft.anchor, to: toSeconds(event.clientX) }); + } + + function trackPointerUp(event: PointerEvent): void { + if (drag !== null) { + setDrag(null); + return; + } + if (draft === null) return; + setDraft(null); + const start = Math.min(draft.anchor, draft.to); + const end = Math.max(draft.anchor, draft.to); + // Under a pixel-scale movement this was a click, and a click on a timeline + // means "show me that moment", not a sliver of a range. + if (end - start <= pixelsToSeconds(3)) { + seek(toSeconds(event.clientX)); + return; + } + const startCell = Math.min(Math.floor(start), last); + const above = Math.ceil(end); + let endCell = above > last ? durationSeconds : above; + if (endCell <= startCell) endCell = boundaryAbove(startCell); + onRangesChange([...ranges, { start_seconds: startCell, end_seconds: endCell }]); + } + + function handlePointerDown(index: number, side: "start" | "end") { + return (event: PointerEvent): void => { + event.preventDefault(); + event.stopPropagation(); + const track = trackRef.current; + if (track !== null) capture(track, event.pointerId); + setDrag({ index, side }); + }; + } + + function handleKeys(index: number, side: "start" | "end") { + return (event: KeyboardEvent): void => { + const nudge = event.shiftKey ? 10 : 1; + const one = ranges[index]; + if (one === undefined) return; + if (event.key === "ArrowLeft" || event.key === "ArrowRight") { + event.preventDefault(); + const from = side === "start" ? one.start_seconds : one.end_seconds; + moveEndpoint(index, side, from + (event.key === "ArrowLeft" ? -nudge : nudge)); + } else if (event.key === "Delete" || event.key === "Backspace") { + event.preventDefault(); + onRangesChange(ranges.filter((_, at) => at !== index)); + } + }; + } + + // The draft as the cells it will commit, so a drag paints whole seconds live. + const draftCells = + draft !== null && Math.abs(draft.to - draft.anchor) > 0 + ? { + start: Math.min(Math.floor(Math.min(draft.anchor, draft.to)), last), + end: + Math.ceil(Math.max(draft.anchor, draft.to)) > last + ? durationSeconds + : Math.ceil(Math.max(draft.anchor, draft.to)), + } + : null; + + // Only the seconds where markers sit are labelled — a full ruler is more + // reading than a selection needs. Deduplicated: touching ranges share one. + const markers = Array.from( + new Set( + ranges + .flatMap((one) => [one.start_seconds, one.end_seconds]) + .concat(draftCells === null ? [] : [draftCells.start, draftCells.end]), + ), + ).sort((a, b) => a - b); + + // The stretches outside the selection, for the scrim that makes a clip read + // against its own imagery: unselected washes out, selected stays full-color. + const gaps: { from: number; to: number }[] = []; + if (merged.length > 0) { + let cursor = 0; + for (const one of merged) { + if (one.start_seconds > cursor) gaps.push({ from: cursor, to: one.start_seconds }); + cursor = one.end_seconds; + } + if (cursor < durationSeconds) gaps.push({ from: cursor, to: durationSeconds }); + } + + const handleClass = cn( + "pointer-events-auto absolute inset-y-0 w-2 cursor-ew-resize rounded-sm bg-primary", + "focus-visible:outline-none focus-visible:ring-3 focus-visible:ring-ring/50", + ); + + return ( +
+ {(src !== null || aside !== undefined) && ( +
+ {src !== null && ( +
+ )} +
+
+ {thumbs.length > 0 && ( + + )} + {gaps.map((gap) => ( +