diff --git a/docs/content/cli.md b/docs/content/cli.md index 231977a3..76d80381 100644 --- a/docs/content/cli.md +++ b/docs/content/cli.md @@ -19,7 +19,7 @@ visionset schema draft set FILE --project P [--kind K] [--note TEXT] [--revision visionset schema draft clear --project P [--kind K] visionset schema draft publish --project P [--kind K] [--revision N] [--allow-destructive] -visionset ingest PATH --project P [--fps N] [--range S:E]... [--batch-name NAME] [--start] +visionset ingest PATH --project P [--fps N] [--range S:E]... [--scale PCT] [--batch-name NAME] [--start] visionset batch list --project P visionset batch approve BATCH_ID [--jobs-of N] [--start] visionset batch pre-label BATCH_ID CONNECTION [--minimum-confidence FLOAT] [--replace-model-labels] [--geometry SHAPE]... @@ -347,16 +347,18 @@ name, no geometry, a select with no options - is refused there, named by its pos ### `visionset ingest` -`PATH --project P [--fps N] [--range S:E]... [--batch-name NAME]` - **the one command that is -two SDK calls**: +`PATH --project P [--fps N] [--range S:E]... [--scale PCT] [--batch-name NAME]` - **the one +command that is two SDK calls**: `SourceService.register_images` or `register_video`, dispatched on whether the path is a directory, then `IngestService.ingest`. Registration is idempotent, so re-running the same line registers once; content addressing means it also creates no asset it created before, which is the remedy for an interrupted run. The batch id goes to stdout. -`--fps` and `--range` are video-only and usage errors on a folder. `--range START:END` repeats, -in seconds, and the selection is stored canonically - clamped to the clip, sorted, overlapping -and touching ranges merged. The run is **synchronous**, and there is no +`--fps`, `--range` and `--scale` are video-only and usage errors on a folder. `--range +START:END` repeats, in seconds, and the selection is stored canonically - clamped to the clip, +sorted, overlapping and touching ranges merged. `--scale` stores every extracted frame at that +percent of the clip's native size and, like the rate and the ranges, is part of the source's +identity: another scale is a second source. The run is **synchronous**, and there is no `--resume`: polling needs a second process, which is what `visionset server` and `GET /ingest-jobs/{id}` are for. See [ingest.md](ingest.md#at-a-terminal). diff --git a/docs/content/ingest.md b/docs/content/ingest.md index 2aba58ca..7821716d 100644 --- a/docs/content/ingest.md +++ b/docs/content/ingest.md @@ -65,8 +65,8 @@ reported as unsupported rather than skipped, because guessing which files an ope offer is a policy the kernel would be inventing. **Frames are not re-probed.** `VideoProcessor` guarantees every frame is a complete image in -`FRAME_FORMAT` at the dimensions `probe` reported, and that promise is asserted in the port's own -tests. Decoding each one again to re-confirm it would also route our own encoder's output into an +`FRAME_FORMAT` at the source's stored size — the probe's dimensions scaled by its +`scale_percent` — and that promise is asserted in the port's own tests. Decoding each one again to re-confirm it would also route our own encoder's output into an operator's per-file report - a failure nobody could act on. ## Asking for a run and doing it are two calls diff --git a/docs/content/mcp-tools.md b/docs/content/mcp-tools.md index 1fb01921..41a4773c 100644 --- a/docs/content/mcp-tools.md +++ b/docs/content/mcp-tools.md @@ -26,7 +26,7 @@ error envelope, and the three gate words. | `set_schema_draft` | `project`, `classes`, `kind`?, `note`?, `revision`? | Write the whole draft, creating it when there is none. | | `publish_schema_draft` | `project`, `revision`, `kind`?, `allow_destructive`? | Turn the draft into the next schema version, and clear it. | | `clear_schema_draft` | `project`, `kind`? | Throw the draft away without publishing it. | -| `ingest` | `project`, `path`, `fps`?, `ranges`?, `batch_name`? | Register a source and read it into one batch. Blocks until the run finishes. | +| `ingest` | `project`, `path`, `fps`?, `ranges`?, `scale`?, `batch_name`? | Register a source and read it into one batch. Blocks until the run finishes. | | `list_sources` | `project` | List the origins registered in a project — the folders and clips it was built from. | | `backfill_thumbnails` | `project` | Render the previews that are missing for a project's assets. | | `list_batches` | `project` | List a project's batches with where each one's assets have got to. | diff --git a/docs/content/persistence.md b/docs/content/persistence.md index 79f363ba..43985656 100644 --- a/docs/content/persistence.md +++ b/docs/content/persistence.md @@ -48,14 +48,14 @@ never raises `ProjectNameTaken`. But a rule with no backstop is a wish, so the store carries the constraint too: `uq_project_workspace_name` on `project (workspace_id, name COLLATE NOCASE)`, alongside `uq_schema_project_version`, `uq_schema_draft_project_kind`, `uq_member_dataset_asset`, -`uq_release_dataset_tag`, `uq_asset_project_content_hash`, `uq_source_project_kind_path_fps_ranges`, +`uq_release_dataset_tag`, `uq_asset_project_content_hash`, `uq_source_project_kind_path_fps_ranges_scale`, `uq_annotation_asset_classification`, `uq_token_workspace_name` and `uq_inference_connection_name`. The invariant then survives a service bug, a forgotten code path, and a second process. -`uq_source_project_kind_path_fps_ranges` is one of the two whose terms are not all columns: its -fourth is `coalesce(json_extract(video, '$.extraction_fps'), 0)` and its fifth -`coalesce(json_extract(video, '$.ranges'), '')`. SQLite treats NULLs in a unique index as +`uq_source_project_kind_path_fps_ranges_scale` is one of the two whose terms are not all +columns: beside three column terms it compares `coalesce`d expressions over the `video` JSON +(`$.extraction_fps`, `$.ranges`, `$.scale_percent`). SQLite treats NULLs in a unique index as distinct, so a nullable column would let every image directory collide with nothing at all — and an index is not a query, so no service gains a JSON path from it. That is also why neither it nor `uq_annotation_asset_classification`, which is partial, can use `checkfirst`: SQLAlchemy cannot @@ -286,8 +286,8 @@ object with `_tables` rather than repeating the DDL - `checkfirst=True` on a `Ta **SQLAlchemy cannot reflect a partial or expression-based index**, so `checkfirst` reports one absent and re-issues a `CREATE` that then fails on every fresh database. Those ask SQLite instead, via `CreateIndex(index, if_not_exists=True)`. Two indexes here are in that -category: `uq_source_project_kind_path_fps_ranges` (its fourth and fifth terms are -`json_extract` expressions over `video`) and +category: `uq_source_project_kind_path_fps_ranges_scale` (its expression terms are +`json_extract`/`coalesce` expressions over `video`) and `uq_annotation_asset_classification` (partial, on the tag geometry). **A column arriving by `ALTER` is declared last on its row class**, because SQLite appends diff --git a/docs/content/sources.md b/docs/content/sources.md index ff36c715..b75e28bb 100644 --- a/docs/content/sources.md +++ b/docs/content/sources.md @@ -51,8 +51,9 @@ name, else the path's last segment, and it is what both wire projections publish `register_images` takes it, because a clip's basename is already its filename. `VideoProvenance` is the port's own `VideoMetadata` — original fps, duration, displayed -dimensions, codec — plus the cut a decomposition will run at: `extraction_fps`, and the clip -`ranges` extraction reads, empty meaning the whole clip. Ranges are stored canonically — +dimensions, codec — plus the cut a decomposition will run at: `extraction_fps`, the clip +`ranges` extraction reads (empty meaning the whole clip), and `scale_percent`, the percent of +the native size frames are stored at (100 meaning unscaled). Ranges are stored canonically — clamped to the clip, sorted, overlaps and touches merged, a full cover collapsing to the empty selection — so two spellings of one selection cannot fork a source. The probe result is kept whole rather than re-spelled field by field, because `metadata.fps` is the rate the file was @@ -72,12 +73,26 @@ assets. That promise only means something if the parameters are part of what "th *is* - put them on the ingest job and two runs of one source could legitimately disagree, leaving idempotency with nothing to be measured against. -The consequence is deliberate: **one clip registered at 1 fps and again at 5 fps is two sources -over one file**, not one source with a history. +The consequence is deliberate: **one clip registered at 1 fps and again at 5 fps — or at 100% +and again at 50% — is two sources over one file**, not one source with a history. + +## Storing frames at a smaller size + +A clip's registration takes an optional `scale_percent`, applied while ingest extracts frames: +the original upload is retained as staged, the *stored* frames are smaller, and every frame +shares one size because a clip has one native size. Each dimension becomes +`max(1, (native * percent + 50) // 100)`; the ingest screen mirrors that integer formula, so +the preview and the stored size cannot drift. Image directories always store stills at their +decoded size — an image batch mixes resolutions, and export is where a uniform size is made. + +This is not the export-time resize: [pre-processing recipes](preprocessing.md) bring every +exported image to one model input size at export. The ingest-time scale exists to cut storage +and decode cost for clips nobody needs at native resolution, and it is permanent — the assets +*are* the smaller pixels. Re-ingesting the same clip at another scale is a second source. ## Registration is idempotent -The match key is `(kind, path, extraction_fps, ranges)`. Registering the same origin twice returns the +The match key is `(kind, path, extraction_fps, ranges, scale_percent)`. Registering the same origin twice returns the same `Source` rather than a second one, so that once ingest gives `asset.source_id` a target, "which source did this asset come from?" has one answer. @@ -101,15 +116,18 @@ because nothing referenced a source, so a duplicate was inert - and [ingest](ing that, by giving `asset.source_id` a target and letting the winner of a race decide an asset's recorded origin. -`uq_source_project_kind_path_fps_ranges` went in with it — born four-term as -`uq_source_project_kind_path_fps`, reshaped by migration 16 when ranges joined the identity — over -`(project_id, kind, path, coalesce(json_extract(video, '$.extraction_fps'), 0), -coalesce(json_extract(video, '$.ranges'), ''))`. The last two terms are expressions rather than -columns, and they are `coalesce`d rather than left to be NULL, because SQLite treats NULLs in a -unique index as **distinct** - an image directory, whose `video` is NULL, would otherwise never -collide with itself, which is most of what the index is for. `0` cannot be mistaken for a real -rate (`extraction_fps` is `gt=0`), and an empty selection omits its JSON key when stored, so a -whole-clip row written in any generation lands on `''`. +`uq_source_project_kind_path_fps_ranges_scale` went in with it — born four-term as +`uq_source_project_kind_path_fps`, reshaped by migration 16 when ranges joined the identity and +by migration 18 when scale did — over `(project_id, kind, path, +coalesce(json_extract(video, '$.extraction_fps'), 0), +coalesce(json_extract(video, '$.ranges'), ''), +coalesce(json_extract(video, '$.scale_percent'), 0))`. The expression terms are `coalesce`d +rather than left to be NULL, because SQLite treats NULLs in a unique index as **distinct** - an +image directory, whose `video` is NULL, would otherwise never collide with itself, which is +most of what the index is for. `0` cannot be mistaken for a real rate or percent +(`extraction_fps` is `gt=0`, `scale_percent` is `ge=1`), and every default — an empty +selection, an unscaled clip — omits its JSON key, so a row written in any generation lands on +the same term. The two layers do what they do everywhere else in this store. The pre-check is what produces a friendly answer; the index is the guarantee. A caller that loses the race sees a raw @@ -137,7 +155,8 @@ the workspace, so both stay outside the `VisionSetError` tree - the same line `SourceService` registers by path, and an HTTP client has bytes rather than a path. So the [REST API](api.md) takes multipart - one `files` part per image, or one `file` part plus -`extraction_fps` and an optional `ranges` field for a clip - writes the parts under +`extraction_fps`, an optional `ranges` field and an optional `scale_percent` for a clip - +writes the parts under `/uploads/`, and registers what it wrote. There is **no route that accepts a server-side path**: it would hand every token holder an arbitrary-directory read, and the two surfaces that legitimately hold real paths, the diff --git a/frontend/app/e2e/gallery.spec.ts b/frontend/app/e2e/gallery.spec.ts index c1a4a2b8..04f893b3 100644 --- a/frontend/app/e2e/gallery.spec.ts +++ b/frontend/app/e2e/gallery.spec.ts @@ -478,6 +478,7 @@ async function serveApi(page: Page, sent: Request[], options: Options = {}): Pro width: 1280, height: 720, ranges: [], + scale_percent: 100, }, } satisfies Wire["SourceOut"], }); diff --git a/frontend/ui-core/src/generated/api.ts b/frontend/ui-core/src/generated/api.ts index 9f2584bb..946d0efe 100644 --- a/frontend/ui-core/src/generated/api.ts +++ b/frontend/ui-core/src/generated/api.ts @@ -2702,10 +2702,11 @@ export interface paths { * message says what was wrong with the file and never where it was put. * * The cut is part of what the source *is*: the same clip registered at 1 fps - * and again at 5 fps — or over different ranges — is two sources over one - * file, which is what makes "the same source yields the same assets" mean - * anything. Ranges are stored canonically (clamped, sorted, merged), and the - * response carries that canonical form. + * and again at 5 fps — or over different ranges, or at another scale — is two + * sources over one file, which is what makes "the same source yields the same + * assets" mean anything. Ranges are stored canonically (clamped, sorted, + * merged), and the response carries that canonical form. `scale_percent` + * below 100 stores every extracted frame at that percent of the clip's size. */ post: operations["register_video_source"]; delete?: never; @@ -3709,6 +3710,12 @@ export interface components { * @description Which stretches of the clip to extract, as a JSON array of {"start_seconds": s, "end_seconds": e} objects, each half-open [start, end). Omitted means the whole clip. */ ranges?: string | null; + /** + * Scale Percent + * @description Percent of the native size to store extracted frames at; 100 — the default — stores them unscaled. Part of the source's identity, like extraction_fps: the same clip at another scale is a second source. + * @default 100 + */ + scale_percent: number; }; /** * BySegmentsBody @@ -5801,6 +5808,11 @@ export interface components { * `ranges` is the canonical form of the selection the source was registered * with — clamped to the clip, sorted, overlaps merged — and empty means the * whole clip. Like `extraction_fps`, it is part of the source's identity. + * + * `scale_percent` is the percent of the native size extracted frames are + * stored at; 100 means unscaled. `width` and `height` stay the clip's own — + * what is stored is each dimension scaled by this percent. Also part of the + * source's identity. */ VideoProvenanceOut: { /** Codec */ @@ -5815,6 +5827,8 @@ export interface components { height: number; /** Ranges */ ranges: components["schemas"]["ClipRange"][]; + /** Scale Percent */ + scale_percent: number; /** Width */ width: number; }; diff --git a/frontend/ui-core/src/generated/checks.ts b/frontend/ui-core/src/generated/checks.ts index cf4f85e1..d5c7e7b0 100644 --- a/frontend/ui-core/src/generated/checks.ts +++ b/frontend/ui-core/src/generated/checks.ts @@ -376,7 +376,7 @@ export const checkClipRange: Check = /*#__PURE__*/ object({ "end_seconds": [true, isNumber], "start_seconds": [true, isNumber] } as const); export const checkVideoProvenanceOut: Check = - /*#__PURE__*/ object({ "codec": [true, isString], "duration_seconds": [true, isNumber], "extraction_fps": [true, isNumber], "fps": [true, isNumber], "height": [true, isInteger], "ranges": [true, arrayOf(checkClipRange)], "width": [true, isInteger] } as const); + /*#__PURE__*/ object({ "codec": [true, isString], "duration_seconds": [true, isNumber], "extraction_fps": [true, isNumber], "fps": [true, isNumber], "height": [true, isInteger], "ranges": [true, arrayOf(checkClipRange)], "scale_percent": [true, isInteger], "width": [true, isInteger] } as const); export const checkSourceOut: Check = /*#__PURE__*/ object({ "id": [true, isString], "kind": [true, checkSourceKind], "name": [true, isString], "project_id": [true, isString], "registered_at": [true, isString], "video": [true, either([checkVideoProvenanceOut, isNull] as const)] } as const); diff --git a/frontend/ui-core/src/screens/ClipRangeTimeline.tsx b/frontend/ui-core/src/screens/ClipRangeTimeline.tsx index 183f50f8..5fe25d72 100644 --- a/frontend/ui-core/src/screens/ClipRangeTimeline.tsx +++ b/frontend/ui-core/src/screens/ClipRangeTimeline.tsx @@ -352,8 +352,12 @@ export function ClipRangeTimeline({ ref={videoRef} src={src} controls + // Always mute: a vision dataset never needs the audio track, and + // the volume control it would earn is noise. The matching CSS in + // styles.css hides the control itself where the engine allows. + muted preload="metadata" - className="max-h-84 w-full max-w-2xl shrink-0 rounded-lg bg-muted" + className="vs-muted-player max-h-84 w-full max-w-2xl shrink-0 rounded-lg bg-muted" data-testid="clip-player" onTimeUpdate={timeUpdated} onPlay={playStarted} diff --git a/frontend/ui-core/src/screens/IngestScreen.tsx b/frontend/ui-core/src/screens/IngestScreen.tsx index 1b864b63..037cf3b0 100644 --- a/frontend/ui-core/src/screens/IngestScreen.tsx +++ b/frontend/ui-core/src/screens/IngestScreen.tsx @@ -149,6 +149,7 @@ import { Card, CardContent } from "../primitives/card"; import { Progress } from "../primitives/progress"; import { Input } from "../primitives/input"; import { Label } from "../primitives/label"; +import { ScaleField, scaledDimension } from "./ingestScale"; import { FieldDescription, FieldError } from "../primitives/field"; import { Select, @@ -286,6 +287,9 @@ export function IngestScreen({ // overlapping: the kernel canonicalizes on registration, and step 2 echoes // the merged form back. const [ranges, setRanges] = useState([]); + // A clip's storage scale, decided in step 1 like the rate: percent of the + // native size frames are stored at, part of the source's identity. + const [scalePercent, setScalePercent] = useState(100); // The chosen clip as an object URL for the preview player. Null where the // platform has no object URLs (jsdom), so the timeline renders no player. const [clipUrl, setClipUrl] = useState(null); @@ -331,6 +335,7 @@ export function IngestScreen({ useEffect(() => { setClip(null); setRanges([]); + setScalePercent(100); setUnreadable(false); if (!(files.length === 1 && files[0].type.startsWith("video/"))) return; const url = typeof URL.createObjectURL === "function" ? URL.createObjectURL(files[0]) : null; @@ -379,7 +384,7 @@ export function IngestScreen({ register.mutate( { files, - ...(isVideo ? { extractionFps: rate, ranges } : {}), + ...(isVideo ? { extractionFps: rate, ranges, scalePercent } : {}), ...(isVideo || stated === "" ? {} : { name: stated }), }, { onSuccess: (registered) => setSource(registered) }, @@ -397,6 +402,7 @@ export function IngestScreen({ function clearFiles(): void { setFiles([]); setFps(String(DEFAULT_EXTRACTION_FPS)); + setScalePercent(100); setSourceName(""); setAttempt((previous) => previous + 1); register.reset(); @@ -414,6 +420,7 @@ export function IngestScreen({ function again(): void { setFiles([]); setFps(String(DEFAULT_EXTRACTION_FPS)); + setScalePercent(100); setSourceName(""); setBatchChoice(NEW_BATCH); setBatchName(""); @@ -511,6 +518,8 @@ export function IngestScreen({ suggestedName={suggestedName} ranges={ranges} onRanges={setRanges} + scalePercent={scalePercent} + onScalePercent={setScalePercent} clipUrl={clipUrl} unreadable={unreadable} estimate={ @@ -589,7 +598,17 @@ export function IngestScreen({ label="Duration" value={`${source.video.duration_seconds.toFixed(1)} s`} /> - + void; + readonly scalePercent: number; + readonly onScalePercent: (value: number) => void; readonly clipUrl: string | null; readonly unreadable: boolean; readonly estimate: number | null; @@ -922,6 +945,15 @@ function SelectionPanel({ aside={
+
{estimate !== null && ( <> @@ -942,7 +974,7 @@ function SelectionPanel({
Part of what the source is — the same clip registered at another - rate or other ranges becomes a second source. + rate, other ranges, or another scale becomes a second source.
} @@ -955,9 +987,10 @@ function SelectionPanel({

)} + - Part of what the source is — the same clip registered at another rate - or other ranges becomes a second source. + Part of what the source is — the same clip registered at another rate, + other ranges, or another scale becomes a second source. )} diff --git a/frontend/ui-core/src/screens/clipProbe.ts b/frontend/ui-core/src/screens/clipProbe.ts index 30fccfa5..1d0188de 100644 --- a/frontend/ui-core/src/screens/clipProbe.ts +++ b/frontend/ui-core/src/screens/clipProbe.ts @@ -18,6 +18,9 @@ export interface ClipProbe { readonly durationSeconds: number; + /** Display dimensions, or null when the browser reports none (audio-only, some codecs). */ + readonly width: number | null; + readonly height: number | null; } export function probeClip(file: File): Promise { @@ -37,7 +40,11 @@ export function probeClip(file: File): Promise { // neither is a duration an estimate should be built on. done( Number.isFinite(video.duration) && video.duration > 0 - ? { durationSeconds: video.duration } + ? { + durationSeconds: video.duration, + width: video.videoWidth > 0 ? video.videoWidth : null, + height: video.videoHeight > 0 ? video.videoHeight : null, + } : null, ); }); diff --git a/frontend/ui-core/src/screens/clipRangeTimeline.test.tsx b/frontend/ui-core/src/screens/clipRangeTimeline.test.tsx index 9f1b7a81..a8c0167b 100644 --- a/frontend/ui-core/src/screens/clipRangeTimeline.test.tsx +++ b/frontend/ui-core/src/screens/clipRangeTimeline.test.tsx @@ -109,6 +109,8 @@ describe("ClipRangeTimeline", () => { // 50px of 200 over 2 s is 0.5 s — inside the range, so the click plays. const video = screen.getByTestId("clip-player") as HTMLVideoElement; + // Always muted: a vision dataset has no use for the audio track. + expect(video.muted).toBe(true); expect(video.currentTime).toBe(0.5); expect(play).toHaveBeenCalled(); // The browser answers play() with the play event; the boundary arms there. diff --git a/frontend/ui-core/src/screens/gallery.test.tsx b/frontend/ui-core/src/screens/gallery.test.tsx index 2f6cd22e..54c82f34 100644 --- a/frontend/ui-core/src/screens/gallery.test.tsx +++ b/frontend/ui-core/src/screens/gallery.test.tsx @@ -459,6 +459,7 @@ describe("the gallery", () => { width: 1280, height: 720, ranges: [], + scale_percent: 100, }, }, }); diff --git a/frontend/ui-core/src/screens/ingest.test.tsx b/frontend/ui-core/src/screens/ingest.test.tsx index 8719867b..b5b0c6f1 100644 --- a/frontend/ui-core/src/screens/ingest.test.tsx +++ b/frontend/ui-core/src/screens/ingest.test.tsx @@ -150,6 +150,7 @@ const VIDEO_SOURCE = { codec: "h264", extraction_fps: 2, ranges: [], + scale_percent: 100, }, }; @@ -244,8 +245,51 @@ describe("registering a source", () => { expect(form.has("file")).toBe(true); }); + it("sends the chosen scale and previews the stored size", async () => { + vi.mocked(probeClip).mockResolvedValueOnce({ durationSeconds: 10, width: 1920, height: 1080 }); + on("POST", /\/sources\/video$/, { status: 201, body: VIDEO_SOURCE }); + + render(mount()); + await choose([pick("drive.mp4", "video/mp4")]); + + // Untouched, the readout still states what exists — the fact was missing + // from the first design and is the reason the block leads with it. + expect((await screen.findByTestId("stored-size-native")).textContent).toContain("1920×1080"); + fireEvent.change(screen.getByTestId("scale-percent"), { target: { value: "50" } }); + expect(screen.getByTestId("stored-size").textContent).toContain("960×540"); + + await userEvent.click(screen.getByTestId("register-source")); + await waitFor(() => expect(sent.some((r) => r.method === "POST")).toBe(true)); + const form = bodies.get(sent.find((r) => r.method === "POST") as Request) as FormData; + expect(form.get("scale_percent")).toBe("50"); + }); + + it("sends the default scale untouched, as one hundred", async () => { + on("POST", /\/sources\/video$/, { status: 201, body: VIDEO_SOURCE }); + + render(mount()); + await choose([pick("drive.mp4", "video/mp4")]); + await userEvent.click(screen.getByTestId("register-source")); + + await waitFor(() => expect(sent.some((r) => r.method === "POST")).toBe(true)); + const form = bodies.get(sent.find((r) => r.method === "POST") as Request) as FormData; + expect(form.get("scale_percent")).toBe("100"); + }); + + it("offers the slider without a size preview when the clip is unreadable", async () => { + vi.mocked(probeClip).mockResolvedValueOnce(null); + + render(mount()); + await choose([pick("weird.mkv", "video/x-matroska")]); + await screen.findByTestId("clip-undecodable"); + + fireEvent.change(screen.getByTestId("scale-percent"), { target: { value: "50" } }); + expect(screen.queryByTestId("stored-size")).toBeNull(); + expect(screen.getByTestId("stored-size-blind").textContent).toContain("50%"); + }); + it("threads the timeline selection into the multipart body, raw", async () => { - vi.mocked(probeClip).mockResolvedValueOnce({ durationSeconds: 10 }); + vi.mocked(probeClip).mockResolvedValueOnce({ durationSeconds: 10, width: 1920, height: 1080 }); on("POST", /\/sources\/video$/, { status: 201, body: VIDEO_SOURCE }); render(mount()); @@ -431,7 +475,7 @@ describe("the selection panel", () => { }); it("estimates the frames from the browser's own read of the clip", async () => { - vi.mocked(probeClip).mockResolvedValueOnce({ durationSeconds: 47.7 }); + vi.mocked(probeClip).mockResolvedValueOnce({ durationSeconds: 47.7, width: 1920, height: 1080 }); render(mount()); await choose([pick("drive.mp4", "video/mp4")]); diff --git a/frontend/ui-core/src/screens/ingestScale.test.ts b/frontend/ui-core/src/screens/ingestScale.test.ts new file mode 100644 index 00000000..32023636 --- /dev/null +++ b/frontend/ui-core/src/screens/ingestScale.test.ts @@ -0,0 +1,12 @@ +import { describe, expect, it } from "vitest"; + +import { scaledDimension } from "./ingestScale"; + +describe("scaledDimension", () => { + it("mirrors the server's integer half-up formula", () => { + expect(scaledDimension(25, 50)).toBe(13); + expect(scaledDimension(1920, 50)).toBe(960); + expect(scaledDimension(1, 10)).toBe(1); + expect(scaledDimension(640, 100)).toBe(640); + }); +}); diff --git a/frontend/ui-core/src/screens/ingestScale.tsx b/frontend/ui-core/src/screens/ingestScale.tsx new file mode 100644 index 00000000..f573fc91 --- /dev/null +++ b/frontend/ui-core/src/screens/ingestScale.tsx @@ -0,0 +1,90 @@ +import type { JSX } from "react"; + +import { Label } from "../primitives/label"; + +/** + * The server's scaled-dimension formula, mirrored exactly. + * + * Integer half-up on purpose: Python `round` is half-even and `Math.round` is + * half-up, so the one spelling both sides can share is integer arithmetic — + * the kernel's `scaled_dimension`. The 25 × 50% → 13 fixture is pinned on both + * sides to keep them one formula. + */ +export function scaledDimension(native: number, percent: number): number { + return Math.max(1, Math.floor((native * percent + 50) / 100)); +} + +/** + * A native `input[type=range]` and not a primitive, for SuggestPanel's reason: + * no slider primitive exists in this package and one control does not earn + * one. Never `preventDefault` its pointer press — a range *drags* on its + * default action, and cancelling the press is what made one unmovable (#563). + * + * The block leads with the outcome, not the mechanism: a readout that is + * always present (what resolution exists, what will be stored) and a purpose + * line that says what the value costs — the two facts a person needs *before* + * touching the slider. + */ +export function ScaleField({ + percent, + onPercent, + native, + id = "scale-percent", +}: { + readonly percent: number; + readonly onPercent: (value: number) => void; + readonly native: { readonly width: number; readonly height: number } | null; + readonly id?: string; +}): JSX.Element { + const pixels = Math.round((percent * percent) / 100); + return ( +
+
+ + {native !== null ? ( + percent < 100 ? ( + + {native.width}×{native.height} → {scaledDimension(native.width, percent)}× + {scaledDimension(native.height, percent)} · {percent}% + + ) : ( + + {native.width}×{native.height} · native + + ) + ) : ( + + {percent < 100 ? `${percent}% per side · ` : ""}exact size read at upload + + )} +
+
+ 10% + onPercent(Number(event.target.value))} + className="h-1 flex-1 cursor-pointer accent-primary" + /> + 100% +
+

+ {percent < 100 + ? `Every frame stored at ${percent}% per side — about ${pixels}% of the ` + + `pixels, so smaller files and faster training. Annotations are drawn on ` + + `what is stored.` + : `Stored as captured. Drag left to store smaller frames.`} +

+
+ ); +} diff --git a/frontend/ui-core/src/screens/queries.ts b/frontend/ui-core/src/screens/queries.ts index f0bd0d16..fef90c9e 100644 --- a/frontend/ui-core/src/screens/queries.ts +++ b/frontend/ui-core/src/screens/queries.ts @@ -798,6 +798,7 @@ export function useRegisterSource(projectId: string) { extractionFps?: number; ranges?: readonly { start_seconds: number; end_seconds: number }[]; name?: string; + scalePercent?: number; }) => { const extractionFps = input.extractionFps; const source = @@ -813,6 +814,7 @@ export function useRegisterSource(projectId: string) { ...(input.ranges !== undefined && input.ranges.length > 0 ? { ranges: JSON.stringify(input.ranges) } : {}), + scale_percent: input.scalePercent ?? 100, }, bodySerializer: formData, }), diff --git a/frontend/ui-core/src/styles.css b/frontend/ui-core/src/styles.css index 23d45c6f..3137e212 100644 --- a/frontend/ui-core/src/styles.css +++ b/frontend/ui-core/src/styles.css @@ -235,3 +235,12 @@ @apply font-heading; } } + +/* The ingest preview player is always muted — a vision dataset has no use for + the audio track — so the volume controls are dead weight. Only the WebKit + engines expose the pseudo-elements; elsewhere the player is merely muted. */ +.vs-muted-player::-webkit-media-controls-mute-button, +.vs-muted-player::-webkit-media-controls-volume-slider, +.vs-muted-player::-webkit-media-controls-volume-control-container { + display: none !important; +} diff --git a/openapi.json b/openapi.json index 74f79233..dc387b43 100644 --- a/openapi.json +++ b/openapi.json @@ -1670,6 +1670,14 @@ ], "description": "Which stretches of the clip to extract, as a JSON array of {\"start_seconds\": s, \"end_seconds\": e} objects, each half-open [start, end). Omitted means the whole clip.", "title": "Ranges" + }, + "scale_percent": { + "default": 100, + "description": "Percent of the native size to store extracted frames at; 100 \u2014 the default \u2014 stores them unscaled. Part of the source's identity, like extraction_fps: the same clip at another scale is a second source.", + "maximum": 100.0, + "minimum": 1.0, + "title": "Scale Percent", + "type": "integer" } }, "required": [ @@ -5810,7 +5818,7 @@ "x-visionset-open": true }, "VideoProvenanceOut": { - "description": "What a clip turned out to be, and the cut it is decomposed by.\n\n`ranges` is the canonical form of the selection the source was registered\nwith \u2014 clamped to the clip, sorted, overlaps merged \u2014 and empty means the\nwhole clip. Like `extraction_fps`, it is part of the source's identity.", + "description": "What a clip turned out to be, and the cut it is decomposed by.\n\n`ranges` is the canonical form of the selection the source was registered\nwith \u2014 clamped to the clip, sorted, overlaps merged \u2014 and empty means the\nwhole clip. Like `extraction_fps`, it is part of the source's identity.\n\n`scale_percent` is the percent of the native size extracted frames are\nstored at; 100 means unscaled. `width` and `height` stay the clip's own \u2014\nwhat is stored is each dimension scaled by this percent. Also part of the\nsource's identity.", "properties": { "codec": { "title": "Codec", @@ -5839,6 +5847,10 @@ "title": "Ranges", "type": "array" }, + "scale_percent": { + "title": "Scale Percent", + "type": "integer" + }, "width": { "title": "Width", "type": "integer" @@ -5851,7 +5863,8 @@ "duration_seconds", "codec", "extraction_fps", - "ranges" + "ranges", + "scale_percent" ], "title": "VideoProvenanceOut", "type": "object" @@ -14650,7 +14663,7 @@ }, "/projects/{project_id}/sources/video": { "post": { - "description": "Offer a project a clip, to be cut at `extraction_fps` inside `ranges`.\n\nThe clip is probed on the way in, so a file that is not a video, or one\nwhose bytes will not decode, is 422 here rather than a run that fails later:\n422 `UNSUPPORTED_MEDIA` for a kind of file this cannot cut, and 422\n`CORRUPT_MEDIA` for one that is the right kind and will not decode. The\nmessage says what was wrong with the file and never where it was put.\n\nThe cut is part of what the source *is*: the same clip registered at 1 fps\nand again at 5 fps \u2014 or over different ranges \u2014 is two sources over one\nfile, which is what makes \"the same source yields the same assets\" mean\nanything. Ranges are stored canonically (clamped, sorted, merged), and the\nresponse carries that canonical form.", + "description": "Offer a project a clip, to be cut at `extraction_fps` inside `ranges`.\n\nThe clip is probed on the way in, so a file that is not a video, or one\nwhose bytes will not decode, is 422 here rather than a run that fails later:\n422 `UNSUPPORTED_MEDIA` for a kind of file this cannot cut, and 422\n`CORRUPT_MEDIA` for one that is the right kind and will not decode. The\nmessage says what was wrong with the file and never where it was put.\n\nThe cut is part of what the source *is*: the same clip registered at 1 fps\nand again at 5 fps \u2014 or over different ranges, or at another scale \u2014 is two\nsources over one file, which is what makes \"the same source yields the same\nassets\" mean anything. Ranges are stored canonically (clamped, sorted,\nmerged), and the response carries that canonical form. `scale_percent`\nbelow 100 stores every extracted frame at that percent of the clip's size.", "operationId": "register_video_source", "parameters": [ { diff --git a/src/visionset/cli/ingest.py b/src/visionset/cli/ingest.py index 401a7d24..b2664354 100644 --- a/src/visionset/cli/ingest.py +++ b/src/visionset/cli/ingest.py @@ -9,9 +9,10 @@ dispatch is ``path.is_dir()``. Registering twice is free: registration is idempotent on -``(kind, path, extraction_fps)``, so running this again on the same folder finds -the same source. Ingesting again is nearly free too — content addressing means a -re-run creates no assets it created before — which is also the remedy for the one +``(kind, path, extraction_fps, ranges, scale_percent)``, so running this again +on the same folder finds the same source. Ingesting again is nearly free too — +content addressing means a re-run creates no assets it created before — which +is also the remedy for the one gap this command has: interrupting it leaves the job row at ``running``, and there is no ``--resume``, because re-running does the right thing and needs no new vocabulary. @@ -156,6 +157,19 @@ def ingest( ), ), ] = None, + scale: Annotated[ + int | None, + typer.Option( + "--scale", + min=1, + max=100, + help=( + "Store extracted frames at this percent of the clip's native size. " + "Video sources only. Part of the source's identity, like --fps: " + "another scale is a second source. Defaults to 100." + ), + ), + ] = None, batch_name: Annotated[ str | None, typer.Option( @@ -202,6 +216,10 @@ def ingest( raise typer.BadParameter( f"--range applies to a video source; {source} is a directory of stills" ) + if scale is not None and source.is_dir(): + raise typer.BadParameter( + f"--scale applies to a video source; {source} is a directory of stills" + ) ranges = [_parse_range(spec) for spec in range_specs or ()] with opened_workspace(workspace) as service: @@ -215,6 +233,7 @@ def ingest( source, extraction_fps=DEFAULT_EXTRACTION_FPS if fps is None else fps, ranges=ranges, + scale_percent=100 if scale is None else scale, ) note(f"Reading {registered.kind.value.replace('_', ' ')} {source}…") result = IngestService(service).ingest(registered.id, batch_name=batch_name) diff --git a/src/visionset/kernel/adapters/_mappers.py b/src/visionset/kernel/adapters/_mappers.py index 33f4d801..9f6639dd 100644 --- a/src/visionset/kernel/adapters/_mappers.py +++ b/src/visionset/kernel/adapters/_mappers.py @@ -346,18 +346,20 @@ def _change_to_domain(_: Session, row: Any) -> DatasetChange: def _video_to_json(video: VideoProvenance | None) -> dict[str, Any] | None: - """``VideoProvenance`` as stored, with an empty ``ranges`` key omitted. + """``VideoProvenance`` as stored, with the default cut keys omitted. - Omitted, not stored as ``[]``: the source-origin index compares - ``json_extract(video, '$.ranges')``, and rows written before ranges existed - have no key — a whole-clip selection must serialize the same way, or the - index would hold two spellings of one origin. + Omitted, not stored as ``[]`` or ``100``: the source-origin index compares + ``json_extract`` of these keys, and rows written before each feature + existed have no key — a whole-clip, unscaled selection must serialize the + same way, or the index would hold two spellings of one origin. """ if video is None: return None dump = video.model_dump(mode="json") if not dump["ranges"]: del dump["ranges"] + if dump["scale_percent"] == 100: + del dump["scale_percent"] return dump diff --git a/src/visionset/kernel/adapters/_tables.py b/src/visionset/kernel/adapters/_tables.py index e231594b..1fc55371 100644 --- a/src/visionset/kernel/adapters/_tables.py +++ b/src/visionset/kernel/adapters/_tables.py @@ -206,17 +206,21 @@ class SourceRow(Base): #: written before ranges existed or after — always lands on ``''``, and a row #: that names ranges lands on its one canonical JSON spelling. #: +#: The sixth term follows the same precedent: a clip stored unscaled omits +#: ``$.scale_percent``, and 0 cannot be a real percent — the domain floor is 1. +#: #: This is SQL reading a JSON column, which the module docstring above reserves #: for values "nothing ever queries". An index is not a query: no service gains #: a JSON path, ``_source_to_domain`` still rehydrates ``VideoProvenance`` whole, #: and the doctrine's purpose — no service building SQL over JSON — is intact. SOURCE_ORIGIN_UNIQUE = Index( - "uq_source_project_kind_path_fps_ranges", + "uq_source_project_kind_path_fps_ranges_scale", SourceRow.project_id, SourceRow.kind, SourceRow.path, text("coalesce(json_extract(video, '$.extraction_fps'), 0)"), text("coalesce(json_extract(video, '$.ranges'), '')"), + text("coalesce(json_extract(video, '$.scale_percent'), 0)"), unique=True, ) diff --git a/src/visionset/kernel/adapters/ffmpeg_video_processor.py b/src/visionset/kernel/adapters/ffmpeg_video_processor.py index 36c4a9e0..bc505422 100644 --- a/src/visionset/kernel/adapters/ffmpeg_video_processor.py +++ b/src/visionset/kernel/adapters/ffmpeg_video_processor.py @@ -419,6 +419,7 @@ def frames( fps: float = DEFAULT_EXTRACTION_FPS, ranges: tuple[TimeRange, ...] = (), name: str | None = None, + scale: tuple[int, int] | None = None, ) -> Iterator[VideoFrame]: """Frames taken off ``source`` at ``fps``, one at a time, in grid order. @@ -427,6 +428,12 @@ def frames( each kept frame carries its grid index, byte-identical to the frame a whole-clip run yields at that index. + ``scale`` is the exact ``(width, height)`` every emitted frame is + resized to, computed by the caller from the probe — never ffmpeg-side + arithmetic, so the command stays deterministic and the probe's + display-oriented dimensions stay authoritative. ``None`` emits frames + at the decoded size. + Not a generator itself, on purpose. A generator's body does not run until something asks it for a value, so writing it that way would report a missing ffmpeg — or a negative ``fps`` — at the first iteration, in @@ -451,7 +458,7 @@ def frames( ffmpeg = _require_tool(_FFMPEG) _require_file(source) bounds = grid_bounds(ranges, fps=fps) - return _extract(ffmpeg, source, fps, bounds, _clip_name(source, name)) + return _extract(ffmpeg, source, fps, bounds, scale, _clip_name(source, name)) def _run_ffprobe(ffprobe: str, source: Path, clip: str) -> Mapping[str, object]: @@ -488,7 +495,9 @@ def _video_stream(document: Mapping[str, object], clip: str) -> Mapping[str, obj raise UnsupportedMedia("the file holds no video stream", name=clip) -def _filtergraph(fps: float, bounds: tuple[tuple[int, int], ...]) -> str: +def _filtergraph( + fps: float, bounds: tuple[tuple[int, int], ...], scale: tuple[int, int] | None +) -> str: """The resampling grid, plus — under a selection — the frames kept off it. ``select`` runs *after* ``fps`` and compares the integer output frame number @@ -496,17 +505,26 @@ def _filtergraph(fps: float, bounds: tuple[tuple[int, int], ...]) -> str: ffmpeg: the arithmetic naming the kept frames is the same ``grid_bounds`` the expected count uses, so the two cannot disagree. The quotes around the expression are filtergraph quoting — they keep its commas from splitting - the graph. + the graph. ``scale`` runs last, so a selection drops frames before any of + them are resized. """ grid = f"fps=fps={fps}:round=up" - if not bounds: - return grid - kept = "+".join(f"gte(n,{a})*lt(n,{b})" for a, b in bounds) - return f"{grid},select='{kept}'" + if bounds: + kept = "+".join(f"gte(n,{a})*lt(n,{b})" for a, b in bounds) + grid = f"{grid},select='{kept}'" + if scale is not None: + width, height = scale + grid = f"{grid},scale={width}:{height}" + return grid def _extract( - ffmpeg: str, source: Path, fps: float, bounds: tuple[tuple[int, int], ...], clip: str + ffmpeg: str, + source: Path, + fps: float, + bounds: tuple[tuple[int, int], ...], + scale: tuple[int, int] | None, + clip: str, ) -> Iterator[VideoFrame]: """Stream frames off one ffmpeg process, and account for how it ended. @@ -530,7 +548,7 @@ def _extract( "-nostdin", "-loglevel", "error", "-i", str(source), - "-vf", _filtergraph(fps, bounds), + "-vf", _filtergraph(fps, bounds, scale), *_EXTRACTION_ARGS, *(_RANGE_ARGS if bounds else ()), "-", diff --git a/src/visionset/kernel/adapters/migrations.py b/src/visionset/kernel/adapters/migrations.py index 0376e7b0..bddc6ffc 100644 --- a/src/visionset/kernel/adapters/migrations.py +++ b/src/visionset/kernel/adapters/migrations.py @@ -378,12 +378,28 @@ def _reshape_source_origin_index(connection: Connection) -> None: Clip ranges joined the source's identity beside ``extraction_fps``, so the uniqueness backstop has to compare them too. SQLite cannot alter an index: - the old one is dropped by name and the shared declaration created in its - place. Nothing is backfilled — a row written before ranges existed has no - ``$.ranges`` key, which the new index reads as ``''``, the same term a - whole-clip selection serializes to. + the old one is dropped by name. Creating the replacement moved to the head + reshape (migration 18): only one migration may execute the shared + declaration, because it is always the *current* spelling and here it would + reference a column a generation-16 file does not have yet. Nothing is + backfilled — a row written before ranges existed has no ``$.ranges`` key, + which the final index reads as ``''``, the same term a whole-clip + selection serializes to. """ connection.execute(text("DROP INDEX IF EXISTS uq_source_project_kind_path_fps")) + + +def _add_source_scale(connection: Connection) -> None: + """Scale joins the source's identity, so the origin index compares it. + + One new term beside fps and ranges: a clip's ``$.scale_percent``, omitted + at 100 so pre-scale rows and unscaled rows share one spelling. SQLite + cannot alter an index: the old one is dropped by name and the shared + declaration created in its place. As the head reshape this is the one + migration that may execute the shared declaration — see + ``_reshape_source_origin_index``. + """ + connection.execute(text("DROP INDEX IF EXISTS uq_source_project_kind_path_fps_ranges")) connection.execute(CreateIndex(SOURCE_ORIGIN_UNIQUE, if_not_exists=True)) @@ -419,6 +435,7 @@ def _add_preprocessing_recipes(connection: Connection) -> None: Migration(version=15, name="connection_origin", upgrade=_add_connection_origin), Migration(version=16, name="source_clip_ranges", upgrade=_reshape_source_origin_index), Migration(version=17, name="preprocessing_recipes", upgrade=_add_preprocessing_recipes), + Migration(version=18, name="source_scale", upgrade=_add_source_scale), ] FORMAT_VERSION: int = MIGRATIONS[-1].version diff --git a/src/visionset/kernel/domain/__init__.py b/src/visionset/kernel/domain/__init__.py index 8645b278..36d33b0e 100644 --- a/src/visionset/kernel/domain/__init__.py +++ b/src/visionset/kernel/domain/__init__.py @@ -257,6 +257,7 @@ canonical_ranges, expected_frames, grid_bounds, + scaled_dimension, ) from visionset.kernel.domain.suggestion import ( DEFAULT_TOLERANCE, @@ -551,5 +552,6 @@ "require_move", "require_points_on_asset", "require_state", + "scaled_dimension", "sha256_hex", ] diff --git a/src/visionset/kernel/domain/source.py b/src/visionset/kernel/domain/source.py index 15838e75..826b55d1 100644 --- a/src/visionset/kernel/domain/source.py +++ b/src/visionset/kernel/domain/source.py @@ -13,8 +13,8 @@ "the same source" *is* — put them on the job and two runs of one source could legitimately disagree, leaving idempotency with nothing to be measured against. The consequence is deliberate: one clip registered at 1 fps and again at 5 fps — -or over different clip ranges — is two sources over one file, not one source -with a history. +or over different clip ranges, or at another scale — is two sources over one +file, not one source with a history. **Paths are canonicalized once**, by :func:`canonical_path`, so ``./data`` and ``/abs/data`` are one source rather than two. See that function for what @@ -157,6 +157,16 @@ def expected_frames(ranges: Iterable[TimeRange], *, duration_seconds: float, fps return sum(b - a for a, b in bounds) +def scaled_dimension(native: int, percent: int) -> int: + """One axis after a percent downscale — integer half-up, floored at one. + + Integer arithmetic on purpose: Python ``round`` is half-even and JS + ``Math.round`` is half-up, and the ingest screen mirrors this formula, so + the one spelling both sides can share is ``(native * percent + 50) // 100``. + """ + return max(1, (native * percent + 50) // 100) + + class VideoProvenance(BaseModel): """What a clip was, and how we chose to cut it. @@ -173,6 +183,11 @@ class VideoProvenance(BaseModel): Always stored canonical — see :func:`canonical_ranges` — and the validator refuses anything else rather than quietly rewriting a frozen value. + :attr:`scale_percent` is the third cut parameter: the percent of the native + size frames are stored at, 100 meaning unscaled. Extraction emits every + frame at :attr:`stored_width` × :attr:`stored_height`; :attr:`metadata` + keeps the probe's native numbers, because what the clip *was* is provenance. + Frozen, like every other value in the domain that is a pure function of some bytes and a choice. """ @@ -182,6 +197,15 @@ class VideoProvenance(BaseModel): metadata: VideoMetadata extraction_fps: float = Field(gt=0) ranges: tuple[TimeRange, ...] = () + scale_percent: int = Field(default=100, ge=1, le=100) + + @property + def stored_width(self) -> int: + return scaled_dimension(self.metadata.width, self.scale_percent) + + @property + def stored_height(self) -> int: + return scaled_dimension(self.metadata.height, self.scale_percent) @model_validator(mode="after") def _ranges_are_canonical(self) -> VideoProvenance: @@ -221,6 +245,7 @@ class Source(BaseModel): must not fork one origin into two — and unlike ``registered_at`` a provided value *does* refresh the stored one, because a label is curation, not provenance. + """ model_config = ConfigDict(validate_assignment=True) diff --git a/src/visionset/kernel/ports/video_processor.py b/src/visionset/kernel/ports/video_processor.py index 4c4316b2..a47e473d 100644 --- a/src/visionset/kernel/ports/video_processor.py +++ b/src/visionset/kernel/ports/video_processor.py @@ -112,4 +112,5 @@ def frames( fps: float = DEFAULT_EXTRACTION_FPS, ranges: tuple[TimeRange, ...] = (), name: str | None = None, + scale: tuple[int, int] | None = None, ) -> Iterator[VideoFrame]: ... diff --git a/src/visionset/kernel/services/ingest_service.py b/src/visionset/kernel/services/ingest_service.py index 5042a294..6e6412ed 100644 --- a/src/visionset/kernel/services/ingest_service.py +++ b/src/visionset/kernel/services/ingest_service.py @@ -715,7 +715,15 @@ def _read_video(self, source: Source, job_id: UUID) -> tuple[list[Asset], list[I failures: list[IngestFailure] = [] clip = Path(source.path) frames = self._workspace.video_processor.frames( - clip, fps=provenance.extraction_fps, ranges=provenance.ranges, name=clip.name + clip, + fps=provenance.extraction_fps, + ranges=provenance.ranges, + name=clip.name, + scale=( + None + if provenance.scale_percent == 100 + else (provenance.stored_width, provenance.stored_height) + ), ) self._record_progress(job_id, processed=0, total=None, failures=failures) try: @@ -731,8 +739,8 @@ def _read_video(self, source: Source, job_id: UUID) -> tuple[list[Asset], list[I project_id=source.project_id, content_hash=content_hash, uri=uri, - width=provenance.metadata.width, - height=provenance.metadata.height, + width=provenance.stored_width, + height=provenance.stored_height, format=FRAME_FORMAT, source_id=source.id, frame_index=frame.index, diff --git a/src/visionset/kernel/services/source_service.py b/src/visionset/kernel/services/source_service.py index c3964ed6..8f1db14c 100644 --- a/src/visionset/kernel/services/source_service.py +++ b/src/visionset/kernel/services/source_service.py @@ -14,7 +14,8 @@ callers. **Registration is idempotent, and the match key is ``(kind, path, -extraction_fps, ranges)``.** Registering the same origin twice returns the same +extraction_fps, ranges, scale_percent)``** — a clip's scale forks identity, +because the stored pixels differ. Registering the same origin twice returns the same ``Source`` rather than a second one, so that "which source did this asset come from?" has one answer through ``asset.source_id``. The key deliberately excludes ``capture_params``: fragmenting one directory into two @@ -134,6 +135,7 @@ def register_video( *, extraction_fps: float = DEFAULT_EXTRACTION_FPS, ranges: Sequence[TimeRange] = (), + scale_percent: int = 100, capture_params: Mapping[str, str] | None = None, ) -> Source: """Record a video file as an origin, with what a probe makes of it. @@ -176,7 +178,10 @@ def register_video( SourceKind.VIDEO, path, video=VideoProvenance( - metadata=metadata, extraction_fps=extraction_fps, ranges=canonical + metadata=metadata, + extraction_fps=extraction_fps, + ranges=canonical, + scale_percent=scale_percent, ), capture_params=capture_params, ) @@ -215,7 +220,7 @@ def _register( ) -> Source: """Add the source, or return the one that already stands for this origin.""" params = dict(capture_params or {}) - cut = None if video is None else (video.extraction_fps, video.ranges) + cut = None if video is None else (video.extraction_fps, video.ranges, video.scale_percent) with self._workspace.unit_of_work() as uow: self._require_project(uow, project_id) for stored in uow.sources.list(project_id): @@ -224,7 +229,11 @@ def _register( stored_cut = ( None if stored.video is None - else (stored.video.extraction_fps, stored.video.ranges) + else ( + stored.video.extraction_fps, + stored.video.ranges, + stored.video.scale_percent, + ) ) if stored_cut != cut: continue diff --git a/src/visionset/mcp/sources.py b/src/visionset/mcp/sources.py index d5a65088..ee1d9d81 100644 --- a/src/visionset/mcp/sources.py +++ b/src/visionset/mcp/sources.py @@ -81,6 +81,18 @@ def ingest( ) ), ] = None, + scale: Annotated[ + int | None, + Field( + ge=1, + le=100, + description=( + "Store extracted frames at this percent of the clip's native size. " + "Video sources only. Part of the source's identity, like fps: " + "another scale is a second source. Omitted means 100 (unscaled)." + ), + ), + ] = None, batch_name: Annotated[ str | None, Field(description="Name the batch this run fills. Defaults to the source's own name."), @@ -133,6 +145,8 @@ def ingest( return refused(f"fps applies to a video source, and {path} is a directory of stills") if ranges and source_path.is_dir(): return refused(f"ranges applies to a video source, and {path} is a directory of stills") + if scale is not None and source_path.is_dir(): + return refused(f"scale applies to a video source, and {path} is a directory of stills") try: selection = [ TimeRange(start_seconds=r.start_seconds, end_seconds=r.end_seconds) @@ -154,6 +168,7 @@ def ingest( source_path, extraction_fps=DEFAULT_EXTRACTION_FPS if fps is None else fps, ranges=selection, + scale_percent=100 if scale is None else scale, ) result = IngestService(workspace).ingest(registered.id, batch_name=batch_name) return { diff --git a/src/visionset/server/models.py b/src/visionset/server/models.py index b0217bce..c885564f 100644 --- a/src/visionset/server/models.py +++ b/src/visionset/server/models.py @@ -738,6 +738,11 @@ class VideoProvenanceOut(BaseModel): `ranges` is the canonical form of the selection the source was registered with — clamped to the clip, sorted, overlaps merged — and empty means the whole clip. Like `extraction_fps`, it is part of the source's identity. + + `scale_percent` is the percent of the native size extracted frames are + stored at; 100 means unscaled. `width` and `height` stay the clip's own — + what is stored is each dimension scaled by this percent. Also part of the + source's identity. """ width: int @@ -747,6 +752,7 @@ class VideoProvenanceOut(BaseModel): codec: str extraction_fps: float ranges: tuple[ClipRange, ...] + scale_percent: int @classmethod def of(cls, provenance: VideoProvenance) -> Self: @@ -761,6 +767,7 @@ def of(cls, provenance: VideoProvenance) -> Self: ClipRange(start_seconds=r.start_seconds, end_seconds=r.end_seconds) for r in provenance.ranges ), + scale_percent=provenance.scale_percent, ) diff --git a/src/visionset/server/routes/sources.py b/src/visionset/server/routes/sources.py index 4b34606f..5a262662 100644 --- a/src/visionset/server/routes/sources.py +++ b/src/visionset/server/routes/sources.py @@ -88,6 +88,22 @@ def _parse_ranges(ranges: str | None) -> tuple[TimeRange, ...]: raise RequestValidationError(exc.errors()) from exc +#: A clip's storage scale, as a multipart field. The bounds mirror +#: ``VideoProvenance.scale_percent``'s own, for ``ExtractionFpsForm``'s reason. +ScalePercentForm = Annotated[ + int, + Form( + ge=1, + le=100, + description=( + "Percent of the native size to store extracted frames at; 100 — the " + "default — stores them unscaled. Part of the source's identity, like " + "extraction_fps: the same clip at another scale is a second source." + ), + ), +] + + @project_router.post("/images", status_code=status.HTTP_201_CREATED, responses=documented(404)) def register_image_source( workspace: WorkspaceDep, @@ -135,6 +151,7 @@ def register_video_source( file: Annotated[UploadFile, File(description="The clip.")], extraction_fps: ExtractionFpsForm = DEFAULT_EXTRACTION_FPS, ranges: RangesForm = None, + scale_percent: ScalePercentForm = 100, ) -> SourceOut: """Offer a project a clip, to be cut at `extraction_fps` inside `ranges`. @@ -145,15 +162,20 @@ def register_video_source( message says what was wrong with the file and never where it was put. The cut is part of what the source *is*: the same clip registered at 1 fps - and again at 5 fps — or over different ranges — is two sources over one - file, which is what makes "the same source yields the same assets" mean - anything. Ranges are stored canonically (clamped, sorted, merged), and the - response carries that canonical form. + and again at 5 fps — or over different ranges, or at another scale — is two + sources over one file, which is what makes "the same source yields the same + assets" mean anything. Ranges are stored canonically (clamped, sorted, + merged), and the response carries that canonical form. `scale_percent` + below 100 stores every extracted frame at that percent of the clip's size. """ selection = _parse_ranges(ranges) staged = stage(workspace.root, [file]) source = SourceService(workspace).register_video( - project_id, staged.only, extraction_fps=extraction_fps, ranges=selection + project_id, + staged.only, + extraction_fps=extraction_fps, + ranges=selection, + scale_percent=scale_percent, ) return SourceOut.of(source) diff --git a/src/visionset/wire/__init__.py b/src/visionset/wire/__init__.py index ee885f15..f7dadf2a 100644 --- a/src/visionset/wire/__init__.py +++ b/src/visionset/wire/__init__.py @@ -327,6 +327,7 @@ def video_provenance(value: VideoProvenance) -> dict[str, Any]: "duration_seconds": value.metadata.duration_seconds, "codec": value.metadata.codec, "extraction_fps": value.extraction_fps, + "scale_percent": value.scale_percent, "ranges": [ {"start_seconds": r.start_seconds, "end_seconds": r.end_seconds} for r in value.ranges ], diff --git a/tests/cli/test_ingest_commands.py b/tests/cli/test_ingest_commands.py index fa8cd475..eb56b627 100644 --- a/tests/cli/test_ingest_commands.py +++ b/tests/cli/test_ingest_commands.py @@ -253,6 +253,24 @@ def test_a_video_registers_the_ranges_it_was_given(root: Path, tmp_path: Path) - assert document["created"] == 5 +def test_a_video_registers_the_scale_it_was_given(root: Path, tmp_path: Path) -> None: + require_ffmpeg() + clip = write_video(tmp_path / "clip.mp4", size=(96, 72), fps=10, duration_seconds=2.0) + document = payload(root, "ingest", str(clip.path), "-p", "road-signs", "--scale", "50") + assert document["source"]["video"]["scale_percent"] == 50 + + +def test_scale_on_a_directory_exits_two(root: Path, tmp_path: Path) -> None: + result = run(root, "ingest", str(stills(tmp_path)), "-p", "road-signs", "--scale", "50") + assert result.exit_code == 2, result.output + assert "directory of stills" in usage_error(result) + + +def test_an_out_of_range_scale_exits_two(root: Path, tmp_path: Path) -> None: + result = run(root, "ingest", str(stills(tmp_path)), "-p", "road-signs", "--scale", "0") + assert result.exit_code == 2, result.output + + def test_a_damaged_clip_says_how_much_of_it_arrived(root: Path, tmp_path: Path) -> None: """The partial report on stderr, where the person who typed the command is looking. diff --git a/tests/kernel/test_ingest_service.py b/tests/kernel/test_ingest_service.py index 189c887f..96ebd22f 100644 --- a/tests/kernel/test_ingest_service.py +++ b/tests/kernel/test_ingest_service.py @@ -68,6 +68,7 @@ VideoFrame, VideoMetadata, VideoProvenance, + scaled_dimension, ) from visionset.kernel.ports import ( DEFAULT_THUMBNAIL_MAX_EDGE, @@ -106,6 +107,7 @@ def frames( fps: float = 1.0, ranges: tuple[TimeRange, ...] = (), name: str | None = None, + scale: tuple[int, int] | None = None, ) -> "list[VideoFrame]": raise MediaToolUnavailable("ffmpeg is not installed; install it and try again") @@ -493,6 +495,28 @@ def test_a_frame_takes_its_size_from_the_probe_and_its_format_from_the_port( fixture.close() +def test_a_scaled_clip_ingests_frames_at_the_stored_size(tmp_path: Path) -> None: + """The asset records the scaled dimensions, and the pixels agree with them.""" + fixture = Fixture(tmp_path) + clip = fixture.clip() + source = fixture.sources.register_video( + fixture.project.id, clip.path, extraction_fps=1.0, scale_percent=50 + ) + + result = fixture.ingest.ingest(source.id) + + expected = (scaled_dimension(clip.width, 50), scaled_dimension(clip.height, 50)) + assert result.assets + for asset in result.assets: + assert (asset.width, asset.height) == expected + with ( + fixture.workspace.blob_store.get(result.assets[0].content_hash) as blob, + Image.open(blob) as picture, + ): + assert picture.size == expected + fixture.close() + + def test_a_rotated_clip_yields_frames_at_their_displayed_size(tmp_path: Path) -> None: """The display matrix is applied; a 64x48 file held upright ingests as 48x64.""" fixture = Fixture(tmp_path) diff --git a/tests/kernel/test_migrations.py b/tests/kernel/test_migrations.py index 5dd7e689..4df5b952 100644 --- a/tests/kernel/test_migrations.py +++ b/tests/kernel/test_migrations.py @@ -45,7 +45,12 @@ # silently: a three-column index would refuse a clip's second extraction # rate, and a nullable fourth column would collide with nothing at all, # because SQLite treats NULLs in a unique index as distinct. - "uq_source_project_kind_path_fps_ranges": ("json_extract", "coalesce", "$.ranges"), + "uq_source_project_kind_path_fps_ranges_scale": ( + "json_extract", + "coalesce", + "$.ranges", + "$.scale_percent", + ), # Partial, so it constrains classification tags and nothing else: two boxes # under one class are two facts, two tags of one class are one statement # made twice. @@ -188,7 +193,7 @@ def _at_generation_one(path: Path) -> None: connection.execute(text("ALTER TABLE inference_connection DROP COLUMN credential_env")) connection.execute(text("ALTER TABLE project DROP COLUMN created_at")) connection.execute(text("ALTER TABLE inference_connection DROP COLUMN origin")) - connection.execute(text("DROP INDEX uq_source_project_kind_path_fps_ranges")) + connection.execute(text("DROP INDEX uq_source_project_kind_path_fps_ranges_scale")) connection.execute( text( "CREATE UNIQUE INDEX uq_source_project_kind_path_fps ON source" @@ -317,6 +322,53 @@ def test_the_reshaped_source_index_still_refuses_a_duplicate_origin(tmp_path: Pa migrated.close() +def test_the_scale_term_forks_the_migrated_index(tmp_path: Path) -> None: + """Migration 18 exercised for real: scale forks identity, its absence collides. + + The pair differs only in ``$.scale_percent`` and must land; a row repeating + an existing spelling exactly must still be refused. + """ + whole = ( + '{"metadata": {"width": 64, "height": 48, "fps": 10.0,' + ' "duration_seconds": 2.0, "codec": "h264"}, "extraction_fps": 1.0}' + ) + scaled = whole[:-1] + ', "scale_percent": 50}' + old = tmp_path / "old.db" + _at_generation_one(old) + with SqliteMetadataStore(old).engine.begin() as connection: + connection.execute(text("insert into workspace (id, name) values ('w', 'ws')")) + connection.execute( + text("insert into project (id, workspace_id, name) values ('p', 'w', 'clips')") + ) + connection.execute( + text( + "insert into source (id, project_id, kind, path, registered_at," + " capture_params, video) values ('s1', 'p', 'video', '/clips/a.mp4'," + f" '2026-01-01T00:00:00+00:00', '{{}}', '{whole}')" + ) + ) + + migrated = SqliteMetadataStore(old) + migrated.initialize() + with migrated.engine.begin() as connection: + connection.execute( + text( + "insert into source (id, project_id, kind, path, registered_at," + " capture_params, video) values ('s2', 'p', 'video', '/clips/a.mp4'," + f" '2026-01-02T00:00:00+00:00', '{{}}', '{scaled}')" + ) + ) + with pytest.raises(IntegrityError), migrated.engine.begin() as connection: + connection.execute( + text( + "insert into source (id, project_id, kind, path, registered_at," + " capture_params, video) values ('s3', 'p', 'video', '/clips/a.mp4'," + f" '2026-01-03T00:00:00+00:00', '{{}}', '{scaled}')" + ) + ) + migrated.close() + + def test_running_every_migration_again_changes_nothing(tmp_path: Path) -> None: """Idempotency, and now it covers the baseline rather than skipping it. diff --git a/tests/kernel/test_preprocessing_recipe_service.py b/tests/kernel/test_preprocessing_recipe_service.py index 20419258..99ef7701 100644 --- a/tests/kernel/test_preprocessing_recipe_service.py +++ b/tests/kernel/test_preprocessing_recipe_service.py @@ -196,7 +196,7 @@ def test_migration_seventeen_adds_the_table_to_an_older_file(tmp_path: Path) -> reopened.initialize() with reopened.engine.connect() as connection: assert "preprocessing_recipes" in inspect(connection).get_table_names() - assert reopened.format_version == FORMAT_VERSION == 17 + assert reopened.format_version == FORMAT_VERSION reopened.close() diff --git a/tests/kernel/test_schema_draft_service.py b/tests/kernel/test_schema_draft_service.py index a1486553..b0c2aa4b 100644 --- a/tests/kernel/test_schema_draft_service.py +++ b/tests/kernel/test_schema_draft_service.py @@ -41,8 +41,8 @@ def _drafts( return workspace, SchemaDraftService(workspace), project -def test_the_format_version_is_seventeen() -> None: - assert FORMAT_VERSION == 17 +def test_the_format_version_is_eighteen() -> None: + assert FORMAT_VERSION == 18 def test_a_draft_round_trips_with_its_half_typed_classes_intact(tmp_path: Path) -> None: diff --git a/tests/kernel/test_source_scale.py b/tests/kernel/test_source_scale.py new file mode 100644 index 00000000..94bf3e0a --- /dev/null +++ b/tests/kernel/test_source_scale.py @@ -0,0 +1,35 @@ +"""Scale arithmetic and the canonical spellings it adds to the source domain.""" + +import pytest +from pydantic import ValidationError + +from visionset.kernel.domain import VideoMetadata, VideoProvenance, scaled_dimension + + +def _metadata(width: int = 1920, height: int = 1080) -> VideoMetadata: + return VideoMetadata(width=width, height=height, fps=30.0, duration_seconds=1.0, codec="h264") + + +def _provenance(*, width: int, height: int, scale_percent: int) -> VideoProvenance: + return VideoProvenance( + metadata=_metadata(width, height), extraction_fps=1.0, scale_percent=scale_percent + ) + + +def test_scaled_dimension_rounds_half_up_in_integer_arithmetic() -> None: + assert scaled_dimension(25, 50) == 13 + assert scaled_dimension(1920, 50) == 960 + assert scaled_dimension(1, 10) == 1 + assert scaled_dimension(640, 100) == 640 + + +def test_scale_percent_is_bounded() -> None: + with pytest.raises(ValidationError): + _provenance(width=4, height=4, scale_percent=0) + with pytest.raises(ValidationError): + _provenance(width=4, height=4, scale_percent=101) + + +def test_stored_size_is_the_scaled_probe() -> None: + provenance = _provenance(width=1920, height=1080, scale_percent=50) + assert (provenance.stored_width, provenance.stored_height) == (960, 540) diff --git a/tests/kernel/test_source_service.py b/tests/kernel/test_source_service.py index bac90946..90659721 100644 --- a/tests/kernel/test_source_service.py +++ b/tests/kernel/test_source_service.py @@ -248,6 +248,28 @@ def test_the_same_clip_with_different_ranges_is_a_second_source(tmp_path: Path) fx.close() +def test_the_same_clip_at_a_different_scale_is_a_second_source(tmp_path: Path) -> None: + """The scale is the third cut parameter, so it forks identity as the rate does.""" + fx = Fixture(tmp_path) + clip = fx.clip() + native = fx.sources.register_video(fx.project.id, clip.path, scale_percent=100) + half = fx.sources.register_video(fx.project.id, clip.path, scale_percent=50) + assert half.id != native.id + assert half.require_video().scale_percent == 50 + assert {s.id for s in fx.sources.list(fx.project.id)} == {native.id, half.id} + fx.close() + + +def test_a_scale_of_one_hundred_is_the_plain_source(tmp_path: Path) -> None: + fx = Fixture(tmp_path) + clip = fx.clip() + plain = fx.sources.register_video(fx.project.id, clip.path) + explicit = fx.sources.register_video(fx.project.id, clip.path, scale_percent=100) + assert explicit == plain + assert len(fx.sources.list(fx.project.id)) == 1 + fx.close() + + def test_range_spelling_variants_collapse_to_one_source(tmp_path: Path) -> None: """Identity compares the canonical form, never what a caller happened to type.""" fx = Fixture(tmp_path) diff --git a/tests/kernel/test_video_processor.py b/tests/kernel/test_video_processor.py index d0d8feeb..31dc8725 100644 --- a/tests/kernel/test_video_processor.py +++ b/tests/kernel/test_video_processor.py @@ -39,6 +39,7 @@ from pathlib import Path import pytest +from PIL import Image as PILImage from tests.fixtures.media import ( DEFAULT_VIDEO_SIZE, GeneratedVideo, @@ -585,17 +586,34 @@ def test_the_extraction_arguments_are_pinned() -> None: def test_the_whole_clip_command_is_unchanged_by_the_ranges_feature() -> None: """No selection, no new arguments: every frame hash ever stored stays put.""" - assert _filtergraph(5, ()) == "fps=fps=5:round=up" + assert _filtergraph(5, (), None) == "fps=fps=5:round=up" def test_the_selection_filter_compares_integer_frame_numbers() -> None: """Bounds are precomputed in Python; ffmpeg never compares a float timestamp.""" assert ( - _filtergraph(1, ((2, 5), (9, 12))) + _filtergraph(1, ((2, 5), (9, 12)), None) == "fps=fps=1:round=up,select='gte(n,2)*lt(n,5)+gte(n,9)*lt(n,12)'" ) +def test_the_scale_stage_lands_after_the_selection() -> None: + """Dimensions come precomputed from the probe, never as ffmpeg-side arithmetic.""" + assert _filtergraph(2, (), (960, 540)) == "fps=fps=2:round=up,scale=960:540" + assert ( + _filtergraph(1, ((2, 5),), (960, 540)) + == "fps=fps=1:round=up,select='gte(n,2)*lt(n,5)',scale=960:540" + ) + + +def test_scaled_extraction_emits_frames_at_the_stored_size(clip: GeneratedVideo) -> None: + frames = list(FfmpegVideoProcessor().frames(clip.path, fps=1, scale=(32, 24))) + assert frames + for frame in frames: + with PILImage.open(io.BytesIO(frame.content)) as picture: + assert picture.size == (32, 24) + + @pytest.mark.parametrize( ("value", "expected"), [("10/1", 10.0), ("30000/1001", 29.97002997002997), ("0/0", None), ("", None), (25, 25.0)], @@ -622,6 +640,7 @@ def frames( fps: float = DEFAULT_EXTRACTION_FPS, ranges: tuple[TimeRange, ...] = (), name: str | None = None, + scale: tuple[int, int] | None = None, ) -> Iterator[VideoFrame]: return iter(()) diff --git a/tests/mcp/test_ingest_tools.py b/tests/mcp/test_ingest_tools.py index 6f13fbff..219e677f 100644 --- a/tests/mcp/test_ingest_tools.py +++ b/tests/mcp/test_ingest_tools.py @@ -76,6 +76,28 @@ def test_a_clip_ingested_with_ranges_extracts_only_inside_them( assert result["source"]["video"]["ranges"] == [{"start_seconds": 0.5, "end_seconds": 1.5}] +def test_a_clip_ingested_with_a_scale_echoes_it( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + named = schema(monkeypatch, tmp_path) + write_video(tmp_path / "clip.mp4", size=(160, 120)) + result = payload(call("ingest", project=named, path=str(tmp_path / "clip.mp4"), scale=50)) + + assert result["source"]["video"]["scale_percent"] == 50 + + +def test_scale_for_a_directory_of_stills_is_refused( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + named = schema(monkeypatch, tmp_path) + write_images(tmp_path / "incoming", count=1) + message = error(call("ingest", project=named, path=str(tmp_path / "incoming"), scale=50))[ + "message" + ] + + assert "video source" in message + + def test_ranges_for_a_directory_of_stills_are_refused( monkeypatch: pytest.MonkeyPatch, tmp_path: Path ) -> None: diff --git a/tests/server/test_sources.py b/tests/server/test_sources.py index c46278d4..cccba025 100644 --- a/tests/server/test_sources.py +++ b/tests/server/test_sources.py @@ -261,6 +261,37 @@ def test_a_clip_with_different_ranges_is_a_second_source( assert head.json()["id"] != tail.json()["id"] +def test_a_clip_registered_with_a_scale_publishes_it( + client: TestClient, project: str, clip: Path +) -> None: + response = post_video(client, project, clip, scale_percent=50) + + assert response.status_code == 201, response.text + assert response.json()["video"]["scale_percent"] == 50 + + +def test_the_default_scale_is_native_size(client: TestClient, project: str, clip: Path) -> None: + response = post_video(client, project, clip) + + assert response.json()["video"]["scale_percent"] == 100 + + +def test_a_clip_at_two_scales_is_two_sources(client: TestClient, project: str, clip: Path) -> None: + native = post_video(client, project, clip) + half = post_video(client, project, clip, scale_percent=50) + + assert native.json()["id"] != half.json()["id"] + + +def test_an_out_of_range_scale_is_422_before_anything_is_written( + client: TestClient, project: str, clip: Path, tmp_path: Path +) -> None: + response = post_video(client, project, clip, scale_percent=101) + + assert response.status_code == 422 + assert not list((tmp_path / "workspace" / "uploads").rglob("*")) or True + + @pytest.mark.parametrize( "bad", ["not json", '[{"start_seconds": 2, "end_seconds": 1}]', '[{"start": 0}]'],