Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/content/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ POST /projects/{project_id}/preprocessing-recipes
GET /projects/{project_id}/preprocessing-recipes/{name}
PUT /projects/{project_id}/preprocessing-recipes/{name} whole-value; a rename is `name` in the body
DELETE /projects/{project_id}/preprocessing-recipes/{name} no confirmation gate
POST /projects/{project_id}/preprocessing-preview one asset through a spec; never cached
POST /projects/{project_id}/preprocessing-preview one asset through a spec, seeded or `showcase`; never cached

GET /inference/connections
POST /inference/connections
Expand Down
5 changes: 4 additions & 1 deletion docs/content/architecture/backend/preprocessing-drivers.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,10 @@ re-encoding, and a resize reads neither argument because it is deterministic.
`visionset.kernel.domain`; `hflip` draws nothing and always mirrors. A driver that read the seed
any other way, or drew from anywhere else, would put its pixels where the geometry transform did
not put the labels. The built-in `rot90` turns counter-clockwise because the kernel's
`_rotated_once` does, and a driver applying that op must turn the same way.
`_rotated_once` does, and a driver applying that op must turn the same way. A preview in
`showcase` mode passes `SHOWCASE_SEED` where a variant seed goes, and the two helpers answer it
with the step at its declared strength - which is why a driver reads them rather than the seed's
bytes, and why the port needs no second argument for it.

The step grammar is closed. Which kinds exist, and which augmentation ops, is decided by the
kernel's `Step` union, so a driver applies one of the kinds the kernel already names and cannot
Expand Down
10 changes: 9 additions & 1 deletion docs/content/preprocessing.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,8 @@ position of that digest. `hflip` draws nothing and always mirrors, for the reaso
draws zero turns: a variant that came out identical to its source would be the base image under a
variant's name. The same recipe over the same bytes
therefore draws the same variant on any machine, and the geometry arithmetic is exact
everywhere.
everywhere. This is the export's path; the preview's `showcase` mode, described below, is the
one place a draw is fixed rather than seeded.

**Byte stability is promised within one environment only.** The pixels a resize or an
enhancement produces depend on the codec and resampling code that produced them, so two
Expand Down Expand Up @@ -167,6 +168,13 @@ manifest with the asset in the train fold - so every variant a spec declares can
whether or not a release exists. The image is capped to 512 pixels on its longer side, labels
scaled to match, and the response is never cached: the spec is the request's own.

The body also takes `showcase`, false by default. With it true the variant's draws are fixed at
each step's declared strength - `hflip` mirrors, `rot90` makes one quarter turn, brightness and
contrast use the full `amount` - so the picture shows what a step *does* rather than one seeded
draw of it. That is a preview's concern only: an export always takes the seeded path above, and
the kernel spells the fixed draws as one sentinel seed (`SHOWCASE_SEED`) that both draw functions
recognise, so the pixel driver and the geometry transform still agree through the same code.

## Refusals

| Error | When |
Expand Down
15 changes: 9 additions & 6 deletions docs/content/ui.md
Original file line number Diff line number Diff line change
Expand Up @@ -267,12 +267,15 @@ it and the footer carries the first one beside a shut *Save recipe*. Those are t
shape, not a kernel rule the client is mirroring: the server still answers 422 to a body the
form did not build.

*Preview* renders three sample assets through `POST /projects/{id}/preprocessing-preview`,
the export's own kernel path over a one-asset manifest: the first three train-fold members
of the newest release with a split — variants are the train fold's — or the project's first
three assets when no release has one. Three columns: the asset as it is, after the resize
step alone, and the first augmented variant; a stage the recipe does not have says so
(*No resize step*) rather than repeating the original. Each cell is one request keyed on
*Preview* renders one sample asset through `POST /projects/{id}/preprocessing-preview`,
the export's own kernel path over a one-asset manifest: the first train-fold member of the
newest release with a split — variants are the train fold's — or the project's first asset
when no release has one. Three columns: the asset as it is, after the resize step alone, and
the last augmentation step ticked, headed by its name (*After horizontal flip*) and asked for
in `showcase` mode — the draws fixed at the step's declared strength, so the cell shows what
the step does rather than one seeded draw of it; an export still takes the seeded path. A
stage the recipe does not have says so (*No resize step*) rather than repeating the
original. Each cell is one request keyed on
the spec it renders, and the spec settles for 400 ms before a cell asks, so typing `640`
does not pay for `6` and `64`. The cell is the member dialog's own picture-with-labels
mechanism (`patterns/StaticAnnotationOverlay.tsx`): the rendered image, and the response's
Expand Down
5 changes: 3 additions & 2 deletions frontend/app/cycle/cycle.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1569,8 +1569,9 @@ test("the whole cycle, from opening the app to a downloaded export", async ({ pa
await expect(page.getByTestId("augment-variants")).toHaveValue("1");
await expect(page.getByTestId("recipe-step-augment")).toHaveAttribute("data-state", "complete");

// The preview: three cells of the first row, each one a real render of a
// frame this walk ingested — the original, the letterbox, and variant 1.
// The preview: three cells of the one row, each one a real render of a
// frame this walk ingested — the original, the letterbox, and the flip
// shown at full strength.
// Generous, because the preview debounces the draft and then goes through
// Pillow three times.
for (const cell of ["original", "resize", "augment"]) {
Expand Down
14 changes: 13 additions & 1 deletion frontend/ui-core/src/generated/api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2189,7 +2189,9 @@ export interface paths {
* train fold, so every variant the spec declares can be seen whether or not a
* release exists. The image is capped to 512 pixels on its longer side, with
* the annotations scaled to match, and comes back base64-encoded beside its
* `media_type`. Never cached: the spec is the request's own.
* `media_type`. Never cached: the spec is the request's own. With `showcase`
* true the variant's draws are fixed at each step's declared strength, so
* the picture shows what a step does rather than one seeded draw of it.
*
* An unknown project is 404 `PROJECT_NOT_FOUND` and an asset outside it 404
* `ASSET_NOT_FOUND`. A step that cannot transform a geometry the asset carries
Expand Down Expand Up @@ -4890,13 +4892,23 @@ export interface components {
*
* `variant` 0 is the base image; `1` to `spec.variants_per_asset` are the
* augmented outputs. A variant the spec does not make is refused.
*
* When `showcase` is true the variant's draws are fixed — hflip mirrors,
* rot90 makes one quarter turn, brightness and contrast use the full
* `amount` — so the image shows the step at its declared strength rather
* than one seeded draw; exports never use it.
*/
PreprocessingPreviewBody: {
/**
* Asset Id
* Format: uuid
*/
asset_id: string;
/**
* Showcase
* @default false
*/
showcase: boolean;
spec: components["schemas"]["RecipeSpecBody"];
/**
* Variant
Expand Down
2 changes: 1 addition & 1 deletion frontend/ui-core/src/patterns/RecipeEditor.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -324,7 +324,7 @@ export function RecipeEditor({
testId="recipe-step-preview"
aside={
<span className="text-xs text-muted-foreground" data-testid="preview-aside">
3 sample assets · seeded
1 sample asset
</span>
}
last
Expand Down
138 changes: 81 additions & 57 deletions frontend/ui-core/src/screens/PreprocessingTab.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,14 @@
*
* `POST /projects/{id}/preprocessing-preview` renders one asset through a spec
* on the same kernel path an export takes, so what the cells show is what the
* archive would hold. Three sample assets, and the choice is the release's: the
* first three train-fold members of the newest release with a split, because
* variants are written for the train fold only; without one, the first three
* assets of the project. The columns are the stages — the asset as it is, after
* the resize step alone, and the first augmented variant — each a request
* keyed on the spec it renders, so a keystroke re-renders only what it changed.
* archive would hold. One sample asset, and the choice is the release's: the
* first train-fold member of the newest release with a split, because variants
* are written for the train fold only; without one, the project's first asset.
* The columns are the stages — the asset as it is, after the resize step alone,
* and the last augmentation step chosen, in `showcase` mode: the draws fixed at
* the step's declared strength, so the cell shows what the step does rather
* than one seeded draw of it. Each cell is a request keyed on the spec it
* renders, so a keystroke re-renders only what it changed.
*
* `PreviewCell` hands the rendered image and its placed annotations to the
* static overlay pattern, so a label is drawn where the export would write it.
Expand Down Expand Up @@ -62,11 +64,13 @@ import {
type Release,
} from "./queries";
import {
AUGMENT_OPS,
canonicalSpec,
draftFromSpec,
draftToSpec,
EMPTY_DRAFT,
sameSpec,
type AugmentStepSpec,
type RecipeDraft,
type RecipeSpec,
} from "./recipeDraft";
Expand Down Expand Up @@ -366,21 +370,25 @@ function Editor({
);
}

/** The asset ids the preview renders: the train fold's first three, or the project's. */
function useSampleAssets(projectId: string, datasetId: string | undefined): readonly string[] | undefined {
/**
* The asset the preview renders: the train fold's first member, or the
* project's first asset; `null` when the project has none, `undefined` while
* the choice is still being read.
*/
function useSampleAsset(projectId: string, datasetId: string | undefined): string | null | undefined {
const releases = useReleases(datasetId);
const newest = newestRelease(releases.data?.items);
const withSplit = newest !== undefined && newest.split !== null && newest.split !== undefined;
const assignment = useReleaseAssignment(withSplit ? newest.id : undefined);
const project = useProjectAssets(projectId, 3);
const project = useProjectAssets(projectId, 1);
if (datasetId !== undefined && releases.data === undefined && !releases.isError) return undefined;
if (withSplit) {
if (assignment.data === undefined && !assignment.isError) return undefined;
const train = assignment.data?.train.slice(0, 3) ?? [];
if (train.length > 0) return train;
const train = assignment.data?.train[0];
if (train !== undefined) return train;
}
if (project.data === undefined) return undefined;
return project.data.items.slice(0, 3).map((asset) => asset.id);
return project.data.items[0]?.id ?? null;
}

function newestRelease(items: readonly Release[] | undefined): Release | undefined {
Expand Down Expand Up @@ -416,17 +424,20 @@ function PreviewGrid({
readonly classes: readonly LabelClass[] | undefined;
readonly onReady: (ready: boolean) => void;
}): JSX.Element {
const samples = useSampleAssets(projectId, datasetId);
const sample = useSampleAsset(projectId, datasetId);
const settled = useSettledSpec(spec);
const resizeSteps = settled === null ? [] : settled.steps.filter((step) => step.kind === "resize");
const resizeOnly: RecipeSpec | null =
settled === null
settled === null ? null : { target: settled.target ?? null, steps: resizeSteps, variants_per_asset: 0 };
const hasResize = resizeSteps.length > 0;
const lastAugment = settled === null ? undefined : lastAugmentStep(settled);
const showcase: RecipeSpec | null =
settled === null || lastAugment === undefined
? null
: { target: settled.target ?? null, steps: settled.steps.filter((step) => step.kind === "resize"), variants_per_asset: 0 };
const hasResize = resizeOnly !== null && resizeOnly.steps.length > 0;
const hasAugment = settled !== null && settled.variants_per_asset > 0;
: { target: settled.target ?? null, steps: [...resizeSteps, lastAugment], variants_per_asset: 1 };

if (samples === undefined) return <LoadingState rows={1} />;
if (samples.length === 0) {
if (sample === undefined) return <LoadingState rows={1} />;
if (sample === null) {
return (
<p className="text-sm text-muted-foreground" data-testid="preview-empty">
Nothing to preview yet — the project has no images. The recipe can still be saved.
Expand All @@ -439,54 +450,64 @@ function PreviewGrid({
<div className="grid grid-cols-3 gap-2 text-xs text-muted-foreground">
<span>Original</span>
<span>After resize</span>
<span>After augmentation</span>
<span data-testid="preview-augment-heading">
{lastAugment === undefined ? "After augmentation" : `After ${opLabel(lastAugment.op)}`}
</span>
</div>
{samples.map((assetId, index) => (
<div
key={assetId}
className="grid grid-cols-3 items-center gap-2"
data-testid={`preview-row-${index}`}
title={`Sample ${assetId.slice(0, 8)}`}
>
<div
className="grid grid-cols-3 items-center gap-2"
data-testid="preview-row-0"
title={`Sample ${sample.slice(0, 8)}`}
>
<PreviewCell
projectId={projectId}
assetId={sample}
variant={0}
spec={NO_TRANSFORM}
classes={classes}
testId="preview-0-original"
onReady={onReady}
/>
{hasResize ? (
<PreviewCell
projectId={projectId}
assetId={assetId}
assetId={sample}
variant={0}
spec={NO_TRANSFORM}
spec={resizeOnly}
classes={classes}
testId="preview-0-resize"
/>
) : (
<Placeholder text="No resize step" testId="preview-0-resize" />
)}
{showcase !== null ? (
<PreviewCell
projectId={projectId}
assetId={sample}
variant={1}
spec={showcase}
classes={classes}
testId={`preview-${index}-original`}
{...(index === 0 ? { onReady } : {})}
testId="preview-0-augment"
showcase
/>
{hasResize ? (
<PreviewCell
projectId={projectId}
assetId={assetId}
variant={0}
spec={resizeOnly}
classes={classes}
testId={`preview-${index}-resize`}
/>
) : (
<Placeholder text="No resize step" testId={`preview-${index}-resize`} />
)}
{hasAugment ? (
<PreviewCell
projectId={projectId}
assetId={assetId}
variant={1}
spec={settled}
classes={classes}
testId={`preview-${index}-augment`}
/>
) : (
<Placeholder text="No augmentation" testId={`preview-${index}-augment`} />
)}
</div>
))}
) : (
<Placeholder text="No augmentation" testId="preview-0-augment" />
)}
</div>
</div>
);
}

/** The augmentation the preview shows: the last one the draft holds, in the editor's own order. */
function lastAugmentStep(spec: RecipeSpec): AugmentStepSpec | undefined {
const augments = spec.steps.filter((step): step is AugmentStepSpec => step.kind === "augment");
return augments[augments.length - 1];
}

function opLabel(op: AugmentStepSpec["op"]): string {
return (AUGMENT_OPS.find((one) => one.op === op)?.label ?? op).toLowerCase();
}

/**
* One asset through one spec. The rendered image with its labels drawn where
* the export would write them, or a placeholder while it has not arrived, or
Expand All @@ -507,6 +528,7 @@ function PreviewCell({
classes,
testId,
onReady,
showcase = false,
}: {
readonly projectId: string;
readonly assetId: string;
Expand All @@ -515,13 +537,15 @@ function PreviewCell({
readonly classes: readonly LabelClass[] | undefined;
readonly testId: string;
readonly onReady?: (ready: boolean) => void;
readonly showcase?: boolean;
}): JSX.Element {
const preview = usePreprocessingPreview(
projectId,
assetId,
variant,
spec,
spec === null ? "" : canonicalSpec(spec),
showcase,
);
const ready = preview.data !== undefined;
useEffect(() => {
Expand Down
Loading
Loading