Skip to content

feat: pre-processing recipes as a project resource, applied at export - #797

Merged
JArmandoAnaya merged 9 commits into
mainfrom
feat/preprocessing-p3-resource
Aug 26, 2026
Merged

feat: pre-processing recipes as a project resource, applied at export#797
JArmandoAnaya merged 9 commits into
mainfrom
feat/preprocessing-p3-resource

Conversation

@JArmandoAnaya

Copy link
Copy Markdown
Contributor

Stack

Position 3 of the pre-processing stack (#785); based on feat/targets-t3-catalog with feat/preprocessing-p2-drivers (PR #793) rebased in.

What changed

A pre-processing recipe is a named project resource, and an export applies one by name beside its target.

  • Resource. preprocessing_recipes table (migration 17, unique (project_id, name)), a preprocessing_recipes repository on the unit of work, and PreprocessingRecipeService with create, list, get, update (whole-value, with rename), delete, for_release (name resolved through the release's project) and preview. Names are slugs. Two new refusals, PreprocessingRecipeNotFound (404) and PreprocessingRecipeNameTaken (409), in the four gated files.
  • Export seam. ReleaseService.export(..., recipe=, recipe_name=, drivers=), and recipe= on check_export and require_export_consent. The manifest is narrowed to the target first and the recipe runs over what is left, so a dropped geometry never reaches a step that refuses it; consent is asked before any transform. Folds are the release's own split over the frozen manifest; variants are generated for the train fold only. The plugin is handed one manifest asset per file to write — the base under its source hash, variant k under <hash>-aug<k> sharing the source's asset_id — and the content reader resolves either key to the driver-transformed bytes. formats/_layout.py::folds_of cuts over base assets only, which keeps a variant in its source's fold.
  • Report. ExportResult gains source_file_count, augmented_file_count, source_annotation_count, augmented_annotation_count and preprocessing; file_count stays the total. visionset-export-report.json gains a preprocessing key (null without a recipe) holding recipe_name, spec, recipe_hash, pillow_version and a mapping of every written image to its source hash, exported digest and variant. tests/formats/test_report_agreement.py holds written == source + augmented per fold on real pixels.
  • REST. POST/GET /projects/{id}/preprocessing-recipes, GET/PUT/DELETE .../{name}, POST /projects/{id}/preprocessing-preview (same kernel path as export, max edge 512, Cache-Control: no-store), and recipe= on POST /releases/{id}/export and GET /releases/{id}/export-compatibility; the job payload carries the recipe as a snapshot. openapi.json and the generated client regenerated; pnpm -r build && pnpm -r test && pnpm -r lint green with no stub changes needed.
  • Wire. wire.recipe_spec, wire.preprocessing_recipe, wire.preprocessing_preview, wire.export_preprocessing; JSON-contract pairs for the recipe and the preview; export_result carries the new fields.
  • CLI. visionset recipe create|list|show|update|delete (--spec FILE or --resize/--augment/--variants/--target), export --recipe.
  • MCP. create_preprocessing_recipe, list_preprocessing_recipes, delete_preprocessing_recipe (only under --allow-destructive, takes confirm), recipe on export_release and check_export; docs/content/mcp-tools.md regenerated.
  • Docs. New docs/content/preprocessing.md; releases.md, cli.md, api.md, mcp.md, persistence.md, the README map, the sidebar, and the changelog.

Decisions taken, flagged for review

  • The dispatch's synthetic-asset mechanism could not be realised literally. ManifestAsset.asset_id is a UUID, so "{asset_id}-aug{k}" cannot live there; and every built-in exporter names its files by content_hash (image_name, the YOLO label .txt, the lane files), so a variant whose content_hash stayed the source's would overwrite its base. The variant's content_hash is therefore "{source_hash}-aug{k}" (variant_content_hash / source_of_content_hash in the kernel), which is what gives B7's <stem>-aug<k> naming for free, and its asset_id stays the source's. Variant annotation ids are uuid5(VARIANT_ID_NAMESPACE, "{id}-aug{k}"), since ManifestAnnotation.id is a UUID too.
  • Exporters recompute folds via folds_of(release, manifest) over the manifest they are handed, which would have re-cut the split over the augmented set. folds_of now cuts over base assets only and looks a variant up by the asset_id it shares with its source.
  • PreprocessingRecipe.id / project_id moved from str to UUID (P1 followed the dispatch's sketch; the repository port is UUID-keyed).
  • ExportResult moved from kernel/domain/release.py to a new kernel/domain/export_report.py, because it now names RecipeSpec and the recipe module already imports release.py. Re-exported from visionset.kernel.domain unchanged.
  • Persistence port. A preprocessing_recipes: Repository[PreprocessingRecipe] property rather than five bespoke methods; the dispatch's add/get/list/replace/delete map onto the repository's own verbs, and name lookup is list(project_id) plus a filter, the ReleaseService.get_by_tag precedent.
  • Table name is preprocessing_recipes as the dispatch spells it; every other table here is singular.
  • pillow_version is PIL.__version__, imported in release_service.py.

Hard stop, not decided here

tests/architecture/test_tracked_file_sizes.py refuses openapi.json at 530,351 bytes against its 512 KiB ceiling. Even with every byte of this branch's new prose removed the file would stay above the ceiling (~522 KB): the growth is three paths, eleven schemas and their 422 entries. The gate's own comment reserves this for a person ("the question is whether the API really grew that much, and the answer is not another zero here"), so the ceiling is untouched and the failure is reported verbatim below.

Checks

Full gate on the final tree (bash scripts/check.sh), verbatim summary:

     38s  python tests
      1s  ruff (lint)
      0s  ruff (format)
      0s  mypy
      0s  import contracts
      5s  frontend build
     29s  frontend tests
     10s  frontend lint
      1s  openapi drift
      1s  generated client drift
      1s  mcp tool reference drift
      0s  export target catalog drift
      0s  version sync
     44s  annotator + app e2e (chromium)
     29s  browser cycle, real server (chromium)
    159s  total

FAILED: python tests ruff (format)

check.sh: FAILED  ran=python,frontend,generated,browser  skipped=docs

The ruff (format) failure was a fenced Python block in docs/content/preprocessing.md, fixed in the last commit; bash scripts/check.sh python rerun on the pushed tree:

FAILED tests/architecture/test_tracked_file_sizes.py::test_no_tracked_file_exceeds_the_size_limit
1 failed, 4744 passed, 33 skipped in 37.99s
FAILED: python tests
check.sh: FAILED  ran=python  skipped=frontend,generated,browser,docs

with the one failure:

E       AssertionError: 1 tracked file(s) over the size limit:
E           openapi.json: 530,351 bytes (limit 524,288)

Docs group (bash scripts/check.sh docs):

check.sh: PASSED  ran=docs  skipped=python,frontend,generated,browser

Browser suites: annotator + app e2e 279 passed (43.0s), real-server cycle 1 passed (28.3s). pnpm test:scripts: 115 pass, 0 fail.

@JArmandoAnaya
JArmandoAnaya force-pushed the feat/targets-t3-catalog branch from b38157d to 1595713 Compare August 26, 2026 11:27
Base automatically changed from feat/targets-t3-catalog to main August 26, 2026 11:33
@JArmandoAnaya
JArmandoAnaya force-pushed the feat/preprocessing-p3-resource branch 2 times, most recently from 9e953b7 to cddd80e Compare August 26, 2026 11:38
A recipe is stored under a slug unique per project, in a table created whole
by migration 17, and read back as the value an export will snapshot. The
service refuses an unknown name and a taken name with their own errors, mapped
on the server and listed in the API reference. Part of the pre-processing epic
(#785).
ReleaseService.export takes a recipe and the installed drivers. The manifest
is narrowed to the target first, then transformed, so a geometry the target
drops never reaches a step that refuses it; folds are the release's own split
over the frozen manifest, and augmented variants are written for the train
fold only. The plugin sees one manifest asset per file — a base image under
its source hash, a variant under <hash>-aug<k> — and the content reader
resolves either to the driver-transformed bytes. The result separates source
from augmented counts, the report gains a preprocessing block with the
snapshot, its hash, the Pillow version and a file mapping, and the export
routes take a recipe by name and carry its snapshot to the job. Part of the
pre-processing epic (#785).
… projections

A project's recipes are a plain resource under
/projects/{id}/preprocessing-recipes with no state gating, and
POST /projects/{id}/preprocessing-preview renders one asset through a spec on
the export's own kernel path, capped to 512 pixels and never cached. The wire
projections pair with the server models under the JSON-contract test, and the
contract and the generated client are regenerated. Part of the pre-processing
epic (#785).
A recipe group with create, list, show, update and delete, taking either a
JSON spec file or the flag form (--resize, --augment, --variants, --target),
and --recipe on export, which resolves the recipe through the release's
project and applies it. Part of the pre-processing epic (#785).
create_preprocessing_recipe and list_preprocessing_recipes are offered by
default; delete_preprocessing_recipe is offered only under --allow-destructive
and takes confirm. The tool reference is regenerated. Part of the
pre-processing epic (#785).
A new preprocessing.md carries the concept, the grammar, the geometry table,
determinism and its scope, the train-only rule, naming and the report; the
release, CLI, API and MCP pages gain the recipe half of exporting, the
persistence page records migration 17, and the changelog the feature. Part
of the pre-processing epic (#785).
@JArmandoAnaya
JArmandoAnaya force-pushed the feat/preprocessing-p3-resource branch from fb7cddb to 64d6cb6 Compare August 26, 2026 11:50
@JArmandoAnaya
JArmandoAnaya merged commit ff05de9 into main Aug 26, 2026
15 checks passed
@JArmandoAnaya
JArmandoAnaya deleted the feat/preprocessing-p3-resource branch August 26, 2026 11:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant