feat: pre-processing recipes as a project resource, applied at export - #797
Merged
Conversation
JArmandoAnaya
force-pushed
the
feat/targets-t3-catalog
branch
from
August 26, 2026 11:27
b38157d to
1595713
Compare
JArmandoAnaya
force-pushed
the
feat/preprocessing-p3-resource
branch
2 times, most recently
from
August 26, 2026 11:38
9e953b7 to
cddd80e
Compare
A recipe is stored under a slug unique per project, in a table created whole by migration 17, and read back as the value an export will snapshot. The service refuses an unknown name and a taken name with their own errors, mapped on the server and listed in the API reference. Part of the pre-processing epic (#785).
ReleaseService.export takes a recipe and the installed drivers. The manifest is narrowed to the target first, then transformed, so a geometry the target drops never reaches a step that refuses it; folds are the release's own split over the frozen manifest, and augmented variants are written for the train fold only. The plugin sees one manifest asset per file — a base image under its source hash, a variant under <hash>-aug<k> — and the content reader resolves either to the driver-transformed bytes. The result separates source from augmented counts, the report gains a preprocessing block with the snapshot, its hash, the Pillow version and a file mapping, and the export routes take a recipe by name and carry its snapshot to the job. Part of the pre-processing epic (#785).
… projections
A project's recipes are a plain resource under
/projects/{id}/preprocessing-recipes with no state gating, and
POST /projects/{id}/preprocessing-preview renders one asset through a spec on
the export's own kernel path, capped to 512 pixels and never cached. The wire
projections pair with the server models under the JSON-contract test, and the
contract and the generated client are regenerated. Part of the pre-processing
epic (#785).
A recipe group with create, list, show, update and delete, taking either a JSON spec file or the flag form (--resize, --augment, --variants, --target), and --recipe on export, which resolves the recipe through the release's project and applies it. Part of the pre-processing epic (#785).
create_preprocessing_recipe and list_preprocessing_recipes are offered by default; delete_preprocessing_recipe is offered only under --allow-destructive and takes confirm. The tool reference is regenerated. Part of the pre-processing epic (#785).
A new preprocessing.md carries the concept, the grammar, the geometry table, determinism and its scope, the train-only rule, naming and the report; the release, CLI, API and MCP pages gain the recipe half of exporting, the persistence page records migration 17, and the changelog the feature. Part of the pre-processing epic (#785).
…s, not the last one walked
JArmandoAnaya
force-pushed
the
feat/preprocessing-p3-resource
branch
from
August 26, 2026 11:50
fb7cddb to
64d6cb6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack
Position 3 of the pre-processing stack (#785); based on
feat/targets-t3-catalogwithfeat/preprocessing-p2-drivers(PR #793) rebased in.What changed
A pre-processing recipe is a named project resource, and an export applies one by name beside its target.
preprocessing_recipestable (migration 17, unique(project_id, name)), apreprocessing_recipesrepository on the unit of work, andPreprocessingRecipeServicewith create, list, get, update (whole-value, with rename), delete,for_release(name resolved through the release's project) andpreview. Names are slugs. Two new refusals,PreprocessingRecipeNotFound(404) andPreprocessingRecipeNameTaken(409), in the four gated files.ReleaseService.export(..., recipe=, recipe_name=, drivers=), andrecipe=oncheck_exportandrequire_export_consent. The manifest is narrowed to the target first and the recipe runs over what is left, so a dropped geometry never reaches a step that refuses it; consent is asked before any transform. Folds are the release's own split over the frozen manifest; variants are generated for the train fold only. The plugin is handed one manifest asset per file to write — the base under its source hash, variantkunder<hash>-aug<k>sharing the source'sasset_id— and the content reader resolves either key to the driver-transformed bytes.formats/_layout.py::folds_ofcuts over base assets only, which keeps a variant in its source's fold.ExportResultgainssource_file_count,augmented_file_count,source_annotation_count,augmented_annotation_countandpreprocessing;file_countstays the total.visionset-export-report.jsongains apreprocessingkey (nullwithout a recipe) holdingrecipe_name,spec,recipe_hash,pillow_versionand amappingof every written image to its source hash, exported digest and variant.tests/formats/test_report_agreement.pyholds written == source + augmented per fold on real pixels.POST/GET /projects/{id}/preprocessing-recipes,GET/PUT/DELETE .../{name},POST /projects/{id}/preprocessing-preview(same kernel path as export, max edge 512,Cache-Control: no-store), andrecipe=onPOST /releases/{id}/exportandGET /releases/{id}/export-compatibility; the job payload carries the recipe as a snapshot.openapi.jsonand the generated client regenerated;pnpm -r build && pnpm -r test && pnpm -r lintgreen with no stub changes needed.wire.recipe_spec,wire.preprocessing_recipe,wire.preprocessing_preview,wire.export_preprocessing; JSON-contract pairs for the recipe and the preview;export_resultcarries the new fields.visionset recipe create|list|show|update|delete(--spec FILEor--resize/--augment/--variants/--target),export --recipe.create_preprocessing_recipe,list_preprocessing_recipes,delete_preprocessing_recipe(only under--allow-destructive, takesconfirm),recipeonexport_releaseandcheck_export;docs/content/mcp-tools.mdregenerated.docs/content/preprocessing.md;releases.md,cli.md,api.md,mcp.md,persistence.md, the README map, the sidebar, and the changelog.Decisions taken, flagged for review
ManifestAsset.asset_idis aUUID, so"{asset_id}-aug{k}"cannot live there; and every built-in exporter names its files bycontent_hash(image_name, the YOLO label.txt, the lane files), so a variant whosecontent_hashstayed the source's would overwrite its base. The variant'scontent_hashis therefore"{source_hash}-aug{k}"(variant_content_hash/source_of_content_hashin the kernel), which is what gives B7's<stem>-aug<k>naming for free, and itsasset_idstays the source's. Variant annotation ids areuuid5(VARIANT_ID_NAMESPACE, "{id}-aug{k}"), sinceManifestAnnotation.idis aUUIDtoo.folds_of(release, manifest)over the manifest they are handed, which would have re-cut the split over the augmented set.folds_ofnow cuts over base assets only and looks a variant up by theasset_idit shares with its source.PreprocessingRecipe.id/project_idmoved fromstrtoUUID(P1 followed the dispatch's sketch; the repository port is UUID-keyed).ExportResultmoved fromkernel/domain/release.pyto a newkernel/domain/export_report.py, because it now namesRecipeSpecand the recipe module already importsrelease.py. Re-exported fromvisionset.kernel.domainunchanged.preprocessing_recipes: Repository[PreprocessingRecipe]property rather than five bespoke methods; the dispatch'sadd/get/list/replace/deletemap onto the repository's own verbs, and name lookup islist(project_id)plus a filter, theReleaseService.get_by_tagprecedent.preprocessing_recipesas the dispatch spells it; every other table here is singular.pillow_versionisPIL.__version__, imported inrelease_service.py.Hard stop, not decided here
tests/architecture/test_tracked_file_sizes.pyrefusesopenapi.jsonat 530,351 bytes against its 512 KiB ceiling. Even with every byte of this branch's new prose removed the file would stay above the ceiling (~522 KB): the growth is three paths, eleven schemas and their 422 entries. The gate's own comment reserves this for a person ("the question is whether the API really grew that much, and the answer is not another zero here"), so the ceiling is untouched and the failure is reported verbatim below.Checks
Full gate on the final tree (
bash scripts/check.sh), verbatim summary:The
ruff (format)failure was a fenced Python block indocs/content/preprocessing.md, fixed in the last commit;bash scripts/check.sh pythonrerun on the pushed tree:with the one failure:
Docs group (
bash scripts/check.sh docs):Browser suites: annotator + app e2e
279 passed (43.0s), real-server cycle1 passed (28.3s).pnpm test:scripts: 115 pass, 0 fail.