feat: pre-processing recipes as a project resource, applied at export

CHANGELOG.md

@@ -13,6 +13,24 @@ nothing was being distributed. This is the first version that is.
 
 ### Added
 
+- **Pre-processing: resize and augmentation recipes applied at export.** Part of the
+  pre-processing epic (#785). A recipe is a named project resource - at most one resize step
+  (`letterbox` or `stretch`) followed by augmentation steps (`hflip`, `brightness_contrast`,
+  `rot90`) and a number of variants per train-fold image - served as
+  `/projects/{id}/preprocessing-recipes` with a preview at
+  `POST /projects/{id}/preprocessing-preview`, `visionset recipe`, and the
+  `create_preprocessing_recipe`, `list_preprocessing_recipes` and (behind `--allow-destructive`)
+  `delete_preprocessing_recipe` tools. An export names one beside its target - `recipe=` on the
+  export and compatibility routes, `visionset export --recipe`, `recipe` on `export_release` and
+  `check_export` - and keeps the spec by value: the job carries a snapshot, and the export report
+  gains a `preprocessing` block with the spec, its hash, the Pillow version and a mapping from
+  every written image to its source. `ExportResult` separates `source_file_count` and
+  `augmented_file_count` from the total. Augmented variants are written for the train fold only
+  and named `<hash>-aug<k>`; a recipe that augments refuses a release published without a split
+  (`AUGMENTATION_REQUIRES_SPLIT`), and a step refuses a geometry it cannot move
+  (`PREPROCESSING_STEP_UNSUPPORTED_GEOMETRY`). Workspace format version 17 adds the
+  `preprocessing_recipes` table.
+
 - **Export targets: a release is exported for the model it will train.** Part of the
   export-targets epic (#784). Every installed format declares the targets it writes for, and the
   catalog is served on every surface: `GET /export-targets`, `visionset target list` and the

docs/content/README.md

@@ -37,6 +37,7 @@ If you are new to VisionSet, start with [install.md](install.md), then continue
 | [media.md](media.md) | Decoding raw media: the two processor ports, the accepted image formats, the orientation policy for stills and clips, pinned thumbnails and seek-free frame extraction, and what their determinism does and does not promise |
 | [datasets.md](datasets.md) | The curated trunk: promotion from a completed batch, what `skipped` keeps out, curation without a `confirm=`, and the append-only change log |
 | [releases.md](releases.md) | The immutable artifact: what a manifest is and is not, why two publishes agree byte for byte, hash verification, and the seeded split recipe |
+| [preprocessing.md](preprocessing.md) | The optional stage at export: resize and augmentation recipes, the grammar, what moves geometry and what moves pixels, determinism and its scope, the train-only rule, and the report |
 | [events.md](events.md) | Domain events: subscribing by type, why emission follows the commit, at-most-once delivery, and what an isolated subscriber failure does |
 | [persistence.md](persistence.md) | The metadata store: repositories, unit of work, table layout, migrations and `format_version` |
 | [examples.md](examples.md) | The six runnable examples: the whole cycle in one pass, ingest on its own, the same cycle driven three ways — over HTTP, from a shell, and over MCP stdio — and the thirty-minute flow that ends at a trainer loading the result, with what each is built to demonstrate |

docs/content/api.md

@@ -609,9 +609,9 @@ argument for branching on `code`.
 | Status | Codes |
 | --- | --- |
 | **401** | `UNAUTHORIZED` — with a `WWW-Authenticate: Bearer` challenge |
-| **404** | `PROJECT_NOT_FOUND` · `SCHEMA_NOT_FOUND` · `SCHEMA_DRAFT_NOT_FOUND` · `BATCH_NOT_FOUND` · `JOB_NOT_FOUND` · `INGEST_JOB_NOT_FOUND` · `BACKGROUND_JOB_NOT_FOUND` · `ASSET_NOT_FOUND` · `SOURCE_NOT_FOUND` · `DATASET_NOT_FOUND` · `ANNOTATION_NOT_FOUND` · `RELEASE_NOT_FOUND` · `TOKEN_NOT_FOUND` · `INFERENCE_CONNECTION_NOT_FOUND` · `ASSET_NOT_IN_JOB` · `ASSET_NOT_IN_DATASET` · `NO_SPLIT_RECIPE` · `EXPORT_FORMAT_NOT_FOUND` · `EXPORT_TARGET_NOT_FOUND` · `THUMBNAIL_NOT_CACHED` · `NOT_FOUND` (no such route) |
+| **404** | `PROJECT_NOT_FOUND` · `SCHEMA_NOT_FOUND` · `SCHEMA_DRAFT_NOT_FOUND` · `BATCH_NOT_FOUND` · `JOB_NOT_FOUND` · `INGEST_JOB_NOT_FOUND` · `BACKGROUND_JOB_NOT_FOUND` · `ASSET_NOT_FOUND` · `SOURCE_NOT_FOUND` · `DATASET_NOT_FOUND` · `ANNOTATION_NOT_FOUND` · `RELEASE_NOT_FOUND` · `TOKEN_NOT_FOUND` · `INFERENCE_CONNECTION_NOT_FOUND` · `ASSET_NOT_IN_JOB` · `ASSET_NOT_IN_DATASET` · `NO_SPLIT_RECIPE` · `EXPORT_FORMAT_NOT_FOUND` · `EXPORT_TARGET_NOT_FOUND` · `THUMBNAIL_NOT_CACHED` · `PREPROCESSING_RECIPE_NOT_FOUND` · `NOT_FOUND` (no such route) |
 | **405** | `METHOD_NOT_ALLOWED` |
-| **409** | `PROJECT_NAME_TAKEN` · `RELEASE_TAG_TAKEN` · `TOKEN_NAME_TAKEN` · `INFERENCE_CONNECTION_NAME_TAKEN` · `WORKSPACE_ALREADY_EXISTS` · `WORKSPACE_NOT_EMPTY` · `SCHEMA_VERSION_CONFLICT` · `INVALID_TRANSITION` · `STALE_WRITE` · `BATCH_NOT_EDITABLE` · `BATCH_IMMUTABLE` · `BATCH_NOT_IN_ANNOTATION` · `ASSET_NOT_WRITABLE` · `JOB_FINISHED` · `BATCH_NOT_COMPLETE` · `JOB_NOT_COMPLETE` · `EMPTY_BATCH` · `EMPTY_RELEASE` · `RELEASE_CONTENT_WOULD_VIOLATE_SCHEMA` · `CONFIRMATION_REQUIRED` · `DESTRUCTIVE_SCHEMA_CHANGE` · `SCHEMA_CHANGE_WOULD_ORPHAN` · `SCHEMA_HAS_NO_DETECTABLE_CLASS` · `UNSERIALIZABLE_MANIFEST` · `LOSSY_EXPORT_NOT_CONSENTED` · `EXPORT_SOURCE_UNREADABLE` · `INFERENCE_CONNECTION_NOT_DOWNLOADABLE` · `INFERENCE_CONNECTION_NOT_CHECKABLE` · `INFERENCE_CONNECTION_NOT_TESTABLE` · `INFERENCE_CONNECTION_MODEL_FIXED` · `WEIGHTS_DAMAGED` · `INFERENCE_CONNECTION_NOT_SET_UP` · `AUGMENTATION_REQUIRES_SPLIT` · `PREPROCESSING_STEP_UNSUPPORTED_GEOMETRY` |
+| **409** | `PROJECT_NAME_TAKEN` · `RELEASE_TAG_TAKEN` · `TOKEN_NAME_TAKEN` · `INFERENCE_CONNECTION_NAME_TAKEN` · `PREPROCESSING_RECIPE_NAME_TAKEN` · `WORKSPACE_ALREADY_EXISTS` · `WORKSPACE_NOT_EMPTY` · `SCHEMA_VERSION_CONFLICT` · `INVALID_TRANSITION` · `STALE_WRITE` · `BATCH_NOT_EDITABLE` · `BATCH_IMMUTABLE` · `BATCH_NOT_IN_ANNOTATION` · `ASSET_NOT_WRITABLE` · `JOB_FINISHED` · `BATCH_NOT_COMPLETE` · `JOB_NOT_COMPLETE` · `EMPTY_BATCH` · `EMPTY_RELEASE` · `RELEASE_CONTENT_WOULD_VIOLATE_SCHEMA` · `CONFIRMATION_REQUIRED` · `DESTRUCTIVE_SCHEMA_CHANGE` · `SCHEMA_CHANGE_WOULD_ORPHAN` · `SCHEMA_HAS_NO_DETECTABLE_CLASS` · `UNSERIALIZABLE_MANIFEST` · `LOSSY_EXPORT_NOT_CONSENTED` · `EXPORT_SOURCE_UNREADABLE` · `INFERENCE_CONNECTION_NOT_DOWNLOADABLE` · `INFERENCE_CONNECTION_NOT_CHECKABLE` · `INFERENCE_CONNECTION_NOT_TESTABLE` · `INFERENCE_CONNECTION_MODEL_FIXED` · `WEIGHTS_DAMAGED` · `INFERENCE_CONNECTION_NOT_SET_UP` · `AUGMENTATION_REQUIRES_SPLIT` · `PREPROCESSING_STEP_UNSUPPORTED_GEOMETRY` |
 | **422** | `VALIDATION_ERROR` · `ASSET_NOT_IN_BATCH` · `ANNOTATION_NOT_FROM_MODEL` · `INVALID_NAME` · `INFERENCE_CONNECTION_INVALID` · `INVALID_SCHEMA` · `UNSUPPORTED_GEOMETRY` · `INVALID_ANNOTATION` · `LABEL_CLASS_NOT_IN_SCHEMA` · `DISALLOWED_GEOMETRY` · `ANNOTATION_GEOMETRY_OUT_OF_BOUNDS` · `DUPLICATE_CLASSIFICATION_TAG` · `MISSING_REQUIRED_ATTRIBUTE` · `UNKNOWN_ATTRIBUTE` · `INVALID_ATTRIBUTE_VALUE` · `INVALID_PARTITION` · `UNKNOWN_JOB_TYPE` · `MEDIA_ERROR` · `UNSUPPORTED_MEDIA` · `CORRUPT_MEDIA` · `UNSUPPORTED_PROMPT` · `PROMPT_POINT_OUT_OF_BOUNDS` · `GEOMETRY_NOT_PRODUCED` |
 | **502** | `INFERENCE_ENDPOINT_UNAVAILABLE` |
 | **503** | `WORKSPACE_BUSY` |

docs/content/cli.md

@@ -35,8 +35,10 @@ visionset job pre-label JOB_ID CONNECTION [--minimum-confidence FLOAT] [--replac
 visionset release publish --tag T --project P [--split TRAIN,VAL,TEST] [--seed N]
 visionset release list --project P
 visionset release verify TAG --project P
-visionset export --project P --release TAG --target T|--format F --out DIR [--allow-lossy]
+visionset export --project P --release TAG --target T|--format F --out DIR [--allow-lossy] [--recipe NAME]
 visionset export --project P --release TAG --target T|--format F --check   # writes nothing; exit 1 = it loses something
+visionset recipe create NAME --project P --spec FILE | --resize letterbox:640x640 --augment hflip --variants 2 --target T
+visionset recipe list|show NAME|update NAME|delete NAME --project P
 visionset format list                                    # no --workspace: it opens nothing
 visionset target list                                    # the models a release can be exported for
 
@@ -459,6 +461,27 @@ means what it looks like. The table is on **stdout** and the summary on stderr,
 classes and nothing else; `--json` prints `visionset.wire.export_compatibility`, the same document
 the REST route and the MCP tool publish.
 
+### `visionset recipe`, and `export --recipe`
+
+`recipe create NAME --project P` stores a [pre-processing recipe](preprocessing.md) under a name,
+and `recipe list`, `recipe show NAME`, `recipe update NAME` and `recipe delete NAME` do what they
+say; `--json` shapes are the wire's, and `recipe show --json` prints under `spec` exactly what
+`--spec FILE` reads back. **Two ways to say what a recipe does, never both**: `--spec FILE` for
+a script, or the flag form at a prompt - `--resize letterbox:640x640` (`--pad` for the grey),
+`--augment hflip,brightness_contrast` (`--amount` for how far), `--variants 2`, `--target yolo11`.
+Mixing the file with a flag, or giving neither, is a usage error at exit 2, and so is a spec
+that breaks the recipe grammar - the rule is named. `recipe update` replaces the spec whole and
+takes `--rename`; `recipe delete` asks nothing, because every export that used the recipe kept
+its own copy.
+
+`export --recipe NAME` applies one: the recipe is resolved through the release's project, every
+image is resized as it says, and augmented variants of the train-fold images are written beside
+their sources. `--check` with a recipe also refuses now what the export would refuse -
+augmentation over a release published without a split, or a step that cannot move a geometry
+the release carries - at exit 1 with the reason. The human output adds one line naming the
+source and augmented counts; `--json` carries them as `source_file_count` and
+`augmented_file_count`, with the recipe under `preprocessing`.
+
 ### `visionset backfill-thumbnails`
 
 `--project P` → `IngestService.backfill_thumbnails`. Renders the previews of assets that have none -

docs/content/mcp-tools.md

@@ -11,7 +11,7 @@ error envelope, and the three gate words.
 
 ## Always offered
 
-54 tools, in the order an agent meets them: make a project, give it a schema, put images in it, work through them, promote, publish, export.
+56 tools, in the order an agent meets them: make a project, give it a schema, put images in it, work through them, promote, publish, export.
 
 | Tool | Takes | What it does |
 | --- | --- | --- |
@@ -60,8 +60,10 @@ error envelope, and the three gate words.
 | `verify_release` | `project`, `tag` | Re-read and re-hash everything a release names, and report what is wrong. |
 | `list_formats` | — | List the export formats installed in this VisionSet, and whether each is lossy. |
 | `list_export_targets` | — | List the models a release can be exported for, each with the format that writes for it. |
-| `check_export` | `project`, `tag`, `target`?, `format`? | Say what a target or a format would drop from a release, without writing anything. |
-| `export_release` | `project`, `tag`, `dest`, `target`?, `format`?, `allow_lossy`? | Write a release to a local directory, for a target or in one of the installed formats. |
+| `check_export` | `project`, `tag`, `target`?, `format`?, `recipe`? | Say what a target or a format would drop from a release, without writing anything. |
+| `export_release` | `project`, `tag`, `dest`, `target`?, `format`?, `allow_lossy`?, `recipe`? | Write a release to a local directory, for a target or in one of the installed formats. |
+| `list_preprocessing_recipes` | `project` | List a project's pre-processing recipes, oldest first, each with its whole spec. |
+| `create_preprocessing_recipe` | `project`, `name`, `spec` | Store a named pre-processing recipe on a project, for `export_release` to apply. |
 | `list_inference_connections` | — | List this workspace's model connections, oldest first. |
 | `model_download_size` | `model_id`, `model_revision` | How big fetching that model's weights would be. Nothing is downloaded. |
 | `create_inference_connection` | `name`, `connection_type`, `model_id`, `model_revision`, `device`?, `precision`?, `endpoint_url`?, `provider_id`?, `credential_env`? | Configure a connection. Nothing is downloaded and nothing is contacted. |
@@ -79,3 +81,4 @@ Absent from the listing unless the server was started for them (#108). A `confir
 | `delete_batch` | `batch_id`, `confirm`? | Delete a batch and how its work was organised. Destructive; requires `confirm=true`. |
 | `delete_project` | `project`, `confirm`? | Delete a project and everything under it. Destructive; requires `confirm=true`. |
 | `delete_inference_connection` | `connection`, `confirm`? | Remove a connection. Annotations keep the model provenance they recorded. |
+| `delete_preprocessing_recipe` | `project`, `name`, `confirm`? | Delete a pre-processing recipe. Destructive; requires `confirm=true`. |

docs/content/mcp.md

@@ -192,8 +192,10 @@ call until the end. #439 has since added a job gate, but it changes none of this
 | `verify_release` | Re-hash every blob a release names. |
 | `list_formats` | Installed exporters, which are lossy, what each can write, and the targets it writes for. |
 | `list_export_targets` | The models a release can be exported for, each with the format it resolves to and its hints. |
-| `check_export` | What a target or a format would drop from a release, before writing anything. Exactly one of `target` and `format`. |
-| `export_release` | Write a release to a local directory, for a target or in a format. `allow_lossy` where needed. |
+| `check_export` | What a target or a format would drop from a release, before writing anything. Exactly one of `target` and `format`; `recipe` also checks that the recipe can run. |
+| `export_release` | Write a release to a local directory, for a target or in a format. `allow_lossy` where needed; `recipe` applies a pre-processing recipe by name. |
+| `create_preprocessing_recipe` | Store a named [pre-processing recipe](preprocessing.md) on a project: a resize step, augmentation steps, and how many variants each train image gets. |
+| `list_preprocessing_recipes` | A project's recipes, each with its whole spec. `name` is what `export_release` takes as `recipe`. |
 
 ### Inference connections
 
@@ -224,6 +226,7 @@ ask and is refused with `INFERENCE_CONNECTION_NOT_TESTABLE`.
 | `delete_batch` | **Destructive.** Removes a batch, its task groups, its jobs and the per-asset progress on them. The **annotations survive** - labels hang off assets, not off batches - and so do the assets themselves. A `completed` batch is refused whatever `confirm` says. |
 | `delete_project` | **Destructive.** Removes the project, its dataset, its batches, its jobs and its annotations. Requires `confirm: true` as well - the parameter is unchanged; what changed is that the tool is not in the listing unless somebody started the server for it. |
 | `delete_inference_connection` | **Destructive.** Removes a model connection's configuration and nothing else: annotations keep their model provenance (identity is copied at write time), and cached weights stay on disk. Requires `confirm: true`. |
+| `delete_preprocessing_recipe` | **Destructive.** Removes a pre-processing recipe and nothing else: every export that used it kept its own copy of the spec. Requires `confirm: true`. |
 
 ## `get_asset_image`, and the coordinate frame
 
@@ -293,7 +296,7 @@ Never merged into one, because they guard different things:
 
 | | guards | on |
 | --- | --- | --- |
-| `confirm` | destroying data | `delete_project`, `delete_batch` |
+| `confirm` | destroying data | `delete_project`, `delete_batch`, `delete_inference_connection`, `delete_preprocessing_recipe` |
 | `allow_destructive` | narrowing a contract | `create_schema_version`, `publish_schema_draft` |
 | `allow_lossy` | emitting an incomplete copy of something that stays intact | `export_release` |
 

docs/content/persistence.md

@@ -169,11 +169,12 @@ MIGRATIONS: list[Migration] = [
     Migration(version=14, name="project_created_at", upgrade=_add_project_created_at),
     Migration(version=15, name="connection_origin", upgrade=_add_connection_origin),
     Migration(version=16, name="source_clip_ranges", upgrade=_reshape_source_origin_index),
+    Migration(version=17, name="preprocessing_recipes", upgrade=_add_preprocessing_recipes),
 ]
-FORMAT_VERSION: int = MIGRATIONS[-1].version  # 16
+FORMAT_VERSION: int = MIGRATIONS[-1].version  # 17
 ```
 
-**Generation 1 is the baseline, and the fifteen entries after it are ordinary migrations.** A long
+**Generation 1 is the baseline, and the sixteen entries after it are ordinary migrations.** A long
 chain of generations got this schema to its present shape while VisionSet was unreleased.
 Every database they could have upgraded was disposable test data inside this repository, so
 what they actually bought was an idempotency argument and an undo line per generation, plus
@@ -186,7 +187,7 @@ force again for every entry appended after the baseline.
 `tests/kernel/test_migrations.py` that builds an old-looking file. The failure is the silent
 kind: a column left in place makes its own migration find the column already there and return
 early, so `test_a_fresh_database_and_a_migrated_one_have_the_same_schema` compares a file
-against itself and passes while proving nothing. The table-creating migrations - 4, 6 and 10 -
+against itself and passes while proving nothing. The table-creating migrations - 4, 6, 10 and 17 -
 are the standing exception: dropping a whole table in the helper would exercise SQLite rather
 than this module.
 
@@ -210,6 +211,8 @@ docstring says which, so a reader does not mistake an honest absence for a forgo
 Migration 16 adds no column at all: it reshapes the source-origin index to carry the
 canonical-ranges term, and backfills nothing — a row written before ranges existed has no
 `$.ranges` key, which the new index reads as the same `''` a whole-clip selection stores.
+Migration 17 creates the `preprocessing_recipes` table whole and backfills nothing: no recipe
+existed before it, and an export that applied none is what every earlier export was.
 
 **There are no downgrade paths, deliberately.** Nothing walks a file backwards and the
 tests no longer do either. A downgrade is a compatibility promise and a promise is owed