Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -682,7 +682,7 @@ jobs:
# suppresses the count line — which is the line that would show this job
# quietly shrinking to nothing.
- name: The reference readers load the exports
run: uv run pytest tests/formats/test_yolo_smoke.py tests/formats/test_coco_smoke.py
run: uv run pytest tests/formats/test_ultralytics_roundtrip.py tests/formats/test_coco_smoke.py

# #49's performance benchmark: frame times during pan, zoom and drag over 200
# boxes and 20 polygons of 32 vertices on a 4K asset, plus a CPU-throttling
Expand Down
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,15 @@ nothing was being distributed. This is the first version that is.

### Changed

- **`yolo` format renamed `ultralytics`; `yolo` accepted as an alias until the next release.**
Part of the export-targets epic (#784). The dialect now derives its task from the release -
segment when it holds a polygon, classify when it holds only classification tags, detect
otherwise - and its `data.yaml` carries `path: .`, one key per fold present, and `names` as a
mapping with no `nc`. A second dialect, `yolov5-yaml`, writes the YOLOv5 descriptor grammar
(`./` paths, `nc`, `names` as a list) for YOLOv7. Every exporter now declares the targets it
writes for, and the registry refuses a target declared by two formats or one promising a
geometry its format never writes.

- **A schema version that only widens the contract now moves every open batch onto it** (#381).
A batch is judged against the version it pinned at approval, and that pin used to move only
when somebody asked for it. Publishing an additive version now takes every batch in
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,7 @@ a deliberate manual run, because each costs minutes or needs its own install.
| Design tokens | part of `pnpm test` — `tests/scripts/design_tokens.test.mjs` refuses a colour inside a class name, and `ui-core`'s `tokens.test.ts` gates the stylesheet against its TypeScript mirror | part of `frontend` |
| Wire action rosters | part of `pnpm test` — `tests/scripts/wire_rosters.test.mjs` holds the two transcriptions of `allowed_actions` (`ui-core`'s `testing/wire.fixtures.ts` and the e2e suite's `_wire.ts`) against the kernel's own answer, `tests/fixtures/wire_capabilities.json`, and against each other, in both directions. Both are typed against the generated union, which catches a withdrawn member and nothing else: the rows are hand-written, and comparing them only to each other once let a member both lacked pass green | part of `frontend` |
| Docs links | part of `pnpm test` — `tests/scripts/docs_links.test.mjs` resolves every internal link and every `#anchor` in every tracked Markdown file, naming the file, line and dead fragment. It reads `git ls-files`, so the set grows with the repository and no list is maintained by hand. External URLs are ignored on purpose: a gate that fails for somebody else's rate limit is one people re-run rather than read. Renaming a heading breaks inbound anchors *silently* — the link just lands at the top of the page — which was a near miss during the `visionset ui` → `visionset server` rename (#329) | part of `frontend` |
| Format smoke (ultralytics, pycocotools) | `uv sync --group yolo --group coco && uv run pytest tests/formats/test_*_smoke.py` — their own groups because ultralytics brings torch **and its wheel ships a top-level `tests` package that shadows this repo's**, so run only those files and `uv sync` again afterwards; skips without them, and CI sets `VISIONSET_REQUIRE_ULTRALYTICS=1` / `VISIONSET_REQUIRE_PYCOCOTOOLS=1` so a broken install goes red | — CI |
| Format smoke (ultralytics, pycocotools) | `uv sync --group yolo --group coco && uv run pytest tests/formats/test_ultralytics_roundtrip.py tests/formats/test_coco_smoke.py` — their own groups because ultralytics brings torch **and its wheel ships a top-level `tests` package that shadows this repo's**, so run only those files and `uv sync` again afterwards; skips without them, and CI sets `VISIONSET_REQUIRE_ULTRALYTICS=1` / `VISIONSET_REQUIRE_PYCOCOTOOLS=1` so a broken install goes red | — CI |
| Inference smoke (local-inference extra) | `uv sync --extra local-inference` then `VISIONSET_REQUIRE_LOCAL_INFERENCE=1 uv run pytest tests/inference tests/architecture/test_optional_runtime.py tests/server/test_inference.py tests/server/test_suggest.py tests/cli/test_inference_commands.py tests/jobs/test_weights_job.py -rs`, and `uv sync` again afterwards. The **with-runtime** half of the matrix — see [the two halves](#the-two-halves-of-the-inference-matrix) below. Roughly two gigabytes of CUDA wheels, which is why it is opt-in locally; CI's `inference-smoke` job runs it | — CI |
| Wheel (build, install, serve) | `bash scripts/build_dist.sh && VISIONSET_REQUIRE_WHEEL=1 uv run pytest tests/packaging` — builds the UI into `_static/`, builds the wheel, installs it in a fresh venv and serves `/app/` from it. Opt-in locally (it costs about a minute); CI's `wheel` job runs it and uploads the artifact | — CI |
| The 30-minute flow | `uv run python examples/thirty_minute_flow.py` — the vision document's success metric end to end. CI's `30-minute flow (wheel, end to end)` job runs it from the **installed wheel** in an empty venv, with `ultralytics` required there | — CI |
Expand Down
2 changes: 1 addition & 1 deletion docs/content/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -326,7 +326,7 @@ GET /ingest-jobs/{job_id} → 200 { "state": "completed", "batch_id":
Export follows the same shape, over the generic surface #328 added:

```
POST /releases/{id}/export?format=yolo → 202 Accepted
POST /releases/{id}/export?format=ultralytics → 202 Accepted
Location: /background-jobs/{job_id}

GET /background-jobs/{job_id} → 200 { "state": "running", "processed": 12, … }
Expand Down
5 changes: 3 additions & 2 deletions docs/content/architecture/backend/formats.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,14 @@ typo answers a `VisionSetError` rather than a `KeyError` and a traceback.

## What ships

Ten plugins in [`pyproject.toml`](../../../../pyproject.toml)'s
Eleven plugins in [`pyproject.toml`](../../../../pyproject.toml)'s
`[project.entry-points."visionset.formats"]`:

| Name | Module |
| --- | --- |
| `dummy` | [`_dummy.py`](../../../../src/visionset/formats/_dummy.py) - writes nothing; the registry's own test subject |
| `yolo` | [`yolo/`](../../../../src/visionset/formats/yolo/) |
| `ultralytics` | [`ultralytics/`](../../../../src/visionset/formats/ultralytics/) - `data.yaml` with `path` and `names` as a mapping; `yolo` is accepted as an alias for one release |
| `yolov5-yaml` | [`yolov5_yaml/`](../../../../src/visionset/formats/yolov5_yaml/) - `data.yaml` with `nc` and `names` as a list; shares [`_yolo_writer.py`](../../../../src/visionset/formats/_yolo_writer.py) with `ultralytics` |
| `coco` | [`coco/`](../../../../src/visionset/formats/coco/) |
| `voc` | [`voc/`](../../../../src/visionset/formats/voc/) |
| `classification` | [`classification/`](../../../../src/visionset/formats/classification/) - `labels.csv`, one row per (image, tag) |
Expand Down
4 changes: 2 additions & 2 deletions docs/content/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -444,8 +444,8 @@ that declaration covers attributes, confidence and provenance - none of which is
of which the per-class table can show. So:

```bash
visionset export --check -p road-signs --release v1.0 -f yolo && \
visionset export -p road-signs --release v1.0 -f yolo --out ./out
visionset export --check -p road-signs --release v1.0 -f ultralytics && \
visionset export -p road-signs --release v1.0 -f ultralytics --out ./out
```

means what it looks like. The table is on **stdout** and the summary on stderr, so `| cut` gets
Expand Down
109 changes: 75 additions & 34 deletions docs/content/releases.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,19 +196,19 @@ against one format's declaration and returns an `ExportCompatibility`:

```json
{
"release_id": "…", "format": "yolo", "compatible": false,
"release_id": "…", "format": "yolov5-yaml", "compatible": false,
"format_is_lossy": true,
"excluded_annotations": 40, "excluded_assets": 40,
"degraded_annotations": 1204, "degraded_assets": 310,
"classes": [
{"label_class": "lane", "geometry": "polygon", "status": "degraded",
"annotations": 1204, "assets": 310,
"reason": "yolo writes a polygon as its bounding box; the shape is lost"},
"reason": "yolov5-yaml writes a polygon as its bounding box; the shape is lost"},
{"label_class": "sign", "geometry": "bbox", "status": "supported",
"annotations": 8800, "assets": 2400, "reason": null},
{"label_class": "weather", "geometry": "classification_tag", "status": "dropped",
"annotations": 40, "assets": 40,
"reason": "yolo cannot place a classification_tag and drops it"}
"reason": "yolov5-yaml cannot place a classification_tag and drops it"}
]
}
```
Expand Down Expand Up @@ -245,7 +245,7 @@ The fix is vocabulary rather than capability. `Exporter` declares two geometry s

- **`supported_geometries`** - written as they stand.
- **`degraded_geometries`** - written, having lost something the kernel could represent. `{polygon}`
for `yolo` and `voc`; empty for `coco`, which writes a polygon as a polygon, and empty for
for `yolov5-yaml` and `voc`; empty for `coco`, which writes a polygon as a polygon, and empty for
`dummy`, which writes nothing at all. The two sets are disjoint, and `supported` wins if a plugin
says both, because resolving a contradiction towards the weaker claim would report a loss that
does not happen.
Expand Down Expand Up @@ -308,10 +308,16 @@ the plugin runs (a plugin that clears its own subdirectory would otherwise take
when an earlier run left one behind. That is what keeps "an exporter that writes nothing reports
zero" true, and keeps exporting twice into one directory agreeing with itself.

### The YOLO detection format
### The YOLO dialects

`yolo` is the first format here that writes anything, and it is a rewrite of v1's rather than a
port. The layout:
Two formats write a YOLO dataset, and they differ only in the grammar of `data.yaml`. Each is a
*dialect*: the wire identifier `format_name` names the descriptor grammar, and the model a person
will train - the *target* - resolves to exactly one dialect. `ultralytics` is what every trainer
from YOLOv3 to YOLO26 in the Ultralytics line reads; `yolov5-yaml` is the older grammar YOLOv7
reads. `yolo`, the former name of `ultralytics`, is accepted as an alias for one release and then
removed.

The layout both share:

```
data.yaml
Expand All @@ -328,7 +334,9 @@ index by sorting the names it found in the labels, so a class nobody had used ye
`data.yaml` - and, worse, drawing the first box of a new class *renumbered every other class*. A
model trained against one export and evaluated against the next is then wrong with nothing to
report it. Here the order is `Manifest.classes`, the project's authored schema order frozen at
publication, and every class gets an index whether or not anything uses it.
publication, and every class gets an index whether or not anything uses it. Indices are positions
in that order, so they are contiguous from zero by construction - a class carries no number of
its own that could leave a gap.

**A read failure aborts.** v1 wrapped the image read in `except Exception: pass` and wrote the
label file anyway, so one lost object produced a training set silently short of an image and
Expand All @@ -344,29 +352,62 @@ suffix, which makes the mapping depend on iteration order and lets one picture i
directories land twice. A hash is stable across machines and runs and cannot collide. The cost is
that the names are not human-readable, which a directory destined for a trainer does not need.

Three details of `data.yaml` are ultralytics' contract rather than ours, and each was measured
against its source rather than assumed:

- **There is no `path:` key.** Ultralytics resolves a relative `path` against its own datasets
directory or the working directory of whatever process loads the file - so the obvious `path: .`
breaks the moment the export is copied. Omitted, it falls back to the yaml's own parent, which
is what makes the directory movable.
- **`train` and `val` are both required**, and a missing key is a `SyntaxError` rather than a
default. A release published without a recipe is one undivided set, so `val` names the training
images: that says "there is no held-out set", where omitting the key says "this file is
malformed". `test` is optional and is written only when it has something in it.
- **`images/` → `labels/` is a string substitution on the resolved image path**, not a configured
location, so those two directory names are load-bearing.

An asset with nothing on it gets an **empty** label file rather than none: ultralytics reads a
missing file as "nobody looked" and an empty one as "somebody looked and there is nothing here",
and a detector needs the second.

`yolo` is `lossy = True` unconditionally, because a label row is five numbers - attributes,
confidence and provenance never survive - so every export in this format asks for consent. Its
`supported_geometries` is `{bbox}`: a polygon is still written, as its axis-aligned bounding box,
but its shape is gone, so #65's report counts it as not carried and says which classes and how
many. A classification tag has no location at all and is dropped rather than given an invented box.
#### `ultralytics`

The descriptor is `path: .`, one key per fold present, and `names` as a mapping from index:

```yaml
path: .
train: images/train
val: images/val
test: images/test
names:
0: "sign"
1: "lane"
```

There is no `nc`; the trainer counts the mapping. `train` and `val` are both required by the
trainer, so a release published without a recipe - one undivided set - still declares `val` and
points it at the training images: that says "there is no held-out set", where omitting the key
would say "this file is malformed". `test` is written only when it has something in it. `path: .`
resolves against the working directory of the process that loads the file, so a training run
starts from inside the export directory. `images/` → `labels/` is a string substitution on the
resolved image path, not a configured location, so those two directory names are load-bearing.

**The task is derived from the release, never chosen.** One export is written for one of the
trainer's tasks:

- `segment` when the release holds any polygon - every polygon is written as its vertices and
every box as its four corners, so nothing located is reduced;
- `classify` when the release holds classification tags and no box or polygon - the export is
then the class tree the trainer reads, `<fold>/<class>/<image>`, one copy of an image per tag
it carries, with a directory for every tag-capable class whether or not anything used it, and
no `data.yaml`;
- `detect` otherwise - `class cx cy w h` per box, clamped into the image.

`supported_geometries` is `{bbox, polygon, classification_tag}` and `degraded_geometries` is
empty. A tag beside a located label has no layout to land in and is not written; `lossy = True`,
so consent is always asked. An asset with nothing on it gets an **empty** label file rather than
none: ultralytics reads a missing file as "nobody looked" and an empty one as "somebody looked and
there is nothing here", and a detector needs the second.

#### `yolov5-yaml`

The descriptor has no `path` key, every split path starts `./` and resolves against the yaml's
own directory, `nc` is an integer, and `names` is a list:

```yaml
train: ./images/train
val: ./images/val
test: ./images/test
nc: 2
names: ["sign", "lane"]
```

Detection only, always. `supported_geometries` is `{bbox}` and `degraded_geometries` is
`{polygon}`: a polygon is written as its axis-aligned bounding box, and the report counts it as
written in a reduced form and says which classes and how many. A classification tag has no
location and is dropped rather than given an invented box.

### The COCO format

Expand Down Expand Up @@ -578,7 +619,7 @@ prints that list without opening a workspace at all.

`--allow-lossy` is the third gate word, never folded into `--yes` or `--allow-destructive`. And
`dummy` writes nothing, so a `file_count` of 0 in its report is an export that ran, not one that
failed. The real ones are [YOLO](#the-yolo-detection-format), [COCO](#the-coco-format) and
failed. The real ones are [the YOLO dialects](#the-yolo-dialects), [COCO](#the-coco-format) and
[VOC](#the-pascal-voc-format) below.

When an export does leave something behind, the names go to **stderr** with the rest of the prose,
Expand Down Expand Up @@ -649,11 +690,11 @@ indistinguishable from a real recipe that said so.

**Export is queued**, and this document used to record the opposite. The limit was that
launch-and-poll needs a row to poll and a row needs a table; #328 gave the product a generic one,
so the argument expired. `yolo` writes one file per image and copies the pixels, which is minutes
so the argument expired. `ultralytics` writes one file per image and copies the pixels, which is minutes
of work behind a request with no way to report progress and every proxy's timeout in front of it.

```
POST /releases/{id}/export?format=yolo → 202 Accepted
POST /releases/{id}/export?format=ultralytics → 202 Accepted
Location: /background-jobs/{job_id}

GET /background-jobs/{job_id} → 200 { "state": "running", … }
Expand Down
2 changes: 1 addition & 1 deletion docs/content/releasing.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@ The acceptance criterion, and it is not satisfied by the upload succeeding:
cd $(mktemp -d)
uv venv && uv pip install --no-cache "visionset==0.0.1b2"
visionset --version # the version you tagged
visionset format list # nine rows: yolo, coco, voc, dummy and the five lane formats
visionset format list # eleven rows: ultralytics, yolov5-yaml, coco, voc, classification, dummy and the five lane formats
```

`format list` is the useful one: it reads installed entry-point metadata, so a non-empty answer
Expand Down
Loading
Loading