feat(scripts): pack, verify and document the KITScenes corpus for local training

[gcordova10]

Problem

Training KITScenes outside the cluster is harder than it should be, and #168 shows
the cost: several of us have trained on different ad-hoc subsets, and the metrics
posted so far cannot be compared with one another.

Three things stand in the way.

The corpus does not fit. The archives are ~3 GB each; extracting the 533-scene
train split needs well over 2 TB, and the archives themselves are ~1.6 TB. Two
contributors have reported this in the thread, along with the fact that streaming
the download does not help, since each archive still lands on disk before packing.

Mistakes surface late. train_il validates a packed corpus against the frozen
split named by KITSCENES_TRAINING_POLICY.validation_manifest and aborts on any
mismatch — but after start-up, once every shard has been scanned. On a corpus that
took hours to pack, that is an expensive place to learn the dataset version was
wrong. And the snapshot moves: it was re-cut to v3.3 while this branch was open, so
a corpus packed against the previous one now has to be repacked.

The setup is not written down. What a contributor needs before the first epoch
lives across issue comments, a Dockerfile, and a couple of error messages.

Fix

Platform/scripts/pack_kitscenes_corpus.py — packs one scene at a time and
deletes as it goes. With --fetch it downloads each archive too, so nothing
accumulates: peak disk is one archive plus one extraction, roughly 8 GB for the whole
corpus instead of 1.6 TB. Measured over the complete train split — 533 partitions, 404
of them non-empty, 42,667 samples — the packed output is 252 KB per sample and 10.2 GB
in total. Resumable, and it stops on its own after three consecutive failures or below
a free-space floor.

Platform/scripts/verify_kitscenes_corpus.py — runs the eight provenance checks
train_il performs, before the run rather than during it, and reports each one
separately. Tar headers and manifest.json only; no camera payload decoded.

Docs/training_on_a_local_machine.md, linked from the README — the prerequisites
that are not in requirements.txt (the KITScenes SDK, its numpy<2.0 constraint,
kubernetes being needed at import time), dataset access, getting and packing the
corpus, which two filters drop scenes and the log line that reports all three counts,
generating the audit, running the training, and what to report so results line up.

Neither tool names a manifest or a version. Both read the manifest the training
policy points at, and the packer takes its dataset version from that file. A checker
pinned to one snapshot would keep passing a corpus training has already started
rejecting, which is the failure it exists to prevent.

Two behaviours are encoded in the scripts because they are silent when wrong: -C is
positional in GNU tar, so trailing it returns 0 while extracting into the working
directory; and data_processing defaults to DATASET_PACK_VERSION while the
KITScenes navigation path uses KITSCENES_NAVIGATION_DATASET_VERSION, which is what
the frozen manifest carries.

Verification

• ruff check and cd Model && mypy . clean — the same commands as CI.
  pytest Platform/tests — 38 passed, 16 of them new: the -C trap, a propagated tar
  failure, partition scanning, each frozen-manifest check failing independently, and
  three that pin both tools to the policy's manifest rather than to a snapshot.
  pytest Model/tests — 914 passed; the 3 failures in test_projection.py are
  pre-existing on upstream/main in this environment (physical_ai_av not
  installed) and reproduce identically on a clean checkout of it.

• Platform/tests is not part of make test, which runs Model/tests, so these do
  not gate CI. They sit next to the two existing Platform/tests files covering the
  other scripts in that directory. They do run from a bare pytest Platform/tests at
  the repository root — that needed a conftest.py putting Model/ on the import
  path, since Model/tests gets that from its own pytest.ini and __init__.py and
  this directory has neither. Adding Platform/tests to make test is a one-line
  Makefile change and all 38 pass without PYTHONPATH, but that is your call, not
  something to slip into a corpus-tooling PR.

• The complete train split was packed by this branch's script, on a laptop, and
  verified 8/8 by this branch's checker. The environment was built from nothing by
  following the page in this PR — no venv, no SDK, no data at the start. 533
  partitions fetched one archive at a time over a USB disk, 63 hours end to end.
  verify_kitscenes_corpus.py then passed every comparison against
  kitscenes_train_dev_v3.json:

  [PASS] packed partitions       533   [PASS] samples             42667
  [PASS] empty partitions        129   [PASS] sample UID digest   d169a7ac…
  [PASS] eligible groups         404   [PASS] dataset version     v3.3
  [PASS] group UID digest   cadcdb21…  [PASS] contract digest     6fb9d857…
  
  == corpus matches the frozen split.
     Holdout: 40 scenes / 3820 samples, group digest 903fec7d…
  

  That is the same holdout Align KITScenes planning loss with rollout evaluation and composite checkpoint selection #176 reports against, so metrics from this corpus are
  comparable with the ones already in the thread.

• Two things the run found, both now in the page. Packing needs cv2, which is in
  neither requirements.txt nor the previous version of this page, and without it
  every partition fails after downloading its 3 GB archive. And a corpus packed
  before the v3.3 re-cut fails exactly two checks — dataset version and contract digest — while counts and group digests still pass, which reads like a corpus
  problem and is not one.

• Trained on, too. One epoch over the verified corpus completed on the same
  laptop — 2 h 20 min, 3.49 samples/s, checkpoint uploaded to a local MinIO and
  registered under its best/final roles. With one caveat that belongs in this
  bullet: the 6 GB card cannot hold the default 256x256 BEV grid, so that epoch used
  the companion PR's --camera_bev_size 64. On main as it stands, a card this
  small still needs the code edit this page documents. Everything the page claims
  about steps 1 to 6 is independent of that.

Only the README line touches an existing file.

Companion PR: #188 exposes the camera BEV grid as a parameter. This page
documents that the default grid does not fit in 6 GB and that the workaround today is
editing the code; that PR is what removes the need to. They are independent — each is
green on its own and neither depends on the other landing.

[riita10069]

I don't think this PR is the right direction, and I don't think it should be merged in its current form.

The core premise seems to be that KITScenes packing and dataset provenance are insufficiently fixed today, but those invariants already exist in the current pipeline:

• The KITScenes source revision is already pinned to a specific commit.
• The KITScenes pack version is already explicitly pinned to KITSCENES_NAVIGATION_DATASET_VERSION (v3.3) by the sharded workflows.
• The train/dev split is already frozen through KITSCENES_TRAINING_POLICY and kitscenes_train_dev_v3.json, including the source revision, dataset version, contract digest, sample inventory, and validation groups.
• Packing itself already exists as data_processing, and KITScenes is already partitioned and fanned out scene-by-scene through the workflow.

So this PR does not establish a missing dataset contract. Instead, it introduces a second orchestration path outside Pipelines that calls the existing data_processing implementation directly and independently reimplements the surrounding lifecycle.

I think that is the wrong abstraction.

If the actual missing capability is "run the existing KITScenes packing path on a local machine without requiring the cluster, while keeping local disk usage bounded", then that should be implemented as a thin local execution adapter around the existing sharded data-preparation path. We should not create and maintain a parallel orchestration mechanism with its own fetching, retry, resume, cleanup, verification, and version-resolution behavior.

Having two paths for producing what is supposed to be the same canonical packed corpus increases, rather than decreases, the risk of divergence over time.

The documentation has the same issue. training_on_a_local_machine.md is nearly 800 lines and attempts to document environment construction, SDK installation quirks, Hugging Face authentication, corpus packing, GPU memory workarounds, local S3, MLflow, training, result reporting, and several implementation-specific failure modes in one document. That is far too much operational state to duplicate in a single PR, and much of it describes the current implementation rather than a stable user-facing interface.

Documentation for local training should describe the small delta between the supported pipeline and local execution. If reproducing the pipeline locally requires an 800-line runbook, that is evidence that the interface should be simplified rather than that the runbook should be merged as-is.

While I appreciate that you're setting up the foundation for implementing learning, it would be helpful if you could align with the project's existing assets.

[gcordova10]

That is a fair objection and I agree with the architectural part: a second orchestration path around the same canonical corpus is a maintenance risk, and a thin local adapter over the existing sharded path is the better shape. I am not going to be able to take that refactor on, so I would rather not leave a half-answer in the tree.

One clarification in case it changes anything, then I will follow whatever you prefer. The premise was not that provenance is unfixed — it is, and the PR relies on that. It was that the corpus does not fit locally (the archives are ~1.6 TB, extraction well over 2 TB) and that the contract is checked after start-up rather than before. The script calls data_processing rather than reimplementing it; what it adds is the surrounding lifecycle, which is exactly the part you are saying should not be duplicated.

On the documentation: you are right that 800 lines is describing the implementation rather than an interface. If a short version is still useful — the delta between the supported pipeline and local execution, without the failure catalogue — I can cut it down and drop the scripts entirely. Otherwise I am happy to close both.