Local MLOps platform that packages, deploys, monitors and rolls back an ML inference service on a single-node kind cluster. An engineer supplies a model bundle and runs one release command; the platform validates the candidate against a fixed release-validation set, deploys it blue/green behind a preview Service, runs correctness and load gates, promotes it, watches Prometheus for the verification window and rolls back automatically if latency or error thresholds are breached.
Measured on Apple M5, 24 GB, macOS 26.6.2 (arm64), kind v0.33.0 go1.27.0 darwin/arm64, Kubernetes v1.37.0; 2 replicas; paced load at 100 req/s from 8 clients; thresholds from benchmarks/thresholds.yaml. Every number below is read from the JSON records by modellaunch summarize.
| Run | Outcome | Measured |
|---|---|---|
20260913T234851Z-release-tiny-1.0.0 |
Released | 165 s from validate to stable, 12 gates passed; accuracy 0.943 >= 0.9; preview p95 4.8 ms; watch p95 4.5 ms over 24 Prometheus samples |
20260913T235450Z-rejected-tiny-1.1.0 |
Rejected at validating | accuracy 0.425 against a 0.9 gate on the fixed release-validation set; no image built, no pods created, stable stays tiny-1.0.0 |
20260913T235238Z-rollback-tiny-1.2.0 |
Automatic rollback | candidate turned slow 45 s after start (slow fault); p95 868 ms breached the 250.0 ms gate on 2 consecutive samples; tiny-1.0.0 serving again 874 ms after detection (fault to detection 15.3 s at 5 s scrape granularity) |
20260913T235451Z-release-flowbench-0.1.0 |
Released (FlowBench CNN), replacing tiny-1.0.0 | 155 s from validate to stable, 12 gates passed; relative_l2 0.466 <= 0.55; preview p95 25.7 ms; watch p95 23.8 ms over 24 Prometheus samples |
Failure drills (make drills, run 20260913T235137Z-drills), written up in docs/INCIDENT_REPORT.md:
| Drill | Result | Measured |
|---|---|---|
| Kill one replica under continuous load | passed | Deleted inference-tiny-1-0-0-566fcb77b7-2msgg at t=15.0s under 94.58 req/s. 0 of 5675 requests failed (none). p95 before/10 s after kill/steady: 6.048 / 7.576 / 7.493 ms. Readiness returned to 2/2 3.0s after the delete. |
| Automatic rollback after a latency fault | passed | tiny-1.2.0 was promoted, then p95 rose to 868 ms after the 45 s fault delay; breach confirmed on 2 consecutive samples (15291 ms after the fault was first observed). Stable reverted to tiny-1.0.0; first healthy response 874 ms after detection. Status: rolled_back. |
| Invalid input to /predict | passed | 7 malformed bodies all answered 422 with a JSON pointer message; /live 200, /ready 200, a valid request 200 afterwards. Prometheus 422 counter rose by 7. |
| Missing weights in a bundle | passed | Release validation rejected the bundle in 0 ms (artifact_integrity: BundleMissingFileError: bundle file 'model.onnx' is missing from artifacts/bundles/tiny/1.0.0-broken.1). Deployed anyway: the container ran (liveness ok, 0 restarts) but never became ready; the readiness gate timed out after 31.0s and the candidate was deleted. The preview Service had no endpoints. |
| Failed quality gate | passed | Candidate tiny-1.1.0 rejected at validating: quality_accuracy 0.425 >= 0.9. Images for it before/after: 0/0; inference deployments before/after: 1/1; stable stays tiny-1.0.0. |
Grafana dashboard provisioned from JSON, under 100 req/s of paced load against the released FlowBench model (make bench, p95 22 ms client-side over 8,715 requests):
The three records replay in a static console: site/index.html, published
on GitHub Pages at https://abdulalimamnun.github.io/modellaunch/ once Pages is enabled
for the repository. It has a record picker, the six-step gate pipeline, the blue/green route
panel, the p95 and error-rate chart drawn from the Prometheus samples against the declared
gates, the event log and the release contract, all read from the committed records. The
playback speed control compresses the replay and is labelled as such; every number on the
page is a recorded measurement. Query parameters select a record and jump to its end, e.g.
index.html?run=rollback&end=1.
Requires uv, colima, docker, kind and kubectl
(brew install colima docker kind kubectl uv, then colima start --cpu 4 --memory 8 --disk 40).
Python 3.12 is pinned in .python-version. Verified from a fresh clone on 2026-09-13.
make setup # uv sync --all-extras + pre-commit hooks
make lint test # ruff, mypy --strict, 141 unit tests (about 7 s)
make train-tiny train-tiny-degraded train-tiny-next # three bundles, about 1 s each on CPUThe inference image serves the bundle with onnxruntime behind FastAPI: /live as soon as
the process binds, /ready only after the bundle hash is verified and a warm-up prediction
has run (0.7 s after docker run on this machine), /predict validated against the
bundle's JSON Schema, /metrics for Prometheus. Full command list, environment variables
and the FlowBench packaging steps: docs/OPERATIONS.md.
make up # kind cluster + Services + Prometheus + Grafana, about 60 s
make release MODEL=artifacts/bundles/tiny/1.0.0 # gated release, about 2.5 min including the 120 s watch
make release MODEL=artifacts/bundles/tiny/1.1.0 SCENARIO=rejected
make release MODEL=artifacts/bundles/tiny/1.2.0 SCENARIO=rollback FAULT_MODE=slow FAULT_DELAY_S=45
make drills # five failure drills, writes docs/INCIDENT_REPORT.md (about 4 min)
make export-site # newest record per scenario into site/records/
curl -s localhost:30080/ready # stable Service via NodePort
open http://localhost:30030 # Grafana dashboard, provisioned from JSON
make downEach release writes releases/<run_id>.json (schema: releases/schema.json). Every
number in it comes from the bundle check, the load generator, kubectl or Prometheus.
Thresholds are declared in benchmarks/thresholds.yaml before any run.
A bundle directory holds manifest.json, model.onnx, preprocess.json,
schemas/request.json, schemas/response.json and evaluation.json. The manifest
records name, semantic version, training-code git revision and a SHA-256 over the payload
files; identical training runs produce identical hashes. Release gates use the committed
release-validation set in benchmarks/validation/ against the thresholds declared in
benchmarks/thresholds.yaml; the final test split is never read by the platform.
Design decisions with rejected alternatives: docs/DECISIONS.md.
flowchart LR
subgraph host [Host: Python 3.12 via uv]
B[bundle: manifest, ONNX, schemas, hash] --> K[k8s wrapper: docker, kind, kubectl]
R[release workflow: state machine, gates, record] --> REC[(releases/*.json)]
L[load generator] --> R
end
subgraph cluster [kind cluster in Colima]
S[(inference-stable :30080)] -. selector .-> BLUE[Deployment vN, 2 replicas]
P[(inference-preview :30081)] -. selector .-> GREEN[Deployment vN+1, 2 replicas]
PROM[Prometheus :30090] -. scrape .-> BLUE
PROM -. scrape .-> GREEN
PROM --> GRAF[Grafana :30030]
end
K -- build, load, apply --> GREEN
R -- selector patch --> S
R -- selector patch --> P
R -- PromQL --> PROM
REC --> SITE[site/: release console]
Details, the state machine and the bundle and record contracts: docs/ARCHITECTURE.md. Every command, copy-pasteable: docs/OPERATIONS.md. A two-minute walk-through: docs/DEMO.md.
- The workflow enforces the gates, not Kubernetes. Readiness probes decide whether a pod gets traffic; the quality, correctness, load and watch gates decide whether a version does. Promotion and rollback are a selector patch on one Service, which is why both take a few hundred milliseconds.
- The serving image never imports PyTorch. Bundles carry an ONNX graph plus a declarative preprocess spec; the 96 MB image runs onnxruntime behind FastAPI and is ready 0.7 s after start.
- Identity without a registry. The image tag encodes the bundle's SHA-256 and the platform's git revision; the Docker image ID and the pod's runtime image ID are recorded. Deployments never reference
:latest. - Thresholds were declared before any run in
benchmarks/thresholds.yaml, are copied into every record, and are the same numbers the Prometheus alert rules and the Grafana threshold lines use (tests check all three agree). - The release-validation set belongs to the platform. A candidate is scored on
benchmarks/validation/<model>.npz, never on anything it ships, and never on a model's final test split. - Honest timing. Rollback duration is breach detection to first healthy response from the previous version on a fresh connection; the time from fault to detection is reported separately with its 5 s scrape granularity. Keep-alive clients follow old pods until they reconnect, and the records say so.
- Every non-obvious choice, with the alternative rejected: docs/DECISIONS.md.
| Phase | Scope | State |
|---|---|---|
| 1 | Scaffold, bundle format, tiny model, unit tests | done |
| 2 | Inference service (/predict, /live, /ready, /metrics), container |
done |
| 3 | kind cluster and manifests | done |
| 4 | Release workflow and release records | done |
| 5 | Prometheus and Grafana as code | done |
| 6 | Load tests, failure drills, incident report | done |
| 7 | Release console (GitHub Pages) | done |
| 8 | FlowBench checkpoint released through the same workflow | done |
| 9 | Fresh-clone reproduction, README polish, demo script | done |
MIT.