A crop-disease inference service on Kubernetes, where the engineering problem is keeping expensive GPUs busy without making a farmer on a bad connection wait for one.
Standalone project. Nothing else needs to exist for it to run.
| Milestone | What | State |
|---|---|---|
| M1 | Cluster, three node pools, CPU tier, gateway, observability, CI | Complete. Runs. Asserted in CI |
| M2 | Models, calibration, escalation rate measured | Complete. The measurement is below, and it is bad news |
| M3 | Router, the ladder, on-device tier | Partly built. The on-device tier and the web app run; the router service is still design |
| M4a | Balloon preemption, autoscaling on demand and queue depth, DRA allocation, GPU sharing | Complete. Everything measured. One measurement contradicted an ADR's reasoning; the ADR now says so |
| M4b | Real GPU node, cold start measured, MIG | Design only. Paid |
| M5 | Canary gated on model quality | Design only |
| M6 | Load, chaos, cost model | Design only |
Doc 06 section 5 says 8 to 15% of traffic should reach the GPU tier, and every cost claim depends on that. M2 was sequenced before the GPU work specifically so this number would arrive before any money did.
Measured escalation rate: 0.0%. That is not a success. Here is why.
Trained on 38,012 PlantVillage images, three epochs, then evaluated twice:
| Tier | Architecture | PlantVillage holdout | PlantDoc (field) | ECE lab → field |
|---|---|---|---|---|
| T0 device | mobilenet_v3_small | 98.76% | 16.05% | 0.057 → 0.257 |
| T1 CPU | mobilenet_v3_large | 98.72% | 16.12% | 0.060 → 0.395 |
| T2 GPU | efficientnet_b0 | 99.48% | 17.21% | 0.053 → 0.383 |
Holdout n=8,147. PlantDoc holdout n=1,290. Same weights, same 38-class label space, two evaluation sets.
The GPU tier is 1.2 points better than the phone on field images. That is what the most expensive compute in the system buys once the data stops looking like a laboratory.
Every routing decision in doc 06 is a comparison against the confidence score, so this is the plot that decides whether the ladder can work at all.
Two things worth noticing. In distribution the curve sits above the diagonal: the model is mildly underconfident on lab images. On field images it sits far below: it claims far more than it delivers. The ECE column above summarises this to one number and hides that the two failures are in opposite directions.
Doc 06 section 4 says to pick the lowest threshold whose covered subset meets the tier's accuracy floor. Each tier's unconditional accuracy (98.9%, 98.6%, 99.5%) already clears its floor (0.90, 0.93, 0.95), so the lowest qualifying threshold is zero. The device tier answers everything, and nothing escalates.
That is the blue curve above: entirely above the floor line, so no selectivity is ever required. The orange curve is the other half of the problem. It never reaches any floor at any coverage, peaking near 45% accuracy when the tier answers only its most confident 3%.
Selective prediction only does anything when the accuracy floor sits above the tier's unconditional accuracy. Below that, the procedure is degenerate and the ladder is inert. The design does not say this, and following it literally produces a three-tier system where the first tier answers everything.
The obvious fix is to choose thresholds on data that resembles deployment. So that was measured too, on a held-out PlantDoc split never used for calibration:
| Thresholds from | Evaluated on | tau | Escalation | End-to-end accuracy | |
|---|---|---|---|---|---|
| A | lab | lab | 0.00 | 0.0% | 98.76% |
| B | lab | field | 0.00 | 0.0% | 16.05% |
| C | field | field | 1.00 | 100.0% | 17.21% |
Both ends are degenerate. With lab thresholds nothing escalates. With field thresholds no threshold reaches any floor, so every tier defers and everything lands on the GPU.
Calibrating for deployment buys 1.2 accuracy points and costs a GPU pool sized for 100% of traffic.
The floors are policy, not measurements, and nothing in the design specifies them, so they are published as a curve rather than defended as a value:
| Floor | Lab escalation | Lab accuracy | Field escalation | Field accuracy |
|---|---|---|---|---|
| 0.80 – 0.97 | 0.0% | 98.76% | 0.0% | 16.05% |
| 0.99 | 0.0% | 98.86% | 22.8% | 17.44% |
Nothing moves at all until the floor exceeds what the models already achieve. At 0.99, sending 22.8% of field traffic to a GPU buys 1.4 accuracy points.
The ladder's premise is that the tiers are differently capable, so that escalating buys something. In distribution they are not differently capable enough to need a ladder: the phone model is already 98.8% and the escalation rate is zero. Out of distribution they are uniformly incapable: 16.1%, 16.1%, 17.2%.
There is no rung worth climbing to. The escalation mechanism is working correctly and delivering nothing, because it is routing between three models that are wrong in the same way.
This does not invalidate the Kubernetes design, which is the actual subject of this project. Queue-depth autoscaling, warm headroom and quality-gated rollouts are all still the right answers to the problems they address. What it invalidates is the cost argument for tiering on this data, and M2 exists precisely so that arrives before M4 provisions a GPU pool.
What would have to change for the ladder to earn its place: a top tier that is actually better on the traffic that reaches it. On this dataset pairing, no amount of threshold tuning produces one.
The design says GPU demand tracks image difficulty, not request volume. Every capacity and cost figure follows from it, so it was measured the same way the escalation rate was: with an experiment that could have said no.
Two phases against the same tier, starting from zero replicas both times.
| Phase | Images | Rate served | Escalation candidates | Woke the GPU tier |
|---|---|---|---|---|
| Volume | lab photographs | 15.9/s | 0 of 1,435 | No |
| Difficulty | field photographs | 4.0/s | 293 of 360 | Yes |
Four times the request rate produced no demand for the accelerated tier at all. A quarter of that rate, of harder images, woke it inside ninety seconds. A request-rate autoscaler would have scaled out hard during the first phase, when there was nothing to do, and sat idle during the second, when there was.
Preemption works, end to end, on real load. No manual scaling anywhere: the load produced escalation candidates, KEDA woke a tier that was at zero, the scheduler evicted a balloon, and a replica served traffic 2.7 seconds later. One balloon yielded and one kept its device, so the next spike still has a warm slot.
And the counterfactual saved nothing, which is the point. The same wake-up with the balloons removed took 3.3 seconds. The gap is smaller than the variation between two runs of the identical case, so it is noise, not a benefit. There is no node to provision on kind and the image is already local, so warm headroom removed a delay that was not there. The experiment is worth running because it says so with a number instead of asking to be believed. The real version needs a real cloud and is M4b.
The GPU node has no GPU. It advertises nvidia.com/gpu as an extended resource
with nothing behind it, which is a complete simulation of everything the
scheduler does with an accelerator and none of what one computes. Every pod
holding one says so through leafline_accelerator_simulated, and a CI policy
test refuses any device-holding pod that stays quiet about it.
Full numbers and method: docs/16-m4a-measurements.md.
A smallholder farmer photographs a diseased leaf. Slow, intermittent connection, cheap phone, and the answer is worth having in seconds.
Serving a vision model to that user is easy. Serving it to ten thousand of them, on GPUs that cost real money and take five to ten minutes to become available when you need another one, is not.
That gap is the project. The subject matter gives it a real offline requirement and a real latency budget. The engineering is Kubernetes.
The usual model-serving deployment is a container behind an HPA on CPU. It breaks twice here.
CPU utilisation says nothing about GPU saturation. An inference pod can have its accelerator fully committed while sitting at low CPU, because the CPU does preprocessing and the GPU does the work. HPA on CPU will never scale it.
Even a perfect signal is too slow. Node provisioning, driver install, image pull, model load into VRAM. Nothing in a request path absorbs that.
Full instructions, including what each command should print and what to do when something does not work, are in docs/14-running-it.md.
uv run poe tools && uv run python cluster/manage.py allCreates a four-node kind cluster on Kubernetes 1.34.8, installs KEDA and Envoy Gateway, builds and loads the service image, generates a local TLS certificate, and deploys everything. One command, from nothing to serving. Then:
uv run poe test:clusterTwenty-nine assertions against the live cluster.
http://127.0.0.1:18080/v1/classify |
The service, through the gateway |
https://127.0.0.1:18443/v1/classify |
The same, TLS terminated at the gateway |
http://127.0.0.1:13000 |
Grafana |
http://127.0.0.1:19090 |
Prometheus |
Host ports deliberately differ from the cluster NodePorts. The 30000-32767 range is contested on a developer machine, and a collision there presents as a connection that hangs rather than one that is refused.
Node pools. One control plane and three workers labelled system,
cpu-inference and gpu-inference. The GPU node carries the production taint
nvidia.com/gpu=present:NoSchedule and has no GPU, so taint behaviour is
testable at zero cost. It is labelled leafline.io/gpu-backing: simulated so
nothing can mistake it for the real thing.
The taint is asserted, not assumed. A probe pod without the toleration is confirmed to stay Pending for a taint reason, and a control test confirms the same pod schedules once it tolerates the taint. Without the control, the first test passes just as happily if the node is simply broken.
T1 serves a real model. FastAPI behind a Service, non-root uid 1001,
read-only root filesystem, all capabilities dropped, restricted Pod Security
enforced at the namespace. Bounded image decode with size, pixel and format caps,
because the input comes from the public internet. The ONNX export is baked into
the image, so a model rollout is an image rollout, which is what the M5 canary
compares. With no model present the service starts on a stub and says so through
leafline_model_is_stub.
Autoscaling is on a custom metric, not CPU. A KEDA ScaledObject drives a
generated HPA from a Prometheus query, with minReplicaCount: 2 because this
tier is what everything else degrades to. spec.fallback is configured so a
Prometheus outage holds replicas rather than reading "no metrics" as "no load".
T2 scales to zero and the balloons hold its node warm. The accelerated tier
starts at zero replicas and is woken by demand rather than by traffic: T1 counts
its own answers below tau_cpu, which is the work the ladder wanted to escalate,
and KEDA activates on that. Two balloon pods at negative priority hold both
devices meanwhile. They run the real inference image with a no-op command, so
preemption frees a node that has already pulled it, and their resource requests
cover an inference pod's on every dimension, so preemption frees enough of
everything and not just a device. Both of those are CI-enforced, because getting
either wrong leaves a mechanism that does nothing while reviewing clean.
Once a real replica arrives, the scheduler evicts a balloon and the replica serves. Asserted against a live scheduler, not inferred from the manifests.
The one measurement that contradicted a design document. ADR-0003 rejects GPU time-slicing in favour of MIG because "time-slicing has unpredictable tails", and its August amendment committed to reporting a local sweep that failed to show that. The sweep ran: from one process to two, throughput grew 1.21x, the median grew 2.01x, and the p99 grew only 1.44x. The tail ratio went down. The distribution shifted rather than spread, which is the well-behaved kind of bad.
The decision survives on a different argument, that 21% more throughput for 101% more median latency is a poor trade against a p95 SLO, and the ADR now says so. Two concurrency levels is thin evidence and the write-up says that too: three processes exhausted a 6 GB card, and time-slicing is proposed at densities up to 48.
DRA is verified, not assumed. ADR-0002 chose Dynamic Resource Allocation over
device plugins because a device plugin can only count: nvidia.com/gpu: 1 cannot
say "a partition with at least 30 GiB", so every pod takes a whole accelerator.
resource.k8s.io/v1 is GA in 1.34 and served by a stock kind cluster, and a DRA
driver's job is to publish devices as attributes in a ResourceSlice, which is an
ordinary API object. So a file stands in for the driver and publishes two
partitions that differ in one attribute. A CEL selector asking for the large one
gets the large one, asking for the small one gets the small one, and asking for a
profile no device has is refused rather than approximated. What a file cannot do
is register a kubelet plugin, so those pods are allocated a device and then never
start, and a test asserts that boundary too.
The GPU tier bounds its own concurrency, which is what creates the queue the
autoscaler reads. A device serves one batch at a time; the tier reproduces that
constraint with a semaphore and measures the wait. Without it leafline_queue_depth
would have been a constant zero and the primary scaling signal would have been
decorative.
The edge. Gateway API v1.6.1 through Envoy Gateway v1.9.0, on the system pool. One HTTPRoute, so every other path gets a 404 at the gateway and never reaches a pod. TLS terminated at the listener. Rate limiting is local rather than global, because global limiting needs Redis in the request path; the cost is that the limit is per proxy pod rather than per cluster.
Observability. Prometheus with annotation-based discovery, Grafana with a provisioned dashboard, both pinned to the system pool.
The on-device tier runs in a browser. web/ is a static page that loads the
6.5 MB ONNX model and classifies entirely in the tab, with no network. Turn wifi
off and it keeps working, which is what FR-L1 asks for.
uv run poe web # http://127.0.0.1:8000
When a cluster is running and the device tier is unsure it asks the CPU tier for
a second opinion. The badge changes to T1 · CPU tier, the model version becomes
the cluster's, and the timing line shows both hops:
When there is no cluster it answers anyway, marked degraded, with a note
written for a farmer rather than an operator. That is ADR-0005: running out of a
better tier is a quality event, not an outage. Verified by deleting the gateway
route and watching the page keep working.
Security is enforced and asserted, not described. Default-deny NetworkPolicy
on both ingress and egress, egress limited to DNS, no service account token
where the API is never called, non-root with a read-only root filesystem and all
capabilities dropped, restricted Pod Security at the namespace, a CORS
allowlist and rate limiting at the edge, and a Content-Security-Policy with no
unsafe-inline and no unsafe-eval anywhere. Trivy scans the image on every
push and CI emits a CycloneDX SBOM; the first scan found four MEDIUM CVEs, all in
a pip the runtime image had never used, and removing it took the report to zero
and removed a supported way for an attacker with code execution to fetch more
code. Eighteen tests fail the build if any of this is weakened. What is not done is listed just as plainly in
docs/13-security.md.
CI. ruff, mypy strict, pytest, kubeconform, the manifest and web security
policy checks, the browser logic under node --test, then a real kind cluster
and the scheduling assertions.
Everything here was measured. Everything not here is a target or an assumption, and is marked as such.
| Measured | Value | What it does and does not mean |
|---|---|---|
| Escalation rate | 0.0% both sets | Not a success. Every threshold selected was zero. See above |
| Accuracy, lab | 98.76 / 98.72 / 99.48% | PlantVillage holdout, n=8,147 |
| Accuracy, field | 16.05 / 16.12 / 17.21% | PlantDoc holdout, n=1,290 |
| Calibration error | 0.053–0.060 lab, 0.257–0.395 field | Underconfident in distribution, badly overconfident out of it |
| ONNX exports | 6.53 / 17.33 / 16.83 MB | All faithful: label agreement 1.000, max confidence delta ~1e-6 |
| On-device model size | 6.53 MB | Design claims ~4 MB quantised. FP32 is over it and quantisation is blocked, so the claim is unverified |
| Device tier latency | 6.1 ms/image | Desktop CPU, 400 images. A floor for a phone, not a prediction |
| T1 image size | 272 MB before the model | Multi-stage, no build toolchain in the runtime layer |
| Image vulnerabilities | 0 after removing pip | The first CI scan found 4 MEDIUM, all in a package installer the runtime never used |
| Tests without a cluster | 130 passing, 0 skipped | The GPU policy checks that used to be vacuous have subjects now and run for real |
| Tests against a live cluster | 29 passing | The above plus balloon preemption, a balloon refused the right to preempt, the no-idle-device invariant, scale-to-zero on one tier with a floor on the other, and DRA allocation selected by CEL |
| Manifests schema-validated | 37 resources, 0 skipped | Including the KEDA, Gateway API, Envoy Gateway and DRA types |
| M2 training run | 68.9 min | RTX 3050 6GB laptop, 3 epochs per tier |
| Demand from easy images | 0 of 1,435 | 15.9 req/s of lab photographs produced no escalation candidates and did not wake the GPU tier |
| Demand from hard images | 293 of 360 | 4.0 req/s of field photographs woke it inside 90 seconds. A quarter of the rate, all of the demand |
| Preemption to serving | 2.7 s observed, 3 s by the API | A balloon evicted and a replica Ready. A floor: the node existed and the image was local |
| The same, with no headroom | 3.3 s observed, 4 s by the API | The difference is inside the run-to-run noise. Headroom saves nothing here, and that is the honest result |
| DRA allocation by CEL selector | right device, every time | Large profile, small profile, and refusal when neither matches. Preparation needs a real driver |
| Time-slicing, 1 to 2 processes | throughput x1.21, p50 x2.01, p99 x1.44 | The tail ratio fell. ADR-0003's stated mechanism is not what happens at this density |
| T1 throughput, one replica | 15.9 to 18.6 req/s | Two replicas, real model, through a port-forward. Not a tuned figure |
Still not measured, and load-bearing:
| Assumption | Needed by | Milestone |
|---|---|---|
| GPU cold start is 5 to 10 minutes | Headroom sizing | M4b |
| Model load on a real accelerator | Whether balloons must pre-load the model | M4b |
| Queue depth beats CPU head to head, with numbers | ADR-0001 | M6 |
| DRA preparation on a node | Binding a real device into a container | M4b |
| The on-device model can reach 4 MB | The download a farmer pays for | M3 |
Found while building. All recorded in docs/12-reality-check.md and corrected in the original documents rather than worked around.
The two that would have deadlocked the system:
The router sheds requests when the estimated wait breaks the SLO, and the GPU tier scales on queue depth from a floor of zero. Together that deadlocks: with zero replicas the wait is infinite, so everything is shed, so nothing is enqueued, so depth stays at zero, so it never scales up. More generally, admission control caps the scaling signal at whatever the budget allows, so the autoscaler only ever sees admitted work and never sees demand.
Balloon pods were specified with the pause image. A pause container does not
pull the inference image, so image pull, one to three minutes of the cold start,
survives preemption. The balloon also requested memory but no CPU while the
inference pod requests two cores, so preempting one frees a GPU and not enough of
everything else. The mechanism would have done nothing while looking correctly
configured. There is a CI check for it now.
The two the measurement found:
Measuring the escalation rate on one in-distribution holdout answers the wrong question, because a lab-captured test split makes the small models look certain about everything.
Threshold selection is degenerate when the accuracy floor sits below the tier's unconditional accuracy, which is the 0.0% escalation rate above.
And one my own reporting code nearly published. verdict() originally
looked only at the escalation rate, and printed "within the assumed 8 to 15%,
the ladder saves what the design claims" for the field result. About a system
that is right one time in six. It now checks accuracy first and says the
escalation rate means nothing when the lower tiers are confidently wrong.
The four M4a added, three of them found by running it:
A pod that holds every accelerator cannot roll forward. A rolling update
surges before it retires, which needs a spare device, and on a full pool there is
none. For the balloons it deadlocks permanently, because preemptionPolicy: Never means a new balloon cannot preempt an old one either. The symptom is a
Pending pod, so it reads as a scheduling problem when it is a strategy one.
maxSurge: 0, now enforced by a policy test.
KEDA's cooldown and the HPA's stabilisation window are different delays that add up. Both were set to 300 seconds in the belief that they were the same knob. The tier sat at one replica for ten minutes after the last request, and the balloon it had preempted stayed Pending for all of it. On rented hardware that is double the idle bill and, worse, double the window with no headroom.
The edge rate limit made the load experiment impossible. 60 requests a minute is an M1 placeholder, and the first run had 3451 of 3600 requests refused with 429, so the volume phase never sent volume. It would have reported the right answer for the wrong reason. That placeholder is about one percent of what a single replica sustains.
Queue depth needed something to make a queue. The design picked it as the scaling signal without noticing the service had none: requests arrived, were classified, and left. A queue forms only behind a bound, so the accelerated tier now bounds itself to one in-flight inference, the way a device does.
The rest: MIG cannot be validated on consumer hardware and the ADR needed a real evidence plan; the readiness-probe policy check was the wrong control and is now a startupProbe check; the router trusted client-supplied confidence and discarded the answer it needed to degrade to; plus naming and QoS corrections.
- Not a diagnosis. Decision support. Every answer carries a confidence and its tier, and anything actionable is flagged for confirmation.
- No treatment or dosing advice. That is agronomy and regulation.
- Not a machine learning project. The model is the workload. Three epochs of transfer learning, no tuning, frozen. When ONNX quantisation blocked, that was recorded and left rather than chased.
- Not multi-cloud. One cluster done properly teaches more than a portable abstraction over three.
The 16% figures are not a claim that this task is impossible. They are what transfer learning on lab images gives you on field photographs with no domain adaptation, no field training data, and a deliberately frozen model. A system built to work in a field would train on field images. The point of measuring it is that the design's cost argument assumed the tiers would differ, and on this data they do not.
PlantDoc covers 28 of PlantVillage's 38 classes, so field evaluation is restricted to that intersection. Its smallest class has two images, so per-class figures there are noisy.
The local cluster cannot demonstrate node provisioning, so the headline cold-start figure stays unmeasured until a real GPU node exists. The development GPU is consumer hardware and cannot do MIG at all.
web/ T0. Static page, on-device ONNX, works offline
services/infer-cpu/ T1. FastAPI, ONNX or stub, bounded image decode
libs/leafline-core/ Response schema. Enforces the tier and degradation contract
models/ Datasets, splits, calibration, the escalation measurement
cluster/ kind config, the driver script CI also uses, and the experiments
deploy/ base, cpu-tier, gpu-tier, dra, gateway, observability
tests/unit/ schema, decode, classifier, serving path
tests/models/ calibration, escalation, splits
tests/policy/ manifest policy checks from docs/09 section 2
tests/cluster/ assertions against a live API server
tools/ pinned binaries, checksum-verified
docs/ the design package, the reality check, and the measurements
Cloning it and running it from nothing:
docs/15-from-a-clean-clone.md. Reproducing M2:
models/README.md, about 70 minutes on a laptop GPU, no
credentials. Reproducing M4a: uv run poe m4a, about 25 minutes, needs only
Docker.
| 00 Index | Reading orders, claims, ADRs |
| 01 Research and analysis | Kubernetes GPU landscape, cold-start numbers, the domain |
| 02 PDD | Goals, non-goals, users, scope, risks |
| 03 Requirements | Functional and non-functional, acceptance criteria |
| 04 HLD | Cluster topology, request path, scaling, rollouts |
| 05 LLD | Manifests, resource specs, router logic, tests |
| 06 Inference ladder | Core contribution one |
| 07 GPU capacity and scaling | Core contribution two |
| 08 Roadmap | Milestones and estimates |
| 09 Test, load and chaos plan | Policy checks, measurements, chaos |
| 10 Cost model | Cost per correct diagnosis |
| 11 Interview pitch | Pitch, demo, likely questions |
| 12 Reality check | Where the design was wrong, and what is still unverified |
| 13 Security | What is enforced and tested, what is unverified, and what is not done |
| 14 Running it | Every command, in order, with what it should print |
| 15 From a clean clone | Start here. Prerequisites, Docker, every command, and what breaks first |
| 16 M4a measurements | Preemption, autoscaling on a signal that is not CPU, DRA allocation, and what a cluster with no GPU can prove |
| ADRs | Six decision records, four since amended |