The one system this course builds. Your team creates this repository once from the template, and every lab from week 2 to graduation is a change to it. There is no week where you start again.
app/ empty. Your service goes here, starting week 2 day 2
docs/ the API contract the Agentic AI cohort integrates against
scripts/ verify-env.sh, which checks your machine against what the labs need
PINS.md every version this course depends on
setup.md how to work in this repository
That is the whole repository, and the shortness of that list is the point. You are not given a finished system to read. You build one, a day at a time, and by week 6 another cohort's agents are calling it.
| Week | Day | What you add |
|---|---|---|
| 2 | Mon | app/ behind an OpenAI-compatible /v1 on CPU |
| 2 | Tue | Dockerfile, and your image on Docker Hub |
| 2 | Wed | Dockerfile.gpu, the same code on a GPU |
| 2 | Thu | compose.yaml, the stack described rather than run by hand |
| 3 | Thu | bench/, the harness that measures all of it |
Each one is a lab, and each one starts from files that day hands you. Lab instructions, decks and quizzes are on the course Drive, one folder per week. This repository is your code.
./scripts/verify-env.sh # checks your machine, writes verify-env-report.jsonThen read setup.md. It is short, and it covers the two things that go wrong:
committing a key, and committing a model.
Measured on a Colab T4, Qwen/Qwen2.5-1.5B-Instruct, 128 generated tokens.
| dtype | predicted GB | measured GB | observed bytes/param | tokens/s |
|---|---|---|---|---|
| fp16 | 3.0 | 3.29 | 2.19 | 31.9 |
| int8 | 1.5 | 1.87 | 1.25 | 6.0 |
| int4 | 0.75 | 1.24 | 0.83 | 15.3 |
Three things the numbers say that the formula does not:
- Measured always exceeds predicted. Parameters times bytes is weights only. The CUDA context, the framework and the activations are the rest, and they do not shrink when the weights do.
- The quantised rows overshoot hardest. int4 predicted 0.75 GB and measured 1.24 GB, so observed bytes per parameter is 0.83 rather than 0.5. Quantisation is applied to most of the weights, not all of them, and the fixed overhead is now a larger share of a smaller number.
- int8 is slower than int4, and both are slower than fp16. Smaller weights do not mean faster. bitsandbytes int8 dequantises on the fly through a path with no fused kernel, so it saves memory and costs speed. That gap is the plant for week 3 Thursday: the bits were never the problem, the kernels were.
Files: generate.py, results.json.
.dockerignore, Dockerfile, and Dockerfile.naive at the repo root.
docker build -t <user>/aidc-serving:cpu-v1 .
docker run --rm -p 8000:8000 -v hf-cache:/home/app/.cache/huggingface \
<user>/aidc-serving:cpu-v1The slim image lands near 1.6 GB. Build Dockerfile.naive alongside it and
read the gap off docker images yourself; that number is your card, not ours.
Four decisions in the Dockerfile, each with a reason:
python:3.11-slim, notpython:3.11. You do not need build toolchains at run time for these wheels.requirements.txtcopied and installed BEFOREapp/. Requirements change rarely, code changes constantly. Separate layers mean a code edit reuses the cached install instead of re-downloading torch every build.--index-urlon the CPU wheel index. Without it pip takes the default CUDA build of torch and the image is roughly 6.5 GB instead of 1.6.chownbeforeUSER app. A volume mountpoint that Docker auto-creates is root-owned, so a container that has already dropped privileges cannot write the first model download into it.
The weights are not in the image, and that is the rule the whole day is shaped around. They arrive at run time into the mounted cache volume, which keeps the image generic: the same image serves any model id.