Model Info

A production health check for the RankShift Serving engagement model: a FastAPI service that reports what is actually loaded in memory, and a Streamlit dashboard that answers three questions in five seconds — is the right kind of artefact deployed, does it have the feature width the last release had, and did it load at all.

Built to docs/SPEC.md. 70 tests.

Three of four in a series: Single Prediction → Batch Prediction → Model Info → Analytics Dashboard

---

Contents

• The problem it solves
• Quick start
• How to read the page
• What the chart is telling you
• Failing well
• The model
• Repository layout
• Verification
• Limitations
• The series

---

The problem it solves

A model deployment can fail in ways that no error surfaces. The service starts, /health says 200, predictions come back — and the artefact is last month's, or the feature join dropped twenty columns, or preprocessing is being applied by a copy of the training code that no longer matches. Nothing throws. The probabilities are simply wrong, quietly, until someone notices a business metric moving.

Three numbers catch most of that, which is why they are the first thing on the page:

Model Type        Feature Count        Version
Pipeline          54                   1.0.0
   │                  │                    │
   │                  │                    └─ is this the release you shipped?
   │                  └─ 54 last time, 32 now? the feature join is broken
   └─ not `Pipeline`? preprocessing is not in the artefact, and can drift

Quick start

With Docker:

docker compose up --build

Or locally:

python -m pip install -r requirements-dev.txt
python -m pip install -e .

Start the API:

uvicorn model_info.api:app --reload

Start the dashboard in a second terminal:

streamlit run app.py

Then open http://localhost:8501. The screenshot above is that page, captured unedited. Point the dashboard at a different API with MODEL_INFO_API_URL.

How to read the page

                    ┌── GET /model/info ──────→ cards · status · sample features
app.py (Streamlit) ─┤
                    └── GET /metrics/features ─→ Top-20 table · bar chart

api.py (FastAPI) ── start-up ──→ best_model.joblib      (the Pipeline)
                             └─→ model_metadata.json    (cached coefficients)

The two blocks are fetched separately and render their own errors. Neither holds a model, and neither computes an importance: what you see is exactly what an API client would get.

Feature Count is the raw width — the 54 columns fed into the ColumnTransformer, not the 182 it expands them into. That distinction is the point of the card: the expanded width moves on its own with the training data's category cardinality, so comparing it across releases would produce false alarms. The raw width only changes when the feature builder does.

What the chart is telling you

Look at the bars. One is two and a half times the next, and the rest are one-hot levels of boolean video flags stacked at a magnitude that is visibly noise.

num__watch_time	0.2307
Largest of the other 181	0.0885
Median of the other 181	0.0197
Coefficients below 0.05	153 of 182

SinglePrediction reached that conclusion by measurement — 25 candidate features screened, 2 kept, the other 23 indistinguishable from noise. This project fits a model on 54 of them and puts every coefficient on screen, and the shape of the chart is that finding.

Which is the honest description of this artefact: a deliberately wide model serving a narrow signal. It is wide because the page is an instrument, and an instrument that can only ever display two rows cannot tell you when a feature pipeline has broken. Full reasoning in docs/MODEL.md.

Failing well

Two failure modes, and neither produces a traceback.

The API is unreachable. Each block reports its own failure, and the page still renders:

The model did not load. /model/info answers 200 with status: not_loaded rather than 500. A 500 would take the status field off the page at the exact moment a reader needs it, so the route is written not to raise. The load reason is sanitised before it leaves the process — the client sees FileNotFoundError while loading the model artefacts, and the path it tried goes to the log (NFR-04).

Under that sits a three-step fallback chain for feature importance: the cached top_coefficients block, then a live read of the fitted Pipeline, then an empty list with a message. Every step logs at WARNING with the reason it fired, because a fallback that fires silently is a fallback nobody knows is load-bearing. The response names which step answered in its source field.

The model

Pipeline(ColumnTransformer(numeric → impute+scale, categorical → impute+one-hot), LogisticRegression). 54 raw features, 182 expanded, ROC-AUC 0.5632 on a chronological hold-out.

Every feature is servable — derivable from user_id, video_id, watch_time and a clock, plus the user and video snapshot tables. The five outcome flags that define the label, engagement_score (ROC-AUC 0.836 on its own), and everything observed during an interaction are excluded before measurement, following SinglePrediction's feature-selection discipline.

Retrain with python -m model_info.train. Details in docs/MODEL.md.

Repository layout

app.py                          Streamlit dashboard — no model, no data
src/model_info/
  api.py                        FastAPI: /model/info, /metrics/features, /health, /predict
  introspection.py              The NFR-03 fallback chains, HTTP-free and unit-tested
  features.py                   The 54-column raw schema and the join that builds it
  train.py                      Fits the Pipeline, caches the coefficients
  config.py                     Paths and limits, all environment-backed
tests/                          70 tests, organised by requirement
docs/
  SPEC.md                       The specification this was built to
  TRACEABILITY.md               Every FR and NFR → implementation → evidence
  API.md                        Endpoint reference
  MODEL.md                      Why a wide Pipeline, and what the chart shows
  DEPLOYMENT.md                 Docker, configuration, post-deploy runbook
  PRODUCTION_READINESS.md       What it would need to carry real traffic

Verification

python -m pytest -q

70 tests, no network, running against the real shipped artefact. Organised by requirement rather than by module, because requirements are what a reviewer is checking — see docs/TRACEABILITY.md for the full map.

Latency, measured over 300 sequential calls after warm-up:

Route	Budget (NFR-01)	p95
/model/info	≤ 200 ms	0.94 ms
/metrics/features	≤ 500 ms	0.93 ms

Both read state resolved at start-up. Nothing touches the disk on the request path.

Limitations

This section lists what is known to be missing or imperfect in what was built. A wider account — what this service would need before it carries real traffic, ordered by risk, with the cost of each remedy — is in docs/PRODUCTION_READINESS.md.

• Coefficients are not causal. cat__is_premium_true at −0.0885 says the fitted linear model assigns premium users slightly lower odds given the other 53 features, not that premium status suppresses engagement.
• One-hot magnitudes need their support checked. The numeric block is standardised so its coefficients are comparable; one-hot columns are not, and a rare level can carry a large coefficient fitted on very little data. min_frequency=0.01 blunts this rather than removing it.
• The page reports what loaded, not what is correct. Model Type: Pipeline and Feature Count: 54 confirm the artefact's shape. They cannot tell you the weights inside it are the ones you trained.
• No model-version history. Version comparison is a manual act: read the card, remember the last release. Storing a series is the obvious next step and is not in this specification.

---

The series

Four repositories, read in this order, are one product line: score one, score many, check what is deployed, then watch it in production.

1. Single Prediction — one prediction per request — feature selection, model choice, calibration and the operating point
2. Batch Prediction — up to 100 rows per call, with per-row fault isolation
3. Model Info (you are here) — what is actually loaded in memory, and what that tells you
4. Analytics Dashboard — traffic and model-output monitoring over a request log

Each repository runs on its own. The cost of that is stated plainly in each Limitations section: features.py, the API skeleton and the model artefact are duplicated across all four.