RankShift

What survives when the exposure policy changes — measured on real logs with a randomized gold standard.

Live page · Experiments · Reproduce · Serving stack

Offline metrics are computed on logs shaped by the very policy being replaced. KuaiRand contains two weeks in which Kuaishou inserted uniformly random videos into live feeds — a genuinely missing-at-random test set. RankShift uses it to measure, rather than assume, what transfers across an exposure shift, what breaks, and which estimators can tell the difference.

Headline

• Ranking transfers; calibration does not. Lift at the top 23.9% of traffic holds across exposure policies (3.43× → 3.36×, −2%); mean predicted ÷ actual probability goes 0.96× → 2.07×. A team validating offline would watch the ranking hold and ship probabilities wrong by a factor of two.
• No single calibrator fixes both. Fitted on randomized traffic it ships there and is rejected on served traffic by the same factor in reverse. Calibration is a property of the exposure policy, not of the model.
• Off-policy estimators, graded against the gold standard, all miss with item-level propensities (IPS 3.00×, DR 3.81× off — bias, not variance). The exposure bias decomposes 3.80× = 1.25× video selection × 3.04× user–video matching; only propensities logged at serving time capture the second term.
• A PyTorch two-tower ties LightGBM at feature parity (AUC 0.8815 vs 0.8811); the pure retrieval tower reaches 0.8246 with no watch signal.
• Everything replicates at 322M interactions, scored in full and out of core, where the calibration break grows to 2.29×.

The six experiments

#	Question	Key number	Script → result
01	Does the upstream "signal ceiling" hold on real logs?	24 of 37 features carry signal; watch_ratio alone 0.7486 AUC	01_signal_probe.py → results/01_signal_probe.json
02	What survives the shift from algorithmic to random exposure?	lift 3.428× → 3.363×; pred÷actual 0.96× → 2.07×	02_exposure_bias.py → results/02_exposure_bias.json
03	Can a calibrator fitted on one policy serve the other?	ECE ↓20× on randomized; 0.961× → 0.481× on served	03_recalibrate.py → results/03_recalibrate.json
04	How wrong are IPS / SNIPS / DM / DR without logged propensities?	all 3.0–3.9× off; 3.80× = 1.25× × 3.04×	04_off_policy.py → results/04_off_policy.json
05	Does a neural ranker beat the GBDT? What does personalization alone carry?	0.8815 vs 0.8811; pure tower 0.8246	05_two_tower.py → results/05_two_tower.json
06	Do the findings survive 120× more data?	322M rows; lift 3.70× → 3.48×; pred÷actual 0.99× → 2.29×	06_scale_27k.py → results/06_scale_27k.json

What's in here

Area	Specifically
Off-policy evaluation	IPS, SNIPS, direct method, doubly robust; bootstrap CIs; effective-sample-size diagnostics; estimators graded against a measurable ground truth
Calibration	Scalar, Platt and isotonic calibrators; ECE vs Brier gating (and why ECE can be gamed); per-policy calibration
Ranking models	LightGBM GBDT; PyTorch two-tower retrieval and two-tower + MLP ranker on Apple-silicon MPS, chronological early stopping
Evaluation design	Identical-window A/B test sets isolating exposure mechanism from temporal drift; base-rate-normalized PR-AUC; leakage fingerprinting for aggregate features
Data engineering at scale	9.9 GB tarball streamed once into column-pruned zstd parquet (48 GB CSV never lands on disk); DuckDB out-of-core joins against a 32M-row table; reservoir sampling to fit, chunked exact scoring of 186M rows
Reproducibility	Every figure written by the script that computes it to a committed results/*.json; dataset fetched, never vendored

Quick start

python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
./scripts/get_data.sh                    # KuaiRand-Pure, 47 MB from Zenodo
.venv/bin/python 01_signal_probe.py      # then 02 … 05

05 needs PyTorch (pip install torch; ~3 min on MPS, falls back to CPU). 06 needs ./scripts/get_data.sh 27k (9.9 GB download, streamed to 3.3 GB of parquet) plus pyarrow and duckdb; it runs in ~104 min on a laptop.

This repository is standalone: it does not depend on the RankShift Serving repos and never modifies the downloaded dataset.

Experiments

01 — Signal probe

Replicates the original feature-selection test on real logs. A feature counts as carrying signal when |AUC − 0.5| clears both three bootstrap sigma and an absolute floor of 0.005 — one rule, applied to both the per-row label and the headline count. The bootstrap resamples at 25%, so its standard error is rescaled by sqrt(frac) to the full sample; without that correction the gate is closer to six sigma than three.

	synthetic	KuaiRand-Pure
request-time features with signal	2 of 25	24 of 37
best single feature	—	watch_ratio @ 0.7486

The 0.58–0.60 ceiling was a property of the synthetic data's corrupted joins, not of engagement prediction.

Leakage fingerprint. Among the 50 daily video aggregates, exposure-side counts carry no signal (show_cnt 0.5031, play_cnt 0.5049) while engagement-side counts carry a lot (like_cnt 0.6282, double_click_cnt 0.6328). Genuine popularity features would move together. That they don't is a signature of label contamination.

02 — Exposure bias

Train on algorithmic exposure, test on two sets covering the identical window (22 Apr – 8 May), so any gap is exposure mechanism rather than temporal drift.

Metric	A · algorithmic	B · random
ROC-AUC	0.8811	0.8639	gap +0.0173, CI [+0.0110, +0.0240]
Lift @ 23.9%	3.428×	3.363×	+2.0% — transfers
Mean pred ÷ actual	0.96×	2.07×	breaks
ECE	0.0028	0.0060	2.1× worse

Engagement under algorithmic exposure runs 3.80× higher than under random — a direct measurement of what the production recommender is worth against a uniform baseline.

The PR-AUC trap. Raw PR-AUC falls 0.2865 → 0.1460, which reads as a 49% collapse. But PR-AUC is bounded below by base rate, and the base rates differ 3.80×. Normalized, it inverts: 13.51× vs 26.20× — the model is 1.94× better on randomized traffic relative to that split's own difficulty.

03 — Recalibration

Fit a calibrator on 9 days of randomized logs (22–30 Apr), apply it forward to the held-out 8 days (1–8 May). Fit and evaluation never share a day.

On randomized traffic — it works. The fitted scalar is 0.5012: halve the probability.

calibrator	pred ÷ actual	ECE	Brier gate
none	2.111×	0.00625	—
scalar	1.058×	0.00051	SHIP
platt	1.054×	0.00031	SHIP
isotonic	1.054×	0.00031	SHIP

ECE improves 20×. Lift is identical at 3.364× for every calibrator — a monotone transform cannot reorder, so calibration is free in ranking terms. One caveat: isotonic is only weakly monotone. It maps distinct scores onto shared values, and those ties move AUC slightly (0.8639 → 0.8631 here, and lift 3.428× → 3.396× on served traffic). Scalar and Platt are strictly monotone and leave both untouched.

On served traffic — it backfires.

calibrator	pred ÷ actual	ECE	Brier gate
none	0.961×	0.00284	—
scalar	0.481×	0.01100	reject
platt	0.489×	0.01084	reject
isotonic	0.491×	0.01079	reject

The correction that fixes randomized traffic breaks served traffic by almost exactly the same factor in the other direction.

Calibration is a property of the exposure policy, not of the model. One model cannot carry one calibrator across both. Notably, the Brier gate — adopted in the original project for an unrelated reason, to stop ECE being gamed by variance collapse — catches this on its own: SHIP on one distribution, reject on the other, with no special-casing.

04 — Off-policy estimators against the gold standard

The random log makes the thing OPE papers assume measurable: the true value of a policy. So instead of using estimators, this experiment grades them. Item-level propensities (video show-frequency, Laplace-smoothed) stand in for the logged propensities production systems rarely have.

Direction 1 — the production case. Estimate the uniform policy's value from the 295K-row standard log; truth is 0.00557 from the random log.

estimator	estimate	×truth	95% CI covers truth?
naive average	0.02121	3.80×	no
IPS	0.01671	3.00×	no
SNIPS	0.02058	3.69×	no
direct method	0.02145	3.85×	no
doubly robust	0.02124	3.81×	no

Every estimator misses, including doubly robust. Not a variance failure — the effective sample size after weighting is 42,633, and every interval is tight. The propensity model is wrong in a way reweighting cannot fix, and DR inherits the direct method's bias because the reward model was itself trained on biased exposure (the same 2× calibration break measured in 02/03).

Direction 2 — the diagnosis. Reverse it: estimate the standard policy's value from the random log, where logging propensities are exactly uniform. IPS with an item-marginal target recovers 0.00694 of a true 0.02121 — 33%. With propensities exact, the remaining error is purely the item-marginal approximation. Which decomposes the exposure bias:

3.80×  measured advantage of the standard policy
  =  1.25×  from which videos it picks        (item marginal, what IPS can see)
  ×  3.04×  from which user it shows them to  (matching, what item-level
                                               propensities cannot express)

Popularity-corrected estimators fix the 1.25×. The 3.04× — the part that is almost all of the value — needs user-conditional propensities, which is to say logged ones. The practical reading: if the logger doesn't record propensities, no estimator in this table reconstructs them afterwards.

One contract detail: the reward model here uses only user and video features. A counterfactual (u, v) pair has no watch time — it never happened — so an OPE reward model lives under a stricter feature contract than the serving model in 02, and reusing the serving model would smuggle an outcome into a counterfactual.

05 — Two-tower neural baselines

Same splits, same metrics as 02, PyTorch on Apple-silicon MPS. Two models: the retrieval-style two-tower (id embeddings + profile/metadata, dot product — no request-time features by construction), and the same towers with the request-time features through an MLP head, which is feature parity with the LightGBM.

model	TEST-A AUC	TEST-B AUC	TEST-A lift	TEST-B lift
LightGBM (02)	0.8811	0.8639	3.428×	3.363×
two-tower + request features	0.8815	0.8519	3.437×	3.256×
two-tower (pure retrieval)	0.8246	0.8007	3.035×	2.929×

Two readings:

• At feature parity, the neural ranker ties the GBDT (0.8815 vs 0.8811 on TEST-A). On a million rows of tabular features, that is the expected result, reported rather than tuned away.
• The pure two-tower reaches 0.8246 with no watch signal at all. Ids and profiles alone carry most of the ranking power — the AUC-level restatement of what 04 measures at the policy level: personalization, not video selection, is where the value lives.

06 — The findings survive 120× more data

Everything above is KuaiRand-Pure: 2.6M interactions, in memory. KuaiRand-27K is 322M interactions across a 32M-video catalogue — ~48 GB unpacked, which does not fit next to itself on this disk, let alone in RAM. The pipeline is shaped like production instead of like a notebook:

• ingest — scripts/convert_27k.py streams the 9.9 GB tarball once and writes column-pruned zstd parquet (3.3 GB); raw CSV never touches the disk. The three video_features_statistic files — half the archive — are skipped at the I/O layer: experiment 01 showed that family is label-contaminated.
• reduce — DuckDB streams the parquet for counts, splits and the out-of-core joins against the 32M-row video table.
• train — LightGBM fits on an 8M-row reservoir sample of the 136M-row train window (1.5 GB in memory): the standard sample-to-fit pattern.
• score — the full 186M-row TEST-A is scored in chunks, so evaluation is exact, not sampled. 1,249 s for the big split; 104 min end to end.

	Pure (2.6M)	27K (322M)
TEST-A AUC	0.8811	0.9147	more data helps
TEST-B AUC	0.8639	0.8831
lift A → B	3.428× → 3.363× (−2%)	3.702× → 3.483× (−6%)	ranking still transfers
pred ÷ actual, A → B	0.96× → 2.07×	0.99× → 2.29×	calibration still breaks
base-rate ratio A/B	3.80×	3.62×

Both headline findings replicate. The calibration break is, if anything, slightly larger at scale — more data sharpens the model's fit to the exposure policy it was trained under, which is precisely the thing that does not transfer.

Reproducing these numbers

Every figure above is written to results/*.json by the script that computes it, and those files are committed. 01_signal_probe.json through 05_two_tower.json are the evidence behind each table; nothing in this README is transcribed by hand.

Caveats

• One model, one seed. The bootstrap CI covers test sampling, not model choice or training randomness.
• play_time_ms and watch_ratio are treated as request-time features, matching the original service contract where watch duration arrives with the request. In a pre-impression ranking setting they would be unavailable and every number here would fall.
• The dataset authors dropped users with fewer than 10 randomly exposed videos, so the random split is not a clean sample of all users.
• Two-week window; the intervention probability is undisclosed for commercial reasons.

Data and attribution

KuaiRand-27K (experiment 06): the same 27,285 users against the full 32,038,725-video catalogue — 136.3M standard interactions in 8–21 Apr, 186.0M in 22 Apr – 8 May, and the same 1,186,059 randomly exposed rows. ./scripts/get_data.sh 27k fetches and converts it.

KuaiRand-Pure: 27,285 users, 7,583 videos. Collected from Kuaishou, 8 Apr – 8 May 2022. The standard logs carry 1,436,609 algorithmically exposed interactions (1,141,112 in 8–21 Apr and 295,497 in 22 Apr – 8 May); the random log carries a further 1,186,059 uniformly exposed ones over 22 Apr – 8 May. The randomized rows are a separate collection, not a subset of the logged traffic.

Chongming Gao, Shijun Li, Yuan Zhang, Jiawei Chen, Biao Li, Wenqiang Lei, Peng Jiang, Xiangnan He. KuaiRand: An Unbiased Sequential Recommendation Dataset with Randomly Exposed Videos. CIKM 2022. arXiv:2208.08696

Source: Zenodo record 10439422. Note a licence discrepancy — the GitHub repo states CC-BY-SA-4.0, the Zenodo record states CC-BY-4.0. Resolve before redistributing any derivative of the data. This repo redistributes none of it; scripts/get_data.sh fetches it from the original source.

Code in this repository is MIT.