Coming Soon: A self-built LLM-powered ML training agent designed to automate hyperparameter search across general machine learning workflows — runs smoke tests automatically, feeds results to an LLM, and recommends the next parameters to try. Used to optimize this model. Release expected in the next few weeks — star this repo to get notified!

The ML Agent: https://github.com/rayzhao27/llm-ml-hyperparameter-tuner

---

About The Project

Two-Stage Sequential Recommendation System: BERT4Rec Retrieval + DIN Reranking

This project implements a production-style two-stage recommender on MovieLens 1M, built from scratch in PyTorch and served via FastAPI:

• Stage 1 — Retrieval: BERT4Rec (Sun et al., 2019) with an optional multi-feature embedding (genre + decade side information) generates top-100 candidates.
• Stage 2 — Reranking: DIN (Zhou et al., 2018) applies target-aware attention over the top-100, trained with mixed random + BERT4Rec-based hard negatives.
• Fusion: Reciprocal Rank Fusion (RRF) combines the two ranker signals at inference time.

The project treats BERT4Rec as the strong single-stage baseline, and DIN as a reranker experiment inspired by production-scale multi-stage architectures. A full debugging trace of how the reranker interacts with retriever quality is included in Reranking Analysis.

Key Results (full-corpus ranking, 3,706 items)

System	HR@10	NDCG@10
BERT4Rec (baseline)	0.2884	0.1603
BERT4Rec + multi-feature	0.2877	0.1611
BERT4Rec + DIN reranker	0.1821	0.0905
BERT4Rec + DIN (RRF fusion, α=0.75)	0.2974	0.1692

Reported benchmark on ML-1M full-ranking: BERT4Rec original code = 0.1518 HR@10 (Petrov & Macdonald, 2022); best public replication = 0.2821. This implementation reaches 0.2884 as a single-stage retriever and 0.2974 with rank fusion.

Core Features:

• Bidirectional Transformer — full self-attention over the entire interaction sequence, no causal mask
• BERT-style Masking — random 20% of items masked during training using the 80/10/10 rule
• Multi-Feature Embedding (optional) — fuses item ID + multi-hot genre + decade bucket via a learned linear projection
• DIN Reranker — target-aware attention with frozen BERT4Rec item embeddings and mixed (random + hard) negatives
• Rank Fusion — RRF combines BERT4Rec and DIN rankings over the same top-K candidate set
• Leave-one-out Evaluation — full-corpus ranking with HR@K, NDCG@K, MRR@K
• Production API — FastAPI inference service with Pydantic validation, seen-item filtering, and Swagger UI
• LLM-powered Hyperparameter Tuning — self-built agent that automates smoke tests and feeds results back to an LLM for parameter recommendations (standalone release coming soon)

Cloud Training: Trained on Google Colab using an NVIDIA L4 GPU — end-to-end training for BERT4Rec (300 epochs) plus DIN (10 epochs) plus RRF fusion evaluation completed in under an hour.

Pipeline Overview:

Training Loss Curve:

Validation Loss Curve:

I hope this project demonstrates a clean, end-to-end ML engineering pipeline — from raw data to a serving API — including how a real reranking experiment surfaces the classic component AUC vs system ranking metric mismatch. Fight on! ✌️

(back to top)

Built With

• Python
• PyTorch
• FastAPI
• Pydantic
• MovieLens 1M
• TensorBoard
• Scikit-learn
• Joblib

(back to top)

Project Structure

bert4rec/
├── pictures/
│   ├── train_loss.png 
│   ├── val_loss.png 
│   └── pipeline.png
│
├── data/
│   ├── download.py                    # stream-downloads MovieLens 1M zip from GroupLens
│   ├── preprocess.py                  # builds chronological user interaction sequences + item side features
│   ├── dataset.py                     # BERT4Rec Dataset with BERT-style masking + DataLoaders
│   ├── din_dataset.py                 # DIN Dataset with multi-prefix training + mixed negatives
│   └── generate_hard_negatives.py     # offline BERT4Rec-based hard negative generation for DIN
│
├── model/
│   ├── embeddings.py                  # item + positional embedding, optional multi-feature fusion
│   ├── attention.py                   # transformer block (pre-LN, multi-head self-attention + FFN)
│   ├── bert4rec.py                    # full BERT4Rec model + inference helper
│   └── din.py                         # DIN reranker with target-aware attention
│
├── training/
│   ├── loss.py                        # masked item prediction loss (CrossEntropy, ignore_index=-100)
│   ├── scheduler.py                   # linear warmup + cosine decay LR schedule
│   ├── trainer.py                     # BERT4Rec training loop with AMP, grad clipping, checkpointing
│   └── train_din.py                   # DIN training loop with BCE + val AUC tracking
│
├── evaluation/
│   ├── metrics.py                     # HR@K, NDCG@K, MRR@K accumulator
│   ├── evaluator.py                   # BERT4Rec single-stage evaluation
│   └── evaluate_pipeline.py           # end-to-end two-stage evaluation + RRF fusion sweep
│
├── api/
│   ├── schemas.py                     # Pydantic request/response models
│   ├── predictor.py                   # single-load model wrapper for inference
│   ├── routes.py                      # GET /health, POST /recommend endpoints
│   └── main.py                        # FastAPI app factory with lifespan startup/shutdown
│
├── visualization/
│   └── save_plots.py                  # export TensorBoard scalars to PNG
│
├── colab/
│   └── bert4rec_colab.ipynb           # Colab training notebook
│
└── .gitignore

(back to top)

System Architecture

Data Pipeline

1. Download: Stream MovieLens 1M zip from GroupLens, extract ratings.dat, movies.dat
2. Preprocess: Encode item IDs (1-based), sort interactions chronologically per user, extract genre multi-hot vectors and decade buckets from movies.dat
3. Split: Leave-one-out — train seq[:-2], val seq[:-1], test seq
4. Mask (BERT4Rec): 20% of items randomly masked using BERT 80/10/10 rule
5. DIN training samples: Multi-prefix expansion over train_seqs only (seq[:i] → seq[i] for each valid i), strictly excluding val and test targets
6. Hard negatives (offline): BERT4Rec scores every DIN training prefix's [MASK] position to generate a top-K hard negative pool per sample; stored to hard_negatives.pkl

Model Pipeline

1. Embedding Layer: Item ID embedding + optional (genre projection + decade embedding) fused via linear → LayerNorm → Dropout
2. Transformer Stack: N=4 bidirectional transformer blocks (pre-LayerNorm)
3. Prediction Head: LayerNorm → Linear(d → vocab_size)
4. Loss: CrossEntropyLoss at masked positions only (ignore_index=-100)
5. DIN Reranker: target attention MLP over behavior × candidate interaction features, BCE loss, item embedding frozen from BERT4Rec

Inference Pipeline

1. User history received via POST /recommend
2. Sequence truncated and left-padded to max_seq_len=200
3. [MASK] appended at the last position
4. BERT4Rec forward → logits at mask position extracted
5. Seen-item logits set to -inf → top-K returned

(The two-stage BERT4Rec + DIN + RRF pipeline is currently exposed via evaluate_pipeline.py for offline evaluation; wiring it into the serving API is future work.)

Data Flow

MovieLens 1M (ratings.dat, movies.dat)
    ↓
Preprocess (encode IDs, sort by timestamp, extract side features)
    ↓
BERT4RecDataset (truncate, pad, mask) ────────────┐
    ↓                                              │
BERTEmbeddings (item + genre + decade + position)  │
    ↓                                              │
TransformerBlock × 4 (bidirectional self-attn)     │  Stage 1: Retrieval
    ↓                                              │
PredictionHead (linear → vocab_size)               │
    ↓                                              │
Top-100 candidates ────────────────────────────────┘
    ↓
DIN (target attention over behavior × candidate) ── Stage 2: Reranking
    ↓
RRF fusion of BERT rank and DIN rank ────────────── Final ordering
    ↓
Top-K recommendations

(back to top)

Architecture Components

Embedding Layer (model/embeddings.py)

• Item embeddings: lookup table E ∈ ℝ^(vocab_size × d_item), padding_idx=0
• Positional embeddings: learned table P ∈ ℝ^(max_seq_len × d), position 0 reserved for PAD
• Multi-feature fusion (optional): genre projection Linear(18 → d_genre) over multi-hot genre vectors + decade embedding lookup, concatenated with item embedding and projected back to hidden_size
• Output: LayerNorm(fused + pos_emb) → Dropout

Transformer Block (model/attention.py)

• Pre-LayerNorm layout — more stable than post-LN at small dataset scale
• nn.MultiheadAttention with batch_first=True, no causal mask
• PAD positions excluded via key_padding_mask
• Feed-forward: Linear(d → 4d) → GELU → Dropout → Linear(4d → d)
• Residual connections around both sub-layers

BERT4Rec Model (model/bert4rec.py)

• Stacks N transformer blocks
• _init_weights: truncated normal (σ=0.02), standard BERT initialization
• recommend(): inference helper — finds last real token, masks it, filters seen items, returns top-K

DIN Reranker (model/din.py)

• Item embedding initialized from BERT4Rec's embeddings.item_embeddings.weight and frozen by default
• Attention unit: for each behavior position, MLP over [b, c, b−c, b*c] produces a scalar weight; PAD positions zeroed
• User representation: weighted sum of behavior embeddings (no softmax, following the original DIN paper)
• Scoring head: MLP over [user_repr, candidate_emb] → BCE-with-logits
• score_candidates(): convenience method that scores K candidates for a single user in one forward pass

Training Loop (training/trainer.py)

• AdamW with parameter groups — bias/LayerNorm excluded from weight decay
• Linear warmup + cosine decay LR schedule
• AMP (automatic mixed precision) on CUDA, graceful fallback on MPS/CPU
• Gradient clipping (max norm 5.0)
• Saves best_model.pt on val loss improvement, latest.pt every epoch
• Signal handler: saves checkpoint on Ctrl-C

DIN Training Loop (training/train_din.py)

• AdamW over trainable parameters only (frozen item embedding excluded)
• BCEWithLogitsLoss, pointwise
• Tracks pairwise-comparison val AUC per epoch, saves best by val AUC
• Supports optional --hard_negatives_path and --mixed_ratio to blend random and BERT4Rec-based hard negatives

Hard Negative Generation (data/generate_hard_negatives.py)

• Enumerates the exact same (user, prefix_len) samples as DINTrainDataset
• Runs BERT4Rec forward on behavior + [MASK] for each sample (batched)
• Filters PAD, MASK, all seen items, and the target from BERT4Rec's top-K logits
• Stores {(user_id, prefix_len): [item_ids]} to data/processed/hard_negatives.pkl

Evaluation (evaluation/evaluator.py)

• Full-corpus ranking — all 3,706 items ranked per user (not sampled negatives)
• Seen-item filtering applied before ranking
• Metrics: HR@K, NDCG@K, MRR@K at K = 5, 10, 20

Two-Stage Pipeline Evaluation (evaluation/evaluate_pipeline.py)

• Runs three evaluations in a single pass over the test set on identical top-K candidate sets:
  1. BERT4Rec alone — full-vocab ranking
  2. DIN reranking — DIN rescores BERT4Rec's top-K, sort by DIN
  3. RRF fusion — sweep over α ∈ {0.0, 0.25, 0.5, 0.75, 1.0}
• All three share the same seen-item filter, ensuring comparable numbers

Inference API (api/)

• POST /recommend — Pydantic-validated request, seen-item filtering, top-K logits
• GET /health — model status, vocab size, device, checkpoint version
• Single-load checkpoint management — model loaded once at startup via FastAPI lifespan
• Auto-generated Swagger UI at /docs

(back to top)

Usage

Prerequisites

git clone https://github.com/rayzhao27/bert4rec.git
cd bert4rec

conda create -n bert4rec python=3.9
conda activate bert4rec

pip install torch torchvision numpy pandas scikit-learn scipy \
            requests tqdm tensorboard joblib fastapi uvicorn pydantic

Download and Preprocess Data

python data/download.py --data_dir data
python data/preprocess.py --data_dir data --min_rating 0

Train BERT4Rec (baseline)

python training/trainer.py \
  --data_dir data \
  --hidden_size 256 \
  --num_hidden_layers 4 \
  --num_attention_heads 4 \
  --intermediate_size 1024 \
  --hidden_dropout_prob 0.2 \
  --attention_probs_dropout 0.2 \
  --learning_rate 1e-3 \
  --warmup_steps 100 \
  --weight_decay 0.01 \
  --epochs 300 \
  --num_workers 0 \
  --checkpoint_dir checkpoints/baseline

Train BERT4Rec with multi-feature embedding

python training/trainer.py \
  --data_dir data \
  --use_features \
  --hidden_size 256 \
  --num_hidden_layers 4 \
  --num_attention_heads 4 \
  --intermediate_size 1024 \
  --hidden_dropout_prob 0.2 \
  --attention_probs_dropout 0.2 \
  --learning_rate 1e-3 \
  --warmup_steps 100 \
  --weight_decay 0.01 \
  --epochs 300 \
  --num_workers 0 \
  --checkpoint_dir checkpoints/multifeature

Generate hard negatives for DIN

python data/generate_hard_negatives.py \
  --bert4rec_ckpt checkpoints/multifeature/best_model.pt \
  --data_dir data \
  --top_k 200 \
  --min_prefix 1 \
  --batch_size 128

Train DIN reranker

python training/train_din.py \
  --bert4rec_ckpt checkpoints/multifeature/best_model.pt \
  --data_dir data \
  --epochs 10 \
  --batch_size 512 \
  --n_neg 4 \
  --learning_rate 1e-3 \
  --hard_negatives_path data/processed/hard_negatives.pkl \
  --mixed_ratio 0.5 \
  --checkpoint_dir checkpoints/din_hard

Monitor Training

python -m tensorboard.main --logdir runs/bert4rec
# open http://localhost:6006

Evaluate BERT4Rec alone

python evaluation/evaluator.py \
  --checkpoint checkpoints/multifeature/best_model.pt \
  --data_dir data \
  --k_values 5 10 20

Evaluate the full two-stage pipeline (with RRF fusion sweep)

python evaluation/evaluate_pipeline.py \
  --bert4rec_ckpt checkpoints/multifeature/best_model.pt \
  --din_ckpt      checkpoints/din_hard/best_din.pt \
  --data_dir      data \
  --recall_k      100 \
  --alphas        0.0 0.25 0.5 0.75 1.0

Start API Server

uvicorn api.main:app --reload --host 0.0.0.0 --port 8000

• Swagger UI: http://localhost:8000/docs
• Health Check: http://localhost:8000/health

API Examples

Health check:

curl http://localhost:8000/health

{
  "status": "ok",
  "model_loaded": true,
  "vocab_size": 3708,
  "model_version": "epoch_60",
  "device": "cuda"
}

Get recommendations:

curl -X POST http://localhost:8000/recommend \
  -H "Content-Type: application/json" \
  -d '{"user_history": [42, 17, 88, 5, 231], "top_k": 10}'

{
  "recommendations": [
    {"item_id": 2652, "score": 5.1263},
    {"item_id": 347,  "score": 5.0701},
    {"item_id": 580,  "score": 4.7346}
  ],
  "model_version": "epoch_60",
  "num_input_items": 5
}

Request Parameters

Parameter	Type	Default	Description
user_history	list[int]	Required	Chronological list of item ids, most recent last
top_k	int	10	Number of recommendations to return (1–100)

(back to top)

Configuration

BERT4Rec Model Configuration

Parameter	Value
hidden_size	256
num_hidden_layers	4
num_attention_heads	4
intermediate_size	1024
max_seq_len	200
mask_prob	0.2
hidden_dropout_prob	0.2
attention_probs_dropout	0.2
vocab_size	3708
total parameters (baseline)	5,179,516
total parameters (multi-feature)	3,429,436

Multi-Feature Embedding Configuration

Parameter	Value
d_item	192
d_genre	32
d_decade	32
num_genres	18
num_decades	10

BERT4Rec Training Configuration

Parameter	Value
optimizer	AdamW (parameter groups)
learning_rate	1e-3
weight_decay	0.01 (non-bias/LN params only)
warmup_steps	100
lr_schedule	linear warmup + cosine decay
grad_clip	5.0
batch_size	256
epochs	300

DIN Configuration

Parameter	Value
item embedding	copied from multi-feature BERT4Rec, frozen
attn_hidden	64
score_hidden	128
dropout	0.2
loss	BCEWithLogitsLoss (pointwise)
batch_size	512
n_neg per sample	4
mixed_ratio (hard : random)	0.5 : 0.5
optimizer	AdamW
learning_rate	1e-3
weight_decay	1e-4
epochs	10
trainable parameters	108,934

RRF Fusion Configuration

Parameter	Value
recall_k	100
rrf_k (smoothing constant)	60
alphas swept	0.0, 0.25, 0.5, 0.75, 1.0

Dataset Statistics

Stat	Value
Users	6,040
Items	3,706
Interactions	1,000,209
Avg sequence length	165.6
Min / Max seq length	20 / 2,314
Genres	18
Decades	10

(back to top)

Cloud Training

Training was offloaded to Google Colab to accelerate experimentation.

GPU Selection

For this workload (embedding-table–bound recommendation model, batch_size 256, ~3.4M parameters), the bottleneck is memory bandwidth and data loading rather than raw TFLOPS. NVIDIA L4 was selected as a good balance of speed, memory (24GB), and Colab compute-unit cost.

Setup

Clone and run directly on Colab or check the notebook under /colab:

!git clone https://github.com/rayzhao27/bert4rec.git
!pip install torch numpy pandas scikit-learn scipy requests tqdm tensorboard joblib fastapi uvicorn pydantic
!python /content/bert4rec/data/download.py --data_dir /content/bert4rec/data
!python /content/bert4rec/data/preprocess.py --data_dir /content/bert4rec/data --min_rating 0
!python /content/bert4rec/training/trainer.py \
  --data_dir /content/bert4rec/data \
  --use_features \
  --hidden_size 256 \
  --num_hidden_layers 4 \
  --num_attention_heads 4 \
  --intermediate_size 1024 \
  --hidden_dropout_prob 0.2 \
  --attention_probs_dropout 0.2 \
  --learning_rate 1e-3 \
  --warmup_steps 100 \
  --weight_decay 0.01 \
  --epochs 300 \
  --num_workers 2

(back to top)

Hyperparameter Tuning Agent

Hyperparameters for this model were explored using a self-built LLM-powered tuning agent currently under active development.

The agent runs automated smoke tests (short training runs), collects val_loss results, and feeds them back to an LLM to recommend the next set of parameters to try — similar in spirit to Bayesian optimization but using an LLM as the surrogate model.

What it does:

• Runs N trials of short training (configurable epochs)
• Parses val_loss from each trial automatically
• Sends results + history to an LLM for next recommendation
• Outputs a final config with rationale

Key finding from this project: Short-horizon smoke tests (10–30 epochs) can miss late-stage overfitting that only appears after epoch 50+. This discovery is being incorporated into the agent's evaluation strategy.

The agent is still under active development and will be released as a standalone repository in the coming weeks. If you're interested, check back soon — or follow the repo to get notified.

(back to top)

Results

BERT4Rec vs Multi-feature (single-stage, full-corpus ranking)

Metric	BERT4Rec	+ multi-feature	Δ
HR@5	0.1929	0.1980	+2.6%
HR@10	0.2884	0.2877	−0.2%
HR@20	0.3932	0.3959	+0.7%
NDCG@5	0.1294	0.1322	+2.2%
NDCG@10	0.1603	0.1611	+0.5%
NDCG@20	0.1868	0.1883	+0.8%
val_loss	5.857	5.800	−1.0%

Multi-feature side information (genre + decade) gives a modest but consistent lift on head-of-list metrics. The effect size is small because ML-1M is a dense dataset where item ID embeddings already carry most of the signal; the same architecture is designed to give larger gains on sparse datasets with cold-start items.

External reference (ML-1M full-ranking) — from Petrov & Macdonald, A Systematic Review and Replicability Study of BERT4Rec for Sequential Recommendation (arXiv:2207.07483), Table 5:

Implementation	HR@10	NDCG@10
BERT4Rec original code	0.1518	0.0806
RecBole BERT4Rec	0.1061	0.0546
BERT4Rec-VAE	0.2394	0.1314
Petrov & Macdonald replication	0.2584	0.1392
Petrov & Macdonald (longer seq)	0.2821	0.1516
This implementation (BERT4Rec baseline)	0.2884	0.1603

Two-Stage Pipeline: BERT4Rec + DIN + RRF Fusion

System	HR@10	NDCG@10
BERT4Rec (baseline retriever)	0.2877	0.1611
BERT4Rec + DIN reranker (pure)	0.1821	0.0905
RRF fusion, α=0.25	0.2341	0.1300
RRF fusion, α=0.50	0.2836	0.1590
RRF fusion, α=0.75	0.2974	0.1692
RRF fusion, α=1.00 (pure BERT rank)	0.2877	0.1611

Pure DIN reranking degrades HR@10 by ~37%. However, RRF fusion with α=0.75 (mostly BERT, some DIN) beats pure BERT by +3.4% HR@10 and +5.0% NDCG@10 — DIN adds incremental signal when used as a supplementary ranker rather than a replacement. See Reranking Analysis for the debugging trace behind this result.

(back to top)

Reranking Analysis

The DIN reranker reached a strong pointwise validation AUC of 0.869 but, when used as a hard reranker over BERT4Rec's top-100 candidates, degraded end-to-end HR@10 from 0.288 to 0.182. This component AUC vs system ranking metric disagreement is a well-known failure pattern in industrial recsys, so I traced it end-to-end rather than swap models.

Bugs ruled out first. Timestep alignment across preprocess → dataset → pipeline, item ID mapping (PAD=0, MASK=vocab_size−1), seen-item filtering symmetry between stages, hard-negative generation context (no future leakage — hard negatives for prefix [A,B] come from BERT4Rec at [A,B,MASK]), label direction in BCEWithLogitsLoss, and frozen embedding integrity all checked out.

Diagnostics.

• History sensitivity test. For a fixed target candidate, DIN score changed by exactly 0.0000 when the user's behavior sequence was shuffled, by +0.022 when replaced with another user's history, and by −0.128 only when replaced with random items. The attention layer appears order-invariant and only weakly personalized.
• Popularity correlation. Within the same BERT4Rec top-100 candidate set, DIN scores had Spearman correlation +0.706 with item popularity, versus +0.081 for BERT4Rec. DIN systematically pushes popular items up and pushes long-tail targets (like popularity=372 for the diagnostic example) down.
• Zero-user_repr ablation. Forcing user_repr = 0 inside DIN's scoring head dropped val AUC from 0.869 to 0.740 (Δ = −0.128). User history is informative for the pointwise task, so the failure at rerank time is more about how the signal is used (rank vs classify) than about whether it exists at all.

Interpretation (hypothesis, not fully proven). Under ML-1M's ID-only features and pointwise BCE objective, DIN may be degenerating into a combination of item-prior scoring and local candidate-to-history similarity matching, rather than learning strongly personalized user-interest representations. This is enough to reach high pointwise AUC on random negatives, but shifts high-popularity items ahead of truly relevant ones when required to rerank an already-strong retriever's top-K. Full causal validation would require a candidate-only baseline and direct attention-weight analysis; these are flagged as future work.

Engineering resolution: rank fusion instead of hard rerank. I applied Reciprocal Rank Fusion over the same top-100 candidate set, sweeping α from 0.0 (pure DIN rank) to 1.0 (pure BERT rank). At α=0.75, RRF fusion beat pure BERT4Rec by +3.4% HR@10 and +5.0% NDCG@10 (see the Results table). DIN as a supplementary signal, downweighted 3:1 relative to BERT, extracts partial value without letting its shortcut behavior dominate.

Takeaways for a production setting.

1. Component-level offline metrics (AUC on training-time negatives) can be misleading — the deployment-relevant signal is end-to-end ranking quality on the actual retrieval-set distribution.
2. When a reranker underperforms the retriever, RRF or other rank-based fusion is a cheap and defensible fallback that lets you extract partial value from an experiment rather than shipping nothing.
3. To fix DIN as a true reranker rather than a fusion signal, next iterations would be popularity-debiased negative sampling (or IPS reweighting) and moving from pointwise BCE to a pairwise (BPR) or listwise (LambdaRank) objective.

(back to top)

Design Decisions

Why pre-LN over post-LN? Pre-LayerNorm is more stable at small dataset scale and does not require an aggressive warmup phase, unlike the post-LN layout in the original BERT paper.

Why keep all ratings? Filtering to rating ≥ 4.0 reduces interaction count by ~40%. Using all interactions gives the model richer sequence data and consistently improves metrics.

Why AdamW parameter groups? Bias terms and LayerNorm weights do not benefit from weight decay the same way weight matrices do. Separating them into two optimizer groups follows standard BERT practice.

Why full-corpus ranking for evaluation? Sampling 100 random negatives inflates metrics by 2–3× and makes results incomparable across papers. Full ranking over all 3,706 items is slower but honest. Krichene & Rendle (2020) show sampled metrics are not consistent estimators, which is why modern replication studies (e.g. Petrov & Macdonald 2022) report full-ranking numbers.

Why 256 hidden size over 384? Experimentation showed 384 hidden size consistently overfits on ML-1M's 6,040 users regardless of dropout and weight decay. Model capacity must match dataset scale.

Why concat + linear fusion for multi-feature embedding (vs sum)? Genre and decade have very different dimensionalities and semantic roles from item ID. A learned linear projection over the concatenation lets the model decide how much weight to give each source, rather than forcing them into the same subspace with additive fusion.

Why frozen BERT4Rec item embedding in DIN? DIN is trained on hundreds of thousands of (user, candidate) samples but only has ~100k trainable parameters. Reusing BERT4Rec's already-learned item representations means DIN only needs to learn the attention and scoring MLPs, converging in a few epochs. Fine-tuning the embedding would risk overwriting BERT4Rec's learned representations with a DIN-specific signal that we already saw was noisy.

Why multi-prefix training for DIN? A single (behavior=full_seq[:-1], target=full_seq[-1]) sample per user gives only 6,040 training examples. Expanding to all valid prefixes over train_seqs (= full_seq[:-2]) produces ~980k samples while strictly excluding the val target (full_seq[-2]) and test target (full_seq[-1]), giving DIN much more supervision without leaking evaluation labels.

Why mixed random + hard negatives? Random unseen negatives are trivially separable from positives (the negative is often a completely unrelated cold-tail item), which lets DIN take a popularity shortcut. Hard negatives drawn from BERT4Rec's top-K expose DIN to the actual reranking distribution. A 50/50 mix preserves training diversity while making the harder samples visible.

Why Reciprocal Rank Fusion over score blending? BERT4Rec logits and DIN logits live on very different scales (unbounded real vs bounded via sigmoid), and different candidate sets have different score distributions. RRF is fully scale-invariant — it operates only on ranks — and is a standard technique in search (Elasticsearch, Vespa) and learning-to-rank late fusion. The RRF smoothing constant k=60 is the value recommended in the original Cormack et al. (2009) paper.

(back to top)

Reference

Sun, F., Liu, J., Wu, J., Pei, C., Lin, X., Ou, W., & Jiang, P. (2019). BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. CIKM 2019. https://arxiv.org/abs/1904.06690

Zhou, G., Zhu, X., Song, C., Fan, Y., Zhu, H., Ma, X., Yan, Y., Jin, J., Li, H., & Gai, K. (2018). Deep Interest Network for Click-Through Rate Prediction. KDD 2018. https://arxiv.org/abs/1706.06978

Petrov, A., & Macdonald, C. (2022). A Systematic Review and Replicability Study of BERT4Rec for Sequential Recommendation. RecSys 2022. https://arxiv.org/abs/2207.07483

Krichene, W., & Rendle, S. (2020). On Sampled Metrics for Item Recommendation. KDD 2020.

Cormack, G. V., Clarke, C. L. A., & Büttcher, S. (2009). Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods. SIGIR 2009.