Sentiment classification with TF-IDF + Linear SVM

A reproducible text-classification project using 3,000 labelled Amazon, IMDb, and Yelp sentences from the UCI Machine Learning Repository. It combines word-level TF-IDF unigrams and bigrams with a linear support vector classifier, then evaluates the model with duplicate-aware cross-validation and a held-out set. This is a sanitized, reproducible rebuild of Waralee's university assignment; course-provided splits, submission files, and student identifiers are excluded. The raw data is downloaded from UCI at runtime and is not committed here.

Results

The fixed model was evaluated once on a deterministic, balanced 600-row held-out partition. Five-fold cross-validation used only the remaining 2,400 training rows.

Metric	5-fold CV, mean ± SD	Held out
Accuracy	0.8008 ± 0.0232	0.8650
Precision	0.8079 ± 0.0195	0.8712
Recall	0.7892 ± 0.0360	0.8567
F1	0.7982 ± 0.0257	0.8639
ROC AUC	0.8849 ± 0.0183	0.9361

The held-out confusion matrix is [[262, 38], [43, 257]] (rows are true negative/positive labels). Performance by source was 0.878 F1 on Amazon, 0.840 on IMDb, and 0.874 on Yelp. The held-out scores are higher than the CV averages, so the cross-validation distribution is the better estimate of typical performance; the held-out result should not be read as a guaranteed 86.5% on new review data.

Data quality and leakage controls

• UCI's documented 3,000 rows were recovered: 1,000 per source and exactly 500 rows per source/label combination.
• Required fields had no missing or empty values, and labels were limited to 0 and 1.
• Normalizing Unicode, case, punctuation, and whitespace found 21 duplicate text groups covering 49 rows (28 extra rows); none had conflicting labels.
• The train/held-out split is stratified by both source and label and grouped by normalized text. This gives 2,400/600 rows with zero normalized-text group overlap.
• A separate character 3–5 gram screen found 15 of 600 held-out rows (2.5%) with cosine similarity of at least 0.90 to a training row. This is a heuristic warning, not proof of duplication; short generic reviews can score highly.

Full machine-readable results are in results/metrics.json.

Method

1. Download the official UCI archive and verify its SHA-256 hash.
2. Repair two embedded physical line breaks in the IMDb source file and validate record counts, classes, missingness, and duplicate structure.
3. Take the first fold from a seeded five-fold StratifiedGroupKFold as the held-out set. Stratification uses source + label; grouping uses normalized text.
4. Fit TF-IDF word unigrams/bigrams (min_df=2, max_df=0.95, sublinear term frequency) and LinearSVC(C=1.0) in a leakage-safe pipeline.
5. Run five group-preserving stratified CV folds on the training partition, then fit once on all training rows and score the untouched held-out rows.

The classifier's decision function is used directly for ROC AUC; no probability calibration is required for ranking performance.

Reproduce

Python 3.10+ is recommended.

python -m venv .venv
# Windows PowerShell
.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt
python src/train.py
python -m jupyter nbconvert --execute --to notebook --inplace analysis.ipynb

The script recreates all files in figures/ plus results/metrics.json and results/cross_validation_folds.csv. The notebook presents the same workflow as a concise reader-facing analysis.

Repository layout

.
├── analysis.ipynb                 # executed narrative analysis
├── DATA_SOURCE.md                 # provenance, licence, and citation
├── figures/                       # generated evaluation figures
├── results/                       # reproduced metrics and CV fold scores
├── src/train.py                   # complete deterministic pipeline
├── requirements.txt
└── README.md

Interpretation and limitations

• The held-out partition contains the same three source domains as training, so this measures within-source generalisation—not performance on a new website, topic, language, or time period.
• UCI selected clearly positive or negative sentences, excluding neutral cases. Real reviews are often longer, mixed, sarcastic, or context-dependent.
• Exact normalized duplicates are isolated by partition, but lexical near-matches remain possible as quantified above.
• Coefficients indicate associations learned from this sample; they are not causal explanations and can encode source-specific vocabulary.

See DATA_SOURCE.md for the UCI citation, DOI, archive hash, and CC BY 4.0 attribution.