Fix train/test leakage from duplicate dataset rows

[Frhnfaya]

What this fixes
Fixes #194.
The combined v2.4 dataset contains duplicate rows that leak across the train/test split, inflating reported accuracy.
Findings (reproducible)

7,718 exact/normalized duplicate rows (5.4% of 145,811)
80 machine-translation artifacts mislabeled as SMS (e.g. "Sorry, I cannot provide a translation…")
10 empty/symbol-only rows
Under config.py's split (test_size=0.2, random_state=42): test→train leakage was 7.4% (2,172 rows). After cleaning: 0.0%.

Changes

clean_ots_dataset.py — reproducible cleaning utility (dedup + artifact removal + unicode normalization) that prints a per-reason audit report.
dataset/sms_spam_phishing_dataset_v2.4.1_dedup.csv — cleaned dataset (138,003 rows; label balance preserved).
test_clean_dataset.py — asserts no duplicates, valid labels, no empty text (3 passed).
CHANGELOG entry.

The original v2.4 file is left untouched; the cleaned set is added alongside it.

[ajamous]

@Frhnfaya thanks — I re-ran the script against the pristine v2.4 combined file and reproduced your numbers exactly (145,811 → 138,003; 7,718 dups + 80 artifacts + 10 junk; leakage 7.4% → 0% under the seed-42 split). Clean work.

One thing before merge: nothing consumes the new file yet. config.py still points current_dataset at v2.4_combined.csv, and even the glob fallback would keep picking the old file (. sorts before _, so v2.4.1_dedup lands earlier in sorted()). Since the PR says "Fixes #194", merging as-is would close the issue while every retrain keeps using the leaky dataset. Either flip current_dataset to the dedup file in this PR, or drop the "Fixes" keyword and we track the switch + retrain as a follow-up — your call, both work for me.

Two smaller notes, fine as follow-ups: ~69 LLM-refusal/translation artifacts survive the marker list ("Please provide more context or clarify your request.", "Sorry, I cannot click on links…", etc.), and because the dedup key is (text, label), a few identical texts with conflicting labels survive (e.g. faceb00k.com as both spam and phishing).

[Frhnfaya]

@ajamous @claude Wired current_dataset to the deduped file (sms_spam_phishing_dataset_v2.4.1_dedup.csv)
so this now closes #194 for real — retrains will use the leakage-free data. Ready to merge.

[ajamous]

Verified locally before merging:

• Re-ran clean_ots_dataset.py on the pristine v2.4 file. The output matches the committed v2.4.1 file row for row, in the same order. The only byte difference is line endings.
• test_clean_dataset.py passes (3/3).
• Label balance is preserved: ham 38.0%, spam 33.5%, phishing 28.5%, same as before within 0.2 points.
• The branch is up to date with main and does not touch any file changed by Platform v2.11.0: close obfuscation and SMPP bypasses, honest benchmarks (model 2.7 unchanged) #229.

Merging. Two follow-ups, tracked separately, not blockers:

1. A wider marker list catches about 300 more translator and assistant replies that survive here, mostly Indonesian ("tidak bisa membuka tautan"), Spanish ("no tiene sentido") and Italian ("non ha un significato"). I will open a small PR that extends TRANSLATION_MARKERS and regenerates the file.
2. Two texts still carry conflicting labels (faceb00k.com as spam and phishing, and one artifact as ham and spam). The dedup key is (text, label) by design, so this is expected, but a labeling rule now exists in docs/LABELING_GUIDE.md and they can be settled against it.

Thanks for the careful work and for wiring current_dataset.