Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions src/mBERT/training/model-training/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Removed 7,718 duplicate rows causing train/test leakage
(7.4% → 0% under config.py split, seed=42).
- Removed 80 machine-translation artifacts and 10 junk-text rows.
- Extended the artifact marker list (Indonesian, Spanish, Italian, French, German,
Russian, Arabic assistant replies): 335 more rows removed, now 137,668 rows.

### Added
- `clean_ots_dataset.py` — reproducible dataset cleaning utility.
Expand Down
18 changes: 17 additions & 1 deletion src/mBERT/training/model-training/clean_ots_dataset.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,22 @@
"மொழிபெயர்ப்பு", "மொழிபெயர்க்கவும்",
"übersetzung:", "hier ist die übersetzung", "traduction:", "voici la traduction",
"traduzione:", "ecco la traduzione", "```",
# Assistant replies that the translation step left in place of a message.
# Phrases are kept specific enough not to match real SMS wording.
"could you please provide more context", "provide more context or clarify",
"i'm sorry, but the text", "i'm sorry, but i cannot", "sorry, i cannot click",
"i cannot click on links", "as an ai", "the text is a link to a website",
"shortened url", "does not seem to be in", "doesn't seem to be in",
"does not have a meaning", "doesn't have a meaning", "remain unchanged", "remains unchanged",
"no tiene sentido", "no tiene un significado", "no hay significado", "no se puede traducir",
"no puedo traducir", "lo siento, pero el texto", "lo siento, no se puede", "sin cambios",
"n'a pas de sens", "ne semble pas avoir de signification", "désolé, ce texte",
"non ha un significato", "non ci sono informazioni comprensibili", "mi dispiace, ma la frase",
"keine bedeutung", "ich bin ein ai-modell", "ich kann den satz nicht",
"текст оставлен без изменений", "не могу предоставить перевод",
"ليس بواجد في اللغة", "لا يمكنني ترجمة",
"membuka tautan", "mengakses tautan", "tidak bisa mengunjungi tautan",
"tidak bisa memfasilitasi", "dapatkah anda memberikan informasi lebih lanjut",
]

PUNCT_ONLY_RE = re.compile(r"^[\W_]+$", re.UNICODE)
Expand Down Expand Up @@ -98,7 +114,7 @@ def clean(in_path: str, out_path: str) -> None:
kept_rows.append((text, label, aug))

with open(out_path, "w", encoding="utf-8", newline="") as f:
w = csv.writer(f)
w = csv.writer(f, lineterminator="\n")
w.writerow(["text", "label", "augmentation_type"])
w.writerows(kept_rows)

Expand Down
Loading
Loading