52 lines (34 loc) · 3.22 KB

Datasets

Text datasets

Parallel corpora

Russian-Udmurt parallel corpora (100k sentences) Hugging Face
Tatoeba (>3600 sentence pairs). Hugging Face / Source
Finugorbib (>30k sentence pairs). Hugging Face / slone/finugorbib
Soviet geography book (>2700 sentence pairs). Hugging Face
FLORES-250, translation benchmark. Hugging Face / Other languages
Estonian-Udmurt Dictionary (EKI) (>106k sentece pairs). Hugging Face
FineTranslations (>27k unfiltered synthetic sentence pairs). Hugging Face

Monolingual corpora

Udmurt news (udmddn.ru and oshmes.info, in total 36k sentences). Hugging Face
Wikipedia dump (more than 43k sentences). Download
MADLAD-400 (651k sentences, 9.5 million words) Hugging Face / All languages
Glot500-c (121k sentences) GitHub
Zerpal (1.4M sentences) Hugging Face
FineWeb2 (13.5 million words) Hugging Face
FinePDFs (however, the quality of Udmurt language texts is really bad) Hugging Face
FineWiki (> 5k texts) Hugging Face

Classification

Finno-Ugric SIB (SIB-SMUGRI) Hugging Face
Zerpal-udmdunne (8,154 rows, 5 labels) Hugging Face
Zerpal-udmurtmedia (15,274 rows, 10 labels) Hugging Face

POS-tagging

Zerpal-pos-tagging (12,392 rows, 17 classes) Hugging Face

NER

WikiANN (the transcription is problematic: Latin and Cyrillic are used inconsistently, Wikipedia Markup is parsed incorrectly, but if you want to use it, see wikiann directory)

Question Answering

MultiWikiQA (5k rows) Hugging Face

Instruction

MURI-IT (2,751 rows) Hugging Face

Audio datasets

Volga Fairytales TTS Dataset (0.23 hours) Hugging Face