Name		Name	Last commit message	Last commit date
Latest commit History 23 Commits
.streamlit		.streamlit
Tokenizer		Tokenizer
__pycache__		__pycache__
iwslt2017_en_de		iwslt2017_en_de
README.md		README.md
app.py		app.py
corpus.txt		corpus.txt
requirements.txt		requirements.txt
test.py		test.py
tokenizer.py		tokenizer.py
train.py		train.py
utils.py		utils.py

Repository files navigation

BPE Tokenizer

About

Custom BPE Tokenizer, trained for 1256 merges. Can be trained further, we can load the previously trained tokenizer and also there's an option to add or remove special tokens that you need.

bpetokenizer-bzeekdnyv9blbbbuoefhpf.streamlit.app/

tokenizer bpe streamlit special-tokens

Readme