Commit 4d080a9
committed
Update
[ghstack-poisoned]1 parent cfda684 commit 4d080a9
1 file changed
Lines changed: 1 addition & 1 deletion
Submodule tokenizers updated 10 files
- benchmark/.gitignore+1
- benchmark/CMakeLists.txt+21
- benchmark/hf_tokenizer_encode_latency.cpp+141
- benchmark/run.sh+15
- include/pytorch/tokenizers/hf_tokenizer.h+9-110
- include/pytorch/tokenizers/regex.h+5-1
- src/hf_tokenizer.cpp+101-15
- src/normalizer.cpp+29-6
- test/test_hf_tokenizer.cpp+65
- test/test_normalizer.cpp+32
0 commit comments