Skip to content

Release Tokenizer 3.6.4#62

Merged
vthorsteinsson merged 2 commits into
masterfrom
perf/profile-tokenizer-hotspots
Jul 10, 2026
Merged

Release Tokenizer 3.6.4#62
vthorsteinsson merged 2 commits into
masterfrom
perf/profile-tokenizer-hotspots

Conversation

@vthorsteinsson

Copy link
Copy Markdown
Member

Summary

  • avoid the redundant second rough-token whitespace pass during normal tokenization
  • retain the pass when HTML escape replacement can introduce whitespace
  • bump the package version to 3.6.4 and document the performance improvement

Performance

On test/toktest_large.txt (149,040 tokens), median CPython tokenization time decreased from 1.905 s to 1.514 s (20.5%). Profiled calls decreased from 9.32M to 7.77M.

Verification

  • pytest: 151 passed
  • ruff check .: passed
  • mypy src/tokenizer: passed
  • source distribution and wheel built successfully
  • twine check: passed
  • local clean-venv wheel smoke test: passed
  • all deep-token fields, including origin spans, matched master on the large corpus

@vthorsteinsson vthorsteinsson merged commit 7ce5d94 into master Jul 10, 2026
6 checks passed
@vthorsteinsson vthorsteinsson deleted the perf/profile-tokenizer-hotspots branch July 10, 2026 12:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant