Tokenizer 3.6.4

Tokenizer 3.6.4 improves tokenization throughput without changing token output or the public API.

What changed

Raw whitespace-delimited tokens are now constructed directly with lazy origin offsets. Normal tokenization skips a redundant second rough-token pass; the additional pass is retained when HTML escape replacement is enabled because replacements such as   can introduce whitespace.

Performance

On test/toktest_large.txt (573 KB, 149,040 output tokens), median CPython tokenization time decreased from 1.905 seconds in 3.6.3 to 1.514 seconds in 3.6.4, a 20.5% reduction. Under cProfile, total calls decreased from 9.32 million to 7.77 million.

Compatibility and verification

• Token kinds, text, values, original text, and materialized origin spans match 3.6.3 across the complete large corpus.
• HTML escape and Unicode replacement behavior is preserved.
• The full suite passes on CPython 3.10 through 3.14 and PyPy 3.11.
• The source distribution and universal wheel pass twine check and a clean-venv installation test.

Upgrade with:

pip install --upgrade tokenizer

Full changes: #62

Version 3.6.3

Highlights

• Improves origin tracking performance by lazily materializing origin_spans, preserving the public token API and index behavior.
• On test/toktest_large.txt (573 KB, about 149k tokens), lazy origin spans were about 8-12% faster and saved about 9 MB peak memory when retaining tokens.
• Fixes valid_ssn() accepting some non-digit inputs.
• Rejects empty Roman numerals instead of returning 0.
• Avoids crashes for oversized numeric HTML escapes with replace_html_escapes=True.
• Updates the supported baseline to Python 3.10+ and CI to current stable Python versions, including Python 3.14.
• Adds uv dependency groups for test, lint, type-check, and release workflows.

Verification

• uv run --group test pytest -q: 151 passed
• uv run --group type mypy src/tokenizer: passed
• uv run --group lint ruff check . --statistics: passed
• twine check /tmp/tokenizer-dist-3.6.3/*: passed
• Installed tokenizer==3.6.3 from PyPI in a temporary virtual environment and verified import plus sample tokenization.

Version 3.6.0

Changes

• Removed the deprecated --handle_kludgy_ordinals CLI flag
• Removed the handle_kludgy_ordinals API option and related constants (KLUDGY_ORDINALS_PASS_THROUGH, KLUDGY_ORDINALS_MODIFY, KLUDGY_ORDINALS_TRANSLATE)
• Kludgy ordinals (e.g. 1sti, 3ja) are now always passed through unchanged as word tokens

CI Improvements

• Fixed PyPy 3.11 build by only installing dev dependencies (including mypy) on Python 3.9
• Added [test] optional dependency for lightweight test-only installations

Version 3.5.5

Version 3.5.5: Maintenance Release

This maintenance release focuses on infrastructure improvements and enhanced package metadata, making Tokenizer more discoverable and easier to work with.

What's Changed

Package Metadata & Infrastructure:

• Modernized pyproject.toml with PEP 517/518 build system configuration
• Enhanced PyPI discoverability with comprehensive keywords (nlp, natural-language-processing, sentence-segmentation, etc.)
• Added project URLs for documentation, issues, and changelog
• Improved package description for better clarity

Development & Tooling:

• Enhanced development dependencies with version pins (pytest>=7.0, ruff>=0.1.0, mypy>=1.0)
• Consolidated tooling configuration (ruff, mypy, pytest) in pyproject.toml
• Type checking improvements across source and test files
• Removed obsolete type: ignore comments

CI/CD:

• Updated build matrix for setuptools compatibility with PyPy
• Now testing: CPython 3.9-3.14, PyPy 3.11

Installation

pip install tokenizer==3.5.5

Full Changelog: 3.5.4...3.5.5

Version 3.5.4: Improved dash and hyphen handling

Summary

Version 3.5.4 improves dash and hyphen handling in the tokenizer, addressing GitHub issue #51 and several related edge cases.

Key Improvements

Dash Handling

• Free-standing hyphens: Hyphens with spaces around them (e.g., word - word) now preserve those spaces instead of being collapsed
• Year ranges: Hyphens between years normalize to en-dash when normalize=True, following Icelandic spelling rules (1914-1918 → 1914–1918)
• Em-dashes: Always treated as centered punctuation with spaces on both sides (word—word → word — word)
• Consecutive dashes: Multiple identical dashes (--, ––, ——) are handled as single tokens and preserve their spacing
• Edge case fix: Correctly handles 1914 -1918 where -1918 might appear to be a negative number but is actually part of a year range

Documentation Updates

• Added comprehensive "Dash and Hyphen Handling" section explaining context-specific behavior
• Updated normalize option descriptions to be specific about dash handling
• Fixed CSV format documentation (was showing outdated 3-column format, now shows current 5-column format)
• Added CSV Output Format section matching JSON documentation style

Testing

• Added comprehensive test suite (test/test_dashes.py) with 93 tests covering all dash scenarios
• All 150 tests pass (93 new + 57 existing)

Installation

pip install --upgrade tokenizer

View on PyPI: https://pypi.org/project/tokenizer/3.5.4/

Version 3.5.3

Changes in Version 3.5.3

This is a documentation fix release to ensure the project description displays correctly on PyPI.

Fix

• Updated pyproject.toml to correctly reference README.md instead of README.rst
• This ensures the project description is properly displayed on PyPI

Installation

pip install --upgrade tokenizer

View on PyPI: https://pypi.org/project/tokenizer/3.5.3/

Version 3.5.2

Changes in Version 3.5.2

Improvements

• BIN_Tuple representation: Now displays as plain tuple format in __str__() and __repr__() methods, matching documentation examples
• Cleaner JSON output: No longer includes empty "t" field for non-text tokens (BEGIN SENT, END SENT)
• Documentation updates: Fixed JSON output examples to reflect actual output including "o" (original) and "s" (span) fields

Bug Fixes

• Fixed test cases to match new JSON output format
• Updated README.md to accurately document JSON output fields

Installation

pip install --upgrade tokenizer

View on PyPI: https://pypi.org/project/tokenizer/3.5.2/

Version 3.5.1

• Fixed bug in composite glyph handling

Full Changelog: 3.5.0...3.5.1

Version 3.5.0

• Minor refactoring and all-round modernization
• Better handling of colon-separated timestamps
• Improved URI handling (more supported schemes)
• New --version flag in command line interface
• Much more extensive test coverage
• Explicit compatibility with CPython 3.14 and PyPy 3.11
• Moved from black to ruff for linting and formatting
• Moved to pyproject.toml for project configuration and metadata

Full Changelog: 3.4.5...3.5.0

Version 3.4.5

• Compatibility with Python 3.13
• Now requires Python 3.9 or later

Full Changelog: 3.4.4...3.4.5