docs: file Limitations as tracked issues and link them from the README - #9
Merged
Conversation
Each limitation now has a GitHub issue carrying the evidence and acceptance criteria (#2-#8); the README keeps the one-line summary and links out. Corrects the pre-tokenization bullet, which claimed the translated patterns were "semantically equivalent to the originals for virtually all practical text". Differential testing against the installed tiktoken shows the GPT-4 pattern diverges on leading-space attachment -- "hello world" splits as ['hello', ' ', 'world'] where cl100k gives ['hello', ' world'] -- inflating pre-token counts by ~69% on English. Tracked in #2 with a verified fix.
Mirdula18
commented
Jul 31, 2026
Mirdula18
left a comment
Mirdula18
left a comment
Owner
Author
There was a problem hiding this comment.
Documentation review completed. The information is complete, easy to understand, and aligns with the implementation. Approved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #1, which merged while this was being written.
Files each README limitation as a tracked issue (#2-#8) and links them from
the README, so the summary and the detail cannot drift apart.
Also corrects a false claim in the pre-tokenization bullet. It said the
translated regexes were "semantically equivalent to the originals for
virtually all practical text". Differential testing against the installed
tiktoken_ext/openai_public.pyshows the GPT-4 pattern diverges on the mostcommon construct in prose:
"hello world"['hello', ' world']['hello', ' ', 'world']"foo_bar"['foo', '_bar']['foo', '_', 'bar']Corpus scale, first 200 KB each: english 41,751 upstream pre-tokens vs 70,557
ours (+69%), python +43%, tamil +20%. Since pre-tokens are the unit BPE
trains and encodes over, this understates every compression number in the
results table.
Tracked in #2 with a one-class fix verified to match upstream 100% across
600 KB spanning three scripts. Not applied here — it changes every published
number and warrants its own PR plus a regeneration.
Docs only; no source changes. 299 tests pass locally, ruff and mypy clean.