Skip to content

Fix uppercase cl100k contraction matching#25

Open
MikeeI wants to merge 1 commit into
tiktoken-go:mainfrom
MikeeI:fix/cl100k-uppercase-contraction
Open

Fix uppercase cl100k contraction matching#25
MikeeI wants to merge 1 commit into
tiktoken-go:mainfrom
MikeeI:fix/cl100k-uppercase-contraction

Conversation

@MikeeI

@MikeeI MikeeI commented Jul 16, 2026

Copy link
Copy Markdown

Summary

  • correct the generated cl100k_base contraction set from the widened [R-Tr-tſ] range to [STstſ]
  • add a regression case requiring uppercase "'RE" to encode as token 95253

Why

The precompiled regexp2cg engine currently treats R/r as part of the case-insensitive s|t alternative. Because that branch is atomic and precedes the re alternative, "'RE" is split into "'R" and "E", producing two tokens instead of the OpenAI-compatible single token. The source regex and vocabulary are already correct; the widened generated ASCII bitmap is the divergence.

Both the pinned regexp2cg v0.7.2 and current v0.9.0 regenerate the widened range, so this is a localized correction to the committed generated engine rather than a dependency bump.

Verification

  • go test -count=1 ./...
  • complete differential check of 10,000 ordinary-text cases against Python tiktoken 0.13.0 across r50k_base, p50k_base, cl100k_base, and o200k_base

Closes #24

@MikeeI

MikeeI commented Jul 16, 2026

Copy link
Copy Markdown
Author

I opened an upstream regexp2cg report with the minimal interpreter-vs-generated-engine reproduction and bitmap analysis: dlclark/regexp2cg#5

@bluescreen10

Copy link
Copy Markdown
Contributor

Thanks for this report and more importantly for opening up an upstream report. Let me see how fast this gets attended upstream. if not I can merge the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Generated cl100k_base regex splits uppercase "'RE" contraction incorrectly

2 participants