Support non-ASCII identifiers - #3082
Merged
Merged
Conversation
soutaro
added a commit
to Shopify/rbs
that referenced
this pull request
Aug 12, 2026
`rbs_lexer_new` reaches `start_pos` by stepping one character at a time from the beginning of the buffer, so only the first byte of a character is a position it can stand on. Inside a character the step goes over `start_pos` and lexing quietly begins at the next one. Nothing downstream can tell: the walk keeps `line`, `column` and `char_pos` consistent, so the result points at a real position that simply is not the one asked for. The shift is caught by accident when what it lands on cannot open a token, but an ASCII letter right after the multibyte character parses cleanly in the wrong place -- and ruby#3082 gives the non-ASCII characters tokens of their own, taking even the accident away. Past the end of the buffer there is nothing left to step over, so the walk never finishes: `rbs_skip` stops moving at EOF while the loop waits for a position it will never reach. `parse_type("Integer", byte_range: 20...30)` hangs on master. The walk is the only thing that knows either, so it says so, and `NULL` carries it out to whoever asked. For the extension that is an `ArgumentError` alongside the reversed and negative ranges, since both are the caller's mistake rather than anything about the source text. `end_pos` keeps taking any value: clamping with a large number instead of measuring the buffer is ordinary, and the lexer stops at the end on its own.
soutaro
added a commit
to Shopify/rbs
that referenced
this pull request
Aug 12, 2026
`rbs_lexer_new` reaches `start_pos` by stepping one character at a time from the beginning of the buffer, so only the first byte of a character is a position it can stand on. Inside a character the step goes over `start_pos` and lexing quietly begins at the next one. Nothing downstream can tell: the walk keeps `line`, `column` and `char_pos` consistent, so the result points at a real position that simply is not the one asked for. The shift is caught by accident when what it lands on cannot open a token, but an ASCII letter right after the multibyte character parses cleanly in the wrong place -- and ruby#3082 gives the non-ASCII characters tokens of their own, taking even the accident away. Past the end of the buffer there is nothing left to step over, so the walk never finishes: `rbs_skip` stops moving at EOF while the loop waits for a position it will never reach. `parse_type("Integer", byte_range: 20...30)` hangs on master. `rbs_lexer_new` returns `NULL` for a `start_pos` it cannot reach, and the extension turns that into an `ArgumentError` alongside the reversed and negative ranges, since all are the caller's mistake rather than anything about the source text. Starting past the end is plain from the buffer's size, so the extension rejects it before parsing; a `NULL` that comes back after that check can only mean a start inside a character. `end_pos` keeps taking any value: clamping with a large number instead of measuring the buffer is ordinary, and the lexer stops at the end on its own.
soutaro
added a commit
to Shopify/rbs
that referenced
this pull request
Aug 12, 2026
`rbs_lexer_new` reaches `start_pos` by stepping one character at a time from the beginning of the buffer, so only the first byte of a character is a position it can stand on. Inside a character the step goes over `start_pos` and lexing quietly begins at the next one. Nothing downstream can tell: the walk keeps `line`, `column` and `char_pos` consistent, so the result points at a real position that simply is not the one asked for. The shift is caught by accident when what it lands on cannot open a token, but an ASCII letter right after the multibyte character parses cleanly in the wrong place -- and ruby#3082 gives the non-ASCII characters tokens of their own, taking even the accident away. Past the end of the buffer there is nothing left to step over, so the walk never finishes: `rbs_skip` stops moving at EOF while the loop waits for a position it will never reach. `parse_type("Integer", byte_range: 20...30)` hangs on master. `rbs_lexer_new` returns `NULL` for a `start_pos` it cannot reach, and the extension turns that into an `ArgumentError` alongside the reversed and negative ranges, since all are the caller's mistake rather than anything about the source text. Starting past the end is plain from the buffer's size, so the extension rejects it before parsing; a `NULL` that comes back after that check can only mean a start inside a character. `end_pos` keeps taking any value: clamping with a large number instead of measuring the buffer is ordinary, and the lexer stops at the end on its own.
soutaro
force-pushed
the
non-ascii-identifiers
branch
from
August 12, 2026 07:33
7416eea to
b83302e
Compare
`word` describes what the class happens to contain rather than what it
is for. Every use is the continuation part of an identifier, and the set
is about to widen to match Ruby's `is_identchar`
ISALNUM(*(ptr)) || (*(ptr)) == '_' || !ISASCII(*(ptr))
which `word` would then describe even less well.
The rule block is realigned so the actions line up again.
src/lexer.c is unchanged. Renaming a named definition affects neither
the generated DFA nor the line numbers it references.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Listing the characters an identifier may contain left no room for
non-ASCII ones. `rbs_next_char` folds every non-ASCII character into a
single code point, and naming that code point in the grammar would tie
the grammar to a value the C side itself calls dummy data.
Subtracting instead states Ruby's own rule
#define is_identchar(p,ptr,end,enc) \
(ISALNUM(*(ptr)) || (*(ptr)) == '_' || !ISASCII(*(ptr)))
without enumerating any punctuation:
ident_char = . \ ([\x00-\x7F] \ [a-zA-Z0-9_]) \ [\uFFFD];
The parentheses matter. `\` is left-associative, so without them this
would subtract the whole ASCII range and never add the word characters
back.
`\uFFFD` is the sentinel `rbs_next_char` reports for a byte that is
invalid under the active encoding. Keeping it out of the class is what
makes such a byte still surface as an ErrorToken instead of being
absorbed into a token, as it already does for `#` comments.
An identifier can now hold a non-ASCII character in any position but the
first: `MI_CONSTANTE_Ñ` is one tUIDENT rather than a tUIDENT followed by
an ErrorToken. A leading non-ASCII character still has no rule, because
nothing yet decides whether it starts a constant or a local.
src/lexer.c grows by 112 lines: two comparisons appended to each of 45
identifier-continuation states, and one more backtracking point.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`identifier` still required `[a-zA-Z_]` in the leading position, so the
non-ASCII members `ident_char` gained could only appear after an ASCII
first character: `:foo日本語` lexed, but `:日本語` did not.
Ruby restricts that position only by excluding ASCII digits -- every
other identifier character may lead one, including a non-ASCII digit, so
`:１２３` is a valid symbol literal. `rbs_next_char` never reports a
non-ASCII character as `[0-9]`, so subtracting the ASCII digits from
`ident_char` expresses exactly that rule:
ident_start = ident_char \ [0-9];
`:123` and `{foo:123}` keep lexing as they did, because the leading
position still bars `[0-9]`. The class is given a name because it states
a rule about identifiers rather than about symbols, and the rules that
lex an identifier take it up next.
The leading `@`s fold into the same rule, which collapses the `:`, `:@`
and `:@@` rules into one and leaves `identifier` with no use, so the
definition goes away.
`foo!`, `foo=`, `@foo` and `@@foo` still asked for `[a-zA-Z_]` at the front. None of them cares about the case of that character, so widening that position to `ident_start` is all they need to accept the same names Ruby does. `@rbs` keeps winning over `@` + identifier: the two match equally far and the keyword is written first.
The rules above this one already take a non-ASCII character anywhere but the first position, and the ones for symbols, instance variables and class variables take it there too. What is left is a bare identifier, and there the first character is not just a character: RBS reads its case to tell a class name from an interface name from an alias name. `rbs_next_char` reports every character outside ASCII as one and the same code point, so a rule cannot read that case on its own. The encoding could be asked, and its answer would be right -- the tables in `rbs_encoding.c` are a copy of prism's, and prism is what decides this in Ruby, so the two agree on every character in every encoding rbs knows. `TypeName#kind` is the problem. It reads `[[:upper:]]`, which is Onigmo's table, not prism's, and the two are different functions: they disagree about 891 characters, among them the full-width capitals of every EUC-JP and Shift_JIS variant, which Ruby does start a constant on. Agreeing would mean carrying prism's table on the Ruby side as well -- 662 code point ranges, 75 single-byte tables and the Japanese byte ranges, kept in step with the C copy forever. So RBS takes less than Ruby gives. A name whose kind is read from its first character has to open with ASCII, and every other name may open with whatever Ruby accepts. Those names get a token of their own: the parser takes `tNONASCIIIDENT` for a method name, a parameter name and a keyword, and nowhere a type name is read, so `class Foo日本語` parses and `class 日本語` does not. An incorrect `byte_range` lands on such a name, so the token it is reported against is a real one now instead of `ErrorToken`.
Covers the names the lexer now takes -- classes, modules, constants, methods, parameters, keywords, instance and class variables, attributes, interfaces, aliases and globals -- and the line the first character draws: a name whose kind is read from it must open with ASCII, everything else need not. The negative cases run three characters through every position that reads a kind, including a titlecase letter, which Ruby does start a constant on. That is the clearest case of RBS taking less than Ruby gives, so it is worth saying in a test. ASCII-8BIT has no invalid bytes, so a high byte there is an identifier character, exactly as it is for Ruby: `\x80abc = 1` assigns a local variable. None of those bytes is ASCII, so a name may not open with one. `TypeName#kind` never sees a name that opens outside ASCII from the parser. One built by hand still can, and the test says what it reads there: nothing.
The three code point tables in this file are a copy of the ones in prism's `src/encoding.c`, and had fallen behind: the version they came from predates the uppercase letters that Unicode has added since. Refreshed against prism 1.9.0, which is the table Ruby 4.0 carries. Only the three lengths and the values change. Nothing in rbs reads them today -- `alpha_char`, `alnum_char` and `isupper_char` have no callers, because the lexer asks the encoding for nothing beyond the width of a character. They are part of the vendored encoding layer all the same, and a vendored copy that has drifted from its source is worse than no copy: the next reader has no way to tell which Unicode version it is looking at.
re2c writes it and `rake confirm_lexer` keeps it honest; the file to review is src/lexer.re. `linguist-generated` collapses it in GitHub diffs and leaves it out of the language statistics.
soutaro
force-pushed
the
non-ascii-identifiers
branch
from
August 12, 2026 07:59
978fad0 to
b3d16a3
Compare
soutaro
marked this pull request as ready for review
August 12, 2026 07:59
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
RBS rejects identifiers that Ruby accepts, such as
def 日本語or@クラス変数. This PR teaches the lexer Ruby's identifier rule —ISALNUM(c) || c == '_' || !ISASCII(c)— so method names, parameter names, instance/class/global variables, and symbols can contain, and start with, characters outside ASCII.The exception is a name whose first character decides its kind: class/module names, interface names, type alias names, and generics type parameters must still start with an ASCII character. Deciding the kind of a non-ASCII character consistently is impractical: the lexer consults prism's Unicode tables in the C extension, while
TypeName#kindasks Onigmo's[[:upper:]], and the two disagree on hundreds of characters — closing the gap would mean shipping a copy of prism's tables on the Ruby side and keeping it in sync forever. This is the one place where RBS accepts less than Ruby.Mechanically, the lexer emits a dedicated token (
tNONASCIIIDENT) for identifiers that open with a non-ASCII character, and the parser accepts it only in positions where the kind question does not come up.The last commit refreshes the Unicode tables in
src/util/rbs_encoding.cfrom prism 1.9.0; they had fallen behind Unicode 16.🤖 Generated with Claude Code