Reject a byte position the lexer cannot start on - #3083
Merged
Conversation
soutaro
force-pushed
the
reject-unreachable-start-pos
branch
2 times, most recently
from
August 12, 2026 07:19
aa5e0a5 to
a29c0f3
Compare
`rbs_lexer_new` reaches `start_pos` by stepping one character at a time from the beginning of the buffer, so only the first byte of a character is a position it can stand on. Inside a character the step goes over `start_pos` and lexing quietly begins at the next one. Nothing downstream can tell: the walk keeps `line`, `column` and `char_pos` consistent, so the result points at a real position that simply is not the one asked for. The shift is caught by accident when what it lands on cannot open a token, but an ASCII letter right after the multibyte character parses cleanly in the wrong place -- and ruby#3082 gives the non-ASCII characters tokens of their own, taking even the accident away. Past the end of the buffer there is nothing left to step over, so the walk never finishes: `rbs_skip` stops moving at EOF while the loop waits for a position it will never reach. `parse_type("Integer", byte_range: 20...30)` hangs on master. `rbs_lexer_new` returns `NULL` for a `start_pos` it cannot reach, and the extension turns that into an `ArgumentError` alongside the reversed and negative ranges, since all are the caller's mistake rather than anything about the source text. Starting past the end is plain from the buffer's size, so the extension rejects it before parsing; a `NULL` that comes back after that check can only mean a start inside a character. `end_pos` keeps taking any value: clamping with a large number instead of measuring the buffer is ordinary, and the lexer stops at the end on its own.
soutaro
force-pushed
the
reject-unreachable-start-pos
branch
from
August 12, 2026 07:21
a29c0f3 to
49d5b5f
Compare
soutaro
marked this pull request as ready for review
August 12, 2026 07:21
soutaro
enabled auto-merge
August 12, 2026 07:22
dak2
pushed a commit
to dak2/rbs
that referenced
this pull request
Aug 12, 2026
ruby#3083 made `rbs_lexer_new` and `rbs_parser_new` return `NULL` for a `start_pos` the lexer cannot start on, and taught the C extension to raise `ArgumentError` for it. The WebAssembly shim went on using the result unchecked: linear memory has no protected page at address 0, so nothing traps there -- the module reads and writes whatever sits at offset 0 and returns a parse failure with an empty result, which the Ruby side then tries to decode as an error blob. That is the `undefined method 'zero?' for nil` behind the three JRuby failures in `RBS::TypeParsingTest`. So the shim checks for `NULL` and reports it: the parse entry points gain a status of their own for a `start_pos` the parser will not take, and `RBS::Parser` turns it into the same `ArgumentError` the extension raises. Negative and reversed ranges get a status too, rather than the parse-error one they shared with an empty blob. The `end_pos` rule had to move as well. The extension takes any `end_pos` -- clamping with a large number instead of measuring the buffer is ordinary, and the lexer stops at the end of the input on its own, because a Ruby string keeps a NUL terminator to stop at. A buffer the host wrote into linear memory has nothing behind it, so the shim rejected anything past the end instead. It now clamps to the buffer, which is the same position the extension stops at, and `parse_type("Integer", byte_range: 0...9999)` parses on JRuby as it does on CRuby. Verified by compiling the shim natively against `src/` under ASan/UBSan (it is plain C) and driving the entry points over exact-sized allocations: the ranges above return their statuses with no read past the end of the buffer, and the pre-fix shim reports the null dereference at `rbs_wasm_parse_type`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AqEsvUoBfRtqECvrGRWTpy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
rbs_lexer_newreachesstart_posby stepping one character at a time from the beginning of the buffer, so the only positions it can start lexing from are the first bytes of characters. Abyte_rangecan hand it two other kinds of position:Inside a character. The step walks over
start_posand lexing quietly begins at the next character boundary. Nothing downstream can tell:line,columnandchar_posall stay consistent — they just describe a position the caller did not ask for. Today the shift often surfaces as a syntax error by accident, because a non-ASCII character cannot open a token; but an ASCII letter right after a multibyte character parses cleanly in the wrong place, and #3082 gives non-ASCII characters tokens of their own, which takes even the accident away.Past the end of the buffer. There is nothing left to step over, so the walk never finishes:
rbs_skipstops moving at EOF while the loop waits for a byte position it will never reach.This PR makes the walk report an unreachable
start_pos:rbs_lexer_newandrbs_parser_newreturnNULL, and the C extension turns that into anArgumentErroralongside the existing reversed/negative range validation, since both are the caller's mistake rather than anything about the source text.end_poskeeps accepting any value: clamping with a large number instead of measuring the buffer is ordinary, and the lexer stops at the end of input on its own.Note: this overlaps with #3082 in
test/rbs/type_parsing_test.rb— both rewritetest_parse__byte_range_incorrect— so whichever lands second needs a trivial rebase.🤖 Generated with Claude Code