Exclude CR from comment tokens to fix an off-by-one Location#end_line on CRLF files

[tufusa]

Background

Currently, the pattern for comment tokens is "#" (. \ [\x00\uFFFD])*:

rbs/src/lexer.re

Lines 61 to 66 in d89e4ad

	"#" (. \ [\x00\uFFFD])* {
	return rbs_next_token(
	lexer,
	lexer->first_token_of_line ? tLINECOMMENT : tCOMMENT
	);
	}

This works correctly in LF environments, but the . pattern excludes only LF, not CR; therefore \r is included at the end of the comment token in CRLF environments such as Windows.

As a result, the value of Location#end_line for comments ends up being one line greater than actually is. This occurs because while Buffer#ranges

rbs/lib/rbs/buffer.rb

Lines 42 to 49 in d89e4ad

	lines.each do |line|
	size0 = line.size
	line = line.chomp
	range = offset...(offset+line.size)
	ranges << range
	offset += size0
	end

correctly removes \r\n (L44), Buffer#pos_to_loc which is passed a value for pos that is one greater than the actual value

rbs/lib/rbs/buffer.rb

Lines 56 to 59 in d89e4ad

	def pos_to_loc(pos)
	index = ranges.bsearch_index do |range|
	pos <= range.end ? true : false
	end

incorrectly finds the next line.

This will cause incorrect behavior in tools built on RBS::Parser.lex, for example, RBS/Layout/ExtraSpacing cop in rubocop-on-rbs (ksss/rubocop-on-rbs#152; in fact, I discovered this issue while investigating the false detection of this cop).

This token behavior itself existed prior to RBS 4, but it looks to become apparent because, starting with RBS 4, String#chomp is called before the character count is performed within Buffer#ranges.

Reproduction

require 'rbs'

src = "class Klass\r\n  # Comment\r\nend\r\n"
buffer = RBS::Buffer.new(name: Pathname("test.rbs"), content: src)
tokens = RBS::Parser.lex(buffer).value

comment = tokens.find { _1.type == :tLINECOMMENT }
puts comment.location.end_line # => 3 (if LF only: 2)

• ruby 4.0.5
• rbs 4.1.2

Changes

• Added \r to the pattern's subtraction group: "#" (. \ [\x00\r\uFFFD])*. This causes the comment token to stop reading as soon as \r appears.
• Added the test RBS::ParserTest#test__lex_crlf next to test__lex. This verifies that \r is treated as a single independent TRIVIA token as usual, even in the case of comments.

Thank you!

[soutaro]

Thanks!