Infinite loop in identifier lexer on non-ASCII byte (hangs on Portuguese/Spanish source)

[arturbersan]

Summary

aspen check hangs indefinitely (100% CPU, no output) when a source file contains a non-ASCII byte in
identifier position. Comments and string literals with the same bytes are handled correctly. The
lexer appears not to advance the cursor past a byte it does not recognize as an identifier character.

Versions

• aspen 1.0.2
• vb6parse 1.2.2
• Linux x86_64, installed via cargo install aspen

Minimal reproduction

mkdir repro && cd repro

# a module whose third line is a single 0xE1 byte in statement position
printf 'Attribute VB_Name = "M"\r\nOption Explicit\r\n\xe1\r\n' > M.bas
printf 'Type=Exe\r\nModule=M; M.bas\r\nStartup="Sub Main"\r\n' > P.vbp

timeout 15 aspen check P.vbp ; echo "exit=$?"    # -> exit=124 (timed out)

Control, identical except for an ASCII identifier:

printf 'Attribute VB_Name = "M"\r\nOption Explicit\r\nDim x As Long\r\n' > OK.bas
printf 'Type=Exe\r\nModule=M; OK.bas\r\nStartup="Sub Main"\r\n' > OK.vbp

timeout 15 aspen check OK.vbp ; echo "exit=$?"   # -> "No errors found", exit=0

Test matrix

Case	Result
Accented char in identifier — cp1252 (0xED)	hang
Accented char in identifier — UTF-8 (0xC3 0xAD)	hang
Bare 0xE1 in statement position	hang
Sub name with an accent	hang
Local variable (Dim á As Long)	hang
Same cases inside a .cls	hang
Same cases with ASCII identifiers	ok (exit 0)
Accented char inside a comment	ok (exit 0)
Accented char inside a string literal	ok (exit 0)

Encoding does not matter — cp1252 and UTF-8 hang identically. Comments and strings handle high bytes
correctly, which points at the identifier lexer specifically.

Evidence it is a spin, not slowness

99.8% CPU with RSS flat at ~3.7 MB (no allocation), still running after 90 s.

Why this matters beyond one file

This is not an edge case for non-English codebases — it is the normal case. Any VB6 source written in
Portuguese or Spanish uses identifiers like Descrição, Código, Início, Tamaño, Año. Measured
on a real corpus of 610 VB6 source files (.bas/.cls/.frm/.ctl): 26 files (4.3%) carry a
non-ASCII byte outside comments and string literals.

Because aspen check runs against a .vbp project, a single such file hangs the whole run —
9 of 9 sample projects in that corpus contain at least one.

Consequence for tooling

A hang is not a usable verdict. It cannot be read as "does not compile", so a timeout wrapper can only
report "parser incomplete", never a real result. That makes aspen unusable in CI for this class of
codebase, which is a shame — it is otherwise the only free, cross-platform VB6 validator that runs
headless on Linux, and it works cleanly on ASCII-only projects.

Suggested direction

Whatever the identifier lexer does on an unrecognized byte, it should consume it (or bail with a parse
error) rather than looping. Deciding whether high bytes are valid in identifiers is a separate
question — VB6 itself accepts them — but even rejecting them with an error would be a strict
improvement over hanging.