Make aspen usable on a non-English VB6 code base: tokenizer hang, designer block, Windows-1252 round trip, exit code

[MarioRial22]

Four fixes found while putting aspen to work on a production VB6 code base: 11 compilable projects, ~660 source files, written in Spanish, so accented characters everywhere. They are separate commits and can be reviewed one at a time, but they are one PR because each one only becomes visible once the previous is out of the way — the hang hides the encoding problem, which hides the exit code problem.

Before: aspen check hung on 6 of the 11 projects, and aspen fmt silently processed 11 of 375 files.
After: check completes on all 11 in 4.0s, and fmt reads all 375.

---

1. tokenize() hangs on a multi-byte character in code

Public Function Añadir() As String   ' never returns, one core at 100%

The tokenizer loop ends with a fallback that consumes one character and reports it as unknown, so the loop always makes progress:

if let Some(token_text) = input.take_count(1) { ... }

SourceStream::take_count works in bytes and returns None when the count would not land on a UTF-8 character boundary. ñ is two bytes, so nothing is consumed, no branch matches on the next pass either, and the loop spins forever.

Non-ASCII inside comments and string literals is fine — take_line_comment and take_string_literal consume those wholesale, so the character never reaches the fallback. It has to sit in code, which an accented identifier does, and VB6 accepts those. A one-byte unknown character in the same position, such as a backtick, exits normally in 0.17s; the difference is purely the boundary check.

Fix: SourceStream::take_character(), which consumes exactly one char however many bytes it occupies. take_count is left alone — its byte semantics are what every other caller wants. The loop also now breaks if the stream reports it is not empty yet nothing can be taken: a tokenizer that cannot advance should stop rather than hang.

2. fmt rewrites the designer block of forms and classes

On a small form here, aspen fmt turned VERSION 5.00 into Version 5.00, flattened every control to column zero and took the file from 3436 to 2796 bytes.

That block is not source code. The VB6 IDE writes it, reads it, and rewrites it in its own layout every time the form is saved, so reformatting it comes straight back as diff noise. It is also where a form's control geometry lives, which makes silently rewriting it the riskiest thing a formatter can do to a VB6 project.

Fix: VersionStatement and PropertiesBlock subtrees are copied through unchanged. The same nodes cover a .cls header, so class files are protected too. Code after the block is still formatted, which the third test pins down. On the same form the change is now 40 bytes instead of 640, all of it in the code section.

3. fmt cannot read Windows-1252, and would corrupt it on write

process_file read with std::fs::read_to_string, which requires valid UTF-8, so every source file holding an accented character was rejected — 364 of 375 files here. The summary still printed 11 of 375 files would be reformatted, presenting the other 364 as fine.

Reading is only half of it: std::fs::write emits the Rust string as UTF-8. Fixing the read alone would have turned "cannot read 364 files" into "silently transcodes 364 files to UTF-8", which VB6 then reads incorrectly — a worse failure, because it is invisible.

Fix: read through SourceFile, the way vb6semantic already does, and add vb6parse::io::encode_windows_1252 to write back. Encoding is refused rather than made lossy: encoding_rs maps a character outside Windows-1252 to ?, losing it silently, so an unrepresentable character is returned as an error with its byte offset. Verified by cycling all 256 byte values through decode and back — Windows-1252 decodes every byte, so the whole range must survive, and it does.

4. fmt exits 0 when files fail

Failures were folded into "unchanged" and the exit code depended only on how many files would be reformatted:

Err(e) => { eprintln!("Error formatting {}: {}", ...); (file, false) }

A run where every file failed to read printed its errors and exited 0. fmt --check therefore passed in exactly the case where it should fail, which makes it worthless as a CI gate. Write mode never exited non-zero at all.

Fix: failures are counted separately, summarised on stderr after the file list, and produce a non-zero exit in both modes.

---

Tests

Nine new tests. The three tokenizer ones hang rather than fail without the fix, which is the point of them.

• lexer::tests::multibyte_character_* — ñ, año, €, 日, covering 2, 3 and 4-byte characters; the whole character is reported as one unknown token and tokenizing continues past it; no regression for the comment and string-literal paths.
• io::source_file::encode_tests::* — accented round trip, all 256 byte values, and refusal to write ? for an unrepresentable character.
• vb6format/tests/designer_block.rs — form block untouched, VERSION casing kept, code after the block still formatted.

cargo test --workspace --locked on Windows: 5761 pass. Eight class_load snapshot tests fail (five audiostation, three cdiu_beat_up_editor) because there is no committed .snap baseline for them, so they store a .snap.new and fail. Those eight fail identically on an unmodified master here — unrelated to this PR.

Not addressed here

Two more things surfaced while testing, left out to keep this reviewable:

• SourceFile::decode (the strict variant) rejects a plain Windows-1252 byte such as 0xF1 with "may not use latin-1 (Windows-1252) code page", which reports the opposite of what happened. The tests here use decode_with_replacement, the path the tools actually take.
• The keyword pass writes On Error Goto for On Error GoTo, and indents line labels that VB6 convention puts at column zero.

Happy to split any of the four out if you would rather take them separately.

[MarioRial22]

Correction to what I wrote above about the eight failing class_load snapshot tests. I said they fail because there is no committed .snap baseline. That is wrong — the baselines are committed, and the failures are an artefact of my machine, not of this repo.

The cause is line endings. The test-data blobs hold LF; this Windows checkout had core.autocrlf=true globally, so they materialised as CRLF, and the snapshots — generated from LF — no longer match. Every diff is text: "\n" against text: "\r\n" and nothing else.

So: those eight are green in your CI, they are unrelated to this PR, and there is nothing here for you to fix. I should have looked at the diff before explaining it rather than after.

The one thing that might be worth having is a note for contributors on Windows to set core.autocrlf=false, since .gitattributes in the superproject does not reach submodule checkouts. Happy to send that separately if you want it.