The BRE engine matches bytes, so . and bracket classes split multi-byte UTF-8
characters. Substituting across the split silently corrupts the text.
$ ved
a
café
.
s/./Y/g
,p
YYYYY
Five replacements for four characters: café is five bytes, and . matched
each byte. GNU ed in a UTF-8 locale prints YYYY.
The corrupting case is a match that covers part of a character:
$ ved
a
café
.
s/f./Z/
w out.txt
wrote 7 bytes to out.txt
f. consumed f and the first byte of é, leaving the second byte orphaned.
run_substitute rebuilds the line with String::from_utf8_lossy
(src/main.rs:768), which turns the orphan into U+FFFD, so the file on disk is
caZ followed by EF BF BD. The original character is gone and no error was
reported. GNU ed produces caZ.
This contradicts the README, which says under "Limitations worth knowing" that
"UTF-8 text with multi-byte characters works".
Scope: the engine is byte-oriented throughout — Atom::Literal(u8),
atom_matches(atom, byte), text: &[u8] — which is correct and fast for ASCII
and is the right representation for the \0NN octal escapes. Making it
character-aware is not a local change. Atom::Dot has to consume a whole UTF-8
sequence, Element::One has to advance by that width, match_star over a
Dot has to step by characters, and bracket expressions have to hold
characters rather than bytes.
A cheaper interim step that removes the data loss without making the engine
character-aware: have run_substitute reject a match whose start or end is not
on a character boundary, and report it rather than lossily converting. That
turns silent corruption into an error message, which is the property a text
editor most needs, and leaves the .-counts-bytes behavior as a documented
limitation until the engine changes.
Found while writing the regex chapters of the ved book. Related: #8, #9.
The BRE engine matches bytes, so
.and bracket classes split multi-byte UTF-8characters. Substituting across the split silently corrupts the text.
Five replacements for four characters:
caféis five bytes, and.matchedeach byte. GNU ed in a UTF-8 locale prints
YYYY.The corrupting case is a match that covers part of a character:
f.consumedfand the first byte ofé, leaving the second byte orphaned.run_substituterebuilds the line withString::from_utf8_lossy(
src/main.rs:768), which turns the orphan into U+FFFD, so the file on disk iscaZfollowed by EF BF BD. The original character is gone and no error wasreported. GNU ed produces
caZ.This contradicts the README, which says under "Limitations worth knowing" that
"UTF-8 text with multi-byte characters works".
Scope: the engine is byte-oriented throughout —
Atom::Literal(u8),atom_matches(atom, byte),text: &[u8]— which is correct and fast for ASCIIand is the right representation for the
\0NNoctal escapes. Making itcharacter-aware is not a local change.
Atom::Dothas to consume a whole UTF-8sequence,
Element::Onehas to advance by that width,match_starover aDothas to step by characters, and bracket expressions have to holdcharacters rather than bytes.
A cheaper interim step that removes the data loss without making the engine
character-aware: have
run_substitutereject a match whose start or end is noton a character boundary, and report it rather than lossily converting. That
turns silent corruption into an error message, which is the property a text
editor most needs, and leaves the
.-counts-bytes behavior as a documentedlimitation until the engine changes.
Found while writing the regex chapters of the ved book. Related: #8, #9.