fix(documents): refuse to save a buffer that was decoded lossily - #379
Conversation
Every read path decodes leniently, so a file in a legacy encoding (GBK,
Big5, Shift-JIS, EUC-KR, CP1251...) opens as U+FFFD mojibake instead of
failing. U+FFFD is not reversible: the original bytes are gone from the
buffer. Writing that buffer back over the source file - auto-save does
it 1.5s after the first keystroke - destroys the document permanently.
Both read commands now report the fidelity of their decode, the tab
carries it, and `documentSession` refuses to write such a buffer over
the file it came from. "Save As" to a different path stays open: the new
file is genuinely UTF-8, so writing it is correct, and it is the user's
way out.
- `decode_utf8_lossy` returns `DecodedText { content, lossy }`. Still one
decoder; it just no longer throws away what it knew.
- `read_file_content_checked` is added rather than changing
`read_file_content`, whose two remaining callers live in a file this
branch must not touch and read the result through `as string` - an
assertion, not a check, so a tuple would type-check and break at
runtime.
- The truncated-preview branch trims to a character boundary *before*
decoding, so a valid UTF-8 file cut mid-character is not misreported
as lossy. That order is load-bearing: reversed, every large CJK
document would be locked out of saving.
- Window restore reads through the checked command too. Without it,
relaunching the app laundered the flag away and the next auto-save
destroyed the document.
- `hasReplacementChars` is a required field of the cross-window transfer
payload and is strictly validated, per that module's existing rule
that a missing field is rejected rather than defaulted. A falsy
default here reads as "safe to overwrite".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
bb942bc to
34bcdd6
Compare
|
Rebased onto #374 added a second entry into
The other two overlaps were mechanical: One new test ( |
The problem
#371 made every read path decode leniently, which is the right call for opening a file: a document in a legacy encoding (GBK, Big5, Shift-JIS, EUC-KR, CP1251…) now opens as U+FFFD mojibake instead of failing outright, and it no longer opens-or-fails purely by size.
What must not follow is writing that buffer back. U+FFFD is not reversible — the original bytes are gone from the buffer, so saving replaces a readable document with permanent mojibake. With auto-save on, that happens 1.5s after the user's first keystroke, before they have any chance to realise the file was never really loaded.
Two paths made it worse than it looks:
≤ max_bytesbranch ofopen_markdown_previewbecame lenient too, so small legacy-encoded files went straight into an editable buffer with nothing marking them;The fix
Both read commands report the fidelity of their decode, the tab carries it, and
documentSessionrefuses to write such a buffer over the file it came from. Every writer — Ctrl+S, the auto-save timer, the close dialogs, the task-checkbox toggle — funnels throughsaveContent/saveContentAs, so that is the one place that has to hold.Save As to a different path stays open. The new file is genuinely UTF-8, so writing it is correct, and it is the user's way out; the flag clears once the buffer has a file of its own. Picking the source file again in the dialog is refused, because that is the same destructive overwrite reached another way.
This is damage control, not encoding support. Markpad still cannot read GBK — that needs a real decoder. This only stops it from overwriting what it could not read.
Details worth a reviewer's eye
decode_utf8_lossyreturnsDecodedText { content, lossy }. Still exactly one decoder (a test pins that); it just no longer throws away whatString::from_utf8already told it.read_file_contentkeeps its shape;read_file_content_checkedis added alongside. Its two remaining callers are inMarkdownViewer.svelteand read the result through(await invoke(…)) as string— an assertion, not a check — so returning a tuple would have type-checked cleanly and turnedrawContentinto an array at runtime. Those two call sites only re-read a file whose tab is already flagged, so they are safe today; migrating them is a small follow-up.hasReplacementCharsis a required field of the cross-window transfer payload, strictly validated, following that module's existing rule that a missing field is rejected rather than defaulted. A falsy default here reads as "safe to overwrite", and the failure mode is a destroyed document. The broker is in-memory and both windows are the same binary, so there is no version skew to tolerate./notes/Legacy.mdfor a tab opened as/notes/legacy.mdis allowed through. Closing that needs the backend to canonicalize both sides; the same-path check is a courtesy on top of the guard that matters.Tests
16 new behaviour tests (
scripts/lossyDecodeSaveGuard.test.ts) plus 6 Rust tests. Run against unmodifiedmaster, 15 of the 16 fail. The one that passes is a deliberate anchor on existing structure — that auto-save and the close dialogs sharesaveContent— so it turns red if a future change routes a writer around the choke point.Follow-up (not in this PR)
When the guard refuses,
saveContentreturnsfalse, and the auto-save timer inMarkdownViewer.sveltetoasts its generic "Auto-save failed" on every re-arm. The specific explanation here is shown once per tab, but the generic one still repeats while typing. Quieting it needsMarkdownViewer.svelte, which #374 is currently changing; it is best done together with migrating the tworead_file_contentcall sites noted above.🤖 Generated with Claude Code