Skip to content

fix(documents): refuse to save a buffer that was decoded lossily - #379

Merged
PathGao merged 1 commit into
sftwrdotdev:masterfrom
PathGao:fix/refuse-saving-lossy-decoded-files
Aug 2, 2026
Merged

fix(documents): refuse to save a buffer that was decoded lossily#379
PathGao merged 1 commit into
sftwrdotdev:masterfrom
PathGao:fix/refuse-saving-lossy-decoded-files

Conversation

@PathGao

@PathGao PathGao commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator

The problem

#371 made every read path decode leniently, which is the right call for opening a file: a document in a legacy encoding (GBK, Big5, Shift-JIS, EUC-KR, CP1251…) now opens as U+FFFD mojibake instead of failing outright, and it no longer opens-or-fails purely by size.

What must not follow is writing that buffer back. U+FFFD is not reversible — the original bytes are gone from the buffer, so saving replaces a readable document with permanent mojibake. With auto-save on, that happens 1.5s after the user's first keystroke, before they have any chance to realise the file was never really loaded.

Two paths made it worse than it looks:

  • the ≤ max_bytes branch of open_markdown_preview became lenient too, so small legacy-encoded files went straight into an editable buffer with nothing marking them;
  • session restore read through the bare command, so relaunching the app laundered the fact away even if it had been known before.

The fix

Both read commands report the fidelity of their decode, the tab carries it, and documentSession refuses to write such a buffer over the file it came from. Every writer — Ctrl+S, the auto-save timer, the close dialogs, the task-checkbox toggle — funnels through saveContent/saveContentAs, so that is the one place that has to hold.

Save As to a different path stays open. The new file is genuinely UTF-8, so writing it is correct, and it is the user's way out; the flag clears once the buffer has a file of its own. Picking the source file again in the dialog is refused, because that is the same destructive overwrite reached another way.

This is damage control, not encoding support. Markpad still cannot read GBK — that needs a real decoder. This only stops it from overwriting what it could not read.

Details worth a reviewer's eye

  • decode_utf8_lossy returns DecodedText { content, lossy }. Still exactly one decoder (a test pins that); it just no longer throws away what String::from_utf8 already told it.
  • read_file_content keeps its shape; read_file_content_checked is added alongside. Its two remaining callers are in MarkdownViewer.svelte and read the result through (await invoke(…)) as string — an assertion, not a check — so returning a tuple would have type-checked cleanly and turned rawContent into an array at runtime. Those two call sites only re-read a file whose tab is already flagged, so they are safe today; migrating them is a small follow-up.
  • The truncation order is load-bearing. The preview trims to a UTF-8 character boundary before decoding, so a valid file cut mid-character is not misreported. Reversed, every large CJK or emoji document would be permanently locked out of saving — a guard worse than the bug. Both directions are pinned by tests.
  • hasReplacementChars is a required field of the cross-window transfer payload, strictly validated, following that module's existing rule that a missing field is rejected rather than defaulted. A falsy default here reads as "safe to overwrite", and the failure mode is a destroyed document. The broker is in-memory and both windows are the same binary, so there is no version skew to tolerate.
  • Known boundary, documented in the code: the same-path comparison is exact string equality, so on a case-insensitive filesystem a Save As typed as /notes/Legacy.md for a tab opened as /notes/legacy.md is allowed through. Closing that needs the backend to canonicalize both sides; the same-path check is a courtesy on top of the guard that matters.

Tests

16 new behaviour tests (scripts/lossyDecodeSaveGuard.test.ts) plus 6 Rust tests. Run against unmodified master, 15 of the 16 fail. The one that passes is a deliberate anchor on existing structure — that auto-save and the close dialogs share saveContent — so it turns red if a future change routes a writer around the choke point.

npm run check   0 errors, 0 warnings
npm test        248 / 248
cargo test      82 / 82

Follow-up (not in this PR)

When the guard refuses, saveContent returns false, and the auto-save timer in MarkdownViewer.svelte toasts its generic "Auto-save failed" on every re-arm. The specific explanation here is shown once per tab, but the generic one still repeats while typing. Quieting it needs MarkdownViewer.svelte, which #374 is currently changing; it is best done together with migrating the two read_file_content call sites noted above.

🤖 Generated with Claude Code

Every read path decodes leniently, so a file in a legacy encoding (GBK,
Big5, Shift-JIS, EUC-KR, CP1251...) opens as U+FFFD mojibake instead of
failing. U+FFFD is not reversible: the original bytes are gone from the
buffer. Writing that buffer back over the source file - auto-save does
it 1.5s after the first keystroke - destroys the document permanently.

Both read commands now report the fidelity of their decode, the tab
carries it, and `documentSession` refuses to write such a buffer over
the file it came from. "Save As" to a different path stays open: the new
file is genuinely UTF-8, so writing it is correct, and it is the user's
way out.

- `decode_utf8_lossy` returns `DecodedText { content, lossy }`. Still one
  decoder; it just no longer throws away what it knew.
- `read_file_content_checked` is added rather than changing
  `read_file_content`, whose two remaining callers live in a file this
  branch must not touch and read the result through `as string` - an
  assertion, not a check, so a tuple would type-check and break at
  runtime.
- The truncated-preview branch trims to a character boundary *before*
  decoding, so a valid UTF-8 file cut mid-character is not misreported
  as lossy. That order is load-bearing: reversed, every large CJK
  document would be locked out of saving.
- Window restore reads through the checked command too. Without it,
  relaunching the app laundered the flag away and the next auto-save
  destroyed the document.
- `hasReplacementChars` is a required field of the cross-window transfer
  payload and is strictly validated, per that module's existing rule
  that a missing field is rejected rather than defaulted. A falsy
  default here reads as "safe to overwrite".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@PathGao
PathGao force-pushed the fix/refuse-saving-lossy-decoded-files branch from bb942bc to 34bcdd6 Compare August 2, 2026 20:02
@PathGao

PathGao commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased onto c983d7a now that #374 has landed. Three files overlapped and one of the resolutions is a behaviour decision worth calling out:

#374 added a second entry into loadMarkdown — a pane that can write (edit or split mode, the F5 / "reload from disk" path) skips the preview and reads the whole file up front, so the editor is never bound to a partial buffer. That branch fills an editable buffer, which makes it exactly the kind of caller that must not use the unchecked command: it would hand the editor unflagged mojibake. It now reads through read_file_content_checked and reports fidelity like the preview branch does.

ensureFullContent is the one place the bare read_file_content survives, and only because it re-reads a file whose tab loadMarkdown already flagged — setTabRawContent does not clear the flag. A test pins that it is the only remaining unchecked call in documentSession, so a future caller cannot quietly join it.

The other two overlaps were mechanical: isTruncated and hasReplacementChars are independent fields, and the two toast keys are independent strings.

One new test (the editable-pane shortcut reports fidelity too), verified to fail when that branch is reverted to the bare command.

npm run check   0 errors, 0 warnings
npm test        304 / 304
cargo test      82 / 82

@PathGao
PathGao merged commit 4f065b4 into sftwrdotdev:master Aug 2, 2026
4 checks passed
@PathGao
PathGao deleted the fix/refuse-saving-lossy-decoded-files branch August 2, 2026 20:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant