perf(transport): eliminate full-string heap reallocation in write_ndjson framing - #75
Merged
Merged
Conversation
…son framing. Closes #50 write_ndjson sanitized raw '\n'/'\r' bytes by collecting the entire payload into a new String via .chars().map(...).collect(), so every multi-megabyte tabular response containing a stray newline (e.g. in an error message or embedded query text) got a full extra heap copy just to swap a handful of bytes for spaces. '\n' and '\r' are single-byte ASCII code points, so this rewrites the sanitization as a byte-slice scan: write the clean slice up to each newline straight to the writer, then a single space byte, and advance past it — no intermediate String is ever allocated. A payload with no newlines (the common case) still goes out in one write_all call, same as before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
write_ndjson(src/transport/framing.rs) sanitized raw\n/\rbytes in a payload by collecting the entire string into a newStringvia.chars().map(...).collect(). Every multi-megabyte tabular response containing a stray newline (an error message, embedded query text, etc.) paid for a full extra heap copy of the whole payload just to swap a handful of bytes for spaces.\nand\rare single-byte ASCII code points, so the sanitization is now a byte-slice scan: the clean slice up to each newline is written straight towriter, followed by a single space byte, andstartadvances past it — no intermediateStringis ever allocated. A payload with no newlines (the common case) still goes out in onewrite_allcall, same as before.Closes #50
Test plan
cargo test— 112/112 passing, including two new regression tests: newline as the very first/last byte plus consecutive newlines (empty slice between them), and a realistic multi-megabyte payloadcargo fmt --all -- --checkcargo clippy --all-targets --all-features -- -D warnings