Skip to content

perf(driver): stream ClickHouse HTTP responses instead of full-text buffering - #74

Merged
ZhuchkaTriplesix merged 1 commit into
devfrom
issue/49-stream-clickhouse-http-responses
Sep 29, 2026
Merged

ZhuchkaTriplesix merged 1 commit into
devfrom
issue/49-stream-clickhouse-http-responses

Conversation

@ZhuchkaTriplesix

Copy link
Copy Markdown
Member

Summary

db.query's tabular path called reqwest::Response::text(), forcing the entire HTTP response body into one contiguous String before any parsing started. For a large analytical SELECT (tens of MB of JSON lines), the raw text and the parsed QueryResult rows coexisted in memory at once, risking OOM under the 256 MB ClickHouse Sandbox ceiling.

  • New ClickHouseClient::post_sql_response (src/driver/client.rs) returns the raw streaming reqwest::Response on success — reusing the existing readonly-setting-conflict retry logic — instead of buffering it into a String.
  • New src/driver/streaming::stream_compact_output reads that response's bytes_stream() through a LinesCodec-based FramedRead and parses rows incrementally as they arrive, stopping — without reading the rest of the network response — once limit rows are parsed or a 200 MB safety byte cap is hit, even when no limit was given at all.
  • Extracted parse_columns/parse_and_normalize_row out of parse_compact_output (src/mapper/row_compact.rs) so both the buffered parser (schema introspection, mock mode, tree building — all small, bounded responses) and the new streaming parser (large query results) share identical row normalization instead of two copies of the same logic.
  • QueryResult gained isTruncated, set whenever limit or the safety cap cut a result short, so callers can tell rows isn't the complete set — also backfilled onto the buffered parse_compact_output path for consistency.
  • handle_query's tabular branch now goes through post_sql_response + stream_compact_output; the non-tabular (mutation/DDL) branch is unchanged since those ClickHouse responses are always small.

Closes #49

Test plan

  • cargo test — 110/110 passing, including new driver::streaming tests (row parsing, limit truncation + isTruncated, chunk-boundary line reassembly — bytes delivered one at a time to prove the codec correctly reassembles lines split across network chunks, empty body, malformed input) and updated row_compact tests for isTruncated
  • cargo fmt --all -- --check
  • cargo clippy --all-targets --all-features -- -D warnings

…uffering. Closes #49

db.query's tabular path called reqwest::Response::text(), forcing the
entire HTTP response body into one contiguous String before any parsing
started. For a large analytical SELECT (tens of MB of JSON lines), the
raw text and the parsed QueryResult rows coexisted in memory at once,
risking OOM under the 256 MB ClickHouse Sandbox ceiling.

- New ClickHouseClient::post_sql_response returns the raw streaming
  reqwest::Response on success (reusing the existing readonly-retry
  logic) instead of buffering it into a String.
- New src/driver/streaming::stream_compact_output reads that response's
  bytes_stream() through a LinesCodec-based FramedRead and parses rows
  incrementally as they arrive, stopping — without reading the rest of
  the network response — once `limit` rows are parsed or a 200 MB
  safety byte cap is hit even when no limit was given.
- Extracted parse_columns/parse_and_normalize_row out of
  parse_compact_output so both the buffered (schema introspection, mock
  mode, tree building) and streaming (large query results) parsers share
  identical row normalization.
- QueryResult gained isTruncated, set whenever limit or the safety cap
  cut a result short, so callers can tell rows isn't the complete set.
- handle_query's tabular branch now goes through post_sql_response +
  stream_compact_output; the non-tabular (mutation/DDL) branch is
  unchanged, since those responses are always small.
@ZhuchkaTriplesix
ZhuchkaTriplesix merged commit 3b12c8b into dev Sep 29, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant