feat(consumer): per-storage max_insert_block_size override#7939
Merged
Conversation
Add an optional per-storage knob, read from snuba.state at `clickhouse_max_insert_block_size:<storage_name>`, that appends `&max_insert_block_size=<n>` to the rust consumer's INSERT URL. Absent or non-positive values leave the setting at ClickHouse's default. Lets us tune insert block size on a single storage at runtime without a deploy or restart, matching the per-storage pattern already used for `clickhouse_load_balancing:<storage_name>`. Co-Authored-By: Claude <noreply@anthropic.com>
Treat the per-storage `clickhouse_max_insert_block_size:<storage_name>` override as a floor only: values below ClickHouse's compiled-in default (1_048_449) are silently dropped instead of being applied. Pulling block size below the server default would not raise it past what ClickHouse already produces on its own, so the URL suffix is now only appended for values that actually take effect. Co-Authored-By: Claude <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 223898c. Configure here.
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
MeredithAnya
approved these changes
May 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
EAP-517
Adds an optional per-storage knob for the rust consumer's ClickHouse INSERTs: when
clickhouse_max_insert_block_size:<storage_name>is set to a positive integer insnuba.state, the consumer appends&max_insert_block_size=<n>to the INSERT URL. Absent or non-positive values leave the setting at ClickHouse's default.Motivation: lets us tune the ClickHouse
max_insert_block_sizeserver setting for a single storage at runtime — for example, to shrink it on a storage hitting memory pressure during inserts — without a deploy or restart.The key shape (
<setting>:<storage_name>) and thesnuba.statebacking intentionally mirror the existing per-storageclickhouse_load_balancing:<storage_name>/clickhouse_load_balancing_first_offset:<storage_name>keys that already drive the samebuild_urlpath.Considered using
sentry-optionsfor the typed schema, but it doesn't support open per-storage maps: the namespace validator force-injectsadditionalProperties: falseat both the top level and inside object-typed options, and it doesn't permitadditionalProperties: { type: integer }, so every storage name would have to be enumerated in the schema (a code change per new storage).snuba.stateis the same lookup mechanism the sibling load-balancing keys already use and supports dynamic keys natively. The tradeoff is no JSON-Schema typing — values are parsed asu64inget_max_insert_block_sizeand non-numeric / non-positive values are silently ignored.Notes for review:
runtime_configcache, so changes propagate within 10s of asnuba.state.set_configcall.0deliberately means "leave unset" rather than "set to 0" (ClickHouse rejects 0 for this setting anyway).