feat(config): add fetch_max_bytes and segment_max_records backup options - #140
Merged
Conversation
- backup.fetch_max_bytes decouples the Kafka Fetch request size from the segment size; previously fetches always requested min(segment_max_bytes, 16MB). Clamped to the protocol's i32 range. - backup.segment_max_records rotates segments once they hold N records, in addition to the existing byte-size and interval thresholds. - Warn on unknown config keys at load time. The config schema ignores unrecognized fields, so until now a typo (or an option that does not exist, e.g. fetch_max_bytes before this change) was silently dropped and users believed the option was applied. Every CLI command that loads a config now logs a warning naming the ignored key path. Requested in osodevops/strimzi-backup-operator#53. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ction - Strip serde_ignored's `?` layer segments so paths read `backup.fetch_max_bytez` instead of `backup.?.fetch_max_bytez`. - Don't warn on the documented `logging` section: it is applied by the CLI layer (RUST_LOG / -v), not deserialized into core Config. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Segment writer: record count never triggers rotation when segment_max_records is unset (pre-v0.16.0 behavior preserved). - Config: a pre-v0.16.0 YAML parses with identical semantics, None for the new options, and zero unknown-key warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First half of osodevops/strimzi-backup-operator#53 (Full Backup Config Options): the two options the reporter wants to tune do not exist in kafka-backup yet — their configs set them, and the config parser silently ignored them. This PR adds both options and makes silent-ignore impossible going forward. (The second half — a Strimzi-style
spec.backup.configpassthrough — lands in the operator.)Added
backup.fetch_max_bytes— maximum bytes requested per Kafka Fetch. Previously fetches always requestedmin(segment_max_bytes, 16MB)with no independent knob. An explicit value wins over that default and is clamped to the protocol's i32 range.backup.segment_max_records— rotate segments once they hold N records, alongside the existingsegment_max_bytes/segment_max_interval_msthresholds.fetch_max_bytes: 16777216andsegment_max_records: 2000000) was silently dropped while the user believed it was applied. All six CLI config-load paths now parse viaConfig::from_yaml_with_warnings(serde_ignored) and log each ignored key path:Version
0.16.0:BackupOptionsandSegmentWriterConfiggained public fields, which is breaking for struct-literal construction ofkafka-backup-coretypes — minor bump per the repo's semver policy. Also adds the missing 0.15.13 changelog entry.Tests (TDD — written first, failed on main)
backup_options_parse_fetch_max_bytes_and_segment_max_recordsconfig_from_yaml_with_warnings_reports_unknown_keys/..._is_quiet_for_known_keysrotates_at_max_segment_records(segment writer)fetch_max_bytes_defaults_to_capped_segment_size,explicit_fetch_max_bytes_wins_over_segment_size,fetch_max_bytes_is_clamped_to_protocol_range(engine)cargo fmt/clippy --all-targets -D warnings/ full suite (248 core + CLI) pass.🤖 Generated with Claude Code