Fix calculation of compressed data buffer size

[CGodiksen]

Due to an addition placed inside a loop, we calculated the size of the compressed data buffers wrong in CompressedDataBuffer::append_compressed_segment_batch().

Before:

let mut compressed_segments_size = 0;
for compressed_segment_batch in compressed_segments.drain(0..) {
    compressed_segments_size += Self::size_of_compressed_segments(&compressed_segment_batch);
    self.compressed_segments.push(compressed_segment_batch);
    self.size_in_bytes += compressed_segments_size;
}

After:

let mut compressed_segments_size = 0;
for compressed_segment_batch in compressed_segments.drain(0..) {
    compressed_segments_size += Self::size_of_compressed_segments(&compressed_segment_batch);
    self.compressed_segments.push(compressed_segment_batch);
}

self.size_in_bytes += compressed_segments_size;

Before, we counted each previous compressed segment on every iteration of the loop, instead of only counting it once. Since we returned the correct value (compressed_segments_size) to the caller (which used it to reserve memory) and then used the wrong value (self.size_in_bytes) when updating the remaining compressed memory when saving compressed data, we added more memory back each time. This meant that the remaining memory would get larger than the reserved memory.

This PR fixes the bug, updates some naming for consistency, and updates the related tests to assert on exact values to catch this issue.

[Copilot]

Pull request overview

Fixes incorrect accounting of in-memory compressed buffer sizes by moving the size_in_bytes update out of the append loop, preventing the compressed-memory “remaining bytes” counter from growing beyond what was actually reserved.

Changes:

• Correct CompressedDataBuffer::append_compressed_segment_batch() to add the total appended size to self.size_in_bytes exactly once.
• Tighten related tests to assert exact buffer-size and memory-pool deltas (instead of only checking > 0 / monotonic growth).
• Minor naming consistency updates around “segment(s)” vs batch elements.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
crates/modelardb_server/src/storage/compressed_data_buffer.rs	Fixes buffer size accounting and strengthens unit tests to validate exact size returned vs stored.
crates/modelardb_server/src/storage/compressed_data_manager.rs	Updates tests to assert precise in-memory buffer sizing and compressed-memory restoration behavior.

Suppressed comments (3)

crates/modelardb_server/src/storage/compressed_data_buffer.rs:173

• This comment says the batch contains "two compressed segments", but the batch actually contains two RecordBatches of compressed segments (one per column). Clarifying the terminology will make the test intent match the CompressedSegmentBatch data model.

        // The batch contains two compressed segments, so the size of both is added to the buffer.
        // The returned size must match since the caller reserves memory based on it.

crates/modelardb_server/src/storage/compressed_data_manager.rs:414

• This comment says the insert adds "two compressed segments", but the batch in this test contains two RecordBatches of compressed segments (one per column). Rewording helps prevent confusion when reasoning about memory sizes.

        // Each insert adds the size of the two compressed segments in the batch.
        assert_eq!(previous_size, 2 * COMPRESSED_SEGMENTS_SIZE);

crates/modelardb_server/src/storage/compressed_data_manager.rs:507

• This comment refers to "two compressed segments", but the buffer created by compressed_segments_record_batch() contains two RecordBatches of compressed segments (one per column). Clarify the wording so it matches the data structure being sized and saved.

        // The remaining memory was set to -1 above. Saving the buffer returns exactly the memory
        // reserved for its two compressed segments.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.