Skip to content

Add support for vacuum to delete files no longer referenced by a table - #344

Merged
CGodiksen merged 31 commits into
mainfrom
dev/support-vacuum
Aug 21, 2025
Merged

Add support for vacuum to delete files no longer referenced by a table#344
CGodiksen merged 31 commits into
mainfrom
dev/support-vacuum

Conversation

@CGodiksen

Copy link
Copy Markdown
Collaborator

This PR adds support for simple VACUUM SQL statements to delete files that are no longer referenced by a table. The PR adds support for VACUUM which vacuums all tables as well as VACUUM table_name and VACUUM table_name_1, table_name_2.

Note that it is only possible to set the retention period using an environment variable for now. #343 has been created to make it possible to specify the retention period in the statement in the future.

@CGodiksen
CGodiksen requested a review from Copilot August 19, 2025 18:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for SQL VACUUM statements to delete files that are no longer referenced by a table. VACUUM can be used to vacuum all tables or specific tables by name, with retention period controlled via environment variable.

  • Adds VACUUM SQL statement parsing and execution support
  • Implements vacuum functionality across all components (server, manager, embedded)
  • Adds configuration for retention period with default of 7 days

Reviewed Changes

Copilot reviewed 15 out of 16 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
protocol.proto Adds retention_period_in_seconds field to Configuration message
parser.rs Adds VACUUM statement parsing using ShowVariable as substitute
delta_lake.rs Implements vacuum_table method using Delta Lake vacuum operations
Cargo.toml Adds chrono dependency for time delta calculations
integration_test.rs Adds comprehensive vacuum integration tests
remote.rs Adds vacuum SQL statement support to Flight service
context.rs Implements vacuum_table method with retention period configuration
configuration.rs Adds retention period configuration management
operations/mod.rs Adds vacuum method to Operations trait
operations/data_folder.rs Implements vacuum for local data folder operations
operations/client.rs Implements vacuum for remote client operations
capi.rs Adds C API binding for vacuum operations
operations.py Adds Python binding for vacuum functionality

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment thread crates/modelardb_storage/src/parser.rs Outdated
Comment thread crates/modelardb_embedded/src/operations/data_folder.rs
Comment thread crates/modelardb_embedded/src/operations/data_folder.rs
Comment thread crates/modelardb_embedded/src/operations/data_folder.rs
Comment thread crates/modelardb_server/src/configuration.rs
Comment thread crates/modelardb_server/src/configuration.rs Outdated
Comment thread crates/modelardb_server/tests/integration_test.rs
Comment thread crates/modelardb_storage/src/parser.rs Outdated
Comment thread crates/modelardb_storage/src/parser.rs
Comment thread crates/modelardb_storage/src/delta_lake.rs
Comment thread crates/modelardb_storage/src/parser.rs Outdated
Comment thread crates/modelardb_storage/src/parser.rs Outdated
Comment thread crates/modelardb_storage/src/parser.rs Outdated
Comment thread crates/modelardb_storage/src/parser.rs
@CGodiksen
CGodiksen merged commit a9c7689 into main Aug 21, 2025
4 checks passed
@CGodiksen
CGodiksen deleted the dev/support-vacuum branch August 21, 2025 09:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants