Add a data storage optimizer

[CGodiksen]

This PR closes #189 by adding support for automatic optimizing and vacuuming through a new DataStorageOptimizer component. The new component compacts the small Apache Parquet files that accumulate for each table into fewer large files as compressed data is written.

Two new configuration values, optimize_target_file_size_in_bytes and vacuum_retention_period_in_seconds, have been added to control the configuration of the DataStorageOptimizer. These values are also used as the default values when an OPTIMIZE or VACUUM query does not have an explicit target size or retention period.

Unit testing has been added in the relevant places, a write-ahead log test proves optimize and vacuum never discard the Delta commit history that we use for crash recovery, and two new integration tests have been added.

[Copilot]

Pull request overview

This PR adds an automatic storage maintenance path to the server by introducing a DataStorageOptimizer that periodically compacts small Parquet files for Delta tables and then vacuums obsolete files, with new configuration plumbing so defaults can be controlled and updated at runtime.

Changes:

• Introduces DataStorageOptimizer and wires it into the storage engine’s compressed-data write path to trigger optimize+vacuum based on an estimated compactable-size threshold.
• Adds two new configuration values (optimize_target_file_size_in_bytes, vacuum_retention_period_in_seconds) across CLI/env, persisted config, Flight protocol, update handling, and integration tests.
• Adds a DataFolder::table_file_sizes() helper and extends tests (including WAL recovery) to cover optimize/vacuum interactions.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
docs/user/README.md	Documents new CLI/env configuration knobs for optimize/vacuum.
crates/modelardb_types/src/flight/protocol.proto	Extends Flight protocol configuration + update-setting enum for new knobs.
crates/modelardb_storage/src/write_ahead_log.rs	Adds WAL recovery test asserting optimize/vacuum don’t break crash recovery history.
crates/modelardb_storage/src/data_folder/mod.rs	Adds table_file_sizes() helper and supports optimizer/transfer size accounting.
crates/modelardb_server/tests/integration_test.rs	Adds integration coverage for reading/updating new configuration values.
crates/modelardb_server/src/storage/mod.rs	Wires DataStorageOptimizer into StorageEngine and exposes setters for runtime updates.
crates/modelardb_server/src/storage/data_transfer.rs	Reuses table_file_sizes() to compute table sizes (refactor/simplification).
crates/modelardb_server/src/storage/data_storage_optimizer.rs	New component implementing auto optimize+vacuum behavior and unit tests.
crates/modelardb_server/src/storage/compressed_data_manager.rs	Invokes optimizer after writing compressed data; threads optimizer into manager.
crates/modelardb_server/src/remote/mod.rs	Uses local config defaults for OPTIMIZE/VACUUM when query omits target/retention; supports updating new settings.
crates/modelardb_server/src/main.rs	Adds CLI/env args for optimize/vacuum configuration.
crates/modelardb_server/src/configuration.rs	Persists/validates new configuration values; adds update methods + tests.
crates/modelardb_auth/src/lib.rs	Minor formatting change in token parsing error mapping.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.