Add a data storage optimizer - #413

Merged

CGodiksen merged 49 commits into

mainfrom

dev/data-optimizer

Aug 10, 2026

Commits on Aug 5, 2026

Add new data optimizer config to proto file
CGodiksen
committed
91ff199
Add data optimizer config to clap arguments
CGodiksen
committed
a77ab4d
Add data optimizer config to Configuration struct
CGodiksen
committed
5847eec
Add data optimizer config to user docs
CGodiksen
committed
9550bef
Fix ordering issue and run Rustfmt
CGodiksen
committed
ced2819
Add a data folder method to return file sizes for a table
CGodiksen
committed
32f1167
Use data folder method to compute table sizes
CGodiksen
committed
8000036
Add struct for DataStorageOptimizer
CGodiksen
committed
410594d
Add try_new for DataStorageOptimizer
CGodiksen
committed
2c712c2
Add method to increase the estimated compactable size
CGodiksen
committed
0ef6439
Add method to optimize and vacuum table and use it
CGodiksen
committed
3cae52c
Add the actual table name to the expect messages
CGodiksen
committed
b01022b
Add test util for data storage optimizer tests
CGodiksen
committed
f56f4c1
Add test for optimizing when size reaches target file size
CGodiksen
committed
d22d2cc
Add test for not opmizing when the estimate is below the target
CGodiksen
committed
3d9f76b
Add test for catching already existing files in try_new
CGodiksen
committed
2a8f686
Run Rustfmt
CGodiksen
committed
df94b12
Add data storage optimizer to compressed data manager
CGodiksen
committed
8375abb
Call data storage optimizer when saving compressed data and fixed test
CGodiksen
committed
6a6073b
Fix clippy issue
CGodiksen
committed
e0060c3
Add methods to DataStorarageOptimizer to set configuration
CGodiksen
committed
e848105
Add methods to StorageEngine to set configuration
CGodiksen
committed
5d89ae5
Add method to update optimize_target_file_size_in_bytes
CGodiksen
committed
94a8a4e
Add method to update vacuum_retention_period_in_seconds
CGodiksen
committed
2153c44
Add test for setting optimize_target_file_size_in_bytes
CGodiksen
committed
55a63cd
Add test for rejecting 0
CGodiksen
committed
fd822e7
Add test for setting vacuum_retention_period_in_seconds
CGodiksen
committed
f54e21a
Add test for setting too large vacuum_retention_period_in_seconds
CGodiksen
committed
4870cae
Add match arms to update data storage optimizer config
CGodiksen
committed
3ebdc79
Use local defaults if a target size or retention period is not specified
CGodiksen
committed
aba2653
Add update config tests to integration tests
CGodiksen
committed
c00e623
Make it clearer that the config is used as defaults
CGodiksen
committed
bab1295
Add test to WAL to ensure that optimize and vacuum does not mess with persisted batch ids
CGodiksen
committed
b7ba75f
Add new configs to test_can_get_configuration
CGodiksen
committed
f88688f
Add test to ensure big files are not included on startup
CGodiksen
committed
bdf6c60

Commits on Aug 6, 2026

Use the physical file count in data storage optimizer tests
CGodiksen
committed
087cec9

Commits on Aug 8, 2026

Update based on comments from @skejserjensen
CGodiksen
committed
e2dfe54

Commits on Aug 10, 2026

Rename DataStorageOptimizer to DataStorageCompactor
CGodiksen
committed
600a225
Rename to compact in storage engine module file
CGodiksen
committed
6e7848a
Rename to compact in compressed data manager
CGodiksen
committed
1872ff3
Rename compaction to merge when talking about optimize in DataFolder
CGodiksen
committed
97ad914
Rename compaction to merge when talking about optimize in context
CGodiksen
committed
2524978
Use merge instead of compaction in WAL and integration test
CGodiksen
committed
89f1631
Use merge instead of compaction when talking about optimize in embedded
CGodiksen
committed
cdf13d8
Use merge instead of compaction in bindings
CGodiksen
committed
70e5530
Use compaction instead of optimization in user docs
CGodiksen
committed
c84d3b1
Merge branch 'main' into dev/data-optimizer
CGodiksen
committed
29a7aa5
Fix table formatting after merge
CGodiksen
committed
1412623
Use compaction when describing what the new config values does
CGodiksen
committed
f554b47