Skip to content

Add support for using Delta Lake vacuum() and Delta Lake optimize() through Arrow Flight #239

Description

@CGodiksen

Delta Lake has functionality for removing stale files and optimizing files, however since we provide a layer over Delta Lake, this functionality is not available to the user of ModelarDB. Actions should be added to Arrow Flight that makes it possible to use this functionality.

When vacuuming or optimizing a table, the metadata Delta Lake should also be vacuumed or optimized since this is cheap and we do not want the user to have to manually vacuum or optimize the metadata Delta Lake.

  • Add new endpoint to server Apache Arrow Flight server to vacuum a table.
  • Add new endpoint to server Apache Arrow Flight server to optimize a table.
  • Consider how to handle passing extra arguments such as retention period to endpoint.
  • Add new endpoint to manager Apache Arrow Flight server to vacuum a table in the entire cluster.
  • Add new endpoint to manager Apache Arrow Flight server to optimize a table in the entire cluster.
  • Consider how to handle passing extra arguments to optimize endpoint.
  • Consider if we should have endpoints to vacuum or optimize all tables.
  • Add integration tests.
  • Add unit tests.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions