Add Iceberg capabilities: hard delete, table location layout, custom partition names #1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…partition names
New capabilities matching dltHub's Iceberg destination:
1. Hard delete support
- Rows with _dlt_deleted_at column set are permanently deleted during merge
- Works with both delete-insert and upsert strategies
- Delete-insert executes hard deletes atomically in same transaction
- Configurable via hard_delete_column config option
2. table_location_layout
- Control directory structure for table files
- Supports patterns: {namespace}, {dataset_name}, {table_name}
- Relative paths are prepended with warehouse location
3. register_new_tables
- Auto-register tables found in storage but missing from catalog
- Useful for backward compatibility and recovery scenarios
- Currently supports local filesystem only
4. Custom partition field names
- iceberg_partition methods now accept optional name parameter
- Example: iceberg_partition.year("ts", "year_field")
- API changed: bucket(num_buckets, column) and truncate(width, column)
to match dltHub's argument order
- Accept single string or list of strings as partition parameter
- Strings are converted to identity PartitionTransforms
- Allows mixing strings with explicit PartitionTransform objects
- Example: partition=["region", iceberg_partition.month("created_at")]
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Adds several capabilities to match dltHub's Iceberg destination:
Hard delete support: Rows with
_dlt_deleted_atcolumn set are permanently deleted during merge operations. Works with both delete-insert (atomic, same transaction) and upsert strategies.table_location_layout: Control directory structure for table files with patterns like
{namespace}/{table_name}.register_new_tables: Auto-register tables found in storage but missing from catalog (local filesystem only for now).
Custom partition field names:
iceberg_partition.year("ts", "year_field")to set custom names instead of auto-generated ones.Breaking Change
The
bucketandtruncatepartition methods now take parameter first to match dltHub's API:iceberg_partition.bucket("column", 10)iceberg_partition.bucket(10, "column")Tests
tests/test_capabilities.pywith tests for hard delete, table location layout, and custom partition names