From 5996d3c0649470326b30644059680ed6b13bb48e Mon Sep 17 00:00:00 2001 From: Mikhail Koviazin Date: Fri, 21 Aug 2026 09:18:11 +0200 Subject: [PATCH 1/2] Cherry-pick of https://github.com/Altinity/ClickHouse/pull/2183 with unresolved conflict markers (resolution in next commit) --- Original cherry-pick message follows: Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2 Iceberg deletion vector support (attempt #2) # Conflicts: # docs/reference/engines/table-engines/integrations/iceberg.mdx # docs/reference/functions/table-functions/iceberg.mdx # docs/reference/functions/table-functions/icebergCluster.mdx # docs/reference/statements/system.mdx # src/Access/Common/AccessType.h # src/AggregateFunctions/AggregateFunctionGroupBitmapData.h # src/Common/ProfileEvents.cpp # src/Core/Settings.cpp # src/Formats/FormatFilterInfo.cpp # src/Interpreters/InterpreterSystemQuery.cpp # src/Parsers/ASTSystemQuery.h # src/Parsers/ParserSystemQuery.cpp # src/Processors/Formats/Impl/Parquet/Reader.cpp # src/Storages/ObjectStorage/DataLakes/Common/AvroForIcebergDeserializer.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataObjectInfo.h # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDeletionVector.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.h # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.h # src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/Mutations.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/Snapshot.h # src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_count_shortcuts.cpp # src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.cpp # src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.h # src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_dv_referenced_data_file.cpp # src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_metrics.cpp # src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_footer_cache.cpp # src/Storages/ObjectStorage/IObjectIterator.h # src/Storages/ObjectStorage/StorageObjectStorageSource.cpp # src/Storages/ObjectStorage/tests/gtest_rendezvous_hashing.cpp # tests/integration/test_storage_iceberg_with_spark/test_deletion_vectors.py # tests/queries/0_stateless/01271_show_privileges.reference # tests/queries/0_stateless/04117_parser_system_query_variants.reference # tests/queries/0_stateless/04117_parser_system_query_variants.sql --- .../table-engines/integrations/iceberg.mdx | 14 +++++ .../functions/table-functions/iceberg.mdx | 14 +++++ .../table-functions/icebergCluster.mdx | 4 ++ docs/reference/statements/system.mdx | 3 + src/Access/Common/AccessType.h | 3 + .../AggregateFunctionGroupBitmapData.h | 5 ++ src/Common/ProfileEvents.cpp | 3 + src/Core/Settings.cpp | 3 + src/Core/SettingsChangesHistory.cpp | 1 + src/Formats/FormatFilterInfo.cpp | 3 + src/Interpreters/InterpreterSystemQuery.cpp | 7 +++ src/Parsers/ASTSystemQuery.cpp | 1 + src/Parsers/ASTSystemQuery.h | 3 + src/Parsers/ParserSystemQuery.cpp | 3 + .../Formats/Impl/Parquet/Reader.cpp | 9 +++ .../Common/AvroForIcebergDeserializer.cpp | 17 ++++++ .../DataLakes/Iceberg/IcebergDataObjectInfo.h | 9 +++ .../Iceberg/IcebergDeletionVector.cpp | 40 +++++++++++++ .../DataLakes/Iceberg/IcebergIterator.cpp | 60 +++++++++++++++++++ .../DataLakes/Iceberg/IcebergIterator.h | 3 + .../DataLakes/Iceberg/IcebergMetadata.cpp | 43 +++++++++++++ .../DataLakes/Iceberg/ManifestFile.cpp | 4 ++ .../DataLakes/Iceberg/ManifestFile.h | 16 +++++ .../Iceberg/ManifestFileIterator.cpp | 3 + .../DataLakes/Iceberg/Mutations.cpp | 5 ++ .../DataLakes/Iceberg/Snapshot.h | 6 ++ .../tests/gtest_iceberg_count_shortcuts.cpp | 3 + .../DataLakes/PuffinDeletionVectorReader.cpp | 23 +++++++ .../DataLakes/PuffinDeletionVectorReader.h | 10 ++++ .../gtest_puffin_dv_referenced_data_file.cpp | 6 ++ .../gtest_puffin_files_cache_metrics.cpp | 45 ++++++++++++++ .../tests/gtest_puffin_footer_cache.cpp | 8 +++ src/Storages/ObjectStorage/IObjectIterator.h | 3 + .../StorageObjectStorageSource.cpp | 15 +++++ .../tests/gtest_rendezvous_hashing.cpp | 4 ++ .../test_deletion_vectors.py | 9 +++ .../01271_show_privileges.reference | 3 + ...117_parser_system_query_variants.reference | 3 + .../04117_parser_system_query_variants.sql | 3 + 39 files changed, 417 insertions(+) diff --git a/docs/reference/engines/table-engines/integrations/iceberg.mdx b/docs/reference/engines/table-engines/integrations/iceberg.mdx index 7a05dab165b5..a523625855d9 100644 --- a/docs/reference/engines/table-engines/integrations/iceberg.mdx +++ b/docs/reference/engines/table-engines/integrations/iceberg.mdx @@ -149,19 +149,33 @@ ClickHouse supports reading Iceberg tables that use the following deletion metho - [Position deletes](https://iceberg.apache.org/spec/#position-delete-files) - [Equality deletes](https://iceberg.apache.org/spec/#equality-delete-files) (supported from version 25.8+) +<<<<<<< HEAD:docs/reference/engines/table-engines/integrations/iceberg.mdx - [Deletion vectors](https://iceberg.apache.org/spec/#deletion-vectors) (Iceberg v3, read-only), stored either in Puffin files or in Delta-style `deletion_vector_*.bin` files using the same `deletion-vector-v1` envelope at the manifest `content_offset` / `content_size_in_bytes` The following limitations apply to deletion vectors: - Only `deletion-vector-v1` blobs are supported (Puffin container or Delta `.bin` slice) +======= +- [Deletion vectors](https://iceberg.apache.org/spec/#deletion-vectors) stored in Puffin files (Iceberg v3, read-only) + +The following limitations apply to deletion vectors: + +- Only `deletion-vector-v1` Puffin blobs are supported +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2):docs/en/engines/table-engines/integrations/iceberg.md - Data files must be in Parquet format - Column-scoped deletion vectors (user column ids in puffin `fields`) are not supported. Writers may set `fields` to `[]` or to the Iceberg reserved `_pos` id (`2147483645`) for file-scoped deletion vectors. - Writing deletion vectors is not supported - `DELETE` / `UPDATE` mutations on Iceberg format version 3+ tables are rejected (writers must not add position-delete files) +<<<<<<< HEAD:docs/reference/engines/table-engines/integrations/iceberg.mdx Parsed deletion vectors can be cached in memory when `use_puffin_files_cache` is enabled and the deletion-vector object has a non-empty `etag`. Empty deletion vectors are cached as well, so repeated reads do not re-fetch the object. Parsed footers for coalesced multi-DV Puffin files are memoized with that cache (same identity: storage, path, `etag`) so slices share one footer parse; Delta `.bin` files have no Puffin footer and skip that memo. The memo shares `puffin_files_cache_size` / max-entry limits and is dropped when the cache is disabled (`puffin_files_cache_size=0`) or cleared. The cache can be cleared with `SYSTEM DROP PUFFIN FILES CACHE`. For [`icebergCluster`](/sql-reference/table-functions/icebergCluster.md) (and `object_storage_cluster`), the initiator loads and materializes each data file's deletion vector while distributing tasks, then sends the resulting row bitmap to workers with the task. Workers apply the bitmap; they do not re-read the Puffin or `.bin` object for that path. On wide v3 tables this can make the initiator a serialization point for deletion-vector I/O and decode. +======= +Parsed deletion vectors can be cached in memory when `use_puffin_files_cache` is enabled and the puffin file has a non-empty `etag`. Empty deletion vectors are cached as well, so repeated reads do not re-fetch the puffin file. Parsed footers for coalesced multi-DV Puffin files are memoized with that cache (same identity: storage, path, `etag`) so slices share one footer parse; the memo shares `puffin_files_cache_size` / max-entry limits and is dropped when the cache is disabled (`puffin_files_cache_size=0`) or cleared. The cache can be cleared with `SYSTEM DROP PUFFIN FILES CACHE`. + +For [`icebergCluster`](/sql-reference/table-functions/icebergCluster.md) (and `object_storage_cluster`), the initiator loads and materializes each data file's deletion vector while distributing tasks, then sends the resulting row bitmap to workers with the task. Workers apply the bitmap; they do not re-read the Puffin blob for that path. On wide v3 tables this can make the initiator a serialization point for deletion-vector I/O and decode. +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2):docs/en/engines/table-engines/integrations/iceberg.md ### Basic usage {#basic-usage} ```sql diff --git a/docs/reference/functions/table-functions/iceberg.mdx b/docs/reference/functions/table-functions/iceberg.mdx index 25c5c9cde65d..d14304abb36a 100644 --- a/docs/reference/functions/table-functions/iceberg.mdx +++ b/docs/reference/functions/table-functions/iceberg.mdx @@ -126,19 +126,33 @@ ClickHouse supports reading Iceberg tables that use the following deletion metho - [Position deletes](https://iceberg.apache.org/spec/#position-delete-files) - [Equality deletes](https://iceberg.apache.org/spec/#equality-delete-files) (supported from version 25.8+) +<<<<<<< HEAD:docs/reference/functions/table-functions/iceberg.mdx - [Deletion vectors](https://iceberg.apache.org/spec/#deletion-vectors) (Iceberg v3, read-only), stored either in Puffin files or in Delta-style `deletion_vector_*.bin` files using the same `deletion-vector-v1` envelope at the manifest `content_offset` / `content_size_in_bytes` The following limitations apply to deletion vectors: - Only `deletion-vector-v1` blobs are supported (Puffin container or Delta `.bin` slice) +======= +- [Deletion vectors](https://iceberg.apache.org/spec/#deletion-vectors) stored in Puffin files (Iceberg v3, read-only) + +The following limitations apply to deletion vectors: + +- Only `deletion-vector-v1` Puffin blobs are supported +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2):docs/en/sql-reference/table-functions/iceberg.md - Data files must be in Parquet format - Column-scoped deletion vectors (user column ids in puffin `fields`) are not supported. Writers may set `fields` to `[]` or to the Iceberg reserved `_pos` id (`2147483645`) for file-scoped deletion vectors. - Writing deletion vectors is not supported - `DELETE` / `UPDATE` mutations on Iceberg format version 3+ tables are rejected (writers must not add position-delete files) +<<<<<<< HEAD:docs/reference/functions/table-functions/iceberg.mdx Parsed deletion vectors can be cached in memory when `use_puffin_files_cache` is enabled and the deletion-vector object has a non-empty `etag`. Empty deletion vectors are cached as well, so repeated reads do not re-fetch the object. Parsed footers for coalesced multi-DV Puffin files are memoized with that cache (same identity: storage, path, `etag`) so slices share one footer parse; Delta `.bin` files have no Puffin footer and skip that memo. The memo shares `puffin_files_cache_size` / max-entry limits and is dropped when the cache is disabled (`puffin_files_cache_size=0`) or cleared. The cache can be cleared with `SYSTEM DROP PUFFIN FILES CACHE`. For [`icebergCluster`](/sql-reference/table-functions/icebergCluster.md) (and `object_storage_cluster`), the initiator loads and materializes each data file's deletion vector while distributing tasks, then sends the resulting row bitmap to workers with the task. Workers apply the bitmap; they do not re-read the Puffin or `.bin` object for that path. On wide v3 tables this can make the initiator a serialization point for deletion-vector I/O and decode. +======= +Parsed deletion vectors can be cached in memory when `use_puffin_files_cache` is enabled and the puffin file has a non-empty `etag`. Empty deletion vectors are cached as well, so repeated reads do not re-fetch the puffin file. Parsed footers for coalesced multi-DV Puffin files are memoized with that cache (same identity: storage, path, `etag`) so slices share one footer parse; the memo shares `puffin_files_cache_size` / max-entry limits and is dropped when the cache is disabled (`puffin_files_cache_size=0`) or cleared. The cache can be cleared with `SYSTEM DROP PUFFIN FILES CACHE`. + +For [`icebergCluster`](/sql-reference/table-functions/icebergCluster.md) (and `object_storage_cluster`), the initiator loads and materializes each data file's deletion vector while distributing tasks, then sends the resulting row bitmap to workers with the task. Workers apply the bitmap; they do not re-read the Puffin blob for that path. On wide v3 tables this can make the initiator a serialization point for deletion-vector I/O and decode. +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2):docs/en/sql-reference/table-functions/iceberg.md ### Basic usage {#basic-usage} diff --git a/docs/reference/functions/table-functions/icebergCluster.mdx b/docs/reference/functions/table-functions/icebergCluster.mdx index 4f50d2cbe508..90f8ec77de5e 100644 --- a/docs/reference/functions/table-functions/icebergCluster.mdx +++ b/docs/reference/functions/table-functions/icebergCluster.mdx @@ -14,7 +14,11 @@ Allows processing files from Apache [Iceberg](https://iceberg.apache.org/) in pa ## Deletion vectors on cluster reads {#deletion-vectors-cluster} +<<<<<<< HEAD:docs/reference/functions/table-functions/icebergCluster.mdx Iceberg v3 [deletion vectors](https://iceberg.apache.org/spec/#deletion-vectors) are loaded on the **initiator** while it distributes tasks: for each data file the initiator reads the Puffin or Delta `.bin` object, validates the `deletion-vector-v1` envelope, materializes deleted row positions, and attaches the bitmap to the task sent to workers. Workers apply that bitmap when reading Parquet; they do not fetch or parse the deletion-vector file again for that path. See [Processing of tables with deleted rows](/sql-reference/table-functions/iceberg.md#deleted-rows) for format limits and caching. +======= +Iceberg v3 [deletion vectors](https://iceberg.apache.org/spec/#deletion-vectors) are loaded on the **initiator** while it distributes tasks: for each data file the initiator reads the Puffin blob, validates it, materializes deleted row positions, and attaches the bitmap to the task sent to workers. Workers apply that bitmap when reading Parquet; they do not fetch or parse the Puffin file again for that path. See [Processing of tables with deleted rows](/sql-reference/table-functions/iceberg.md#deleted-rows) for format limits and caching. +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2):docs/en/sql-reference/table-functions/icebergCluster.md On wide tables with many deletion vectors, initiator-side decode and per-task bitmap serialization can become a bottleneck even when Parquet reads are well parallelized across the cluster. diff --git a/docs/reference/statements/system.mdx b/docs/reference/statements/system.mdx index 9e083a92e529..8612fb002ddf 100644 --- a/docs/reference/statements/system.mdx +++ b/docs/reference/statements/system.mdx @@ -134,6 +134,7 @@ Clears the per-URL Confluent Schema Registry caches used by the `AvroConfluent` Clears the parquet metadata cache. +<<<<<<< HEAD:docs/reference/statements/system.mdx ## SYSTEM CLEAR|DROP PAIMON METADATA CACHE {#drop-paimon-metadata-cache} Clears the in-memory cache of parsed Paimon metadata files (manifest lists and manifests). @@ -141,6 +142,8 @@ Clears the in-memory cache of parsed Paimon metadata files (manifest lists and m Clears the cache of preprocessed constant polygons used by the function [`pointInPolygon`](/reference/functions/regular-functions/geo/coordinates#pointinpolygon). The configured size limit (the `point_in_polygon_cache_size` server setting) is left unchanged, so the cache keeps accepting entries afterwards. To disable the cache instead, set `point_in_polygon_cache_size` to `0`. +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2):docs/en/sql-reference/statements/system.md ## SYSTEM DROP PUFFIN FILES CACHE {#drop-puffin-files-cache} Clears the Puffin files cache used for parsed Iceberg puffin file content such as deletion vectors. diff --git a/src/Access/Common/AccessType.h b/src/Access/Common/AccessType.h index 768721c15813..8556c56f683c 100644 --- a/src/Access/Common/AccessType.h +++ b/src/Access/Common/AccessType.h @@ -332,7 +332,10 @@ enum class AccessType : uint8_t M(SYSTEM_DROP_PAIMON_METADATA_CACHE, "SYSTEM CLEAR PAIMON_METADATA_CACHE, SYSTEM DROP PAIMON_METADATA_CACHE", GLOBAL, SYSTEM_DROP_CACHE) \ M(SYSTEM_DROP_AVRO_SCHEMA_CACHE, "SYSTEM CLEAR AVRO SCHEMA CACHE, SYSTEM DROP AVRO SCHEMA CACHE, DROP AVRO SCHEMA CACHE", GLOBAL, SYSTEM_DROP_CACHE) \ M(SYSTEM_DROP_PARQUET_METADATA_CACHE, "SYSTEM DROP PARQUET_METADATA_CACHE", GLOBAL, SYSTEM_DROP_CACHE) \ +<<<<<<< HEAD M(SYSTEM_DROP_POINT_IN_POLYGON_CACHE, "SYSTEM CLEAR POINT IN POLYGON CACHE, SYSTEM DROP POINT IN POLYGON CACHE, DROP POINT IN POLYGON CACHE", GLOBAL, SYSTEM_DROP_CACHE) \ +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) M(SYSTEM_DROP_PUFFIN_FILES_CACHE, "SYSTEM DROP PUFFIN_FILES_CACHE", GLOBAL, SYSTEM_DROP_CACHE) \ M(SYSTEM_PREWARM_PRIMARY_INDEX_CACHE, "SYSTEM PREWARM PRIMARY INDEX, PREWARM PRIMARY INDEX CACHE, PREWARM PRIMARY INDEX", GLOBAL, SYSTEM_DROP_CACHE) \ M(SYSTEM_DROP_PRIMARY_INDEX_CACHE, "SYSTEM CLEAR PRIMARY INDEX CACHE, SYSTEM DROP PRIMARY INDEX, DROP PRIMARY INDEX CACHE, DROP PRIMARY INDEX", GLOBAL, SYSTEM_DROP_CACHE) \ diff --git a/src/AggregateFunctions/AggregateFunctionGroupBitmapData.h b/src/AggregateFunctions/AggregateFunctionGroupBitmapData.h index c7539ee882a8..274b3566e6c6 100644 --- a/src/AggregateFunctions/AggregateFunctionGroupBitmapData.h +++ b/src/AggregateFunctions/AggregateFunctionGroupBitmapData.h @@ -673,7 +673,12 @@ class RoaringBitmapWithSmallSet : private boost::noncopyable } /** +<<<<<<< HEAD * Return new set of the smallest `limit` values (as UnsignedT) which are no less than `range_start`. +======= + * Return new set of the smallest `limit` values in set which is no less than `range_start`. + * It's used in subset and currently only support UInt32 +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) */ UInt64 rb_limit(UInt64 range_start, UInt64 limit, RoaringBitmapWithSmallSet & r1) const /// NOLINT { diff --git a/src/Common/ProfileEvents.cpp b/src/Common/ProfileEvents.cpp index 7e7d268a88ad..d44c5f880f3e 100644 --- a/src/Common/ProfileEvents.cpp +++ b/src/Common/ProfileEvents.cpp @@ -124,10 +124,13 @@ M(IcebergMetadataReturnedObjectInfos, "Total number of returned object infos from iceberg iterator.", ValueType::Number) \ M(IcebergMinMaxNonPrunedDeleteFiles, "Total number of accepted data files-position delete file pairs by minmax analysis from pairs suitable by partitioning and sequence number.", ValueType::Number) \ M(IcebergMinMaxPrunedDeleteFiles, "Total number of accepted data files-position delete file pairs by minmax analysis from pairs suitable by partitioning and sequence number.", ValueType::Number) \ +<<<<<<< HEAD M(OneLakeAccessTokenRequests, "Number of access token requests issued by the OneLake catalog via the Entra ID refresh token grant.", ValueType::Number) \ M(OneLakeAccessTokenRequestFailures, "Number of failed access token requests issued by the OneLake catalog via the Entra ID refresh token grant (including expired or revoked refresh tokens).", ValueType::Number) \ M(OneLakeAccessTokenRequestMicroseconds, "Total time spent requesting access tokens via the Entra ID refresh token grant in the OneLake catalog.", ValueType::Microseconds) \ M(OneLakeAccessTokenExpirations, "Number of times the cached OneLake access token was found expired and renewed transparently.", ValueType::Number) \ +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) M(PuffinFilesRead, "Number of Puffin files read (footer or deletion vector blob).", ValueType::Number) \ M(PuffinFileReadMicroseconds, "Total time spent reading Puffin files.", ValueType::Microseconds) \ M(PuffinFilesCacheHits, "Number of times parsed Puffin file content has been found in the cache.", ValueType::Number) \ diff --git a/src/Core/Settings.cpp b/src/Core/Settings.cpp index 881dd7186dd6..01499541342c 100644 --- a/src/Core/Settings.cpp +++ b/src/Core/Settings.cpp @@ -6051,6 +6051,7 @@ Minimum time of delay between 2 background compaction operations. DECLARE(Seconds, iceberg_compaction_data_cleanup, 60 * 60 * 3, R"( The time after which the data will be deleted. )", 0) \ +<<<<<<< HEAD DECLARE(UInt64, iceberg_compaction_commit_batch_size, 100, R"( Number of merged data files that background Iceberg compaction accumulates before publishing them in a new snapshot. @@ -6059,6 +6060,8 @@ how long already merged files stay unpublished while compaction keeps finding ne of publishing only when compaction runs out of candidates - a state that is never reached while the table keeps receiving new data files, so every merged output is then written to object storage and never referenced by a snapshot. )", 0) \ +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) DECLARE(Bool, use_puffin_files_cache, true, R"( If turned on, Iceberg reads may utilize the Puffin files cache for parsed puffin file content such as deletion vectors. diff --git a/src/Core/SettingsChangesHistory.cpp b/src/Core/SettingsChangesHistory.cpp index 441325235ea7..94ab67aa25ab 100644 --- a/src/Core/SettingsChangesHistory.cpp +++ b/src/Core/SettingsChangesHistory.cpp @@ -220,6 +220,7 @@ const VersionToSettingsChangesMap & getSettingsChangesHistory() { {"use_puffin_files_cache", false, true, "Enables cache of parsed Puffin file content such as deletion vectors."}, }); + addSettingsChanges(settings_changes_history, "26.6", { {"analyzer_compatibility_apply_final_to_all_joined_tables", true, false, "Fixed a bug in the analyzer where FINAL on the left-most table of a JOIN was incorrectly applied to the other joined tables as well. previous_value=true so `compatibility` with versions before 26.6 restores the old behavior."}, diff --git a/src/Formats/FormatFilterInfo.cpp b/src/Formats/FormatFilterInfo.cpp index cee9aa77ad60..1f87b0720eb3 100644 --- a/src/Formats/FormatFilterInfo.cpp +++ b/src/Formats/FormatFilterInfo.cpp @@ -86,6 +86,7 @@ bool FormatFilterInfo::hasFilter() const /// Any of these can reduce the number of rows emitted by the reader pipeline. /// Count-from-files cache must not be populated when they are present. return filter_actions_dag != nullptr || row_level_filter != nullptr || prewhere_info != nullptr; +<<<<<<< HEAD } namespace @@ -103,6 +104,8 @@ namespace return true; return false; } +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) } Block FormatFilterInfo::buildKeyConditionInputs( diff --git a/src/Interpreters/InterpreterSystemQuery.cpp b/src/Interpreters/InterpreterSystemQuery.cpp index e070cd3aafcf..bd9aca33b9d0 100644 --- a/src/Interpreters/InterpreterSystemQuery.cpp +++ b/src/Interpreters/InterpreterSystemQuery.cpp @@ -505,10 +505,13 @@ BlockIO InterpreterSystemQuery::execute() #else throw Exception(ErrorCodes::SUPPORT_IS_DISABLED, "The server was compiled without the support for Parquet"); #endif +<<<<<<< HEAD case Type::CLEAR_POINT_IN_POLYGON_CACHE: getContext()->checkAccess(AccessType::SYSTEM_DROP_POINT_IN_POLYGON_CACHE); clearPointInPolygonCache(); break; +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) case Type::CLEAR_PUFFIN_FILES_CACHE: getContext()->checkAccess(AccessType::SYSTEM_DROP_PUFFIN_FILES_CACHE); system_context->clearPuffinFilesCache(); @@ -2826,6 +2829,7 @@ AccessRightsElements InterpreterSystemQuery::getRequiredAccessForDDLOnCluster() required_access.emplace_back(AccessType::SYSTEM_DROP_AVRO_SCHEMA_CACHE); break; case Type::CLEAR_PARQUET_METADATA_CACHE: +<<<<<<< HEAD required_access.emplace_back(AccessType::SYSTEM_DROP_PARQUET_METADATA_CACHE); break; case Type::CLEAR_POINT_IN_POLYGON_CACHE: @@ -2834,6 +2838,9 @@ AccessRightsElements InterpreterSystemQuery::getRequiredAccessForDDLOnCluster() case Type::CLEAR_PUFFIN_FILES_CACHE: required_access.emplace_back(AccessType::SYSTEM_DROP_PUFFIN_FILES_CACHE); break; +======= + case Type::CLEAR_PUFFIN_FILES_CACHE: +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) case Type::CLEAR_PRIMARY_INDEX_CACHE: required_access.emplace_back(AccessType::SYSTEM_DROP_PRIMARY_INDEX_CACHE); break; diff --git a/src/Parsers/ASTSystemQuery.cpp b/src/Parsers/ASTSystemQuery.cpp index bf7cd7f177dc..8c10b1d1ae62 100644 --- a/src/Parsers/ASTSystemQuery.cpp +++ b/src/Parsers/ASTSystemQuery.cpp @@ -635,6 +635,7 @@ void ASTSystemQuery::formatImpl(WriteBuffer & ostr, const FormatSettings & setti case Type::CLEAR_PARQUET_METADATA_CACHE: case Type::CLEAR_POINT_IN_POLYGON_CACHE: case Type::CLEAR_AVRO_SCHEMA_CACHE: + case Type::CLEAR_PUFFIN_FILES_CACHE: case Type::DROP_OBJECT_STORAGE_LIST_OBJECTS_CACHE: case Type::CLEAR_PUFFIN_FILES_CACHE: case Type::RESET_COVERAGE: diff --git a/src/Parsers/ASTSystemQuery.h b/src/Parsers/ASTSystemQuery.h index 1439e9c1743e..2f280e410ae4 100644 --- a/src/Parsers/ASTSystemQuery.h +++ b/src/Parsers/ASTSystemQuery.h @@ -49,7 +49,10 @@ class ASTSystemQuery : public IAST, public ASTQueryWithOnCluster CLEAR_ICEBERG_METADATA_CACHE, CLEAR_PAIMON_METADATA_CACHE, CLEAR_PARQUET_METADATA_CACHE, +<<<<<<< HEAD CLEAR_POINT_IN_POLYGON_CACHE, +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) CLEAR_PUFFIN_FILES_CACHE, CLEAR_FILESYSTEM_CACHE, CLEAR_DISTRIBUTED_CACHE, diff --git a/src/Parsers/ParserSystemQuery.cpp b/src/Parsers/ParserSystemQuery.cpp index 4ac8beaf354b..f0de9c7e39b7 100644 --- a/src/Parsers/ParserSystemQuery.cpp +++ b/src/Parsers/ParserSystemQuery.cpp @@ -322,7 +322,10 @@ bool ParserSystemQuery::parseImpl(IParser::Pos & pos, ASTPtr & node, Expected & {"DROP ICEBERG METADATA CACHE", Type::CLEAR_ICEBERG_METADATA_CACHE}, {"DROP PAIMON METADATA CACHE", Type::CLEAR_PAIMON_METADATA_CACHE}, {"DROP PARQUET METADATA CACHE", Type::CLEAR_PARQUET_METADATA_CACHE}, +<<<<<<< HEAD {"DROP POINT IN POLYGON CACHE", Type::CLEAR_POINT_IN_POLYGON_CACHE}, +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) {"DROP PUFFIN FILES CACHE", Type::CLEAR_PUFFIN_FILES_CACHE}, {"DROP PUFFIN_FILES_CACHE", Type::CLEAR_PUFFIN_FILES_CACHE}, {"DROP FILESYSTEM CACHE", Type::CLEAR_FILESYSTEM_CACHE}, diff --git a/src/Processors/Formats/Impl/Parquet/Reader.cpp b/src/Processors/Formats/Impl/Parquet/Reader.cpp index 4c5e176286da..967989634430 100644 --- a/src/Processors/Formats/Impl/Parquet/Reader.cpp +++ b/src/Processors/Formats/Impl/Parquet/Reader.cpp @@ -13,7 +13,10 @@ #include #include #include +<<<<<<< HEAD #include +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) #include #include #include @@ -361,6 +364,7 @@ void Reader::getHyperrectangleForRowGroup(const parq::RowGroup * meta, Hyperrect } } +<<<<<<< HEAD bool Reader::spatialBboxStatsHaveNoNulls(const parq::RowGroup & meta, size_t spatial_key_condition_idx) const { for (size_t bbox_pc_idx : spatial_key_condition_bbox_col_indices.at(spatial_key_condition_idx)) @@ -374,6 +378,8 @@ bool Reader::spatialBboxStatsHaveNoNulls(const parq::RowGroup & meta, size_t spa return true; } +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) std::vector buildRowGroupGlobalOffsets(const parq::FileMetaData & file_metadata) { if (file_metadata.num_rows < 0) @@ -743,6 +749,7 @@ void Reader::prefilterAndInitRowGroups(const std::optionalcolumns.size() != total_primitive_columns_in_file) throw Exception(ErrorCodes::INCORRECT_DATA, "Row group {} has unexpected number of columns: {} != {}", row_group_idx, meta->columns.size(), total_primitive_columns_in_file); +<<<<<<< HEAD /// Lazy materialization: skip row groups that contain none of the requested rows. std::pair requested_rows_slice {0, 0}; if (rows_to_read) @@ -755,6 +762,8 @@ void Reader::prefilterAndInitRowGroups(const std::optionalbegin()), size_t(end_it - rows_to_read->begin())}; } +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) Hyperrectangle hyperrectangle(extended_sample_block.columns(), Range::createWholeUniverse()); if ((options.format.parquet.filter_push_down && format_filter_info->key_condition) || !spatial_key_conditions.empty()) diff --git a/src/Storages/ObjectStorage/DataLakes/Common/AvroForIcebergDeserializer.cpp b/src/Storages/ObjectStorage/DataLakes/Common/AvroForIcebergDeserializer.cpp index a9534abdd383..a08a1bd75b42 100644 --- a/src/Storages/ObjectStorage/DataLakes/Common/AvroForIcebergDeserializer.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Common/AvroForIcebergDeserializer.cpp @@ -17,6 +17,7 @@ #include #include #include +#include #include #include @@ -332,7 +333,10 @@ ParsedManifestFileEntryPtr AvroForIcebergDeserializer::createParsedManifestFileE case FileContentType::POSITION_DELETE: { /// reference_file_path can be absent in schema for some reason, though it is present in specification: https://iceberg.apache.org/spec/#manifests const bool is_puffin = Poco::toLower(file_format) == "puffin"; +<<<<<<< HEAD const bool has_dv_offsets = content_offset.has_value() && content_size_in_bytes.has_value(); +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) std::optional lower_reference_data_file_path; std::optional upper_reference_data_file_path; bool bounds_set_by_referenced_data_file = false; @@ -348,9 +352,15 @@ ParsedManifestFileEntryPtr AvroForIcebergDeserializer::createParsedManifestFileE bounds_set_by_referenced_data_file = true; } } +<<<<<<< HEAD /// Parquet position deletes may fall back to file-path column bounds. Deletion /// vectors must use the dedicated referenced_data_file field only. if (!bounds_set_by_referenced_data_file && !is_puffin && !has_dv_offsets) +======= + /// Parquet position deletes may fall back to file-path column bounds. Puffin deletion + /// vectors must use the dedicated referenced_data_file field only. + if (!bounds_set_by_referenced_data_file && !is_puffin) +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) { if (auto it = value_for_bounds.find(IcebergPositionDeleteTransform::data_file_path_column_field_id); it != value_for_bounds.end()) @@ -365,17 +375,24 @@ ParsedManifestFileEntryPtr AvroForIcebergDeserializer::createParsedManifestFileE if (is_puffin) { +<<<<<<< HEAD if (!has_dv_offsets) +======= + if (!content_offset.has_value() || !content_size_in_bytes.has_value()) +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) { throw Exception( DB::ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, "Puffin deletion vector entry in manifest file '{}' is missing content_offset or content_size_in_bytes", manifest_file_path); } +<<<<<<< HEAD } if (is_puffin || has_dv_offsets) { +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) requireDirectReferencedDataFileForPuffinDeletionVector( bounds_set_by_referenced_data_file, lower_reference_data_file_path, manifest_file_path); } diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataObjectInfo.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataObjectInfo.h index c5062f74ecc5..637438b12f7b 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataObjectInfo.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataObjectInfo.h @@ -100,6 +100,15 @@ struct IcebergDataObjectInfo : public ObjectInfo, std::enable_shared_from_this clone() const override { return std::make_shared(*this); } +<<<<<<< HEAD +======= + ObjectStoragePtr getResolvedStorage() const { return resolved_storage; } + + void setResolvedStorage(ObjectStoragePtr storage) { resolved_storage = std::move(storage); } + + void addEqualityDeleteObject(const Iceberg::ProcessedManifestFileEntryPtr & equality_delete_object, const String & resolved_storage_path); + +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) Iceberg::IcebergObjectSerializableInfo info; }; diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDeletionVector.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDeletionVector.cpp index 7f77e3f40e3c..d47c4f6cb319 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDeletionVector.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDeletionVector.cpp @@ -18,7 +18,10 @@ #include #include +<<<<<<< HEAD #include +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) namespace DB { @@ -132,9 +135,13 @@ DataLakeObjectMetadata::ExcludedRowsPtr loadDeletionVectorUncached( ContextPtr context, LoggerPtr log, bool disable_filesystem_cache, +<<<<<<< HEAD FooterBlobsPtr preloaded_footer, PuffinFilesCache * footer_cache = nullptr, const std::optional & footer_key = {}) +======= + FooterBlobsPtr preloaded_footer) +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) { RelativePathWithMetadata puffin_object{puffin_path}; auto read_settings = context->getReadSettings(); @@ -151,6 +158,7 @@ DataLakeObjectMetadata::ExcludedRowsPtr loadDeletionVectorUncached( if (!file_size) throw Exception(ErrorCodes::BAD_ARGUMENTS, "Cannot determine Puffin file size for '{}'", puffin_path); +<<<<<<< HEAD const auto container = detectIcebergDeletionVectorContainer( *seekable, *file_size, content_offset, content_size_in_bytes, puffin_path); @@ -180,6 +188,18 @@ DataLakeObjectMetadata::ExcludedRowsPtr loadDeletionVectorUncached( expected_data_file.serialize(), expected_cardinality); } +======= + FooterBlobsPtr footer_owner = preloaded_footer; + if (!footer_owner) + footer_owner = std::make_shared>(readPuffinFooterBlobsFromSeekable(*seekable, *file_size)); + + bindDeletionVectorBlob( + *footer_owner, + content_offset, + content_size_in_bytes, + expected_data_file.serialize(), + expected_cardinality); +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) auto deleted_positions = readDeletionVectorFromPuffin( *read_buffer, content_offset, content_size_in_bytes, expected_cardinality); @@ -195,9 +215,14 @@ DataLakeObjectMetadata::ExcludedRowsPtr loadDeletionVectorUncached( LOG_DEBUG( log, +<<<<<<< HEAD "Loaded deletion vector from file '{}' ({}) for data file '{}': {} deleted rows", puffin_path, container == IcebergDeletionVectorContainer::Puffin ? "Puffin" : "Delta .bin", +======= + "Loaded deletion vector from puffin file '{}' for data file '{}': {} deleted rows", + puffin_path, +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) expected_data_file.serialize(), deleted_positions.size()); @@ -390,10 +415,21 @@ DataLakeObjectMetadata::ExcludedRowsPtr loadDeletionVector( } /// Footer is keyed by file identity only, so N DV slices in one coalesced Puffin share one parse. +<<<<<<< HEAD /// Resolve the footer only on a Puffin deletion-vector cache miss (nested memo lookup). /// Delta `.bin` objects have no Puffin footer; detection inside the uncached loader skips memo. return cache->getOrSetDeletionVector(*cache_key, [&]() { +======= + /// Resolve the footer only on a deletion-vector cache miss (nested memo lookup). + return cache->getOrSetDeletionVector(*cache_key, [&]() + { + auto footer = cache->getOrSetFooter(*footer_key, [&]() + { + return readFooterBlobs(object_storage, puffin_path, context, log, /*disable_filesystem_cache=*/ true); + }); + +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) return loadDeletionVectorUncached( object_storage, puffin_path, @@ -405,9 +441,13 @@ DataLakeObjectMetadata::ExcludedRowsPtr loadDeletionVector( context, log, true, +<<<<<<< HEAD nullptr, cache.get(), footer_key); +======= + footer); +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) }); } diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.cpp index 3ccc4248d9c8..3ad4b92a1b90 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.cpp @@ -317,10 +317,39 @@ IcebergIterator::IcebergIterator( , callback(std::move(callback_)) , table_schema_id(table_snapshot_->schema_id) { +<<<<<<< HEAD /// Decoding any manifest reads settings from the context, so a missing one is fatal either way. if (!local_context) throw DB::Exception(DB::ErrorCodes::LOGICAL_ERROR, "Context is required to construct IcebergIterator"); +======= + auto delete_file = deletes_iterator.next(); + while (delete_file.has_value()) + { + if (delete_file.value()->parsed_entry->equality_ids.has_value()) + { + equality_deletes_files.emplace_back(std::move(delete_file.value())); + } + else if (delete_file.value()->parsed_entry->isDeletionVector()) + { + deletion_vector_files.emplace_back(std::move(delete_file.value())); + } + else + { + parquet_position_deletes_files.emplace_back(std::move(delete_file.value())); + } + delete_file = deletes_iterator.next(); + } + LOG_DEBUG( + logger, + "Taken {} deletion vector files, {} parquet position delete files and {} equality delete files in iceberg iterator", + deletion_vector_files.size(), + parquet_position_deletes_files.size(), + equality_deletes_files.size()); + std::sort(equality_deletes_files.begin(), equality_deletes_files.end()); + std::sort(deletion_vector_files.begin(), deletion_vector_files.end()); + std::sort(parquet_position_deletes_files.begin(), parquet_position_deletes_files.end()); +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) producer_task = std::make_unique( [this, thread_group = CurrentThread::getGroup()]() { @@ -490,6 +519,14 @@ ObjectInfoPtr IcebergIterator::next(size_t) const auto & data_file_path = object_info->info.data_object_file_path_key; bool has_deletion_vector = false; +<<<<<<< HEAD +======= + object_info->info.requires_external_storage = (storage_to_use != object_storage); + + const auto & data_file_path = object_info->info.data_object_file_path_key; + bool has_deletion_vector = false; + +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) for (const auto & deletion_vector : defineDeletesSpan(manifest_file_entry, deletion_vector_files, /* is_equality_delete */ false, logger)) { @@ -516,6 +553,7 @@ ObjectInfoPtr IcebergIterator::next(size_t) } const auto & parsed_entry = deletion_vector->parsed_entry; +<<<<<<< HEAD /// For icebergCluster, next() runs on the initiator's task-distribution path: DV /// I/O / CRC / roaring materialization happen here, then excluded_rows is sent on @@ -524,6 +562,20 @@ ObjectInfoPtr IcebergIterator::next(size_t) auto excluded_rows = Iceberg::loadDeletionVector( object_storage, persistent_components.path_resolver.resolve(parsed_entry->file_path_key), +======= + const auto puffin_metadata_path = parsed_entry->file_path_key.serialize(); + auto [puffin_storage, puffin_key] = resolveObjectStorageForPath( + persistent_components.table_location, puffin_metadata_path, + object_storage, *secondary_storages, local_context, + persistent_components.path_resolver); + + /// For icebergCluster, next() runs on the initiator's task-distribution path: DV + /// I/O / CRC / roaring materialization happen here, then excluded_rows is sent on + /// the wire per task. Workers apply the bitmap and do not re-read the Puffin blob. + auto excluded_rows = Iceberg::loadDeletionVector( + puffin_storage, + puffin_key, +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) parsed_entry->content_offset.value(), parsed_entry->content_size_in_bytes.value(), data_file_path, @@ -539,7 +591,11 @@ ObjectInfoPtr IcebergIterator::next(size_t) has_deletion_vector = true; LOG_DEBUG( logger, +<<<<<<< HEAD "Attached deletion vector from file `{}` to data file `{}`", +======= + "Attached deletion vector from puffin file `{}` to data file `{}`", +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) parsed_entry->file_path_key, data_file_path); } @@ -580,7 +636,11 @@ ObjectInfoPtr IcebergIterator::next(size_t) lower.has_value() ? lower->serialize() : "[no lower bound]", upper.has_value() ? upper->serialize() : "[no upper bound]"); object_info->addPositionDeleteObject( +<<<<<<< HEAD position_delete, persistent_components.path_resolver.resolve(position_delete->parsed_entry->file_path_key)); +======= + position_delete, position_delete->parsed_entry->file_path_key.serialize()); +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) } } } diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.h index 6b27c8a37c81..cd49d4bdd9eb 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.h @@ -110,8 +110,11 @@ class IcebergIterator : public IObjectIterator ConcurrentBoundedQueue blocking_queue; std::unique_ptr producer_task; IDataLakeMetadata::FileProgressCallback callback; +<<<<<<< HEAD /// Filled once under `deletes_mutex` and never mutated afterwards, so `next` may read them /// unguarded once it has gone through `ensureDeletesReady`. +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) std::vector deletion_vector_files; std::vector parquet_position_deletes_files; std::vector equality_deletes_files; diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp index 3fa4471b3f17..47e834a66cd5 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp @@ -10,6 +10,10 @@ #include #include #include +<<<<<<< HEAD +======= +#include +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) #include #include #include @@ -1310,6 +1314,7 @@ std::optional IcebergMetadata::totalRows(ContextPtr local_context) const && *actual_data_snapshot->total_equality_delete_rows > 0) return {}; +<<<<<<< HEAD /// Row counts stored in the metadata layers above the manifest files are not used as /// data sources, because writers derive them instead of measuring them against the data: /// - the snapshot summary's `total-records` is maintained incrementally (parent total @@ -1325,10 +1330,22 @@ std::optional IcebergMetadata::totalRows(ContextPtr local_context) const /// required field in every format version, so summing it over the live data files is /// exact, at the cost of opening the manifest files (served from the Iceberg metadata /// cache on repeated queries). +======= + /// Do not trust snapshot-summary `total-records` for the answer. Those totals are optional, + /// writer-maintained incrementally, and a single bad commit can poison every later snapshot. + /// Sum required per-data-file `record_count` from manifests when there are no live delete + /// files; otherwise fail closed to a real scan. Summary is compared only for a mismatch warning. + /// + /// Manifest-list `added_rows_count`/`existing_rows_count` are not used (some writers stamp them + /// from snapshot summary and can report 0 after compaction). Subtracting live position-delete / + /// deletion-vector `record_count` from data-file totals is also unsafe (duplicates, stale + /// references, DV supersession of parquet position deletes). +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) UInt64 result = 0; for (const auto & manifest_list_entry : actual_data_snapshot->manifest_list_entries) { auto manifest_file_ptr = getManifestFileEntriesHandle( +<<<<<<< HEAD object_storage, persistent_components, local_context, log, manifest_list_entry, actual_table_state_snapshot.schema_id); /// Live delete files make an exact metadata-only count impossible: @@ -1346,12 +1363,38 @@ std::optional IcebergMetadata::totalRows(ContextPtr local_context) const /// nullopt means a corrupted manifest file with a negative `record_count`: fail /// closed to a real scan instead of returning a wrong count. +======= + object_storage, persistent_components, local_context, log, manifest_list_entry, actual_table_state_snapshot.schema_id, *secondary_storages); + + if (!manifest_file_ptr.getFilesWithoutDeleted(FileContentType::EQUALITY_DELETE).empty() + || !manifest_file_ptr.getFilesWithoutDeleted(FileContentType::POSITION_DELETE).empty()) + return {}; + /// nullopt means a negative / overflowing per-file `record_count`: fail closed to a + /// real scan instead of returning a wrong count. Do not use optional column + /// `value_counts` here — nested fields can report element counts larger than rows. +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) auto manifest_rows = manifest_file_ptr.getRowsCountInAllFilesExcludingDeleted(FileContentType::DATA); if (!manifest_rows.has_value()) return {}; /// Per-manifest sums are capped at Int64::max; still guard the cross-manifest total. if (common::addOverflow(result, static_cast(*manifest_rows), result)) return {}; +<<<<<<< HEAD +======= + } + + if (auto summary_total_rows = actual_data_snapshot->getTotalRows(); + summary_total_rows.has_value() && *summary_total_rows != result) + { + LOG_WARNING( + log, + "Iceberg snapshot summary of table {} claims {} total rows, but its manifest files describe {} rows. " + "The snapshot summary is inconsistent with the table data (possibly a corrupted commit in the table " + "history), using the row count from the manifest files", + persistent_components.table_location, + *summary_total_rows, + result); +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) } const auto summary_total_rows = actual_data_snapshot->getTotalRows(); diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.cpp index 201fdcd3a7eb..9e70640da7b5 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.cpp @@ -92,7 +92,11 @@ void requireDirectReferencedDataFileForPuffinDeletionVector( { throw DB::Exception( DB::ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, +<<<<<<< HEAD "Deletion vector entry in manifest file '{}' is missing referenced_data_file", +======= + "Puffin deletion vector entry in manifest file '{}' is missing referenced_data_file", +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) manifest_file_path); } } diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.h index 7b49fdd76d09..1e9c221ff3a7 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.h @@ -30,6 +30,7 @@ struct ColumnInfo #include #include +#include namespace DB::Iceberg { @@ -104,14 +105,24 @@ struct ParsedManifestFileEntry : boost::noncopyable Int64 record_count; Int64 file_size_in_bytes; +<<<<<<< HEAD /// Iceberg v3 deletion vector metadata (`content_offset` / `content_size_in_bytes`). /// Present for Puffin containers and for Delta-style `.bin` files that store the same envelope. +======= + /// Iceberg v3 deletion vector metadata (position delete entries with puffin format) +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) std::optional content_offset; std::optional content_size_in_bytes; bool isDeletionVector() const { +<<<<<<< HEAD return content_offset.has_value() && content_size_in_bytes.has_value(); +======= + return Poco::toLower(file_format) == "puffin" + && content_offset.has_value() + && content_size_in_bytes.has_value(); +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) } ParsedManifestFileEntry( @@ -184,9 +195,14 @@ std::optional getRecordCountInAllFilesExcludingDeleted( std::optional getBytesSizeInAllDataFilesExcludingDeleted( const std::vector & files); +<<<<<<< HEAD /// Deletion vectors (Puffin or Delta `.bin`) must identify the data file via the dedicated /// `referenced_data_file` manifest field (non-empty). Position-delete lower/upper bounds /// must not be used as a fallback. +======= +/// Puffin deletion vectors must identify the data file via the dedicated `referenced_data_file` +/// manifest field (non-empty). Position-delete lower/upper bounds must not be used as a fallback. +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) void requireDirectReferencedDataFileForPuffinDeletionVector( bool set_from_referenced_data_file_field, const std::optional & referenced_path, diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp index cecfc51c3c4f..96e8353ce2f3 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp @@ -88,6 +88,7 @@ bool ManifestFileIterator::ManifestFileEntriesHandle::areAllDataFilesSortedBySor bool ManifestFileIterator::ManifestFileEntriesHandle::areAllDataFilesEligibleForLazyMaterialization(Int32 table_schema_id) const { +<<<<<<< HEAD /// Equality deletes force reading all physical columns of the data files they apply to /// (see IcebergMetadata::getInitialSchemaByPath), so the pruned main read is impossible. if (!equality_delete_files->empty()) @@ -117,6 +118,8 @@ std::optional ManifestFileIterator::ManifestFileEntriesHandle::getRowsCo /// silently produce a wrong -- or, after the conversion to size_t, absurdly huge -- /// count) and rather than rejected (the count is only an optimization, a malformed /// value must not make the table unreadable). +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) return getRecordCountInAllFilesExcludingDeleted(getFilesWithoutDeleted(content)); } diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/Mutations.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/Mutations.cpp index 0d2416b854f9..667115fdfd8d 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/Mutations.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/Mutations.cpp @@ -45,8 +45,13 @@ namespace DB::ErrorCodes extern const int BAD_ARGUMENTS; extern const int LOGICAL_ERROR; extern const int LIMIT_EXCEEDED; +<<<<<<< HEAD extern const int NOT_IMPLEMENTED; extern const int SUPPORT_IS_DISABLED; +======= +extern const int SUPPORT_IS_DISABLED; +extern const int QUERY_WAS_CANCELLED; +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) } namespace DB::DataLakeStorageSetting diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/Snapshot.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/Snapshot.h index 8ed1dc9b798a..1a6948f5aedc 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/Snapshot.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/Snapshot.h @@ -16,10 +16,16 @@ struct IcebergDataSnapshot DB::ManifestFileCacheKeys manifest_list_entries; Int64 snapshot_id; Int64 schema_id_on_snapshot_commit; +<<<<<<< HEAD /// Row-count hint from the snapshot summary (`total-records`). Only used to log a /// warning when it disagrees with the row count derived from the manifest files; never /// used as a data source, because the summary is maintained incrementally by writers /// and a corrupted commit in the table history poisons it silently. +======= + /// From snapshot summary (`total-records`). Compared to the manifest-derived count for a + /// mismatch warning only — never used as the trivial COUNT answer. Summary totals are + /// maintained incrementally by writers and can be poisoned by a bad commit in table history. +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) std::optional total_rows; std::optional total_bytes; std::optional total_position_delete_rows; diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_count_shortcuts.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_count_shortcuts.cpp index 1e00063187ba..2dd69cfc093f 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_count_shortcuts.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_count_shortcuts.cpp @@ -124,7 +124,10 @@ ProcessedManifestFileEntryPtr makeDataEntryForRecordCount( /*row_number=*/0, ManifestEntryStatus::ADDED, /*written_sequence_number=*/std::nullopt, +<<<<<<< HEAD /*written_file_sequence_number=*/std::nullopt, +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) /*written_snapshot_id=*/std::nullopt, DB::Row{}, std::move(columns_infos), diff --git a/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.cpp b/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.cpp index 92a11a6457d0..e89936c56454 100644 --- a/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.cpp +++ b/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.cpp @@ -8,8 +8,11 @@ #include #include #include +<<<<<<< HEAD #include #include +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) #include #include @@ -109,6 +112,7 @@ void checkDeletionVectorBlobReadLimits(Int64 length, std::optional expec throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector blob is too small"); } +<<<<<<< HEAD bool isPuffinFileMagic(const UInt8 * header) { return std::memcmp(header, PUFFIN_FILE_MAGIC, sizeof(PUFFIN_FILE_MAGIC)) == 0; @@ -148,11 +152,28 @@ void validateDeletionVectorEnvelope(const UInt8 * header, Int64 length) if (common::addOverflow(static_cast(combined_length), UInt64{8}, expected_blob_size)) throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector combined length: {}", combined_length); +======= +void validateDeletionVectorEnvelope(const UInt8 * header, Int64 length) +{ + const UInt32 combined_length = readBigEndianUInt32(header); + if (std::memcmp(header + sizeof(UInt32), DELETION_VECTOR_MAGIC, sizeof(DELETION_VECTOR_MAGIC)) != 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector magic"); + + if (combined_length < sizeof(DELETION_VECTOR_MAGIC)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector combined length: {}", combined_length); + + UInt64 expected_blob_size = 0; + if (common::addOverflow(static_cast(combined_length), UInt64{8}, expected_blob_size)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector combined length: {}", combined_length); + + if (static_cast(length) != expected_blob_size) +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) throw Exception( ErrorCodes::BAD_ARGUMENTS, "Deletion vector blob size {} does not match combined length {}", length, combined_length); +<<<<<<< HEAD } } @@ -229,6 +250,8 @@ IcebergDeletionVectorContainer detectIcebergDeletionVectorContainer( throwUnknownDeletionVectorContainer(path, header, content_offset); UNREACHABLE(); +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) } std::vector deserializeDeletionVectorV1Blob(std::string_view blob_bytes, std::optional expected_cardinality) diff --git a/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.h b/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.h index 4412015160d1..e4bd365bc0d7 100644 --- a/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.h +++ b/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.h @@ -4,11 +4,18 @@ #include #include +<<<<<<< HEAD #include #include #include #include #include +======= +#include +#include +#include +#include +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) namespace DB { @@ -45,6 +52,7 @@ void validatePuffinBlobBounds(Int64 offset, Int64 length, size_t file_size, std: /// Iceberg deletion-vector-v1 envelope magic (`0xD1D33964`), shared with SQL `Puffin` decode. inline constexpr UInt8 DELETION_VECTOR_MAGIC[4] = {0xD1, 0xD3, 0x39, 0x64}; +<<<<<<< HEAD /// Puffin file magic (`PFA1`), shared with SQL `Puffin` decode. inline constexpr UInt8 PUFFIN_FILE_MAGIC[4] = {0x50, 0x46, 0x41, 0x31}; @@ -73,6 +81,8 @@ IcebergDeletionVectorContainer detectIcebergDeletionVectorContainer( Int64 content_size_in_bytes, std::string_view path); +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) /// Fail closed before envelope peek / full allocate. Shared by SQL `Puffin` and Iceberg loaders. /// Order: cardinality ceiling, then length bounds (`length < 0`, absolute blob cap, min envelope). void checkDeletionVectorBlobReadLimits(Int64 length, std::optional expected_cardinality); diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_dv_referenced_data_file.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_dv_referenced_data_file.cpp index 855a8ac25f7c..09ed8326e077 100644 --- a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_dv_referenced_data_file.cpp +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_dv_referenced_data_file.cpp @@ -8,11 +8,14 @@ #include #include +<<<<<<< HEAD #include #include #include #include +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) using namespace DB; using namespace DB::Iceberg; @@ -73,6 +76,7 @@ TEST(PuffinDeletionVectorReferencedDataFile, RejectsEmptyPath) } } +<<<<<<< HEAD namespace { @@ -113,4 +117,6 @@ TEST(IcebergDeletionVectorClassification, OffsetsClassifyAsDeletionVectorRegardl EXPECT_FALSE(makePositionDeleteEntry("parquet", 1, std::nullopt)->isDeletionVector()); } +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) #endif diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_metrics.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_metrics.cpp index 966869851857..50c0791ebac5 100644 --- a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_metrics.cpp +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_metrics.cpp @@ -40,8 +40,13 @@ TEST(PuffinFilesCacheMetrics, ClearDuringLoadCountsAsMissNotHit) ASSERT_TRUE(key.has_value()); auto & counters = CurrentThread::getProfileEvents(); +<<<<<<< HEAD const auto hits_before = counters[ProfileEvents::PuffinFilesCacheHits]; const auto misses_before = counters[ProfileEvents::PuffinFilesCacheMisses]; +======= + const auto hits_before = counters[ProfileEvents::PuffinFilesCacheHits].load(); + const auto misses_before = counters[ProfileEvents::PuffinFilesCacheMisses].load(); +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) size_t load_calls = 0; const auto result = cache.getOrSetDeletionVector( @@ -58,8 +63,13 @@ TEST(PuffinFilesCacheMetrics, ClearDuringLoadCountsAsMissNotHit) ASSERT_EQ(load_calls, 1u); ASSERT_NE(result, nullptr); EXPECT_TRUE(result->rb_contains(1)); +<<<<<<< HEAD EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheHits] - hits_before, 0u); EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheMisses] - misses_before, 1u); +======= + EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheHits].load() - hits_before, 0u); + EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheMisses].load() - misses_before, 1u); +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) /// Entry was not inserted after clear; the next lookup must load again. const auto second = cache.getOrSetDeletionVector(*key, [&]() @@ -69,8 +79,13 @@ TEST(PuffinFilesCacheMetrics, ClearDuringLoadCountsAsMissNotHit) }); ASSERT_EQ(load_calls, 2u); ASSERT_NE(second, nullptr); +<<<<<<< HEAD EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheMisses] - misses_before, 2u); EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheHits] - hits_before, 0u); +======= + EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheMisses].load() - misses_before, 2u); + EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheHits].load() - hits_before, 0u); +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) } TEST(PuffinFilesCacheMetrics, WaiterOfClearDiscardedLoadCountsAsMiss) @@ -81,8 +96,13 @@ TEST(PuffinFilesCacheMetrics, WaiterOfClearDiscardedLoadCountsAsMiss) "Local:////test-prefix", "puffin.bin", "etag-waiter", 100, 200, "data/file-w.parquet", 1, 100); ASSERT_TRUE(key.has_value()); +<<<<<<< HEAD const auto hits_before = ProfileEvents::global_counters[ProfileEvents::PuffinFilesCacheHits]; const auto misses_before = ProfileEvents::global_counters[ProfileEvents::PuffinFilesCacheMisses]; +======= + const auto hits_before = ProfileEvents::global_counters[ProfileEvents::PuffinFilesCacheHits].load(); + const auto misses_before = ProfileEvents::global_counters[ProfileEvents::PuffinFilesCacheMisses].load(); +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) std::promise load_started; auto load_started_future = load_started.get_future(); @@ -151,8 +171,13 @@ TEST(PuffinFilesCacheMetrics, OrdinaryHitAndMissCounters) ASSERT_TRUE(key.has_value()); auto & counters = CurrentThread::getProfileEvents(); +<<<<<<< HEAD const auto hits_before = counters[ProfileEvents::PuffinFilesCacheHits]; const auto misses_before = counters[ProfileEvents::PuffinFilesCacheMisses]; +======= + const auto hits_before = counters[ProfileEvents::PuffinFilesCacheHits].load(); + const auto misses_before = counters[ProfileEvents::PuffinFilesCacheMisses].load(); +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) size_t load_calls = 0; auto load_fn = [&]() @@ -165,8 +190,13 @@ TEST(PuffinFilesCacheMetrics, OrdinaryHitAndMissCounters) ASSERT_NE(cache.getOrSetDeletionVector(*key, load_fn), nullptr); EXPECT_EQ(load_calls, 1u); +<<<<<<< HEAD EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheMisses] - misses_before, 1u); EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheHits] - hits_before, 1u); +======= + EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheMisses].load() - misses_before, 1u); + EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheHits].load() - hits_before, 1u); +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) } TEST(PuffinFilesCacheMetrics, HitRemainsHitWhenCacheClearedAfterLookup) @@ -187,6 +217,7 @@ TEST(PuffinFilesCacheMetrics, HitRemainsHitWhenCacheClearedAfterLookup) ASSERT_NE(cache.getOrSetDeletionVector(*key, load_fn), nullptr); +<<<<<<< HEAD const auto hits_before = counters[ProfileEvents::PuffinFilesCacheHits]; const auto misses_before = counters[ProfileEvents::PuffinFilesCacheMisses]; @@ -194,10 +225,24 @@ TEST(PuffinFilesCacheMetrics, HitRemainsHitWhenCacheClearedAfterLookup) EXPECT_EQ(load_calls, 1u); EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheHits] - hits_before, 1u); EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheMisses] - misses_before, 0u); +======= + const auto hits_before = counters[ProfileEvents::PuffinFilesCacheHits].load(); + const auto misses_before = counters[ProfileEvents::PuffinFilesCacheMisses].load(); + + ASSERT_NE(cache.getOrSetDeletionVector(*key, load_fn), nullptr); + EXPECT_EQ(load_calls, 1u); + EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheHits].load() - hits_before, 1u); + EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheMisses].load() - misses_before, 0u); +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) /// Clearing after the hit must not rewrite the already-recorded hit as a miss. The old /// contains()-after-getOrSet path could race here with SYSTEM DROP PUFFIN FILES CACHE. cache.clear(); +<<<<<<< HEAD EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheHits] - hits_before, 1u); EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheMisses] - misses_before, 0u); +======= + EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheHits].load() - hits_before, 1u); + EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheMisses].load() - misses_before, 0u); +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) } diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_footer_cache.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_footer_cache.cpp index cd7ea18fd553..4d1e261ad5b0 100644 --- a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_footer_cache.cpp +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_footer_cache.cpp @@ -84,7 +84,11 @@ TEST(PuffinFooterMemo, CoalescedSlicesShareOneFooterParse) ASSERT_TRUE(key_b.has_value()); auto & counters = ProfileEvents::global_counters; +<<<<<<< HEAD const auto files_read_before = counters[ProfileEvents::PuffinFilesRead]; +======= + const auto files_read_before = counters[ProfileEvents::PuffinFilesRead].load(); +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) size_t footer_loads = 0; auto load_footer = [&]() @@ -122,7 +126,11 @@ TEST(PuffinFooterMemo, CoalescedSlicesShareOneFooterParse) EXPECT_EQ(footer_loads, 1u); /// One footer parse (`PuffinFilesRead` in readPuffinFooter) plus two blob reads. +<<<<<<< HEAD EXPECT_EQ(counters[ProfileEvents::PuffinFilesRead] - files_read_before, 3u); +======= + EXPECT_EQ(counters[ProfileEvents::PuffinFilesRead].load() - files_read_before, 3u); +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) } TEST(PuffinFooterMemo, ClearDropsFooterEntries) diff --git a/src/Storages/ObjectStorage/IObjectIterator.h b/src/Storages/ObjectStorage/IObjectIterator.h index 467474e9d8e3..b96530cbc631 100644 --- a/src/Storages/ObjectStorage/IObjectIterator.h +++ b/src/Storages/ObjectStorage/IObjectIterator.h @@ -55,10 +55,13 @@ struct ObjectInfo FileBucketInfoPtr file_bucket_info; +<<<<<<< HEAD /// Lazy materialization: if set, read only these rows of the file. /// Sorted absolute row indexes within the file, see FormatFilterInfo::rows_to_read. std::shared_ptr> rows_to_read; +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) /// Polymorphic copy: preserves the dynamic type (e.g. `IcebergDataObjectInfo` with equality / /// position deletes, resolved storage, and metadata path) where a plain copy would slice it. virtual std::shared_ptr clone() const { return std::make_shared(*this); } diff --git a/src/Storages/ObjectStorage/StorageObjectStorageSource.cpp b/src/Storages/ObjectStorage/StorageObjectStorageSource.cpp index 1bed63dba006..48f2d3c8e1ef 100644 --- a/src/Storages/ObjectStorage/StorageObjectStorageSource.cpp +++ b/src/Storages/ObjectStorage/StorageObjectStorageSource.cpp @@ -242,6 +242,7 @@ static bool canUseCountFromFilesCache(const ObjectInfoPtr & object_info) return true; } +<<<<<<< HEAD static bool readsIdentityPartitionColumn( const ActionsDAG & dag, const std::vector> & identity_partition_columns) { @@ -306,6 +307,8 @@ static std::optional buildIdentityPartitionColumnsDag( return dag; } +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) StorageObjectStorageSource::StorageObjectStorageSource( const StorageID & storage_id_, String name_, @@ -927,9 +930,14 @@ Chunk StorageObjectStorageSource::generate() /// reduce rows seen by generate(), while the cache key is file identity only. if (reader.getInputFormat() && read_context->getSettingsRef()[Setting::use_cache_for_count_from_files] && format_filter_info && !format_filter_info->hasFilter() +<<<<<<< HEAD && canUseCountFromFilesCache(reader.getObjectInfo()) && !reader.getObjectInfo()->rows_to_read) addNumRowsToCache(*reader.getObjectInfo(), total_rows_in_file); +======= + && canUseCountFromFilesCache(reader.getObjectInfo())) + addNumRowsToCache(reader.getObjectInfo(), total_rows_in_file); +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) total_rows_in_file = 0; chassert(reader_future.valid()); @@ -1287,7 +1295,11 @@ StorageObjectStorageSource::ReaderHolder StorageObjectStorageSource::createReade const bool effective_need_only_count = need_only_count; #endif +<<<<<<< HEAD const bool can_use_count_cache = effective_need_only_count && !headers_requested +======= + const bool can_use_count_cache = effective_need_only_count +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) && context_->getSettingsRef()[Setting::use_cache_for_count_from_files] && canUseCountFromFilesCache(object_info); @@ -1577,6 +1589,7 @@ StorageObjectStorageSource::ReaderHolder StorageObjectStorageSource::createReade builder.init(Pipe(input_format)); +<<<<<<< HEAD if (!identity_partition_columns.empty()) { if (auto dag = buildIdentityPartitionColumnsDag(builder.getHeader(), identity_partition_columns)) @@ -1589,6 +1602,8 @@ StorageObjectStorageSource::ReaderHolder StorageObjectStorageSource::createReade } } +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) /// Deletion vectors (and selection vectors) address absolute file row numbers via /// `ChunkInfoRowNumbers`. Iceberg equality deletes use a plain `FilterTransform` that /// shrinks the chunk without maintaining `applied_filter`, so DV must run first — diff --git a/src/Storages/ObjectStorage/tests/gtest_rendezvous_hashing.cpp b/src/Storages/ObjectStorage/tests/gtest_rendezvous_hashing.cpp index 1c4216c61329..1fbc9ec45b10 100644 --- a/src/Storages/ObjectStorage/tests/gtest_rendezvous_hashing.cpp +++ b/src/Storages/ObjectStorage/tests/gtest_rendezvous_hashing.cpp @@ -1,8 +1,12 @@ #include +<<<<<<< HEAD #include #include #include #include +======= + +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) #include #include diff --git a/tests/integration/test_storage_iceberg_with_spark/test_deletion_vectors.py b/tests/integration/test_storage_iceberg_with_spark/test_deletion_vectors.py index 1d5fd3e64d63..3013eeae10d6 100644 --- a/tests/integration/test_storage_iceberg_with_spark/test_deletion_vectors.py +++ b/tests/integration/test_storage_iceberg_with_spark/test_deletion_vectors.py @@ -1,5 +1,8 @@ +<<<<<<< HEAD import json import os +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) import uuid import pytest @@ -42,6 +45,7 @@ def _strip_file_uri_scheme(path): return path +<<<<<<< HEAD PUFFIN_MAGIC = b"PFA1" PUFFIN_FOOTER_TRAILER_SIZE = 12 ICEBERG_WAREHOUSE = "/var/lib/clickhouse/user_files/iceberg_data" @@ -259,6 +263,8 @@ def convert_spark_puffin_dvs_to_delta_bin(table_name): return len(conversions) +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) def add_equality_deletes_by_id(spark, table_name, ids): """Commit an Iceberg equality-delete file for the given `id` values. @@ -364,6 +370,7 @@ def test_deletion_vectors(started_cluster_iceberg_with_spark, storage_type, run_ @pytest.mark.parametrize("run_on_cluster", [False, True]) @pytest.mark.parametrize("storage_type", ["s3", "azure", "local"]) +<<<<<<< HEAD def test_deletion_vectors_delta_bin(started_cluster_iceberg_with_spark, storage_type, run_on_cluster): """Iceberg v3 DVs stored as Delta `.bin` files (Databricks UniForm), not Puffin. @@ -426,6 +433,8 @@ def test_deletion_vectors_delta_bin(started_cluster_iceberg_with_spark, storage_ @pytest.mark.parametrize("run_on_cluster", [False, True]) @pytest.mark.parametrize("storage_type", ["s3", "azure", "local"]) +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) def test_deletion_vectors_aggregates(started_cluster_iceberg_with_spark, storage_type, run_on_cluster): """Aggregates over Iceberg v3 tables must ignore rows covered by deletion vectors.""" if storage_type == "local" and run_on_cluster: diff --git a/tests/queries/0_stateless/01271_show_privileges.reference b/tests/queries/0_stateless/01271_show_privileges.reference index 5f16f7ce7a47..bca6cea7a73f 100644 --- a/tests/queries/0_stateless/01271_show_privileges.reference +++ b/tests/queries/0_stateless/01271_show_privileges.reference @@ -137,7 +137,10 @@ SYSTEM DROP ICEBERG METADATA CACHE ['SYSTEM CLEAR ICEBERG_METADATA_CACHE','SYSTE SYSTEM DROP PAIMON METADATA CACHE ['SYSTEM CLEAR PAIMON_METADATA_CACHE','SYSTEM DROP PAIMON_METADATA_CACHE'] GLOBAL SYSTEM DROP CACHE SYSTEM DROP AVRO SCHEMA CACHE ['SYSTEM CLEAR AVRO SCHEMA CACHE','SYSTEM DROP AVRO SCHEMA CACHE','DROP AVRO SCHEMA CACHE'] GLOBAL SYSTEM DROP CACHE SYSTEM DROP PARQUET METADATA CACHE ['SYSTEM DROP PARQUET_METADATA_CACHE'] GLOBAL SYSTEM DROP CACHE +<<<<<<< HEAD SYSTEM DROP POINT IN POLYGON CACHE ['SYSTEM CLEAR POINT IN POLYGON CACHE','SYSTEM DROP POINT IN POLYGON CACHE','DROP POINT IN POLYGON CACHE'] GLOBAL SYSTEM DROP CACHE +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) SYSTEM DROP PUFFIN FILES CACHE ['SYSTEM DROP PUFFIN_FILES_CACHE'] GLOBAL SYSTEM DROP CACHE SYSTEM PREWARM PRIMARY INDEX CACHE ['SYSTEM PREWARM PRIMARY INDEX','PREWARM PRIMARY INDEX CACHE','PREWARM PRIMARY INDEX'] GLOBAL SYSTEM DROP CACHE SYSTEM DROP PRIMARY INDEX CACHE ['SYSTEM CLEAR PRIMARY INDEX CACHE','SYSTEM DROP PRIMARY INDEX','DROP PRIMARY INDEX CACHE','DROP PRIMARY INDEX'] GLOBAL SYSTEM DROP CACHE diff --git a/tests/queries/0_stateless/04117_parser_system_query_variants.reference b/tests/queries/0_stateless/04117_parser_system_query_variants.reference index 197276c6dac6..39ada1de3f2a 100644 --- a/tests/queries/0_stateless/04117_parser_system_query_variants.reference +++ b/tests/queries/0_stateless/04117_parser_system_query_variants.reference @@ -16,7 +16,10 @@ SYSTEM CLEAR ENCRYPTION HEADERS CACHE SYSTEM CLEAR COMPILED EXPRESSION CACHE SYSTEM CLEAR ICEBERG METADATA CACHE SYSTEM CLEAR PARQUET METADATA CACHE +<<<<<<< HEAD SYSTEM CLEAR POINT IN POLYGON CACHE +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) SYSTEM CLEAR PUFFIN FILES CACHE SYSTEM CLEAR PUFFIN FILES CACHE SYSTEM CLEAR FILESYSTEM CACHE diff --git a/tests/queries/0_stateless/04117_parser_system_query_variants.sql b/tests/queries/0_stateless/04117_parser_system_query_variants.sql index 9093c599264c..749b90faca20 100644 --- a/tests/queries/0_stateless/04117_parser_system_query_variants.sql +++ b/tests/queries/0_stateless/04117_parser_system_query_variants.sql @@ -26,7 +26,10 @@ EXPLAIN SYNTAX SYSTEM DROP ENCRYPTION HEADERS CACHE; EXPLAIN SYNTAX SYSTEM DROP COMPILED EXPRESSION CACHE; EXPLAIN SYNTAX SYSTEM DROP ICEBERG METADATA CACHE; EXPLAIN SYNTAX SYSTEM DROP PARQUET METADATA CACHE; +<<<<<<< HEAD EXPLAIN SYNTAX SYSTEM DROP POINT IN POLYGON CACHE; +======= +>>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) EXPLAIN SYNTAX SYSTEM DROP PUFFIN FILES CACHE; EXPLAIN SYNTAX SYSTEM DROP PUFFIN_FILES_CACHE; EXPLAIN SYNTAX SYSTEM DROP FILESYSTEM CACHE; From 4827d9ce8b8302e6e29b3b707f5d112320cf8ade Mon Sep 17 00:00:00 2001 From: Andrey Zvonov <32552679+zvonand@users.noreply.github.com> Date: Wed, 23 Sep 2026 23:05:18 +0200 Subject: [PATCH 2/2] Resolve conflicts in cherry-pick of #2183 PR #2183 (Iceberg deletion vector support) is already present on this branch: it was ported earlier in the pipeline (commits "Cherry-pick of .../pull/2183 ..." + "Resolve conflicts in cherry-pick of #2183") and then extended by the port of #2271 (Delta-style `.bin` deletion-vector containers, footer memo, DV classification by content offsets). Every conflict hunk therefore had "ours" holding the same change in its newer, superset form, so all hunks are resolved to "ours": - docs (iceberg.mdx, icebergCluster.mdx, system.mdx): ours documents the Puffin *and* Delta `.bin` containers and keeps the neighbouring Paimon / point-in-polygon cache sections. - AccessType.h, ASTSystemQuery.h, ParserSystemQuery.cpp, InterpreterSystemQuery.cpp, 01271_show_privileges.reference, 04117_parser_system_query_variants.{sql,reference}: SYSTEM DROP PUFFIN FILES CACHE already exists; the conflicts only concerned the neighbouring POINT IN POLYGON cache entries, which are kept. - ProfileEvents.cpp / Settings.cpp: append-only registries - only the rows already on the branch are kept (Puffin rows are already there). - Iceberg sources (AvroForIcebergDeserializer, IcebergDeletionVector, IcebergIterator, IcebergMetadata, ManifestFile*, Mutations, Snapshot, PuffinDeletionVectorReader, StorageObjectStorageSource, IObjectIterator, gtests): ours is the superset (content-offset based DV detection, path_resolver instead of the removed resolveObjectStorageForPath / requires_external_storage, deferred ensureDeletesReady, Delta `.bin` container detection, footer memo, lazy materialization). Three artifacts the replay left outside the markers are undone because "ours" already carries the same content: - ASTSystemQuery.cpp: removed the duplicated `case Type::CLEAR_PUFFIN_FILES_CACHE:` label (would not compile). - AvroForIcebergDeserializer.cpp: removed the duplicated `#include `. - ManifestFile.h: removed the `#include ` that was only needed by the older `Poco::toLower(file_format) == "puffin"` form of `isDeletionVector()`, which ours replaced. - SettingsChangesHistory.cpp: removed the stray blank line; the `use_puffin_files_cache` row is already present exactly once. Net effect: the tree is identical to the pre-cherry-pick state, i.e. the feature stays in the form already ported to antalya-26.8. --- .../table-engines/integrations/iceberg.mdx | 14 ----- .../functions/table-functions/iceberg.mdx | 14 ----- .../table-functions/icebergCluster.mdx | 4 -- docs/reference/statements/system.mdx | 3 - src/Access/Common/AccessType.h | 3 - .../AggregateFunctionGroupBitmapData.h | 5 -- src/Common/ProfileEvents.cpp | 3 - src/Core/Settings.cpp | 3 - src/Core/SettingsChangesHistory.cpp | 1 - src/Formats/FormatFilterInfo.cpp | 3 - src/Interpreters/InterpreterSystemQuery.cpp | 7 --- src/Parsers/ASTSystemQuery.cpp | 1 - src/Parsers/ASTSystemQuery.h | 3 - src/Parsers/ParserSystemQuery.cpp | 3 - .../Formats/Impl/Parquet/Reader.cpp | 9 --- .../Common/AvroForIcebergDeserializer.cpp | 17 ------ .../DataLakes/Iceberg/IcebergDataObjectInfo.h | 9 --- .../Iceberg/IcebergDeletionVector.cpp | 40 ------------- .../DataLakes/Iceberg/IcebergIterator.cpp | 60 ------------------- .../DataLakes/Iceberg/IcebergIterator.h | 3 - .../DataLakes/Iceberg/IcebergMetadata.cpp | 43 ------------- .../DataLakes/Iceberg/ManifestFile.cpp | 4 -- .../DataLakes/Iceberg/ManifestFile.h | 16 ----- .../Iceberg/ManifestFileIterator.cpp | 3 - .../DataLakes/Iceberg/Mutations.cpp | 5 -- .../DataLakes/Iceberg/Snapshot.h | 6 -- .../tests/gtest_iceberg_count_shortcuts.cpp | 3 - .../DataLakes/PuffinDeletionVectorReader.cpp | 23 ------- .../DataLakes/PuffinDeletionVectorReader.h | 10 ---- .../gtest_puffin_dv_referenced_data_file.cpp | 6 -- .../gtest_puffin_files_cache_metrics.cpp | 45 -------------- .../tests/gtest_puffin_footer_cache.cpp | 8 --- src/Storages/ObjectStorage/IObjectIterator.h | 3 - .../StorageObjectStorageSource.cpp | 15 ----- .../tests/gtest_rendezvous_hashing.cpp | 4 -- .../test_deletion_vectors.py | 9 --- .../01271_show_privileges.reference | 3 - ...117_parser_system_query_variants.reference | 3 - .../04117_parser_system_query_variants.sql | 3 - 39 files changed, 417 deletions(-) diff --git a/docs/reference/engines/table-engines/integrations/iceberg.mdx b/docs/reference/engines/table-engines/integrations/iceberg.mdx index a523625855d9..7a05dab165b5 100644 --- a/docs/reference/engines/table-engines/integrations/iceberg.mdx +++ b/docs/reference/engines/table-engines/integrations/iceberg.mdx @@ -149,33 +149,19 @@ ClickHouse supports reading Iceberg tables that use the following deletion metho - [Position deletes](https://iceberg.apache.org/spec/#position-delete-files) - [Equality deletes](https://iceberg.apache.org/spec/#equality-delete-files) (supported from version 25.8+) -<<<<<<< HEAD:docs/reference/engines/table-engines/integrations/iceberg.mdx - [Deletion vectors](https://iceberg.apache.org/spec/#deletion-vectors) (Iceberg v3, read-only), stored either in Puffin files or in Delta-style `deletion_vector_*.bin` files using the same `deletion-vector-v1` envelope at the manifest `content_offset` / `content_size_in_bytes` The following limitations apply to deletion vectors: - Only `deletion-vector-v1` blobs are supported (Puffin container or Delta `.bin` slice) -======= -- [Deletion vectors](https://iceberg.apache.org/spec/#deletion-vectors) stored in Puffin files (Iceberg v3, read-only) - -The following limitations apply to deletion vectors: - -- Only `deletion-vector-v1` Puffin blobs are supported ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2):docs/en/engines/table-engines/integrations/iceberg.md - Data files must be in Parquet format - Column-scoped deletion vectors (user column ids in puffin `fields`) are not supported. Writers may set `fields` to `[]` or to the Iceberg reserved `_pos` id (`2147483645`) for file-scoped deletion vectors. - Writing deletion vectors is not supported - `DELETE` / `UPDATE` mutations on Iceberg format version 3+ tables are rejected (writers must not add position-delete files) -<<<<<<< HEAD:docs/reference/engines/table-engines/integrations/iceberg.mdx Parsed deletion vectors can be cached in memory when `use_puffin_files_cache` is enabled and the deletion-vector object has a non-empty `etag`. Empty deletion vectors are cached as well, so repeated reads do not re-fetch the object. Parsed footers for coalesced multi-DV Puffin files are memoized with that cache (same identity: storage, path, `etag`) so slices share one footer parse; Delta `.bin` files have no Puffin footer and skip that memo. The memo shares `puffin_files_cache_size` / max-entry limits and is dropped when the cache is disabled (`puffin_files_cache_size=0`) or cleared. The cache can be cleared with `SYSTEM DROP PUFFIN FILES CACHE`. For [`icebergCluster`](/sql-reference/table-functions/icebergCluster.md) (and `object_storage_cluster`), the initiator loads and materializes each data file's deletion vector while distributing tasks, then sends the resulting row bitmap to workers with the task. Workers apply the bitmap; they do not re-read the Puffin or `.bin` object for that path. On wide v3 tables this can make the initiator a serialization point for deletion-vector I/O and decode. -======= -Parsed deletion vectors can be cached in memory when `use_puffin_files_cache` is enabled and the puffin file has a non-empty `etag`. Empty deletion vectors are cached as well, so repeated reads do not re-fetch the puffin file. Parsed footers for coalesced multi-DV Puffin files are memoized with that cache (same identity: storage, path, `etag`) so slices share one footer parse; the memo shares `puffin_files_cache_size` / max-entry limits and is dropped when the cache is disabled (`puffin_files_cache_size=0`) or cleared. The cache can be cleared with `SYSTEM DROP PUFFIN FILES CACHE`. - -For [`icebergCluster`](/sql-reference/table-functions/icebergCluster.md) (and `object_storage_cluster`), the initiator loads and materializes each data file's deletion vector while distributing tasks, then sends the resulting row bitmap to workers with the task. Workers apply the bitmap; they do not re-read the Puffin blob for that path. On wide v3 tables this can make the initiator a serialization point for deletion-vector I/O and decode. ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2):docs/en/engines/table-engines/integrations/iceberg.md ### Basic usage {#basic-usage} ```sql diff --git a/docs/reference/functions/table-functions/iceberg.mdx b/docs/reference/functions/table-functions/iceberg.mdx index d14304abb36a..25c5c9cde65d 100644 --- a/docs/reference/functions/table-functions/iceberg.mdx +++ b/docs/reference/functions/table-functions/iceberg.mdx @@ -126,33 +126,19 @@ ClickHouse supports reading Iceberg tables that use the following deletion metho - [Position deletes](https://iceberg.apache.org/spec/#position-delete-files) - [Equality deletes](https://iceberg.apache.org/spec/#equality-delete-files) (supported from version 25.8+) -<<<<<<< HEAD:docs/reference/functions/table-functions/iceberg.mdx - [Deletion vectors](https://iceberg.apache.org/spec/#deletion-vectors) (Iceberg v3, read-only), stored either in Puffin files or in Delta-style `deletion_vector_*.bin` files using the same `deletion-vector-v1` envelope at the manifest `content_offset` / `content_size_in_bytes` The following limitations apply to deletion vectors: - Only `deletion-vector-v1` blobs are supported (Puffin container or Delta `.bin` slice) -======= -- [Deletion vectors](https://iceberg.apache.org/spec/#deletion-vectors) stored in Puffin files (Iceberg v3, read-only) - -The following limitations apply to deletion vectors: - -- Only `deletion-vector-v1` Puffin blobs are supported ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2):docs/en/sql-reference/table-functions/iceberg.md - Data files must be in Parquet format - Column-scoped deletion vectors (user column ids in puffin `fields`) are not supported. Writers may set `fields` to `[]` or to the Iceberg reserved `_pos` id (`2147483645`) for file-scoped deletion vectors. - Writing deletion vectors is not supported - `DELETE` / `UPDATE` mutations on Iceberg format version 3+ tables are rejected (writers must not add position-delete files) -<<<<<<< HEAD:docs/reference/functions/table-functions/iceberg.mdx Parsed deletion vectors can be cached in memory when `use_puffin_files_cache` is enabled and the deletion-vector object has a non-empty `etag`. Empty deletion vectors are cached as well, so repeated reads do not re-fetch the object. Parsed footers for coalesced multi-DV Puffin files are memoized with that cache (same identity: storage, path, `etag`) so slices share one footer parse; Delta `.bin` files have no Puffin footer and skip that memo. The memo shares `puffin_files_cache_size` / max-entry limits and is dropped when the cache is disabled (`puffin_files_cache_size=0`) or cleared. The cache can be cleared with `SYSTEM DROP PUFFIN FILES CACHE`. For [`icebergCluster`](/sql-reference/table-functions/icebergCluster.md) (and `object_storage_cluster`), the initiator loads and materializes each data file's deletion vector while distributing tasks, then sends the resulting row bitmap to workers with the task. Workers apply the bitmap; they do not re-read the Puffin or `.bin` object for that path. On wide v3 tables this can make the initiator a serialization point for deletion-vector I/O and decode. -======= -Parsed deletion vectors can be cached in memory when `use_puffin_files_cache` is enabled and the puffin file has a non-empty `etag`. Empty deletion vectors are cached as well, so repeated reads do not re-fetch the puffin file. Parsed footers for coalesced multi-DV Puffin files are memoized with that cache (same identity: storage, path, `etag`) so slices share one footer parse; the memo shares `puffin_files_cache_size` / max-entry limits and is dropped when the cache is disabled (`puffin_files_cache_size=0`) or cleared. The cache can be cleared with `SYSTEM DROP PUFFIN FILES CACHE`. - -For [`icebergCluster`](/sql-reference/table-functions/icebergCluster.md) (and `object_storage_cluster`), the initiator loads and materializes each data file's deletion vector while distributing tasks, then sends the resulting row bitmap to workers with the task. Workers apply the bitmap; they do not re-read the Puffin blob for that path. On wide v3 tables this can make the initiator a serialization point for deletion-vector I/O and decode. ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2):docs/en/sql-reference/table-functions/iceberg.md ### Basic usage {#basic-usage} diff --git a/docs/reference/functions/table-functions/icebergCluster.mdx b/docs/reference/functions/table-functions/icebergCluster.mdx index 90f8ec77de5e..4f50d2cbe508 100644 --- a/docs/reference/functions/table-functions/icebergCluster.mdx +++ b/docs/reference/functions/table-functions/icebergCluster.mdx @@ -14,11 +14,7 @@ Allows processing files from Apache [Iceberg](https://iceberg.apache.org/) in pa ## Deletion vectors on cluster reads {#deletion-vectors-cluster} -<<<<<<< HEAD:docs/reference/functions/table-functions/icebergCluster.mdx Iceberg v3 [deletion vectors](https://iceberg.apache.org/spec/#deletion-vectors) are loaded on the **initiator** while it distributes tasks: for each data file the initiator reads the Puffin or Delta `.bin` object, validates the `deletion-vector-v1` envelope, materializes deleted row positions, and attaches the bitmap to the task sent to workers. Workers apply that bitmap when reading Parquet; they do not fetch or parse the deletion-vector file again for that path. See [Processing of tables with deleted rows](/sql-reference/table-functions/iceberg.md#deleted-rows) for format limits and caching. -======= -Iceberg v3 [deletion vectors](https://iceberg.apache.org/spec/#deletion-vectors) are loaded on the **initiator** while it distributes tasks: for each data file the initiator reads the Puffin blob, validates it, materializes deleted row positions, and attaches the bitmap to the task sent to workers. Workers apply that bitmap when reading Parquet; they do not fetch or parse the Puffin file again for that path. See [Processing of tables with deleted rows](/sql-reference/table-functions/iceberg.md#deleted-rows) for format limits and caching. ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2):docs/en/sql-reference/table-functions/icebergCluster.md On wide tables with many deletion vectors, initiator-side decode and per-task bitmap serialization can become a bottleneck even when Parquet reads are well parallelized across the cluster. diff --git a/docs/reference/statements/system.mdx b/docs/reference/statements/system.mdx index 8612fb002ddf..9e083a92e529 100644 --- a/docs/reference/statements/system.mdx +++ b/docs/reference/statements/system.mdx @@ -134,7 +134,6 @@ Clears the per-URL Confluent Schema Registry caches used by the `AvroConfluent` Clears the parquet metadata cache. -<<<<<<< HEAD:docs/reference/statements/system.mdx ## SYSTEM CLEAR|DROP PAIMON METADATA CACHE {#drop-paimon-metadata-cache} Clears the in-memory cache of parsed Paimon metadata files (manifest lists and manifests). @@ -142,8 +141,6 @@ Clears the in-memory cache of parsed Paimon metadata files (manifest lists and m Clears the cache of preprocessed constant polygons used by the function [`pointInPolygon`](/reference/functions/regular-functions/geo/coordinates#pointinpolygon). The configured size limit (the `point_in_polygon_cache_size` server setting) is left unchanged, so the cache keeps accepting entries afterwards. To disable the cache instead, set `point_in_polygon_cache_size` to `0`. -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2):docs/en/sql-reference/statements/system.md ## SYSTEM DROP PUFFIN FILES CACHE {#drop-puffin-files-cache} Clears the Puffin files cache used for parsed Iceberg puffin file content such as deletion vectors. diff --git a/src/Access/Common/AccessType.h b/src/Access/Common/AccessType.h index 8556c56f683c..768721c15813 100644 --- a/src/Access/Common/AccessType.h +++ b/src/Access/Common/AccessType.h @@ -332,10 +332,7 @@ enum class AccessType : uint8_t M(SYSTEM_DROP_PAIMON_METADATA_CACHE, "SYSTEM CLEAR PAIMON_METADATA_CACHE, SYSTEM DROP PAIMON_METADATA_CACHE", GLOBAL, SYSTEM_DROP_CACHE) \ M(SYSTEM_DROP_AVRO_SCHEMA_CACHE, "SYSTEM CLEAR AVRO SCHEMA CACHE, SYSTEM DROP AVRO SCHEMA CACHE, DROP AVRO SCHEMA CACHE", GLOBAL, SYSTEM_DROP_CACHE) \ M(SYSTEM_DROP_PARQUET_METADATA_CACHE, "SYSTEM DROP PARQUET_METADATA_CACHE", GLOBAL, SYSTEM_DROP_CACHE) \ -<<<<<<< HEAD M(SYSTEM_DROP_POINT_IN_POLYGON_CACHE, "SYSTEM CLEAR POINT IN POLYGON CACHE, SYSTEM DROP POINT IN POLYGON CACHE, DROP POINT IN POLYGON CACHE", GLOBAL, SYSTEM_DROP_CACHE) \ -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) M(SYSTEM_DROP_PUFFIN_FILES_CACHE, "SYSTEM DROP PUFFIN_FILES_CACHE", GLOBAL, SYSTEM_DROP_CACHE) \ M(SYSTEM_PREWARM_PRIMARY_INDEX_CACHE, "SYSTEM PREWARM PRIMARY INDEX, PREWARM PRIMARY INDEX CACHE, PREWARM PRIMARY INDEX", GLOBAL, SYSTEM_DROP_CACHE) \ M(SYSTEM_DROP_PRIMARY_INDEX_CACHE, "SYSTEM CLEAR PRIMARY INDEX CACHE, SYSTEM DROP PRIMARY INDEX, DROP PRIMARY INDEX CACHE, DROP PRIMARY INDEX", GLOBAL, SYSTEM_DROP_CACHE) \ diff --git a/src/AggregateFunctions/AggregateFunctionGroupBitmapData.h b/src/AggregateFunctions/AggregateFunctionGroupBitmapData.h index 274b3566e6c6..c7539ee882a8 100644 --- a/src/AggregateFunctions/AggregateFunctionGroupBitmapData.h +++ b/src/AggregateFunctions/AggregateFunctionGroupBitmapData.h @@ -673,12 +673,7 @@ class RoaringBitmapWithSmallSet : private boost::noncopyable } /** -<<<<<<< HEAD * Return new set of the smallest `limit` values (as UnsignedT) which are no less than `range_start`. -======= - * Return new set of the smallest `limit` values in set which is no less than `range_start`. - * It's used in subset and currently only support UInt32 ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) */ UInt64 rb_limit(UInt64 range_start, UInt64 limit, RoaringBitmapWithSmallSet & r1) const /// NOLINT { diff --git a/src/Common/ProfileEvents.cpp b/src/Common/ProfileEvents.cpp index d44c5f880f3e..7e7d268a88ad 100644 --- a/src/Common/ProfileEvents.cpp +++ b/src/Common/ProfileEvents.cpp @@ -124,13 +124,10 @@ M(IcebergMetadataReturnedObjectInfos, "Total number of returned object infos from iceberg iterator.", ValueType::Number) \ M(IcebergMinMaxNonPrunedDeleteFiles, "Total number of accepted data files-position delete file pairs by minmax analysis from pairs suitable by partitioning and sequence number.", ValueType::Number) \ M(IcebergMinMaxPrunedDeleteFiles, "Total number of accepted data files-position delete file pairs by minmax analysis from pairs suitable by partitioning and sequence number.", ValueType::Number) \ -<<<<<<< HEAD M(OneLakeAccessTokenRequests, "Number of access token requests issued by the OneLake catalog via the Entra ID refresh token grant.", ValueType::Number) \ M(OneLakeAccessTokenRequestFailures, "Number of failed access token requests issued by the OneLake catalog via the Entra ID refresh token grant (including expired or revoked refresh tokens).", ValueType::Number) \ M(OneLakeAccessTokenRequestMicroseconds, "Total time spent requesting access tokens via the Entra ID refresh token grant in the OneLake catalog.", ValueType::Microseconds) \ M(OneLakeAccessTokenExpirations, "Number of times the cached OneLake access token was found expired and renewed transparently.", ValueType::Number) \ -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) M(PuffinFilesRead, "Number of Puffin files read (footer or deletion vector blob).", ValueType::Number) \ M(PuffinFileReadMicroseconds, "Total time spent reading Puffin files.", ValueType::Microseconds) \ M(PuffinFilesCacheHits, "Number of times parsed Puffin file content has been found in the cache.", ValueType::Number) \ diff --git a/src/Core/Settings.cpp b/src/Core/Settings.cpp index 01499541342c..881dd7186dd6 100644 --- a/src/Core/Settings.cpp +++ b/src/Core/Settings.cpp @@ -6051,7 +6051,6 @@ Minimum time of delay between 2 background compaction operations. DECLARE(Seconds, iceberg_compaction_data_cleanup, 60 * 60 * 3, R"( The time after which the data will be deleted. )", 0) \ -<<<<<<< HEAD DECLARE(UInt64, iceberg_compaction_commit_batch_size, 100, R"( Number of merged data files that background Iceberg compaction accumulates before publishing them in a new snapshot. @@ -6060,8 +6059,6 @@ how long already merged files stay unpublished while compaction keeps finding ne of publishing only when compaction runs out of candidates - a state that is never reached while the table keeps receiving new data files, so every merged output is then written to object storage and never referenced by a snapshot. )", 0) \ -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) DECLARE(Bool, use_puffin_files_cache, true, R"( If turned on, Iceberg reads may utilize the Puffin files cache for parsed puffin file content such as deletion vectors. diff --git a/src/Core/SettingsChangesHistory.cpp b/src/Core/SettingsChangesHistory.cpp index 94ab67aa25ab..441325235ea7 100644 --- a/src/Core/SettingsChangesHistory.cpp +++ b/src/Core/SettingsChangesHistory.cpp @@ -220,7 +220,6 @@ const VersionToSettingsChangesMap & getSettingsChangesHistory() { {"use_puffin_files_cache", false, true, "Enables cache of parsed Puffin file content such as deletion vectors."}, }); - addSettingsChanges(settings_changes_history, "26.6", { {"analyzer_compatibility_apply_final_to_all_joined_tables", true, false, "Fixed a bug in the analyzer where FINAL on the left-most table of a JOIN was incorrectly applied to the other joined tables as well. previous_value=true so `compatibility` with versions before 26.6 restores the old behavior."}, diff --git a/src/Formats/FormatFilterInfo.cpp b/src/Formats/FormatFilterInfo.cpp index 1f87b0720eb3..cee9aa77ad60 100644 --- a/src/Formats/FormatFilterInfo.cpp +++ b/src/Formats/FormatFilterInfo.cpp @@ -86,7 +86,6 @@ bool FormatFilterInfo::hasFilter() const /// Any of these can reduce the number of rows emitted by the reader pipeline. /// Count-from-files cache must not be populated when they are present. return filter_actions_dag != nullptr || row_level_filter != nullptr || prewhere_info != nullptr; -<<<<<<< HEAD } namespace @@ -104,8 +103,6 @@ namespace return true; return false; } -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) } Block FormatFilterInfo::buildKeyConditionInputs( diff --git a/src/Interpreters/InterpreterSystemQuery.cpp b/src/Interpreters/InterpreterSystemQuery.cpp index bd9aca33b9d0..e070cd3aafcf 100644 --- a/src/Interpreters/InterpreterSystemQuery.cpp +++ b/src/Interpreters/InterpreterSystemQuery.cpp @@ -505,13 +505,10 @@ BlockIO InterpreterSystemQuery::execute() #else throw Exception(ErrorCodes::SUPPORT_IS_DISABLED, "The server was compiled without the support for Parquet"); #endif -<<<<<<< HEAD case Type::CLEAR_POINT_IN_POLYGON_CACHE: getContext()->checkAccess(AccessType::SYSTEM_DROP_POINT_IN_POLYGON_CACHE); clearPointInPolygonCache(); break; -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) case Type::CLEAR_PUFFIN_FILES_CACHE: getContext()->checkAccess(AccessType::SYSTEM_DROP_PUFFIN_FILES_CACHE); system_context->clearPuffinFilesCache(); @@ -2829,7 +2826,6 @@ AccessRightsElements InterpreterSystemQuery::getRequiredAccessForDDLOnCluster() required_access.emplace_back(AccessType::SYSTEM_DROP_AVRO_SCHEMA_CACHE); break; case Type::CLEAR_PARQUET_METADATA_CACHE: -<<<<<<< HEAD required_access.emplace_back(AccessType::SYSTEM_DROP_PARQUET_METADATA_CACHE); break; case Type::CLEAR_POINT_IN_POLYGON_CACHE: @@ -2838,9 +2834,6 @@ AccessRightsElements InterpreterSystemQuery::getRequiredAccessForDDLOnCluster() case Type::CLEAR_PUFFIN_FILES_CACHE: required_access.emplace_back(AccessType::SYSTEM_DROP_PUFFIN_FILES_CACHE); break; -======= - case Type::CLEAR_PUFFIN_FILES_CACHE: ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) case Type::CLEAR_PRIMARY_INDEX_CACHE: required_access.emplace_back(AccessType::SYSTEM_DROP_PRIMARY_INDEX_CACHE); break; diff --git a/src/Parsers/ASTSystemQuery.cpp b/src/Parsers/ASTSystemQuery.cpp index 8c10b1d1ae62..bf7cd7f177dc 100644 --- a/src/Parsers/ASTSystemQuery.cpp +++ b/src/Parsers/ASTSystemQuery.cpp @@ -635,7 +635,6 @@ void ASTSystemQuery::formatImpl(WriteBuffer & ostr, const FormatSettings & setti case Type::CLEAR_PARQUET_METADATA_CACHE: case Type::CLEAR_POINT_IN_POLYGON_CACHE: case Type::CLEAR_AVRO_SCHEMA_CACHE: - case Type::CLEAR_PUFFIN_FILES_CACHE: case Type::DROP_OBJECT_STORAGE_LIST_OBJECTS_CACHE: case Type::CLEAR_PUFFIN_FILES_CACHE: case Type::RESET_COVERAGE: diff --git a/src/Parsers/ASTSystemQuery.h b/src/Parsers/ASTSystemQuery.h index 2f280e410ae4..1439e9c1743e 100644 --- a/src/Parsers/ASTSystemQuery.h +++ b/src/Parsers/ASTSystemQuery.h @@ -49,10 +49,7 @@ class ASTSystemQuery : public IAST, public ASTQueryWithOnCluster CLEAR_ICEBERG_METADATA_CACHE, CLEAR_PAIMON_METADATA_CACHE, CLEAR_PARQUET_METADATA_CACHE, -<<<<<<< HEAD CLEAR_POINT_IN_POLYGON_CACHE, -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) CLEAR_PUFFIN_FILES_CACHE, CLEAR_FILESYSTEM_CACHE, CLEAR_DISTRIBUTED_CACHE, diff --git a/src/Parsers/ParserSystemQuery.cpp b/src/Parsers/ParserSystemQuery.cpp index f0de9c7e39b7..4ac8beaf354b 100644 --- a/src/Parsers/ParserSystemQuery.cpp +++ b/src/Parsers/ParserSystemQuery.cpp @@ -322,10 +322,7 @@ bool ParserSystemQuery::parseImpl(IParser::Pos & pos, ASTPtr & node, Expected & {"DROP ICEBERG METADATA CACHE", Type::CLEAR_ICEBERG_METADATA_CACHE}, {"DROP PAIMON METADATA CACHE", Type::CLEAR_PAIMON_METADATA_CACHE}, {"DROP PARQUET METADATA CACHE", Type::CLEAR_PARQUET_METADATA_CACHE}, -<<<<<<< HEAD {"DROP POINT IN POLYGON CACHE", Type::CLEAR_POINT_IN_POLYGON_CACHE}, -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) {"DROP PUFFIN FILES CACHE", Type::CLEAR_PUFFIN_FILES_CACHE}, {"DROP PUFFIN_FILES_CACHE", Type::CLEAR_PUFFIN_FILES_CACHE}, {"DROP FILESYSTEM CACHE", Type::CLEAR_FILESYSTEM_CACHE}, diff --git a/src/Processors/Formats/Impl/Parquet/Reader.cpp b/src/Processors/Formats/Impl/Parquet/Reader.cpp index 967989634430..4c5e176286da 100644 --- a/src/Processors/Formats/Impl/Parquet/Reader.cpp +++ b/src/Processors/Formats/Impl/Parquet/Reader.cpp @@ -13,10 +13,7 @@ #include #include #include -<<<<<<< HEAD #include -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) #include #include #include @@ -364,7 +361,6 @@ void Reader::getHyperrectangleForRowGroup(const parq::RowGroup * meta, Hyperrect } } -<<<<<<< HEAD bool Reader::spatialBboxStatsHaveNoNulls(const parq::RowGroup & meta, size_t spatial_key_condition_idx) const { for (size_t bbox_pc_idx : spatial_key_condition_bbox_col_indices.at(spatial_key_condition_idx)) @@ -378,8 +374,6 @@ bool Reader::spatialBboxStatsHaveNoNulls(const parq::RowGroup & meta, size_t spa return true; } -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) std::vector buildRowGroupGlobalOffsets(const parq::FileMetaData & file_metadata) { if (file_metadata.num_rows < 0) @@ -749,7 +743,6 @@ void Reader::prefilterAndInitRowGroups(const std::optionalcolumns.size() != total_primitive_columns_in_file) throw Exception(ErrorCodes::INCORRECT_DATA, "Row group {} has unexpected number of columns: {} != {}", row_group_idx, meta->columns.size(), total_primitive_columns_in_file); -<<<<<<< HEAD /// Lazy materialization: skip row groups that contain none of the requested rows. std::pair requested_rows_slice {0, 0}; if (rows_to_read) @@ -762,8 +755,6 @@ void Reader::prefilterAndInitRowGroups(const std::optionalbegin()), size_t(end_it - rows_to_read->begin())}; } -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) Hyperrectangle hyperrectangle(extended_sample_block.columns(), Range::createWholeUniverse()); if ((options.format.parquet.filter_push_down && format_filter_info->key_condition) || !spatial_key_conditions.empty()) diff --git a/src/Storages/ObjectStorage/DataLakes/Common/AvroForIcebergDeserializer.cpp b/src/Storages/ObjectStorage/DataLakes/Common/AvroForIcebergDeserializer.cpp index a08a1bd75b42..a9534abdd383 100644 --- a/src/Storages/ObjectStorage/DataLakes/Common/AvroForIcebergDeserializer.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Common/AvroForIcebergDeserializer.cpp @@ -17,7 +17,6 @@ #include #include #include -#include #include #include @@ -333,10 +332,7 @@ ParsedManifestFileEntryPtr AvroForIcebergDeserializer::createParsedManifestFileE case FileContentType::POSITION_DELETE: { /// reference_file_path can be absent in schema for some reason, though it is present in specification: https://iceberg.apache.org/spec/#manifests const bool is_puffin = Poco::toLower(file_format) == "puffin"; -<<<<<<< HEAD const bool has_dv_offsets = content_offset.has_value() && content_size_in_bytes.has_value(); -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) std::optional lower_reference_data_file_path; std::optional upper_reference_data_file_path; bool bounds_set_by_referenced_data_file = false; @@ -352,15 +348,9 @@ ParsedManifestFileEntryPtr AvroForIcebergDeserializer::createParsedManifestFileE bounds_set_by_referenced_data_file = true; } } -<<<<<<< HEAD /// Parquet position deletes may fall back to file-path column bounds. Deletion /// vectors must use the dedicated referenced_data_file field only. if (!bounds_set_by_referenced_data_file && !is_puffin && !has_dv_offsets) -======= - /// Parquet position deletes may fall back to file-path column bounds. Puffin deletion - /// vectors must use the dedicated referenced_data_file field only. - if (!bounds_set_by_referenced_data_file && !is_puffin) ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) { if (auto it = value_for_bounds.find(IcebergPositionDeleteTransform::data_file_path_column_field_id); it != value_for_bounds.end()) @@ -375,24 +365,17 @@ ParsedManifestFileEntryPtr AvroForIcebergDeserializer::createParsedManifestFileE if (is_puffin) { -<<<<<<< HEAD if (!has_dv_offsets) -======= - if (!content_offset.has_value() || !content_size_in_bytes.has_value()) ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) { throw Exception( DB::ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, "Puffin deletion vector entry in manifest file '{}' is missing content_offset or content_size_in_bytes", manifest_file_path); } -<<<<<<< HEAD } if (is_puffin || has_dv_offsets) { -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) requireDirectReferencedDataFileForPuffinDeletionVector( bounds_set_by_referenced_data_file, lower_reference_data_file_path, manifest_file_path); } diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataObjectInfo.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataObjectInfo.h index 637438b12f7b..c5062f74ecc5 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataObjectInfo.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDataObjectInfo.h @@ -100,15 +100,6 @@ struct IcebergDataObjectInfo : public ObjectInfo, std::enable_shared_from_this clone() const override { return std::make_shared(*this); } -<<<<<<< HEAD -======= - ObjectStoragePtr getResolvedStorage() const { return resolved_storage; } - - void setResolvedStorage(ObjectStoragePtr storage) { resolved_storage = std::move(storage); } - - void addEqualityDeleteObject(const Iceberg::ProcessedManifestFileEntryPtr & equality_delete_object, const String & resolved_storage_path); - ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) Iceberg::IcebergObjectSerializableInfo info; }; diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDeletionVector.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDeletionVector.cpp index d47c4f6cb319..7f77e3f40e3c 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDeletionVector.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergDeletionVector.cpp @@ -18,10 +18,7 @@ #include #include -<<<<<<< HEAD #include -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) namespace DB { @@ -135,13 +132,9 @@ DataLakeObjectMetadata::ExcludedRowsPtr loadDeletionVectorUncached( ContextPtr context, LoggerPtr log, bool disable_filesystem_cache, -<<<<<<< HEAD FooterBlobsPtr preloaded_footer, PuffinFilesCache * footer_cache = nullptr, const std::optional & footer_key = {}) -======= - FooterBlobsPtr preloaded_footer) ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) { RelativePathWithMetadata puffin_object{puffin_path}; auto read_settings = context->getReadSettings(); @@ -158,7 +151,6 @@ DataLakeObjectMetadata::ExcludedRowsPtr loadDeletionVectorUncached( if (!file_size) throw Exception(ErrorCodes::BAD_ARGUMENTS, "Cannot determine Puffin file size for '{}'", puffin_path); -<<<<<<< HEAD const auto container = detectIcebergDeletionVectorContainer( *seekable, *file_size, content_offset, content_size_in_bytes, puffin_path); @@ -188,18 +180,6 @@ DataLakeObjectMetadata::ExcludedRowsPtr loadDeletionVectorUncached( expected_data_file.serialize(), expected_cardinality); } -======= - FooterBlobsPtr footer_owner = preloaded_footer; - if (!footer_owner) - footer_owner = std::make_shared>(readPuffinFooterBlobsFromSeekable(*seekable, *file_size)); - - bindDeletionVectorBlob( - *footer_owner, - content_offset, - content_size_in_bytes, - expected_data_file.serialize(), - expected_cardinality); ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) auto deleted_positions = readDeletionVectorFromPuffin( *read_buffer, content_offset, content_size_in_bytes, expected_cardinality); @@ -215,14 +195,9 @@ DataLakeObjectMetadata::ExcludedRowsPtr loadDeletionVectorUncached( LOG_DEBUG( log, -<<<<<<< HEAD "Loaded deletion vector from file '{}' ({}) for data file '{}': {} deleted rows", puffin_path, container == IcebergDeletionVectorContainer::Puffin ? "Puffin" : "Delta .bin", -======= - "Loaded deletion vector from puffin file '{}' for data file '{}': {} deleted rows", - puffin_path, ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) expected_data_file.serialize(), deleted_positions.size()); @@ -415,21 +390,10 @@ DataLakeObjectMetadata::ExcludedRowsPtr loadDeletionVector( } /// Footer is keyed by file identity only, so N DV slices in one coalesced Puffin share one parse. -<<<<<<< HEAD /// Resolve the footer only on a Puffin deletion-vector cache miss (nested memo lookup). /// Delta `.bin` objects have no Puffin footer; detection inside the uncached loader skips memo. return cache->getOrSetDeletionVector(*cache_key, [&]() { -======= - /// Resolve the footer only on a deletion-vector cache miss (nested memo lookup). - return cache->getOrSetDeletionVector(*cache_key, [&]() - { - auto footer = cache->getOrSetFooter(*footer_key, [&]() - { - return readFooterBlobs(object_storage, puffin_path, context, log, /*disable_filesystem_cache=*/ true); - }); - ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) return loadDeletionVectorUncached( object_storage, puffin_path, @@ -441,13 +405,9 @@ DataLakeObjectMetadata::ExcludedRowsPtr loadDeletionVector( context, log, true, -<<<<<<< HEAD nullptr, cache.get(), footer_key); -======= - footer); ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) }); } diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.cpp index 3ad4b92a1b90..3ccc4248d9c8 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.cpp @@ -317,39 +317,10 @@ IcebergIterator::IcebergIterator( , callback(std::move(callback_)) , table_schema_id(table_snapshot_->schema_id) { -<<<<<<< HEAD /// Decoding any manifest reads settings from the context, so a missing one is fatal either way. if (!local_context) throw DB::Exception(DB::ErrorCodes::LOGICAL_ERROR, "Context is required to construct IcebergIterator"); -======= - auto delete_file = deletes_iterator.next(); - while (delete_file.has_value()) - { - if (delete_file.value()->parsed_entry->equality_ids.has_value()) - { - equality_deletes_files.emplace_back(std::move(delete_file.value())); - } - else if (delete_file.value()->parsed_entry->isDeletionVector()) - { - deletion_vector_files.emplace_back(std::move(delete_file.value())); - } - else - { - parquet_position_deletes_files.emplace_back(std::move(delete_file.value())); - } - delete_file = deletes_iterator.next(); - } - LOG_DEBUG( - logger, - "Taken {} deletion vector files, {} parquet position delete files and {} equality delete files in iceberg iterator", - deletion_vector_files.size(), - parquet_position_deletes_files.size(), - equality_deletes_files.size()); - std::sort(equality_deletes_files.begin(), equality_deletes_files.end()); - std::sort(deletion_vector_files.begin(), deletion_vector_files.end()); - std::sort(parquet_position_deletes_files.begin(), parquet_position_deletes_files.end()); ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) producer_task = std::make_unique( [this, thread_group = CurrentThread::getGroup()]() { @@ -519,14 +490,6 @@ ObjectInfoPtr IcebergIterator::next(size_t) const auto & data_file_path = object_info->info.data_object_file_path_key; bool has_deletion_vector = false; -<<<<<<< HEAD -======= - object_info->info.requires_external_storage = (storage_to_use != object_storage); - - const auto & data_file_path = object_info->info.data_object_file_path_key; - bool has_deletion_vector = false; - ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) for (const auto & deletion_vector : defineDeletesSpan(manifest_file_entry, deletion_vector_files, /* is_equality_delete */ false, logger)) { @@ -553,7 +516,6 @@ ObjectInfoPtr IcebergIterator::next(size_t) } const auto & parsed_entry = deletion_vector->parsed_entry; -<<<<<<< HEAD /// For icebergCluster, next() runs on the initiator's task-distribution path: DV /// I/O / CRC / roaring materialization happen here, then excluded_rows is sent on @@ -562,20 +524,6 @@ ObjectInfoPtr IcebergIterator::next(size_t) auto excluded_rows = Iceberg::loadDeletionVector( object_storage, persistent_components.path_resolver.resolve(parsed_entry->file_path_key), -======= - const auto puffin_metadata_path = parsed_entry->file_path_key.serialize(); - auto [puffin_storage, puffin_key] = resolveObjectStorageForPath( - persistent_components.table_location, puffin_metadata_path, - object_storage, *secondary_storages, local_context, - persistent_components.path_resolver); - - /// For icebergCluster, next() runs on the initiator's task-distribution path: DV - /// I/O / CRC / roaring materialization happen here, then excluded_rows is sent on - /// the wire per task. Workers apply the bitmap and do not re-read the Puffin blob. - auto excluded_rows = Iceberg::loadDeletionVector( - puffin_storage, - puffin_key, ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) parsed_entry->content_offset.value(), parsed_entry->content_size_in_bytes.value(), data_file_path, @@ -591,11 +539,7 @@ ObjectInfoPtr IcebergIterator::next(size_t) has_deletion_vector = true; LOG_DEBUG( logger, -<<<<<<< HEAD "Attached deletion vector from file `{}` to data file `{}`", -======= - "Attached deletion vector from puffin file `{}` to data file `{}`", ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) parsed_entry->file_path_key, data_file_path); } @@ -636,11 +580,7 @@ ObjectInfoPtr IcebergIterator::next(size_t) lower.has_value() ? lower->serialize() : "[no lower bound]", upper.has_value() ? upper->serialize() : "[no upper bound]"); object_info->addPositionDeleteObject( -<<<<<<< HEAD position_delete, persistent_components.path_resolver.resolve(position_delete->parsed_entry->file_path_key)); -======= - position_delete, position_delete->parsed_entry->file_path_key.serialize()); ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) } } } diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.h index cd49d4bdd9eb..6b27c8a37c81 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergIterator.h @@ -110,11 +110,8 @@ class IcebergIterator : public IObjectIterator ConcurrentBoundedQueue blocking_queue; std::unique_ptr producer_task; IDataLakeMetadata::FileProgressCallback callback; -<<<<<<< HEAD /// Filled once under `deletes_mutex` and never mutated afterwards, so `next` may read them /// unguarded once it has gone through `ensureDeletesReady`. -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) std::vector deletion_vector_files; std::vector parquet_position_deletes_files; std::vector equality_deletes_files; diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp index 47e834a66cd5..3fa4471b3f17 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp @@ -10,10 +10,6 @@ #include #include #include -<<<<<<< HEAD -======= -#include ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) #include #include #include @@ -1314,7 +1310,6 @@ std::optional IcebergMetadata::totalRows(ContextPtr local_context) const && *actual_data_snapshot->total_equality_delete_rows > 0) return {}; -<<<<<<< HEAD /// Row counts stored in the metadata layers above the manifest files are not used as /// data sources, because writers derive them instead of measuring them against the data: /// - the snapshot summary's `total-records` is maintained incrementally (parent total @@ -1330,22 +1325,10 @@ std::optional IcebergMetadata::totalRows(ContextPtr local_context) const /// required field in every format version, so summing it over the live data files is /// exact, at the cost of opening the manifest files (served from the Iceberg metadata /// cache on repeated queries). -======= - /// Do not trust snapshot-summary `total-records` for the answer. Those totals are optional, - /// writer-maintained incrementally, and a single bad commit can poison every later snapshot. - /// Sum required per-data-file `record_count` from manifests when there are no live delete - /// files; otherwise fail closed to a real scan. Summary is compared only for a mismatch warning. - /// - /// Manifest-list `added_rows_count`/`existing_rows_count` are not used (some writers stamp them - /// from snapshot summary and can report 0 after compaction). Subtracting live position-delete / - /// deletion-vector `record_count` from data-file totals is also unsafe (duplicates, stale - /// references, DV supersession of parquet position deletes). ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) UInt64 result = 0; for (const auto & manifest_list_entry : actual_data_snapshot->manifest_list_entries) { auto manifest_file_ptr = getManifestFileEntriesHandle( -<<<<<<< HEAD object_storage, persistent_components, local_context, log, manifest_list_entry, actual_table_state_snapshot.schema_id); /// Live delete files make an exact metadata-only count impossible: @@ -1363,38 +1346,12 @@ std::optional IcebergMetadata::totalRows(ContextPtr local_context) const /// nullopt means a corrupted manifest file with a negative `record_count`: fail /// closed to a real scan instead of returning a wrong count. -======= - object_storage, persistent_components, local_context, log, manifest_list_entry, actual_table_state_snapshot.schema_id, *secondary_storages); - - if (!manifest_file_ptr.getFilesWithoutDeleted(FileContentType::EQUALITY_DELETE).empty() - || !manifest_file_ptr.getFilesWithoutDeleted(FileContentType::POSITION_DELETE).empty()) - return {}; - /// nullopt means a negative / overflowing per-file `record_count`: fail closed to a - /// real scan instead of returning a wrong count. Do not use optional column - /// `value_counts` here — nested fields can report element counts larger than rows. ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) auto manifest_rows = manifest_file_ptr.getRowsCountInAllFilesExcludingDeleted(FileContentType::DATA); if (!manifest_rows.has_value()) return {}; /// Per-manifest sums are capped at Int64::max; still guard the cross-manifest total. if (common::addOverflow(result, static_cast(*manifest_rows), result)) return {}; -<<<<<<< HEAD -======= - } - - if (auto summary_total_rows = actual_data_snapshot->getTotalRows(); - summary_total_rows.has_value() && *summary_total_rows != result) - { - LOG_WARNING( - log, - "Iceberg snapshot summary of table {} claims {} total rows, but its manifest files describe {} rows. " - "The snapshot summary is inconsistent with the table data (possibly a corrupted commit in the table " - "history), using the row count from the manifest files", - persistent_components.table_location, - *summary_total_rows, - result); ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) } const auto summary_total_rows = actual_data_snapshot->getTotalRows(); diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.cpp index 9e70640da7b5..201fdcd3a7eb 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.cpp @@ -92,11 +92,7 @@ void requireDirectReferencedDataFileForPuffinDeletionVector( { throw DB::Exception( DB::ErrorCodes::ICEBERG_SPECIFICATION_VIOLATION, -<<<<<<< HEAD "Deletion vector entry in manifest file '{}' is missing referenced_data_file", -======= - "Puffin deletion vector entry in manifest file '{}' is missing referenced_data_file", ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) manifest_file_path); } } diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.h index 1e9c221ff3a7..7b49fdd76d09 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.h @@ -30,7 +30,6 @@ struct ColumnInfo #include #include -#include namespace DB::Iceberg { @@ -105,24 +104,14 @@ struct ParsedManifestFileEntry : boost::noncopyable Int64 record_count; Int64 file_size_in_bytes; -<<<<<<< HEAD /// Iceberg v3 deletion vector metadata (`content_offset` / `content_size_in_bytes`). /// Present for Puffin containers and for Delta-style `.bin` files that store the same envelope. -======= - /// Iceberg v3 deletion vector metadata (position delete entries with puffin format) ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) std::optional content_offset; std::optional content_size_in_bytes; bool isDeletionVector() const { -<<<<<<< HEAD return content_offset.has_value() && content_size_in_bytes.has_value(); -======= - return Poco::toLower(file_format) == "puffin" - && content_offset.has_value() - && content_size_in_bytes.has_value(); ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) } ParsedManifestFileEntry( @@ -195,14 +184,9 @@ std::optional getRecordCountInAllFilesExcludingDeleted( std::optional getBytesSizeInAllDataFilesExcludingDeleted( const std::vector & files); -<<<<<<< HEAD /// Deletion vectors (Puffin or Delta `.bin`) must identify the data file via the dedicated /// `referenced_data_file` manifest field (non-empty). Position-delete lower/upper bounds /// must not be used as a fallback. -======= -/// Puffin deletion vectors must identify the data file via the dedicated `referenced_data_file` -/// manifest field (non-empty). Position-delete lower/upper bounds must not be used as a fallback. ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) void requireDirectReferencedDataFileForPuffinDeletionVector( bool set_from_referenced_data_file_field, const std::optional & referenced_path, diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp index 96e8353ce2f3..cecfc51c3c4f 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp @@ -88,7 +88,6 @@ bool ManifestFileIterator::ManifestFileEntriesHandle::areAllDataFilesSortedBySor bool ManifestFileIterator::ManifestFileEntriesHandle::areAllDataFilesEligibleForLazyMaterialization(Int32 table_schema_id) const { -<<<<<<< HEAD /// Equality deletes force reading all physical columns of the data files they apply to /// (see IcebergMetadata::getInitialSchemaByPath), so the pruned main read is impossible. if (!equality_delete_files->empty()) @@ -118,8 +117,6 @@ std::optional ManifestFileIterator::ManifestFileEntriesHandle::getRowsCo /// silently produce a wrong -- or, after the conversion to size_t, absurdly huge -- /// count) and rather than rejected (the count is only an optimization, a malformed /// value must not make the table unreadable). -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) return getRecordCountInAllFilesExcludingDeleted(getFilesWithoutDeleted(content)); } diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/Mutations.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/Mutations.cpp index 667115fdfd8d..0d2416b854f9 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/Mutations.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/Mutations.cpp @@ -45,13 +45,8 @@ namespace DB::ErrorCodes extern const int BAD_ARGUMENTS; extern const int LOGICAL_ERROR; extern const int LIMIT_EXCEEDED; -<<<<<<< HEAD extern const int NOT_IMPLEMENTED; extern const int SUPPORT_IS_DISABLED; -======= -extern const int SUPPORT_IS_DISABLED; -extern const int QUERY_WAS_CANCELLED; ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) } namespace DB::DataLakeStorageSetting diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/Snapshot.h b/src/Storages/ObjectStorage/DataLakes/Iceberg/Snapshot.h index 1a6948f5aedc..8ed1dc9b798a 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/Snapshot.h +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/Snapshot.h @@ -16,16 +16,10 @@ struct IcebergDataSnapshot DB::ManifestFileCacheKeys manifest_list_entries; Int64 snapshot_id; Int64 schema_id_on_snapshot_commit; -<<<<<<< HEAD /// Row-count hint from the snapshot summary (`total-records`). Only used to log a /// warning when it disagrees with the row count derived from the manifest files; never /// used as a data source, because the summary is maintained incrementally by writers /// and a corrupted commit in the table history poisons it silently. -======= - /// From snapshot summary (`total-records`). Compared to the manifest-derived count for a - /// mismatch warning only — never used as the trivial COUNT answer. Summary totals are - /// maintained incrementally by writers and can be poisoned by a bad commit in table history. ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) std::optional total_rows; std::optional total_bytes; std::optional total_position_delete_rows; diff --git a/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_count_shortcuts.cpp b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_count_shortcuts.cpp index 2dd69cfc093f..1e00063187ba 100644 --- a/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_count_shortcuts.cpp +++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_count_shortcuts.cpp @@ -124,10 +124,7 @@ ProcessedManifestFileEntryPtr makeDataEntryForRecordCount( /*row_number=*/0, ManifestEntryStatus::ADDED, /*written_sequence_number=*/std::nullopt, -<<<<<<< HEAD /*written_file_sequence_number=*/std::nullopt, -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) /*written_snapshot_id=*/std::nullopt, DB::Row{}, std::move(columns_infos), diff --git a/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.cpp b/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.cpp index e89936c56454..92a11a6457d0 100644 --- a/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.cpp +++ b/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.cpp @@ -8,11 +8,8 @@ #include #include #include -<<<<<<< HEAD #include #include -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) #include #include @@ -112,7 +109,6 @@ void checkDeletionVectorBlobReadLimits(Int64 length, std::optional expec throw Exception(ErrorCodes::BAD_ARGUMENTS, "Deletion vector blob is too small"); } -<<<<<<< HEAD bool isPuffinFileMagic(const UInt8 * header) { return std::memcmp(header, PUFFIN_FILE_MAGIC, sizeof(PUFFIN_FILE_MAGIC)) == 0; @@ -152,28 +148,11 @@ void validateDeletionVectorEnvelope(const UInt8 * header, Int64 length) if (common::addOverflow(static_cast(combined_length), UInt64{8}, expected_blob_size)) throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector combined length: {}", combined_length); -======= -void validateDeletionVectorEnvelope(const UInt8 * header, Int64 length) -{ - const UInt32 combined_length = readBigEndianUInt32(header); - if (std::memcmp(header + sizeof(UInt32), DELETION_VECTOR_MAGIC, sizeof(DELETION_VECTOR_MAGIC)) != 0) - throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector magic"); - - if (combined_length < sizeof(DELETION_VECTOR_MAGIC)) - throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector combined length: {}", combined_length); - - UInt64 expected_blob_size = 0; - if (common::addOverflow(static_cast(combined_length), UInt64{8}, expected_blob_size)) - throw Exception(ErrorCodes::BAD_ARGUMENTS, "Invalid deletion vector combined length: {}", combined_length); - - if (static_cast(length) != expected_blob_size) ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) throw Exception( ErrorCodes::BAD_ARGUMENTS, "Deletion vector blob size {} does not match combined length {}", length, combined_length); -<<<<<<< HEAD } } @@ -250,8 +229,6 @@ IcebergDeletionVectorContainer detectIcebergDeletionVectorContainer( throwUnknownDeletionVectorContainer(path, header, content_offset); UNREACHABLE(); -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) } std::vector deserializeDeletionVectorV1Blob(std::string_view blob_bytes, std::optional expected_cardinality) diff --git a/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.h b/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.h index e4bd365bc0d7..4412015160d1 100644 --- a/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.h +++ b/src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.h @@ -4,18 +4,11 @@ #include #include -<<<<<<< HEAD #include #include #include #include #include -======= -#include -#include -#include -#include ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) namespace DB { @@ -52,7 +45,6 @@ void validatePuffinBlobBounds(Int64 offset, Int64 length, size_t file_size, std: /// Iceberg deletion-vector-v1 envelope magic (`0xD1D33964`), shared with SQL `Puffin` decode. inline constexpr UInt8 DELETION_VECTOR_MAGIC[4] = {0xD1, 0xD3, 0x39, 0x64}; -<<<<<<< HEAD /// Puffin file magic (`PFA1`), shared with SQL `Puffin` decode. inline constexpr UInt8 PUFFIN_FILE_MAGIC[4] = {0x50, 0x46, 0x41, 0x31}; @@ -81,8 +73,6 @@ IcebergDeletionVectorContainer detectIcebergDeletionVectorContainer( Int64 content_size_in_bytes, std::string_view path); -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) /// Fail closed before envelope peek / full allocate. Shared by SQL `Puffin` and Iceberg loaders. /// Order: cardinality ceiling, then length bounds (`length < 0`, absolute blob cap, min envelope). void checkDeletionVectorBlobReadLimits(Int64 length, std::optional expected_cardinality); diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_dv_referenced_data_file.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_dv_referenced_data_file.cpp index 09ed8326e077..855a8ac25f7c 100644 --- a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_dv_referenced_data_file.cpp +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_dv_referenced_data_file.cpp @@ -8,14 +8,11 @@ #include #include -<<<<<<< HEAD #include #include #include #include -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) using namespace DB; using namespace DB::Iceberg; @@ -76,7 +73,6 @@ TEST(PuffinDeletionVectorReferencedDataFile, RejectsEmptyPath) } } -<<<<<<< HEAD namespace { @@ -117,6 +113,4 @@ TEST(IcebergDeletionVectorClassification, OffsetsClassifyAsDeletionVectorRegardl EXPECT_FALSE(makePositionDeleteEntry("parquet", 1, std::nullopt)->isDeletionVector()); } -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) #endif diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_metrics.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_metrics.cpp index 50c0791ebac5..966869851857 100644 --- a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_metrics.cpp +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_files_cache_metrics.cpp @@ -40,13 +40,8 @@ TEST(PuffinFilesCacheMetrics, ClearDuringLoadCountsAsMissNotHit) ASSERT_TRUE(key.has_value()); auto & counters = CurrentThread::getProfileEvents(); -<<<<<<< HEAD const auto hits_before = counters[ProfileEvents::PuffinFilesCacheHits]; const auto misses_before = counters[ProfileEvents::PuffinFilesCacheMisses]; -======= - const auto hits_before = counters[ProfileEvents::PuffinFilesCacheHits].load(); - const auto misses_before = counters[ProfileEvents::PuffinFilesCacheMisses].load(); ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) size_t load_calls = 0; const auto result = cache.getOrSetDeletionVector( @@ -63,13 +58,8 @@ TEST(PuffinFilesCacheMetrics, ClearDuringLoadCountsAsMissNotHit) ASSERT_EQ(load_calls, 1u); ASSERT_NE(result, nullptr); EXPECT_TRUE(result->rb_contains(1)); -<<<<<<< HEAD EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheHits] - hits_before, 0u); EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheMisses] - misses_before, 1u); -======= - EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheHits].load() - hits_before, 0u); - EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheMisses].load() - misses_before, 1u); ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) /// Entry was not inserted after clear; the next lookup must load again. const auto second = cache.getOrSetDeletionVector(*key, [&]() @@ -79,13 +69,8 @@ TEST(PuffinFilesCacheMetrics, ClearDuringLoadCountsAsMissNotHit) }); ASSERT_EQ(load_calls, 2u); ASSERT_NE(second, nullptr); -<<<<<<< HEAD EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheMisses] - misses_before, 2u); EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheHits] - hits_before, 0u); -======= - EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheMisses].load() - misses_before, 2u); - EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheHits].load() - hits_before, 0u); ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) } TEST(PuffinFilesCacheMetrics, WaiterOfClearDiscardedLoadCountsAsMiss) @@ -96,13 +81,8 @@ TEST(PuffinFilesCacheMetrics, WaiterOfClearDiscardedLoadCountsAsMiss) "Local:////test-prefix", "puffin.bin", "etag-waiter", 100, 200, "data/file-w.parquet", 1, 100); ASSERT_TRUE(key.has_value()); -<<<<<<< HEAD const auto hits_before = ProfileEvents::global_counters[ProfileEvents::PuffinFilesCacheHits]; const auto misses_before = ProfileEvents::global_counters[ProfileEvents::PuffinFilesCacheMisses]; -======= - const auto hits_before = ProfileEvents::global_counters[ProfileEvents::PuffinFilesCacheHits].load(); - const auto misses_before = ProfileEvents::global_counters[ProfileEvents::PuffinFilesCacheMisses].load(); ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) std::promise load_started; auto load_started_future = load_started.get_future(); @@ -171,13 +151,8 @@ TEST(PuffinFilesCacheMetrics, OrdinaryHitAndMissCounters) ASSERT_TRUE(key.has_value()); auto & counters = CurrentThread::getProfileEvents(); -<<<<<<< HEAD const auto hits_before = counters[ProfileEvents::PuffinFilesCacheHits]; const auto misses_before = counters[ProfileEvents::PuffinFilesCacheMisses]; -======= - const auto hits_before = counters[ProfileEvents::PuffinFilesCacheHits].load(); - const auto misses_before = counters[ProfileEvents::PuffinFilesCacheMisses].load(); ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) size_t load_calls = 0; auto load_fn = [&]() @@ -190,13 +165,8 @@ TEST(PuffinFilesCacheMetrics, OrdinaryHitAndMissCounters) ASSERT_NE(cache.getOrSetDeletionVector(*key, load_fn), nullptr); EXPECT_EQ(load_calls, 1u); -<<<<<<< HEAD EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheMisses] - misses_before, 1u); EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheHits] - hits_before, 1u); -======= - EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheMisses].load() - misses_before, 1u); - EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheHits].load() - hits_before, 1u); ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) } TEST(PuffinFilesCacheMetrics, HitRemainsHitWhenCacheClearedAfterLookup) @@ -217,7 +187,6 @@ TEST(PuffinFilesCacheMetrics, HitRemainsHitWhenCacheClearedAfterLookup) ASSERT_NE(cache.getOrSetDeletionVector(*key, load_fn), nullptr); -<<<<<<< HEAD const auto hits_before = counters[ProfileEvents::PuffinFilesCacheHits]; const auto misses_before = counters[ProfileEvents::PuffinFilesCacheMisses]; @@ -225,24 +194,10 @@ TEST(PuffinFilesCacheMetrics, HitRemainsHitWhenCacheClearedAfterLookup) EXPECT_EQ(load_calls, 1u); EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheHits] - hits_before, 1u); EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheMisses] - misses_before, 0u); -======= - const auto hits_before = counters[ProfileEvents::PuffinFilesCacheHits].load(); - const auto misses_before = counters[ProfileEvents::PuffinFilesCacheMisses].load(); - - ASSERT_NE(cache.getOrSetDeletionVector(*key, load_fn), nullptr); - EXPECT_EQ(load_calls, 1u); - EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheHits].load() - hits_before, 1u); - EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheMisses].load() - misses_before, 0u); ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) /// Clearing after the hit must not rewrite the already-recorded hit as a miss. The old /// contains()-after-getOrSet path could race here with SYSTEM DROP PUFFIN FILES CACHE. cache.clear(); -<<<<<<< HEAD EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheHits] - hits_before, 1u); EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheMisses] - misses_before, 0u); -======= - EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheHits].load() - hits_before, 1u); - EXPECT_EQ(counters[ProfileEvents::PuffinFilesCacheMisses].load() - misses_before, 0u); ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) } diff --git a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_footer_cache.cpp b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_footer_cache.cpp index 4d1e261ad5b0..cd7ea18fd553 100644 --- a/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_footer_cache.cpp +++ b/src/Storages/ObjectStorage/DataLakes/tests/gtest_puffin_footer_cache.cpp @@ -84,11 +84,7 @@ TEST(PuffinFooterMemo, CoalescedSlicesShareOneFooterParse) ASSERT_TRUE(key_b.has_value()); auto & counters = ProfileEvents::global_counters; -<<<<<<< HEAD const auto files_read_before = counters[ProfileEvents::PuffinFilesRead]; -======= - const auto files_read_before = counters[ProfileEvents::PuffinFilesRead].load(); ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) size_t footer_loads = 0; auto load_footer = [&]() @@ -126,11 +122,7 @@ TEST(PuffinFooterMemo, CoalescedSlicesShareOneFooterParse) EXPECT_EQ(footer_loads, 1u); /// One footer parse (`PuffinFilesRead` in readPuffinFooter) plus two blob reads. -<<<<<<< HEAD EXPECT_EQ(counters[ProfileEvents::PuffinFilesRead] - files_read_before, 3u); -======= - EXPECT_EQ(counters[ProfileEvents::PuffinFilesRead].load() - files_read_before, 3u); ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) } TEST(PuffinFooterMemo, ClearDropsFooterEntries) diff --git a/src/Storages/ObjectStorage/IObjectIterator.h b/src/Storages/ObjectStorage/IObjectIterator.h index b96530cbc631..467474e9d8e3 100644 --- a/src/Storages/ObjectStorage/IObjectIterator.h +++ b/src/Storages/ObjectStorage/IObjectIterator.h @@ -55,13 +55,10 @@ struct ObjectInfo FileBucketInfoPtr file_bucket_info; -<<<<<<< HEAD /// Lazy materialization: if set, read only these rows of the file. /// Sorted absolute row indexes within the file, see FormatFilterInfo::rows_to_read. std::shared_ptr> rows_to_read; -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) /// Polymorphic copy: preserves the dynamic type (e.g. `IcebergDataObjectInfo` with equality / /// position deletes, resolved storage, and metadata path) where a plain copy would slice it. virtual std::shared_ptr clone() const { return std::make_shared(*this); } diff --git a/src/Storages/ObjectStorage/StorageObjectStorageSource.cpp b/src/Storages/ObjectStorage/StorageObjectStorageSource.cpp index 48f2d3c8e1ef..1bed63dba006 100644 --- a/src/Storages/ObjectStorage/StorageObjectStorageSource.cpp +++ b/src/Storages/ObjectStorage/StorageObjectStorageSource.cpp @@ -242,7 +242,6 @@ static bool canUseCountFromFilesCache(const ObjectInfoPtr & object_info) return true; } -<<<<<<< HEAD static bool readsIdentityPartitionColumn( const ActionsDAG & dag, const std::vector> & identity_partition_columns) { @@ -307,8 +306,6 @@ static std::optional buildIdentityPartitionColumnsDag( return dag; } -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) StorageObjectStorageSource::StorageObjectStorageSource( const StorageID & storage_id_, String name_, @@ -930,14 +927,9 @@ Chunk StorageObjectStorageSource::generate() /// reduce rows seen by generate(), while the cache key is file identity only. if (reader.getInputFormat() && read_context->getSettingsRef()[Setting::use_cache_for_count_from_files] && format_filter_info && !format_filter_info->hasFilter() -<<<<<<< HEAD && canUseCountFromFilesCache(reader.getObjectInfo()) && !reader.getObjectInfo()->rows_to_read) addNumRowsToCache(*reader.getObjectInfo(), total_rows_in_file); -======= - && canUseCountFromFilesCache(reader.getObjectInfo())) - addNumRowsToCache(reader.getObjectInfo(), total_rows_in_file); ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) total_rows_in_file = 0; chassert(reader_future.valid()); @@ -1295,11 +1287,7 @@ StorageObjectStorageSource::ReaderHolder StorageObjectStorageSource::createReade const bool effective_need_only_count = need_only_count; #endif -<<<<<<< HEAD const bool can_use_count_cache = effective_need_only_count && !headers_requested -======= - const bool can_use_count_cache = effective_need_only_count ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) && context_->getSettingsRef()[Setting::use_cache_for_count_from_files] && canUseCountFromFilesCache(object_info); @@ -1589,7 +1577,6 @@ StorageObjectStorageSource::ReaderHolder StorageObjectStorageSource::createReade builder.init(Pipe(input_format)); -<<<<<<< HEAD if (!identity_partition_columns.empty()) { if (auto dag = buildIdentityPartitionColumnsDag(builder.getHeader(), identity_partition_columns)) @@ -1602,8 +1589,6 @@ StorageObjectStorageSource::ReaderHolder StorageObjectStorageSource::createReade } } -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) /// Deletion vectors (and selection vectors) address absolute file row numbers via /// `ChunkInfoRowNumbers`. Iceberg equality deletes use a plain `FilterTransform` that /// shrinks the chunk without maintaining `applied_filter`, so DV must run first — diff --git a/src/Storages/ObjectStorage/tests/gtest_rendezvous_hashing.cpp b/src/Storages/ObjectStorage/tests/gtest_rendezvous_hashing.cpp index 1fbc9ec45b10..1c4216c61329 100644 --- a/src/Storages/ObjectStorage/tests/gtest_rendezvous_hashing.cpp +++ b/src/Storages/ObjectStorage/tests/gtest_rendezvous_hashing.cpp @@ -1,12 +1,8 @@ #include -<<<<<<< HEAD #include #include #include #include -======= - ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) #include #include diff --git a/tests/integration/test_storage_iceberg_with_spark/test_deletion_vectors.py b/tests/integration/test_storage_iceberg_with_spark/test_deletion_vectors.py index 3013eeae10d6..1d5fd3e64d63 100644 --- a/tests/integration/test_storage_iceberg_with_spark/test_deletion_vectors.py +++ b/tests/integration/test_storage_iceberg_with_spark/test_deletion_vectors.py @@ -1,8 +1,5 @@ -<<<<<<< HEAD import json import os -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) import uuid import pytest @@ -45,7 +42,6 @@ def _strip_file_uri_scheme(path): return path -<<<<<<< HEAD PUFFIN_MAGIC = b"PFA1" PUFFIN_FOOTER_TRAILER_SIZE = 12 ICEBERG_WAREHOUSE = "/var/lib/clickhouse/user_files/iceberg_data" @@ -263,8 +259,6 @@ def convert_spark_puffin_dvs_to_delta_bin(table_name): return len(conversions) -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) def add_equality_deletes_by_id(spark, table_name, ids): """Commit an Iceberg equality-delete file for the given `id` values. @@ -370,7 +364,6 @@ def test_deletion_vectors(started_cluster_iceberg_with_spark, storage_type, run_ @pytest.mark.parametrize("run_on_cluster", [False, True]) @pytest.mark.parametrize("storage_type", ["s3", "azure", "local"]) -<<<<<<< HEAD def test_deletion_vectors_delta_bin(started_cluster_iceberg_with_spark, storage_type, run_on_cluster): """Iceberg v3 DVs stored as Delta `.bin` files (Databricks UniForm), not Puffin. @@ -433,8 +426,6 @@ def test_deletion_vectors_delta_bin(started_cluster_iceberg_with_spark, storage_ @pytest.mark.parametrize("run_on_cluster", [False, True]) @pytest.mark.parametrize("storage_type", ["s3", "azure", "local"]) -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) def test_deletion_vectors_aggregates(started_cluster_iceberg_with_spark, storage_type, run_on_cluster): """Aggregates over Iceberg v3 tables must ignore rows covered by deletion vectors.""" if storage_type == "local" and run_on_cluster: diff --git a/tests/queries/0_stateless/01271_show_privileges.reference b/tests/queries/0_stateless/01271_show_privileges.reference index bca6cea7a73f..5f16f7ce7a47 100644 --- a/tests/queries/0_stateless/01271_show_privileges.reference +++ b/tests/queries/0_stateless/01271_show_privileges.reference @@ -137,10 +137,7 @@ SYSTEM DROP ICEBERG METADATA CACHE ['SYSTEM CLEAR ICEBERG_METADATA_CACHE','SYSTE SYSTEM DROP PAIMON METADATA CACHE ['SYSTEM CLEAR PAIMON_METADATA_CACHE','SYSTEM DROP PAIMON_METADATA_CACHE'] GLOBAL SYSTEM DROP CACHE SYSTEM DROP AVRO SCHEMA CACHE ['SYSTEM CLEAR AVRO SCHEMA CACHE','SYSTEM DROP AVRO SCHEMA CACHE','DROP AVRO SCHEMA CACHE'] GLOBAL SYSTEM DROP CACHE SYSTEM DROP PARQUET METADATA CACHE ['SYSTEM DROP PARQUET_METADATA_CACHE'] GLOBAL SYSTEM DROP CACHE -<<<<<<< HEAD SYSTEM DROP POINT IN POLYGON CACHE ['SYSTEM CLEAR POINT IN POLYGON CACHE','SYSTEM DROP POINT IN POLYGON CACHE','DROP POINT IN POLYGON CACHE'] GLOBAL SYSTEM DROP CACHE -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) SYSTEM DROP PUFFIN FILES CACHE ['SYSTEM DROP PUFFIN_FILES_CACHE'] GLOBAL SYSTEM DROP CACHE SYSTEM PREWARM PRIMARY INDEX CACHE ['SYSTEM PREWARM PRIMARY INDEX','PREWARM PRIMARY INDEX CACHE','PREWARM PRIMARY INDEX'] GLOBAL SYSTEM DROP CACHE SYSTEM DROP PRIMARY INDEX CACHE ['SYSTEM CLEAR PRIMARY INDEX CACHE','SYSTEM DROP PRIMARY INDEX','DROP PRIMARY INDEX CACHE','DROP PRIMARY INDEX'] GLOBAL SYSTEM DROP CACHE diff --git a/tests/queries/0_stateless/04117_parser_system_query_variants.reference b/tests/queries/0_stateless/04117_parser_system_query_variants.reference index 39ada1de3f2a..197276c6dac6 100644 --- a/tests/queries/0_stateless/04117_parser_system_query_variants.reference +++ b/tests/queries/0_stateless/04117_parser_system_query_variants.reference @@ -16,10 +16,7 @@ SYSTEM CLEAR ENCRYPTION HEADERS CACHE SYSTEM CLEAR COMPILED EXPRESSION CACHE SYSTEM CLEAR ICEBERG METADATA CACHE SYSTEM CLEAR PARQUET METADATA CACHE -<<<<<<< HEAD SYSTEM CLEAR POINT IN POLYGON CACHE -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) SYSTEM CLEAR PUFFIN FILES CACHE SYSTEM CLEAR PUFFIN FILES CACHE SYSTEM CLEAR FILESYSTEM CACHE diff --git a/tests/queries/0_stateless/04117_parser_system_query_variants.sql b/tests/queries/0_stateless/04117_parser_system_query_variants.sql index 749b90faca20..9093c599264c 100644 --- a/tests/queries/0_stateless/04117_parser_system_query_variants.sql +++ b/tests/queries/0_stateless/04117_parser_system_query_variants.sql @@ -26,10 +26,7 @@ EXPLAIN SYNTAX SYSTEM DROP ENCRYPTION HEADERS CACHE; EXPLAIN SYNTAX SYSTEM DROP COMPILED EXPRESSION CACHE; EXPLAIN SYNTAX SYSTEM DROP ICEBERG METADATA CACHE; EXPLAIN SYNTAX SYSTEM DROP PARQUET METADATA CACHE; -<<<<<<< HEAD EXPLAIN SYNTAX SYSTEM DROP POINT IN POLYGON CACHE; -======= ->>>>>>> 4b7cecaa3cf (Merge pull request #2183 from Altinity/feature/antalya-26.6/iceberg-puffin-deletion-vectors-read-2) EXPLAIN SYNTAX SYSTEM DROP PUFFIN FILES CACHE; EXPLAIN SYNTAX SYSTEM DROP PUFFIN_FILES_CACHE; EXPLAIN SYNTAX SYSTEM DROP FILESYSTEM CACHE;