CAS: answer directory probes inside a part from the part manifest, no LIST per part file

[filimonov]

On a cas disk, a directory probe on a path inside a part (<table>/<part>/<file>) was routed to the table-subdirectory branch, which runs an S3 LIST of the table's _files/ prefix. MergeTree makes this probe for every checksum entry of every part while it loads (MergeTreeDataPartChecksum::checkSize), so one restart of a 1,672-part server issued 77k LISTs in three minutes, got 537 503 Slow Down and failed 139 uploads.

This PR gives such a path its own shape, PartFile, and answers it from the part's folder manifest: a plain file is not a directory, a nested directory is. No LIST is issued when the part resolves. When it does not resolve (a table-level subdirectory that happens to look like a part), the old branch runs unchanged.

Measured on ATTACH TABLE: CASRootList was 105 for 20 parts and 1005 for 200 parts; it is now 5 for both.

One visible change: a nested non-projection directory inside a part now reports present, its children and non-empty. Before, it reported absent and empty. Projection directories keep their answers.

Tests: routing cases in gtest_ca_wiring, a new gtest_cas_directory_probes (zero LIST for resolved parts, exact one-LIST oracle for unresolved paths, a failed manifest read throws and never lists, a 50-part load profile) and the stateless test 05053 (DETACH/ATTACH of 20 and 200 parts, equal LIST counts read from the ATTACH query's own ProfileEvents).

Closes: #2439

Changelog category (leave one):

• Performance Improvement

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

cas disk: loading a table no longer issues one object-store LIST per part file. Directory probes inside a part are answered from the part manifest, so a restart costs a fixed number of LISTs per table instead of one per file.

Documentation entry for user-facing changes

• Documentation is written (mandatory for new features)

[github-actions]

Workflow [PR], commit [58f8458]

[alsugiliazova]

PR #2440 Distributed-Systems Audit

AI audit note: This review was generated by AI (GPT-5.6 Sol).

Confirmed defects

Timing / ordering

Medium: A post-repoint probe can use the previous manifest

Impact. existsDirectory() or listDirectory() can return stale results after a ref has been repointed. A removed nested directory may still appear, or a newly added directory may appear absent.

Anchor.

• src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedMetadataStorage.cpp
  • ContentAddressedMetadataStorage::existsDirectory, PartFile branch at line 1752
  • ContentAddressedMetadataStorage::listDirectory, PartFile branch at line 1943
• src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Parts/PartFolderAccess.cpp
  • CachedPartFolderAccess::getView, lines 154–216
  • CachedPartFolderAccess::buildView, lines 235–274

Minimal trigger.

1. Reader A resolves ref R to manifest M1, becomes the cold-build leader, and stalls while reading M1.
2. A writer repoints R to M2 and clears the retained view.
3. Reader B starts after the repoint and resolves R to M2.
4. Reader B joins Reader A's in-flight future because the single-flight map is keyed only by PartRefKey::cacheKey().
5. Reader A completes. Reader B returns and retains the M1 view without comparing it with its resolved M2 ID.

Why this is a defect. Retained-cache hits compare cached->manifestId() with the caller's resolved manifest ID. The single-flight follower path returns future.get() without the equivalent validation. Cache invalidation during the repoint does not cancel or generation-fence an older in-flight build.

Fix direction. Key in-flight builds by (PartRefKey, ManifestId), or compare the completed view with resolved->manifest_id and rebuild on mismatch.

Regression test direction. Block the M1 manifest GET, repoint the ref to M2, start a second directory probe, release M1, and assert that the second probe observes only M2.

Limits

This was a static review of PR head 58f8458c707. No runtime reproduction was performed, and no review comment was posted to GitHub.

[filimonov]

PR #2440 Distributed-Systems Audit

The mechanics are as described: a follower of a cold CachedForLoad build in CachedPartFolderAccess::buildView returns the leader's view without comparing its manifestId with the id the follower resolved. The warm path in getView does compare.

It is not introduced by this PR, so I am not changing it here:

• The single-flight path came with the Parts layer (9d08bc0). This PR does not touch Parts/; it adds two getView(..., CachedForLoad) call sites next to about fifteen existing ones.
• CachedForLoad is documented as stale-tolerant. Reads that need freshness use ForceFresh, which does not coalesce.
• The exposure is the calls that join one in-flight manifest GET. The next access sees the id mismatch and rebuilds.
• For the two probes this PR adds (existsDirectory / listDirectory inside a part) the answer differs only if an in-place repoint changes the directory set of a committed part.

The follower check is cheap (compare and rebuild on mismatch) and is tracked separately, together with the question whether a writer can read its own in-place change through a coalesced build.