Retry asynchronous table loading after transient CAS mount failures

[pauloschade]

1 — Previous behavior

When a CAS mount lost its lease, table-loading tasks could fail while storage was unavailable. The mount could subsequently recover, but failed or canceled loading tasks retained their errors.

2 — Problem

Tables remained unavailable after the underlying storage recovered. Restoring access required restarting ClickHouse, even though the mount was already healthy.

3 — Fix

Retry table construction during asynchronous loading when it encounters the identified transient CAS mount failure. Keep permanent failures terminal and preserve the existing lease fencing. Allow shutdown to interrupt recovery without waiting indefinitely.

---

Production example

Our deployment uses one writer and two readers backed by shared S3 storage. During an incident, S3 returned 503 Slow Down, and the writer lost its CAS lease. The mount recovered automatically, but table-loading tasks remained failed or canceled until we restarted the affected server. After deploying the patch, another S3 throttling episode caused the writer to lose and recover its lease during startup. All 1,043 loading tasks subsequently completed without another restart, and our query and RLS checks passed.

Changelog category

• Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry

Recover asynchronous table loading after transient CAS mount failures without requiring a server restart.

Documentation entry for user-facing changes

• No user-facing configuration changes; standalone reproduction instructions are included in tests/integration/test_cas_load_recovery/README.md.

Validation

The included Docker Compose reproduction is standalone and is not wired into the upstream pytest runner. On ARM64 Linux, the unmodified build reproduced persistent loading failures after remount; the patched build recovered writer and reader without a recovery restart, preserved rows and RLS, continued rejecting terminal mount states, and shut down during a fault in 3.87 seconds. The six changed C++ files match the deployed and tested patch; this submission excludes our image packaging and deployment scripts.

Published test evidence: https://github.com/use-kayro/ClickHouse/releases/tag/v26.6.4.20001-kayro-cas1-rc2

CI/CD Options

Regression jobs to run:

• CAS (content-addressed storage; Antalya only)

[filimonov]

During an incident, S3 returned 503 Slow Down

This one should be addressed in #2440
(it was released last week: https://github.com/Altinity/ClickHouse/releases/tag/v26.6.8.20001.altinityantalya )

And most probably after that the problem will go away.

CAS mount lost its lease

This one i'm looking on (making the leases more robust).

Regarding the retries in async loader - it may make sense, but we're trying to limit the impact on the upstream code and avoid all kinds of if (cas) ... whereever possible. I'll keep the PR open for now, please let me know if it will still be needed after #2440