Describe the bug
On a cluster restart with a previously working and functional table with CAS disk, async loader fails to load the table on one of 2 replicas.
<Error> executeQuery: Code: 722. DB::Exception: Waited job failed: Code: 696. DB::Exception: Load job 'startup table {table}' -> Code: 695. DB::Exception: Load job 'load table {table}' failed: Code: 210. DB::Exception: content-addressed disk 'cas' -- mount lease not held; backing may be temporarily unreachable; the operation is admitted again once the disk recovers to Live; TRANSIENT unavailability, not damage: Cannot attach table {table} from metadata file store/8f8/8f8fb353-d4cd-48d9-bac7-e7b2d4ac2f34/{table}.sql from query ATTACH TABLE {table_definition}. (NETWORK_ERROR)
Note: Table name and definition redacted.
To Reproduce
Don't have consistent steps to reproduce, I restarted a cluster that had CAS table. Issue may be transient as this is not the first time I have restarted the table, although a following node restart did not resolve the issue.
Expected behaviour
CAS table should load on restart without failure.
Screenshots
N/A
Key information
Provide relevant runtime details.
- Altinity Cloud Manger
- Project Antalya Build Version: 26.6.4.20001.altinityantalya
- Cloud provider: AWS
- Object storage: S3
Additional context
Possibly related to #2431. There were many AWS 503 errors leading up to this async load fail error. However, the replica that successfully loaded the table also had the same errors and it loaded the table anyway.
Describe the bug
On a cluster restart with a previously working and functional table with CAS disk, async loader fails to load the table on one of 2 replicas.
Note: Table name and definition redacted.
To Reproduce
Don't have consistent steps to reproduce, I restarted a cluster that had CAS table. Issue may be transient as this is not the first time I have restarted the table, although a following node restart did not resolve the issue.
Expected behaviour
CAS table should load on restart without failure.
Screenshots
N/A
Key information
Provide relevant runtime details.
Additional context
Possibly related to #2431. There were many AWS 503 errors leading up to this async load fail error. However, the replica that successfully loaded the table also had the same errors and it loaded the table anyway.