Skip to content

CAS gc with conditional batch deletes - #2534

Draft
k-morozov wants to merge 16 commits into
antalya-26.6from
cas_batch_deletes
Draft

k-morozov wants to merge 16 commits into
antalya-26.6from
cas_batch_deletes

Conversation

@k-morozov

@k-morozov k-morozov commented Oct 9, 2026 •

Copy link
Copy Markdown

Changelog category (leave one):

  • Performance Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

CAS garbage collection deletes retired blob bodies on S3 with batched DeleteObjects requests that carry a per-object ETag.

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

@k-morozov k-morozov linked an issue Oct 9, 2026 that may be closed by this pull request
@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Workflow [PR], commit [a95fe39]

k-morozov and others added 15 commits October 9, 2026 18:18
Replace the parallel `io_results`, `io_errors` and `delete_submitted`
vectors in `Gc::redeleteBlobs` with one `RedeleteBlobState` per entry.
The observation result (`live_token`) is now separate from the delete
result (`del`), so `performRedeleteIo` becomes `observeRetiredBlob`.

Rename `applyRedeleteOutcome` to `recordRedeleteSuccess` and
`reportRedeleteFailure` to `recordRedeleteFailure`: the delete has
already run when they are called, and `Outcome` already names the
persisted `GcOutcomes` log. Rename the test hook and test names to match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Move the selection of token-matched entries and their split into
conditional delete chunks out of `Gc::redeleteBlobs` into
`Gc::prepareConditionalDeleteChunks`. The second wave now only submits
the prepared chunks, so `matched`, `delete_begin` and `delete_indices`
are gone.

The function and its value types are public so that the new
`CASGCConditionalDeleteChunks` tests can call it without a pool or a
backend request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Move the conditional delete chunk size calculation out of
`Gc::redeleteBlobs` into a method next to `Gc::bulkDeleteChunkKeys`.
No behavior change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add `Gc::runRedeleteIoJobs` and use it for both the observation wave
and the conditional delete wave in `Gc::redeleteBlobs`. It runs jobs
inline when `concurrency <= 1` or there is at most one job, and on the
GC I/O pool otherwise. It returns per-job errors and the enqueue
refusal.

`jobs_scheduled` still counts only observation jobs submitted to the
pool. The delete wave now also runs inline at `concurrency <= 1`, and
both waves use one injected enqueue refusal message.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Split the tail of `Gc::redeleteBlobs`:
- `Gc::recordRedeleteResults` records failures and successes and returns
  the error with the lowest entry index;
- `Gc::scheduleBlobMetaDeletes` schedules `.meta` deletion for `Removed`
  and `Gone` blobs;
- `Gc::redeleteBlobs` rethrows the error after both steps.

`Gc::recordRedeleteSuccess` no longer schedules `.meta` deletion, so a
later batch `.meta` delete changes only `Gc::scheduleBlobMetaDeletes`.
If recording throws, no `.meta` deletion is scheduled for the round; the
next round records the blobs as `Absent` and schedules it then.

Rename `RedeleteBlobState::del` to `removal` (it can come from the
observation wave, not only from a delete) and the wave results to
`observe_jobs` and `delete_jobs`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@k-morozov

Copy link
Copy Markdown
Author

rustfs/rustfs#8518

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CAS: batches for pending_deletes

2 participants