feat: retention-window garbage collection — soft-delete GC + expired-share sweep + purge endpoint (#65 core)#106
Closed
ravirajsinh45 wants to merge 19 commits into
Closed
feat: retention-window garbage collection — soft-delete GC + expired-share sweep + purge endpoint (#65 core)#106ravirajsinh45 wants to merge 19 commits into
ravirajsinh45 wants to merge 19 commits into
Conversation
This was referenced Jul 7, 2026
Closed
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A retention-window garbage collector that reclaims storage from soft-deleted data — the core of #65, building on the upload-orphan reaper (#101). A daily job hard-deletes rows soft-deleted longer than
SOFT_DELETE_RETENTION_DAYS(default30,0disables) and deletes their S3 objects, cascading the fullProject → Folder → Asset → Version → MediaFile / Comment / Approval / ShareLinktree; long-expired share links are swept into soft-delete first; and a superadminPOST /admin/purgetriggers a run on demand. Backend only, no DB migration (reuses existingdeleted_at).Changes
tasks/cleanup_tasks.py:_purge_project/_folder/_asset/_version/_comment/_share_linkwalk the tree child-first (there are no ORM relationships /ondelete=CASCADE, so ordering is explicit), reclaiming S3 (delete_object/delete_prefix) best-effort via_safe, deleting rows via bulk.delete(synchronize_session=False)._purge_soft_deletedprocesses aged soft-deleted roots top-down;_expire_share_linkssoft-deletes links expired beyond the retention window (recently-expired stay owner-editable; 410 read-time enforcement already existed and is now regression-tested).SOFT_DELETE_RETENTION_DAYSconfig (default 30,<=0disables — same convention as the reaper), documented in.env.example.cleanup_soft_deletedCelery beat task (03:00 UTC) = expire-sweep + purge; the sole commit boundary is the task wrapper (helpers onlyflush()).POST /admin/purge(superadmin-only): enqueues the same task and returns202(async — never blocks the request on a long cascade); reclaimed counts are logged by the worker.Deferred (tracked follow-ups, not in this PR): the S3 orphan sweeper (list bucket keys, delete those with no DB row — the riskiest slice; deserves a report-only-first posture and its own review) and
CollectionShare.expires_at(no consumption endpoint exists).Verified invariants (independently checked in review): never deletes live data (descends only from aged soft-deleted roots); FK-safe against real Postgres (every inbound FK covered — now guarded by a test);
0disables before any cutoff is computed; S3 reclaimed before the row; storage-usage accounting is unchanged (committed-sum already excludes soft-deleted).Part of #65.
Testing
SetupGuardMiddleware503 failures on unrelated auth-gatedclienttests — present onmain; CI's gate is a>=40 passedfloor. The new GC tests are DB-state-independent / bypass the guard.)