Skip to content

feat: retention-window garbage collection — soft-delete GC + expired-share sweep + purge endpoint (#65 core)#106

Closed
ravirajsinh45 wants to merge 19 commits into
mainfrom
feat/retention-window-gc
Closed

feat: retention-window garbage collection — soft-delete GC + expired-share sweep + purge endpoint (#65 core)#106
ravirajsinh45 wants to merge 19 commits into
mainfrom
feat/retention-window-gc

Conversation

@ravirajsinh45

Copy link
Copy Markdown
Contributor

Summary

A retention-window garbage collector that reclaims storage from soft-deleted data — the core of #65, building on the upload-orphan reaper (#101). A daily job hard-deletes rows soft-deleted longer than SOFT_DELETE_RETENTION_DAYS (default 30, 0 disables) and deletes their S3 objects, cascading the full Project → Folder → Asset → Version → MediaFile / Comment / Approval / ShareLink tree; long-expired share links are swept into soft-delete first; and a superadmin POST /admin/purge triggers a run on demand. Backend only, no DB migration (reuses existing deleted_at).

Changes

  • Cascade GC in tasks/cleanup_tasks.py: _purge_project/_folder/_asset/_version/_comment/_share_link walk the tree child-first (there are no ORM relationships / ondelete=CASCADE, so ordering is explicit), reclaiming S3 (delete_object / delete_prefix) best-effort via _safe, deleting rows via bulk .delete(synchronize_session=False).
  • _purge_soft_deleted processes aged soft-deleted roots top-down; _expire_share_links soft-deletes links expired beyond the retention window (recently-expired stay owner-editable; 410 read-time enforcement already existed and is now regression-tested).
  • SOFT_DELETE_RETENTION_DAYS config (default 30, <=0 disables — same convention as the reaper), documented in .env.example.
  • Daily cleanup_soft_deleted Celery beat task (03:00 UTC) = expire-sweep + purge; the sole commit boundary is the task wrapper (helpers only flush()).
  • POST /admin/purge (superadmin-only): enqueues the same task and returns 202 (async — never blocks the request on a long cascade); reclaimed counts are logged by the worker.
  • Tests: real-Postgres cascade coverage (full tree, retention-window boundary, standalone reaper-version-on-live-asset, disabled path), share-link expiry sweep + 410 regression lock, admin-purge 403/202, and a metadata-driven FK-completeness guard that fails CI if a future schema adds an inbound FK the cascade doesn't handle.

Deferred (tracked follow-ups, not in this PR): the S3 orphan sweeper (list bucket keys, delete those with no DB row — the riskiest slice; deserves a report-only-first posture and its own review) and CollectionShare.expires_at (no consumption endpoint exists).

Verified invariants (independently checked in review): never deletes live data (descends only from aged soft-deleted roots); FK-safe against real Postgres (every inbound FK covered — now guarded by a test); 0 disables before any cutoff is computed; S3 reclaimed before the row; storage-usage accounting is unchanged (committed-sum already excludes soft-deleted).

Part of #65.

Testing

  • Backend tests pass — 19 new GC tests pass; zero new regressions. (The suite also shows ~26 pre-existing SetupGuardMiddleware 503 failures on unrelated auth-gated client tests — present on main; CI's gate is a >=40 passed floor. The new GC tests are DB-state-independent / bypass the guard.)
  • Frontend builds — N/A (backend only)
  • Tested manually in browser — N/A (backend only)

@ravirajsinh45

Copy link
Copy Markdown
Contributor Author

Superseded by #114 — the full #65 GC stack is consolidated into a single release-candidate branch (integration/gc-65) for one combined adversarial review + CI gate. Closing in favor of #114; the fixes from the integration review live there.

@ravirajsinh45 ravirajsinh45 deleted the feat/retention-window-gc branch July 9, 2026 09:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant