Skip to content

Add cron job worker to delete expired uploads #441

@Syndesi

Description

@Syndesi

When resumable uploads are created, they get an expiration date after which they become obsolete.

This is done in order to prevent abuse by creating many uploads and keeping them alive forever.

While the expiration date is already working, they still block space in the S3 storage bucket and the Upload-nodes themself are not of interest as well. Therefore we should create a cron job, which periodically queries the database for old uploads and deletes both partial uploads from S3 as well as removes the nodes.

Addition: Objects in the upload bucket, which are older than reasonably possible, e.g. more than 24h, should be automatically removed as well.

Acceptance criteria:

  • Add new configuration variable file.expiredUploadCanBeDeletedAfterExpirationInSeconds, with a default value of 1 hour.
  • Create cron job which queries the database for expired uploads.
    • Upload nodes are deleted.
    • Upload artifacts are deleted from S3.
    • Delete objects in the S3 upload bucket older than e.g. 24h (configurable? n times upload lifetime?).
      Note: This should be explicitly documented / warned against.
  • Document feature
  • Add unit tests, where possible
  • Add feature tests

Metadata

Metadata

Assignees

No one assigned

    Labels

    FeatureIntroducing new capabilities.

    Type

    No type

    Projects

    Status

    No status

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions