Skip to content

fix: recover expired Redis task reservations - #280

Open
suguanYang wants to merge 1 commit into
mainfrom
fix/wangbinqi/restore-expired-celery-reservations
Open

fix: recover expired Redis task reservations#280
suguanYang wants to merge 1 commit into
mainfrom
fix/wangbinqi/restore-expired-celery-reservations

Conversation

@suguanYang

Copy link
Copy Markdown
Contributor

Summary

  • add a 30-second Celery Beat recovery task for expired Redis reservations
  • invoke Kombu restore_visible(interval=1) through a fresh broker connection
  • guard duplicate Beat firings and close broker resources safely
  • add worker contract coverage

Evidence

  • staging had two expired reservations in Redis after the configured 4,500-second visibility window
  • an approved one-time Kombu restore requeued both reservations
  • staging then redelivered the parse task and billing idempotency reported Job already charged
  • local Celery 5.5.3/Kombu 5.5.4 gevent drills confirmed the reservation format and recovery behavior

Verification

  • focused worker contracts: 8 passed
  • Ruff: passed
  • git diff --check: passed

This PR is review-only. Do not merge, promote to staging, deploy, or rerun the test job without explicit approval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant