[hard] Add dead-letter queue for failed policy deployment jobs

[davedumto]

Summary

Policy deployment jobs that repeatedly fail are currently retried indefinitely with no dead-letter handling.

Requirements

• Route jobs exceeding max retry count to a dead-letter queue
• Add an admin endpoint to inspect and requeue dead-lettered jobs
• Add tests covering the retry-then-deadletter path
• Add alerting when the dead-letter queue depth exceeds a threshold

[felladaniel36-hash]

@felladaniel36-hash has applied to work on this issue as part of the Stellar Wave Program's 8th wave.

Will like to work on this please sir, won't disappoint the team

ℹ️ Repo Maintainers: To accept this application, review their application or assign @felladaniel36-hash to this issue.

[onyemaechiezekiel9]

@onyemaechiezekiel9 has applied to work on this issue as part of the Stellar Wave Program's 8th wave.

Hello boss I'd love to work on this issue

ℹ️ Repo Maintainers: To accept this application, review their application or assign @onyemaechiezekiel9 to this issue.

[drips-wave]

Congratulations, @onyemaechiezekiel9! 🎉 Your application was accepted by the repo's maintainers, and the issue is due on August 31, 2026.

🧑‍💻 @onyemaechiezekiel9: Please resolve the issue such that the repo's maintainers have enough time to review your contribution before the due date. You'll earn Points for completing the issue on-time, which will make you eligible for a share of the Stellar Wave Program's reward pool.

Warning

When opening a PR, please link it to this issue to ensure it gets tracked accurately. Points are awarded when this issue is marked as completed by the maintainer.

🤠 Repo maintainers: Please keep an eye on the contributor's progress and review their work before the due date. You can manage this issue, including adjusting its complexity and points, here.

🌊 Happy Wave 🌊

[drips-wave]

This issue has been marked as completed by a Drips Wave moderator for @onyemaechiezekiel9 as part of the Stellar Wave Program's 8th Wave 🥳

Moderator's reason: [Auto-assessed] The PR was successfully merged by the maintainers (mergedAt: 2026-08-28T21:00:49Z), and it comprehensively addresses Issue #290 by implementing the dead-letter queue (DLQ) store, admin API endpoints, retry logic with backoff, monitoring dashboards, alert rules, and complete test suites.

😎 @onyemaechiezekiel9: You earned 200 Points for completing this issue! After the current Wave ends, you'll be eligible for a percentage of the Wave's reward pool based on the percentage of total points you've earned. Learn more here.

🧑‍💻 Repo maintainers: How'd the contributor do? Leave a review to share your experience working with them.

ℹ️ Note: This issue was marked complete via moderation. Points were issued even though the GitHub issue remains open.