Overview
Build backend/jobs/ to run background work including historical backfills. Design idempotency and poison-message handling in from the start: a job that fails partway through needs to be safely retryable without double-applying what it already committed, and a job whose input will fail every retry needs to stop and surface for investigation rather than loop forever.
Why this is hard
Idempotency is a property of the combination of a job's logic and its checkpointing, not something bolted on generically at the queue level — a backfill job that inserts rows will double-insert on retry unless every insert is an upsert. Poison-message handling additionally requires distinguishing "failed because of a transient issue" from "will fail every single time", which generic retry-with-backoff can't tell apart without an explicit error taxonomy.
What to build
Suggested layout:
backend/jobs/
mod.rs
error.rs # explicit retryable vs. terminal error taxonomy
scheduler.rs # retry-with-backoff for retryable, immediate quarantine for terminal
dead_letter.rs # quarantine state + full failure history + alert
backfill.rs # example job built on this, upsert-based
backend/tests/
partial_failure_retry_test.rs
poison_message_test.rs
Implementation steps:
- Build every job's writes as upserts, confirmed safe to repeat, never a blind append or increment.
- Define the retryable/terminal error taxonomy in
error.rs that job handlers return.
- Build
dead_letter.rs: after N retries or a terminal error, quarantine with full failure history and alert, rather than silently dropping or infinitely retrying.
- Write
partial_failure_retry_test.rs (kill a job at partial completion, re-run, assert no duplication) and poison_message_test.rs (malformed input, assert quarantine within the expected retry ceiling).
Acceptance criteria
- Every job type is confirmed idempotent.
- A dead-letter/quarantine path exists with alerting.
- Both tests above pass.
Overview
Build
backend/jobs/to run background work including historical backfills. Design idempotency and poison-message handling in from the start: a job that fails partway through needs to be safely retryable without double-applying what it already committed, and a job whose input will fail every retry needs to stop and surface for investigation rather than loop forever.Why this is hard
Idempotency is a property of the combination of a job's logic and its checkpointing, not something bolted on generically at the queue level — a backfill job that inserts rows will double-insert on retry unless every insert is an upsert. Poison-message handling additionally requires distinguishing "failed because of a transient issue" from "will fail every single time", which generic retry-with-backoff can't tell apart without an explicit error taxonomy.
What to build
Suggested layout:
Implementation steps:
error.rsthat job handlers return.dead_letter.rs: after N retries or a terminal error, quarantine with full failure history and alert, rather than silently dropping or infinitely retrying.partial_failure_retry_test.rs(kill a job at partial completion, re-run, assert no duplication) andpoison_message_test.rs(malformed input, assert quarantine within the expected retry ceiling).Acceptance criteria