Skip to content

Make enrichment backlog durable across quota windows #44

Description

@batbrainy

Problem

Enrichment demand can exceed the 40-request hourly allowance. Quota exhaustion must defer
entity work to later windows, not convert delay into a terminal outcome. The previous
skipped_budget and age-out path discarded enrichment intent even though actor and
repository rows already provided a durable work record.

Required behavior

  • Keep pending and retryable_failure entities actionable until enrichment succeeds or
    establishes an entity-specific terminal outcome.
  • Select the oldest backlog row first within each entity class across quota windows.
  • Give durable backlog work priority within the enrichment allowance, with 20/20
    actor/repository guarantees and borrowing.
  • Attempt refreshes only after the never-enriched backlog is observed empty.
  • Remove skipped_budget and migrate historical skipped rows back into actionable states.
  • Publish backlog size, oldest wait, claimability, and allowance usage through /status.
  • Keep polling/control work isolated from quota-paced enrichment queue depth.

Acceptance criteria

  • Quota denial leaves the entity durable and retryable in a later window.
  • Backlog order survives restarts and window rollover.
  • Historical skipped rows are restored without losing prior failure context.
  • Tests cover rollover, FIFO ordering, borrowing, retry/backoff, recovery, queue isolation,
    and status reporting.
  • Documentation describes backlog growth as the capacity limitation rather than dropped
    coverage.

Implementation

Tracked by PR #43.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions