Skip to content

Add backfill script for historical time-series data in scripts/ #49

Description

@christabel888

Context: The new time-series tables (#45-#47) start empty -- dashboards will look broken/sparse until historical data is backfilled from existing ingested ledger/event data.

Where:

  • scripts/ -- new backfill script, following the existing pattern in src/jobs/backfill.rs and scripts/

What to do:

Suggested approach:

  1. Follow src/jobs/backfill.rs's existing pattern for a one-off/re-runnable backfill script.
  2. Backfill Design and migrate time-series schema for daily network metrics #45-Design and migrate time-series schema for wallet balance history #47's tables from whatever historical ledger/contract-event data already exists -- this is bounded by how far back ingestion already goes, it can't manufacture history that was never ingested.
  3. Make it idempotent/resumable (safe to re-run, safe to interrupt) since backfilling real historical volume can take a while.

Watch out for:

  • Running this against production data for the first time is a good candidate for a dry-run mode first (compute and log, don't write) before actually writing rows.

Definition of done:

  • Running the script populates historical data for the new dashboards without requiring a wait period

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions