Skip to content

[RELIABILITY][EXPORTS] Stream paginated report exports instead of materializing entire datasets #182

Description

@Obiajulu-gif

Summary

Report and ledger exports build complete query results and CSV line arrays in memory. Data growth can exhaust serverless memory or time limits and produce partial operator downloads.

Repository evidence

  • app/api/admin/reports/export/route.ts materializes report datasets before calling toCsv.
  • app/api/transactions/ledger/export/route.ts and lib/security/audit-export.ts accumulate full row arrays/strings.

Scope

  • Page through stable sort keys and stream CSV with backpressure.
  • Set explicit row/time budgets and return an async export job for oversized requests if needed.
  • Record export status, row count, checksum, and terminal failure.

Acceptance criteria

  • Peak memory is bounded independently of export row count.
  • Rows are neither duplicated nor skipped under the documented snapshot policy.
  • Interrupted exports fail visibly and never appear complete.

Tests

  • Large-dataset memory/backpressure test.
  • Pagination boundary, concurrent-write snapshot, disconnect, and checksum tests.

Non-goals

  • Adding new report fields or changing CSV formula-safety behavior.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

GrantFox OSSIssue tracked in GrantFox OSSMaybe RewardedIssue may be eligible for a GrantFox rewardThird CampaignCampaign: Third Campaign

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions