Make history-index rebuild and schema migration chunked and resumable instead of unbounded single-transaction loops

[Cedarich]

• Complexity: Hard
• Labels: Soroban, contracts, migration, performance, Hard
• Overview: Every migration and recovery routine in the contract is written as an unbounded single-transaction loop over the full payment set. rebuild_payment_history_index makes several complete passes over every payment record and sorts all of them in contract memory; migrate_settlement_refs makes another; and migrate_schema_v0_to_v1 runs both in one call. A Soroban transaction has a hard CPU, memory, and ledger-access budget, so past some payment count these entrypoints stop being callable at all — permanently. The migration path and the corruption-recovery path both fail precisely on the deployments large enough to need them.
• Details:
  • A single rebuild_payment_history_index call performs, in one transaction: is_index_complete → history_entries_exist looping 0..count with a persistent has() per slot; clear_history_index looping 0..count with a remove() per slot; collect_all_payment_records looping 0..payment_count with a get_payment_log_entry plus a get_payment per record; a full in-memory sort; then write_history_index, which calls clear_history_index a second time before writing every record back with a TTL extension each.
  • That is roughly five to six full passes over the payment set plus an allocation holding every PaymentRecord at once. The second clear_history_index inside write_history_index is redundant — rebuild_payment_history_index has already cleared — so a meaningful fraction of the work is pure waste.
  • sort_records_by_timestamp collects every record into an alloc::vec::Vec before sorting. Memory, not just CPU, scales linearly with total payments.
  • migrate_schema_v0_to_v1 calls rebuild_payment_history_index and then migrate_settlement_refs, which independently walks all payments again resolving each through get_payment. The v0-to-v1 migration therefore cannot complete on a large deployment, and there is no partial-progress state — a failed attempt reverts entirely and the next attempt starts over.
  • The doc comment acknowledges the shape ("may be O(n) in the number of payments. For large deployments, consider calling this during maintenance windows") but a maintenance window does not raise the per-transaction resource budget. There is no window in which an over-budget transaction succeeds.
  • upgrade_storage_schema makes this reachable on the ordinary path. When current == target_version it checks is_history_index_consistent and, if inconsistent, calls rebuild_payment_history_index. So upgrade_storage — documented as "Safe to call multiple times — idempotent" — quietly attempts a full unbounded rebuild on every invocation once the index looks inconsistent. Combined with the counter drift described in Define a retention policy and TTL strategy so payment records are not archived out of the audit log #465, a mature deployment reaches a state where every upgrade_storage call attempts an unbounded rebuild that can never succeed.
  • The tests hide this completely: the migration tests in migration.rs use three to five records, so they exercise the logic but never the budget.
• Scope:
  • Restructure the rebuild and migration routines to process a bounded range per call, with persisted progress state so an operator can drive them to completion across multiple transactions.
  • Make partial progress durable and resumable, so an interrupted or over-budget attempt does not discard the work already done.
  • Remove the redundant second clear_history_index inside write_history_index, and eliminate the duplicate full passes where a single pass would do.
  • Avoid holding every payment record in contract memory at once; establish ordering without an all-records in-memory sort, or sort within bounded chunks.
  • Fold the settlement-reference migration into the same chunked, resumable mechanism rather than a second independent full scan.
  • Stop upgrade_storage from silently launching an unbounded rebuild on the already-current path; report the inconsistency and let the operator drive a chunked rebuild explicitly.
  • Expose progress so ops tooling can tell how far a migration has advanced and what remains.
  • Add tests that build a payment history large enough to exceed a single transaction's budget and assert that migration and rebuild still complete across calls.
• Technical scope:
  • soroban/contracts/invoice-payment/src/migration.rs
  • soroban/contracts/invoice-payment/src/migration_helpers.rs
  • soroban/contracts/invoice-payment/src/storage.rs
  • soroban/contracts/invoice-payment/src/lib.rs
  • soroban/contracts/invoice-payment/src/test.rs
  • soroban/client/src/soroban-invoice-client.ts
• Acceptance criteria:
  • A single rebuild or migration call performs a bounded amount of work regardless of total payment count.
  • An operator can complete a full rebuild and a full schema migration on a history far larger than one transaction's budget, by repeated calls.
  • Progress is persisted, so an interrupted run resumes rather than restarting.
  • No routine holds the entire payment set in contract memory at once.
  • The redundant clear pass is removed and each record is read and written the minimum number of times.
  • upgrade_storage no longer triggers an unbounded rebuild implicitly; the inconsistency is reported and the rebuild is driven explicitly.
  • Migration progress and remaining work are observable through a read method and surfaced by the TypeScript client.
  • Tests exercise a history large enough to exceed a single-transaction budget and assert completion across multiple calls.

[DatabyPrisca]

Please can you give me this issue? I have experience in this and I am sure to bring value to this project

[Johnpii1]

Hello 👋
Please can I work on this

[grantfox-oss]

🦊 GrantFox — @Johnpii1 has been assigned to this issue as part of the Third Campaign campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #480)
2. Your PR will be reviewed by the ZyntariHQ maintainers

Good luck! Track your progress on GrantFox.