Implement reconciliation service for on-chain/off-chain payment state drift

[devJaja]

Summary

The reconciliation.ts service exists but needs certification that it fully detects and corrects drift between payments DB records and actual Stellar chain state. Scenarios: (1) a payment released on-chain but not marked in DB (crash between submit and DB write), (2) DB says released but no on-chain claim exists, (3) escrow Locked but never released and task expired.

Priority

P1 — Payment state drift can cause double-payments or lost funds.

Files to work on

backend/src/services/reconciliation.ts
backend/src/payment/payment.ts
backend/src/db/payments.ts

Requirements

• Audit reconciliation.ts for correctness against all drift scenarios
• Ensure drift detection covers:
  • DB locked but no on-chain claimable balance found → mark orphaned
  • DB released but no on-chain release tx found → identify missing Horizon record
  • DB released but on-chain shows unreleased → flag for re-submit
  • Expired escrow (task failed, escrow never refunded) → schedule refund
• Implement automatic remediation for unambiguous cases
• Add metrics: reconcile.driftDetected, reconcile.remediated, categorized by type
• Emit RECONCILIATION_EVENT for any drift detected and any remediation
• Add integration tests with a mocked Horizon API simulating each drift scenario
• Ensure remediation is idempotent (no double-refund on re-run)

Acceptance Criteria

• All 4 drift scenarios are detected by the reconciliation run
• Unambiguous remediation is applied automatically
• No drift is remediated twice (idempotent)
• Metrics increment correctly on drift and remediation
• Events emitted for each drift/remediation
• Reconciliation runs on a schedule (default 60s) and on-demand
• Tests cover all 4 drift scenarios

[achebejohnpaul]

Am a full-Stack Engineer focused on frontend, backend, APIs, automation, and modern web technologies. Passionate about solving real-world problems, contributing to open source, and delivering clean, scalable, review-ready solutions.