Skip to content

Rehmy implementation gf/4 - #460

Merged
Wilfred007 merged 3 commits into
Protocol-Guild:mainfrom
rhemy-arc:Rehmy-implementation-gf/4
Aug 16, 2026
Merged

Rehmy implementation gf/4#460
Wilfred007 merged 3 commits into
Protocol-Guild:mainfrom
rhemy-arc:Rehmy-implementation-gf/4

Conversation

@rhemy-arc

Copy link
Copy Markdown
Contributor

Summary
Adds row-level locking (FOR UPDATE SKIP LOCKED) to the payroll schedule executor to prevent concurrent pods from executing the same schedule simultaneously. Without this fix, every backend pod running the '* * * * *' cron job independently queries and processes the same due schedules, causing duplicate Stellar payments to employees.
Changes

  • Migration 028_schedule_row_locking.sql — Adds locked_by (VARCHAR) and locked_at (TIMESTAMPTZ) columns to schedules table with a partial index for efficient claim queries.
  • scheduleExecutor.ts — Replaces the plain SELECT in processDueSchedules() with an atomic UPDATE ... FROM (SELECT ... FOR UPDATE SKIP LOCKED) claim pattern. Each claimed row is stamped with hostname-pid. Adds releaseClaim() (clears lock after execution) and releaseStaleClaims() (reclaims rows from crashed pods after 5-minute timeout). Updates recordExecution() to clear the lock inside its transaction.
  • scheduleExecutor.test.ts — Updates existing tests for the new claim query pattern and adds a concurrent pod safety test suite covering two-pod race simulation, FOR UPDATE SKIP LOCKED assertion, and stale claim cleanup.
    Testing
  • 12 schedule executor tests passing (npx jest --testPathPatterns=scheduleExecutor.test.ts)
  • Concurrency test simulates two pods (A and B) running processDueSchedules() concurrently against the same due schedule — asserts exactly one execution occurs
  • Existing tests updated to verify BEGIN/UPDATE FOR UPDATE SKIP LOCKED/COMMIT transaction pattern
  • Lock-clear assertions added to recordExecution tests
    Tradeoffs
  • Claim-then-execute vs hold-lock-during-execute: Chose claim-then-execute to avoid holding PostgreSQL row locks during slow Stellar RPC calls. The tradeoff is a brief window where a crash leaves a stale claim — mitigated by the 5-minute stale claim cleanup.
  • locked_by column vs advisory locks: Used column-based locking over pg_advisory_lock because it survives connection pooling (advisory locks are session-scoped) and provides visibility into which pod holds which claim via direct table inspection.
  • 5-minute stale timeout: Balances crash recovery speed against avoiding double-execution during legitimately slow operations. Can be tuned.
    Architecture
    Pod A (cron tick) Pod B (cron tick)
    │ │
    ├─ releaseStaleClaims() ├─ releaseStaleClaims()
    ├─ BEGIN ├─ BEGIN
    ├─ UPDATE ... FOR UPDATE SKIP LOCKED ├─ UPDATE ... FOR UPDATE SKIP LOCKED
    │ → claims schedule 42 │ → skips schedule 42 (locked)
    ├─ COMMIT ├─ COMMIT
    ├─ executeSchedule(42) ├─ (no schedules claimed)
    ├─ recordExecution(42) │
    │ → clears locked_by │
    └─ releaseClaim(42) └─ done
    Out of scope
  • Changes to schedule creation or cron cadence
  • part49Jobs.ts daily jobs (separate concern)
  • idempotencyMiddleware wiring to the background executor path

Closes #448

“rhemy-arc” added 3 commits August 16, 2026 07:50
Adds locked_by and locked_at columns to the schedules table to support
claim-based concurrency control. Includes an index for efficient
lookup of claimable rows (active + due + unlocked).
Replace the plain SELECT in processDueSchedules() with an atomic
UPDATE ... FROM (SELECT ... FOR UPDATE SKIP LOCKED) pattern that marks
each claimed row with the pod's hostname+pid. This prevents two pods
from executing the same schedule simultaneously.

Also adds releaseClaim() to clear locks after execution and
releaseStaleClaims() to reclaim rows from crashed pods (5min timeout).
Update existing processDueSchedules tests to match the new claim query
(BEGIN/UPDATE FOR UPDATE SKIP LOCKED/COMMIT pattern) and add lock-clear
assertions to recordExecution tests.
@Wilfred007
Wilfred007 merged commit 87de719 into Protocol-Guild:main Aug 16, 2026
1 check passed
@Wilfred007

Copy link
Copy Markdown
Contributor

Thank you @rhemy-arc for your contributions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Payroll scheduler has no row locking — duplicate payment execution risk across pod replicas

2 participants