Summary
AutoRebalancerService adds jobs to the portfolio-check queue without a deduplication key. A manual forceCheck() and the 30-minute repeatable scheduler can coexist in the queue simultaneously, causing concurrent workers to pick up two jobs for the same portfolio and attempt overlapping rebalances:
// backend/src/services/autoRebalancer.ts — forceCheck()
await queue.add('force-portfolio-check', { triggeredBy: 'manual' }, { priority: 1 })
// startup in start()
await queue.add('startup-portfolio-check', { triggeredBy: 'startup' }, { priority: 1 })Neither job carries a BullMQ jobId (deduplication key). BullMQ's default behaviour is to enqueue all jobs unconditionally, so N calls to forceCheck() produce N independent jobs.
Impact
1. Concurrent Stellar transaction submission race
Two workers processing the same portfolio simultaneously both call StellarService to build and submit a rebalance transaction. Stellar's sequence number mechanism means one will succeed and one will return tx_bad_seq — but both workers may already have partially updated off-chain state (rebalance history, WebSocket push), leaving the backend and contract in an inconsistent state.
2. On-chain cooldown bypass through job timing
Worker A processes portfolio 42 at T=0, succeeds, sets last_rebalance = T. Worker B, delayed in the queue, processes the same portfolio at T=60 seconds. The contract's execute_rebalance checks current_time < last_rebalance + 3600 — this rejects Worker B correctly on-chain, but Worker B has already consumed circuit-breaker budget, written a failed rebalance history entry, and potentially sent a webhook notification for a rebalance that never happened.
3. MAX_AUTO_REBALANCES_PER_DAY counter drift
The service declares MAX_AUTO_REBALANCES_PER_DAY = 3 but enforces it nowhere in the queue layer. Duplicate jobs can trigger more than 3 rebalances per portfolio per day, violating the documented safety limit.
Steps to Reproduce
- Start with Redis + ≥ 2 BullMQ workers.
- Call
POST /api/rebalancer/force-check three times in rapid succession.
- Observe: three
force-portfolio-check jobs in the queue, all processed concurrently for the same portfolio set.
- Check rebalance history — multiple entries for the same portfolio within seconds of each other.
Root Cause
BullMQ supports job-level deduplication via a stable jobId. Without it, every queue.add() call creates a unique job regardless of payload similarity.
Suggested Fix
Job-level deduplication
// forceCheck()
await queue.add(
'force-portfolio-check',
{ triggeredBy: 'manual' },
{
priority: 1,
jobId: 'force-check-singleton', // BullMQ deduplicates by jobId
}
)Per-portfolio locks in the worker
// In the worker processor
const lockKey = `rebalance-lock:${portfolioId}`
const acquired = await redis.set(lockKey, '1', 'NX', 'EX', 60)
if (!acquired) {
logger.warn(`[WORKER] Skipping ${portfolioId} — rebalance already in progress`)
return
}
// ... process rebalance
await redis.del(lockKey)Enforce MAX_AUTO_REBALANCES_PER_DAY in the worker
Check rebalance history count before processing; skip and log if limit is reached.
References
backend/src/services/autoRebalancer.ts — start(), forceCheck(), MAX_AUTO_REBALANCES_PER_DAYcontracts/src/lib.rs — execute_rebalance cooldown check (current_time < last_rebalance + 3600)backend/src/api/routes.ts — POST /rebalancer/force-check endpoint
Severity: High — concurrent workers submit duplicate on-chain transactions and corrupt off-chain rebalance history
Summary
AutoRebalancerServiceadds jobs to theportfolio-checkqueue without a deduplication key. A manualforceCheck()and the 30-minute repeatable scheduler can coexist in the queue simultaneously, causing concurrent workers to pick up two jobs for the same portfolio and attempt overlapping rebalances:Neither job carries a BullMQ
jobId(deduplication key). BullMQ's default behaviour is to enqueue all jobs unconditionally, soNcalls toforceCheck()produceNindependent jobs.Impact
1. Concurrent Stellar transaction submission race
Two workers processing the same portfolio simultaneously both call
StellarServiceto build and submit a rebalance transaction. Stellar's sequence number mechanism means one will succeed and one will returntx_bad_seq— but both workers may already have partially updated off-chain state (rebalance history, WebSocket push), leaving the backend and contract in an inconsistent state.2. On-chain cooldown bypass through job timing
Worker A processes portfolio 42 at T=0, succeeds, sets
last_rebalance = T. Worker B, delayed in the queue, processes the same portfolio at T=60 seconds. The contract'sexecute_rebalancecheckscurrent_time < last_rebalance + 3600— this rejects Worker B correctly on-chain, but Worker B has already consumed circuit-breaker budget, written a failed rebalance history entry, and potentially sent a webhook notification for a rebalance that never happened.3. MAX_AUTO_REBALANCES_PER_DAY counter drift
The service declares
MAX_AUTO_REBALANCES_PER_DAY = 3but enforces it nowhere in the queue layer. Duplicate jobs can trigger more than 3 rebalances per portfolio per day, violating the documented safety limit.Steps to Reproduce
POST /api/rebalancer/force-checkthree times in rapid succession.force-portfolio-checkjobs in the queue, all processed concurrently for the same portfolio set.Root Cause
BullMQ supports job-level deduplication via a stable
jobId. Without it, everyqueue.add()call creates a unique job regardless of payload similarity.Suggested Fix
Job-level deduplication
Per-portfolio locks in the worker
Enforce MAX_AUTO_REBALANCES_PER_DAY in the worker
Check rebalance history count before processing; skip and log if limit is reached.
References
backend/src/services/autoRebalancer.ts—start(),forceCheck(),MAX_AUTO_REBALANCES_PER_DAYcontracts/src/lib.rs—execute_rebalancecooldown check (current_time < last_rebalance + 3600)backend/src/api/routes.ts—POST /rebalancer/force-checkendpointSeverity: High — concurrent workers submit duplicate on-chain transactions and corrupt off-chain rebalance history