[Bug] Webhook Listener: Database initialization fails due to short lock retry window

[SteveJRobertson]

Description

Cold-start schema creation often takes longer than the current 1.5s exponential backoff window, causing secondary PM2 workers to exhaust their retries and crash with "Lock file is already being held" errors.

Current config:

• DB_INIT_MAX_ATTEMPTS = 6
• DB_INIT_INITIAL_BACKOFF_MS = 50
• DB_INIT_MAX_BACKOFF_MS = 1000
  Total window: ~1.5 seconds.

Tasks

1. Increase Retry Window:
   • Update DB_INIT_MAX_ATTEMPTS to 15 or increase the backoff multipliers to ensure the total wait time is at least 20 seconds.
2. Robust Locking:
   • Consider updating the proper-lockfile call to use its internal retries option if it simplifies the code, or simply extend the existing exponential backoff logic.
3. Update Tests:
   • Add/update unit tests to verify the extended retry logic.

PO Refinement (Round 1)

Acceptance Criteria

• All PM2 cluster workers successfully start without fatal "Lock file is already being held" errors during cold starts.
• The total initialization retry window is at least 20 seconds.
• Unit tests confirm the correct number of retries and backoff progression.

Scope Boundary

• IN SCOPE: Updating constants and retry logic in apps/webhook-listener/src/db/schema.ts.
• OUT OF SCOPE: Changing the locking strategy (e.g., no Redis or DB-based locks).
• OUT OF SCOPE: Modifying ecosystem.config.js or PM2 settings.

Nx Projects Affected

• webhook-listener (strictly code/test changes in src/db/).

Dependencies

• None. (Follow-up to [Bug] Webhook Listener startup race condition in PM2 Cluster Mode #120).

Effort Estimate

• XS (Extra Small). Surgical constant update.

Validation Strategy

• Unit Tests: Mock proper-lockfile to simulate a held lock and assert retry count.
• Manual: Deploy to Mac Mini and perform pm2 restart webhook-listener to verify clean cluster boot.