Add distributed lock so background sweep workers run as a single active instance

[meshackyaro]

Description

The background sweep workers — GigExpiryWorkerService (src/gig/gig-expiry-worker.service.ts) and RepinWorkerService (src/ipfs-pinning/repin-worker.service.ts) — each register their own process-local setInterval on onModuleInit. In any deployment running more than one instance behind a load balancer, every instance runs its own copy of each sweep independently, so a single expiring gig or degraded IPFS pin gets processed N times, dispatching N duplicate webhooks per event.

This is independent of where the underlying gig/pin data eventually lives (see the persistence spike, #181) — it's specifically about making sure only one instance's timer is ever "active" at a time. The backend already has a Redis client wired up (REDIS_CLIENT in src/common/redis/redis.module.ts) via the rate-limiting module, so the primitive needed (a distributed lock) is already available without adding a new infra dependency.

Component

Backend

Difficulty

🔴 Hard

Tasks

• Implement a Redis-backed lease/heartbeat lock (e.g. SET key value NX PX <ttl> + periodic renewal) shared by both worker services
• Only the instance currently holding the lock should execute runOnce(); other instances should skip their tick
• Handle lock handoff cleanly on graceful shutdown (onModuleDestroy) and on lock-holder crash (TTL expiry)
• Add tests simulating multiple instances racing for the lock, and confirming exactly one wins per tick
• Document the new REDIS_URL-dependency behavior when Redis is unavailable (currently REDIS_CLIENT can be null — decide whether workers should fail closed or fall back to per-instance execution with a warning)

Acceptance Criteria

• Running 3+ instances of the backend against the same Redis only ever executes one runOnce() sweep per interval tick, verified by a test or integration harness
• Lock is released/expires correctly on instance shutdown or crash so the sweep resumes on another instance without manual intervention
• Existing gig-expiry-worker.service.spec.ts / repin-worker.service.spec.ts tests still pass
• Behavior when REDIS_URL is unset is explicitly defined and documented

Estimated Time

2-3 days

[nyuiela]

I'd like to take this.

I looked at backend/src/gig/gig-expiry-worker.service.ts on current HEAD. onModuleInit starts setInterval with no lock, so every replica will sweep findExpirable() on the same cadence. The IPFS re-pin worker has the same shape.

Plan:

• Add a Redis/DB lease so only one instance runs the gig-expiry and re-pin sweeps.
• Skip the tick if the lock is held; expire the lease so a dead replica recovers.
• Cover double-start in tests (second instance does not call findExpirable).

please assign this to me if that works.

[ritachukwuneye]

@ritachukwuneye has applied to work on this issue as part of the Stellar Wave Program's 8th wave.

Hi maintainer, kindly assign this issue to me please.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @ritachukwuneye to this issue.