Distributed Job Queue System

Enterprise-grade distributed job processing platform built with Java 21, Spring Boot 3, PostgreSQL, Redis, and Docker.

Architecture

Client → API Service → PostgreSQL (metadata) + Redis (queue)
                              ↓
                    Worker Service(s) → Job Handlers
                              ↓
                    Dead Letter Queue + Monitoring

• API Service (jobqueue-api) — REST API for job management, auth, stats. Never executes jobs.
• Worker Service (jobqueue-worker) — Consumes from Redis Streams, executes jobs, handles retries/DLQ.
• PostgreSQL — Job metadata, users, workers, attempt history.
• Redis — Main queue (Streams), priority queue (Sorted Sets), delayed jobs, rate limiting, DLQ.

Features

Feature	Implementation
Authentication	JWT + RBAC (USER, ADMIN)
Job Queue	Redis Streams + Sorted Sets
Priority	HIGH(1) → MEDIUM(5) → LOW(10)
Scheduling	Redis Sorted Set with score = timestamp
Retries	Exponential backoff: baseDelay × 2^attempt
Dead Letter Queue	PostgreSQL + Redis Stream
Idempotency	Idempotency-Key header + unique DB constraint
Rate Limiting	Redis token bucket per job type
Worker Heartbeat	Every 5s, inactive after 30s
Crash Recovery	RUNNING jobs > 10 min → retry
Monitoring	Prometheus + Grafana
Real-time Updates	WebSocket /ws/jobs
API Docs	Swagger UI at /swagger-ui.html

Job Types (Strategy Pattern)

• SEND_EMAIL
• GENERATE_REPORT
• RESIZE_IMAGE
• WEBHOOK_DELIVERY
• DATA_CLEANUP

Quick Start

Prerequisites

• Java 21
• Maven 3.9+
• Docker & Docker Compose

Run with Docker Compose

docker-compose up -d

Services:

Service	URL
Frontend UI	http://localhost:3002
API	http://localhost:8080
Swagger UI	http://localhost:8080/swagger-ui.html
Prometheus	http://localhost:9090
Grafana	http://localhost:3001 (admin/admin)

Run Frontend Locally (dev mode)

cd jobqueue-frontend
npm install
npm run dev

Opens at http://localhost:3002 with API proxied to http://localhost:8080.

Run Locally (without Docker)

Start PostgreSQL and Redis, then:

# Terminal 1 — API
mvn spring-boot:run -pl jobqueue-api

# Terminal 2 — Worker
mvn spring-boot:run -pl jobqueue-worker -Dspring-boot.run.arguments=--worker.name=worker-1

Default Admin User

Email: admin@jobqueue.com
Password: admin123

API Examples

Register & Login

# Register
curl -X POST http://localhost:8080/api/auth/register \
  -H "Content-Type: application/json" \
  -d '{"email":"user@example.com","password":"password123"}'

# Login
curl -X POST http://localhost:8080/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"admin@jobqueue.com","password":"admin123"}'

Create a Job

curl -X POST http://localhost:8080/api/jobs \
  -H "Authorization: Bearer <TOKEN>" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: unique-key-123" \
  -d '{
    "type": "SEND_EMAIL",
    "priority": "HIGH",
    "payload": {"to": "user@example.com", "subject": "Hello"}
  }'

Get Statistics (Admin)

curl http://localhost:8080/api/stats \
  -H "Authorization: Bearer <ADMIN_TOKEN>"

Dead Letter Queue (Admin)

# List DLQ
curl http://localhost:8080/api/dlq \
  -H "Authorization: Bearer <ADMIN_TOKEN>"

# Replay a DLQ job
curl -X POST http://localhost:8080/api/dlq/{id}/replay \
  -H "Authorization: Bearer <ADMIN_TOKEN>"

Project Structure

jobqueue-parent/
├── jobqueue-common/     # Shared entities, DTOs, enums, utilities
├── jobqueue-api/        # REST API service (port 8080)
├── jobqueue-worker/     # Worker service (port 8081)
├── jobqueue-frontend/   # React dashboard (port 3002)
├── docker-compose.yml
├── prometheus/
└── grafana/

Testing

# Unit tests
mvn test

# Integration tests (requires Docker for Testcontainers)
mvn verify -pl jobqueue-api

Job Lifecycle

PENDING → QUEUED → RUNNING → COMPLETED
                    ↓
                  FAILED → RETRYING → RUNNING → ...
                    ↓
              DEAD_LETTERED

Resume Highlights

• Built a distributed job processing platform using Java, Spring Boot, PostgreSQL, Redis, and Docker
• Designed async queue architecture with priority scheduling, retries, dead-letter queues, and delayed jobs
• Implemented worker heartbeat, crash recovery, idempotency, JWT auth, and rate limiting
• Added observability with Prometheus, Grafana, structured JSON logging, and WebSocket real-time updates
• Scalable multi-worker architecture capable of handling thousands of concurrent jobs

License

MIT

Deploy on Render

Host the API, worker, frontend, PostgreSQL, and Redis on Render using the included blueprint:

# render.yaml defines all services — apply via Render Dashboard → New → Blueprint

See RENDER_DEPLOY.md for step-by-step instructions.