Skip to content

Commit 206d8bc

Browse files
timan-ztiman-z
committed
Working on documentation. Can't see what these mermaid diagrams look like in VsCode
1 parent 042847d commit 206d8bc

6 files changed

Lines changed: 7054 additions & 15 deletions

File tree

‎README.md‎

Lines changed: 243 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -4,35 +4,124 @@
44
<u>Date</u>: <b>11/30/2025</b> (core backend system, *frontend dashboard finished by 12/8/2025*).<br>
55
<u>Description</u>: <b>This is a Production-Grade Distributed Task Queue System Built with Java, Spring Boot, Redis, PostgreSQL, GraphQL, REST APIs & React-TypeScript (among various other technologies).</b>
66

7+
## Table of Contents
8+
- [Overview](#overview)
9+
- [Key Features](#key-features)
10+
711
## Overview
812

9-
**SpringQueuePro** is a distributed, fault-tolerant task processing platform that draws inspiration from real services like **Celery**, **BullMQ**, and **AWS SQS**. It was built and implemented from scratch in **Java** and **Spring Boot 3**. It features **persistent tasks**, **Redis-backed distributed locking**, **automatic retries with exponential backoff**, **JWT authentication**, **GraphQL & REST APIs**, and a **real-time React-TS dashboard**. Fully instrumented with **Micrometer + Prometheus** and integration-tested using **Testcontainers**. (As of 2025/12/09 its backend and related databases are hosted on **Railway**; its frontend dashboard on **Netlify**).
13+
**SpringQueuePro** is a distributed, fault-tolerant task processing platform that draws inspiration from real services like **Celery**, **BullMQ**, and **AWS SQS**. It was built and implemented from scratch in **Java** and **Spring Boot 3**. It features **persistent tasks**, **Redis-backed distributed locking**, **automatic retries with exponential backoff**, **JWT authentication**, **GraphQL & REST APIs**, and a **real-time React-TS dashboard**. Fully instrumented with **Micrometer + Prometheus** and integration-tested using **Testcontainers**. SpringQueuePro is designed as a backend-focused demonstration of how modern distributed task queues are architected in production environments, emphasizing correctness, observability, and fault tolerance over raw throughput.
14+
15+
(As of 2025/12/09 its backend and related databases are hosted on **Railway**; its frontend dashboard on **Netlify**).
16+
17+
## Why the SpringQueue-"*Pro*"?
18+
- **SpringQueuePro** is a professional, production-grade evolution of my aptly-titled **SpringQueue** project, an earlier primitive system that mimicked the same core functionality but tightly-coupled and limited in its extensibility, modularity, and production-like features. **SpringQueue** (***base***) was an intentionally skeletal job queue system and little more than an implementation of the Producer-Consumer model created to help learn the basics of Spring Boot. (*Additionally, it served to refresh my Java fundamentals after an absence from using the language. Not to mention an excuse to work with concurrency patterns in Java*).
19+
20+
- SpringQueue itself is a parallel implementation of **GoQueue**, an even earlier job queue project I built — also implementing the Producer-Consumer model, made to practice concurrency and thread-pool patterns in Go (Golang) using Go primitives like goroutines, channels, and mutexes. SpringQueue was a translation of GoQueue but refactored to make it more "idiomatically Java" (e.g., using an ExecutorService as the core thread manager).
21+
22+
## Key Features
23+
- Distributed task processing with durable persistence (PostgreSQL)
24+
- Redis-backed distributed locking to prevent double execution (race conditions).
25+
- Atomic task state transitions (**QUEUED → IN_PROGRESS → COMPLETED / FAILED**)
26+
- Automatic retries with exponential backoff
27+
- Pluggable TaskHandler registry for extensible job logic
28+
- JWT authentication with access + refresh tokens and rotation
29+
- Role-based access control (RBAC) across GraphQL and REST APIs
30+
- Fully instrumented with Micrometer + Prometheus
31+
- Integration-tested using Testcontainers (Postgres + Redis)
32+
- Dockerized for local development and cloud deployment
33+
34+
## System Architecture (Conceptual)
35+
36+
At a high level, the SpringQueuePro system is comprised of:
37+
38+
- **API Layer (GraphQL + REST)**
39+
Exposes authenticated endpoints for task creation, inspection, and administration. All APIs are protected using a stateless JWT-based security model with role-based access control (RBAC).
40+
41+
- **Persistence Layer (PostgreSQL)**
42+
Acts as the system of record for all tasks. Each task follows a strict lifecycle
43+
(**QUEUED → IN_PROGRESS → COMPLETED / FAILED**) enforced via atomic state transitions to prevent race conditions and duplicate execution.
44+
45+
- **Distributed Coordination Layer (Redis)**
46+
Provides distributed locks and ephemeral coordination primitives. Redis-backed locks ensure that only one worker may claim and process a task at a time, even under concurrent execution. (*The traditiional caching "fast-lookup" utility of Redis is—of course—used too, but this is its most significant purpose*).
47+
48+
- **Execution Layer (ExecutorService Worker Pool)**
49+
A configurable thread pool executes tasks asynchronously. Workers claim tasks transactionally, execute handler logic, and update task state deterministically.
50+
51+
- **Processing & Retry Orchestration**
52+
Task execution is orchestrated via a dedicated `ProcessingService` that centralizes retry policies, exponential backoff, failure handling, and re-enqueue logic.
53+
54+
- **Security Layer (Spring Security + JWT)**
55+
Authentication and authorization are enforced via a stateless filter chain with access/refresh token rotation, Redis-backed refresh token storage, and role-based access control across all APIs.
56+
57+
- **Observability Layer (Micrometer + Actuator + Prometheus)**
58+
Provides first-class visibility into task throughput, processing latency, retries, failures, queue depth, JVM health, database connections, and Redis availability.
59+
60+
- **Presentation Layer (React + TypeScript Dashboard)**
61+
A lightweight UI for interacting with the system, visualizing queue state, and monitoring backend health.
62+
63+
All components are containerized using Docker and designed to run identically in local, CI, and cloud environments.
64+
65+
# TO-DO: INSERT MERMAID DIAGRAM HERE!!!
66+
67+
68+
69+
70+
```
71+
flowchart LR
72+
Client[Client<br/>(React Dashboard / API Clients)]
73+
74+
subgraph API[Spring Boot Application]
75+
Auth[Spring Security<br/>JWT + RBAC]
76+
GQL[GraphQL API]
77+
REST[REST API]
78+
79+
Core[Queue & Processing Core]
80+
end
81+
82+
subgraph Core
83+
TS[TaskService]
84+
QS[QueueService]
85+
PS[ProcessingService]
86+
H[TaskHandlers]
87+
end
88+
89+
DB[(PostgreSQL)]
90+
Redis[(Redis)]
91+
Metrics[(Micrometer / Prometheus)]
92+
93+
Client -->|HTTP + JWT| Auth
94+
Auth --> GQL
95+
Auth --> REST
96+
97+
GQL --> TS
98+
REST --> TS
99+
100+
TS --> DB
101+
TS --> QS
102+
103+
QS --> PS
104+
PS --> DB
105+
PS --> Redis
106+
PS --> H
107+
108+
API --> Metrics
109+
```
110+
111+
10112

11-
### System Architecture Summary
12113

13-
- Like any professional distributed job queue system, SpringQueuePro is built around a fully event-driven, asynchronous task execution pipeline with durable persistence, coordinated concurrency, and robust reliability guarantees.
14114

15-
- At its core, SpringQueuePro models a distributed worker-thread architecture: tasks are persisted in PostgreSQL, coordinated via atomic state transitions, claimed safely using Redis-based distributed locks, and executed on a configurable ExecutorService worker pool. This design guarantees idempotent task processing, prevents race conditions when multiple workers (threads) compete for the same job, and ensures that each task is processed exactly once (or retired safely under controlled backoff).
16115

17-
- The platform supports automatic retries with exponential backoff, dead-task prevention, and stateful lifecycle management (**QUEUED → IN_PROGRESS → COMPLETED / FAILED**). Processing logic is fully decoupled using a TaskHandler registry, enabling clean extensibility and domain separation.
18116

19-
- All APIs — both GraphQL and REST — are protected through a modern JWT authentication system with access + refresh tokens, token rotation, revocation, and a Redis-backed refresh token store. The security pipeline uses a stateless Spring Security filter chain, custom authentication filters, and strict authorization boundaries around internal queue management endpoints.
20117

21-
- SpringQueuePro implements role-based access control (RBAC) using Spring Security’s stateless filter chain. Each request passes through a JWT authentication filter that validates tokens, loads user roles from the database, and attaches an authenticated UserDetails principal to the security context. Both GraphQL and REST routes enforce fine-grained authorization rules — public endpoints (login/register) are open, while all internal queue management APIs require authenticated users with the proper roles, ensuring strong separation between public interfaces and privileged system operations.
22118

23-
- SpringQueuePro is fully observability-ready. Using Micrometer, Spring Actuator, and Prometheus, the system records metrics for:
24-
task throughput, retry rates, worker pool utilization, processing duration histograms, queue depth, API call counts, Postgres/Redis health, and JVM resource usage. These metrics are validated using Testcontainers-powered integration tests, ensuring correctness across real Postgres + Redis environments.
25119

26-
- A lightweight React + TypeScript Dashboard provides a visual interface for interacting with the system — allowing user authentication, task creation, queue inspection, and health monitoring of the backend services. **The entire stack is containerized with Docker**.
27120

28-
For my own sake, SpringQueuePro — aside from the practical experience gained in building such a complex system using Java and Spring Boot — was meant to be an exercise to better my expertise in distributed systems, concurrency control, queue design, cloud-native observability (*this will be expanded on in the **Future Improvements** section*), security architecture, and full-stack application development, all packaged in a clean, maintainable, and extensible software engineering project.
29121

30-
## Why the SpringQueue-"*Pro*"?
31-
- **SpringQueuePro** is a professional, production-grade evolution of my aptly-titled **SpringQueue** project, an earlier primitive system that mimicked the same core functionality but tightly-coupled and limited in its extensibility, modularity, and production-like features. **SpringQueue** (***base***) was an intentionally skeletal job queue system and little more than an implementation of the Producer-Consumer model created to help learn the basics of Spring Boot. (*Additionally, it served to refresh my Java fundamentals after an absence from using the language. Not to mention an excuse to work with concurrency patterns in Java*).
32122

33-
- SpringQueue itself is a parallel implementation of **GoQueue**, an even earlier job queue project I built — also implementing the Producer-Consumer model, made to practice concurrency and thread-pool patterns in Go (Golang) using Go primitives like goroutines, channels, and mutexes. SpringQueue was a translation of GoQueue but refactored to make it more "idiomatically Java" (e.g., using an ExecutorService as the core thread manager).
34123
---
35-
## Key Features
124+
## Key Features (old)
36125

37126
### 1. Persistent, Strongly-Typed Task Model
38127

@@ -493,6 +582,9 @@ Testing approach:
493582

494583
---
495584

585+
586+
587+
496588
## Architecture Design
497589

498590
### High-Level Component Diagram
@@ -746,3 +838,139 @@ The React dashboard uses these operations behind the scenes to:
746838
* Fire bursts of task create requests.
747839
* Observe processing latency and retry behavior.
748840
* Validate system under quasi-real load.
841+
842+
843+
844+
845+
### Project Structure
846+
```
847+
src
848+
├── main
849+
│ ├── java
850+
│ │ └── com
851+
│ │ └── springqprobackend
852+
│ │ └── springqpro
853+
│ │ ├── config
854+
│ │ │ ├── ExecutorConfig.java
855+
│ │ │ ├── GlobalExceptionHandler.java
856+
│ │ │ ├── ProcessingMetricsConfig.java
857+
│ │ │ ├── QueueProperties.java
858+
│ │ │ ├── RedisConfig.java
859+
│ │ │ ├── SecurityConfig.java
860+
│ │ │ └── TaskHandlerProperties.java
861+
│ │ ├── controller
862+
│ │ │ ├── auth
863+
│ │ │ │ └── AuthenticationController.java
864+
│ │ │ ├── graphql
865+
│ │ │ │ ├── GraphiQLRedirectController.java
866+
│ │ │ │ └── TaskGraphQLController.java
867+
│ │ │ ├── rest
868+
│ │ │ │ ├── ProcessingEventsController.java
869+
│ │ │ │ ├── ProducerController.java
870+
│ │ │ │ ├── SystemHealthController.java
871+
│ │ │ │ └── TaskRestController.java
872+
│ │ │ └── controllerRecords.java
873+
│ │ ├── domain
874+
│ │ │ ├── entity
875+
│ │ │ │ ├── TaskEntity.java
876+
│ │ │ │ └── UserEntity.java
877+
│ │ │ ├── event
878+
│ │ │ │ └── TaskCreatedEvent.java
879+
│ │ │ └── exception
880+
│ │ │ └── TaskProcessingException.java
881+
│ │ ├── enums
882+
│ │ │ ├── TaskStatus.java
883+
│ │ │ └── TaskType.java
884+
│ │ ├── handlers
885+
│ │ │ ├── DataCleanUpHandler.java
886+
│ │ │ ├── DefaultHandler.java
887+
│ │ │ ├── EmailHandler.java
888+
│ │ │ ├── FailAbsHandler.java
889+
│ │ │ ├── FailHandler.java
890+
│ │ │ ├── NewsLetterHandler.java
891+
│ │ │ ├── ReportHandler.java
892+
│ │ │ ├── SmsHandler.java
893+
│ │ │ ├── TakesLongHandler.java
894+
│ │ │ └── TaskHandler.java
895+
│ │ ├── listeners
896+
│ │ │ └── TaskCreatedListener.java
897+
│ │ ├── mapper
898+
│ │ │ └── TaskMapper.java
899+
│ │ ├── models
900+
│ │ │ ├── Task.java
901+
│ │ │ └── TaskHandlerRegistry.java
902+
│ │ ├── redis
903+
│ │ │ ├── RedisDistributedLock.java
904+
│ │ │ ├── RedisTokenStore.java
905+
│ │ │ ├── Redis_Lua_Note.md
906+
│ │ │ └── TaskRedisRepository.java
907+
│ │ ├── repository
908+
│ │ │ ├── TaskRepository.java
909+
│ │ │ └── UserRepository.java
910+
│ │ ├── runtime
911+
│ │ │ └── Worker.java
912+
│ │ ├── security
913+
│ │ │ ├── dto
914+
│ │ │ │ ├── AuthRequest.java
915+
│ │ │ │ ├── AuthResponse.java
916+
│ │ │ │ ├── LoginRequest.java
917+
│ │ │ │ ├── RefreshRequest.java
918+
│ │ │ │ └── RegisterRequest.java
919+
│ │ │ ├── CustomUserDetailsService.java
920+
│ │ │ ├── JwtAuthenticationFilter.java
921+
│ │ │ ├── JwtUtil.java
922+
│ │ │ └── RefreshTokenService.java
923+
│ │ ├── service
924+
│ │ │ ├── ProcessingService.java
925+
│ │ │ ├── QueueService.java
926+
│ │ │ └── TaskService.java
927+
│ │ ├── util
928+
│ │ │ ├── RealSleeper.java
929+
│ │ │ └── Sleeper.java
930+
│ │ └── SpringQueueProApplication.java
931+
│ └── resources
932+
│ ├── graphql
933+
│ │ └── schema.graphqls
934+
│ ├── static
935+
│ │ └── graphiql
936+
│ │ └── index.html
937+
│ ├── templates
938+
│ ├── application-prod.yml
939+
│ ├── application.properties
940+
│ └── application.yml
941+
└── test
942+
├── java
943+
│ └── com
944+
│ └── springqprobackend
945+
│ └── springqpro
946+
│ ├── config
947+
│ │ └── RedisTestConfig.java
948+
│ ├── handlers
949+
│ │ ├── DefaultHandlerTests.java
950+
│ │ └── FailHandlerTests.java
951+
│ ├── integration
952+
│ │ ├── AuthJwtIntegrationTest.java
953+
│ │ ├── CreateAndProcessTaskIntegrationTest.java
954+
│ │ ├── OwnershipGraphQLIntegrationTest.java
955+
│ │ ├── ProcessingConcurrencyIntegrationTest.java
956+
│ │ ├── RedisDistributedLockIntegrationTest.java
957+
│ │ ├── RedisPingIntegrationTest.java
958+
│ │ ├── RetryBehaviorIntegrationTest.java
959+
│ │ ├── TaskCacheIntegrationTest.java
960+
│ │ └── TaskGraphQLIntegrationTest.java
961+
│ ├── models
962+
│ │ └── TaskHandlerRegistryTests.java
963+
│ ├── runtime
964+
│ │ └── WorkerTests.java
965+
│ ├── service
966+
│ │ └── QueueServiceTests.java
967+
│ ├── testcontainers
968+
│ │ ├── BasePostgresContainer.java
969+
│ │ ├── BaseRedisContainer.java
970+
│ │ ├── IntegrationTestBase.java
971+
│ │ └── RedisIntegrationTestBase.java
972+
│ └── SpringQueueProApplicationTests.java
973+
└── resources
974+
├── application-test.properties
975+
└── application-test.yml
976+
```

0 commit comments

Comments
 (0)