🚀 Distributed File Storage System

A distributed file storage platform built using Spring Boot microservices that separates metadata management from physical file storage while supporting chunk replication, integrity verification, concurrent downloads, heartbeat monitoring, and automatic self-healing.

📖 Overview

Modern distributed storage systems separate metadata management from physical storage to improve scalability, fault tolerance, and maintainability.

This project demonstrates those principles by implementing a Distributed File Storage System (DFS) composed of two independent Spring Boot microservices:

Metadata Service – Coordinates uploads/downloads, maintains file metadata, manages storage nodes, detects failures, and performs replica recovery.
Storage Service – Stores file chunks on local disk, serves chunk requests, registers with the metadata service, and periodically sends heartbeat updates.

Files are divided into configurable chunks, replicated across multiple storage nodes, verified using SHA-256 checksums, and reconstructed during download using parallel retrieval with CompletableFuture.

🎯 Problem Statement

Traditional file storage systems often tightly couple metadata management with physical file storage, making them difficult to scale, recover from failures, and distribute across multiple storage nodes.

This project addresses these challenges by implementing a Distributed File Storage System (DFS) that separates metadata management from chunk storage through a microservices architecture.

The Metadata Service acts as the system coordinator, maintaining file metadata, chunk mappings, storage node health, and replication information, while independent Storage Services are responsible for persisting file chunks on local storage.

To improve reliability and resilience, the system implements chunk replication, SHA-256 integrity verification, scheduled heartbeat monitoring, automatic failure detection, and self-healing replica recovery. Files are reconstructed through concurrent chunk retrieval, demonstrating fundamental concepts used in modern distributed storage systems.

✨ Key Features

📦 Distributed Storage

Microservices-based architecture
Metadata and storage separation
Chunk-based file storage
Configurable chunk size

🔁 Fault Tolerance

Replication Factor = 2
Automatic storage node registration
Scheduled heartbeat monitoring
Node failure detection
Automatic re-replication (Self-Healing)

🔒 Data Integrity

SHA-256 checksum generation
Integrity verification during downloads
Path traversal protection

⚡ Performance

Concurrent chunk downloads using CompletableFuture
Round-robin chunk allocation
REST-based inter-service communication
Local disk storage using Java NIO

🏗️ System Architecture

Architecture Highlights

Metadata Service acts as the coordinator for all file operations.
Storage Services store replicated file chunks on the local file system.
PostgreSQL stores metadata, chunk mappings, storage node information, and heartbeat status.
Services communicate using REST APIs via RestTemplate.
Storage nodes automatically register with the Metadata Service during startup.
Scheduled heartbeats enable node health monitoring and failure detection.
Self-healing logic automatically restores missing replicas when a node becomes unavailable.

🎯 Core Capabilities

Capability	Implementation
File Chunking	Configurable chunk size
Chunk Replication	Replication Factor = 2
Load Distribution	Round-Robin Allocation
Integrity Verification	SHA-256 Checksums
Parallel Downloads	CompletableFuture
Node Discovery	Automatic Registration
Health Monitoring	Scheduled Heartbeats
Failure Detection	Metadata Scheduler
Replica Recovery	Automatic Self-Healing
Storage Backend	Local File System
Metadata Storage	PostgreSQL
Communication	REST APIs

🛠️ Technology Stack

Category	Technologies
Language	Java 25
Framework	Spring Boot
Database	PostgreSQL
ORM	Spring Data JPA (Hibernate)
Build Tool	Maven
Communication	REST APIs, RestTemplate
Concurrency	CompletableFuture
File Storage	Java NIO Local File System
Integrity Verification	SHA-256
Scheduling	Spring Scheduler (@Scheduled)
Code Generation	Lombok

📂 Repository Structure

dfs-java-project/
│
├── README.md
├── LICENSE
├── .gitignore
│
├── docs/
│   ├── architecture.md
│   ├── api-reference.md
│   ├── deployment.md
│   ├── design-decisions.md
│   ├── workflow.md
│   └── images/
│       └── architecture.png
│
├── metadata-service/
│   ├── src/
│   ├── pom.xml
│   └── ...
│
└── storage-service/
    ├── src/
    ├── pom.xml
    └── ...

🔄 System Workflow

The Distributed File Storage System follows a coordinated workflow where the Metadata Service orchestrates all file operations while Storage Services are responsible for persisting and serving file chunks.

File Upload

Client uploads a file to the Metadata Service.
Metadata Service validates active storage nodes.
File metadata is created in PostgreSQL.
The file is divided into configurable chunks.
SHA-256 checksums are generated for every chunk.
Chunks are distributed across storage nodes using a round-robin strategy.
Each chunk is replicated to two storage nodes.
Chunk metadata and replica mappings are stored.
Upload response is returned to the client.

File Download

Client requests a file.
Metadata Service retrieves chunk metadata.
Active storage nodes are identified.
Chunks are downloaded concurrently using CompletableFuture.
SHA-256 integrity verification is performed.
Chunks are merged in their original order.
The reconstructed file is streamed back to the client.

Storage Node Lifecycle

Storage nodes register automatically during startup.
Heartbeats are sent every five seconds.
Metadata Service monitors node health.
Nodes missing heartbeats for more than thirty seconds are marked as DOWN.
Replica recovery is triggered automatically whenever redundancy is reduced.

📡 Service Communication

                Client
                   │
                   ▼
        Metadata Service
      (Coordinator / Name Node)
                   │
     REST (RestTemplate)
                   │
        ┌──────────┴──────────┐
        ▼                     ▼
 Storage Service 1     Storage Service 2
        │                     │
        └──────────┴──────────┘
                   ▼
          Local File Storage

The Metadata Service coordinates all client requests while Storage Services remain stateless storage nodes responsible for storing and retrieving file chunks. Communication between services is implemented using REST APIs with RestTemplate.

🎯 Design Highlights

Separation of metadata management from physical storage.
Configurable chunk-based file storage.
Round-robin chunk allocation strategy.
Replication factor of 2 for fault tolerance.
SHA-256 checksum verification for data integrity.
Concurrent downloads using CompletableFuture.
Automatic storage node registration.
Scheduled heartbeat monitoring.
Failure detection with automatic node status updates.
Self-healing replica recovery when redundancy is reduced.
Local filesystem-backed chunk persistence using Java NIO.

⚙️ Prerequisites

Before running the project, ensure the following software is installed:

Software	Version
Java	25
Maven	3.x or later
PostgreSQL	14+
Git	Latest

🚀 Getting Started

1. Clone the Repository

git clone https://github.com/<your-username>/dfs-java-project.git
cd dfs-java-project

2. Configure PostgreSQL

Create a PostgreSQL database named:

dfs_metadata

Update the database credentials in:

metadata-service/src/main/resources/application.properties

Example:

spring.datasource.url=jdbc:postgresql://localhost:5432/dfs_metadata
spring.datasource.username=postgres
spring.datasource.password=your_password

3. Start the Metadata Service

cd metadata-service

./mvnw spring-boot:run

Runs on

http://localhost:8081

4. Start Storage Service

Open another terminal.

cd storage-service

./mvnw spring-boot:run

Runs on

http://localhost:8082

After startup, the storage service automatically registers itself with the Metadata Service and begins sending heartbeat messages.

🖥️ Running Multiple Storage Nodes

To simulate a distributed storage cluster locally, launch multiple storage service instances using different ports and storage directories.

Storage Node 1

java -jar storage-service.jar \
--server.port=8082 \
--dfs.storage.dir=storage1

Storage Node 2

java -jar storage-service.jar \
--server.port=8083 \
--dfs.storage.dir=storage2

Storage Node 3

java -jar storage-service.jar \
--server.port=8084 \
--dfs.storage.dir=storage3

Each instance automatically registers with the Metadata Service and participates in chunk storage and replication.

📡 REST API Overview

Metadata Service

Method	Endpoint	Description
POST	`/files/upload`	Upload a file
GET	`/files/{fileId}`	Retrieve file metadata
GET	`/files/download/{fileId}`	Download a file
DELETE	`/files/{fileId}`	Delete file metadata
POST	`/nodes/register`	Register a storage node
POST	`/nodes/heartbeat`	Process heartbeat requests

Storage Service

Method	Endpoint	Description
POST	`/chunks/upload`	Upload a chunk
GET	`/chunks/{chunkName}`	Download a chunk
POST	`/heartbeat/start`	Resume heartbeat
POST	`/heartbeat/stop`	Pause heartbeat

📊 Metadata Database

The Metadata Service stores all coordination information inside PostgreSQL.

Main Entities

Entity	Purpose
File	Stores uploaded file metadata
Chunk	Stores chunk mapping and checksum information
StorageNode	Tracks registered storage nodes and heartbeat status

The actual binary file chunks are not stored in the database. Instead, they are persisted on the local file system of the storage services, while PostgreSQL maintains metadata and replica mappings.

🔐 Reliability Features

The system incorporates several reliability mechanisms commonly found in distributed storage systems.

Automatic storage node registration
Heartbeat-based health monitoring
Failure detection
Chunk replication
Automatic replica recovery
SHA-256 checksum validation
Path traversal protection
Concurrent chunk retrieval

📚 Documentation

Additional project documentation is available in the docs/ directory.

Document	Description
`architecture.md`	Detailed explanation of the system architecture and service interactions
`workflow.md`	End-to-end upload, download, heartbeat, and replication workflows
`api-reference.md`	Complete REST API reference with request and response examples
`deployment.md`	Local deployment guide and multi-node setup instructions
`design-decisions.md`	Design choices, trade-offs, and architectural decisions

⚠️ Current Limitations

This project was developed as a learning-oriented implementation of a Distributed File Storage System. Some production-level capabilities are intentionally simplified or remain future enhancements.

Current limitations include:

Replication factor is currently hardcoded to 2.
Storage node allocation is based on round-robin scheduling rather than available disk space.
Metadata is stored in a single PostgreSQL instance without replication.
Chunk uploads are performed synchronously.
File chunk deletion on storage nodes is not performed when metadata is deleted.
Configuration values such as coordinator URLs are partially hardcoded.
Files are processed in memory during upload and download.

These limitations provide opportunities for future improvements and experimentation.

🚀 Future Enhancements

Potential improvements include:

Configurable replication factor
Distributed metadata service
Storage node load balancing
Dynamic storage capacity awareness
Automatic orphan chunk cleanup
Asynchronous chunk uploads
Streaming upload/download for large files
Authentication and authorization
Service discovery
Docker & Docker Compose support
Kubernetes deployment
Monitoring with Prometheus and Grafana
Distributed tracing
OpenAPI (Swagger) documentation
Web-based administration dashboard

🤝 Contributing

Contributions, suggestions, and improvements are welcome.

If you would like to contribute:

Fork the repository.
Create a feature branch.
Commit your changes.
Open a Pull Request.

Please ensure that any contribution maintains code quality and includes appropriate documentation where applicable.

📄 License

This project is licensed under the MIT License.

See the LICENSE file for more information.

👨💻 Author

Developed as part of a learning project to explore the design and implementation of distributed storage systems using Java and Spring Boot.

⭐ If you found this project helpful

If this repository helped you understand distributed file storage systems, consider giving it a ⭐ on GitHub.

It helps increase the visibility of the project and supports continued improvements.

Name		Name	Last commit message	Last commit date
docs/images		docs/images
metadata-service		metadata-service
storage-service		storage-service
.gitignore		.gitignore
README.md		README.md

Latest commit

History

Folders and files

Repository files navigation