Skip to content

DOCUMENTATION_INDEX.md

Latest commit

 

History

History
291 lines (228 loc) · 9.96 KB

DOCUMENTATION_INDEX.md

File metadata and controls

291 lines (228 loc) · 9.96 KB
Raw

StreamFlow Documentation Index

Welcome to the StreamFlow documentation! This index will help you find the right documentation for your needs.

Quick Links

🚀 Getting Started

📚 Learning Resources

👨‍💻 For Developers

🚢 For Operations

📊 Project Status

Documentation by Role

I'm a Student / Learning Distributed Systems

Start here:

  1. README - Understand what StreamFlow is
  2. Learning Guide - Deep dive into concepts
  3. Architecture - How it all works together
  4. Phase 1: Storage Layer - Log-structured storage
  5. Phase 2: Network Layer - Binary protocol
  6. Phase 3: Consumer Groups - Load balancing

Try it out:

I'm a Developer / Contributor

Start here:

  1. Developer Guide - Setup your environment
  2. Project Structure - Understand the codebase
  3. Architecture - System design
  4. API Reference - API contracts

Building and Testing:

Code References:

I'm an Operator / DevOps Engineer

Start here:

  1. Deployment Guide - Deploy to production
  2. Docker Deployment - Containerized setup
  3. Configuration - Tuning parameters
  4. Monitoring Setup - Prometheus + Grafana

Operations:

Quick Deploy:

# Docker Compose (easiest)
docker-compose up -d

# Access services:
# - Broker: localhost:9092
# - Admin API: http://localhost:8080/api/swagger-ui.html
# - Prometheus: http://localhost:9090
# - Grafana: http://localhost:3000 (admin/admin)

I'm Using the API

Start here:

  1. API Reference - Complete API documentation
  2. Admin REST API - HTTP endpoints
  3. Producer Client API - Java producer
  4. Consumer Client API - Java consumer

Examples:

Swagger UI:

# Start admin API
mvn spring-boot:run -f admin/pom.xml

# Open Swagger UI
open http://localhost:8080/api/swagger-ui.html

Documentation by Topic

Architecture & Design

Document Description
Architecture Complete system architecture
Storage Architecture Log-structured storage
Network Protocol Binary protocol specification
Replication Leader-follower replication
Consumer Groups Group coordination

Development

Document Description
Developer Guide Complete development guide
Getting Started Setup environment
Building Build and test
Adding Features Extend the platform
Code Style Coding conventions

Deployment

Document Description
Deployment Guide Complete deployment guide
Local Deployment Run locally
Docker Deployment Containerized setup
Production Deployment Production-ready setup
Monitoring Prometheus + Grafana

API Documentation

Document Description
API Reference Complete API reference
REST API Admin REST endpoints
Binary Protocol Wire protocol
Producer API Producer client
Consumer API Consumer client
Error Codes All error codes

Learning

Document Description
Learning Guide Educational deep dive
Phase 1: Storage Storage layer implementation
Phase 2: Network Network layer implementation
Phase 3: Consumer Groups Consumer groups
Phase 5: Admin API Admin API implementation

Complete File List

Main Documentation (11 files)

docs/
├── DOCUMENTATION_INDEX.md           # This file
├── ARCHITECTURE.md                  # System architecture (600+ lines)
├── DEVELOPER_GUIDE.md               # Developer guide (800+ lines)
├── DEPLOYMENT_GUIDE.md              # Deployment guide (700+ lines)
├── API_REFERENCE.md                 # API reference (800+ lines)
├── LEARNING_GUIDE.md                # Learning guide (400+ lines)
├── phase1-storage-layer.md          # Phase 1 (300+ lines)
├── phase2-network-layer.md          # Phase 2 (300+ lines)
├── phase3-consumer-groups.md        # Phase 3 (200+ lines)
└── phase5-admin-api.md              # Phase 5 (300+ lines)

Project Documentation (3 files)

├── README.md                        # Project overview
├── IMPLEMENTATION_COMPLETE.md       # Implementation status
└── PROJECT_STRUCTURE.md             # Directory structure

Configuration (10 files)

├── pom.xml                          # Parent POM
├── common/pom.xml
├── broker/pom.xml
├── client/pom.xml
├── admin/pom.xml
├── Dockerfile.broker
├── Dockerfile.admin
├── docker-compose.yml
├── monitoring/prometheus.yml
└── admin/src/main/resources/application.yml

Documentation Statistics

  • Total Documentation Files: 11
  • Total Lines of Documentation: ~4,700 lines
  • Total Project Files: 78 files
  • Total Lines of Code: ~7,500 lines
  • Documentation-to-Code Ratio: 63%

Quick Command Reference

Build

mvn clean install              # Build all modules
mvn test                       # Run all tests

Run Locally

# Terminal 1: Broker
java -jar broker/target/streamflow-broker-1.0.0-SNAPSHOT.jar

# Terminal 2: Admin API
java -jar admin/target/streamflow-admin-1.0.0-SNAPSHOT.jar

Run with Docker

docker-compose up -d           # Start all services
docker-compose logs -f         # View logs
docker-compose down            # Stop all services

Test APIs

# Create topic
curl -X POST http://localhost:8080/api/topics \
  -H "Content-Type: application/json" \
  -d '{"name":"test","partitions":3,"replicationFactor":1}'

# Get topic info
curl http://localhost:8080/api/topics/test | jq

# Health check
curl http://localhost:8080/api/broker/health | jq

External Resources

Related Technologies

Learning Resources

Contributing

We welcome contributions! See Developer Guide - Adding New Features

Support

Last Updated: 2026-03-15 Version: 1.0.0 Status: Complete (6/6 phases) ✅

Happy learning and building with StreamFlow! 🚀