[WIP] Update README to include project introduction and architecture

[Copilot]

Comprehensive README Update Plan

• Project Introduction section
  • Clear description of distributed social media backend
  • Mention CS6650 course project
  • Learning goals and motivation
• System Architecture section
  • Text-based architecture diagram
  • Document microservices, communication, databases, infrastructure
• Core Features section
  • Document User Service
  • Document Social Graph Service
  • Document Post Service
  • Document Timeline Service
  • Document Web Service
• Technology Stack section
• Key Experiments & Results section
  • Timeline retrieval experiments
  • Storage experiments
  • Post consistency testing
• Project Evolution Timeline section
• Development Activity section
• Getting Started section
• Project Structure section
• Team & License section

Original prompt

Overview

Replace the minimal README with a comprehensive document that introduces the CS6650 distributed social media backend project, showcases the system architecture, highlights experimental results, and demonstrates the project's evolution over time.

Current State

The current README.md contains only a single line: # CS6550-Project

Requirements

1. Project Introduction

• Clear description of what the project is: a distributed Twitter-like social media backend
• Explain the motivation: learning distributed systems concepts, comparing design patterns (push vs pull vs hybrid timeline generation)
• Mention this is a CS6650 course project
• Key learning goals: scalability, microservices architecture, database design trade-offs

2. System Architecture

Create a text-based architecture diagram showing:

• Microservices: User Service, Social Graph Service, Post Service, Timeline Service, Web Service
• Communication: gRPC for service-to-service, HTTP/REST for external APIs
• Databases: RDS PostgreSQL (user data), DynamoDB (posts, social graph, timelines)
• Infrastructure: AWS ECS, Application Load Balancer, Service Connect
• IaC: Terraform for infrastructure management

3. Core Features

Document each microservice:

User Service (Python/Flask, gRPC + HTTP, Port 8080/50051)

• User registration and authentication
• Profile management
• RDS PostgreSQL backend

Social Graph Service (Go, gRPC + HTTP, Port 8085/50052)

• Follow/unfollow relationships
• Followers/following lists with pagination
• DynamoDB with list format for O(1) lookups
• Power-law distributed test data (mega users, influencers, active, casual)

Post Service (Python/Flask, HTTP, Port 8083)

• Create posts
• Smart fan-out based on follower count
• Push/pull/hybrid strategies
• DynamoDB backend

Timeline Service (Python/Flask, HTTP, Port 8084)

• Timeline generation (push/pull/hybrid models)
• Redis caching for pull model optimization
• Integration with Post and Social Graph services

Web Service (Go/Gin, Port 3000)

• Frontend HTML interface
• Aggregates backend microservices

4. Technology Stack

List all technologies:

• Languages: Python, Go
• Frameworks: Flask, Gin, gRPC
• Databases: PostgreSQL (RDS), DynamoDB, Redis
• Infrastructure: AWS ECS Fargate, ALB, VPC, CloudWatch
• IaC: Terraform with modular design
• Containerization: Docker, ECR
• Service Discovery: AWS ECS Service Connect
• Testing: Locust for load testing, Python scripts

5. Key Experiments & Results

Highlight the experimental work documented in the repository:

Timeline Retrieval Experiments (PR #31)

• Compared push vs pull vs hybrid fan-out strategies
• Tests at 5K, 25K, 100K user scales
• Results showing latency trade-offs and throughput characteristics
• Located in: tests/timeline_retrieval/

Storage Experiments (PR #32)

• Storage comparison across different strategies
• Push model: $0.1984/month for 5K users (high timeline storage)
• Pull model: $0.0015/month (minimal storage, compute-intensive)
• Hybrid model: $0.1859/month (balanced approach)
• Results in: tests/storage-experiment/results/

Post Consistency Testing (PR #34)

• Testing data consistency across distributed services

6. Project Evolution Timeline

Show development progression using commit/PR history:

Phase 1: Foundation (Nov 5-7)

• Initial user service implementation (PR user service #1, Feature/user service #2, Feature/user service #3)
• Basic infrastructure setup

Phase 2: Core Services (Nov 7-9)

• User service with gRPC (PR Feature/user service #6)
• Post service initialization (PR post service init #7)
• Timeline service development (PR Feature/timeline service #8, transplant timeline service user service gRPC to real one #9, Feature/timeline service #10)

Phase 3: Integration (Nov 9-12)

• Service integration and bug fixes (PR Feature/post service merge #11-20)
• Post/Timeline service coordination
• Social Graph service merge (PR merge Dev/social graph services to main #21)

Phase 4: Experiments & Optimization (Nov 12-29)

• Timeline retrieval experiments (PR Feature/timeline service #22, Feature/post timeline service #23, Feature/timeline service #25, Feature/timeline service #29, Feature/timeline service #30, Experiments/timeline retrieval #31)
• Hybrid threshold tuning (PR move hybrid threshold to enviroment variables #24, keep the hybrid threshold of post service align with timeline service #28)
• Storage experiments (PR storage experiments #32)
• Consistency testing (PR add results #33, added posts inconsistency test #34)

7. Development Activity

Showcase the 34 pull requests and 30+ commits demonstrating iterative development, experimentation, and refinement

8. Getting Started

Include:

• Prerequisites (AWS account, Terraform, Docker, Python, Go)
• Clone instructions
• Infrastructure deployment with Terraform
• Service deployment steps
• Testing instructions
• Reference to CURL_COMMANDS.md for API examples

9. Project Structure

Show the repository organization:

├── services/          # Microservices
│   ├── user-service/
│   ├── social-graph-services/
│   ├── post-service/
│   └── timeline-service/
├── web-service/       # Frontend
├── terraform/         # Infrastructure as Code
├── tests/            # Experiments and load tests
│   ├── timeline_retrieval/
│   └── storage-experiment/
├── proto/            # Protocol buffer definitions
└── scripts/          # Utility scripts

10. Team & License

• Mention contributors: @PCBZ, @woodsxwu, @nagi0310, @yi...

This pull request was created as a result of the following prompt from Copilot chat.

Overview

Replace the minimal README with a comprehensive document that introduces the CS6650 distributed social media backend project, showcases the system architecture, highlights experimental results, and demonstrates the project's evolution over time.

Current State

The current README.md contains only a single line: # CS6550-Project

Requirements

1. Project Introduction

• Clear description of what the project is: a distributed Twitter-like social media backend
• Explain the motivation: learning distributed systems concepts, comparing design patterns (push vs pull vs hybrid timeline generation)
• Mention this is a CS6650 course project
• Key learning goals: scalability, microservices architecture, database design trade-offs

2. System Architecture

Create a text-based architecture diagram showing:

• Microservices: User Service, Social Graph Service, Post Service, Timeline Service, Web Service
• Communication: gRPC for service-to-service, HTTP/REST for external APIs
• Databases: RDS PostgreSQL (user data), DynamoDB (posts, social graph, timelines)
• Infrastructure: AWS ECS, Application Load Balancer, Service Connect
• IaC: Terraform for infrastructure management

3. Core Features

Document each microservice:

User Service (Python/Flask, gRPC + HTTP, Port 8080/50051)

• User registration and authentication
• Profile management
• RDS PostgreSQL backend

Social Graph Service (Go, gRPC + HTTP, Port 8085/50052)

• Follow/unfollow relationships
• Followers/following lists with pagination
• DynamoDB with list format for O(1) lookups
• Power-law distributed test data (mega users, influencers, active, casual)

Post Service (Python/Flask, HTTP, Port 8083)

• Create posts
• Smart fan-out based on follower count
• Push/pull/hybrid strategies
• DynamoDB backend

Timeline Service (Python/Flask, HTTP, Port 8084)

• Timeline generation (push/pull/hybrid models)
• Redis caching for pull model optimization
• Integration with Post and Social Graph services

Web Service (Go/Gin, Port 3000)

• Frontend HTML interface
• Aggregates backend microservices

4. Technology Stack

List all technologies:

• Languages: Python, Go
• Frameworks: Flask, Gin, gRPC
• Databases: PostgreSQL (RDS), DynamoDB, Redis
• Infrastructure: AWS ECS Fargate, ALB, VPC, CloudWatch
• IaC: Terraform with modular design
• Containerization: Docker, ECR
• Service Discovery: AWS ECS Service Connect
• Testing: Locust for load testing, Python scripts

5. Key Experiments & Results

Highlight the experimental work documented in the repository:

Timeline Retrieval Experiments (PR #31)

• Compared push vs pull vs hybrid fan-out strategies
• Tests at 5K, 25K, 100K user scales
• Results showing latency trade-offs and throughput characteristics
• Located in: tests/timeline_retrieval/

Storage Experiments (PR #32)

• Storage comparison across different strategies
• Push model: $0.1984/month for 5K users (high timeline storage)
• Pull model: $0.0015/month (minimal storage, compute-intensive)
• Hybrid model: $0.1859/month (balanced approach)
• Results in: tests/storage-experiment/results/

Post Consistency Testing (PR #34)

• Testing data consistency across distributed services

6. Project Evolution Timeline

Show development progression using commit/PR history:

Phase 1: Foundation (Nov 5-7)

• Initial user service implementation (PR user service #1, Feature/user service #2, Feature/user service #3)
• Basic infrastructure setup

Phase 2: Core Services (Nov 7-9)

• User service with gRPC (PR Feature/user service #6)
• Post service initialization (PR post service init #7)
• Timeline service development (PR Feature/timeline service #8, transplant timeline service user service gRPC to real one #9, Feature/timeline service #10)

Phase 3: Integration (Nov 9-12)

• Service integration and bug fixes (PR Feature/post service merge #11-20)
• Post/Timeline service coordination
• Social Graph service merge (PR merge Dev/social graph services to main #21)

Phase 4: Experiments & Optimization (Nov 12-29)

• Timeline retrieval experiments (PR Feature/timeline service #22, Feature/post timeline service #23, Feature/timeline service #25, Feature/timeline service #29, Feature/timeline service #30, Experiments/timeline retrieval #31)
• Hybrid threshold tuning (PR move hybrid threshold to enviroment variables #24, keep the hybrid threshold of post service align with timeline service #28)
• Storage experiments (PR storage experiments #32)
• Consistency testing (PR add results #33, added posts inconsistency test #34)

7. Development Activity

Showcase the 34 pull requests and 30+ commits demonstrating iterative development, experimentation, and refinement

8. Getting Started

Include:

• Prerequisites (AWS account, Terraform, Docker, Python, Go)
• Clone instructions
• Infrastructure deployment with Terraform
• Service deployment steps
• Testing instructions
• Reference to CURL_COMMANDS.md for API examples

9. Project Structure

Show the repository organization:

├── services/          # Microservices
│   ├── user-service/
│   ├── social-graph-services/
│   ├── post-service/
│   └── timeline-service/
├── web-service/       # Frontend
├── terraform/         # Infrastructure as Code
├── tests/            # Experiments and load tests
│   ├── timeline_retrieval/
│   └── storage-experiment/
├── proto/            # Protocol buffer definitions
└── scripts/          # Utility scripts

10. Team & License

• Mention contributors: @PCBZ, @woodsxwu, @nagi0310, @yixu9-hub
• Reference MIT License

Writing Style

• Clear, professional, and technical
• Use emojis sparingly for visual organization (🏗️, 🚀, 🧪, 📊)
• Include code blocks for architecture diagrams
• Keep it informative but concise
• Emphasize the learning journey and experimental nature

References

• Link to individual service READMEs for detailed documentation
• Reference experiment result files
• Link to CURL_COMMANDS.md for API usage examples

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.