Skip to content

Project Plan

Jump to bottom
PCBZ edited this page Dec 3, 2025 · 11 revisions

Detailed Project Plan: Distributed Social Media Platform

1. Executive Summary

Project Goal: Design and implement a distributed social media platform demonstrating core distributed systems concepts including fan-out algorithms, CAP theorem trade-offs, and horizontal scaling strategies.

Architecture Overview: 4-microservice system (User, Post, Social Graph, Timeline Services) deployed on AWS ECS Fargate with Infrastructure as Code using Terraform.

Key Deliverables:

  • Post and Timeline generation with 3 fan-out strategies (Push, Pull, Hybrid)
  • Performance analysis across different user scales
  • Comprehensive scalability testing and CAP theorem validation

2. Technical Architecture

2.1 System Design

graph TB
    Users[👤 Users] --> ALB[Application Load Balancer]
    
    ALB --> AGW[API Gateway Service<br/>ECS Fargate<br/>Request Router]
    

    subgraph "Microservices Layer"
        AGW --> US[User Service<br/>ECS Fargate]
        AGW --> SGS[Social Graph Service<br/>ECS Fargate] 
        AGW --> PS[Post Service<br/>ECS Fargate]
        AGW --> TS[Timeline Service<br/>ECS Fargate]
    end
    
    subgraph "Data Layer"
        US --> RDS[(RDS PostgreSQL<br/>User Profiles<br/>Authentication)]
        SGS --> DDB3[(DynamoDB<br/>Social Relationships)]
        PS --> DDB1[(DynamoDB<br/>Posts Storage)]
        TS --> DDB2[(DynamoDB<br/>Timeline Cache)]
    end
    
    subgraph "Message Queue Layer"
        PS --> SNS[SNS Topic<br/>Post Created Events]
        SNS --> SQS[SQS Queue<br/>Timeline Updates]
        SQS --> TS
    end
Loading

2.2 Fan-out Strategy Implementation

  • Push Model: Pre-compute timelines for all followers upon post creation
  • Pull Model: Generate timelines on-demand during read requests
  • Hybrid Model: Celebrity users (50K+ followers) use pull strategy, regular users use push strategy

2.3 Service Responsibilities

Service Responsibilities Database Key Features
User Service profile management PostgreSQL User registration, profile data
Post Service Content creation DynamoDB Create Post
Social Graph Service Relationship management Neptune Follow/unfollow, follower lists
Timeline Service Timeline generation/retrieval DynamoDB Fan-out algorithms, timeline caching

3. Implementation Phases

Phase 1: Core Infrastructure Setup + Core Service Development (Week 1-2)

Objectives:

  • Establish AWS ECS Fargate environment
  • Configure Terraform Infrastructure as Code
  • Set up basic Go/Gin microservice framework
  • Initialize database instances (PostgreSQL, DynamoDB)
  • Implement fundamental CRUD operations for each service
  • Establish inter-service communication patterns
  • Create basic API endpoints

Deliverables:

  • Terraform scripts for AWS infrastructure
  • Basic Go/Gin service templates
  • Database schemas and connections
  • Docker containerization with multi-arch support
  • AWS Service Connect configuration
  • User Service: Registration, profile management APIs
  • Post Service: Create, post functionality
  • Social Graph Service: Follow/unfollow relationship management
  • Timeline Service: Basic timeline retrieval infrastructure
  • RPC interface definitions using protobuf
  • SNS/SQS configuration

Phase 3: Fan-out Algorithm Implementation (Week 3)

Objectives:

  • Implement 3 distinct fan-out strategies
  • Optimize for different user scenarios

Deliverables:

  • Push Fan-out: Pre-computed timeline updates
  • Pull Fan-out: On-demand timeline generation
  • Hybrid Fan-out: Celebrity detection (50K+ followers threshold)

Phase 4: Performance Testing & Analysis (Week 4)

Objectives:

  • Comprehensive load testing across different scales
  • Performance comparison between 3 fan-out strategies

Deliverables:

  • Locust load testing configuration
  • Performance metrics for 5K, 25K, 100K user scenarios
  • Latency, throughput, and resource utilization analysis
  • Horizontal scaling verification

4. Technology Stack

Backend Services

  • Programming Language: Go
  • Web Framework: Gin
  • Communication: RPC with protobuf message definitions

Data Storage

  • User Data: PostgreSQL
  • Relationship: Neptune
  • Posts & Timelines: DynamoDB (NoSQL for scalability)
  • Caching: Redis ElastiCache (optional, time permitting)

Infrastructure & Deployment

  • Container Platform: AWS ECS Fargate
  • Load Balancing: Application Load Balancer
  • Service Discovery: AWS Service Connect
  • Infrastructure as Code: Terraform
  • Container Registry: Amazon ECR
  • Monitoring: AWS CloudWatch

Development & Testing

5. Core Experiments

We will tests with 3 user scales： 5K, 25K, 100K. User distribution:

Attribute Regular Users Influencers Celebrities
Percentage 85% 14% 1%
Follower Count 10-100 100-50,000 50,000-500,000
Following Count 50-200 100-500 50-200

Post Inconsistency

  • Overview: Post different scale, request timline retrievel time.
  • Result: Different scales with inconsistency ratio.

Timeline Retrievel Time

  • Overview: Locust test
  • Result: Response time.

Database Storage

  • Overview: Calculate database storage.
  • Result: Different scale Post-service and Timeline-service DynamoDB

Throughput, AutoScale

  • Overview: Test different load RPS
  • Result: RPS and ECS count.

Clone this wiki locally