LogPulse

Introduction

LogPulse is a cloud-native backend solution designed to handle massive scale log ingestion and real-time analytics. It addresses the challenge of "Peak Shaving" in high-concurrency scenarios by decoupling the ingestion layer from the storage layer.

Built with a Microservices mindset, LogPulse ensures data consistency and system resilience through asynchronous messaging and a robust caching strategy.

Table of Contents

• Getting Started
  • System Requirements
  • Prerequisites
  • Quick Start
  • Manual Start
  • Troubleshooting
• Kubernetes Deployment
  • Prerequisites
  • Deploy
  • Horizontal Pod Autoscaler
  • Teardown
• API Usage Examples
• Key Features
• Architecture
• Performance Benchmarks
  • Data Validation
  • Stress Test Analysis
• Rate Limiting
• Design Decisions & Trade-offs
• Project Layout
• License

Getting Started

System Requirements

Since this stack involves heavy infrastructure (Elasticsearch, Kafka), please ensure your environment meets the minimum requirements:

• RAM: 4GB minimum free memory (8GB recommended).
• Disk: 10GB free space.
• Note: If you are running on low memory, consider disabling Kibana in docker-compose.yml to save resources.

Prerequisites

• Docker & Docker Compose installed.
• Make (Optional, for simplified commands).

Quick Start (Recommended)

We provide a Makefile to simplify common operations.

1. Clone the repository

   git clone https://github.com/Yupoer/logpulse.git
   cd logpulse

2. Run the application

   make run

   This command will automatically build the images and start all services (App, MySQL, Redis, Kafka, ES, Kibana) in the background.

   Note: By default, make run initializes 3 Go Application Replicas (API + Worker) and 3 Kafka partitions/consumers behind an Nginx Load Balancer to simulate a production-ready distributed environment.

3. Stop the application

   make stop

Manual Start (Without Make)

If you are on Windows (without WSL) or don't have make installed, you can use the raw Docker commands:

# Start services
docker-compose -f deployments/docker-compose.yml up -d --build

# Stop services
docker-compose -f deployments/docker-compose.yml down

Verify Status

docker-compose ps
# OR if you configured it in Makefile:
# make ps

Troubleshooting

If you encounter bind: address already in use or Windows WinNAT port issues:

1. Open the .env file in the root directory.
2. Change the conflicting port (e.g., change KIBANA_PORT from 5601 to 5602).
3. Run make run again.

Kubernetes Deployment

LogPulse ships with a complete set of Kubernetes manifests under k8s/ for running the full stack in any Kubernetes cluster (local via minikube/kind or cloud-managed).

Architecture overview

Manifest	Purpose
k8s/config-secret.yaml	ConfigMap (non-secret env vars) + Secret (DB credentials)
k8s/backends.yaml	Stateful backends: MySQL, Redis, Zookeeper, Kafka, Elasticsearch
k8s/app.yaml	logpulse-app Deployment + ClusterIP Service
k8s/hpa.yaml	HorizontalPodAutoscaler (CPU-based, 2–6 replicas)

Design note: Backends use single-replica Deployments with emptyDir volumes (data is ephemeral) — appropriate for demo/testing. In production, use StatefulSet + PVC or a managed cloud service. The Kubernetes Service names (mysql, redis, kafka, elasticsearch) are intentionally identical to the Docker Compose service names so the app config requires zero changes between environments.

K8s Prerequisites

• A running Kubernetes cluster (minikube, kind, Docker Desktop, or cloud-managed).
• kubectl configured to target your cluster.
• The logpulse:v1 image available in your cluster's registry (or built locally with docker build -t logpulse:v1 . and loaded via minikube image load logpulse:v1).
• Metrics Server installed (required for HPA): kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Deploy

Apply the manifests in order:

# 1. Create config and secrets
kubectl apply -f k8s/config-secret.yaml

# 2. Deploy stateful backends (MySQL, Redis, Kafka, Elasticsearch)
kubectl apply -f k8s/backends.yaml

# 3. Deploy the LogPulse application
kubectl apply -f k8s/app.yaml

# 4. Enable Horizontal Pod Autoscaler
kubectl apply -f k8s/hpa.yaml

Verify everything is running:

kubectl get pods
kubectl get svc
kubectl get hpa

Access the API (port-forward for local testing):

kubectl port-forward svc/logpulse 8080:80
# API now available at http://localhost:8080

Horizontal Pod Autoscaler

The HPA automatically scales logpulse-app between 2 and 6 replicas based on average CPU utilization:

Setting	Value
Min replicas	2
Max replicas	6
Scale-up trigger	Average CPU > 50%

Under load (e.g., a k6 stress test), the HPA detects CPU pressure and provisions additional pods within seconds, then scales back down when traffic subsides.

# Watch the HPA in real-time during a load test
kubectl get hpa logpulse-hpa --watch

Teardown

kubectl delete -f k8s/hpa.yaml
kubectl delete -f k8s/app.yaml
kubectl delete -f k8s/backends.yaml
kubectl delete -f k8s/config-secret.yaml

API Usage Examples

Quick Test (VS Code)

We provide an apiTest.http file for convenient testing directly within VS Code.

1. Install the REST Client extension.
2. Open the apiTest.http file in this repository.
3. Click the Send Request link that appears above each API call to interact with your running services.

1. Ingest a Log (Producer)

Send a log entry to the system. The API will respond immediately (Async).

curl -X POST http://localhost:8080/logs \
  -H "Content-Type: application/json" \
  -d '{
    "service": "payment-service",
    "level": "error",
    "message": "Transaction failed due to timeout",
    "timestamp": "2023-12-05T10:00:00Z"
  }'

2. Search Logs (Consumer & Reader)

Search logs via Elasticsearch.

curl "http://localhost:8080/logs/search?q=timeout&level=error"

Key Features

• High Concurrency Ingestion: Utilizing Kafka as a buffer to handle traffic spikes and prevent database overload (Peak Shaving).
• Full-Text Search: Integrated Elasticsearch for efficient log indexing and fuzzy search capabilities (CQRS Pattern).
• Rate Limiting: Implemented Redis (Token Bucket / Counter) to protect the API from abuse (DDoS protection).
• Clean Architecture: Codebase structured into Controller, Service, and Repository layers with Dependency Injection, ensuring testability and maintainability.
• Fully Containerized: "One-Click Deployment" for the entire stack (App, DB, Broker, Search) using Docker Compose.
• Kubernetes Ready: Full Kubernetes manifests for production-grade deployments with ConfigMap/Secret separation and CPU-based Horizontal Pod Autoscaling (2–6 replicas).
• CI/CD Pipeline: Automated linting, testing, and image building via GitHub Actions.
• DevOps Ready: Implemented Graceful Shutdown, Health Checks (liveness/readiness probes), and resource limits for zero-downtime deployments.

Architecture

The system follows an Event-Driven Architecture with separated read/write paths:

graph TB
    Client[Client]
    Nginx[Nginx Load Balancer]
    
    API["API Cluster<br/>(x3 Replicas)"]
    Kafka[Kafka Broker<br/>3 Partitions]
    Worker["Worker Cluster<br/>(x3 Consumers)"]
    
    Redis[(Redis<br/>Cache)]
    MySQL[(MySQL<br/>Primary DB)]
    ES[(Elasticsearch<br/>Search Engine)]
    
    %% Write Flow
    Client -->|POST /logs| Nginx
    Nginx -->|Round Robin| API
    API -->|Rate Limit Check| Redis
    API -.->|Async Push| Kafka
    Kafka -->|Batch Pull| Worker
    Worker -->|Persist| MySQL
    Worker -->|Index| ES
    
    %% Read Flow
    API -->|GET /logs/:id<br/>Cache Hit| Redis
    Redis -.->|Cache Miss<br/>Fallback| MySQL
    

Loading

Flow Explanation

Write Path (Async):

1. Client sends log → Nginx load balancer
2. Nginx distributes to one of 3 API replicas (round-robin)
3. API checks rate limit via Redis
4. API pushes log to Kafka (async, returns immediately)
5. Kafka distributes to 3 partitions for parallel processing
6. 3 Workers (consumer group) pull batches from partitions
7. Workers write to MySQL (persistence) and Elasticsearch (search indexing)

Read Path (Sync):

1. Client queries log → Nginx → API replica
2. API checks Redis cache first (Cache-Aside pattern)
3. Cache Hit: Return immediately
4. Cache Miss: Fetch from MySQL, cache in Redis, then return

Why this architecture?

• API Cluster (x3): Handle high concurrency, zero downtime during deployment
• Kafka Broker: Decouples ingestion from storage (peak shaving), allows backpressure handling
• 3 Partitions: Enables parallel consumption by 3 workers for higher throughput
• Worker Cluster (x3): Matches partition count for optimal performance

CI/CD Pipeline: GitHub Actions handles automated linting (GolangCI-Lint), unit testing, and Docker image building/pushing on every code push.

Performance Benchmarks

Load testing results using k6 with up to 600 concurrent VUs over a 7-minute stress test.

Data Validation

Elasticsearch Document Inspection: Expanded view of an ingested log entry showing all indexed fields and values. This verifies that the complete data pipeline (Go API → Kafka → Elasticsearch) preserves data integrity and correctly maps structured fields (service_name, level, message, timestamp, etc.) for full-text search capabilities.

Stress Test Analysis

Scenario: 600 concurrent VUs generating mixed workloads (write/read/search operations) over 7 minutes.

Key Observations:

• Sustained Throughput: 1,768 req/s average across all operations
• System Stability: Consistent performance throughout the test duration (flat response time curve indicates no memory leaks or degradation)
• Peak Traffic Handling: Successfully processed 743,453 total requests with 99.93% write success rate
• Latency Control: P95 response time maintained below 300ms even under peak load

k6 Load Test Results

Metric	Result
Total Requests	743,453
Throughput	1,768 req/s
Write Success Rate	99.93%
Read Success Rate	100.00%
Search Success Rate	100.00%
P95 Latency	292.41ms
Avg Response Time	211.63ms

Detailed Test Report

█ THRESHOLDS

  http_req_duration
  ✓ 'p(95)<1000' p(95)=292.41ms

  read_success_rate
  ✓ 'rate>0.90' rate=100.00%

  search_success_rate
  ✓ 'rate>0.90' rate=100.00%

  write_success_rate
  ✓ 'rate>0.90' rate=99.93%

█ TOTAL RESULTS

  checks_total.......: 756080 1798.735725/s
  checks_succeeded...: 99.94% 755651 out of 756080
  checks_failed......: 0.05%  429 out of 756080

  ✗ write status is 201
    ↳  99% — ✓ 702559 / ✗ 429
  ✓ read status is 200 or 404
  ✓ search status is 200
  ✓ search returns array

  CUSTOM
  read_duration..................: avg=256ms   p(95)=314ms
  search_duration................: avg=339ms   p(95)=400ms
  write_duration.................: avg=207ms   p(95)=291ms

  HTTP
  http_req_duration..............: avg=211.63ms p(95)=292.41ms
  http_req_failed................: 0.05% 429 out of 743453
  http_reqs......................: 743453 1768.695735/s

  EXECUTION
  iterations.....................: 743352 1768.455453/s
  vus............................: max=600
  running time...................: 7m00.3s

Run Sample Stress Test

We use k6 for load testing. Install k6 and run:

# Install k6 (macOS)
brew install k6

# Install k6 (Windows via Chocolatey)
choco install k6

# Run the stress test (ensure the app is running first)
k6 run test/k6/stress_test.js

Test Scenarios Configuration

The stress test includes 3 concurrent scenarios:

Scenario	Peak VUs	Duration	Description
write_load	500	7 min	High-throughput log ingestion via POST /logs
read_load	60	4 min	Random log retrieval via GET /logs/:id
search_load	40	4 min	Keyword search via GET /logs/search?q=

Thresholds:

• P95 response time < 1000ms
• Write/Read/Search success rate > 90%

Test Data Samples

// Services tested
const services = ['auth-service', 'payment-service', 'order-service', 'user-service', 'notification-service'];

// Log levels
const levels = ['INFO', 'WARN', 'ERROR', 'DEBUG'];

// Sample messages
const messages = [
    'User login successful via OAuth',
    'Database connection timeout during transaction',
    'Processing order items',
    'Cache miss, fetching from database',
    'Payment processed successfully',
    // ...
];

// Search keywords
const searchKeywords = ['login', 'timeout', 'order', 'payment', 'ERROR', 'auth-service', 'user'];

Rate Limiting

LogPulse implements a Redis-based Token Bucket rate limiter to protect the API from excessive traffic and DDoS attacks.

Features

• Token Bucket Algorithm: Smooth rate limiting with configurable burst capacity
• Distributed: Powered by Redis, ensuring consistent rate limiting across all API replicas
• Graceful Response: Returns 429 Too Many Requests when limit is exceeded

How it works:

The Token Bucket algorithm allows a configurable burst capacity (default: 100 tokens) for handling traffic spikes, then limits requests to a sustained rate (default: 50 requests/sec). Each request consumes 1 token. When the bucket is empty, requests are rejected with 429 Too Many Requests until tokens refill.

Configuration

Edit the .env file in the root directory to adjust rate limiting:

# --- Rate Limiting (Token Bucket) ---
RATE_LIMIT_ENABLED=true
RATE_LIMIT_CAPACITY=100    # Max burst requests (bucket capacity)
RATE_LIMIT_RATE=50         # Tokens per second refill rate

After modifying parameters, restart the application:

make restart

Testing

Run the dedicated k6 test to verify rate limiting behavior:

k6 run test/k6/ratelimit_test.js

Expected output: With default config (capacity=100, rate=50/sec), approximately 12-15% of requests should be allowed, and 85-88% rate limited.

Design Decisions & Trade-offs

• Why Kafka over RabbitMQ?
  • LogPulse requires high-throughput sequential writing. Kafka's log-based storage offers superior performance for peak shaving (100k+ msg/sec) compared to RabbitMQ's complex routing.
• Why Elasticsearch?
  • MySQL performs poorly on fuzzy text search (LIKE %...%). ES provides Inverted Indexing, enabling O(1) search complexity for log keywords.
• Hybrid Data Strategy (The "Write-Async, Read-Aside" Pattern)
  • Ingestion (Write): We use Asynchronous Write via Kafka. This ensures the API remains low-latency (<10ms) even if the storage layer is under heavy load.
  • Retrieval (Read): We employ the Cache-Aside Pattern for specific log retrieval. Data is loaded into Redis only upon request (Lazy Loading), optimizing memory usage by not caching the entire log stream.

Project Layout

The project follows the Standard Go Project Layout:

.
├── cmd/
│   └── api/
│       └── main.go            # Application entry point
├── configs/
│   └── config.yaml            # Configuration file
├── deployments/
│   └── docker-compose.yml     # Docker Compose infrastructure definition
├── k8s/
│   ├── config-secret.yaml     # ConfigMap + Secret
│   ├── backends.yaml          # MySQL, Redis, Zookeeper, Kafka, Elasticsearch
│   ├── app.yaml               # LogPulse Deployment + ClusterIP Service
│   └── hpa.yaml               # Horizontal Pod Autoscaler (2–6 replicas)
├── internal/
│   ├── config/                # Configuration loading
│   ├── domain/                # Domain models
│   ├── handler/               # HTTP Handlers (Gin)
│   ├── repository/            # Data Access (MySQL, Redis, ES, Kafka)
│   └── service/               # Business Logic
├── pkg/
│   └── utils/                 # Shared utilities
├── nginx/
│   └── nginx.conf             # Nginx Load Balancer Configuration
├── test/
│   └── k6/
│       ├── stress_test.js     # k6 stress test (600 VUs, 7 min)
│       ├── ratelimit_test.js  # k6 rate limit validation test
│       └── testReport/        # Saved k6 test results
├── .env                       # Environment variables (auto-created by 'make run')
├── .golangci.yml              # Linting configuration
├── Dockerfile                 # Container definition
├── Makefile                   # Management commands
└── README.md

License

Distributed under the MIT License. See LICENSE for more information.