Kafka vs. Redis Streams Benchmark 🚀

A high-performance benchmarking tool written in Go to compare the throughput and latency of Apache Kafka and Redis Streams.

This project is built for both local testing (via Docker) and cloud-scale experiments (via Google Cloud Platform).

---

📊 Why This Project?

Modern distributed systems often need to choose between Kafka and Redis Streams for message streaming. This project provides a practical benchmark to measure:

• Throughput: Messages per second successfully delivered.
• Latency: End-to-end processing delay (P95).
• Resource Usage: CPU and memory consumption.

Run the benchmark under configurable loads to gain insights into which system best fits your performance requirements.

---

🛠️ Core Technologies

• Language: Go
• Messaging Systems: Apache Kafka, Redis Streams
• Containerization: Docker & Docker Compose
• Optional Cloud Deployment: Google Cloud Platform (Cloud Run, Cloud Build, Memorystore, Confluent Cloud)

---

⚡ Quick Start

Prerequisites

• Go 1.21+
• Docker
• Docker Compose

---

Local Setup & Execution

1. Clone the Repository

   git clone https://github.com/Zyx-98/message-queue-benchmark.git
   cd message-queue-benchmark

2. Start the Services
   This will spin up Kafka, Zookeeper, and Redis containers with preconfigured resource limits (2 CPUs, 4GiB RAM).

   docker-compose up -d
   docker-compose ps   # verify services are running

3. Install Go Dependencies

   go mod tidy

4. Run Benchmarks

   # Kafka benchmark
   go run . kafka
   
   # Redis Streams benchmark
   go run . redis

👉 Once complete, results (throughput & latency) are printed in the terminal.

---

Monitor Performance Locally

Track real-time CPU and RAM usage:

docker stats kafka redis

---

☁️ Cloud Deployment (GCP)

This project includes configs for Cloud Build and Cloud Run Jobs, making it easy to run large-scale benchmarks.

• CI/CD:
  • cloudbuild.yaml builds & pushes Docker images to Artifact Registry.
  • Automatically triggers Cloud Run jobs.
• Infrastructure:
  • Kafka: Managed service (e.g., Confluent Cloud).
  • Redis: Google Cloud Memorystore.
  • Compute: Cloud Run (Jobs).
• Metrics:
  • Throughput & Latency → available in Cloud Run job logs.
  • CPU & RAM → visible in the GCP Console under job metrics.

---

⚙️ Configuration

Benchmark parameters can be tuned in config/config.go:

Parameter	Description	Example
MessageSize	Size of each message in bytes	1024
MessageRate	Target messages per second	10000
TestDuration	Total benchmark duration (in seconds)	60

---

📑 Output

Redis

Throughput: 6409.98 msg/s
P95 Latency: 9 ms
CPU Usage: 41.12%
Memory Usage: 2.1 GiB

---