High-Throughput Distributed Log Ingestor

A scalable, distributed system designed to handle high-volume log ingestion using Golang, Apache Kafka, and PostgreSQL. This project demonstrates an event-driven architecture that decouples log reception from storage, ensuring low latency for clients and high durability for data.

Architecture

The system is split into three core components:

1. The API Producer (Go): A REST API that accepts logs via HTTP. It utilizes a Worker Pool pattern to handle concurrent requests efficiently and pushes messages to a Kafka topic.
2. The Message Broker (Kafka): Acts as a durable buffer, handling backpressure and ensuring no logs are lost even if the database is temporarily down.
3. The Consumer (Go): A separate microservice that reads from Kafka in real-time and persists structured logs into PostgreSQL.

graph LR
    Client[Client / Script] -- HTTP POST --> API[Go API Server]
    subgraph "Producer Service"
        API -- Channel --> Workers[Worker Pool]
        Workers -- Push --> Kafka
    end
    
    Kafka[Apache Kafka] -- Pull --> Consumer[Go Consumer Service]
    
    subgraph "Consumer Service"
        Consumer -- Insert --> DB[(PostgreSQL)]
    end

Loading