Skip to content
BranchesTags

Repository files navigation

LogSage 🌲

AI-powered log analysis with ML anomaly detection and intelligent root cause analysis.

React FastAPI Python Tailwind ChromaDB

A production-grade log analysis platform combining Drain3 clustering, Isolation Forest ML anomaly detection, ChromaDB vector memory, and Gemini AI for intelligent root cause analysis with code location extraction.

Why LogSage?

  • ML-Powered Anomaly Detection — Isolation Forest with 5-feature analysis
  • Root Cause Analysis — Extracts exact code locations (file:line:function) from logs
  • Vector Memory — ChromaDB stores incident history for learning and fast retrieval
  • Multi-Format Support — DLT automotive logs, standard text logs, JSON, syslog
  • Health Scoring — Multi-factor weighted scoring (-50% critical, -30% errors, -15% anomalies)
  • Clean Architecture — Modular, scalable, professional codebase

Quick Start

Prerequisites

  • Node.js 18+
  • Python 3.10+
  • Gemini API key, optional (create at https://aistudio.google.com/apikey) — parsing, clustering, and anomaly detection work without it; AI insights and RCA need it
  1. Backend
cd backend
cp .env.example .env         # add your GEMINI_API_KEY here (optional)
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python api.py

Backend: http://localhost:8000 — API docs: http://localhost:8000/docs

  1. Frontend
# from project root
npm install
npm run dev

Frontend: http://localhost:5173 (API calls are proxied to the backend automatically)

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        Frontend (React)                          │
│  ┌─────────────┐  ┌──────────────┐  ┌─────────────────────┐   │
│  │  Dashboard  │  │  AnomalyPanel│  │  LoadingAnimation   │   │
│  └─────────────┘  └──────────────┘  └─────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘
                              ↓ API Calls
┌─────────────────────────────────────────────────────────────────┐
│                      Backend (FastAPI)                           │
│  ┌────────────────┐  ┌────────────────┐  ┌──────────────────┐ │
│  │  api.py        │  │  rca_engine.py │  │  chroma_service  │ │
│  │  - Upload      │  │  - ML Analysis │  │  - Vector Store  │ │
│  │  - Health      │  │  - Code Extract│  │  - Embeddings    │ │
│  └────────────────┘  └────────────────┘  └──────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
         ↓                     ↓                      ↓
┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐
│   Drain3 Engine  │  │  Isolation Forest│  │   Gemini 2.5     │
│   - Clustering   │  │  - 5 Features    │  │   - RAG Analysis │
│   - Templates    │  │  - Contamination │  │   - Confidence   │
└──────────────────┘  └──────────────────┘  └──────────────────┘

Core Features

1. ML Anomaly Detection

  • Isolation Forest with 5-feature analysis (frequency, cluster size, template length, wildcards, error keywords)
  • Hybrid approach combining ML predictions with heuristics (rare patterns, error keywords, small clusters)
  • Dynamic contamination rate (50-70% for small datasets, 45% for larger)
  • Severity classification (High/Medium/Low) based on anomaly scores

2. Root Cause Analysis (RCA)

  • Code location extraction using 6+ regex patterns (Java, Python, C#, generic file:line)
  • Frequency analysis to find primary code location with confidence scores
  • Vector similarity search in ChromaDB to find similar past incidents
  • Gemini-powered diagnosis with context from code locations and historical data
  • ChromaDB caching for instant retrieval of previously analyzed incidents

3. Multi-Format Log Processing

  • DLT automotive logs (Date||Time||ECU||AppID||Message format)
  • Standard text logs (syslog, application logs)
  • JSON logs with structured data
  • Auto-detection and normalization across formats

4. Health Scoring System

health_score = 100.0
health_score -= critical_ratio * 50   # Critical errors: -50% max
health_score -= error_ratio * 30      # Errors: -30% max  
health_score -= anomaly_ratio * 15    # Anomalies: -15% max
health_score -= warning_ratio * 5     # Warnings: -5% max
health_score -= high_severity_ratio * 10  # High severity: -10% max

5. Vector Memory (ChromaDB)

  • Incident memory collection stores all RCA results with embeddings
  • Similarity search retrieves similar past incidents for context
  • Cluster insights caches AI-generated insights for fast retrieval
  • Persistent storage in backend/chroma_db/ directory

API Endpoints

Method Endpoint Purpose
GET /api/health Check API health, Gemini config, ChromaDB status
POST /api/upload Upload & parse logs with Drain3 + Isolation Forest
POST /api/analyze-rca Root Cause Analysis with code extraction & AI diagnosis
GET /api/rca-history Retrieve historical RCA results from ChromaDB
POST /api/analyze-clusters Generate AI insights for all clusters (Gemini + caching)
POST /api/insight/:cluster_id Get AI insight for specific cluster

RCA Workflow

User clicks "Analyze Root Cause" on anomaly
            ↓
┌─────────────────────────────────────────┐
│ 1. Check ChromaDB Cache                 │
│    - Search by cluster_id               │
│    - Return if found (instant!)         │
└─────────────────────────────────────────┘
            ↓ Cache Miss
┌─────────────────────────────────────────┐
│ 2. Extract Features (5 features)        │
│    - Frequency %, cluster size, etc.    │
└─────────────────────────────────────────┘
            ↓
┌─────────────────────────────────────────┐
│ 3. Isolation Forest ML                  │
│    - Compute anomaly score (0-1)        │
│    - Classify severity (High/Med/Low)   │
└─────────────────────────────────────────┘
            ↓
┌─────────────────────────────────────────┐
│ 4. Extract Code Locations               │
│    - Regex: Java, Python, C#, generic   │
│    - Find primary: file:line:function   │
└─────────────────────────────────────────┘
            ↓
┌─────────────────────────────────────────┐
│ 5. ChromaDB Vector Search               │
│    - Retrieve 5 similar past incidents  │
│    - Get their solutions & RCA results  │
└─────────────────────────────────────────┘
            ↓
┌─────────────────────────────────────────┐
│ 6. Gemini AI Analysis (RAG)             │
│    - Context: code + history + metrics  │
│    - Generate: root cause + confidence  │
└─────────────────────────────────────────┘
            ↓
┌─────────────────────────────────────────┐
│ 7. Store in ChromaDB                    │
│    - Cache for future lookups           │
│    - Add to incident memory             │
└─────────────────────────────────────────┘
            ↓
    Display RCA result with:
    - Root cause diagnosis
    - Code location (file:line:function)
    - Confidence score (0.0-1.0)
    - Recommended action
    - Technical details

Example upload (curl)

curl -X POST http://localhost:8000/api/upload -F "file=@/path/to/logfile.txt"

Example RCA request

curl -X POST http://localhost:8000/api/analyze-rca \
  -H "Content-Type: application/json" \
  -d '{
    "template": "Connection timeout to database",
    "log_samples": ["2025-01-15 10:23:45 ERROR Connection timeout..."],
    "cluster_id": 42,
    "frequency": 0.05,
    "cluster_size": 15
  }'

Project Structure

LogSage/
├── backend/
│   ├── api.py                 # FastAPI endpoints (upload, RCA, health)
│   ├── requirements.txt       # Python dependencies
│   ├── .env.example           # Environment variable template
│   └── src/
│       ├── rca_engine.py      # RCA engine (ML, code extraction, Gemini)
│       ├── chroma_service.py  # ChromaDB vector store wrapper
│       ├── enhanced_parser.py # Drain3 log parser
│       ├── dlt_parser.py      # DLT automotive log parser
│       ├── format_detector.py # DLT vs standard log detection
│       ├── config.py          # Environment-based configuration
│       └── universal_log_processor.py  # Multi-format log processor
├── src/
│   ├── components/
│   │   ├── Dashboard.jsx      # Main dashboard with stats
│   │   ├── AnomalyPanel.jsx   # RCA trigger & display
│   │   ├── LogView.jsx        # Parsed logs display
│   │   ├── Features.jsx       # Features showcase
│   │   └── Docs.jsx           # API documentation
│   └── services/
│       └── api.js             # API client
├── Dockerfile                 # Multi-stage build (frontend + backend)
├── render.yaml                # One-click Render deployment
└── package.json               # Frontend dependencies

Configuration

All configuration is via environment variables (see backend/.env.example):

Variable Default Purpose
GEMINI_API_KEY — Enables AI insights & RCA (optional)
LLM_MODEL gemini-2.5-flash Gemini model to use
PORT 8000 Backend port
CORS_ORIGINS localhost dev ports Comma-separated allowed origins

The frontend uses relative /api URLs — proxied by Vite in dev, served by FastAPI in production. Set VITE_API_URL only if the backend runs on a different origin.

Deployment

The Dockerfile builds the React frontend and serves it from FastAPI as a single container. Deploy to Render in one click:

  1. Push this repo to GitHub
  2. Render Dashboard → New → Blueprint → select the repo (render.yaml is picked up)
  3. Set GEMINI_API_KEY in the environment settings (optional)

Or run it anywhere with Docker:

docker build -t logsage .
docker run -p 8000:8000 -e GEMINI_API_KEY=your-key logsage

Technology Stack

Frontend:

  • React 19.1.1 + Vite
  • Tailwind CSS v4
  • Clean color palette: olive (#154001), deepgreen (#2d6b0f), gold (#877C46)

Backend:

  • FastAPI + Uvicorn
  • Python 3.8+

ML & AI:

  • scikit-learn (Isolation Forest, StandardScaler)
  • Google Generative AI (Gemini 2.5 Flash)
  • ChromaDB (vector store + embeddings)

Log Processing:

  • Drain3 (log clustering & template mining)
  • Universal log processor (multi-format support)

Troubleshooting

  • Port 8000 in use: set PORT to another port, or stop the other process
  • 500 on upload: check file size (<10MB recommended) and backend logs
  • No AI responses: ensure GEMINI_API_KEY is set in backend/.env

Development tips

  • Backend dev with auto-reload: RELOAD=1 python api.py
  • Frontend dev: npm run dev
  • Lint: npm run lint (frontend), PEP8 (backend)

License

MIT

Made with care — the LogSage Team 🌿

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages