LogAI - Multi-Stage Hybrid Log Classifier

LogAI is a hybrid log classification system that combines traditional techniques with advanced NLP models (Sentence Transformers and LLaMA 3.3) to automatically cluster and categorize system logs.

🚀 Features

• Accepts raw logs via CSV input
• Converts log messages into embeddings using Sentence Transformers
• Clusters similar logs using DBSCAN
• Applies Regex-based classification for known log patterns
• Uses:
  • Sentence Transformers + Logistic Regression for medium-sized unknown clusters
  • LLaMA 3.3 via GroqCloud API for rare/unclassified logs
• Exposes a FastAPI /classify endpoint to automate the entire process
• Outputs the results into output.csv

---

🔧 Tech Stack

• Python
• Sentence Transformers
• DBSCAN (from scikit-learn)
• Regex (Python's re module)
• LLaMA 3.3 (via GroqCloud API)
• FastAPI
• Uvicorn

---

📦 Setup

1. Clone the repo

   git clone https://github.com/rprahadeep/LogAI.git
   cd LogAI

2. Install dependencies

   pip install -r requirements.txt

3. Configure environment variables

   Create a .env file in the root directory:

   GROQ_API_KEY=your_groq_api_key

---

▶️ Run the FastAPI Server

uvicorn server:app --reload

---

📬 API Usage

POST /classify

• Description: Upload a CSV file containing a logs column
• Response: Saves output.csv locally with an added category column

Sample Request (using curl):

curl -X POST "http://127.0.0.1:8000/classify" \
  -F "file=@logs.csv"

---

📁 Output

The server will generate an output.csv file with the original logs and their predicted categories.

---

🧠 Model Strategy

1. Embeddings: Convert logs into vector space using Sentence Transformers
2. Clustering: Use DBSCAN to identify log groupings
3. Classification:
   • Regex for known patterns
   • Sentence Transformer + Logistic Regression for common unknowns
   • LLaMA 3.3 for rare or uncategorized logs

---