CyberLogIQ - Advanced Log Analysis Platform

CyberLogIQ is a powerful, web-based log analysis platform designed for security professionals and system administrators. It provides comprehensive log ingestion, analysis, and reporting capabilities with a focus on security threat detection and operational intelligence.

🚀 Key Features

📊 Multi-Format Log Ingestion

• Supported Formats: Apache, Nginx, Syslog, Windows Event Logs, SSH, Firewall logs, JSON, CSV
• Large File Support: Handle files up to 6GB with optimized memory management
• Automatic Format Detection: Intelligent parsing for heterogeneous log sources

🔍 Advanced Security Analysis

• Threat Detection: SQL injection, XSS, LFI/RFI, command injection, brute force attacks, DDoS patterns
• Real-time Analysis: Live monitoring and immediate threat identification
• Anomaly Detection: Behavioral analysis and outlier detection
• IP Geolocation: Visual mapping of suspicious activities

📈 Comprehensive Reporting

• Interactive Dashboards: Real-time data visualization and metrics
• Custom SQL Queries: Direct database access with pagination support
• Export Capabilities: CSV, JSON, and PDF report generation
• Scheduled Reports: Automated reporting for regular security audits

🛡️ Enterprise-Grade Security

• Role-Based Access Control: Admin, Analyst, and Read-only user roles
• Secure Authentication: CSRF protection and session management
• Data Isolation: Separate workspaces and access controls
• Audit Logging: Comprehensive activity tracking

🏗️ Architecture Overview

Technology Stack

Backend:

• Framework: Flask 3.0.3 with Werkzeug
• Database: SQLite with WAL journaling for high concurrency
• Data Processing: Pandas for efficient data manipulation
• Concurrency: Gevent for asynchronous operations
• Reporting: ReportLab for PDF generation

Frontend:

• UI Framework: Bootstrap 5.1.3 with responsive design
• Charts & Visualizations: Chart.js and custom D3.js components
• Data Tables: DataTables with server-side processing
• Maps: Leaflet for geolocation visualization
• Icons: Font Awesome 6.0.0

Application Structure

CyberLogIQ/
├── app/                          # Application core
│   ├── __init__.py              # Flask app factory
│   ├── config.py                # Configuration management
│   ├── routes.py                # API and UI endpoints
│   └── utils/                   # Business logic
│       ├── database_manager.py  # Database operations
│       ├── log_parser.py        # Log parsing engine
│       ├── report_generator.py  # Report generation
│       └── security_analyzer.py # Threat detection
├── databases/                   # SQLite database files
├── static/                      # Static assets
│   ├── css/style.css           # Custom styles
│   └── js/script.js            # Frontend logic
├── templates/                   # Jinja2 templates
│   ├── analysis/               # Analysis views
│   ├── base.html               # Main layout
│   └── various pages           # UI components
├── uploads/                     # Uploaded log files
└── tests/                       # Test suite

📦 Installation

Prerequisites

• Python 3.10 or higher
• 4GB RAM minimum (8GB recommended for large files)
• 10GB+ free disk space for log storage
• Network access for CDN resources

Quick Start

1. Clone the repository

   git clone <repository-url>
   cd CyberLogIQ

2. Create virtual environment

   python -m venv .venv
   source .venv/bin/activate  # Linux/Mac
   # or
   .venv\Scripts\activate    # Windows

3. Install dependencies

   pip install -r requirements.txt

4. Configure the application

   # Edit app/config.py for custom settings
   # Set upload limits, database paths, and security settings

5. Run the application

   python app.py

6. Access the application Open your browser and navigate to http://localhost:5000

Docker Deployment (Optional)

docker build -t cyberlogiq .
docker run -p 5000:5000 -v $(pwd)/databases:/app/databases cyberlogiq

🎯 Usage Guide

1. Upload Log Files

• Navigate to the upload interface
• Select one or multiple log files
• Monitor upload progress with real-time feedback
• Files are automatically parsed and stored

2. Analyze Data

• Use the interactive dashboard for overview metrics
• Apply filters and search criteria
• View security threats in dedicated analysis panels
• Explore geographical data on interactive maps

3. Generate Reports

• Create custom SQL queries with the query builder
• Export results to CSV, JSON, or PDF formats
• Schedule regular reports for ongoing monitoring
• Share findings with team members

4. Manage Users

• Admin users can create and manage accounts
• Assign appropriate roles and permissions
• Monitor user activity through audit logs

� User Management

CyberLogIQ features a robust Role-Based Access Control (RBAC) system that enables secure user management with granular permissions.

User Roles and Permissions

🔧 Administrator

• Full system access with unlimited privileges
• User management: Create, edit, disable, and delete user accounts
• System configuration: Modify application settings and security parameters
• Database operations: Purge logs, reset systems, and manage storage
• Audit log access: Review all user activities and system events

🔍 Security Analyst

• Log analysis: Full access to analysis tools and dashboards
• File operations: Upload, process, and analyze log files
• Reporting: Generate and export security reports (PDF, CSV, JSON)
• Threat investigation: Access to all security detection features
• Real-time monitoring: Live log analysis and alert monitoring

👀 Read-Only User

• View-only access to dashboards and reports
• No modification capabilities for any data
• Search and filter functionality for log exploration
• Export limited to pre-approved report templates

User Account Lifecycle

Account Creation

1. Initial Setup: Default admin account created during first launch
2. Admin Creation: Administrators can create new users through the admin interface
3. Role Assignment: Each user is assigned one of the three security roles
4. Temporary Passwords: New users receive temporary credentials with forced password change

Account Management

• Password Policies: Enforced password complexity and expiration rules
• Account Status: Users can be enabled/disabled without deletion
• Session Control: Configurable session timeouts and concurrent login limits
• Audit Trail: All user actions are logged for security review

Security Features

Authentication

• Secure Login: CSRF-protected authentication system
• Session Management: Secure cookie-based sessions with expiration
• Password Hashing: BCrypt-based password storage
• Brute Force Protection: Automatic account locking after failed attempts

Access Control

• Route Protection: All endpoints enforce role-based permissions
• API Security: REST endpoints validate user permissions
• Data Isolation: Users can only access authorized data and functions
• Audit Logging: Comprehensive activity monitoring across all operations

Administrative Interface

The admin dashboard provides comprehensive user management capabilities:

User Management Panel

• User List: View all users with filtering and search capabilities
• Account Creation: Create new users with role assignment
• Profile Editing: Modify user details, roles, and status
• Password Reset: Force password resets for security compliance
• Account Deactivation: Temporarily disable accounts without deletion

Security Settings

• Role Configuration: Define and modify permission sets
• Password Policies: Set complexity requirements and expiration rules
• Session Settings: Configure timeout durations and security policies
• Audit Review: Access comprehensive activity logs and reports

Best Practices

1. Least Privilege: Assign users only the permissions they need
2. Regular Reviews: Periodically audit user accounts and permissions
3. Password Rotation: Enforce regular password changes for sensitive roles
4. Monitoring: Regularly review audit logs for suspicious activities
5. Backup: Maintain backups of user configuration and permission sets

API Access

User management is also available through REST API endpoints (admin role required):

# List all users
GET /api/users

# Create new user
POST /api/users
{
    "username": "newuser",
    "role": "analyst",
    "email": "user@example.com"
}

# Update user permissions
PUT /api/users/{user_id}
{
    "role": "readonly",
    "is_active": true
}

�🔧 Configuration

Key Configuration Options

Edit app/config.py to customize:

# Upload settings
UPLOAD_FOLDER = 'uploads/'
MAX_CONTENT_LENGTH = 6 * 1024 * 1024 * 1024  # 6GB

# Database settings
DATABASE_PATH = 'databases/logs.db'

# Security settings
SESSION_TIMEOUT = 3600  # 1 hour
CSRF_ENABLED = True

# Analysis settings
THREAT_DETECTION_RULES = {
    'sql_injection': True,
    'xss': True,
    'brute_force': True
}

Environment Variables

Set these environment variables for production deployment:

export FLASK_ENV=production
export SECRET_KEY=your-secret-key
export DATABASE_URL=sqlite:///databases/production.db

🧪 Testing

Run the test suite to ensure everything works correctly:

python -m pytest test_database_manager.py
python -m pytest test_log_parser.py 
python -m pytest test_security.py

🚀 Performance Optimization

For Large Log Files

• Enable WAL mode in SQLite for better concurrency
• Use appropriate indexing strategies
• Implement chunked processing for memory efficiency
• Consider SSD storage for database files

Database Optimization

-- Recommended indexes
CREATE INDEX idx_log_timestamp ON log_entries(timestamp);
CREATE INDEX idx_log_source_ip ON log_entries(source_ip);
CREATE INDEX idx_log_severity ON log_entries(severity);

🔒 Security Considerations

Best Practices

• Always change default admin credentials
• Use HTTPS in production environments
• Regularly update dependencies for security patches
• Implement proper firewall rules for the application
• Monitor and rotate audit logs regularly

Data Protection

• Log files may contain sensitive information
• Implement appropriate access controls
• Consider encryption for sensitive data storage
• Regular security audits recommended

📊 Monitoring and Maintenance

Health Checks

• Monitor database growth and performance
• Check disk space for uploads directory
• Review application logs for errors
• Regular backup of database files

Performance Metrics

• Response times for key operations
• Memory usage during large file processing
• Database query performance
• User activity and system load

🤝 Contributing

We welcome contributions! Please follow these guidelines:

1. Fork the repository
2. Create a feature branch
3. Write tests for new functionality
4. Ensure code follows existing style
5. Submit a pull request with description

Development Setup

git clone <fork-url>
cd CyberLogIQ
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pip install -r requirements-dev.txt  # for development tools

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🆘 Support

For support and questions:

• Check the documentation in this README
• Review existing issues on GitHub
• Create a new issue for bugs or feature requests
• Contact the maintainers for security concerns

🔮 Roadmap

Planned Features

• Real-time log streaming support
• Advanced machine learning anomaly detection
• Integration with SIEM systems
• Custom alert rules and notifications
• Multi-tenant support
• API for external integrations
• Enhanced visualization options

Known Limitations

• Concurrent user limitations for very large operations
• Memory usage scales with log file size
• Geographic data requires external GeoLite2 database
• Some advanced parsing may require custom rules

---

CyberLogIQ - Making log analysis accessible and powerful for security professionals worldwide. 🛡️