🛡️ Network Security — Phishing URL Detection System

An end-to-end Machine Learning system for detecting phishing URLs — built with a production-grade MLOps pipeline.

---

📌 Overview

Network Security is a full-stack MLOps project that detects phishing websites using machine learning. It goes beyond a simple notebook — implementing a complete, production-ready pipeline covering data ingestion from MongoDB, data validation with drift detection, feature transformation, model training with hyperparameter tuning, experiment tracking via MLflow & DagsHub, and deployment as a REST API using FastAPI inside a Docker container with CI/CD via GitHub Actions.

🔍 The model analyzes network-level features of URLs to classify them as phishing or legitimate — a real-world cybersecurity problem with direct industry relevance.

---

✨ Key Highlights

Feature	Description
🗄️ Data Source	Phishing dataset stored and fetched from MongoDB Atlas
✅ Data Validation	Schema checks + KS-test based data drift detection
⚙️ Feature Engineering	KNN Imputer for missing values, preprocessing pipeline serialized as .pkl
🤖 Model Selection	Automated comparison of 5 classifiers with GridSearchCV
📊 Experiment Tracking	All runs tracked via MLflow integrated with DagsHub
🚀 REST API	Full prediction & training endpoints via FastAPI
🐳 Containerized	Fully Dockerized application
🔄 CI/CD	Automated pipeline via GitHub Actions

---

🏗️ System Architecture

Raw Data (CSV)
      │
      ▼
┌─────────────────┐
│  MongoDB Atlas  │  ← Data Storage
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Data Ingestion │  ← Pulls data, splits into Train/Test
└────────┬────────┘
         │
         ▼
┌──────────────────┐
│ Data Validation  │  ← Schema validation + Drift Detection (KS-Test)
└────────┬─────────┘
         │
         ▼
┌────────────────────┐
│ Data Transformation│  ← KNN Imputer, Preprocessing → saved as .pkl
└────────┬───────────┘
         │
         ▼
┌─────────────────┐
│  Model Trainer  │  ← Trains 5 models, selects best, logs to MLflow
└────────┬────────┘
         │
         ▼
┌──────────────────┐
│  FastAPI Server  │  ← /train and /predict endpoints
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│  Docker + CI/CD  │  ← Containerized & deployed via GitHub Actions
└──────────────────┘

---

🧠 ML Pipeline — Deep Dive

1. 📥 Data Ingestion

• Connects to MongoDB Atlas and exports the NetworkData collection as a Pandas DataFrame
• Handles missing values (na → NaN) and drops internal MongoDB _id field
• Splits data into train (80%) / test (20%) and saves to the artifact store

2. ✅ Data Validation

• Validates that train and test datasets conform to the expected schema (column count, column names)
• Detects data drift using the Kolmogorov-Smirnov (KS) test on each feature column
• Generates a drift report (report.yaml) flagging drifted columns with p-values

3. 🔧 Data Transformation

• Applies KNN Imputer (k=3, weights=uniform) to handle missing values
• Serializes the complete preprocessing pipeline as preprocessing.pkl for inference consistency

4. 🤖 Model Training & Selection

Trains and evaluates 5 classification models using cross-validated GridSearchCV:

Model	Hyperparameters Tuned
Random Forest	n_estimators
Decision Tree	criterion
Gradient Boosting	learning_rate, subsample, n_estimators
Logistic Regression	—
AdaBoost	learning_rate, n_estimators

• Selects the best model by test score
• Evaluates with F1-Score, Precision, and Recall
• Logs all metrics and models to MLflow / DagsHub

5. 📊 Experiment Tracking (MLflow + DagsHub)

• Every training run is logged with metrics: f1_score, precision, recall
• Models are registered in the MLflow Model Registry
• Experiments are tracked remotely on DagsHub

---

🚀 API Endpoints

Once the server is running, navigate to http://localhost:8000/docs for the interactive Swagger UI.

Method	Endpoint	Description
GET	/	Redirects to Swagger UI
GET	/train	Triggers the full training pipeline
POST	/predict	Upload a CSV file and get phishing predictions

Prediction Response

The /predict endpoint accepts a CSV file, runs inference using the saved model + preprocessor, and returns an HTML table with a predicted_column appended (1 = Phishing, -1 = Legitimate).

---

🗂️ Project Structure

NetworkSecurity/
├── networksecurity/
│   ├── components/               # Core ML pipeline stages
│   │   ├── data_ingestion.py
│   │   ├── data_validation.py
│   │   ├── data_transformation.py
│   │   └── model_trainer.py
│   ├── pipeline/
│   │   └── training_pipeline.py  # Orchestrates full pipeline
│   ├── entity/
│   │   ├── config_entity.py      # Configuration dataclasses
│   │   └── artifact_entity.py    # Artifact dataclasses
│   ├── constants/
│   │   └── training_pipeline/    # All constants & hyperparams
│   ├── utils/                    # Utility functions
│   └── exception/ & logging/     # Custom exception & logger
├── templates/
│   └── table.html                # Prediction results UI
├── app.py                        # FastAPI application
├── main.py                       # Run pipeline manually
├── push_data.py                  # Load CSV data into MongoDB
├── Dockerfile
├── requirements.txt
├── setup.py
└── .github/workflows/main.yml    # CI/CD pipeline

---

⚙️ Setup & Installation

Prerequisites

• Python 3.10+
• MongoDB Atlas account
• DagsHub account (for MLflow tracking)
• Docker (optional, for containerized run)

1. Clone the Repository

git clone https://github.com/RudrakshJani/NetworkSecurity.git
cd NetworkSecurity

2. Create a Virtual Environment

python -m venv venv
source venv/bin/activate        # Linux/macOS
venv\Scripts\activate           # Windows

3. Install Dependencies

pip install -r requirements.txt

4. Configure Environment Variables

Create a .env file in the project root:

MONGODB_URL_KEY=your_mongodb_atlas_connection_string
MLFLOW_TRACKING_URI=your_dagshub_mlflow_uri
MLFLOW_TRACKING_USERNAME=your_dagshub_username
MLFLOW_TRACKING_PASSWORD=your_dagshub_token

5. Push Data to MongoDB (First-time setup)

python push_data.py

6. Run the Application

python app.py

Visit http://localhost:8000/docs to access the API.

---

🐳 Docker

Build the Image

docker build -t networksecurity:latest .

Run the Container

docker run -p 8000:8000 --env-file .env networksecurity:latest

---

🔄 CI/CD Pipeline

This project uses GitHub Actions for continuous integration. On every push to main:

1. Checkout — Pulls the latest code
2. Lint — Code quality check
3. Unit Tests — Automated test execution

The workflow is defined in .github/workflows/main.yml.

---

🛠️ Tech Stack

Category	Technology
Language	Python 3.10+
ML Framework	Scikit-learn
API Framework	FastAPI + Uvicorn
Database	MongoDB Atlas
Experiment Tracking	MLflow + DagsHub
Containerization	Docker
CI/CD	GitHub Actions
Data Processing	Pandas, NumPy
Config Management	PyYAML, python-dotenv

---

📈 Why This Project Stands Out

• ✅ Not just a notebook — fully modular, production-grade codebase
• ✅ MLOps best practices — artifact management, experiment tracking, model registry
• ✅ Real-world problem — phishing detection is a top cybersecurity challenge
• ✅ Scalable design — pipeline components are loosely coupled and independently testable
• ✅ End-to-end — from raw data in MongoDB to a live REST API in Docker

---

👨‍💻 Author

Rudraksh Jani 📧 rudrakshjani05@gmail.com

---

📄 License

This project is licensed under the MIT License — see the LICENSE file for details.

---

If you found this project useful or interesting, please consider giving it a ⭐ on GitHub!