Skip to content

Implement various supervised and unsupervised learning algorithms fro… #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Mattral merged 1 commit into from
Jun 3, 2026
Merged

Implement various supervised and unsupervised learning algorithms fro… #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
name: CI

on:
push:
branches: [main]
pull_request:
branches: [main]

jobs:
test:
name: Test Python ${{ matrix.python-version }}
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.10", "3.11", "3.12"]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: pip
- run: pip install -e ".[dev]"
- run: pytest tests/ --cov=mlscratch --cov-report=xml --cov-fail-under=88
- uses: actions/upload-artifact@v4
with:
name: coverage-${{ matrix.python-version }}
path: coverage.xml

lint:
name: Lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
cache: pip
- run: pip install ruff black
- run: ruff check src/ tests/
- run: black --check src/ tests/
283 changes: 114 additions & 169 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,214 +1,159 @@
# ML-AI-Algorithms-from-scratch

# AI, ML, DL, and RL Demystified: From Scratch to Understanding
A structured, educational repository of from-scratch ML/AI/RL/Bayesian algorithms.

Welcome to my comprehensive repository dedicated to unraveling the mysteries of Artificial Intelligence (AI), Machine Learning (ML), Deep Learning (DL), and Reinforcement Learning (RL). 🚀

## Purpose & Positioning

This repository is a **learning-first, from-scratch implementation collection** of core AI, Machine Learning, Deep Learning, Reinforcement Learning, and Bayesian algorithms.

It is designed for readers who:
- Already know *what* these algorithms are
- Want to understand **how they work internally**
- Prefer readable, step-by-step implementations over optimized or production-grade code

This is **not** a production library or benchmarking suite.
Instead, the focus is on **algorithmic intuition, mathematical flow, and code transparency**.


## Who This Repository Is NOT For

This repository may not be ideal if you are looking for:
- Highly optimized or GPU-accelerated implementations
- Drop-in replacements for scikit-learn, PyTorch, or TensorFlow
- State-of-the-art performance benchmarks
- Large-scale dataset pipelines

The goal here is **understanding**, not performance.
This project is evolving from a collection of standalone scripts into a clean, `pip`-installable Python package under `src/mlscratch/`.

---

### How to Navigate This Repository

If you're new to the repository, a recommended learning path is:

1. **Supervised Learning**
- Linear & Logistic Regression
- k-Nearest Neighbors
- Decision Trees
2. **Unsupervised Learning**
- K-Means
- PCA
- Gaussian Mixture Models
3. **Neural Networks**
- Single-Layer Perceptron
- Multi-Layer Perceptron
- CNNs and RNNs
4. **Reinforcement Learning**
- Q-Learning
- Deep Q-Networks
- Policy-based methods
5. **Bayesian Learning**
- Bayesian Inference
- Bayesian Neural Networks

Each folder is self-contained and can be explored independently.
## Current Status

- Standardized package layout under `src/mlscratch/`
- Verified supervised algorithms with `pytest`
- `README.md` updated to reflect current package state
- Added package-level implementations for:
- `LinearRegression`
- `LogisticRegression`
- `LassoRegression`
- `RidgeRegression`
- `KNeighborsClassifier`
- `DecisionTreeClassifier`
- `RandomForestClassifier`
- `GaussianNB`
- `LinearSVMClassifier`
- Next implementation focus: unsupervised algorithms, beginning with `KMeans`

---

# Repo Structure
## Project Structure

```
│
ML-AI-Algorithms-from-scratch/
├── LICENSE
├── README.md <- The top-level README for developers/collaborators using this project.
├── neural_network <- Folder for Neural Network implementations
│ ├── AutoEncoder
│ ├── BoltzmannMachine
│ ├── GenerativeAdversarialNetwork
│ ├── HopfieldNetwork
│ ├── LongShortTermMemoryLSTM
│ ├── MultiLayerPerceptronClassification
│ ├── MultiLayerPerceptronRegression
│ ├── RadialBasisFunctionNetworks
│ ├── SelfAttentionMechanism
│ ├── SimpleCNN
│ ├── SimpleEncoderDecoder
│ ├── SimpleRNN
│ ├── SingleLayerPerceptronClassification
│ ├── SingleLayerPerceptronRegression
│ ├── TitanicSurvialBySingleLayerPerceptron
│ └── Transformer
│
├── reinforcement_learning <- Folder for Reinforcement Learning implementations
│ ├── Deep Deterministic Policy Gradients
│ ├── Deep Q Network
│ ├── Soft Actor Crtic
│ ├── Proximal Policy Optimization
│ └── QLearning
│
├── supervised <- Folder for Supervised Learning implementations
│ ├── DecisionTrees
│ ├── KnearestNeighbour
│ ├── LassoRegression
│ ├── LinearRegression
│ ├── LogisticRegression
│ ├── Naive Bayes
│ ├── RandomForest
│ ├── RidgeRegression
│ └── SupportVectorMachines
│
├── unsupervised <- Folder for Unsupervised Learning implementations
│ ├── AprioriAlgorithm
│ ├── Density-Based Spatial Clustering of Applications with Noise
│ ├── Expectation-Maximization
│ ├── Gaussian Mixture Model
│ ├── HierarchicalClustering
│ ├── IndependentComponentAnalysis
│ ├── K-MedoidsClustering
│ ├── KMeansPlusPlus
│ ├── PrincipalComponentAnalysis
│ ├── SelfOrganizing Map
│ ├── kmeanclustering
│ └── tSNE
│
└── Bayesian Learning
├── BayesianInference
├── BayesianNetwork
├── Gibbs Sampling
├── Metropolis-Hastings Algorithm
├── Bayesian Neural Networks
├── BayesianLinearRegression
└── Variational Inference
├── README.md
├── pyproject.toml
├── src/
│ └── mlscratch/
│ ├── __init__.py
│ ├── supervised/
│ │ ├── __init__.py
│ │ ├── linear_regression.py
│ │ ├── logistic_regression.py
│ │ ├── lasso_regression.py
│ │ ├── ridge_regression.py
│ │ ├── knn.py
│ │ ├── decision_tree.py
│ │ ├── random_forest.py
│ │ ├── naive_bayes.py
│ │ └── svm.py
│ └── unsupervised/ <- in progress
├── tests/
│ ├── conftest.py
│ ├── supervised/
│ │ ├── test_linear_regression.py
│ │ ├── test_logistic_regression.py
│ │ ├── test_lasso_regression.py
│ │ ├── test_ridge_regression.py
│ │ ├── test_knn.py
│ │ ├── test_decision_tree.py
│ │ ├── test_random_forest.py
│ │ ├── test_naive_bayes.py
│ │ └── test_svm.py
│ └── unsupervised/ <- coming next
```

## Design Philosophy

Across all implementations, the following principles are applied:

- Prefer explicit loops over vectorized one-liners when it improves clarity
- Separate model logic, loss computation, and parameter updates
- Avoid high-level ML libraries to expose core mechanics
- Keep implementations concise and inspectable

Many design choices intentionally trade performance for readability.

---

## What to Expect
## What This Repository Is For

Are you eager to grasp the core concepts of these cutting-edge technologies? Look no further! In this repository, we've meticulously crafted implementations of fundamental algorithms from scratch, accompanied by detailed explanations and documentation. Our mission is to empower learners by providing hands-on experience in building these algorithms, fostering a deeper understanding of the underlying principles.
This repo is intended as an educational reference for learners who want to understand the internal mechanics of algorithms, not as a production-ready library.

---

## How to Learn Effectively With This Repository

To get the most value from this repository:

1. Read the code line-by-line
2. Add print statements or visualizations
3. Modify hyperparameters and observe behavior
4. Re-implement the same algorithm in a different style
5. Compare similar algorithms across folders
It prioritizes:

This repository is meant to be **actively explored**, not passively read.
- clarity over micro-optimization
- math-first explanations
- algorithmic correctness through tests
- reproducible minimal examples

---

## Why Learn From Scratch?

Understanding AI, ML, DL, and RL can be a daunting task, especially for beginners. Yet, I believe that building these algorithms from the ground up offers unparalleled insights. By diving into the code, you'll gain a profound understanding of the inner workings, demystifying the complex algorithms that power the technology around us.

## What Sets This Apart?

- **Educational Focus:** Every algorithm is meticulously implemented with educational purposes in mind.
- **Comprehensive Documentation:** Each implementation is accompanied by thorough explanations, ensuring you not only run the code but understand it.
- **Progressive Complexity:** Starting from simpler concepts, we gradually delve into more advanced algorithms, allowing you to build your knowledge progressively.

## Explore my Implementations
## Installation

- **Neural Networks:** Dive into the realm of neural networks, from basic perceptrons to advanced architectures like LSTMs and Transformers.
- **Reinforcement Learning:** Understand the dynamics of reinforcement learning through implementations of DDPG, DQN, PPO, and Q-learning.
- **Supervised Learning:** Explore classical supervised learning algorithms, including decision trees, regression models, and support vector machines.
- **Unsupervised Learning:** Delve into the mysteries of unsupervised learning with implementations like k-means, PCA, and GMM.
```bash
python -m pip install -e .
python -m pip install -e .[dev]
```

## Who Is This For?
The repository is designed to work with Python 3.10+.

Whether you're a student, a curious enthusiast, or a seasoned developer looking to solidify your understanding, this repository is designed for you. Our step-by-step implementations and detailed documentation cater to learners at all levels.
---

Ready to embark on this exciting journey? Let's code, learn, and demystify the world of AI together! 🌐✨
## Quick Start

```python
from mlscratch.supervised import (
OrdinaryLeastSquares,
LogisticRegression,
LassoRegression,
RidgeRegression,
KNeighborsClassifier,
DecisionTreeClassifier,
RandomForestClassifier,
GaussianNB,
LinearSVMClassifier,
)

model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)
predictions = model.predict(X_test)
```

---

## Educational Content
## Testing

## Conceptual Background (Why These Implementations Matter)
Run the supervised test suite:

The implementations in this repository are grounded in the following learning paradigms:
```bash
python -m pytest tests/supervised -q
```

The repository uses `pytest` and is configured with `pytest-cov` for coverage reporting.

### What is Supervised Learning?
---

Supervised learning is a type of machine learning where the algorithm is trained on a labeled dataset. In a labeled dataset, each input data point is associated with the corresponding correct output, allowing the algorithm to learn the mapping between inputs and outputs. The goal is for the algorithm to make accurate predictions on new, unseen data.
## Package Goals

### What is Unsupervised Learning?
The long-term goal is to make this repository a best-in-class educational reference by:

Unsupervised learning involves training algorithms on unlabeled datasets. Unlike supervised learning, there are no predefined output labels. Instead, the algorithm discovers patterns, structures, or relationships within the data on its own. Common tasks in unsupervised learning include clustering and dimensionality reduction.
- standardizing module structure
- enforcing tests for correctness against `scikit-learn` baselines
- adding benchmark-driven performance comparisons
- documenting math and algorithmic intuition consistently

### What are Neural Networks?
---

Neural networks are a class of machine learning models inspired by the structure and function of the human brain. They consist of interconnected nodes, or neurons, organized into layers. Neural networks can learn complex patterns and representations through training on labeled data. Deep learning, a subset of neural networks, involves architectures with multiple layers (deep neural networks).
## Next Work

### What is Reinforcement Learning?
The next active task is to migrate unsupervised algorithms into `src/mlscratch/unsupervised/`, starting with a clean `KMeans` implementation and its test coverage.

Reinforcement learning is a type of machine learning where an agent learns to make decisions by interacting with an environment. The agent receives feedback in the form of rewards or penalties based on the actions it takes. The goal is for the agent to learn a policy that maximizes cumulative reward over time. Reinforcement learning is commonly used in applications such as game playing, robotics, and autonomous systems.
After that, work will continue through the remaining `feedback.md` roadmap:

### What is Bayesian Learning?
- unsupervised algorithms (`KMeans`, `PCA`, `GMM`, `DBSCAN`, `SOM`, `tSNE`)
- neural network modules
- reinforcement algorithms
- Bayesian algorithms

Bayesian learning is a statistical framework that combines prior knowledge with new evidence to update and refine our beliefs about uncertain quantities. Unlike traditional machine learning approaches that focus solely on point estimates, Bayesian learning provides a probabilistic framework for reasoning about uncertainty.
---

## Notes for Contributors

## Usage
If you want to help improve this repository, focus on:

Each algorithm is provided as a standalone Python script. You can run these scripts to see the algorithms in action. Additionally, the code is extensively documented to help you understand the implementation details.
- adding `src/mlscratch/` modules for remaining algorithms
- matching the package template used by existing supervised implementations
- writing tests that compare output to `scikit-learn` or other reliable baselines
- keeping documentation concise and mathematically rigorous



Loading
Loading