Skip to content
BranchesTags

Repository files navigation

LLM-Assisted Code-Change Risk Predictor

This project predicts whether a code change may be risky.

The current beginner version uses a small sample dataset with simple code-change features:

  • number of files changed
  • number of lines added
  • number of lines deleted

The model learns from previous examples and predicts whether a new code change looks risky or not risky.

Later, this project will use real Git commit data, CodeBERT embeddings, and LLM-generated explanations for code reviewers.

Why This Project Matters

During code review, some changes are more likely to cause bugs than others.

For example, a change that modifies many files and many lines may need more careful review than a very small change.

This project explores whether machine learning can help reviewers identify risky code changes earlier.

Current Stage

The current version is a beginner baseline.

It uses:

  • a small fake commit dataset
  • basic static features
  • Logistic Regression
  • simple risk prediction
  • saved model results

Current input features:

files_changed
lines_added
lines_deleted

Current label:

risky

Where:

0 = not risky
1 = risky

Project Structure

LLM-Assisted-Code-Change-Risk-Predictor/
├── data/
│   └── sample_commits.csv
├── outputs/
│   ├── figures/
│   └── tables/
│       └── simple_results.txt
├── src/
│   ├── load_dataset.py
│   ├── prepare_xy.py
│   ├── train_simple_model.py
│   ├── test_simple_model.py
│   ├── predict_new_change.py
│   └── save_simple_results.py
├── README.md
└── requirements.txt

Setup on Arch Linux

Create a virtual environment:

python -m venv .venv

Activate it:

source .venv/bin/activate

Install dependencies:

pip install -r requirements.txt

Check setup:

python src/check_setup.py

How to Run

Load and inspect the dataset:

python src/load_dataset.py

Separate inputs and output:

python src/prepare_xy.py

Train a simple model:

python src/train_simple_model.py

Test the model:

python src/test_simple_model.py

Predict one new code change:

python src/predict_new_change.py

Save experiment results:

python src/save_simple_results.py

Example Output

Example prediction:

New code change:
files_changed = 6
lines_added = 220
lines_deleted = 90

Risk prediction:
RISKY

Risk probability:
0.96

Reviewer note:
This change may need careful review because it modifies 6 files and changes 310 total lines.

Real Dataset: ApacheJIT

This project uses ApacheJIT as the first real commit-level defect prediction dataset.

Download the training CSV:

mkdir -p data/raw data/processed
wget -O data/raw/apachejit_train.csv "https://zenodo.org/records/5907002/files/apachejit_train.csv?download=1"

Load and inspect the dataset:

python src/load_real_dataset.py

This creates:

data/processed/apachejit_train_processed.csv

Real Baseline Results

The first real baseline uses ApacheJIT commit-level metrics.

Models compared:

  • Logistic Regression
  • Random Forest

Metrics:

  • accuracy
  • precision
  • recall
  • F1 score

Result table is saved at:

outputs/tables/final_real_model_comparison.csv

Markdown version:

outputs/tables/final_real_model_comparison.md
model accuracy precision recall f1_score
Random Forest 0.7213 0.7167 0.7322 0.7243
Logistic Regression 0.6822 0.7359 0.5685 0.6414

What the Model Learns

The current model learns simple patterns such as:

Small code changes are usually lower risk.
Large code changes are usually higher risk.

This is only a beginner baseline.

The goal is not to claim this fake-data model is useful yet.

The goal is to build the full pipeline step by step.

Planned Improvements

Next, this project will add:

  • better static features
  • real commit data
  • proper evaluation tables
  • Random Forest baseline
  • CodeBERT code embeddings
  • static + CodeBERT fusion model
  • reviewer-facing LLM explanations
  • error analysis

Final Project Goal

The final system should take a code change or commit diff and output:

Risk score: 82%

Explanation:
This change may be risky because it modifies many files, changes complex logic, and resembles previous risky commits.

The long-term goal is to build an explainable AI4SE system for commit-level defect risk prediction.

Status

Current status:

Stage 2/3 complete:
Tiny fake dataset + simple ML baseline + saved results.

Next stage:

Improve the baseline with more features and cleaner evaluation.

Reading Real Git Commits

Install PyDriller:

pip install pydriller

Clone a sample repository:

mkdir -p external_repos
git clone https://github.com/pallets/markupsafe.git external_repos/markupsafe

Read sample commits:

python src/read_git_commits.py

This creates:

outputs/tables/sample_git_commits.csv

Label Problem

When we mine commits from a normal Git repository using PyDriller, we can extract commit features such as:

  • commit message
  • files changed
  • lines added
  • lines deleted
  • total churn

However, normal Git history does not automatically tell us whether a commit is risky or bug-inducing.

For supervised training, this project uses ApacheJIT because it already includes commit-level risky/clean labels.

Mined Git commits are useful for feature extraction practice and future prediction demos.

CodeBERT Tokenizer

This project uses the Hugging Face tokenizer for:

microsoft/codebert-base

The tokenizer turns cleaned diff text into tokens and token IDs.

Run:

python src/load_codebert_tokenizer.py
python src/tokenize_clean_diff.py

Outputs:

outputs/tables/codebert_tokenizer_example.csv
outputs/tables/clean_diff_tokens.csv

Reusable CodeBERT Features

After creating CodeBERT embeddings, the project saves them in reusable formats.

Run:

python src/save_codebert_features_reusable.py
python src/load_saved_codebert_features.py

Outputs:

data/features/codebert_embedding_features.csv
data/features/codebert_embedding_matrix.npy
data/features/codebert_embedding_metadata.csv
data/features/codebert_embedding_feature_names.txt
outputs/tables/codebert_feature_export_summary.csv

The .npy file stores the numeric embedding matrix for fast model training.

Toy Fusion Model Comparison

This project compares three feature sets:

  • Static-only features
  • CodeBERT-only embeddings
  • Static + CodeBERT fusion features

Run:

python src/create_toy_fusion_dataset.py
python src/train_toy_fusion_classifier.py
python src/save_final_fusion_comparison.py

Outputs:

outputs/tables/final_toy_fusion_model_comparison.csv
outputs/tables/final_toy_fusion_model_comparison.md
outputs/figures/final_toy_fusion_f1_comparison.png

Note: this is a toy fusion experiment using demo labels. The real fusion experiment requires static features and CodeBERT embeddings from the same labeled commits.

Real ApacheJIT Fusion Pipeline (Option C)

This pipeline mines real commit diffs from the ApacheJIT commit_id + project fields, creates CodeBERT embeddings, joins them with ApacheJIT static features and labels, and trains a real fusion classifier.

Run the full pipeline:

python scripts/run_real_fusion_pipeline.py --max-commits 2000 --resume

Or run step by step:

python scripts/mine_apachejit_diffs.py --max-commits 2000 --resume
python scripts/create_apachejit_codebert_embeddings.py --resume
python scripts/finetune_codebert_apachejit.py
python scripts/create_apachejit_finetuned_embeddings.py --resume
python scripts/reduce_fusion_embeddings.py \
  --input-path data/embeddings/apachejit_finetuned_embeddings.csv \
  --output-path data/embeddings/apachejit_finetuned_embeddings_pca.csv \
  --pca-model-path models/fusion_finetuned_pca.joblib \
  --selection-path outputs/tables/real_fusion_finetuned_pca_selection.csv
python scripts/create_real_fusion_dataset.py --embedding-source finetuned_pca
python scripts/train_real_fusion_classifier.py --model random_forest --tune-hyperparams --tune-threshold
python scripts/save_final_real_fusion_comparison.py
python scripts/summarize_results_for_report.py

Key inputs:

data/processed/apachejit_train_processed.csv

Key outputs:

data/processed/apachejit_diff_index.csv
data/processed/apachejit_fusion_dataset.csv
data/embeddings/apachejit_codebert_embeddings.csv
outputs/tables/final_real_fusion_model_comparison.csv
outputs/tables/report_results_summary.csv

Notes:

  • --max-commits 0 mines all ApacheJIT commits (slow; requires network access for PyDriller).
  • Default --max-commits 2000 uses a stratified sample for a practical first real evaluation.
  • Mining uses GitHub repositories such as https://github.com/apache/groovy.
  • Resume flags let you continue mining and embedding after interruptions.

Reviewer-Facing Explanations

The project creates grounded reviewer-facing explanations from evidence packets.

Run:

python src/create_explanation_evidence_packets.py
python src/create_safe_llm_prompts.py
python src/generate_sample_explanations.py

Outputs:

outputs/evidence/explanation_evidence_packets.jsonl
outputs/prompts/llm_explanation_prompts.jsonl
outputs/explanations/sample_llm_explanations.csv
outputs/explanations/sample_llm_explanations.jsonl
outputs/explanations/sample_llm_explanations.md

The explanations are review aids. They do not prove that a bug exists.

Sample Reviewer-Facing Explanation

The system predicts whether a code change may be risky and generates a grounded explanation for reviewers.

Example output is saved at:

outputs/explanations/final_explanation_examples_short.md

The explanation is based on evidence such as:

  • risk probability
  • files changed
  • lines added
  • lines deleted
  • total churn
  • changed file paths

The explanation is a review aid, not proof that a bug exists.

Full Static Pipeline Demo

Run the full static-feature risk prediction pipeline:

python scripts/run_pipeline.py --input sample_inputs/sample_commit.json

This runs:

  1. model training
  2. prediction for one sample commit
  3. reviewer-facing explanation

Main outputs:

models/static_risk_model.joblib
outputs/tables/final_training_metrics.csv
outputs/predictions/prediction_result.txt
outputs/explanations/prediction_explanation.txt
outputs/pipeline/pipeline_summary.txt

The prediction and explanation are review aids, not proof that a bug exists.

Project Structure

code-change-risk-predictor/
├── configs/              # configuration files
├── data/
│   ├── raw/              # original downloaded datasets
│   ├── processed/        # cleaned datasets
│   ├── features/         # saved feature matrices
│   └── embeddings/       # saved embedding files
├── examples/             # small example diffs and inputs
├── models/               # trained model files
├── notebooks/            # optional exploration notebooks
├── outputs/
│   ├── tables/           # CSV result tables
│   ├── figures/          # plots and confusion matrices
│   ├── explanations/     # generated explanations
│   ├── evidence/         # explanation evidence packets
│   ├── error_analysis/   # false positive/negative analysis
│   ├── pipeline/         # pipeline summaries and logs
│   ├── predictions/      # prediction outputs
│   └── prompts/          # LLM prompt templates
├── scripts/              # command-line entry points
├── src/                  # Python source and lesson scripts
├── tests/                # future tests
├── README.md
├── requirements.txt
└── .gitignore

Check the structure:

python scripts/check_project_structure.py

Configuration

Project settings are stored in:

configs/default.yaml

The config includes:

  • project name
  • dataset paths
  • model settings
  • CodeBERT settings
  • output folders
  • prediction paths
  • explanation safety rule

Check the config with:

python scripts/check_config.py

Logging

The project includes a reusable logging utility:

src/logging_utils.py

Check logging with:

python scripts/check_logging.py

This writes logs to:

logs/project.log

Log files are ignored by Git because they are generated during runs.

Tests

Run the test suite with:

pytest

The tests check:

  • required project folders and files
  • config file sections
  • sample prediction input validity
  • prediction input feature compatibility, if trained feature names exist

Final Report

The project report is stored in:

reports/final_report.md

It describes the problem, dataset, method, experiments, explanations, error analysis, limitations, and conclusion.

Web App

The project includes a full-stack web application for reviewer-facing risk prediction.

  • Backend: FastAPI (backend/) serves the saved static ApacheJIT model, grounded explanations, metrics, error analysis, and the final report.
  • Frontend: React + Vite (frontend/) provides a dashboard UI for prediction, evaluation summaries, and project documentation views.
  • Current production model: The web app uses the saved static ApacheJIT Random Forest model (models/static_risk_model.joblib) trained on commit-level static features.
  • Experimental extension path: CodeBERT embeddings and static + CodeBERT fusion workflows exist in the repository as research/demo code, but they are not the current production inference path.
  • Safety note: All predictions and explanations are review aids. They do not prove that a bug exists.

Run the backend

From the repository root, activate your Python environment if needed, then:

cd backend
pip install -r requirements.txt
uvicorn app.main:app --reload

The API runs at http://localhost:8000.

Run the frontend

In a second terminal:

cd frontend
npm install
npm run dev

Open the app at:

http://localhost:5173

API docs

Interactive OpenAPI docs:

http://localhost:8000/docs

Web app pages

  • Dashboard — API health, model status, key metrics
  • Predict Risk — submit commit features, view prediction and grounded explanation
  • Model Metrics — training and report summary tables
  • Error Analysis — false positives, false negatives, grouped errors
  • Report — final project report markdown
  • Settings — model paths, feature names, API configuration

See WEB_APP_PLAN.md for architecture, endpoints, and future roadmap.

Troubleshooting

Model file missing

If prediction returns 503 or model info shows model_loaded: false, train the static model first:

python scripts/train_model.py

This creates:

models/static_risk_model.joblib
models/static_feature_names.txt

Backend not running

If the frontend shows a network error or cannot load dashboard data, start the FastAPI server:

cd backend
uvicorn app.main:app --reload

CORS issue

The backend allows http://localhost:5173 by default. If you run the frontend on a different host or port, update CORS in backend/app/main.py or run Vite on port 5173.

Frontend cannot reach API

The frontend uses VITE_API_BASE_URL with default http://localhost:8000.

Check:

  1. Backend is running on port 8000
  2. Browser can open http://localhost:8000/health
  3. Set a custom API URL when needed:
cd frontend
VITE_API_BASE_URL=http://localhost:8000 npm run dev

Backend tests

cd backend
pytest

Docker

Run the full web app with Docker Compose from the repository root:

docker compose up --build

This starts:

  • Backend on http://localhost:8000
  • Frontend on http://localhost:5173

The backend container mounts project files from the host:

  • models/
  • configs/
  • outputs/
  • reports/

The frontend is built with:

VITE_API_BASE_URL=http://localhost:8000

API docs remain available at http://localhost:8000/docs.

Stop the stack with:

docker compose down

Local development without Docker still works using the backend and frontend commands above.

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages