diff --git a/README.md b/README.md index a938a3d..2f2afe9 100644 --- a/README.md +++ b/README.md @@ -1,182 +1,314 @@ # CodeSecAudit AI -**Rule-based + RAG security code review engine for Python.** +Enterprise AI Pull Request Reviewer for Security, Code Quality, and RAG-based Secure Coding Guidance -CodeSecAudit AI detects OWASP Top 10 vulnerabilities in Python code using -deterministic pattern matching (7 rules), optionally augmented with -retrieval-augmented generation (RAG) over OWASP cheat sheets (2,833 ChromaDB -docs). Ships with a CLI, FastAPI, Streamlit UI + dashboard, GitHub Action, and -a lightweight Docker image (~500 MB; RAG optional). +CodeSecAudit AI acts like an AI senior engineer inside GitHub pull requests, detecting risky code, posting inline comments, suggesting fixes, and tracking review analytics before code is merged. -## Features +[![Python](https://img.shields.io/badge/Python-3.11+-blue)](https://www.python.org/) +[![FastAPI](https://img.shields.io/badge/FastAPI-0.115-009688)](https://fastapi.tiangolo.com/) +[![Docker](https://img.shields.io/badge/Docker-compose-2496ED)](https://docs.docker.com/compose/) +[![Hugging Face Dataset](https://img.shields.io/badge/HF-Dataset-FFD21E)](https://huggingface.co/datasets/OMCHOKSI108/CodeSecAudit-RAG) +[![Hugging Face Space](https://img.shields.io/badge/HF-RAG%20Service-FFD21E)](https://OMCHOKSI108-codereview-agent.hf.space) +[![Kaggle](https://img.shields.io/badge/Kaggle-Notebook-20BEFF)](https://www.kaggle.com/code/omchoksi04/codereview) +[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) -- **7 rule-based detectors** — CWE-94 (code injection), CWE-89 (SQLi), - CWE-78 (command injection), CWE-328 (weak hash), CWE-798 (hardcoded creds), - CWE-22 (path traversal), CWE-918 (SSRF) -- **Optional RAG** — ChromaDB index of OWASP cheat sheets used by the fixer -- **Autofix generation** — template-based suggestions per CWE -- **Risk scoring** — weighted severity → 0–100 score with verdict -- **Review history** — SQLite persistence via FastAPI CRUD endpoints -- **GitHub Action** — automatic PR comments (summary + inline, max 10 lines) -- **Streamlit apps** — review interface + analytics dashboard -- **Docker Compose** — 3-service stack (API, review UI, dashboard) +--- -## Quick Start +## Live Links -```bash -# Install lightweight (rules-only, no heavy ML deps) -pip install -e ".[api,ui]" +| Asset | Link | +|---|---| +| Kaggle Notebook | https://www.kaggle.com/code/omchoksi04/codereview | +| Hugging Face Dataset | https://huggingface.co/datasets/OMCHOKSI108/CodeSecAudit-RAG | +| RAG Service (live) | https://OMCHOKSI108-codereview-agent.hf.space | +| GitHub Repo | https://github.com/OMCHOKSI108/codesec-audit-dataset | +| Deploy PR | https://github.com/OMCHOKSI108/codesec-audit-dataset/pull/1 | + +--- -# Start the API -uvicorn api.main:app --port 8003 +## What Problem It Solves + +- **Manual PR reviews are slow** — context-switching between diff view and security checklists costs teams hours per week. +- **Security issues slip through** — CWE-94 (code injection), CWE-89 (SQLi), and CWE-78 (command injection) are routinely missed in handwritten reviews. +- **Junior developers need guidance** — without inline, specific feedback, recurring vulnerabilities go unfixed. +- **Teams need review history** — without analytics, there is no way to track security debt or review velocity across repos. + +CodeSecAudit AI addresses all four: automated detection with OWASP-aligned rules, optional RAG-based secure coding guidance, inline PR comments with fix suggestions, and a dashboard for review analytics. + +--- -# Start the review UI -streamlit run ui/app.py --server.port 8501 +## Product Features + +| Feature | Details | +|---|---| +| **GitHub Action PR review** | Triggers on `opened`, `synchronize`, `reopened` — reviews `.py`, `.js`, `.ts` files | +| **Summary comment** | Posts a markdown table with verdict, risk score, CWE breakdown, and suggested fixes | +| **Inline file/line comments** | Posts comments directly on vulnerable lines in the PR diff (capped at 10) | +| **7 CWE detectors** | CWE-94, CWE-89, CWE-78, CWE-328, CWE-798, CWE-22, CWE-918 | +| **Severity + risk score** | Weighted severity produces a 0–100 risk score and verdict (APPROVE / WARNING / REQUEST_CHANGES) | +| **Suggested fixes** | Template-based per CWE — tells the developer exactly what to change | +| **RAG secure guidance** | Optional retrieval from 2,833 OWASP cheat sheet chunks for contextual advice | +| **Review history DB** | Every review saved to SQLite with search by repo, PR number, verdict, date | +| **Dashboard analytics** | Verdict distribution, risk trends, CWE breakdown, review detail viewer | +| **Usage-limit SaaS design** | Schema ready for free-tier caps, plan enforcement, and owner contact | +| **Resend email workflow design** | Welcome, usage guide, limit-reached notification templates | +| **MongoDB Atlas design** | 7-collection schema for users, installations, reviews, plans, email events | -# Start the dashboard -streamlit run ui/dashboard.py --server.port 8502 +--- + +## Architecture + +```mermaid +flowchart TD + A[GitHub Pull Request] --> B[GitHub Action / Future GitHub App] + B --> C[Changed Files + Diff Parser] + C --> D[review_engine Critic] + D --> E[Remote RAG Service on Hugging Face] + E --> F[OWASP Secure Coding Guidance] + D --> G[Fixer + Risk Scoring] + F --> G + G --> H[PR Summary Comment] + G --> I[Inline PR Comments] + G --> J[FastAPI Review History API] + J --> K[SQLite MVP / MongoDB SaaS] + K --> L[Streamlit Dashboard] ``` -## Usage +For detailed architecture including the review pipeline, RAG service internals, and SaaS future, see [docs/architecture.md](docs/architecture.md). -### CLI +--- -```bash -python scripts/review_code.py path/to/code.py +## Repository Structure + +``` +review_engine/ Core: critic, fixer, retriever, pipeline, schemas, remote RAG client +rag_service/ Standalone FastAPI microservice for RAG (deployed on HF Space) +review_store/ SQLite persistence layer with repository pattern +api/ FastAPI application with review and history endpoints +ui/ Streamlit apps: review interface + analytics dashboard +scripts/ CLI, evaluation, smoke test, RAG index builder, deploy helpers +examples/ Demo files: vulnerable_pr_demo.py, safe_pr_demo.py +eval/ Golden test cases for regression testing (12 cases) +deploy/ HF Space Dockerfile, start script, deploy README +docs/ Architecture, API reference, GitHub Action, RAG service, Docker, deployment +.github/workflows/ GitHub Action workflow definition ``` -### API +--- +## Dataset — CodeSecAudit-RAG + +The custom dataset powers both rule-based evaluation and the RAG retrieval service. + +| Metric | Value | +|---|---| +| Total review records | 28,548 | +| RAG corpus chunks | 2,833 | +| Embedding model | `all-MiniLM-L6-v2` (384-dim) | +| Similarity | Cosine | +| Sources | CodeXGLUE + OWASP Benchmark Python + OWASP Cheat Sheet Series | + +**Hugging Face Dataset**: https://huggingface.co/datasets/OMCHOKSI108/CodeSecAudit-RAG + +**Build the index locally**: ```bash -curl -X POST http://localhost:8003/review \ - -H "Content-Type: application/json" \ - -d '{"code": "eval(user_input)", "use_rag": false}' +python scripts/build_rag_index.py ``` -### Python +--- -```python -from review_engine.pipeline import review_code +## Kaggle Notebook -result = review_code("eval(user_input)", use_rag=False) -print(result["summary"]) # Found 1 issue(s): CWE-94. Risk score: 25/100... -``` +Explore the dataset, RAG corpus, and a prototype Critic → Retriever → Fixer pipeline on Kaggle: -## Services +https://www.kaggle.com/code/omchoksi04/codereview -| Service | Port | Description | -|------------|-------|--------------------------------------| -| API | 8003 | FastAPI review + history endpoints | -| Review UI | 8501 | Streamlit code review interface | -| Dashboard | 8502 | Analytics dashboard with charts | +The notebook demonstrates: +- Dataset exploration (28,548 records from CodeXGLUE + OWASP) +- RAG corpus embedding and similarity search +- Prototype review pipeline -## Docker +--- + +## Quickstart -See [docs/docker.md](docs/docker.md) for build options including RAG mode. +### Local (no Docker) ```bash -docker compose build -docker compose up -d +pip install -e ".[dev]" +python scripts/review_code.py --code "eval(user_input)" --json ``` -## Evaluation +### Docker Compose (lightweight, rules-only) ```bash -python scripts/evaluate_reviewer.py +docker compose up --build ``` -Runs 12 golden cases spanning all 7 supported CWEs. All pass. +### Access the services -## Project Structure +| Service | URL | +|---|---| +| API | http://localhost:8003 | +| API docs (Swagger) | http://localhost:8003/docs | +| Review UI | http://localhost:8501 | +| Dashboard | http://localhost:8502 | -``` -├── review_engine/ # Core: critic, fixer, retriever, pipeline, schemas -├── review_store/ # SQLite persistence layer -├── api/ # FastAPI application -├── ui/ # Streamlit apps (review + dashboard) -├── scripts/ # CLI, evaluation, smoke test, RAG index builder -├── eval/ # Golden test cases -├── docs/ # Documentation -└── data/ # RAG index, datasets (gitignored except index) -``` +See [docs/docker.md](docs/docker.md) for RAG mode and build options. -## GitHub App +--- -The GitHub App is the primary way to automate PR reviews for your repositories. -Once installed, it listens on `pull_request` events and posts inline comments. +## GitHub Action Usage -> **Status**: Coming soon. Until then, use the [GitHub Action](.github/workflows/pr_review.yml) -> as a manual MVP fallback. +The workflow at `.github/workflows/codesec-audit.yml` triggers automatically on `pull_request: [opened, synchronize, reopened]`. -[![Install GitHub App](https://img.shields.io/badge/GitHub%20App-Install-blue)]() +**What it does**: +1. Checks out the PR branch +2. Installs the CodeSecAudit package +3. Runs `scripts/github_pr_review.py` — reviews changed `.py`, `.js`, `.ts` files +4. Posts a **summary comment** with verdict, risk score, and issue table +5. Posts **inline comments** on vulnerable lines (max 10) +6. Skips lines without duplicates (detected via issue fingerprint) + +**Limitations**: +- `use_rag=False` in CI (RAG caching not yet configured) +- Non-blocking — always exits 0 +- No paid LLMs; all detection is rule-based + +### Dry-run locally + +```bash +python scripts/github_pr_review.py --files examples/vulnerable_pr_demo.py --dry-run +``` + +Full documentation: [docs/github_action.md](docs/github_action.md) --- -## Try Demo +## RAG Service -Try the review engine live without installing anything: +The RAG retrieval service runs as a **separate Hugging Face Space** to keep the main deployment lightweight (~500 MB). -[![Hugging Face Space](https://img.shields.io/badge/Hugging%20Face-Demo-yellow)]() +**Live endpoint**: https://OMCHOKSI108-codereview-agent.hf.space -The demo runs the rules-only engine (no RAG, no auth, single-user). +| Endpoint | Method | Description | +|---|---|---| +| `/health` | GET | Health check with index status | +| `/rag/search` | POST | Search OWASP guidance by query | ---- +### Enable remote RAG in your environment -## Deployment Architecture +```env +CODESEC_ENABLE_RAG=true +CODESEC_RAG_MODE=remote +CODESEC_RAG_SERVICE_URL=https://OMCHOKSI108-codereview-agent.hf.space +CODESEC_RAG_API_KEY=your-key-here +``` -| Component | Host | Purpose | -|-----------------|--------------------|---------------------------------| -| Demo (rules) | Hugging Face Space | Public lightweight playground | -| API + Dashboard | Render / Railway | Production backend | -| Database | MongoDB Atlas | Users, reviews, usage | -| Email | Resend | Notifications | -| GitHub App | GitHub Marketplace | Final PR integration | +Fallback: if the remote RAG service is unreachable, the review completes in rules-only mode with a `rag_error` in metadata. -See [docs/deployment_strategy.md](docs/deployment_strategy.md) for details. +Full documentation: [docs/rag_service.md](docs/rag_service.md) --- -## Environment Variables +## Evaluation -Copy `.env.example` to `.env` and fill in your values: +12 golden cases span all 7 supported CWEs and verify correctness after every change. ```bash -cp .env.example .env +python scripts/evaluate_reviewer.py +python scripts/smoke_test_demo_files.py ``` -Key variables: +| Metric | Status | +|---|---| +| Golden cases | 12 | +| Passing | 12/12 | +| Multi-hit detection | 3 per rule max, 20 total max | +| Safe file | Returns `APPROVE` with 0 issues | +| Vulnerable demo | Returns `REQUEST_CHANGES` with multiple CWEs | + +Full documentation: [docs/evaluation.md](docs/evaluation.md) + +--- + +## API Endpoints -| Variable | Description | -|----------------------------|-----------------------------------| -| `PUBLIC_WEBSITE_URL` | Public website URL | -| `MONGODB_URI` | MongoDB Atlas connection string | -| `RESEND_API_KEY` | Resend API key for emails | -| `GITHUB_APP_ID` | GitHub App ID | -| `GITHUB_CLIENT_ID` | GitHub OAuth client ID | -| `GITHUB_CLIENT_SECRET` | GitHub OAuth client secret | -| `GITHUB_WEBHOOK_SECRET` | GitHub webhook secret token | -| `GITHUB_PRIVATE_KEY_BASE64`| GitHub App private key (base64) | -| `FREE_PR_REVIEWS_PER_MONTH`| Free tier PR review limit (30) | -| `CODESEC_ENABLE_RAG` | Enable RAG retrieval (`true`/`false`) | -| `OWNER_CONTACT_EMAIL` | Contact email for limit overrides | +| Endpoint | Method | Description | +|---|---|---| +| `/` | GET | Root info — uptime, version, configuration | +| `/health` | GET | Health check with RAG index status | +| `/review` | POST | Review code (no persistence) | +| `/review/code` | POST | Review code and save to history DB | +| `/reviews` | GET | List past reviews (paginated) | +| `/reviews/{id}` | GET | Single review with full issue details | +| `/stats` | GET | Aggregated analytics | -Never commit real secrets to version control. +Full API reference with request/response examples: [docs/api_reference.md](docs/api_reference.md) --- -## Owner Contact +## Deployment Strategy -For questions, custom plans, or limit increases: +| Component | Host | Status | +|---|---|---| +| RAG Service | Hugging Face Space | **Live** | +| API + Dashboard | Render / Railway | Planned | +| Database | MongoDB Atlas | Designed | +| Email | Resend | Designed | +| GitHub App | GitHub Marketplace | Planned | -📧 **omchoksi108@gmail.com** +See [docs/deployment_strategy.md](docs/deployment_strategy.md) and [docs/deployment_status.md](docs/deployment_status.md). --- -## RAG Index +## SaaS Future (Planned) + +The user flow for the SaaS version: + +1. Developer clicks **Install** on the GitHub App README badge +2. GitHub App installs in the developer's repo or org +3. Developer signs in via **GitHub OAuth** +4. Dashboard shows review history, analytics, and settings +5. Free tier: **30 PR reviews / month** +6. Limit reached → Resend email with upgrade prompt +7. Contact owner for custom plans + +Owner contact: **omchoksi108@gmail.com** + +Current status: +- [x] Datasets + RAG corpus +- [x] Kaggle notebook +- [x] CLI reviewer +- [x] FastAPI API +- [x] GitHub Action +- [x] Inline PR comments +- [x] Evaluation +- [x] Review history DB +- [x] Dashboard +- [x] Docker +- [x] Remote RAG service (deployed) +- [ ] Render deployment +- [ ] MongoDB Atlas integration +- [ ] Resend email implementation +- [ ] GitHub OAuth +- [ ] GitHub App install flow +- [ ] Usage limit enforcement -The ChromaDB index (2,833 docs, 384-dim, cosine) is built from OWASP Python -cheat sheets and CodeXGLUE security commits. Build it locally: +--- -```bash -python scripts/build_rag_index.py -``` +## Responsible Use + +CodeSecAudit AI is a **defensive security tool**: +- It detects vulnerabilities — it does not exploit them. +- It is not a replacement for professional SAST tools or manual security review. +- It is designed for **educational and secure-code-review purposes**. +- Do not use it for offensive automation or vulnerability research on systems you do not own. + +--- + +## Owner Contact + +For questions, custom plans, or limit increases: + +**omchoksi108@gmail.com** diff --git a/docs/api_reference.md b/docs/api_reference.md new file mode 100644 index 0000000..456b583 --- /dev/null +++ b/docs/api_reference.md @@ -0,0 +1,258 @@ +# API Reference + +Base URL: `http://localhost:8003` (local) or your deployed URL. + +## `GET /` + +Root endpoint with application info and configuration. + +**Response**: +```json +{ + "app": "CodeSecAudit AI API", + "version": "0.6.0", + "description": "RAG-powered security code review engine", + "uptime": "0h 12m 34s", + "uptime_seconds": 754, + "start_time_iso": "2026-06-22T12:00:00Z", + "configuration": { + "rag_index_path": "data/final/rag_index", + "top_k_default": 3, + "database": "data/app/reviews.db", + "review_rules": 7, + "rag_index_loaded": true + } +} +``` + +--- + +## `GET /health` + +Health check with RAG index status. + +**Response**: +```json +{ + "status": "ok", + "engine_version": "0.6.0", + "collection": "owasp_rag", + "documents": 2833, + "uptime_seconds": 754 +} +``` + +--- + +## `POST /review` + +Review source code without saving to history. + +**Request**: +```json +{ + "code": "eval(user_input)", + "file_path": "demo.py", + "top_k": 3, + "use_rag": true +} +``` + +| Field | Type | Default | Description | +|---|---|---|---| +| `code` | string | — | Source code to analyze (required, min 1 char) | +| `file_path` | string | `null` | Source file path (for display) | +| `top_k` | integer | `3` | Number of RAG results (1–20) | +| `use_rag` | boolean | `true` | Enable RAG retrieval | + +**Response**: +```json +{ + "summary": "Found 1 issue(s): CWE-94. Risk score: 35/100. Verdict: WARNING.", + "risk_score": 35, + "verdict": "WARNING", + "issues": [ + { + "cwe_id": "CWE-94", + "severity": "critical", + "message": "Code injection via eval()", + "line": 1, + "snippet": "eval(user_input)", + "file_path": "demo.py", + "suggested_fix": "Avoid using eval() with untrusted input. Use ast.literal_eval() or a safe parser instead." + } + ], + "metadata": { + "engine_version": "0.6.0", + "rag_used": true, + "rag_error": null, + "total_issues_found": 1, + "total_issues_reported": 1, + "deduplication_skipped": 0, + "limit_capped_by_rule": 0, + "limit_capped_total": false + } +} +``` + +--- + +## `POST /review/code` + +Review source code and save the result to the review history database. + +**Request**: +```json +{ + "code": "eval(user_input)", + "file_path": "demo.py", + "source": "api", + "repo": "owner/repo", + "pr_number": 12, + "commit_sha": "abc123def456", + "top_k": 3, + "use_rag": false +} +``` + +| Field | Type | Default | Description | +|---|---|---|---| +| `code` | string | — | Source code to analyze (required) | +| `file_path` | string | `null` | Source file path | +| `source` | string | `"api"` | Origin (`api`, `cli`, `github-action`) | +| `repo` | string | `null` | GitHub `owner/repo` | +| `pr_number` | integer | `null` | Pull request number | +| `commit_sha` | string | `null` | Commit SHA | +| `top_k` | integer | `3` | Number of RAG results (1–20) | +| `use_rag` | boolean | `true` | Enable RAG retrieval | + +**Response**: +```json +{ + "review_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", + "summary": "Found 1 issue(s): CWE-94. Risk score: 35/100. Verdict: WARNING.", + "risk_score": 35, + "verdict": "WARNING", + "issues": [...], + "metadata": {...}, + "created_at": "2026-06-22T12:34:56Z" +} +``` + +--- + +## `GET /reviews` + +List past reviews, newest first. + +**Query Parameters**: + +| Parameter | Default | Max | Description | +|---|---|---|---| +| `limit` | 50 | 200 | Number of reviews to return | +| `offset` | 0 | — | Pagination offset | + +**Request**: +```bash +curl "http://localhost:8003/reviews?limit=10&offset=0" +``` + +**Response**: +```json +{ + "reviews": [ + { + "review_id": "a1b2c3d4-...", + "source": "api", + "repo": "owner/repo", + "pr_number": 12, + "file_path": "demo.py", + "risk_score": 35, + "verdict": "WARNING", + "summary": "Found 1 issue(s): CWE-94...", + "created_at": "2026-06-22T12:34:56Z" + } + ], + "total": 42, + "limit": 10, + "offset": 0 +} +``` + +--- + +## `GET /reviews/{review_id}` + +Get a single review with full issue details. + +**Request**: +```bash +curl "http://localhost:8003/reviews/a1b2c3d4-..." +``` + +**Response**: +```json +{ + "review_id": "a1b2c3d4-...", + "source": "api", + "repo": "owner/repo", + "pr_number": 12, + "commit_sha": "abc123", + "file_path": "demo.py", + "risk_score": 35, + "verdict": "WARNING", + "summary": "Found 1 issue(s): CWE-94...", + "issues": [ + { + "cwe_id": "CWE-94", + "severity": "critical", + "message": "Code injection via eval()", + "line": 1, + "snippet": "eval(user_input)", + "file_path": "demo.py", + "suggested_fix": "Avoid using eval() with untrusted input..." + } + ], + "metadata": { + "engine_version": "0.6.0", + "rag_used": false, + "total_issues_found": 1, + "total_issues_reported": 1 + }, + "created_at": "2026-06-22T12:34:56Z" +} +``` + +Returns `404` if review not found. + +--- + +## `GET /stats` + +Aggregated analytics from the review history database. + +**Response**: +```json +{ + "total_reviews": 42, + "verdict_counts": { + "APPROVE": 15, + "WARNING": 20, + "REQUEST_CHANGES": 7 + }, + "average_risk_score": 28.5, + "high_risk_reviews": 5, + "total_issues": 63 +} +``` + +--- + +## Error Responses + +| Status | Description | +|---|---| +| `400` | Invalid request body (e.g., empty code) | +| `404` | Review not found | +| `422` | Validation error (e.g., `top_k` out of range) | +| `500` | Internal server error | diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..e1ef842 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,124 @@ +# Architecture + +## High-Level System Diagram + +```mermaid +flowchart TD + A[GitHub Pull Request] --> B[GitHub Action / Future GitHub App] + B --> C[Changed Files + Diff Parser] + C --> D[review_engine Critic] + D --> E[Remote RAG Service on Hugging Face] + E --> F[OWASP Secure Coding Guidance] + D --> G[Fixer + Risk Scoring] + F --> G + G --> H[PR Summary Comment] + G --> I[Inline PR Comments] + G --> J[FastAPI Review History API] + J --> K[SQLite MVP / MongoDB SaaS] + K --> L[Streamlit Dashboard] + K --> M[Analytics API] + L --> M +``` + +## Component Overview + +| Component | Language | Role | +|---|---|---| +| `review_engine` | Python | Core: critic, fixer, retriever, risk scorer, pipeline | +| `rag_service` | Python | Standalone FastAPI microservice for RAG (HF Space) | +| `review_store` | Python | SQLite persistence with repository pattern | +| `api` | Python | FastAPI application exposing review + history endpoints | +| `ui` (review) | Python | Streamlit interface for submitting code reviews | +| `ui` (dashboard) | Python | Streamlit interface for analytics and history | +| `scripts/` | Python | CLI, evaluation, deploy, smoke test helpers | +| `.github/workflows/` | YAML | GitHub Action definition | + +## Review Pipeline + +```mermaid +sequenceDiagram + participant PR as Pull Request + participant GA as GitHub Action + participant RE as review_engine + participant RS as RAG Service + participant DB as Review Store + participant UI as Dashboard + + PR->>GA: opened / synchronize + GA->>RE: review_code(code, use_rag) + RE->>RE: critic.scan() → list of issues + alt RAG mode = remote + RE->>RS: POST /rag/search (query) + RS-->>RE: OWASP guidance chunks + else RAG mode = local + RE->>RE: RAGRetriever.search() + end + RE->>RE: fixer.generate_fixes(issues, guidance) + RE->>RE: risk_score.compute(issues) → score + verdict + RE-->>GA: ReviewResult + GA->>GA: post summary comment + GA->>GA: post inline comments (max 10) + GA->>DB: save review record + DB-->>UI: analytics data +``` + +## RAG Service Architecture + +```mermaid +flowchart LR + A[Client] --> B[FastAPI /rag/search] + B --> C{API Key Check} + C -->|Missing / Invalid| D[401 Unauthorized] + C -->|Optional / Matching| E[RagIndex] + E --> F[Corpus JSONL from HF Dataset] + E --> G[all-MiniLM-L6-v2 Embeddings] + E --> H[Numpy Cosine Similarity Search] + H --> I[Top-K Results] + I --> B +``` + +The RAG service: +- Loads the corpus from Hugging Face Dataset (`OMCHOKSI108/CodeSecAudit-RAG`) on startup +- Embeds all chunks using `sentence-transformers/all-MiniLM-L6-v2` (384-dim) +- Searches via numpy cosine similarity (no heavy DB driver) +- Supports optional `X-CodeSec-RAG-Key` auth header (production: required) +- Returns up to `top_k` results with rank, score, title, CWE ID, and content + +### Why a separate service? + +RAG dependencies (sentence-transformers, PyTorch) add ~7 GB to the Docker image. By deploying the RAG service as a **separate Hugging Face Space**, the main API and dashboard stay lightweight (~500 MB) and call RAG over HTTP when needed. + +## SaaS Future Architecture + +```mermaid +flowchart TD + A[GitHub App Webhook] --> B[API Server - Render] + B --> C[MongoDB Atlas] + B --> D[Resend Email] + B --> E[RAG Service - HF Space] + C --> F[Users Collection] + C --> G[Installations Collection] + C --> H[Reviews Collection] + C --> I[Usage Events Collection] + C --> J[Plans Collection] + C --> K[Email Events Collection] + F --> L[Dashboard - Streamlit] + L --> M[GitHub OAuth Login] + L --> N[Usage Stats + Analytics] + D --> O[Welcome Email] + D --> P[Limit Reached Email] + D --> Q[Usage Guide Email] +``` + +See [docs/deployment_strategy.md](deployment_strategy.md) and [docs/saas_data_model.md](saas_data_model.md) for full details. + +## Key Design Decisions + +| Decision | Rationale | +|---|---| +| Rule-based detection (no LLM API) | Zero cost per review, deterministic, no API keys needed | +| Remote RAG via HTTP | Keeps main image ~500 MB; RAG deps live only on HF Space | +| Numpy cosine similarity over ChromaDB | Simpler, no heavy DB driver, 2,833 chunks fit in memory | +| SQLite MVP → MongoDB SaaS | SQLite is zero-config for development; MongoDB for production scale | +| `use_rag=False` in CI | Avoids downloading embedding model on every workflow run | +| Non-blocking CI (exit 0) | Prevents broken builds from false positives | diff --git a/docs/demo.md b/docs/demo.md new file mode 100644 index 0000000..e222dc1 --- /dev/null +++ b/docs/demo.md @@ -0,0 +1,106 @@ +# Demo + +## Local Demo (No Docker) + +```bash +# Install +pip install -e ".[dev]" + +# CLI review +python scripts/review_code.py --code "eval(user_input)" --json +``` + +Expected output: JSON with 1 issue (CWE-94), risk score 35, verdict WARNING. + +### Manual review with file + +```bash +python scripts/review_code.py --file examples/vulnerable_pr_demo.py --json +``` + +Expected: multiple CWEs (94, 328, 798, 22, 78), risk score > 50, verdict REQUEST_CHANGES. + +### Safe file + +```bash +python scripts/review_code.py --file examples/safe_pr_demo.py --json +``` + +Expected: 0 issues, risk score 0, verdict APPROVE. + +## Docker Demo + +```bash +# Start all services +docker compose up --build + +# In another terminal +curl -X POST http://localhost:8003/review \ + -H "Content-Type: application/json" \ + -d '{"code": "eval(user_input)"}' +``` + +### Access the services + +| Service | URL | +|---|---| +| API | http://localhost:8003 | +| API docs (Swagger) | http://localhost:8003/docs | +| Review UI | http://localhost:8501 | +| Dashboard | http://localhost:8502 | + +### Dashboard after seeding data + +```bash +# Submit a few reviews +curl -X POST http://localhost:8003/review/code \ + -H "Content-Type: application/json" \ + -d '{"code": "eval(user_input)", "file_path": "demo.py", "use_rag": false}' + +curl -X POST http://localhost:8003/review/code \ + -H "Content-Type: application/json" \ + -d '{"code": "def add(a, b): return a + b", "file_path": "safe.py", "use_rag": false}' +``` + +Then open http://localhost:8502 for the dashboard. + +## GitHub Dry-Run Demo + +```bash +# Simulates what the GitHub Action produces +python scripts/github_pr_review.py --files examples/vulnerable_pr_demo.py --dry-run +``` + +Expected output: +- Summary markdown with verdict, risk score, issue table +- Inline Comment Plan showing which lines would get comments + +## Smoke Test + +```bash +./scripts/docker_smoke_test.sh +``` + +Verifies: health endpoint → review API → review detail → stats → Streamlit UI pages. + +## RAG Service Demo + +```bash +# Health +curl https://OMCHOKSI108-codereview-agent.hf.space/health + +# Search +curl -X POST https://OMCHOKSI108-codereview-agent.hf.space/rag/search \ + -H "Content-Type: application/json" \ + -d '{"query": "SQL injection prevention", "top_k": 2}' +``` + +## Kaggle Notebook + +Explore the dataset and prototype pipeline on Kaggle: + +https://www.kaggle.com/code/omchoksi04/codereview + +## Screenshot Checklist + +See [docs/screenshots/README.md](screenshots/README.md) for the full screenshot capture guide. diff --git a/docs/deployment_status.md b/docs/deployment_status.md new file mode 100644 index 0000000..dd862c8 --- /dev/null +++ b/docs/deployment_status.md @@ -0,0 +1,67 @@ +# Deployment Status + +## Live Services + +| Service | URL | Status | +|---|---|---| +| RAG Service | https://OMCHOKSI108-codereview-agent.hf.space | **Live** | +| GitHub Repo | https://github.com/OMCHOKSI108/codesec-audit-dataset | Active | +| Hugging Face Dataset | https://huggingface.co/datasets/OMCHOKSI108/CodeSecAudit-RAG | Published | +| Deploy PR | https://github.com/OMCHOKSI108/codesec-audit-dataset/pull/1 | Merged | + +## RAG Service Verification + +All checks pass against the live remote RAG service. + +### Health check + +```json +GET /health +{ + "status": "ok", + "index_loaded": true, + "total_chunks": 2833, + "embedding_count": 2833, + "embedding_model": "sentence-transformers/all-MiniLM-L6-v2" +} +``` + +### Search test + +```json +POST /rag/search {"query": "SQL injection", "top_k": 2} +{ + "results": [ + {"rank": 1, "score": 0.704, "title": "SQL Injection Prevention Cheat Sheet", "cwe_id": "CWE-89"}, + {"rank": 2, "score": 0.686, "title": "SQL Injection Prevention Cheat Sheet", "cwe_id": "CWE-89"} + ], + "metadata": { + "query_time_ms": 154.58, + "total_chunks": 2833 + } +} +``` + +### Main API integration + +```text +CODESEC_RAG_MODE=remote +CODESEC_RAG_SERVICE_URL=https://OMCHOKSI108-codereview-agent.hf.space + +Result: rag_used=True, rag_error=None, verdict=WARNING, risk_score=35 +``` + +### Golden evaluation + +```text +12 golden cases: 12/12 passing (with remote RAG enabled) +``` + +## Pending Deployments + +| Component | Host | Status | Notes | +|---|---|---|---| +| API + Dashboard | Render / Railway | Planned | Awaiting SaaS backend prep | +| MongoDB Atlas | MongoDB Cloud | Planned | Schema designed but not deployed | +| Resend Email | Resend | Planned | Workflows designed but not implemented | +| GitHub App | GitHub Marketplace | Planned | Requires GitHub OAuth + App registration | diff --git a/docs/github_action.md b/docs/github_action.md new file mode 100644 index 0000000..f4406c1 --- /dev/null +++ b/docs/github_action.md @@ -0,0 +1,120 @@ +# GitHub Action — CodeSecAudit AI Review + +## Workflow File + +`.github/workflows/codesec-audit.yml` + +```yaml +name: CodeSecAudit AI Review +on: + pull_request: + types: [opened, synchronize, reopened] +permissions: + contents: read + pull-requests: write +jobs: + review: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + - name: Install package + run: | + pip install --upgrade pip + pip install -e ".[dev]" + - name: Run security review + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + GITHUB_REPOSITORY: ${{ github.repository }} + GITHUB_EVENT_PATH: ${{ github.event_path }} + GITHUB_SHA: ${{ github.sha }} + GITHUB_BASE_REF: ${{ github.base_ref }} + GITHUB_HEAD_REF: ${{ github.head_ref }} + run: python scripts/github_pr_review.py +``` + +## Triggers + +- `opened` — new pull request +- `synchronize` — new commits pushed to the PR branch +- `reopened` — closed PR reopened + +## Permissions + +- `contents: read` — to checkout the repository +- `pull-requests: write` — to post and update PR comments + +## Supported File Types + +- `.py`, `.js`, `.jsx`, `.ts`, `.tsx` + +## Ignored Paths + +``` +node_modules/ dist/ build/ +.venv/ venv/ data/ +release/ notebooks/ .ipynb_checkpoints/ +__pycache__/ *.min.js *.lock +package-lock.json pnpm-lock.yaml +yarn.lock +``` + +Files larger than 200 KB are also skipped. + +## Output + +### Summary Comment + +After the action completes, a PR comment is posted containing: + +- **Verdict** (APPROVE / WARNING / REQUEST_CHANGES) +- **Risk score** (0–100) +- **Issues Found** table with columns: Location, CWE, Severity, Issue, Suggested Fix +- **Notes** section with engine version, RAG status, and limits info + +### Inline Comments + +Inline comments are posted directly on vulnerable added lines in the PR diff: + +- Only on **added lines** (green `+` lines in the diff) +- Deduplication: same `path + line + CWE` is skipped +- Cap: maximum **10 inline comments** per PR +- Non-commentable issues (not in diff) remain in the summary table only + +### Fingerprint Deduplication + +Each inline comment includes a hidden fingerprint. If a comment with the same path, line number, and CWE already exists, the new comment is skipped. This prevents duplicate comments when new commits are pushed. + +## Dry Run + +Test locally without calling the GitHub API: + +```bash +python scripts/github_pr_review.py --files examples/vulnerable_pr_demo.py --dry-run +``` + +The `--dry-run` flag: +- Prints the summary markdown to the terminal +- Shows the Inline Comment Plan table +- Does not call any GitHub API +- Does not save to the review database (use `--save` to save) + +```bash +# Dry run with recent git diff +python scripts/github_pr_review.py --dry-run + +# Dry run with save to review DB +python scripts/github_pr_review.py --dry-run --files examples/vulnerable_pr_demo.py --save +``` + +## Limitations + +- **`use_rag=False` in CI** — RAG retrieval is disabled in the GitHub Action to avoid downloading the embedding model (~80 MB) and ChromaDB index on every run. Will be enabled after `actions/cache` is configured. +- **Non-blocking** — the action always exits 0. Issues do not fail the build or block merging. +- **No paid LLMs** — all detection is rule-based, all fixes are template-based. +- **Inline comment cap** — maximum 10 inline comments per PR. Additional issues are summarized in the table only. +- **Summary only fallback** — if the GitHub API fails to return patch data, the script falls back to git-based detection, producing a summary comment without inline comments. diff --git a/docs/project_status.md b/docs/project_status.md new file mode 100644 index 0000000..2a731ad --- /dev/null +++ b/docs/project_status.md @@ -0,0 +1,42 @@ +# Project Status + +## Component Status + +| Component | Status | Notes | +|---|---|---| +| Dataset (CodeSecAudit-RAG) | **Done** | 28,548 records, 2,833 RAG chunks, published to HF + Kaggle | +| Kaggle Notebook | **Done** | Dataset exploration + RAG prototype | +| Review Engine (core) | **Done** | Critic, fixer, retriever, risk scorer, pipeline | +| GitHub Action | **Done** | PR triggers, summary + inline comments, max caps | +| Inline PR Comments | **Done** | 10-comment cap, duplicate detection via fingerprint | +| Evaluation Framework | **Done** | 12 golden cases, all passing | +| Review History DB (SQLite) | **Done** | CRUD endpoints, pagination, stats | +| Streamlit Dashboard | **Done** | Stats cards, charts, filters, review detail viewer | +| Docker Compose | **Done** | 3-service stack, lightweight default, RAG optional | +| RAG Service (HF Space) | **Deployed** | Live at https://OMCHOKSI108-codereview-agent.hf.space | +| Remote RAG Client | **Done** | `search_remote_rag()`, `remote_rag_available()` in pipeline | +| Deployment Scripts | **Done** | `prepare_hf_rag_space.py`, `deploy_hf_rag_space.py`, smoke tests | +| Render / Railway Deployment | **Pending** | Not yet deployed | +| MongoDB Atlas Integration | **Pending** | Schema designed; code not implemented | +| Resend Email Implementation | **Pending** | Workflows designed; code not implemented | +| GitHub OAuth | **Pending** | Not yet implemented | +| GitHub App Install Flow | **Pending** | Not yet implemented | +| Usage Limit Enforcement | **Pending** | Schema designed; enforcement not yet built | + +## Design Artifacts + +| Artifact | File | +|---|---| +| SaaS Data Model (7 MongoDB collections) | [docs/saas_data_model.md](saas_data_model.md) | +| Email Workflows (4 email types) | [docs/email_workflows.md](email_workflows.md) | +| Usage Limits (free tier: 30 reviews/month) | [docs/usage_limits.md](usage_limits.md) | +| Deployment Strategy | [docs/deployment_strategy.md](deployment_strategy.md) | + +## Known Gaps + +- No authentication anywhere (API, dashboard, RAG service are public in demo mode) +- SQLite is single-writer; concurrent CI runs may queue +- No data retention policy; DB size grows unbounded +- GitHub Action uses `use_rag=False` until caching is configured +- Docker RAG mode adds ~7 GB and ~8 min build time +- HF Space Docker SDK has a known API bug with `huggingface_hub v1.19.0` (workaround: manual Space creation + git push) diff --git a/docs/rag_service.md b/docs/rag_service.md index d97bead..7d4c220 100644 --- a/docs/rag_service.md +++ b/docs/rag_service.md @@ -1,16 +1,35 @@ # RAG Service Architecture +**Live endpoint**: https://OMCHOKSI108-codereview-agent.hf.space + > **Production**: Always set `RAG_API_KEY` on the RAG Space. Without it, the service is **public** — anyone with the Space URL can query your RAG index. The main API should also set `CODESEC_RAG_API_KEY` to match. ## Why Separate RAG? -RAG dependencies (sentence-transformers, ChromaDB, PyTorch) add ~7 GB to the +RAG dependencies (sentence-transformers, PyTorch) add ~7 GB to the Docker image. For free-tier deployment, they should live in exactly one place: the Hugging Face Space RAG service. All other services (API, dashboard, website) stay lightweight (~500 MB) and call the RAG service over HTTP when needed. +## Live Verification + +```bash +# Health +curl https://OMCHOKSI108-codereview-agent.hf.space/health + +# Search +curl -X POST https://OMCHOKSI108-codereview-agent.hf.space/rag/search \ + -H "Content-Type: application/json" \ + -d '{"query":"SQL injection","top_k":2}' +``` + +Expected health response: +```json +{"status":"ok","index_loaded":true,"total_chunks":2833,"embedding_count":2833} +``` + ## Architecture ``` @@ -61,13 +80,16 @@ curl -X POST http://localhost:7860/rag/search \ ## Deploy to Hugging Face Space +> **Status**: The RAG service is already live at https://OMCHOKSI108-codereview-agent.hf.space. +> The instructions below document the deployment process for reference or to redeploy to a new Space. + ### Method A — Python deploy script (recommended) Requires `huggingface_hub` installed. Automates packaging, Space creation, file upload, and secrets. ```bash export HF_TOKEN="your_huggingface_write_token" -export HF_RAG_SPACE_ID="OMCHOKSI108/codesec-rag-service" +export HF_RAG_SPACE_ID="OMCHOKSI108/codereview-agent" export RAG_API_KEY="your-shared-rag-key" python scripts/deploy_hf_rag_space.py @@ -92,7 +114,7 @@ python scripts/prepare_hf_rag_space.py # 2. Push to Hugging Face cd dist/huggingface-rag-space git init -git remote add origin https://huggingface.co/spaces/OMCHOKSI108/codesec-rag-service +git remote add origin https://huggingface.co/spaces/OMCHOKSI108/codereview-agent git add . git commit -m "deploy CodeSecAudit RAG service" git push origin main @@ -143,7 +165,7 @@ curl -X POST http://localhost:7860/rag/search \ ### Verify the remote service ```bash -export CODESEC_RAG_SERVICE_URL=https://your-space.hf.space +export CODESEC_RAG_SERVICE_URL=https://OMCHOKSI108-codereview-agent.hf.space export CODESEC_RAG_API_KEY=your-key python scripts/check_remote_rag_service.py @@ -152,7 +174,7 @@ python scripts/check_remote_rag_service.py Expected output: ``` -Checking remote RAG service at: https://your-space.hf.space +Checking remote RAG service at: https://OMCHOKSI108-codereview-agent.hf.space [1/2] Health check: status: ok @@ -181,10 +203,22 @@ OK: Remote RAG service is healthy and responding # In your main API .env: CODESEC_ENABLE_RAG=true CODESEC_RAG_MODE=remote -CODESEC_RAG_SERVICE_URL=https://your-space.hf.space +CODESEC_RAG_SERVICE_URL=https://OMCHOKSI108-codereview-agent.hf.space CODESEC_RAG_API_KEY=your-internal-key ``` +Verify the connection: + +```bash +python scripts/check_main_api_remote_rag.py +``` + +Expected output: + +``` +OK: Remote RAG used successfully in review +``` + ## Fallback Behavior If the remote RAG service is unreachable: diff --git a/docs/screenshots/README.md b/docs/screenshots/README.md new file mode 100644 index 0000000..8ff2beb --- /dev/null +++ b/docs/screenshots/README.md @@ -0,0 +1,52 @@ +# Screenshots + +This directory should contain screenshots for documentation, demos, and the README. + +## Screenshot Checklist + +Capture the following screenshots and save them in this directory: + +| # | Screenshot | Description | Status | +|---|---|---|---| +| 1 | `pr-summary-comment.png` | GitHub PR page showing the CodeSecAudit summary comment at the bottom | Pending | +| 2 | `inline-comment-eval.png` | Inline comment on an `eval()` line in the PR diff view | Pending | +| 3 | `github-action-run.png` | Successful GitHub Action run log in the Actions tab | Pending | +| 4 | `dashboard-stats.png` | Streamlit dashboard with stats cards, verdict chart, and recent reviews | Pending | +| 5 | `review-detail.png` | Review detail page showing full issue breakdown | Pending | +| 6 | `api-docs.png` | FastAPI Swagger UI at `/docs` showing all endpoints | Pending | +| 7 | `kaggle-notebook.png` | Kaggle notebook preview showing dataset exploration | Pending | +| 8 | `hf-dataset.png` | Hugging Face dataset page for CodeSecAudit-RAG | Pending | +| 9 | `rag-service-health.png` | RAG service health endpoint response | Pending | + +## How to Capture + +### GitHub Screenshots (1–3) + +1. Create a test PR with `examples/vulnerable_pr_demo.py` +2. Wait for the GitHub Action to complete +3. Capture the PR comments section (screenshot 1) +4. Click on an inline comment in the Files Changed tab (screenshot 2) +5. Go to the Actions tab and open the workflow run (screenshot 3) + +### Dashboard (4–5) + +1. Run `docker compose up --build` +2. Submit a few reviews: + ```bash + curl -X POST http://localhost:8003/review/code -H "Content-Type: application/json" \ + -d '{"code": "eval(user_input)", "file_path": "demo.py"}' + ``` +3. Open http://localhost:8502 and capture the dashboard (screenshot 4) +4. Click a review to open the detail view (screenshot 5) + +### API Docs (6) + +1. Run `docker compose up --build` +2. Open http://localhost:8003/docs +3. Capture the Swagger UI page (screenshot 6) + +### External Services (7–9) + +1. Open https://www.kaggle.com/code/omchoksi04/codereview and capture (screenshot 7) +2. Open https://huggingface.co/datasets/OMCHOKSI108/CodeSecAudit-RAG and capture (screenshot 8) +3. Run `curl https://OMCHOKSI108-codereview-agent.hf.space/health` and capture the response (screenshot 9)