MCQ Quality Validation Utility

Overview

MCQ Quality Validation Utility is an enterprise-grade, AI-powered automated quality assurance system designed to validate and improve Multiple Choice Questions (MCQs) in bulk. It processes entire Excel spreadsheets containing hundreds or thousands of MCQs, systematically checking them against multiple quality dimensions using advanced LLM-based analysis and deterministic validation logic.

This tool was created to address a critical gap in educational content management: manual quality review of MCQs is time-consuming, inconsistent, and error-prone. By automating quality checks through a multi-gate validation pipeline, this utility ensures consistent, high-quality assessment questions while drastically reducing manual review burden.

---

Why This Project Was Created

The Problem

Educational institutions, online learning platforms, and assessment companies face several challenges with MCQ management:

1. Scale: Processing thousands of MCQs manually is impractical and time-consuming
2. Consistency: Manual reviewers apply subjective criteria, leading to inconsistent quality standards
3. Error Detection: Common issues (incorrect answers, ambiguous options, factual errors) are easily missed in bulk reviews
4. Time Sensitivity: Questions containing outdated references or version-specific content become problematic over time
5. Pedagogical Alignment: Difficulty labels often don't match actual cognitive complexity
6. Language Quality: Grammar, punctuation, and clarity issues can confuse learners
7. Answer Verification: Given answer keys may be incorrect, requiring independent validation

The Solution

This utility implements a 6-gate quality pipeline (G0–G5) plus independent solving and answer validation, enabling:

• Automated structural validation
• Question-option coherence checking
• Temporal stability verification
• Cognitive difficulty validation
• Language and grammar proofing
• Industry-agnostic content verification
• Independent answer key verification
• Option improvement suggestions
• Excel export with detailed feedback and actionable recommendations

---

Architecture & Workflow

High-Level Flow

Excel Input (XLSX)
       ↓
   Parse MCQ Data
       ↓
   Create Payloads (solve-only + full)
       ↓
   ┌─ G0: Input Validation (structure check)
   │
   ├─ G1: Question Alignment (coherence check)
   │
   ├─ G2: Stability (time/version sensitivity)
   │
   ├─ (G3 + G5) Parallel: Difficulty + Industry Agnostic
   │
   ├─ G4: Proofreading (language/grammar)
   │
   ├─ SOLVE: Independent MCQ Solving
   │
   ├─ ANSWER_VALIDATION: Verify provided answer key
   │
   ├─ IMPROVE (Conditional): Fix options if needed
   │
   └─ FINALIZE: Rollup results → Excel columns
       ↓
   Enriched Excel Output (with detailed validation results)

Technical Stack

• Framework: FastAPI (web service)
• LLM Integration: Azure OpenAI (configurable via environment variables)
• Graph Orchestration: LangGraph (multi-stage pipeline)
• Data Format: Excel (XLSX) input/output via openpyxl
• Async: ThreadPoolExecutor for parallel gate execution (G3 + G5 run simultaneously)
• Caching: DiskCache for LLM result memoization (optional)

---

Performance & Scalability

• Batch Processing: Designed for thousands of MCQs
• Parallel Gates: G3 and G5 execute simultaneously (ThreadPoolExecutor)
• Caching: LLM results optionally cached to reduce API calls
• Async: Background job processing; UI remains responsive
• Typical Speed: ~10-30 seconds per MCQ (depends on LLM latency + complexity)

Project Structure

QC/
├── app.py                    # FastAPI application entry point
├── api.py                    # API routes and job management
├── workflow.py               # LangGraph pipeline definition & logic
├── schemas.py                # Pydantic data models
├── llm.py                    # LLM interaction wrapper
├── excel_actions.py          # Excel parsing & writing
├── __init__.py
├── requirements.txt          # Python dependencies
├── README.md                 # This file
├── prompts/                  # LLM system prompts
│   ├── system_prompt.md      # Base system instruction for LLM
│   ├── g0_input_validation.md
│   ├── g1_question_alignment.md
│   ├── g2_stability.md
│   ├── g3_difficulty.md
│   ├── g4_proofread.md
│   ├── g5_industry_agnostic.md
│   ├── solve_mcq.md
│   ├── answer_validation.md
│   └── improve_options.md
├── qc_cache/                 # DiskCache for LLM memoization (auto-created)
└── .env                      # Environment variables (credentials)

Features:

• Structured output (JSON mode) ensures parseable responses
• Schema validation catches malformed outputs
• Error handling for LLM failures
• Retry logic for transient failures

---