FYP Summary Evaluation Refactor

This project is a refactored version of the analysis code from my final year project, Evaluating the Effectiveness of AI-Generated Summaries for Video Lectures.

The original thesis workflow was notebook-based and involved calculating ROUGE and G-Eval scores, cleaning the results manually, and then running statistical tests in a separate notebook. This refactor separates the reusable parts of that workflow into a small Python package with clearer structure, validation, and tests.

Project Structure

FYP Code Refactored/
├── data/
│   ├── rouge_scores.csv
│   ├── geval_scores.csv
│   └── survey_preference_counts.csv
├── outputs/
│   ├── rouge_wilcoxon_results.csv
│   ├── geval_wilcoxon_results.csv
│   ├── survey_binomial_results.csv
│   └── statistical_results_combined.csv
├── scripts/
│   └── run_results_analysis.py
├── src/
│   └── fyp_evaluation/
│       ├── __init__.py
│       ├── constants.py
│       ├── data_loader.py
│       ├── geval_evaluation.py
│       ├── results_analysis.py
│       ├── rouge_evaluation.py
│       └── statistical_tests.py
└── tests/
    ├── test_geval_evaluation.py
    ├── test_rouge_evaluation.py
    └── test_statistical_tests.py

What This Refactor Does

• Loads processed ROUGE, G-Eval, and survey result tables from CSV files.
• Re-runs Wilcoxon signed-rank tests for objective evaluation scores.
• Re-runs binomial tests for survey preference counts.
• Provides reusable ROUGE evaluation logic.
• Separates G-Eval prompt construction and score parsing from the external LLM API call.
• Adds tests for statistical calculations, ROUGE evaluation, and G-Eval validation.

Setup

Create and activate a virtual environment:

python -m venv .venv
.\.venv\Scripts\activate

Install dependencies:

pip install -r requirements.txt

Running the Analysis

From the project root, run:

$env:PYTHONPATH="src"
python scripts/run_results_analysis.py

The generated output files are written to:

outputs/

Running Tests

From the project root, run:

$env:PYTHONPATH="src"
pytest -q

At the time of writing, the test suite contains 14 passing tests covering statistical tests, ROUGE evaluation, and G-Eval validation.

Notes on the Refactor

The original thesis code was written in Jupyter notebooks because the project was exploratory and research-focused. This refactor demonstrates how the analysis could be structured more clearly for maintainability, reproducibility, and code review.

The processed CSV files in data/ were created from the final thesis result tables. The original raw survey exports and Excel workbooks are intentionally excluded from this refactored version because they are not required by the code and may contain participant-related information.

Possible Future Improvements

• Move original transcripts and summaries into structured JSON files.
• Automatically rerun ROUGE evaluation across all lecture clips.
• Add an optional command for rerunning G-Eval using a configured API key.
• Cache G-Eval outputs to avoid repeated external API calls.
• Add generated charts for objective and subjective evaluation results.
• Package the project properly with pyproject.toml.