An academic project to evaluate and compare the performance of PEGASUS, T5, LED, and GPT-2 on the task of abstractive summarization of movie subtitles.
This project tackles the challenging task of generating coherent, abstractive summaries from noisy and dialogue-rich movie subtitle data. Unlike well-structured news articles, subtitles present unique difficulties due to their conversational nature and fragmented narrative.
Our goal was to systematically benchmark four state-of-the-art Transformer-based models to understand their capabilities and limitations in this specific domain, providing valuable insights for summarizing dialogue-heavy text.
As a primary contributor to this two-person team, I was responsible for the following key areas:
- Foundational Development: Developed the initial codebase, established the complete project workflow, and performed early-stage program optimizations.
- Data Engineering: Managed the entire data pipeline, including cleaning and preprocessing subtitles from the OpenSubtitles dataset and aligning them with the CMU Movie Summary Corpus.
- Model Implementation & Training: Implemented the fine-tuning process for all four models (PEGASUS, T5, LongT5, LED, GPT-2) using the Hugging Face Transformers library.
- Performance Analysis: Conducted a comprehensive quantitative evaluation using ROUGE, BLEU, and BERTScore metrics to analyze and compare the performance of each model.
- Languages: Python
- Libraries & Frameworks: PyTorch, Hugging Face Transformers, Scikit-learn, Pandas, NLTK
For a detailed breakdown of the methodology, experiments, results, and in-depth analysis, please see the full project report included in this repository.
This project was completed in collaboration with Yansheng Ma.