NLP Model Benchmarking Suite for Structured Information Extraction

TL;DR

A desktop NLP benchmarking system that evaluates and compares Hugging Face models on structured information extraction tasks (job postings).

It provides a full pipeline:

• Model discovery and selection (Hugging Face integration)
• Batch inference across multiple architectures (QA, NER, Text2Text)
• Semantic evaluation using embedding-based cosine similarity
• Fair and standardized model comparison
• Automated report generation and visualization

The goal is to make NLP model evaluation reproducible, comparable, and fully automated within a single GUI-based environment. A desktop application for systematically evaluating and comparing Hugging Face NLP models on structured information extraction tasks from job posting datasets. Built during an internship, this tool automates the full pipeline — from model discovery and selection to inference, semantic evaluation, and report generation — through a unified PyQt5 GUI.

---

Overview

Extracting structured fields (company name, job title, location, work type, etc.) from unstructured job posting descriptions is a non-trivial NLP task. Different model architectures (QA, NER, Text2Text) approach this problem differently, and evaluating them objectively at scale requires a reproducible, automated pipeline.

This project provides exactly that: a self-contained benchmarking environment where you can browse models on Hugging Face, select candidates, run them against a labeled dataset, and compare their performance through semantic similarity scoring — all from a single desktop interface.

---

Why This Project Exists

Standard NLP benchmarks rarely account for heterogeneous architectures under the same evaluation conditions. This system was designed to enable objective comparison of QA-based, NER-based, and generative models under a unified evaluation protocol — specifically for structured information extraction from job posting descriptions.

Rather than relying on exact string matching, evaluation is grounded in semantic similarity, allowing fair comparison across architectures that produce differently-phrased but semantically equivalent outputs.

---

Use Case / Problem Statement

This project was developed to address a key limitation in NLP research and applied ML workflows:

Lack of standardized, reproducible benchmarking pipelines for structured information extraction from unstructured text.

In real-world applications such as job boards, recruitment systems, and talent analytics platforms, critical information (e.g., company name, job title, location, work type) must be extracted from noisy, unstructured text.

However:

• Different NLP architectures (QA, NER, Text2Text) produce non-comparable outputs
• Evaluation is often inconsistent (string matching, manual inspection, or task-specific metrics)
• Reproducibility across models and datasets is difficult
• There is no unified interface for model selection, execution, and evaluation

This system provides a unified benchmarking environment that:

• Standardizes structured extraction evaluation
• Enables fair comparison across heterogeneous model architectures
• Introduces semantic evaluation via embedding-based similarity
• Automates the full ML evaluation pipeline from inference to reporting

Features

• Model Discovery UI — Search and browse Hugging Face models with metadata (downloads, likes, author, tags). Select models and export your curated list as JSON.
• Multi-Architecture Support — Supports question-answering, text-generation, and NER pipeline types.
• Automated Batch Inference — Runs each selected model sequentially against the dataset, isolating outputs into individual SQLite databases per run.
• Semantic Evaluation via Cosine Similarity — Model answers are embedded using intfloat/multilingual-e5-large-instruct and compared against ground-truth vectors using cosine similarity — going beyond exact string matching.
• Two Evaluation Strategies:
  • Standard Evaluation: Score ≥ threshold → success.
  • Fair Evaluation: Skips samples where the ground-truth value is not present in the source text, preventing unfair penalization of the model.
• Automated Report Generation — Per-model, per-field scores are persisted into a report database for cross-model comparison.
• Visual Report Viewer — Inspect results through a dedicated report UI within the same application.
• Self-Initializing Ingestion Pipeline — The system automatically initializes SQLite schemas, resolves dataset sources (local → remote → fallback), and prepares a fully runnable environment at startup, eliminating manual setup steps.

---

Architecture

Pipeline Flow

Model Selection UI → Batch Runner → SQLite Outputs → Vectorization → Cosine Evaluation → Report DB → Visualization UI

Code Structure

.
├── main_menu.py              # Application entry point (PyQt5 MainWindow)
├── build_report_backend.py   # Evaluation orchestrator & report writer
├── view_report_ui.py         # Report visualization UI
│
├── core/
│   ├── algorithm.py          # Model runner & pipeline orchestration
│   ├── vectorize.py          # Sentence embedding & .pt vector storage
│   ├── eval.py               # Cosine-based evaluation logic
│   ├── db_service.py         # SQLite abstraction layer
│   ├── db_prep.py            # Database initialization utilities
│   ├── query.py              # Dataset query helpers
│   └── models/
│       ├── model_qa.py       # Question-Answering pipeline wrapper
│       ├── model_ner.py      # Named Entity Recognition pipeline wrapper
│       └── model_text2text.py # Text-to-Text generation wrapper
│
├── pages/
│   ├── model_search_page.py  # Hugging Face model browser UI
│   ├── gui.py                # Report UI widget
│   ├── file_dynamic.py       # Dynamic page loader (run models / vectorization)
│   ├── terminal.py           # In-app terminal output widget
│   └── search_models.py      # Model search logic
│
├── model_lists/              # Exported model selection JSONs
│   ├── selected_models_1.json
│   ├── selected_models_2.json
│   └── selected_models_3.json
│
└── img/
    └── icon.png

---

Evaluated Fields

The benchmark targets six structured fields extracted from job posting descriptions:

Field	Description
company_name	Name of the hiring company
title	Job title / position name
summary	Role summary or description excerpt
location	Geographic location of the role
application_type	Application method (e.g., online)
work_type	Employment type (full-time, remote…)

---

How It Works

Step 1 — Select Models

Launch the application and navigate to File → Search Models. Browse Hugging Face models filtered by task type. Select your candidates using the checkboxes and export the list via File → Export Selected Models. This saves a JSON file under model_lists/.

Step 2 — Run Inference

Navigate to AI Operations → Run Models. The runner reads your model list, copies a template database for each model, and executes inference against the dataset. Each model's outputs are stored in a separate .db file under a versioned model_outputs/test_run_XXX/ folder. A model_output_map.json is generated to track run metadata.

# Programmatic usage
from core.algorithm import algorithm

algorithm.run_all_qa_models(
    number_rows=1000,
    models_json_path="model_lists/selected_models_2.json"
)

Step 3 — Vectorize

Navigate to AI Operations → Run Vectorization, or run programmatically:

from core.vectorize import vectorize

vectorize.run_vectorization_from_json_combined(
    num_rowsx=1000,
    json_path="model_outputs/test_run_001/model_output_map.json",
    vectorize_dataset=True  # Set False if dataset vectors already exist
)

This encodes both the ground-truth dataset columns and the model answers using a multilingual sentence transformer, saving tensors as .pt files under vectors/.

Step 4 — Evaluate & Generate Report

from build_report_backend import report

report.run_all_model_reports_from_json(
    config_path="model_outputs/test_run_001/model_output_map.json",
    report_db_path="report/report.db"
)

Each model receives a score per field (0–100), stored in the report database.

Step 5 — View Results

Navigate to File → Model Answers Interface to inspect per-model, per-field results visually.

---

Installation

Requirements

• Python 3.11+
• CUDA-capable GPU — strongly recommended for the vectorization step (CPU fallback is supported but slow)

Setup

git clone https://github.com/<your-username>/<repo-name>.git
cd <repo-name>
make install

make install will:

1. Locate a compatible Python 3.11+ interpreter on your system
2. Create an isolated virtual environment under .venv/
3. Attempt to detect your CUDA environment and select an appropriate PyTorch build (CUDA 12.x / 11.x / CPU)
4. Install all pinned dependencies from requirements.txt
5. Run an import verification check and report installed versions

Available Commands

Command	Description
make install	Create venv and install all dependencies
make run	Launch the desktop application
make check	Verify all critical imports are working
make clean	Remove the virtual environment
make reinstall	Full clean + fresh install

Run

make run

Dependency notes: Three core packages are version-pinned for mutual compatibility: transformers==4.54.1, sentence-transformers==5.0.0, huggingface_hub==0.34.3. PyTorch is installed separately by the setup process to match your hardware. Do not upgrade these packages independently.

---

Dataset

The project expects a SQLite database at dataset/data.db containing a JobPostings table with at minimum the six evaluation fields listed above plus a description column used as the model input context.

---

Output Structure

model_outputs/
└── test_run_001/
    ├── model1_output.db       # Raw model answers (SQLite)
    ├── model2_output.db
    └── model_output_map.json  # Run metadata & completion status

vectors/
├── dataset/
│   └── dataset_n_1000.pt     # Ground-truth embeddings
└── answers/
    └── test_run_001/
        ├── model1_output.pt   # Per-model answer embeddings
        └── model2_output.pt

report/
└── report.db                  # Final per-model, per-field scores

---

Example Model List Format

[
  {
    "source": "question-answering",
    "data": [
      { "model_name": "deepset/roberta-base-squad2" },
      { "model_name": "deepset/bert-base-cased-squad2" }
    ]
  }
]

---

Limitations

• Evaluation relies on embedding-based cosine similarity, which measures semantic proximity rather than strict factual correctness — high scores do not guarantee exact field extraction.
• NER pipeline support is experimental; performance on the target fields was limited during development.
• GPU acceleration is required for practical vectorization at scale; CPU inference is functional but significantly slower.
• Model performance is sensitive to prompt phrasing; the included question templates were tuned for the job posting domain specifically.

---

Tech Stack

• Python 3.11
• PyQt5 — Desktop GUI framework
• Hugging Face Transformers — NLP model inference
• Sentence Transformers — Semantic embedding (multilingual-e5-large-instruct)
• PyTorch — Tensor operations & vector storage
• SQLite — Lightweight storage for dataset, model outputs, and reports
• Pandas — Data manipulation and evaluation logic

---