DRIFTS / CO2 Hydrogenation Literature Mining Pipeline

An automated pipeline that extracts structured data from academic literature on CO2 hydrogenation catalysis, to support a SHAP-based machine learning model for catalyst design guidance. Built during a research internship at the University of Manchester, supervised by Dr. Xiaolei Fan, Dr. Jiyizhe Zhang and Dr. Niu Jiangqi.

The pipeline works across two paper corpora:

• DRIFTS set — ~43 main papers + 33 SI papers rich in Diffuse Reflectance Infrared Fourier Transform Spectroscopy (DRIFTS) data
• Cu-catalyst set — ~95 papers on Cu-based catalysts

It extracts catalyst descriptors, reaction conditions, DRIFTS peak assignments, reaction intermediates, and performance metrics from paper text, then links these into a reaction-pathway knowledge graph.

Repository Structure

Text Extractor/

Live and batch extraction of paper-level metadata (catalyst composition, reaction conditions, product selectivity, etc.) into structured JSON, using Claude's forced tool-use extraction.

• batch_extractor.py — batch-mode extraction across both corpora (--corpus drifts / --corpus cu_catalyst) via Anthropic's Message Batches API
• build_paper_database.py — builds a paper-level Excel database from the batch extractor's JSON outputs

Extraction Prompts Files/

Shared prompt and schema definitions used by the extractors above.

• system_prompt.py — general extraction rules sent via the API's system parameter
• json_schema.py — field-by-field schema description sent as part of the extraction prompt
• Pydantic_schema_provider_agnostic.py — a provider-agnostic Pydantic schema usable across Gemini, OpenAI, and Anthropic tool-calling interfaces

Excel Extraction Pipeline/

Extracts DRIFTS peak data directly from papers' supplementary Excel files (where raw wavenumber/intensity data is already digitized), and assigns peaks to reaction intermediates via a controlled-vocabulary lookup table.

• inspect_excel_structure.py — run first on any new paper's supplementary Excel file, to check sheet layout before extraction
• drifts_excel_pipeline.py — parses DRIFTS peak data from Excel and assigns peaks to intermediates
• intermediate_lookup.py — loader for the wavenumber → intermediate controlled-vocabulary table
• intermediate_lookup.json — the controlled-vocabulary data itself (master table of DRIFTS wavenumber → reaction intermediate assignments), kept as plain data separate from the loader so it can be edited/reviewed on its own
• test_run_multi_papers.py — per-paper extraction configs (Excel path, figures, wavenumber ranges) extended across multiple papers

Drifts Pathway Extractor (Opus)/

Second-pass extraction of the mechanistic reaction-pathway narrative (intermediate-to-intermediate reaction steps, spectator species, competing pathways) from paper text and SI, using Claude Opus for extraction judgment calls, then merges results into the paper-level database.

• drifts_pathway_schema.py — Pydantic schema for pathway narratives
• drifts_pathway_extractor.py — live extractor over paper PDFs
• drifts_pathway_batch_extractor.py — batch-mode version via the Message Batches API, scoped to DRIFTS-corpus papers already flagged has_drifts
• merge_drifts_database.py — merges pathway JSON and digitized peak JSON outputs into the paper-level Excel database as two new sheets: DRIFTS_Reaction_Steps and DRIFTS_Peak_Assignments

knowledge_graph_prototype.py

Builds a catalyst → intermediate → product knowledge graph from the merged DRIFTS_Reaction_Steps sheet, using NetworkX. Nodes are catalysts, intermediates, and products; edge weight reflects how many papers support a given reaction step. Outputs a presentation PNG plus queryable graph objects (.gpickle and .graphml, the latter Neo4j-compatible via APOC).

Pipeline Flow

PDF papers
   |
   v
Text Extractor/  (paper-level metadata: catalyst, conditions, selectivity)
   |
   +--> Excel Extraction Pipeline/  (DRIFTS peaks from supplementary Excel)
   |
   +--> Drifts Pathway Extractor (Opus)/  (mechanistic reaction-step narrative)
   |
   v
merge_drifts_database.py  -->  paper-level Excel database
   |
   v
knowledge_graph_prototype.py  -->  reaction-pathway knowledge graph
   |
   v
(planned) feature engineering + SHAP-based ML model for catalyst design

Requirements

• Python 3.10
• Key libraries: pdfplumber, networkx, pydantic, openpyxl, scipy
• Anthropic API key (ANTHROPIC_API_KEY environment variable) for the extraction scripts

Status

Internship deliverables complete: batch extraction pipeline for both corpora, merged DRIFTS database, and a static knowledge graph prototype. Ongoing work (possible dissertation continuation) includes intermediate name normalization via controlled vocabulary, and building toward SHAP-based structure-performance modeling.