PhilParse

PhilParse converts philosophical texts into structured, sequence-aware knowledge graphs that can be queried to interrogate argumentative form and logical consistency. Text is decomposed into atomic units of meaning (sentences), each classified and linked to its neighbors according to a formal ontology. The result is a graph that preserves the linear reading order while exposing the logical relationships between ideas.

Approach

The pipeline has three stages:

1. OCR and parsing — PDFs are OCR'd with Mistral's OCR API and parsed into a hierarchical structure (chapters → sections → paragraphs → atoms). Chapter boundaries are taken from PDF metadata when available, with regex-based detection as a fallback.
2. Atomic classification — Each atom is classified into one of 18 categories (Claim, Premise, Conclusion, Rebuttal, Concession, Implication, Definition, Stipulation, Example, Distinction, Position Statement, Quotation, Citation, Roadmap, Thesis, Problem Statement, Inquiry, Error).
3. Local-context linking — Each atom is linked to preceding atoms in its local context using typed relationships (Supports, Rebuts, Clarifies, Illustrates, Implies, Quantifies, Addresses, Outlines, Attributes, Cites, Continues). Relationships are validated against the ontology's valid_sources / valid_targets rules.

Atoms are processed sequentially within a local context (paragraph/section) while chapters are processed in parallel, which keeps the LLM context window bounded and produces naturally closed subgraphs corresponding to self-contained arguments.

Ontology-driven adaptability

Classification and linking are driven entirely by two files:

• src/models/taxonomy.json — the set of atom classes.
• src/models/ontology.json — the relationship types and their valid source/target class pairs.

The LLM prompt (src/llm/prompts/atom_graph.md) and the database schema are agnostic to the specific domain. To retarget PhilParse to a different domain (legal reasoning, scientific argumentation, policy debate, etc.), the classification schema only needs to be changed once — in these two files — and the rest of the pipeline carries over unchanged.

Status

The backend is functional end-to-end: upload a PDF, parse it, construct the atomic graph, and query the graph over the API. The following are not yet implemented: metagraph (high-level overlay) is scaffolded in src/graph/metagraph.py but not wired into the API; vector embeddings have columns in the schema but are not populated or queried; no frontend exists.

Project structure

├── src/
│   ├── api/
│   │   ├── api.py              # FastAPI app: document CRUD, processing pipeline, graph queries
│   │   └── models.py           # Pydantic request/response models
│   ├── database/
│   │   └── pgvector.py         # PostgreSQL repository (hierarchy, atoms, relationships)
│   ├── graph/
│   │   ├── construct_graph.py  # Atomic graph construction (parallel chapters, sequential atoms)
│   │   ├── metagraph.py        # Overlay summarization graph (scaffolded, not wired in)
│   │   └── ontology.md         # Human-readable ontology/taxonomy reference
│   ├── llm/
│   │   ├── llm_client.py       # Mistral client with rate limiting and prompt caching
│   │   └── prompts/
│   │       ├── atom_graph.md   # Classification + relationship extraction prompt
│   │       └── summarize.md    # Summarization prompt for the metagraph
│   ├── models/
│   │   ├── ontology.json       # Relationship ontology with validation rules
│   │   ├── taxonomy.json       # Atom classification taxonomy
│   │   └── schema.json         # Parsed-document JSON schema
│   ├── preprocessing/
│   │   ├── metadata.py         # PDF metadata / ToC extraction for chapter boundaries
│   │   ├── ocr.py              # Mistral OCR with chapter-aware chunking
│   │   ├── parse.py            # Metadata-first, regex-fallback text parser
│   │   └── clean.py            # Text cleaning/normalization
│   └── main.py                # Uvicorn entrypoint
├── postgres/docker/
│   ├── Dockerfile              # PostgreSQL 17 + pgvector
│   └── init/PGVECTOR_INIT.SQL  # Schema: documents, document_structure, atoms, relationships
├── docker-compose.yml
├── Dockerfile
└── requirements.txt

Getting started

1. Prerequisites: Docker and Docker Compose.
2. Configure: copy .env.example to .env and set MISTRAL_API_KEY and the Postgres credentials.
3. Launch: docker-compose up starts the app and a PostgreSQL instance with pgvector.
4. API docs: http://localhost:8000/docs once running.

API

All endpoints are prefixed with /api.

Method	Path	Purpose
GET	/documents	List documents (paginated).
GET	/documents/{id}	Retrieve a single document.
DELETE	/documents/{id}	Delete a document and all associated data.
POST	/documents/process	Upload a PDF, run OCR, parse structure, and store.
POST	/documents/{id}/graph	Start atomic graph construction in the background.
GET	/documents/{id}/graph/progress	Poll background graph construction status.
POST	/documents/{id}/process	Run the full parse → graph pipeline in the background.
GET	/documents/{id}/structure	Get the hierarchical structure tree.
GET	/documents/{id}/graph/context	Get atoms and relationships within a structure element.
GET	/atoms/{id}/neighborhood	Get an atom and its directly connected neighbors.

Roadmap

• Metagraph: wire in src/graph/metagraph.py to produce an overlay graph summarizing self-contained arguments and exposing cross-argument relationships, enabling navigation between atomic detail and high-level argument structure.
• Semantic search: populate the vector columns and add endpoints for similarity queries over atoms and structures.
• Frontend: file upload, a reader view with atoms color-coded by classification, and an interactive graph visualization.
• Logical validation: automated detection of fallacies, circular reasoning, and argumentative inconsistencies against the constructed graph.
• Cross-document analysis: compare argumentative structures across texts.
• Cost reduction: the current LLM-driven pipeline is expensive (e.g. one test text costs ~10.5M input / 1.5M output tokens); once the schema stabilizes, task-specific classifiers trained on LLM outputs could replace the general-purpose calls.

License

See LICENSE.