feat: Ground truth evaluation system - #9
Merged
Conversation
- Created comprehensive ground truth schema v4 (footnotes, citations, bibliography) - Added candidate page finder spike (31_candidate_page_finder.py) - Created symlink test_files -> spikes/sample_pdfs - Draft proprietary citations design doc (needs revision) - Added candidate reports for Kant, Heidegger, Derrida texts Next: Fix margin detection, organize sample PDFs, research segmentation tools
Ground Truth Schema (v1.1.0): - Hierarchical document-centric structure with multi-page support - Added sous_rature element type for under-erasure text - Removed ocr_confidence (confidence belongs to pipeline, not truth) - Added scan_quality and difficulty fields for test stratification - Normalized bboxes (0-1 coordinates), language codes (ISO 639-1) Scripts: - generate_draft.py: Auto-generate draft YAML from PDF extraction - visualize.py: HTML review interface with bbox overlays - validate.py: Schema validation with sous_rature support Test Planning: - COMPREHENSIVE_TEST_PLAN.md: Full evaluation methodology - LAYOUT_TEST_PLAN.md: Layout segmentation testing approach - selected_pages.json: 6 test pages across difficulty levels - baseline_results.json: Initial extraction baseline Spikes: - 32_layout_segmentation.py: Docling layout model exploration - docling_batched.py: Batch processing for layout analysis
Comprehensive design for ground truth annotation interface: Layout: - Sidebar: mode selection (new/edit), page nav, status, zoom - Main: PDF viewer (left) + YAML editor (right) - Bottom: Element panel + action bar Key Features: - Current Page / Full Document YAML view toggle - Annotation status management (pending/annotated/verified) - Live validation with error display - Unsaved changes indicator Implementation Phases: - Phase 1 (MVP): View, YAML edit, validate, save - Phase 2: Undo/redo, element panel editing - Phase 3: Re-extraction integration - Phase 4: Drawable canvas, keyboard shortcuts Dependencies: streamlit, streamlit-ace, streamlit-drawable-canvas
Pipeline Integration (PIPELINE_INTEGRATION.md): - Ground truth → normalized element format - ScholarDocument → normalized format - Element matching algorithm with similarity scoring - Evaluation script design (single doc + batch) - Regression testing integration - CLI interface specification Testing Methodology (TESTING_METHODOLOGY.md): - Three-level evaluation: detection, localization, content - Metrics: precision, recall, F1, text similarity - Baseline format and threshold guidelines - Comparison workflows: regression, A/B, error analysis - Test stratification by difficulty/type/document - Continuous improvement loop - Reporting: CLI, HTML, JSON formats - CI/CD integration with GitHub Actions
- Add render_page_image() returning PIL Image for st.image() display - Support highlight_region_id for current selection feedback - Add ground-truth dependency group (streamlit, streamlit-ace, etc.) Prepares visualize.py for Annotation UI Phase 1 integration.
Evaluation Library (ground_truth/lib/): - normalize.py: Convert GT and ScholarDocument to common format - matching.py: Element matching with configurable thresholds - metrics.py: Precision, recall, F1, and aggregate metrics - reports.py: CLI, JSON, and HTML report generation Scripts: - annotate_ui.py: Streamlit UI for viewing/editing ground truth YAML - evaluate.py: CLI for running extraction evaluation Tests: - 44 unit tests for normalize, matching, metrics modules - Test fixtures (minimal_valid.yaml, multi_page_footnote.yaml) Updates .gitignore to exclude lib/ but include ground_truth/lib/.
- Add test_ground_truth_regression.py for CI/CD integration - Skips if no verified ground truth documents exist - Tests extraction meets baseline thresholds - Validates ground truth file consistency - Add compare.py for A/B evaluation comparison - Compare baseline vs candidate metrics - CLI and HTML output formats - --fail-on-regression flag for CI - Add initial baseline placeholder (ground_truth/baselines/initial.json)
STACK.md, INTEGRATIONS.md, ARCHITECTURE.md, STRUCTURE.md, CONVENTIONS.md, TESTING.md, CONCERNS.md — 1772 lines total.
ScholarDoc: PDF extraction library producing rich ScholarDocument representation for RAG, Anki, citation management, and corpus analysis workflows.
Files: - ground_truth_evaluation.md - pdf_extraction_architecture.md - experimentation_frameworks.md - repo_structure.md - SUMMARY.md Key findings: - Architecture: Reader->IR->Writer consensus pattern confirmed; enrich IR with bboxes/fonts/confidence - Evaluation: Layered metrics (CER/WER -> F1 -> domain-specific); jiwer+rapidfuzz for computation - Critical pitfall: Zero verified GT documents exist despite built evaluation library
4 research documents (GT evaluation, PDF architectures, experimentation frameworks, repo structure) plus synthesis. 5-phase roadmap with 9 requirements mapped. Config and state tracking initialized.
ScholarGT is now an independent GT annotation platform, not just a ScholarDoc support tool. Milestone 1 redesigned around 5 phases: schema design, extractor interface, experimentation framework, annotation tool, and design validation. ScholarDoc improvements deferred to milestone 2. 15 requirements defined across 6 categories.
Superseded by GSD (Get Shit Done) framework. Old infrastructure: - 12 agents (analyze-logs, code-reviewer, diagnose, etc.) - 15 commands (auto, checkpoint, debug, explore, etc.) - 6 hooks (block-dangerous, enforce-branch, post-edit, etc.) - 27 session logs, 5 archived plan logs, signals dir These served the earlier ad-hoc workflow and are no longer needed.
GSD (Get Shit Done) replaces the old ad-hoc workflow with structured phase planning and execution: - 11 GSD agents (planner, executor, verifier, debugger, etc.) - GSD commands and workflow infrastructure - GSD hooks (check-update, statusline) replacing old Python hooks - Simplified settings.json with GSD hook configuration - .claude-template for new project bootstrapping Also updates: - .serena/project.yml: updated config fields
- Remove !spikes/sample_pdfs/*.pdf exception from .gitignore - Add spikes/output/ to .gitignore (generated spike results) - Create spikes/sample_pdfs/MANIFEST.md documenting 20 test corpus PDFs - PDFs were already untracked in prior commit; gitignore now prevents re-tracking
These planning documents are superseded by .planning/ROADMAP.md but contain prior-art context valuable for Phase 1 (schema), Phase 3 (testing methodology), and Phase 4 (annotation tool) design work. - DIAGNOSTIC_PLAN.md: Early ground truth diagnostic analysis - IMPLEMENTATION_PLAN.md: Original implementation strategy - README.md: Ground truth directory overview
- Add 00-02-SUMMARY.md documenting infrastructure swap - Update STATE.md: plan 2/3, 6% progress, session continuity
- 00-01-SUMMARY.md: documents gitignore updates, PDF manifest, ground_truth commits - STATE.md: advance to plan 3 of 3, record metrics and decisions - ROADMAP.md: check off 00-01 and 00-02 plan completion
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Complete ground truth evaluation system for measuring and tracking extraction quality.
Schema & Documentation
Evaluation Library (
ground_truth/lib/)Scripts
Testing
test_ground_truth_regression.pyCI/CD Ready
--fail-on-regressionflag for compare.pyTest plan