Hidden Texas History (HTH)
Hidden Texas History (HTH) is an open-source historic-document research framework for turning archival source material into reproducible, reviewable research evidence.
HTH is designed for difficult historical collections: photographed manuscripts, bound registers, irregular page geometry, degraded images, handwritten records, and source material that must retain a clear chain of provenance. The framework separates source preservation, processing, calibration, validation, publication, and later research stages so that every result can be traced back to the exact source material and software that produced it.
The current reference collection is HTH-0001 — San Antonio Baptisms, 1788–1824 and 1858–1898, but the pipeline and detector framework are intentionally collection-independent.
HTH provides a reproducible pipeline for:
- acquiring or importing archival source material while preserving source identity and provenance;
- extracting and normalizing document images;
- detecting the physical page or document region in difficult captures;
- calibrating and comparing classical, learned, and fused document detectors;
- validating detector behavior against frozen Golden Sets;
- recording machine-readable evidence, diagnostics, timing, and build provenance;
- publishing curated production results separately from test and calibration history; and
- providing a foundation for transcription, translation, indexing, citation, and historical research.
The project favors preserved evidence over destructive cleanup. Source material remains identifiable and reproducible while derived images, geometry, analysis, and research outputs are generated as separate artifacts.
authorized archival source
↓
source release + manifest + SHA-256 provenance
↓
STAGE_PREPROCESS
↓
STAGE_DETECT_CURRENT
↓
approved detector inference
↓
STAGE_VALIDATE_OUTPUTS
↓
STAGE_PUBLISH_PRODUCTION / STAGE_PUBLISH_TEST
detector research/calibration (separate path)
↓
STAGE_DETECT_CANDIDATES → Golden Set evaluation → persisted calibration intelligence
↓
transcription → translation → indexing → citation → research
Stage names are stable HTH concepts rather than GitHub Actions implementation details. The same pipeline architecture can therefore be executed by GitHub-hosted, self-hosted, or future execution environments without changing the research model.
HTH currently carries 47 registered document/page detectors spanning classical geometry, learned historical-document segmentation, OpenCV primitives, and three generations of fusion. The authoritative inventory is the Document Detector Catalog.
Historic document images are rarely clean scans. Pages may be skewed, cropped, shadowed, surrounded by viewer chrome or background material, partially obscured, or captured at inconsistent scale. HTH treats physical-page detection as a measurable research-engineering problem rather than a one-off image-processing step.
The detector framework supports multiple independent approaches, including classical image-processing methods, historical-document neural models, and detector fusions. Detectors expose their geometry, confidence, timing, diagnostics, and debug evidence through a common framework.
Calibration is evaluated against manually approved Golden Sets using reproducible parameter spaces and persisted calibration evidence. Regression reports track metrics such as average IoU, minimum IoU, standard deviation, failures, parameter influence, search coverage, and runtime. Production inference resolves the strongest compatible approved calibration rather than silently adopting an unverified result.
HTH-0001, the original five-page Golden Set, is frozen so historical calibration evidence retains a stable identity. Future Golden Sets can broaden the corpus without changing the meaning of earlier results.
HTH deliberately separates three responsibilities:
Source repository
Collection identity, source metadata, manifests, rights/provenance notes,
and immutable source-release ownership.
Pipeline repository (this repository)
Code, configuration, workflows, tests, detector implementations,
calibration machinery, schemas, and documentation.
Results / publication repository
Curated production outputs plus isolated test and calibration history.
Large immutable source masters are distributed as source-release assets rather than being embedded in normal Git history. Release manifests record asset names, sizes, SHA-256 hashes, source repository identity, release identity, and source commit provenance. Pipeline publications similarly record the exact source, pipeline commit, workflow run, detector calibration, and publication commit used to create them.
The goal is straightforward: a historical conclusion should remain auditable back through the derived evidence to the source material that supports it.
The active production collection contains 928 pages derived from San Antonio baptismal records covering 1788–1824 and 1858–1898. It is being used both for historical research and as the proving corpus for HTH's reusable document-processing architecture.
The broader historical objective that started the project is to locate, document, transcribe, translate, and contextualize primary-source evidence concerning Juana Navarro Alsbury, an Alamo survivor, while building tooling useful well beyond that individual research question.
HTH's source model uses stable collection and page identities so improved source renditions can replace older captures without discarding annotations, calibration history, citations, or research relationships.
HTH is source-neutral and does not assume that archival material is freely redistributable or available through unattended automation.
Acquisition must use access permitted by the source custodian: an official download mechanism, an approved API or integration, material supplied directly by a repository, or other authorized access. HTH records provenance and source identity so that acquisition method, rights status, and later source replacements can remain explicit rather than becoming hidden implementation details.
The framework is designed to benefit from direct archive-quality source images when authorized access is available. Lower-quality historical captures can remain as provenance while a better rendition becomes the preferred processing source.
| Path | Purpose |
|---|---|
hth/ |
Core Python pipeline, detector, calibration, reporting, and provenance code |
config/ |
Pipeline, detector, Golden Set, and analysis configuration |
.github/workflows/ |
Preprocess, regression, calibration, optimizer, review, freeze, and reporting workflows |
tests/ |
Repository regression and invariant test suite |
tools/ |
Research, review, source-release, and operational tooling |
docs/ |
Architecture, detector, calibration, acquisition, publication, and development documentation |
Useful starting points:
- Architecture — repository responsibilities, canonical stages, reporting, and durable design principles.
- Workflow architecture — CI/execution organization and workflow responsibilities.
- Preprocessor — source extraction and preprocessing behavior.
- Canonical Build Evidence — deterministic build identity, audit, reuse, and forced-verification contract.
- Chromatic normalization — conservative color-cast assessment, bounded methods, held-out validation, and CBE integration.
- Restoration normalization — four-stage denoising/artifact suppression followed by conservative sharpening/detail enhancement, with immutable releases and CBE reuse.
- Physical geometry evaluation — geometry-analysis and validation model.
- Regression — detector calibration, metrics, search spaces, and reporting conventions.
- Canonical learned-evidence cache — reusable neural evidence identity, local/release resolution, validation, and recovery.
- Calibration selection — how approved detector evidence is selected for inference.
- Golden Set — reference geometry and frozen Golden Set policy.
- Document detector review — review workflow for detector output.
- Source releases — immutable source assets and verification contract.
- Acquisition pipeline — source-neutral acquisition architecture and authorization boundary.
- Publication — production/test publication and provenance reporting.
- Project status — historical project status and objectives.
- HTH development standard — authoritative engineering conventions for contributors.
Detector-specific documentation is also maintained under docs/ for the implemented classical, learned, and fusion approaches.
HTH is built around a few durable rules:
- Preserve the source. Derived processing never substitutes silently for source evidence.
- Make provenance executable. Builds record enough identity to reproduce and audit their inputs and decisions.
- Measure detector quality. Golden Sets, regression, parameter calibration, and persisted evidence replace visual guesswork.
- Keep algorithms interchangeable. Detector-specific behavior stays behind common interfaces so better methods can be evaluated without rebuilding the pipeline around them.
- Separate production from experimentation. Smoke tests, calibration, optimizer runs, historical evidence, and production publication have distinct responsibilities.
- Fail explicitly. Missing candidates, invalid geometry, incompatible calibration evidence, corrupted model artifacts, and persistence failures should be visible rather than silently accepted.
- Design for additional collections. Collection-specific source configuration remains separable from the reusable HTH framework.
HTH uses Python's standard unittest framework for repository tests. The GitHub Actions workflows exercise production preprocessing, smoke/regression calibration, detector optimization, Golden Set validation, document review, and report generation.
For repository conventions and contribution requirements, read docs/README.md before making implementation changes.
Physical-page detection and reproducible preprocessing are foundational stages, not the final research product. HTH is intended to carry verified archival evidence forward into:
source preservation
→ page/document normalization
→ transcription
→ translation
→ structured record extraction
→ indexing and entity linkage
→ citation
→ historical analysis
→ reproducible publication
The long-term objective is a transparent research framework in which both people and automated tools can assist with difficult historical records without losing the evidence trail that makes the research trustworthy.