Consolidate Week 7–9 evaluation architecture

[moryev]

Purpose

Consolidate the evaluation architecture that accumulated during Weeks 7–9 into a smaller set of generic, composable interfaces before the package-wide reorganization tracked in #3.

This issue exists because the current scientific experiments are valid, but their implementation contains several parallel evaluation paths that were useful during incremental roadmap development and should not become the permanent architecture.

Current architectural pressure

The codebase now contains multiple partially overlapping concepts, including:

• A/B-specific preparation and result structures;
• A/C/D/E instrument/acquisition evaluation paths;
• A–F robustness/generalization helpers;
• generic prediction tables and summary helpers;
• baseline, ML, and classical-fitting evaluation with different result shapes;
• Week 9 uncertainty outputs with their own semantics;
• upcoming experimental, Bayesian, and persistence workflows that must integrate without creating more parallel stacks.

Goal

Define a coherent evaluation layer that separates:

frozen scientific benchmark definitions
        ↓
generic preparation / prediction / evaluation machinery
        ↓
estimator-specific diagnostics where genuinely necessary
        ↓
standardized result/reporting objects

The result should make it possible to add estimators, representations, test regimes, and uncertainty outputs without repeatedly creating a new experiment-specific evaluation module.

Proposed work

Inventory

• Inventory Week 7–9 evaluation entry points.
• Inventory result dataclasses and report objects.
• Inventory duplicated prediction-table, summary-table, degradation, and validation logic.
• Map which interfaces are generic versus tied to one historical Day/Week experiment.

Prepared data

• Define one generic representation for prepared development/final-test data where practical.
• Preserve frozen A–F benchmark identities and provenance.
• Avoid re-simulating or mutating untouched final-test data during evaluation.

Estimator evaluation

• Define a generic multi-estimator / multi-test evaluation path for baselines and sklearn-style estimators.
• Decide how classical reconvolution plugs into the same reporting layer while retaining fit-specific diagnostics and failure handling.
• Decide how Week 9 interval outputs and uncertainty scores attach to predictions without conflating their semantics.
• Ensure future Bayesian results from Post-roadmap: Bayesian Poisson inference for TCSPC lifetime estimation #4 and experimental measurements from Post-Week-9: experimental TCSPC data ingestion, processing, and evaluation #6 can be represented without synthetic-only assumptions.

Metrics and degradation

• Standardize ordinary prediction metrics (MAE, RMSE, bias, percentile absolute errors, etc.).
• Standardize reference-versus-OOD degradation calculations.
• Standardize interval/calibration metrics separately from heuristic uncertainty-score diagnostics.
• Keep model-mismatch diagnostics distinct from ordinary in-model error metrics where scientifically necessary.

Result objects

• Review objects such as A/B-specific, representation-specific, and instrument/acquisition benchmark result dataclasses.
• Consolidate result classes where this reduces duplication without hiding estimator-specific information.
• Prefer scientific/domain naming over roadmap-specific naming in reusable interfaces.
• Preserve thin compatibility wrappers temporarily where needed for notebooks/tests during migration.

Frozen benchmark configuration

• Separate generic evaluation infrastructure from the fixed benchmark estimator/test definitions used to reproduce roadmap results.
• Ensure the canonical Week 8 A–F suite remains reproducible after consolidation.
• Ensure Week 9 reference results remain reproducible after consolidation.

Scope discipline

Do not combine this issue with the physical package move in #3.

First clarify and consolidate evaluation concepts and interfaces; then let #3 place those clarified responsibilities into subpackages.

Do not opportunistically change scientific benchmark definitions, random seeds, test regimes, or metric semantics merely to simplify code.

Acceptance criteria

• Major Week 7–9 evaluation paths are inventoried.
• Duplicated A/B versus C–E/A–F logic is reduced where the underlying operation is genuinely the same.
• Generic and frozen-benchmark layers are clearly separated.
• Result objects and metric semantics are documented and internally consistent.
• Classical fitting, ML, uncertainty, and future Bayesian/experimental outputs have a clear integration point.
• Existing benchmark tests pass.
• Reference numerical results are unchanged except for intentional, documented fixes.
• Public-facing reusable names no longer need to depend on Day/Week terminology where practical.

Timing

Implement after #2 API stabilization decisions are sufficiently clear and before #3 package restructuring.

#2 API stabilization
    ↓
#11 evaluation consolidation
    ↓
#3 package restructuring