Context
The effect-size section reports the raw effect shift (peak.effect.mean - baseline.effect.mean) and a ± spread, but a raw shift is not comparable across concepts or models whose scorers live on different scales. A standardized effect size — the shift divided by the pooled seed-to-seed standard deviation (a Cohen's-d-style ratio) — makes "how big, in noise units" legible and comparable.
Proposal
Add a pure function that computes a standardized effect size for a SweepPoint against the baseline, reusing the existing per-seed aggregation (_combine already yields mean and population std per point) rather than recomputing spread. Surface it in the effect-size summary lines.
Acceptance criteria
- Add
standardized_effect(point: SweepPoint, baseline: SweepPoint) -> float in src/steerbench/report.py: (point.effect.mean - baseline.effect.mean) divided by the pooled std of the two points' effect.spread; return 0.0 (not a division error) when the pooled std is 0.0.
- Reuse the mean/std already carried on
MetricStat; do not re-derive spread from raw rows.
- Add it to
_effect_summary_lines so the report card names the standardized effect at the sweet spot alongside the raw shift.
- Because the input
spread is a seed-to-seed std, the metric is only meaningful at >= 3 seeds; state this in the docstring and have the summary line report the seed count (n_seeds) next to the standardized value so mean +/- std and seed count travel together. Note seeds/hardware/wall-clock belong in the run provenance.
- CPU-only unit tests in
tests/test_report.py on a hand-built curve: correct ratio for a known case, and 0.0 for zero pooled std. No model download.
ruff check ., ruff format --check ., mypy src, pytest -q all pass.
Context
The effect-size section reports the raw effect shift (
peak.effect.mean - baseline.effect.mean) and a±spread, but a raw shift is not comparable across concepts or models whose scorers live on different scales. A standardized effect size — the shift divided by the pooled seed-to-seed standard deviation (a Cohen's-d-style ratio) — makes "how big, in noise units" legible and comparable.Proposal
Add a pure function that computes a standardized effect size for a
SweepPointagainst the baseline, reusing the existing per-seed aggregation (_combinealready yields mean and population std per point) rather than recomputing spread. Surface it in the effect-size summary lines.Acceptance criteria
standardized_effect(point: SweepPoint, baseline: SweepPoint) -> floatinsrc/steerbench/report.py:(point.effect.mean - baseline.effect.mean)divided by the pooled std of the two points'effect.spread; return0.0(not a division error) when the pooled std is0.0.MetricStat; do not re-derive spread from raw rows._effect_summary_linesso the report card names the standardized effect at the sweet spot alongside the raw shift.spreadis a seed-to-seed std, the metric is only meaningful at >= 3 seeds; state this in the docstring and have the summary line report the seed count (n_seeds) next to the standardized value so mean +/- std and seed count travel together. Note seeds/hardware/wall-clock belong in the run provenance.tests/test_report.pyon a hand-built curve: correct ratio for a known case, and0.0for zero pooled std. No model download.ruff check .,ruff format --check .,mypy src,pytest -qall pass.