Summary
#1465 added a warning when a .bca-baseline.toml entry has measured past
its recorded value. That warning cannot close the staleness class, by
construction, and this is the follow-up that can.
The residual gap
bca check's warning is emitted from filter_by_baseline, which only sees
violations that reached Baseline::classify. A function whose metric stopped
breaching its threshold entirely produces no Violation at all, so it
never reaches classify and its stale entry is invisible to that path.
So the warning covers offenders still above their limits; it cannot see an
entry whose offender is fully fixed. The looser such an entry, the longer it
survives, and nothing forces a refresh.
Proposal — option 3 from #1465
A scheduled workflow that regenerates and reports drift:
make self-scan-write-baseline-headroom
git diff --exit-code .bca-baseline.toml
A non-empty diff means the checked-in baseline no longer describes the tree.
File an issue (or open a PR with the regenerated file) when it does.
That catches both classes — decreases still above the limit, and entries
whose offender no longer violates at all — because it compares the whole
generated file rather than per-violation values.
Precedent to follow
.github/workflows/mutation-test.yml and benchmark.yml are both quarterly
crons with workflow_dispatch, permissions: {contents: read, issues: write}, and concurrency groups; mutation-test's "File issue on escaped
mutants" step is the template for the issue-filing half. Their cron dates are
deliberately offset two weeks from each other (benchmark.yml says so in a
comment) — pick a third non-colliding date and say why.
Notes
- Use
make self-scan-write-baseline-headroom, never the bare
self-scan-write-baseline: the checked-in file carries tier = "soft", headroom = 0.95 provenance, and the bare target would make the soft-tier
gate re-fire on untouched files.
bca diff-baseline already models a downward move (it has an improved
bucket) — reuse its vocabulary so the two features speak the same language.
Related
Summary
#1465 added a warning when a
.bca-baseline.tomlentry has measured pastits recorded value. That warning cannot close the staleness class, by
construction, and this is the follow-up that can.
The residual gap
bca check's warning is emitted fromfilter_by_baseline, which only seesviolations that reached
Baseline::classify. A function whose metric stoppedbreaching its threshold entirely produces no
Violationat all, so itnever reaches
classifyand its stale entry is invisible to that path.So the warning covers offenders still above their limits; it cannot see an
entry whose offender is fully fixed. The looser such an entry, the longer it
survives, and nothing forces a refresh.
Proposal — option 3 from #1465
A scheduled workflow that regenerates and reports drift:
A non-empty diff means the checked-in baseline no longer describes the tree.
File an issue (or open a PR with the regenerated file) when it does.
That catches both classes — decreases still above the limit, and entries
whose offender no longer violates at all — because it compares the whole
generated file rather than per-violation values.
Precedent to follow
.github/workflows/mutation-test.ymlandbenchmark.ymlare both quarterlycrons with
workflow_dispatch,permissions: {contents: read, issues: write}, andconcurrencygroups; mutation-test's "File issue on escapedmutants" step is the template for the issue-filing half. Their cron dates are
deliberately offset two weeks from each other (
benchmark.ymlsays so in acomment) — pick a third non-colliding date and say why.
Notes
make self-scan-write-baseline-headroom, never the bareself-scan-write-baseline: the checked-in file carriestier = "soft", headroom = 0.95provenance, and the bare target would make the soft-tiergate re-fire on untouched files.
bca diff-baselinealready models a downward move (it has animprovedbucket) — reuse its vocabulary so the two features speak the same language.
Related
AGENTS.mdandrecipes/baselines.mdall state this residual gap explicitly.