Skip to content

fix(report): make the projected readiness panel trustworthy - #32

Open
joeymussalli wants to merge 1 commit into
trustabl:mainfrom
joeymussalli:fix/projection-panel-correctness
Open

fix(report): make the projected readiness panel trustworthy#32
joeymussalli wants to merge 1 commit into
trustabl:mainfrom
joeymussalli:fix/projection-panel-correctness

Conversation

@joeymussalli

Copy link
Copy Markdown

The projection panel is what a user triages from, and two defects made it lie. First, one malformed finding voided the whole panel: removed() evaluated sw($f.severity) * $f.confidence, and jq cannot multiply a number by null, so a single finding with no confidence (or "confidence": null, or no tool_name) aborted the entire program with exit 5. stderr went to /dev/null, read got nothing, and the fallbacks rendered the failure as a report: P_CRIT/P_CH/P_CHM/P_CHML fell back to $SCORE but P_ALL fell back to a hardcoded 100, so total failure printed as "Readiness 59 -> 100 (+41)" with every point of that headroom attributed to "Fix +info" — findings docs/EVALUATION.md calls "not defects" that "never fail a build on their own". The user worked exactly the wrong queue and nothing in the log said the projection had failed.

Second, the ladder mixed two different scores. projected() derives its baseline as min over .readiness[], but the deltas subtracted $SCORE, which comes from .overall_score — weighted across the surfaces found, explicitly not a min. With 49 clean tools and one tool at weighted_severity 1.5 the panel printed "Readiness 95 -> 64 (+-31)" and "Fix critical 95 -> 50 (+-45)": fixing every critical finding was reported as a 45-point drop, and "+-45" is malformed output on top of being wrong. trustabl-summary.md — the artifact people paste into a PR — carried "+-31" without even the console's "estimate, not a re-scan" disclaimer.

Changes:

  • Default a missing/non-numeric confidence to 1.0 (full weight, matching the severity weight's own default) and a missing/non-string tool_name to a sentinel that no readiness row can match, so an unattributable finding is simply never credited. Both defaults are type-guarded, so they cannot alter a projection computed from well-formed data.
  • Guard .weighted_severity the same way (non-numeric -> 0) so a malformed readiness row cannot abort the program either.
  • Emit the projection's own baseline, p100(projected([])), as a sixth field, and anchor the whole panel to it — header, "Projected all" gauge, and every rung of the ladder now share one baseline. Since resolving findings can only lower weighted severity, the ladder is non-decreasing by construction; a clamp enforces it regardless.
  • Return empty when .readiness is absent or empty rather than a vacuous 100 for every rung, and keep jq's stderr: on any failure the log gets a WARNING naming the cause, and every rung falls back to $SCORE — no headroom known, never a fabricated perfect score.
  • Format all deltas with printf '%+d', so "+-45" can no longer be printed anywhere.
  • Give trustabl-summary.md the same estimate/not-a-re-scan disclaimer the console carries, and, when the projection baseline differs from the weighted readiness score, one line in both outputs explaining that the baseline is the lowest-scoring surface while readiness is weighted across all of them.

Verified with bash -n and by driving the real report and summary blocks
against fixtures. A well-formed repo is unchanged: 58 -> 88 -> 90 -> 95 ->
100, +42, byte for byte. The same repo with a finding missing confidence, with "confidence": null, or with weird types now renders numbers instead of aborting; missing tool_name renders a conservative +30 instead of a fabricated "+42, all of it in info". The 49-clean/1-bad case now reads 50 -> 64 (+14) with a monotone ladder instead of 95 -> 64 (+-31). Empty .findings still reports +0; absent .readiness reports +0 with a WARNING naming the reason, as does an unparseable trustabl.json. No "+-" appears in any console or markdown output across the fixture set.

The report header now reads "Projected 58 -> 88" rather than "Readiness ...". Anchoring the header to the projection baseline is what makes its three numbers share a baseline, but leaving the label as "Readiness" put two different numbers under that one word -- the header's, and the "Readiness now" gauge's, which still shows the weighted score. Renaming the header line is the smaller half of the fix and removes the ambiguity.

The projection panel is what a user triages from, and two defects made it
lie. First, one malformed finding voided the whole panel: removed()
evaluated sw($f.severity) * $f.confidence, and jq cannot multiply a number
by null, so a single finding with no `confidence` (or `"confidence": null`,
or no `tool_name`) aborted the entire program with exit 5. stderr went to
/dev/null, `read` got nothing, and the fallbacks rendered the failure as a
report: P_CRIT/P_CH/P_CHM/P_CHML fell back to $SCORE but P_ALL fell back to
a hardcoded 100, so total failure printed as "Readiness 59 -> 100 (+41)"
with every point of that headroom attributed to "Fix +info" — findings
docs/EVALUATION.md calls "not defects" that "never fail a build on their
own". The user worked exactly the wrong queue and nothing in the log said
the projection had failed.

Second, the ladder mixed two different scores. projected() derives its
baseline as min over .readiness[], but the deltas subtracted $SCORE, which
comes from .overall_score — weighted across the surfaces found, explicitly
not a min. With 49 clean tools and one tool at weighted_severity 1.5 the
panel printed "Readiness 95 -> 64 (+-31)" and "Fix critical 95 -> 50
(+-45)": fixing every critical finding was reported as a 45-point drop, and
"+-45" is malformed output on top of being wrong. trustabl-summary.md — the
artifact people paste into a PR — carried "**+-31**" without even the
console's "estimate, not a re-scan" disclaimer.

Changes:

- Default a missing/non-numeric `confidence` to 1.0 (full weight, matching
  the severity weight's own default) and a missing/non-string `tool_name`
  to a sentinel that no readiness row can match, so an unattributable
  finding is simply never credited. Both defaults are type-guarded, so
  they cannot alter a projection computed from well-formed data.
- Guard `.weighted_severity` the same way (non-numeric -> 0) so a malformed
  readiness row cannot abort the program either.
- Emit the projection's own baseline, p100(projected([])), as a sixth
  field, and anchor the whole panel to it — header, "Projected all" gauge,
  and every rung of the ladder now share one baseline. Since resolving
  findings can only lower weighted severity, the ladder is non-decreasing
  by construction; a clamp enforces it regardless.
- Return `empty` when .readiness is absent or empty rather than a vacuous
  100 for every rung, and keep jq's stderr: on any failure the log gets a
  WARNING naming the cause, and every rung falls back to $SCORE — no
  headroom known, never a fabricated perfect score.
- Format all deltas with `printf '%+d'`, so "+-45" can no longer be
  printed anywhere.
- Give trustabl-summary.md the same estimate/not-a-re-scan disclaimer the
  console carries, and, when the projection baseline differs from the
  weighted readiness score, one line in both outputs explaining that the
  baseline is the lowest-scoring surface while readiness is weighted
  across all of them.

Verified with `bash -n` and by driving the real report and summary blocks
against fixtures. A well-formed repo is unchanged: 58 -> 88 -> 90 -> 95 ->
100, +42, byte for byte. The same repo with a finding missing `confidence`,
with `"confidence": null`, or with weird types now renders numbers instead
of aborting; missing `tool_name` renders a conservative +30 instead of a
fabricated "+42, all of it in info". The 49-clean/1-bad case now reads
50 -> 64 (+14) with a monotone ladder instead of 95 -> 64 (+-31). Empty
.findings still reports +0; absent .readiness reports +0 with a WARNING
naming the reason, as does an unparseable trustabl.json. No "+-" appears in
any console or markdown output across the fixture set.

The report header now reads "Projected  58 -> 88" rather than "Readiness ...".
Anchoring the header to the projection baseline is what makes its three numbers
share a baseline, but leaving the label as "Readiness" put two different numbers
under that one word -- the header's, and the "Readiness now" gauge's, which
still shows the weighted score. Renaming the header line is the smaller half of
the fix and removes the ambiguity.
@joeymussalli

Copy link
Copy Markdown
Author

Closing in favor of #4 (bradAGI), which rewrites projected readiness against the engine surfaces — this would conflict heavily.

@joeymussalli

Copy link
Copy Markdown
Author

Reopened — closed earlier by mistake. Leaving this up per author request.

@joeymussalli
joeymussalli force-pushed the fix/projection-panel-correctness branch from 01531d7 to 5279eeb Compare August 24, 2026 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant