fix(report): make the projected readiness panel trustworthy - #32
Open
joeymussalli wants to merge 1 commit into
Open
fix(report): make the projected readiness panel trustworthy#32joeymussalli wants to merge 1 commit into
joeymussalli wants to merge 1 commit into
Conversation
The projection panel is what a user triages from, and two defects made it lie. First, one malformed finding voided the whole panel: removed() evaluated sw($f.severity) * $f.confidence, and jq cannot multiply a number by null, so a single finding with no `confidence` (or `"confidence": null`, or no `tool_name`) aborted the entire program with exit 5. stderr went to /dev/null, `read` got nothing, and the fallbacks rendered the failure as a report: P_CRIT/P_CH/P_CHM/P_CHML fell back to $SCORE but P_ALL fell back to a hardcoded 100, so total failure printed as "Readiness 59 -> 100 (+41)" with every point of that headroom attributed to "Fix +info" — findings docs/EVALUATION.md calls "not defects" that "never fail a build on their own". The user worked exactly the wrong queue and nothing in the log said the projection had failed. Second, the ladder mixed two different scores. projected() derives its baseline as min over .readiness[], but the deltas subtracted $SCORE, which comes from .overall_score — weighted across the surfaces found, explicitly not a min. With 49 clean tools and one tool at weighted_severity 1.5 the panel printed "Readiness 95 -> 64 (+-31)" and "Fix critical 95 -> 50 (+-45)": fixing every critical finding was reported as a 45-point drop, and "+-45" is malformed output on top of being wrong. trustabl-summary.md — the artifact people paste into a PR — carried "**+-31**" without even the console's "estimate, not a re-scan" disclaimer. Changes: - Default a missing/non-numeric `confidence` to 1.0 (full weight, matching the severity weight's own default) and a missing/non-string `tool_name` to a sentinel that no readiness row can match, so an unattributable finding is simply never credited. Both defaults are type-guarded, so they cannot alter a projection computed from well-formed data. - Guard `.weighted_severity` the same way (non-numeric -> 0) so a malformed readiness row cannot abort the program either. - Emit the projection's own baseline, p100(projected([])), as a sixth field, and anchor the whole panel to it — header, "Projected all" gauge, and every rung of the ladder now share one baseline. Since resolving findings can only lower weighted severity, the ladder is non-decreasing by construction; a clamp enforces it regardless. - Return `empty` when .readiness is absent or empty rather than a vacuous 100 for every rung, and keep jq's stderr: on any failure the log gets a WARNING naming the cause, and every rung falls back to $SCORE — no headroom known, never a fabricated perfect score. - Format all deltas with `printf '%+d'`, so "+-45" can no longer be printed anywhere. - Give trustabl-summary.md the same estimate/not-a-re-scan disclaimer the console carries, and, when the projection baseline differs from the weighted readiness score, one line in both outputs explaining that the baseline is the lowest-scoring surface while readiness is weighted across all of them. Verified with `bash -n` and by driving the real report and summary blocks against fixtures. A well-formed repo is unchanged: 58 -> 88 -> 90 -> 95 -> 100, +42, byte for byte. The same repo with a finding missing `confidence`, with `"confidence": null`, or with weird types now renders numbers instead of aborting; missing `tool_name` renders a conservative +30 instead of a fabricated "+42, all of it in info". The 49-clean/1-bad case now reads 50 -> 64 (+14) with a monotone ladder instead of 95 -> 64 (+-31). Empty .findings still reports +0; absent .readiness reports +0 with a WARNING naming the reason, as does an unparseable trustabl.json. No "+-" appears in any console or markdown output across the fixture set. The report header now reads "Projected 58 -> 88" rather than "Readiness ...". Anchoring the header to the projection baseline is what makes its three numbers share a baseline, but leaving the label as "Readiness" put two different numbers under that one word -- the header's, and the "Readiness now" gauge's, which still shows the weighted score. Renaming the header line is the smaller half of the fix and removes the ambiguity.
Author
|
Closing in favor of #4 (bradAGI), which rewrites projected readiness against the engine surfaces — this would conflict heavily. |
Author
|
Reopened — closed earlier by mistake. Leaving this up per author request. |
joeymussalli
force-pushed
the
fix/projection-panel-correctness
branch
from
August 24, 2026 20:05
01531d7 to
5279eeb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The projection panel is what a user triages from, and two defects made it lie. First, one malformed finding voided the whole panel: removed() evaluated sw($f.severity) * $f.confidence, and jq cannot multiply a number by null, so a single finding with no
confidence(or"confidence": null, or notool_name) aborted the entire program with exit 5. stderr went to /dev/null,readgot nothing, and the fallbacks rendered the failure as a report: P_CRIT/P_CH/P_CHM/P_CHML fell back to $SCORE but P_ALL fell back to a hardcoded 100, so total failure printed as "Readiness 59 -> 100 (+41)" with every point of that headroom attributed to "Fix +info" — findings docs/EVALUATION.md calls "not defects" that "never fail a build on their own". The user worked exactly the wrong queue and nothing in the log said the projection had failed.Second, the ladder mixed two different scores. projected() derives its baseline as min over .readiness[], but the deltas subtracted $SCORE, which comes from .overall_score — weighted across the surfaces found, explicitly not a min. With 49 clean tools and one tool at weighted_severity 1.5 the panel printed "Readiness 95 -> 64 (+-31)" and "Fix critical 95 -> 50 (+-45)": fixing every critical finding was reported as a 45-point drop, and "+-45" is malformed output on top of being wrong. trustabl-summary.md — the artifact people paste into a PR — carried "+-31" without even the console's "estimate, not a re-scan" disclaimer.
Changes:
confidenceto 1.0 (full weight, matching the severity weight's own default) and a missing/non-stringtool_nameto a sentinel that no readiness row can match, so an unattributable finding is simply never credited. Both defaults are type-guarded, so they cannot alter a projection computed from well-formed data..weighted_severitythe same way (non-numeric -> 0) so a malformed readiness row cannot abort the program either.emptywhen .readiness is absent or empty rather than a vacuous 100 for every rung, and keep jq's stderr: on any failure the log gets a WARNING naming the cause, and every rung falls back to $SCORE — no headroom known, never a fabricated perfect score.printf '%+d', so "+-45" can no longer be printed anywhere.Verified with
bash -nand by driving the real report and summary blocksagainst fixtures. A well-formed repo is unchanged: 58 -> 88 -> 90 -> 95 ->
100, +42, byte for byte. The same repo with a finding missing
confidence, with"confidence": null, or with weird types now renders numbers instead of aborting; missingtool_namerenders a conservative +30 instead of a fabricated "+42, all of it in info". The 49-clean/1-bad case now reads 50 -> 64 (+14) with a monotone ladder instead of 95 -> 64 (+-31). Empty .findings still reports +0; absent .readiness reports +0 with a WARNING naming the reason, as does an unparseable trustabl.json. No "+-" appears in any console or markdown output across the fixture set.The report header now reads "Projected 58 -> 88" rather than "Readiness ...". Anchoring the header to the projection baseline is what makes its three numbers share a baseline, but leaving the label as "Readiness" put two different numbers under that one word -- the header's, and the "Readiness now" gauge's, which still shows the weighted score. Renaming the header line is the smaller half of the fix and removes the ambiguity.