docs(evaluation): correct the report-only instructions - #41
Open
joeymussalli wants to merge 1 commit into
Open
Conversation
Step 1 of "Trialling it" says to run in report-only mode, and "Running it here"
tells you how:
SEVERITY_THRESHOLD: high # report-only: omit this
Omitting it does not produce a report-only run. It leaves SEVERITY_THRESHOLD at
its default of `none`, which only disables *this script's* severity gate. The
trustabl binary still exits 1 on any finding at medium or above, and the script
honours that exit independently of either threshold.
So a reader following the documented evaluation procedure gets a failed
pipeline on their very first trial run -- exactly the outcome step 1 exists to
avoid -- and the note in the code fence tells them the one thing that will not
help. The actual switch, `|| true` on the invoking command, is documented in
codepipeline/README.md and codecatalyst/README.md but not in the guide that
tells you to use it.
The Gating table has the same gap from the other direction. It lists four
controls without saying that two of them belong to the binary and two to this
script, which invites the reading that SEVERITY_THRESHOLD sets the bar. It does
not; it adds a second bar. `SEVERITY_THRESHOLD: critical` does not permit high
findings.
- Say plainly that report-only is not the default and cannot be reached by
leaving thresholds unset, with the `|| true` form for both platforms.
- Warn to remove `|| true` at step 4, since it also suppresses the failure
raised when the scan did not complete.
- Add a "Whose behaviour" column to the Gating table and explain that the
thresholds are additional gates layered on the binary's default.
- Fix the misleading code-fence comment and cross-reference Gating.
Documentation only; no code changes.
Author
|
Closing in favor of #36 (Paramjeet) on the report-only / omitted SEVERITY_THRESHOLD docs. |
Author
|
Reopened — closed earlier by mistake. Leaving this up per author request. |
6 tasks
joeymussalli
force-pushed
the
docs/evaluation-report-only
branch
from
August 24, 2026 20:05
8b6929a to
f1f171c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 1 of "Trialling it" says to run in report-only mode, and "Running it here" tells you how:
Omitting it does not produce a report-only run. It leaves SEVERITY_THRESHOLD at its default of
none, which only disables this script's severity gate. The trustabl binary still exits 1 on any finding at medium or above, and the script honours that exit independently of either threshold.So a reader following the documented evaluation procedure gets a failed pipeline on their very first trial run -- exactly the outcome step 1 exists to avoid -- and the note in the code fence tells them the one thing that will not help. The actual switch,
|| trueon the invoking command, is documented in codepipeline/README.md and codecatalyst/README.md but not in the guide that tells you to use it.The Gating table has the same gap from the other direction. It lists four controls without saying that two of them belong to the binary and two to this script, which invites the reading that SEVERITY_THRESHOLD sets the bar. It does not; it adds a second bar.
SEVERITY_THRESHOLD: criticaldoes not permit high findings.|| trueform for both platforms.|| trueat step 4, since it also suppresses the failure raised when the scan did not complete.Documentation only; no code changes.