From f1f171c1ea2f57cea4dfc6f936d7c319ed1dc723 Mon Sep 17 00:00:00 2001 From: Joey Mussalli Date: Mon, 24 Aug 2026 15:09:38 -0400 Subject: [PATCH] docs(evaluation): correct the report-only instructions Step 1 of "Trialling it" says to run in report-only mode, and "Running it here" tells you how: SEVERITY_THRESHOLD: high # report-only: omit this Omitting it does not produce a report-only run. It leaves SEVERITY_THRESHOLD at its default of `none`, which only disables *this script's* severity gate. The trustabl binary still exits 1 on any finding at medium or above, and the script honours that exit independently of either threshold. So a reader following the documented evaluation procedure gets a failed pipeline on their very first trial run -- exactly the outcome step 1 exists to avoid -- and the note in the code fence tells them the one thing that will not help. The actual switch, `|| true` on the invoking command, is documented in codepipeline/README.md and codecatalyst/README.md but not in the guide that tells you to use it. The Gating table has the same gap from the other direction. It lists four controls without saying that two of them belong to the binary and two to this script, which invites the reading that SEVERITY_THRESHOLD sets the bar. It does not; it adds a second bar. `SEVERITY_THRESHOLD: critical` does not permit high findings. - Say plainly that report-only is not the default and cannot be reached by leaving thresholds unset, with the `|| true` form for both platforms. - Warn to remove `|| true` at step 4, since it also suppresses the failure raised when the scan did not complete. - Add a "Whose behaviour" column to the Gating table and explain that the thresholds are additional gates layered on the binary's default. - Fix the misleading code-fence comment and cross-reference Gating. Documentation only; no code changes. --- docs/EVALUATION.md | 38 +++++++++++++++++++++++++++++++------- 1 file changed, 31 insertions(+), 7 deletions(-) diff --git a/docs/EVALUATION.md b/docs/EVALUATION.md index 4835905..6e53fc9 100644 --- a/docs/EVALUATION.md +++ b/docs/EVALUATION.md @@ -28,6 +28,18 @@ signed channel, so a scan picks up new detections without upgrading the binary. well. You are checking whether the findings are real, not whether the build passes. +> Report-only is **not** the default, and it is not something you get by leaving +> the thresholds unset. `trustabl` itself exits non-zero on any finding at medium +> or above, and the scanner script honours that regardless of +> `SEVERITY_THRESHOLD`. To get a genuinely advisory run, append `|| true` to the +> command that invokes the script: +> +> - CodeBuild: `- bash "$CODEBUILD_SRC_DIR/scan/trustabl-scan.sh" || true` +> - CodeCatalyst: `- Run: bash scan/trustabl-scan.sh || true` +> +> Remember to take it back off at step 4 — with `|| true` in place nothing can +> ever fail the build, including a scanner that did not run. + **2. Read the inventory before the findings.** If the tool and agent counts look wrong, the scan is pointed at the wrong path or your SDK is not being detected. A score computed over the wrong inventory is meaningless. @@ -102,12 +114,21 @@ Projections come from the same formula, not a re-scan. Any one of these can fail the run. -| Control | Effect | -|---|---| -| default | Fails on any finding at medium or above | -| severity threshold | Fails when the worst finding reaches the level you set | -| risk score threshold | Fails when risk reaches a number you set | -| strict | Lowers the bar to any finding of low or above | +| Control | Effect | Whose behaviour | +|---|---|---| +| default | Fails on any finding at medium or above | the `trustabl` binary | +| severity threshold | Fails when the worst finding reaches the level you set | this script | +| risk score threshold | Fails when risk reaches a number you set | this script | +| strict | Lowers the bar to any finding of low or above | the `trustabl` binary | + +The distinction in the last column matters. The two thresholds are **additional** +gates layered on top; neither can relax the binary's own medium-and-above +default. Setting `SEVERITY_THRESHOLD: critical` does not permit high findings — +it adds a second condition to a build that the binary already failed. + +Only `|| true` on the invoking command turns gating off entirely, and it turns +off *everything*, including the failure the script raises when the scan did not +complete. A common progression is report-only, then `high`, then `medium` once the backlog is clear. @@ -136,9 +157,12 @@ Vendor `scan/` plus the config for your platform into your repo, then set variables on the CodeBuild project or CodeCatalyst workflow: ```yaml -SEVERITY_THRESHOLD: high # report-only: omit this +SEVERITY_THRESHOLD: high # an ADDITIONAL gate, not a relaxation ``` +Omitting it does not give you report-only — see [Gating](#gating). For an +advisory run, append `|| true` to the command that invokes the script. + Variables are **UPPER_SNAKE**: `SEVERITY_THRESHOLD`, `RISK_SCORE_THRESHOLD`, `STRICT`, `DETECTORS`, `VERSION`.