feat: report-only trials and Security Hub ASFF export

.gitignore

@@ -3,6 +3,7 @@ trustabl.json
 trustabl.sarif
 trustabl-summary.md
 trustabl.env
+trustabl.asff.json
 .trustabl-bin/
 trustabl/
 # Claude tooling

CHANGELOG.md

@@ -4,6 +4,18 @@ All notable changes to the Trustabl AWS plugin are documented here.
 Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/);
 versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Added
+
+- `REPORT_ONLY=true` for evaluation-guide trials: publishes artifacts without
+  gating, but still fails on scanner errors (exit 2). Replaces the documented
+  `|| true` workaround, which swallowed dead-scanner failures.
+- ASFF export (`scan/to-asff.sh`, `trustabl.asff.json`) and optional
+  `SECURITY_HUB=true` import into AWS Security Hub via
+  `securityhub:BatchImportFindings`.
+- Offline tests for the ASFF converter (`test/test-asff.sh`).
+
 
 ## [0.1.0] — 2026-06-17
 

README.md

@@ -28,12 +28,15 @@ One scanner (`scan/trustabl-scan.sh`), two integrations:
 | `RISK_SCORE_THRESHOLD` | `0` | Fail when risk (100 − readiness) >= N. `0` disables. |
 | `SEVERITY_THRESHOLD` | `none` | Fail when any finding >= severity (`none/low/medium/high/critical`). |
 | `GITHUB_TOKEN` | _(none)_ | Optional — avoids GitHub's 60 req/hr anonymous rate limit. |
+| `REPORT_ONLY` | `false` | Scan and publish artifacts without gating. Scanner errors (exit 2) still fail the build. |
+| `SECURITY_HUB` | `false` | Import ASFF findings into AWS Security Hub (`securityhub:BatchImportFindings`). |
 
 ## Outputs
 
-`trustabl.json`, `trustabl.sarif`, `trustabl-summary.md`, and `trustabl.env`
+`trustabl.json`, `trustabl.sarif`, `trustabl-summary.md`, `trustabl.env`
 (`TRUSTABL_READINESS_SCORE`, `TRUSTABL_RISK_SCORE`, `TRUSTABL_MAX_SEVERITY`,
-`TRUSTABL_FINDINGS_COUNT`, `TRUSTABL_EXIT_CODE`).
+`TRUSTABL_FINDINGS_COUNT`, `TRUSTABL_EXIT_CODE`), and `trustabl.asff.json` (ASFF
+for Security Hub).
 
 ## A note on "AWS Marketplace"
 

codecatalyst/README.md

@@ -26,6 +26,7 @@ Copy into your repo, keeping the layout, then commit + push:
 ```
 your-repo/
 ├── scan/trustabl-scan.sh                    # the scanner
+├── scan/to-asff.sh                          # JSON → Security Hub ASFF
 └── .codecatalyst/workflows/trustabl.yaml    # the workflow (from codecatalyst/workflows/)
 ```
 
@@ -45,9 +46,9 @@ The workflow triggers on push to `main`. CodeCatalyst → your project →
 `trustabl.sarif` / `trustabl-summary.md`, surfaces findings in the **Reports**
 tab (SARIF), and **fails the run on any medium-or-higher finding**.
 
-> **Report-only (don't block)?** trustabl fails on medium+ by default. To make
-> it advisory, change the workflow `Run:` line to:
-> `- Run: bash scan/trustabl-scan.sh || true`
+> **Report-only (don't block)?** Set `REPORT_ONLY=true` in the workflow
+> Variables. That publishes artifacts and still fails on scanner errors
+> (exit 2). Do not use `|| true` — that also swallows a dead scanner.
 
 ### CLI?
 CodeCatalyst workflows are **file-driven** — there's no separate CLI to create

codecatalyst/workflows/trustabl.yaml

@@ -27,6 +27,10 @@ Actions:
           Value: "none"
         - Name: RISK_SCORE_THRESHOLD
           Value: "0"
+        - Name: REPORT_ONLY
+          Value: "false"
+        - Name: SECURITY_HUB
+          Value: "false"
     Configuration:
       Steps:
         - Run: bash scan/trustabl-scan.sh
@@ -37,6 +41,8 @@ Actions:
             - trustabl.json
             - trustabl.sarif
             - trustabl-summary.md
+            - trustabl.env
+            - trustabl.asff.json
       # Surfaces findings in the CodeCatalyst Reports tab.
       # NOTE: verify the SARIF report Format enum against current CodeCatalyst
       # docs before relying on it (the schema evolves).

codepipeline/README.md

@@ -11,11 +11,14 @@
 
 A gate failure exits non-zero -> the CodeBuild action fails -> the pipeline
 stage fails. Artifacts (`trustabl.json`, `trustabl.sarif`,
-`trustabl-summary.md`) are emitted to the artifact bucket.
+`trustabl-summary.md`, `trustabl.env`, `trustabl.asff.json`) are emitted to the
+artifact bucket.
 
-**Optional — Security Hub:** convert findings to ASFF and
-`aws securityhub batch-import-findings` to surface them in Security Hub (needs
-Security Hub enabled + IAM `securityhub:BatchImportFindings`). Not in v0.1.0.
+**Optional — Security Hub:** set `SECURITY_HUB=true` on the CodeBuild project.
+The scanner writes `trustabl.asff.json` and then calls
+`aws securityhub batch-import-findings`. Needs Security Hub enabled in the
+region and IAM `securityhub:BatchImportFindings` plus `sts:GetCallerIdentity`
+on the CodeBuild role.
 
 ## Quickstart — from zero (console)
 
@@ -28,6 +31,7 @@ Copy these two into your repo, keeping the layout, then commit + push:
 ```
 your-repo/
 ├── scan/trustabl-scan.sh        # the scanner
+├── scan/to-asff.sh              # JSON → Security Hub ASFF
 └── codepipeline/buildspec.yml   # tells CodeBuild to run it
 ```
 
@@ -37,7 +41,8 @@ Console → **CodeBuild → Create build project**:
 - **Buildspec:** "Use a buildspec file" → path `codepipeline/buildspec.yml`
 - **Service role:** let CodeBuild create one (it needs CloudWatch Logs)
 - *(optional)* **Env vars:** `SEVERITY_THRESHOLD`, `RISK_SCORE_THRESHOLD`,
-  `VERSION` (pin a tag), `GITHUB_TOKEN` (dodges GitHub's 60-req/hr anon limit)
+  `VERSION` (pin a tag), `GITHUB_TOKEN` (dodges GitHub's 60-req/hr anon limit),
+  `REPORT_ONLY=true` (trial without gating), `SECURITY_HUB=true` (import ASFF)
 - **`GITHUB_TOKEN` is a credential** — add it with type **Secrets Manager** or
   **Parameter Store**, not as a plaintext environment variable. Plaintext env
   vars are readable by anyone with `codebuild:BatchGetProjects`.
@@ -53,9 +58,10 @@ prints the readiness report, uploads `trustabl.json` / `trustabl.sarif` /
 `trustabl-summary.md` as artifacts, and **fails the stage if any finding is
 medium-or-higher** — so the pipeline stops on unsafe agent code.
 
-> **Want report-only (don't block the pipeline)?** trustabl fails on medium+ by
-> default. To make it advisory, change the build command in your buildspec to:
-> `- bash "$CODEBUILD_SRC_DIR/scan/trustabl-scan.sh" || true`
+> **Want report-only (don't block the pipeline)?** Set `REPORT_ONLY=true` on
+> the CodeBuild project. That publishes artifacts and still fails the build on
+> scanner errors (exit 2). Do not use `|| true` — that also swallows a dead
+> scanner, which [docs/EVALUATION.md](../docs/EVALUATION.md) says not to trust.
 
 ### CLI (alternative to steps 2–3)
 After vendoring (step 1), replace `<you>/<repo>` and `<acct>`:

codepipeline/buildspec.yml

@@ -13,7 +13,9 @@ version: 0.2
 #   TARGET (default "."), VERSION (default "latest"), DETECTORS, STRICT,
 #   RULES_REF, RULES_REPO, SARIF_FILE, JSON_FILE,
 #   RISK_SCORE_THRESHOLD (default 0), SEVERITY_THRESHOLD (default none),
-#   GITHUB_TOKEN (optional — dodges GitHub's 60 req/hr anon rate limit).
+#   GITHUB_TOKEN (optional — dodges GitHub's 60 req/hr anon rate limit),
+#   REPORT_ONLY (default false — scan without gating; scanner errors still fail),
+#   SECURITY_HUB (default false — import findings into AWS Security Hub).
 
 env:
   variables:
@@ -37,3 +39,5 @@ artifacts:
     - trustabl.json
     - trustabl.sarif
     - trustabl-summary.md
+    - trustabl.env
+    - trustabl.asff.json

docs/EVALUATION.md

@@ -137,18 +137,22 @@ Vendor `scan/` plus the config for your platform into your repo, then set
 variables on the CodeBuild project or CodeCatalyst workflow:
 
 ```yaml
-SEVERITY_THRESHOLD: high             # report-only: omit this
+REPORT_ONLY: "true"                  # step 1: artifacts only; scanner errors still fail
+SEVERITY_THRESHOLD: high             # step 4: turn gating on
 ```
 
 Variables are **UPPER_SNAKE**: `SEVERITY_THRESHOLD`, `RISK_SCORE_THRESHOLD`,
-`STRICT`, `DETECTORS`, `VERSION`.
+`STRICT`, `DETECTORS`, `VERSION`, `REPORT_ONLY`, `SECURITY_HUB`.
+
+Do not use `|| true` around the scan to get report-only behaviour. That also
+swallows exit 2 (scanner/I/O failure), and those results must not be trusted.
 
 ## Where the results appear
 
 | Surface | What you get |
 |---|---|
 | **Build log** | The readiness panel |
-| **Artifacts** | `trustabl.json`, `trustabl.sarif`, `trustabl-summary.md`, `trustabl.env` |
+| **Artifacts** | `trustabl.json`, `trustabl.sarif`, `trustabl-summary.md`, `trustabl.env`, `trustabl.asff.json` |
 | **CodeCatalyst Reports tab** | SARIF report, when configured in the workflow |
 
 A gate failure exits non-zero, which fails the CodeBuild action and therefore the
@@ -157,5 +161,6 @@ pipeline stage.
 `trustabl.env` exposes `TRUSTABL_EXIT_CODE`, `TRUSTABL_READINESS_SCORE`,
 `TRUSTABL_RISK_SCORE` and `TRUSTABL_MAX_SEVERITY` for downstream steps.
 
-For a trial, run the CodeBuild project on its own before wiring it into a
-pipeline. It exercises the same scan without gating anything real.
+For a trial, set `REPORT_ONLY=true` and run the CodeBuild project on its own
+before wiring it into a pipeline. That exercises the same scan without gating
+a real stage, while still failing if the scanner itself errors.

scan/to-asff.sh

@@ -0,0 +1,135 @@
+#!/usr/bin/env bash
+# Convert a Trustabl JSON ScanResult into AWS Security Hub ASFF (2018-10-08)
+# and optionally import it with `aws securityhub batch-import-findings`.
+#
+# Usage:
+#   to-asff.sh [json] [out] [resource-id]
+#   to-asff.sh --import [json] [out] [resource-id]
+#
+# Defaults: JSON_FILE/trustabl.json, trustabl.asff.json, TARGET/repo.
+#
+# Does not require AWS credentials unless --import is set. Placeholders are
+# used for ProductArn / AwsAccountId and rewritten at import time from
+# `aws sts get-caller-identity` + AWS_REGION.
+set -euo pipefail
+
+IMPORT=false
+if [ "${1:-}" = "--import" ]; then IMPORT=true; shift; fi
+
+JSON="${1:-${JSON_FILE:-trustabl.json}}"
+OUT="${2:-trustabl.asff.json}"
+RESOURCE="${3:-${TARGET:-repository}}"
+
+if [ ! -f "$JSON" ]; then
+  echo "to-asff: JSON result not found: $JSON" >&2
+  exit 2
+fi
+if ! command -v jq >/dev/null 2>&1; then
+  echo "to-asff: jq is required" >&2
+  exit 2
+fi
+
+REGION="${AWS_REGION:-${AWS_DEFAULT_REGION:-us-east-1}}"
+ACCOUNT="${AWS_ACCOUNT_ID:-000000000000}"
+NOW=$(date -u +%Y-%m-%dT%H:%M:%S.000Z)
+GENERATOR="trustabl"
+
+# Security Hub BatchImportFindings accepts a JSON array of findings.
+jq --arg region "$REGION" \
+   --arg account "$ACCOUNT" \
+   --arg now "$NOW" \
+   --arg gen "$GENERATOR" \
+   --arg resource "$RESOURCE" \
+   --arg product "arn:aws:securityhub:${REGION}:${ACCOUNT}:product/${ACCOUNT}/default" \
+   '
+   def sev_label($s):
+     if $s == "critical" then "CRITICAL"
+     elif $s == "high" then "HIGH"
+     elif $s == "medium" then "MEDIUM"
+     elif $s == "low" then "LOW"
+     else "INFORMATIONAL" end;
+   def clip($s; $n):
+     ($s | tostring) as $t
+     | if ($t | length) <= $n then $t else $t[0:$n-1] + "…" end;
+   . as $root
+   | ($root.findings // []) as $findings
+   | [ range(0; $findings | length) as $i
+       | $findings[$i]
+       | . as $f
+       | ($f.severity // $f.level // "info") as $sev
+       | ($f.rule_id // $f.id // $f.check_id // $f.title // "finding") as $fid
+       | ($f.title // $f.rule_id // $f.message // "Trustabl finding") as $title
+       | ($f.explanation // $f.message // $f.description // $f.title // "No description provided") as $desc
+       | ($f.file_path // $f.file // $f.path // $f.location.file // $f.location.uri // $resource) as $file
+       | ("Software and Configuration Checks/Code Analysis/Agent Reliability") as $type
+       | {
+           SchemaVersion: "2018-10-08",
+           Id: ("trustabl/" + ($i | tostring) + "/" + ($fid | tostring) + "/" + ($file | tostring)),
+           ProductArn: $product,
+           GeneratorId: $gen,
+           AwsAccountId: $account,
+           Types: [$type],
+           CreatedAt: $now,
+           UpdatedAt: $now,
+           Severity: { Label: sev_label($sev) },
+           Title: clip($title; 256),
+           Description: clip($desc; 1024),
+           Remediation: {
+             Recommendation: {
+               Text: clip(($f.suggested_fix // $f.fix // $f.remediation // $f.suggestion // "See the Trustabl finding for the suggested fix."); 512)
+             }
+           },
+           Resources: [
+             {
+               Type: "Other",
+               Id: clip($file; 512),
+               Partition: "aws",
+               Region: $region
+             }
+           ],
+           ProductFields: {
+             "trustabl/rule": (($f.rule_id // $f.id // "") | tostring),
+             "trustabl/tool": (($f.tool_name // $f.tool // "") | tostring),
+             "trustabl/readiness": (($root.overall_score // "") | tostring)
+           },
+           FindingProviderFields: {
+             Severity: { Label: sev_label($sev), Original: ($sev | tostring) },
+             Types: [$type]
+           }
+         }
+     ]
+   ' "$JSON" > "$OUT"
+
+COUNT=$(jq 'length' "$OUT")
+echo "to-asff: wrote $COUNT finding(s) to $OUT"
+
+if [ "$IMPORT" != "true" ]; then
+  exit 0
+fi
+
+if ! command -v aws >/dev/null 2>&1; then
+  echo "to-asff: aws CLI is required for --import" >&2
+  exit 2
+fi
+
+ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
+REGION="${AWS_REGION:-${AWS_DEFAULT_REGION:?set AWS_REGION or AWS_DEFAULT_REGION to import}}"
+PRODUCT="arn:aws:securityhub:${REGION}:${ACCOUNT}:product/${ACCOUNT}/default"
+
+TMP=$(mktemp)
+jq --arg account "$ACCOUNT" --arg region "$REGION" --arg product "$PRODUCT" \
+  'map(.AwsAccountId = $account | .ProductArn = $product | .Resources |= map(.Region = $region))' \
+  "$OUT" > "$TMP"
+mv "$TMP" "$OUT"
+
+# Security Hub accepts at most 100 findings per BatchImportFindings call.
+TOTAL=$(jq 'length' "$OUT")
+OFFSET=0
+while [ "$OFFSET" -lt "$TOTAL" ]; do
+  CHUNK=$(mktemp)
+  jq -c --argjson skip "$OFFSET" '.[$skip:$skip+100]' "$OUT" > "$CHUNK"
+  aws securityhub batch-import-findings --region "$REGION" --findings "file://${CHUNK}"
+  rm -f "$CHUNK"
+  OFFSET=$(( OFFSET + 100 ))
+done
+echo "to-asff: imported $TOTAL finding(s) into Security Hub ($REGION)"

scan/trustabl-scan.sh

@@ -13,7 +13,7 @@
 # Inputs are environment variables (all optional; sensible defaults):
 #   TARGET VERSION DETECTORS STRICT RULES_REF RULES_REPO
 #   SARIF_FILE JSON_FILE RISK_SCORE_THRESHOLD SEVERITY_THRESHOLD
-#   BRANCH GITHUB_TOKEN DEBUG
+#   BRANCH GITHUB_TOKEN DEBUG REPORT_ONLY SECURITY_HUB
 
 # ---- inputs (env, with defaults) ----
 TARGET="${TARGET:-.}"
@@ -27,6 +27,8 @@ JSON_FILE="${JSON_FILE:-trustabl.json}"
 RISK_THRESHOLD="${RISK_SCORE_THRESHOLD:-0}"
 SEV_THRESHOLD="${SEVERITY_THRESHOLD:-none}"
 BRANCH_INPUT="${BRANCH:-}"
+REPORT_ONLY="${REPORT_ONLY:-false}"
+SECURITY_HUB="${SECURITY_HUB:-false}"
 [ "${DEBUG:-false}" = "true" ] && set -x
 
 set -e
@@ -255,6 +257,14 @@ if [ "$ST" != "none" ] && [ "$ST" != "" ]; then
   fi
 fi
 
+# REPORT_ONLY implements docs/EVALUATION.md step 1 ("scan without gating
+# first") without `|| true`, which would also swallow scanner errors (exit 2).
+if [ "$FAIL" = "1" ] && [ "$REPORT_ONLY" = "true" ] && [ "$NATIVE_CODE" != "2" ]; then
+  echo "REPORT_ONLY=true: not failing the build (${REASONS[*]})"
+  FAIL=0
+  REASONS=()
+fi
+
 GREEN=$'\e[1;32m'; RED=$'\e[1;31m'; RESET=$'\e[0m'
 # No native step-summary UI on AWS; write the markdown to an artifact instead.
 SUMMARY="trustabl-summary.md"
@@ -288,9 +298,22 @@ SUMMARY="trustabl-summary.md"
   echo ""
 } >> "$SUMMARY"
 
+# Always emit ASFF next to the other artifacts. SECURITY_HUB=true also
+# batch-imports; that path is fail-closed (missing aws CLI, IAM, or Hub).
+ASFF_SCRIPT="$(cd "$(dirname "$0")" && pwd)/to-asff.sh"
+if [ -x "$ASFF_SCRIPT" ] || [ -f "$ASFF_SCRIPT" ]; then
+  if [ "$SECURITY_HUB" = "true" ]; then
+    bash "$ASFF_SCRIPT" --import "$JSON_FILE" trustabl.asff.json "$REPO"
+  else
+    bash "$ASFF_SCRIPT" "$JSON_FILE" trustabl.asff.json "$REPO" || echo "WARNING: ASFF conversion failed"
+  fi
+fi
+
 if [ "$FAIL" = "1" ]; then
   printf '%b\n' "${RED}✗ Failed due to: ${REASONS[*]}${RESET}"
   echo "### ❌ Failed — ${REASONS[*]}" >> "$SUMMARY"
+  # Report-only trials still distinguish a dead scanner from a gate hit.
+  if [ "$REPORT_ONLY" = "true" ] && [ "$NATIVE_CODE" = "2" ]; then exit 2; fi
   exit 1
 fi