diff --git a/.gitignore b/.gitignore index e40aed2..855823d 100644 --- a/.gitignore +++ b/.gitignore @@ -3,6 +3,7 @@ trustabl.json trustabl.sarif trustabl-summary.md trustabl.env +trustabl.asff.json .trustabl-bin/ trustabl/ # Claude tooling diff --git a/CHANGELOG.md b/CHANGELOG.md index a084669..9592160 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,18 @@ All notable changes to the Trustabl AWS plugin are documented here. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +### Added + +- `REPORT_ONLY=true` for evaluation-guide trials: publishes artifacts without + gating, but still fails on scanner errors (exit 2). Replaces the documented + `|| true` workaround, which swallowed dead-scanner failures. +- ASFF export (`scan/to-asff.sh`, `trustabl.asff.json`) and optional + `SECURITY_HUB=true` import into AWS Security Hub via + `securityhub:BatchImportFindings`. +- Offline tests for the ASFF converter (`test/test-asff.sh`). + ## [0.1.0] — 2026-06-17 diff --git a/README.md b/README.md index 474ea56..2fe79da 100644 --- a/README.md +++ b/README.md @@ -28,12 +28,15 @@ One scanner (`scan/trustabl-scan.sh`), two integrations: | `RISK_SCORE_THRESHOLD` | `0` | Fail when risk (100 − readiness) >= N. `0` disables. | | `SEVERITY_THRESHOLD` | `none` | Fail when any finding >= severity (`none/low/medium/high/critical`). | | `GITHUB_TOKEN` | _(none)_ | Optional — avoids GitHub's 60 req/hr anonymous rate limit. | +| `REPORT_ONLY` | `false` | Scan and publish artifacts without gating. Scanner errors (exit 2) still fail the build. | +| `SECURITY_HUB` | `false` | Import ASFF findings into AWS Security Hub (`securityhub:BatchImportFindings`). | ## Outputs -`trustabl.json`, `trustabl.sarif`, `trustabl-summary.md`, and `trustabl.env` +`trustabl.json`, `trustabl.sarif`, `trustabl-summary.md`, `trustabl.env` (`TRUSTABL_READINESS_SCORE`, `TRUSTABL_RISK_SCORE`, `TRUSTABL_MAX_SEVERITY`, -`TRUSTABL_FINDINGS_COUNT`, `TRUSTABL_EXIT_CODE`). +`TRUSTABL_FINDINGS_COUNT`, `TRUSTABL_EXIT_CODE`), and `trustabl.asff.json` (ASFF +for Security Hub). ## A note on "AWS Marketplace" diff --git a/codecatalyst/README.md b/codecatalyst/README.md index 8d03778..136ba9e 100644 --- a/codecatalyst/README.md +++ b/codecatalyst/README.md @@ -26,6 +26,7 @@ Copy into your repo, keeping the layout, then commit + push: ``` your-repo/ ├── scan/trustabl-scan.sh # the scanner +├── scan/to-asff.sh # JSON → Security Hub ASFF └── .codecatalyst/workflows/trustabl.yaml # the workflow (from codecatalyst/workflows/) ``` @@ -45,9 +46,9 @@ The workflow triggers on push to `main`. CodeCatalyst → your project → `trustabl.sarif` / `trustabl-summary.md`, surfaces findings in the **Reports** tab (SARIF), and **fails the run on any medium-or-higher finding**. -> **Report-only (don't block)?** trustabl fails on medium+ by default. To make -> it advisory, change the workflow `Run:` line to: -> `- Run: bash scan/trustabl-scan.sh || true` +> **Report-only (don't block)?** Set `REPORT_ONLY=true` in the workflow +> Variables. That publishes artifacts and still fails on scanner errors +> (exit 2). Do not use `|| true` — that also swallows a dead scanner. ### CLI? CodeCatalyst workflows are **file-driven** — there's no separate CLI to create diff --git a/codecatalyst/workflows/trustabl.yaml b/codecatalyst/workflows/trustabl.yaml index 190a499..1603970 100644 --- a/codecatalyst/workflows/trustabl.yaml +++ b/codecatalyst/workflows/trustabl.yaml @@ -27,6 +27,10 @@ Actions: Value: "none" - Name: RISK_SCORE_THRESHOLD Value: "0" + - Name: REPORT_ONLY + Value: "false" + - Name: SECURITY_HUB + Value: "false" Configuration: Steps: - Run: bash scan/trustabl-scan.sh @@ -37,6 +41,8 @@ Actions: - trustabl.json - trustabl.sarif - trustabl-summary.md + - trustabl.env + - trustabl.asff.json # Surfaces findings in the CodeCatalyst Reports tab. # NOTE: verify the SARIF report Format enum against current CodeCatalyst # docs before relying on it (the schema evolves). diff --git a/codepipeline/README.md b/codepipeline/README.md index ce3e81e..fdb97ad 100644 --- a/codepipeline/README.md +++ b/codepipeline/README.md @@ -11,11 +11,14 @@ A gate failure exits non-zero -> the CodeBuild action fails -> the pipeline stage fails. Artifacts (`trustabl.json`, `trustabl.sarif`, -`trustabl-summary.md`) are emitted to the artifact bucket. +`trustabl-summary.md`, `trustabl.env`, `trustabl.asff.json`) are emitted to the +artifact bucket. -**Optional — Security Hub:** convert findings to ASFF and -`aws securityhub batch-import-findings` to surface them in Security Hub (needs -Security Hub enabled + IAM `securityhub:BatchImportFindings`). Not in v0.1.0. +**Optional — Security Hub:** set `SECURITY_HUB=true` on the CodeBuild project. +The scanner writes `trustabl.asff.json` and then calls +`aws securityhub batch-import-findings`. Needs Security Hub enabled in the +region and IAM `securityhub:BatchImportFindings` plus `sts:GetCallerIdentity` +on the CodeBuild role. ## Quickstart — from zero (console) @@ -28,6 +31,7 @@ Copy these two into your repo, keeping the layout, then commit + push: ``` your-repo/ ├── scan/trustabl-scan.sh # the scanner +├── scan/to-asff.sh # JSON → Security Hub ASFF └── codepipeline/buildspec.yml # tells CodeBuild to run it ``` @@ -37,7 +41,8 @@ Console → **CodeBuild → Create build project**: - **Buildspec:** "Use a buildspec file" → path `codepipeline/buildspec.yml` - **Service role:** let CodeBuild create one (it needs CloudWatch Logs) - *(optional)* **Env vars:** `SEVERITY_THRESHOLD`, `RISK_SCORE_THRESHOLD`, - `VERSION` (pin a tag), `GITHUB_TOKEN` (dodges GitHub's 60-req/hr anon limit) + `VERSION` (pin a tag), `GITHUB_TOKEN` (dodges GitHub's 60-req/hr anon limit), + `REPORT_ONLY=true` (trial without gating), `SECURITY_HUB=true` (import ASFF) - **`GITHUB_TOKEN` is a credential** — add it with type **Secrets Manager** or **Parameter Store**, not as a plaintext environment variable. Plaintext env vars are readable by anyone with `codebuild:BatchGetProjects`. @@ -53,9 +58,10 @@ prints the readiness report, uploads `trustabl.json` / `trustabl.sarif` / `trustabl-summary.md` as artifacts, and **fails the stage if any finding is medium-or-higher** — so the pipeline stops on unsafe agent code. -> **Want report-only (don't block the pipeline)?** trustabl fails on medium+ by -> default. To make it advisory, change the build command in your buildspec to: -> `- bash "$CODEBUILD_SRC_DIR/scan/trustabl-scan.sh" || true` +> **Want report-only (don't block the pipeline)?** Set `REPORT_ONLY=true` on +> the CodeBuild project. That publishes artifacts and still fails the build on +> scanner errors (exit 2). Do not use `|| true` — that also swallows a dead +> scanner, which [docs/EVALUATION.md](../docs/EVALUATION.md) says not to trust. ### CLI (alternative to steps 2–3) After vendoring (step 1), replace `/` and ``: diff --git a/codepipeline/buildspec.yml b/codepipeline/buildspec.yml index a77d0fc..10d69b4 100644 --- a/codepipeline/buildspec.yml +++ b/codepipeline/buildspec.yml @@ -13,7 +13,9 @@ version: 0.2 # TARGET (default "."), VERSION (default "latest"), DETECTORS, STRICT, # RULES_REF, RULES_REPO, SARIF_FILE, JSON_FILE, # RISK_SCORE_THRESHOLD (default 0), SEVERITY_THRESHOLD (default none), -# GITHUB_TOKEN (optional — dodges GitHub's 60 req/hr anon rate limit). +# GITHUB_TOKEN (optional — dodges GitHub's 60 req/hr anon rate limit), +# REPORT_ONLY (default false — scan without gating; scanner errors still fail), +# SECURITY_HUB (default false — import findings into AWS Security Hub). env: variables: @@ -37,3 +39,5 @@ artifacts: - trustabl.json - trustabl.sarif - trustabl-summary.md + - trustabl.env + - trustabl.asff.json diff --git a/docs/EVALUATION.md b/docs/EVALUATION.md index 6a1205a..f26d68e 100644 --- a/docs/EVALUATION.md +++ b/docs/EVALUATION.md @@ -137,18 +137,22 @@ Vendor `scan/` plus the config for your platform into your repo, then set variables on the CodeBuild project or CodeCatalyst workflow: ```yaml -SEVERITY_THRESHOLD: high # report-only: omit this +REPORT_ONLY: "true" # step 1: artifacts only; scanner errors still fail +SEVERITY_THRESHOLD: high # step 4: turn gating on ``` Variables are **UPPER_SNAKE**: `SEVERITY_THRESHOLD`, `RISK_SCORE_THRESHOLD`, -`STRICT`, `DETECTORS`, `VERSION`. +`STRICT`, `DETECTORS`, `VERSION`, `REPORT_ONLY`, `SECURITY_HUB`. + +Do not use `|| true` around the scan to get report-only behaviour. That also +swallows exit 2 (scanner/I/O failure), and those results must not be trusted. ## Where the results appear | Surface | What you get | |---|---| | **Build log** | The readiness panel | -| **Artifacts** | `trustabl.json`, `trustabl.sarif`, `trustabl-summary.md`, `trustabl.env` | +| **Artifacts** | `trustabl.json`, `trustabl.sarif`, `trustabl-summary.md`, `trustabl.env`, `trustabl.asff.json` | | **CodeCatalyst Reports tab** | SARIF report, when configured in the workflow | A gate failure exits non-zero, which fails the CodeBuild action and therefore the @@ -157,5 +161,6 @@ pipeline stage. `trustabl.env` exposes `TRUSTABL_EXIT_CODE`, `TRUSTABL_READINESS_SCORE`, `TRUSTABL_RISK_SCORE` and `TRUSTABL_MAX_SEVERITY` for downstream steps. -For a trial, run the CodeBuild project on its own before wiring it into a -pipeline. It exercises the same scan without gating anything real. +For a trial, set `REPORT_ONLY=true` and run the CodeBuild project on its own +before wiring it into a pipeline. That exercises the same scan without gating +a real stage, while still failing if the scanner itself errors. diff --git a/scan/to-asff.sh b/scan/to-asff.sh new file mode 100755 index 0000000..5c5cb91 --- /dev/null +++ b/scan/to-asff.sh @@ -0,0 +1,135 @@ +#!/usr/bin/env bash +# Convert a Trustabl JSON ScanResult into AWS Security Hub ASFF (2018-10-08) +# and optionally import it with `aws securityhub batch-import-findings`. +# +# Usage: +# to-asff.sh [json] [out] [resource-id] +# to-asff.sh --import [json] [out] [resource-id] +# +# Defaults: JSON_FILE/trustabl.json, trustabl.asff.json, TARGET/repo. +# +# Does not require AWS credentials unless --import is set. Placeholders are +# used for ProductArn / AwsAccountId and rewritten at import time from +# `aws sts get-caller-identity` + AWS_REGION. +set -euo pipefail + +IMPORT=false +if [ "${1:-}" = "--import" ]; then IMPORT=true; shift; fi + +JSON="${1:-${JSON_FILE:-trustabl.json}}" +OUT="${2:-trustabl.asff.json}" +RESOURCE="${3:-${TARGET:-repository}}" + +if [ ! -f "$JSON" ]; then + echo "to-asff: JSON result not found: $JSON" >&2 + exit 2 +fi +if ! command -v jq >/dev/null 2>&1; then + echo "to-asff: jq is required" >&2 + exit 2 +fi + +REGION="${AWS_REGION:-${AWS_DEFAULT_REGION:-us-east-1}}" +ACCOUNT="${AWS_ACCOUNT_ID:-000000000000}" +NOW=$(date -u +%Y-%m-%dT%H:%M:%S.000Z) +GENERATOR="trustabl" + +# Security Hub BatchImportFindings accepts a JSON array of findings. +jq --arg region "$REGION" \ + --arg account "$ACCOUNT" \ + --arg now "$NOW" \ + --arg gen "$GENERATOR" \ + --arg resource "$RESOURCE" \ + --arg product "arn:aws:securityhub:${REGION}:${ACCOUNT}:product/${ACCOUNT}/default" \ + ' + def sev_label($s): + if $s == "critical" then "CRITICAL" + elif $s == "high" then "HIGH" + elif $s == "medium" then "MEDIUM" + elif $s == "low" then "LOW" + else "INFORMATIONAL" end; + def clip($s; $n): + ($s | tostring) as $t + | if ($t | length) <= $n then $t else $t[0:$n-1] + "…" end; + . as $root + | ($root.findings // []) as $findings + | [ range(0; $findings | length) as $i + | $findings[$i] + | . as $f + | ($f.severity // $f.level // "info") as $sev + | ($f.rule_id // $f.id // $f.check_id // $f.title // "finding") as $fid + | ($f.title // $f.rule_id // $f.message // "Trustabl finding") as $title + | ($f.explanation // $f.message // $f.description // $f.title // "No description provided") as $desc + | ($f.file_path // $f.file // $f.path // $f.location.file // $f.location.uri // $resource) as $file + | ("Software and Configuration Checks/Code Analysis/Agent Reliability") as $type + | { + SchemaVersion: "2018-10-08", + Id: ("trustabl/" + ($i | tostring) + "/" + ($fid | tostring) + "/" + ($file | tostring)), + ProductArn: $product, + GeneratorId: $gen, + AwsAccountId: $account, + Types: [$type], + CreatedAt: $now, + UpdatedAt: $now, + Severity: { Label: sev_label($sev) }, + Title: clip($title; 256), + Description: clip($desc; 1024), + Remediation: { + Recommendation: { + Text: clip(($f.suggested_fix // $f.fix // $f.remediation // $f.suggestion // "See the Trustabl finding for the suggested fix."); 512) + } + }, + Resources: [ + { + Type: "Other", + Id: clip($file; 512), + Partition: "aws", + Region: $region + } + ], + ProductFields: { + "trustabl/rule": (($f.rule_id // $f.id // "") | tostring), + "trustabl/tool": (($f.tool_name // $f.tool // "") | tostring), + "trustabl/readiness": (($root.overall_score // "") | tostring) + }, + FindingProviderFields: { + Severity: { Label: sev_label($sev), Original: ($sev | tostring) }, + Types: [$type] + } + } + ] + ' "$JSON" > "$OUT" + +COUNT=$(jq 'length' "$OUT") +echo "to-asff: wrote $COUNT finding(s) to $OUT" + +if [ "$IMPORT" != "true" ]; then + exit 0 +fi + +if ! command -v aws >/dev/null 2>&1; then + echo "to-asff: aws CLI is required for --import" >&2 + exit 2 +fi + +ACCOUNT=$(aws sts get-caller-identity --query Account --output text) +REGION="${AWS_REGION:-${AWS_DEFAULT_REGION:?set AWS_REGION or AWS_DEFAULT_REGION to import}}" +PRODUCT="arn:aws:securityhub:${REGION}:${ACCOUNT}:product/${ACCOUNT}/default" + +TMP=$(mktemp) +jq --arg account "$ACCOUNT" --arg region "$REGION" --arg product "$PRODUCT" \ + 'map(.AwsAccountId = $account | .ProductArn = $product | .Resources |= map(.Region = $region))' \ + "$OUT" > "$TMP" +mv "$TMP" "$OUT" + +# Security Hub accepts at most 100 findings per BatchImportFindings call. +TOTAL=$(jq 'length' "$OUT") +OFFSET=0 +while [ "$OFFSET" -lt "$TOTAL" ]; do + CHUNK=$(mktemp) + jq -c --argjson skip "$OFFSET" '.[$skip:$skip+100]' "$OUT" > "$CHUNK" + aws securityhub batch-import-findings --region "$REGION" --findings "file://${CHUNK}" + rm -f "$CHUNK" + OFFSET=$(( OFFSET + 100 )) +done +echo "to-asff: imported $TOTAL finding(s) into Security Hub ($REGION)" diff --git a/scan/trustabl-scan.sh b/scan/trustabl-scan.sh old mode 100644 new mode 100755 index caa6314..ff77b15 --- a/scan/trustabl-scan.sh +++ b/scan/trustabl-scan.sh @@ -13,7 +13,7 @@ # Inputs are environment variables (all optional; sensible defaults): # TARGET VERSION DETECTORS STRICT RULES_REF RULES_REPO # SARIF_FILE JSON_FILE RISK_SCORE_THRESHOLD SEVERITY_THRESHOLD -# BRANCH GITHUB_TOKEN DEBUG +# BRANCH GITHUB_TOKEN DEBUG REPORT_ONLY SECURITY_HUB # ---- inputs (env, with defaults) ---- TARGET="${TARGET:-.}" @@ -27,6 +27,8 @@ JSON_FILE="${JSON_FILE:-trustabl.json}" RISK_THRESHOLD="${RISK_SCORE_THRESHOLD:-0}" SEV_THRESHOLD="${SEVERITY_THRESHOLD:-none}" BRANCH_INPUT="${BRANCH:-}" +REPORT_ONLY="${REPORT_ONLY:-false}" +SECURITY_HUB="${SECURITY_HUB:-false}" [ "${DEBUG:-false}" = "true" ] && set -x set -e @@ -255,6 +257,14 @@ if [ "$ST" != "none" ] && [ "$ST" != "" ]; then fi fi +# REPORT_ONLY implements docs/EVALUATION.md step 1 ("scan without gating +# first") without `|| true`, which would also swallow scanner errors (exit 2). +if [ "$FAIL" = "1" ] && [ "$REPORT_ONLY" = "true" ] && [ "$NATIVE_CODE" != "2" ]; then + echo "REPORT_ONLY=true: not failing the build (${REASONS[*]})" + FAIL=0 + REASONS=() +fi + GREEN=$'\e[1;32m'; RED=$'\e[1;31m'; RESET=$'\e[0m' # No native step-summary UI on AWS; write the markdown to an artifact instead. SUMMARY="trustabl-summary.md" @@ -288,9 +298,22 @@ SUMMARY="trustabl-summary.md" echo "" } >> "$SUMMARY" +# Always emit ASFF next to the other artifacts. SECURITY_HUB=true also +# batch-imports; that path is fail-closed (missing aws CLI, IAM, or Hub). +ASFF_SCRIPT="$(cd "$(dirname "$0")" && pwd)/to-asff.sh" +if [ -x "$ASFF_SCRIPT" ] || [ -f "$ASFF_SCRIPT" ]; then + if [ "$SECURITY_HUB" = "true" ]; then + bash "$ASFF_SCRIPT" --import "$JSON_FILE" trustabl.asff.json "$REPO" + else + bash "$ASFF_SCRIPT" "$JSON_FILE" trustabl.asff.json "$REPO" || echo "WARNING: ASFF conversion failed" + fi +fi + if [ "$FAIL" = "1" ]; then printf '%b\n' "${RED}✗ Failed due to: ${REASONS[*]}${RESET}" echo "### ❌ Failed — ${REASONS[*]}" >> "$SUMMARY" + # Report-only trials still distinguish a dead scanner from a gate hit. + if [ "$REPORT_ONLY" = "true" ] && [ "$NATIVE_CODE" = "2" ]; then exit 2; fi exit 1 fi diff --git a/test/fixtures/scan.json b/test/fixtures/scan.json new file mode 100644 index 0000000..9e47f4c --- /dev/null +++ b/test/fixtures/scan.json @@ -0,0 +1,39 @@ +{ + "overall_score": 0.42, + "findings": [ + { + "rule_id": "TOOL-001", + "severity": "critical", + "title": "Unsandboxed shell tool", + "explanation": "Agent tool runs a caller-controlled shell command.", + "suggested_fix": "Constrain the argv and drop the raw shell.", + "file_path": "src/tools/shell.py", + "start_line": 12, + "end_line": 18, + "tool_name": "run_shell", + "confidence": 0.91 + }, + { + "rule_id": "MCP-007", + "severity": "medium", + "title": "Caller-controlled URL fetch", + "explanation": "MCP tool fetches a URL from the prompt.", + "suggested_fix": "Allowlist destinations and pin the scheme.", + "file_path": "src/mcp/fetch.ts", + "start_line": 40, + "end_line": 40, + "tool_name": "fetch_url", + "confidence": 0.8 + }, + { + "rule_id": "META-002", + "severity": "info", + "title": "META: SDK not fully inventoried", + "explanation": "Dynamic agent construction skipped.", + "file_path": "src/agent.py", + "start_line": 1, + "end_line": 1, + "confidence": 1 + } + ] +} diff --git a/test/test-asff.sh b/test/test-asff.sh new file mode 100755 index 0000000..df9833b --- /dev/null +++ b/test/test-asff.sh @@ -0,0 +1,70 @@ +#!/usr/bin/env bash +# Offline tests for scan/to-asff.sh. No AWS credentials, no trustabl binary. +set -euo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +ASFF="$ROOT/scan/to-asff.sh" +FIX="$ROOT/test/fixtures/scan.json" +OUT="$(mktemp)" +trap 'rm -f "$OUT"' EXIT + +bash "$ASFF" "$FIX" "$OUT" "demo/repo" + +python3 - "$OUT" <<'PY' +import json, sys +path = sys.argv[1] +findings = json.load(open(path)) +assert isinstance(findings, list), findings +assert len(findings) == 3, len(findings) +labels = {f["Severity"]["Label"] for f in findings} +assert labels == {"CRITICAL", "MEDIUM", "INFORMATIONAL"}, labels +ids = [] +for f in findings: + assert f["SchemaVersion"] == "2018-10-08" + assert f["Id"].startswith("trustabl/") + assert f["ProductArn"].startswith("arn:aws:securityhub:") + assert f["AwsAccountId"] + assert f["GeneratorId"] == "trustabl" + assert f["Title"] + assert f["Description"] + assert f["Resources"][0]["Type"] == "Other" + assert len(f["Title"]) <= 256 + assert len(f["Description"]) <= 1024 + assert "CVE" not in "".join(f["Types"]) + assert f["FindingProviderFields"]["Types"] == f["Types"] + assert f["FindingProviderFields"]["Severity"]["Label"] == f["Severity"]["Label"] + ids.append(f["Id"]) +assert len(ids) == len(set(ids)), ids +# info/META must not be labelled as a defect severity +info = next(f for f in findings if "META" in f["Title"]) +assert info["Severity"]["Label"] == "INFORMATIONAL" +shell = next(f for f in findings if f["Id"].endswith("src/tools/shell.py")) +assert "caller-controlled shell" in shell["Description"] +assert "Constrain the argv" in shell["Remediation"]["Recommendation"]["Text"] +assert shell["Resources"][0]["Id"] == "src/tools/shell.py" +assert shell["ProductFields"]["trustabl/rule"] == "TOOL-001" +print("ok: 3 findings, severity mapping, ASFF shape") +PY + +# Empty findings -> empty array, still valid ASFF input +EMPTY="$(mktemp)" +echo '{"overall_score": 1, "findings": []}' > "$EMPTY" +bash "$ASFF" "$EMPTY" "$OUT" "demo/repo" +python3 - "$OUT" <<'PY' +import json, sys +findings = json.load(open(sys.argv[1])) +assert findings == [] +print("ok: empty findings") +PY +rm -f "$EMPTY" + +# --import without aws CLI must fail closed (exit 2), not skip +if command -v aws >/dev/null 2>&1; then + echo "skip: aws CLI present; import fail-closed tested only when absent" +else + set +e + bash "$ASFF" --import "$FIX" "$OUT" "demo/repo" + rc=$? + set -e + [ "$rc" -eq 2 ] || { echo "expected exit 2 without aws CLI, got $rc"; exit 1; } + echo "ok: --import fail-closed without aws CLI" +fi