feat(ci): phase C — authenticated ZAP scan + flaky test quarantine

.agents/skills/gitops-principles/SKILL.md

@@ -0,0 +1,19 @@
+# GitOps Principles Skill — prei
+
+## When to load
+- Before editing anything under `.github/workflows/`
+- Before touching deployment config, the GitOps manifest repo, or `docker-compose.yml`
+- When reasoning about rollback, canary, or progressive-delivery behavior
+
+## Principles
+
+1. **Git is the source of truth.** Config, workflows, deployment state — all in git. Never modify running infrastructure directly.
+2. **Immutable artifacts.** The Docker image is the deployable unit. Test the artifact, not the source. Never rebuild for deployment.
+3. **PR gates are deploy gates.** Every merge to `main` is a deploy candidate. Broken `main` blocks all PRs — fix main CI before merging anything else.
+4. **Declarative pipelines.** Workflows describe DESIRED STATE: what artifacts, what gates, what triggers. Not imperative scripts.
+5. **Artifact verification.** Every PR that touches workflows or deployment config must be verified against the live container via `post-deployment.yml`.
+6. **Rollback is `git revert`.** Rollback to a previous commit on the GitOps manifest repo. The rollback job in `post-deployment.yml` is a safety net, not the primary mechanism.
+7. **Observability built-in.** Every workflow step logs `job-start` / `job-finish` timestamps. Build times, test results, deploy status are traceable.
+8. **Progressive delivery.** Canary → staging → production (see `docs/DEPLOYMENT_STRATEGY.md`). Never ship to 100% in one step.
+9. **Naming is infrastructure.** PR titles follow Conventional Commits. Commit messages describe intent. Tags trigger deployments.
+10. **Branch discipline.** All work happens on feature branches off `main` (trunk-based development, short-lived). Never commit directly to `main`. Branch naming: `feat/<slug>`, `fix/<slug>`, `chore/<slug>`, `docs/<slug>`. Every branch opens a PR through CI gates before merge.

.github/scripts/flaky_report.py

@@ -0,0 +1,140 @@
+#!/usr/bin/env python3
+"""Detect flaky tests from a pytest --report-log JSONL file.
+
+A test is "flaky this run" when pytest-rerunfailures had to rerun it
+(outcome == "rerun" on an earlier "call" report) and the final "call"
+report for the same nodeid passed. Tests that still fail after retries
+are real failures, not flaky ones, and are left alone.
+
+Modes:
+  --mode report  Print a markdown summary (and append to
+                 $GITHUB_STEP_SUMMARY if set). Does not touch the ledger.
+  --mode write   Same summary, plus updates docs/quality/flaky_tests.json
+                 and tests/.flaky_quarantine.txt. Intended to run only
+                 from the single main-branch writer job.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from datetime import datetime, timezone
+from pathlib import Path
+
+DEFAULT_LEDGER = Path("docs/quality/flaky_tests.json")
+DEFAULT_QUARANTINE = Path("tests/.flaky_quarantine.txt")
+DEFAULT_THRESHOLD = 3
+
+
+def find_flaky_nodeids(report_log_path: Path) -> list[str]:
+    """Return nodeids whose call reports show rerun-then-pass this run."""
+    call_outcomes: dict[str, list[str]] = {}
+
+    if not report_log_path.exists():
+        return []
+
+    with report_log_path.open() as f:
+        for line in f:
+            line = line.strip()
+            if not line:
+                continue
+            record = json.loads(line)
+            if record.get("$report_type") != "TestReport":
+                continue
+            if record.get("when") != "call":
+                continue
+            nodeid = record.get("nodeid")
+            outcome = record.get("outcome")
+            if nodeid is None or outcome is None:
+                continue
+            call_outcomes.setdefault(nodeid, []).append(outcome)
+
+    flaky = []
+    for nodeid, outcomes in call_outcomes.items():
+        if "rerun" in outcomes and outcomes[-1] == "passed":
+            flaky.append(nodeid)
+    return sorted(flaky)
+
+
+def load_ledger(ledger_path: Path) -> dict:
+    if not ledger_path.exists():
+        return {}
+    with ledger_path.open() as f:
+        data: dict = json.load(f)
+        return data
+
+
+def update_ledger(ledger: dict, flaky_nodeids: list[str], threshold: int) -> dict:
+    now = datetime.now(timezone.utc).isoformat()
+    for nodeid in flaky_nodeids:
+        entry = ledger.get(nodeid, {"count": 0, "first_seen": now})
+        entry["count"] = entry.get("count", 0) + 1
+        entry.setdefault("first_seen", now)
+        entry["last_seen"] = now
+        entry["quarantined"] = entry["count"] >= threshold
+        ledger[nodeid] = entry
+    return dict(sorted(ledger.items()))
+
+
+def write_quarantine(ledger: dict, quarantine_path: Path) -> None:
+    quarantined = sorted(
+        nodeid for nodeid, entry in ledger.items() if entry.get("quarantined")
+    )
+    quarantine_path.parent.mkdir(parents=True, exist_ok=True)
+    with quarantine_path.open("w") as f:
+        for nodeid in quarantined:
+            f.write(f"{nodeid}\n")
+
+
+def render_summary(flaky_nodeids: list[str], ledger: dict) -> str:
+    if not flaky_nodeids:
+        return "No flaky tests detected this run.\n"
+
+    lines = [
+        "| Test | Cumulative count | Quarantined |",
+        "| --- | --- | --- |",
+    ]
+    for nodeid in flaky_nodeids:
+        entry = ledger.get(nodeid, {})
+        count = entry.get("count", "1 (this run)")
+        quarantined = "yes" if entry.get("quarantined") else "no"
+        lines.append(f"| `{nodeid}` | {count} | {quarantined} |")
+    return "\n".join(lines) + "\n"
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--report-log", type=Path, required=True)
+    parser.add_argument("--mode", choices=["report", "write"], required=True)
+    parser.add_argument("--ledger", type=Path, default=DEFAULT_LEDGER)
+    parser.add_argument("--quarantine", type=Path, default=DEFAULT_QUARANTINE)
+    parser.add_argument("--threshold", type=int, default=DEFAULT_THRESHOLD)
+    args = parser.parse_args()
+
+    flaky_nodeids = find_flaky_nodeids(args.report_log)
+    ledger = load_ledger(args.ledger)
+
+    if args.mode == "write":
+        ledger = update_ledger(ledger, flaky_nodeids, args.threshold)
+        args.ledger.parent.mkdir(parents=True, exist_ok=True)
+        with args.ledger.open("w") as f:
+            json.dump(ledger, f, indent=2, sort_keys=True)
+            f.write("\n")
+        write_quarantine(ledger, args.quarantine)
+
+    summary = render_summary(flaky_nodeids, ledger)
+    sys.stdout.write(summary)
+
+    step_summary = os.environ.get("GITHUB_STEP_SUMMARY")
+    if step_summary:
+        with open(step_summary, "a") as f:
+            f.write("\n## Flaky test report\n\n")
+            f.write(summary)
+
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

.github/workflows/ci-quality.yml

@@ -104,13 +104,23 @@ jobs:
           coverage run --parallel-mode -m pytest tests/ core/tests/ tests_bdd/ prei/pipeline/tests/ \
             -q --tb=short \
             -k "not e2e and not docker and not integration and not container and not startup and not add_to_pipeline and not acceptance and not export and not unreachable_url"
+      - name: Flaky test report
+        if: always()
+        run: python .github/scripts/flaky_report.py --report-log .pytest-report.jsonl --mode report
       - name: Upload coverage data
         uses: actions/upload-artifact@v7
         with:
           name: coverage-unit
           path: .coverage.*
           include-hidden-files: true
           retention-days: 1
+      - name: Upload test report log
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: report-log-unit
+          path: .pytest-report.jsonl
+          retention-days: 1
 
   tests-integration:
     name: "🔗 Integration Tests"
@@ -132,13 +142,23 @@ jobs:
           coverage run --parallel-mode -m pytest tests/ core/tests/ \
             -q --tb=short \
             -k "integration"
+      - name: Flaky test report
+        if: always()
+        run: python .github/scripts/flaky_report.py --report-log .pytest-report.jsonl --mode report
       - name: Upload coverage data
         uses: actions/upload-artifact@v7
         with:
           name: coverage-integration
           path: .coverage.*
           include-hidden-files: true
           retention-days: 1
+      - name: Upload test report log
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: report-log-integration
+          path: .pytest-report.jsonl
+          retention-days: 1
 
   tests-e2e:
     name: "🌐 E2E Tests"
@@ -160,13 +180,23 @@ jobs:
           coverage run --parallel-mode -m pytest tests/ \
             -q --tb=short \
             -k "e2e or docker or container or startup or add_to_pipeline or export"
+      - name: Flaky test report
+        if: always()
+        run: python .github/scripts/flaky_report.py --report-log .pytest-report.jsonl --mode report
       - name: Upload coverage data
         uses: actions/upload-artifact@v7
         with:
           name: coverage-e2e
           path: .coverage.*
           include-hidden-files: true
           retention-days: 1
+      - name: Upload test report log
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: report-log-e2e
+          path: .pytest-report.jsonl
+          retention-days: 1
 
   typecheck:
     name: "🔷 Typecheck"
@@ -221,6 +251,69 @@ jobs:
       - name: Verify KPI calculations
         run: python -m pytest tests/test_finance_math.py -v --tb=short
 
+  # ── Authenticated security scan (ephemeral CI instance, not live deploy) ──
+  # See docs/KNOWN_LIMITATIONS.md LIMIT-22: this scans a freshly-migrated,
+  # freshly-seeded CI-only instance, not the real deployment. The existing
+  # unauthenticated full scan in post-deployment.yml still runs against the
+  # live artifact and is unchanged.
+
+  zap-authenticated-scan:
+    name: "🛡️ Authenticated ZAP Scan"
+    runs-on: ubuntu-latest
+    env:
+      DJANGO_SETTINGS_MODULE: investor_app.settings_test
+      SECRET_KEY: ci-zap-scan-key
+      DATABASE_URL: "sqlite:////tmp/zap_scan.sqlite3"
+      ZAP_AUTH_USERNAME: zap-ci-scan-only
+    steps:
+      - uses: actions/checkout@v7
+      - uses: ./.github/actions/python-setup
+      - name: Generate throwaway scan credential
+        run: |
+          echo "ZAP_AUTH_PASSWORD=$(python3 -c 'import secrets; print(secrets.token_urlsafe(24))')" >> "$GITHUB_ENV"
+      - name: Install dependencies
+        run: pip install -r requirements.txt -q
+      - name: Migrate
+        run: python manage.py migrate --noinput
+      - name: Collect static files
+        run: python manage.py collectstatic --noinput --verbosity 0
+      - name: Seed ZAP scan user
+        run: python manage.py seed_zap_scan_user
+      - name: Render ZAP auth context
+        run: |
+          CREDS_B64=$(printf 'username=%s&password=%s' "$ZAP_AUTH_USERNAME" "$ZAP_AUTH_PASSWORD" | base64 -w0)
+          sed "s/__ZAP_AUTH_CREDS_B64__/$CREDS_B64/" .zap/prei-auth-context.xml > .zap/prei-auth-context-runtime.xml
+      - name: Start server
+        run: nohup python manage.py runserver 0.0.0.0:8000 > server.log 2>&1 &
+      - name: Wait for healthy
+        run: |
+          set -e
+          HEALTHY=0
+          for i in $(seq 1 60); do
+            STATUS=$(curl -sf -o /dev/null -w "%{http_code}" http://localhost:8000/health/ 2>/dev/null || echo "000")
+            if [ "$STATUS" = "200" ]; then
+              echo "Healthy after ~$((i * 2))s"
+              HEALTHY=1
+              break
+            fi
+            sleep 2
+          done
+          if [ "$HEALTHY" != "1" ]; then
+            echo "::error::Server never became healthy within 120s"
+            cat server.log || true
+            exit 1
+          fi
+      - name: OWASP ZAP Authenticated Full Scan
+        uses: zaproxy/action-full-scan@v0.13.0
+        with:
+          target: "http://localhost:8000"
+          allow_issue_writing: false
+          fail_action: true
+          cmd_options: "-a -n /zap/wrk/.zap/prei-auth-context-runtime.xml -U zap-ci-scan-only -I"
+      - name: Server logs
+        if: always()
+        run: cat server.log || true
+
   # ── Coverage combine (gate: all test jobs must pass) ──────────────────────
 
   coverage:
@@ -253,7 +346,7 @@ jobs:
 
   pr-gates-pass:
     name: "All Gates Passed"
-    needs: [pr-title, lint, secrets, django-checks, tests-unit, tests-integration, tests-e2e, typecheck, acceptance-check, finance-math, coverage]
+    needs: [pr-title, lint, secrets, django-checks, tests-unit, tests-integration, tests-e2e, typecheck, acceptance-check, finance-math, coverage, zap-authenticated-scan]
     runs-on: ubuntu-latest
     if: always()
     steps:
@@ -271,6 +364,7 @@ jobs:
           if [ "${{ needs.acceptance-check.result }}" != "success" ]; then FAILED="$FAILED acceptance"; fi
           if [ "${{ needs.finance-math.result }}" != "success" ]; then FAILED="$FAILED finance-math"; fi
           if [ "${{ needs.coverage.result }}" != "success" ]; then FAILED="$FAILED coverage"; fi
+          if [ "${{ needs.zap-authenticated-scan.result }}" != "success" ]; then FAILED="$FAILED zap-authenticated-scan"; fi
           if [ -n "$FAILED" ]; then
             echo "❌ Failed gates:${FAILED}"
             exit 1

.github/workflows/docker-publish.yml

@@ -105,7 +105,7 @@ jobs:
     needs: build-image
     runs-on: ubuntu-latest
     permissions:
-      contents: read
+      contents: write
       packages: read
     steps:
       - name: "job-start"
@@ -216,6 +216,29 @@ jobs:
             prei-live-test \
             python -m pytest tests_bdd/ -q --tb=short 2>&1
           echo "=== All BDD acceptance tests passed ==="
+      - name: Extract test report log
+        if: always()
+        run: docker cp prei-live-test:/app/.pytest-report.jsonl .pytest-report.jsonl 2>/dev/null || true
+      - name: Update flaky test ledger
+        if: always()
+        run: |
+          if [ -f .pytest-report.jsonl ]; then
+            python3 .github/scripts/flaky_report.py --report-log .pytest-report.jsonl --mode write
+          else
+            echo "No report log found, skipping flaky ledger update"
+          fi
+      - name: Commit flaky test ledger
+        if: always()
+        run: |
+          if git diff --quiet -- docs/quality/flaky_tests.json tests/.flaky_quarantine.txt; then
+            echo "No ledger changes"
+          else
+            git config user.name "github-actions[bot]"
+            git config user.email "github-actions[bot]@users.noreply.github.com"
+            git add docs/quality/flaky_tests.json tests/.flaky_quarantine.txt
+            git commit -m "chore(ci): update flaky test ledger [skip ci]"
+            git push origin HEAD:main
+          fi
       - name: "job-finish"
         if: always()
         run: |

.github/workflows/post-deployment.yml

@@ -107,7 +107,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: OWASP ZAP Full Scan
-        uses: zaproxy/action-full-scan@v1
+        uses: zaproxy/action-full-scan@v0.13.0
         with:
           target: ${{ needs.smoke.outputs.target }}
           allow_issue_writing: false

.gitignore

@@ -82,4 +82,6 @@ test-report.md
 verification-output.md
 verification-report.md
 .coverage
+.pytest-report.jsonl
+.zap/prei-auth-context-runtime.xml
 drafts/