Feat: Add AIAC evaluation framework and eval suite (aiac/eval/)

aiac/.gitignore

@@ -13,5 +13,10 @@ test/pdp/policy/rego_out/
 # committed fixture and is not under rego_out/, so it stays tracked.
 test/integration/rego_out/
 
+# policy-eval-scenarios generated Rego + per-run pass/fail/skip reports
+# (eval/test_policy_pipeline_eval.py); regenerated per run.
+eval/rego_out/
+eval/reports/
+
 # uc1-onboarding demo's captured .rego snapshots — regenerated per run, not source of truth.
 demo/use-cases/uc1-onboarding/generated/

aiac/CLAUDE.md

@@ -70,12 +70,15 @@ ls src/aiac/<subsystem>/               # drill into any layer
 **Unit test command:**
 
 ```bash
-.venv/bin/pytest test/ -m "not integration"
+.venv/bin/pytest test/
 ```
 
-The whole `test/` tree collects and runs green — no `--ignore` flags are needed.
-(This wasn't always true: the Policy Computation Engine was migrated to the SPM
-store surface in Wave 3, which resolved the earlier PCE-chain collection
+`pyproject.toml`'s `addopts` defaults `-m` to excluding every live-infra marker
+(`integration`, `eval_extended`, `eval_consistency`,
+`eval_robustness`), so a bare invocation never makes a real LLM/Keycloak
+call. The whole `test/` tree collects and runs green — no `--ignore` flags are
+needed. (This wasn't always true: the Policy Computation Engine was migrated to
+the SPM store surface in Wave 3, which resolved the earlier PCE-chain collection
 failures.)
 
 The `-m "not integration"` expression needs no external services. The live-LLM
@@ -120,6 +123,16 @@ set -a; . test/integration/.env; set +a
 
 When the cluster is not wired or the env is unset, the suite **skips cleanly** (it never false-passes).
 
+A passed `-m` always overrides the default, so this opts back into exactly
+`integration` (not the heavier markers below). Three heavier, narrower-infra
+markers exist alongside it — `eval_extended` (same live infra as
+`integration`, many more PRB/LLM calls), `eval_consistency` and
+`eval_robustness` (LLM only, no Keycloak/`opa`) — each invoked the same
+way, e.g. `pytest eval/ -m eval_extended`. See
+`docs/specs/eval/policy-eval-scenarios.md` and
+`docs/specs/eval/policy-eval-robustness-consistency.md` for their
+runbooks.
+
 **Smoke test** (requires live service at `AIAC_PDP_CONFIG_URL`, default `http://127.0.0.1:7071`):
 
 ```bash

aiac/demo/assets/agents/github_agent/k8s/github-agent-deployment.yaml

@@ -94,34 +94,23 @@ spec:
             seccompProfile:
               type: RuntimeDefault
           env:
-            # LLM configuration - Ollama (local LLM)
-            # Matches upstream .env.ollama from agent-examples repo
+            # LLM configuration - OpenAI-compatible endpoint (litellm proxy)
             - name: TASK_MODEL_ID
-              value: "ollama/ibm/granite4:latest"
-            # Ollama API base URL. Required by litellm (used by crewai >=1.10).
-            # For Docker Desktop / Kind: http://host.docker.internal:11434
-            # For in-cluster Ollama:     http://ollama.ollama.svc:11434
+              value: "Azure/gpt-5-mini-2025-08-07"
             - name: LLM_API_BASE
-              value: "http://host.docker.internal:11434"
-            - name: OLLAMA_API_BASE
-              value: "http://host.docker.internal:11434"
+              value: "https://ete-litellm.ai-models.vpc-int.res.ibm.com/v1"
             - name: LLM_API_KEY
-              value: "ollama"
+              valueFrom:
+                secretKeyRef:
+                  name: openai-secret
+                  key: apikey
+            - name: OPENAI_API_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: openai-secret
+                  key: apikey
             - name: MODEL_TEMPERATURE
               value: "0"
-            # For OpenAI (uncomment and set your key):
-            # - name: TASK_MODEL_ID
-            #   value: "gpt-4.1-nano"
-            # - name: LLM_API_KEY
-            #   valueFrom:
-            #     secretKeyRef:
-            #       name: openai-secret
-            #       key: apikey
-            # - name: OPENAI_API_KEY
-            #   valueFrom:
-            #     secretKeyRef:
-            #       name: openai-secret
-            #       key: apikey
 
             # Agent service settings
             # PORT tells the agent where to listen for A2A traffic.
@@ -206,6 +195,7 @@ metadata:
   namespace: team1
 spec:
   type: agent
+  mtlsMode: disabled
   targetRef:
     apiVersion: apps/v1
     kind: Deployment

aiac/demo/assets/tools/github_tool/k8s/github-tool-deployment.yaml

@@ -130,6 +130,7 @@ metadata:
   namespace: team1
 spec:
   type: tool
+  mtlsMode: disabled
   targetRef:
     apiVersion: apps/v1
     kind: Deployment

aiac/docs/specs/PRD.md

@@ -126,7 +126,7 @@ Nine components across five Kubernetes Pods plus a Python library layer, all imp
                │                      |
     (𝘶𝘴𝘦𝘳𝘴, 𝘳𝘰𝘭𝘦𝘴, 𝘤𝘭𝘪𝘦𝘯𝘵𝘴)    (𝘈𝘶𝘵𝘩𝘰𝘳𝘪𝘻𝘢𝘵𝘪𝘰𝘯𝘗𝘰𝘭𝘪𝘤𝘺 𝘊𝘙)
 ┌──────────────┼──────────────────────┼───────────────────┐
-│  Rossoctl Interface Pod              │                   │
+│  Rossoctl Interface Pod             │                   │
 │              │                      │                   │
 │      ┌───────┴──────┐      ┌────────┴───────┐           │
 │      │  IdP Config  │      │  PDP Policy    │           │
@@ -597,6 +597,8 @@ Beyond the marker-gated pytest tests above, individual integration tests are spe
 | PDP Policy Writer — `generate_rego.py` | Standalone launcher (no Docker) that boots the OPA stub locally, applies a `PolicyModel` through `aiac.pdp.policy.library`, and writes the generated Rego to a known directory for manual inspection. Write-only; not `@pytest.mark.integration`. | [integration-test/pdp-policy-writer.md](integration-test/pdp-policy-writer.md) |
 | `policy-pipeline` — `policy_pipeline.py` | Standalone launcher (no Docker) driving the full identity→policy pipeline — provisions a Keycloak realm + entities, runs the three PRB mappings, applies via the PCE, and writes the generated Rego to a known directory for manual inspection. Write-only; not `@pytest.mark.integration`. | [integration-test/policy-pipeline.md](integration-test/policy-pipeline.md) |
 | `uc1-onboarding-pipeline` — a **ladder** of UC-1 onboarding tests | Discovery-driven sibling of `policy-pipeline` validating the **phase-1** deliverable against **one** in-cluster AIAC stack (OPA filesystem-stub writer, single abstract `policy.md`): with `github-agent` + a simplified `github-tool` **already deployed and registered** as Keycloak clients, three gradual rungs drive **real UC-1 onboarding** (`POST /apply/service/{id}`) — agent-only, agent→tool, tool→agent — and assert the generated Rego with `opa eval` (verdicts from `scenario_uc1.py`). Rungs 2/3 assert onboarding-**order-independence**. A fourth two-policy rung is **deferred** (two-stack topology discarded). Same scenario facts/tables as `policy-pipeline`; Rego semantically similar (not byte-identical). `@pytest.mark.integration`. | [integration-test/uc1-onboarding-pipeline.md](integration-test/uc1-onboarding-pipeline.md) |
+| `policy-eval-scenarios` — `test_policy_pipeline_eval.py` + guardrail tests | Generalized evaluation suite extending `policy-pipeline`'s single-agent/single-tool proof to ten scenarios: baseline-scale (many entities, names decoupled from roles, one agent→agent delegation grant), missing-details (emergent unreachability/zero-access under deny-by-default, a broad-sounding clause narrowed by an explicit qualifier, wildcard-grant expansion), adversarial-authoring (misleading names/descriptions, an identity/boundary-confusion probe, empty descriptions), and ambiguous-and-contradictory / adversarial-injection-and-edge-cases (whole-document `xfail` checks against the PRB directly, no Keycloak or `opa`). The eight heavy scenarios (`@pytest.mark.eval_extended`, scenario modules under `eval/scenarios/` except `agent_delegation`) assert full per-cell `opa eval` truth tables; the two light scenarios (`@pytest.mark.integration`) assert PRB-level rejection. | [eval/policy-eval-scenarios.md](eval/policy-eval-scenarios.md) |
+| `policy-eval-robustness-consistency` — `test_policy_pipeline_consistency.py` + `test_policy_pipeline_robustness.py` | Companion to `policy-eval-scenarios`, reusing its 8-scenario corpus to check the PRB's raw grant decisions (no OPA/PCE/k8s) for **consistency** (`@pytest.mark.eval_consistency`: N repeated runs on the same input, exact grant-set equality) and **robustness** (`@pytest.mark.eval_robustness`: mechanical text/order perturbation + a hand-reworded semantic-sibling corpus under `eval/scenarios_perturbed/`, both checked against the truth-table oracle). No Keycloak/`opa` needed — only `LLM_BASE_URL`/`LLM_MODEL`/`LLM_API_KEY`. | [eval/policy-eval-robustness-consistency.md](eval/policy-eval-robustness-consistency.md) |
 
 Tracking issues: the live-Keycloak pytest integration tests in `testing/5.1-integration-tests.md`; the PDP Policy Writer integration test in `testing/5.2-pdp-writer-integration-test.md`; the policy-pipeline integration test in `testing/5.3-policy-pipeline-integration-test.md`; the UC-1 onboarding pipeline integration-test ladder in `testing/5.4-uc1-onboarding-integration-test.md` (epic) with rungs `testing/5.4.1`/`5.4.2`/`5.4.3` and the deferred two-policy `testing/5.4.4`.