From dad32a2c6b537eb316c0b3dfa549b55e2a26638c Mon Sep 17 00:00:00 2001 From: BohdanMar Date: Wed, 26 Aug 2026 14:04:28 +0200 Subject: [PATCH 1/8] docs: add guide for catching policy violations early with ITs Guide users in setting up Integration Tests that surface release-time policy violations earlier in the development workflow, based on the findings from EC-1904. --- .../ROOT/pages/early-policy-violations.adoc | 199 ++++++++++++++++++ modules/ROOT/partials/contents.adoc | 1 + 2 files changed, 200 insertions(+) create mode 100644 modules/ROOT/pages/early-policy-violations.adoc diff --git a/modules/ROOT/pages/early-policy-violations.adoc b/modules/ROOT/pages/early-policy-violations.adoc new file mode 100644 index 00000000..2092ea93 --- /dev/null +++ b/modules/ROOT/pages/early-policy-violations.adoc @@ -0,0 +1,199 @@ += Catching policy violations early with Integration Tests + +By default, the Conforma integration test in Konflux runs with +`pipeline_intention` set to `staging`. This means some policy rules that are +enforced at release time are skipped during integration testing. As a result, +you may only discover certain violations when you attempt a release. + +This guide explains which rules are skipped, and how to configure an +additional integration test that surfaces release-time violations earlier in +your development workflow. + +== What is checked at each stage + +Policy rules use the `pipeline_intention` parameter to determine when they +should run. The default integration test uses `staging`, while the release +pipeline uses `release`. + +.Rules by pipeline_intention +[cols="3,1,1,1",options="header"] +|=== +| Rule | staging | release | production + +| Most policy rules (signatures, provenance, trusted tasks, etc.) +| Yes +| Yes +| Yes + +| `quay_expiration.expires_label` +| Yes +| Yes +| Yes + +| `olm.unpinned_snapshot_references` +| Yes +| Yes +| Yes + +| `olm.unpinned_related_images` +| Yes +| Yes +| Yes + +| `olm.inaccessible_related_images` +| Yes +| Yes +| Yes + +| `olm.unmapped_references` +| Yes +| Yes +| Yes + +| `schedule.weekday_restriction` +| No +| Yes +| Yes + +| `schedule.date_restriction` +| No +| Yes +| Yes +|=== + +The `schedule` rules are intentionally release-only -- they restrict _when_ a +release can happen, which is not relevant during integration testing. + +The majority of policy rules, including signature verification, provenance +checks, and trusted task validation, run at both `staging` and `release`. +This means the default integration test already catches most violations. + +== Setting up a release-aligned integration test + +If you want the closest possible match to release-time checks during +integration testing, you can create an additional `IntegrationTestScenario` +that uses the same `EnterpriseContractPolicy` as your release pipeline. + +=== Step 1: Find your release policy configuration + +Your release policy is defined in the `ReleasePlanAdmission` in your managed +namespace. Ask your release engineering or SRE team for the +`EnterpriseContractPolicy` (ECP) name or configuration used in your release +pipeline. + +=== Step 2: Create a non-blocking integration test + +Create a new `IntegrationTestScenario` that references the same policy +configuration as your release pipeline. Setting `STRICT` to `false` makes +this test informational -- it reports violations without blocking your builds. + +include::partial$oc_login.adoc[] + +Create a file called `release-check-its.yaml`: + +[,yaml] +---- +apiVersion: appstudio.redhat.com/v1alpha1 +kind: IntegrationTestScenario +metadata: + name: release-policy-check +spec: + application: + resolverRef: + resolver: git + params: + - name: url + value: https://github.com/conforma/tekton-catalog + - name: revision + value: main + - name: pathInRepo + value: pipelines/enterprise-contract/0.1/enterprise-contract.yaml + params: + - name: POLICY_CONFIGURATION + value: + - name: STRICT + value: "false" +---- + +Replace `` with your application name, and +`` with the ECP used by your release pipeline (either a +CR name like `enterprise-contract-service/default` or a git URL). + +Apply it to your namespace: + +[,shell] +---- +$ oc create -f release-check-its.yaml +---- + +=== Step 3: Review results + +After your next build completes, the integration test runs and reports any +policy violations that would occur at release time. Because `STRICT` is set +to `false`, the test always passes, but the results show which rules would +fail. + +You can view the results in the Konflux UI under your application's +integration tests, or inspect the pipeline run directly: + +[,shell,subs="+quotes"] +---- +$ oc get pipelinerun --selector test.appstudio.openshift.io/scenario=release-policy-check --sort-by='.status.startTime' -o name | tail -1 | xargs oc logs -c step-report +---- + +== Considerations + +=== Schedule rules will always fail + +The `schedule.weekday_restriction` and `schedule.date_restriction` rules +check whether a release is happening on a permitted day or date. These rules +will likely report violations during integration testing since they are +evaluated against the current time rather than a planned release time. This +is expected and can be safely ignored in the integration test results. + +=== OLM rules may not pass until release-ready + +For OLM (Operator Lifecycle Manager) operators, the following rules may +report violations during integration testing that resolve themselves closer +to release time: + +* `olm.unpinned_related_images` -- related images may not be pinned until the + release process pins them. +* `olm.inaccessible_related_images` -- images may not be published to their + final registry location until release. +* `olm.unmapped_references` -- similar to the above, references may not be + fully mapped until release. + +These are informational during integration testing. If they consistently fail, +it may indicate an issue worth investigating. + +=== Keeping policies in sync + +If the release ECP is updated, your integration test will automatically pick +up the changes (assuming you reference the same ECP). This ensures your +early checks stay aligned with what the release pipeline enforces. + +== Making the test blocking + +Once you are confident that your integration test results are clean, you can +make the test blocking by changing `STRICT` to `true`: + +[,shell,subs="+quotes"] +---- +$ oc edit integrationtestscenario release-policy-check +---- + +Change the `STRICT` parameter: + +[,yaml] +---- + - name: STRICT + value: "true" +---- + +With `STRICT` set to `true`, any policy violation will cause the integration +test to fail, preventing the snapshot from being released. + +NOTE: If you make this test blocking, be aware that `schedule` rule violations +will block your builds even when the underlying code is fine. Consider +excluding those rules in your integration test ECP if you enable blocking mode. diff --git a/modules/ROOT/partials/contents.adoc b/modules/ROOT/partials/contents.adoc index 30628b09..fb7e7c3b 100644 --- a/modules/ROOT/partials/contents.adoc +++ b/modules/ROOT/partials/contents.adoc @@ -8,6 +8,7 @@ ** xref:reproducing-a-konflux-conforma-report.adoc[Reproducing a Konflux Conforma report locally] ** xref:custom-config.adoc[Using custom configuration] ** xref:custom-data.adoc[Using custom data] +** xref:early-policy-violations.adoc[Catching policy violations early with Integration Tests] ** xref:hitchhikers-guide.adoc[Hitchhiker's Guide to Conforma] * xref:slsa.adoc[Conforma & SLSA] From 2b074fccdf9ce9f9b24e9b92e5ab68a42993b463 Mon Sep 17 00:00:00 2001 From: BohdanMar Date: Thu, 27 Aug 2026 15:10:36 +0200 Subject: [PATCH 2/8] docs: emphasize POLICY_CONFIGURATION and multiple ITS usage Update based on review feedback: highlight POLICY_CONFIGURATION as the key parameter, add guidance on multiple ITS with different policies, fix apiVersion to v1beta2, and add real-world example values. --- .../ROOT/pages/early-policy-violations.adoc | 26 ++++++++++++------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/modules/ROOT/pages/early-policy-violations.adoc b/modules/ROOT/pages/early-policy-violations.adoc index 2092ea93..164721af 100644 --- a/modules/ROOT/pages/early-policy-violations.adoc +++ b/modules/ROOT/pages/early-policy-violations.adoc @@ -68,18 +68,26 @@ The majority of policy rules, including signature verification, provenance checks, and trusted task validation, run at both `staging` and `release`. This means the default integration test already catches most violations. -== Setting up a release-aligned integration test +== Using POLICY_CONFIGURATION to match your release policy -If you want the closest possible match to release-time checks during -integration testing, you can create an additional `IntegrationTestScenario` -that uses the same `EnterpriseContractPolicy` as your release pipeline. +The key to catching release-time violations early is the +`POLICY_CONFIGURATION` parameter. The default integration test uses a +generic policy, but your release pipeline likely uses a specific +`EnterpriseContractPolicy` (ECP) tailored to your product. By creating an +additional `IntegrationTestScenario` that references the same ECP as your +release pipeline, you can surface violations before you attempt a release. + +You can have multiple enterprise-contract integration tests, each with a +different `POLICY_CONFIGURATION` value. For example, one for basic validation +and another matching your release policy. === Step 1: Find your release policy configuration Your release policy is defined in the `ReleasePlanAdmission` in your managed namespace. Ask your release engineering or SRE team for the `EnterpriseContractPolicy` (ECP) name or configuration used in your release -pipeline. +pipeline. The value is typically in the format `namespace/name`, for example +`rhtap-releng-tenant/registry-rhtap-contract`. === Step 2: Create a non-blocking integration test @@ -93,7 +101,7 @@ Create a file called `release-check-its.yaml`: [,yaml] ---- -apiVersion: appstudio.redhat.com/v1alpha1 +apiVersion: appstudio.redhat.com/v1beta2 kind: IntegrationTestScenario metadata: name: release-policy-check @@ -110,14 +118,14 @@ spec: value: pipelines/enterprise-contract/0.1/enterprise-contract.yaml params: - name: POLICY_CONFIGURATION - value: + value: / - name: STRICT value: "false" ---- Replace `` with your application name, and -`` with the ECP used by your release pipeline (either a -CR name like `enterprise-contract-service/default` or a git URL). +`/` with the ECP used by your release pipeline +(for example, `rhtap-releng-tenant/registry-rhtap-contract`). Apply it to your namespace: From 30b6c5b43faa4c3b6a4a58b4f873bdd0bce439a3 Mon Sep 17 00:00:00 2001 From: BohdanMar Date: Thu, 27 Aug 2026 15:11:16 +0200 Subject: [PATCH 3/8] docs: add git URL format for POLICY_CONFIGURATION Document both cluster reference (namespace/name) and git URL formats for specifying the ECP, so teams can manage policies in-cluster or in version control. --- modules/ROOT/pages/early-policy-violations.adoc | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/modules/ROOT/pages/early-policy-violations.adoc b/modules/ROOT/pages/early-policy-violations.adoc index 164721af..37b5fcf8 100644 --- a/modules/ROOT/pages/early-policy-violations.adoc +++ b/modules/ROOT/pages/early-policy-violations.adoc @@ -123,9 +123,20 @@ spec: value: "false" ---- -Replace `` with your application name, and -`/` with the ECP used by your release pipeline -(for example, `rhtap-releng-tenant/registry-rhtap-contract`). +Replace `` with your application name, and set +`POLICY_CONFIGURATION` to the ECP used by your release pipeline. The value +can be specified in two ways: + +* **Cluster reference** -- `namespace/name` format pointing to an + `EnterpriseContractPolicy` CR in the cluster, for example + `rhtap-releng-tenant/registry-rhtap-contract`. +* **Git URL** -- `git::github.com/org/repo//path/?ref=branchorsha` format + pointing to a `policy.yaml` (or `policy.json`) file in a git repository. + This lets teams manage their ECP in version control without creating + cluster resources. + +Teams can choose the approach that fits their workflow -- create ECP records +in their own tenant namespace, or point to a policy file in git. Apply it to your namespace: From 936d8b104ece88e7f3f84b686322df042257953a Mon Sep 17 00:00:00 2001 From: BohdanMar Date: Mon, 31 Aug 2026 10:40:12 +0200 Subject: [PATCH 4/8] docs: address review feedback - Remove unexplained production column from table - Soften "will always fail" to "will likely fail" - Fix nav entry to sentence case --- modules/ROOT/pages/early-policy-violations.adoc | 14 +++----------- modules/ROOT/partials/contents.adoc | 2 +- 2 files changed, 4 insertions(+), 12 deletions(-) diff --git a/modules/ROOT/pages/early-policy-violations.adoc b/modules/ROOT/pages/early-policy-violations.adoc index 37b5fcf8..609fff8e 100644 --- a/modules/ROOT/pages/early-policy-violations.adoc +++ b/modules/ROOT/pages/early-policy-violations.adoc @@ -16,49 +16,41 @@ should run. The default integration test uses `staging`, while the release pipeline uses `release`. .Rules by pipeline_intention -[cols="3,1,1,1",options="header"] +[cols="3,1,1",options="header"] |=== -| Rule | staging | release | production +| Rule | staging | release | Most policy rules (signatures, provenance, trusted tasks, etc.) | Yes | Yes -| Yes | `quay_expiration.expires_label` | Yes | Yes -| Yes | `olm.unpinned_snapshot_references` | Yes | Yes -| Yes | `olm.unpinned_related_images` | Yes | Yes -| Yes | `olm.inaccessible_related_images` | Yes | Yes -| Yes | `olm.unmapped_references` | Yes | Yes -| Yes | `schedule.weekday_restriction` | No | Yes -| Yes | `schedule.date_restriction` | No | Yes -| Yes |=== The `schedule` rules are intentionally release-only -- they restrict _when_ a @@ -162,7 +154,7 @@ $ oc get pipelinerun --selector test.appstudio.openshift.io/scenario=release-pol == Considerations -=== Schedule rules will always fail +=== Schedule rules will likely fail The `schedule.weekday_restriction` and `schedule.date_restriction` rules check whether a release is happening on a permitted day or date. These rules diff --git a/modules/ROOT/partials/contents.adoc b/modules/ROOT/partials/contents.adoc index fb7e7c3b..9cb85f20 100644 --- a/modules/ROOT/partials/contents.adoc +++ b/modules/ROOT/partials/contents.adoc @@ -8,7 +8,7 @@ ** xref:reproducing-a-konflux-conforma-report.adoc[Reproducing a Konflux Conforma report locally] ** xref:custom-config.adoc[Using custom configuration] ** xref:custom-data.adoc[Using custom data] -** xref:early-policy-violations.adoc[Catching policy violations early with Integration Tests] +** xref:early-policy-violations.adoc[Catching policy violations early with integration tests] ** xref:hitchhikers-guide.adoc[Hitchhiker's Guide to Conforma] * xref:slsa.adoc[Conforma & SLSA] From bcdd375bc482d71dd75add3b33ca9f33e639360b Mon Sep 17 00:00:00 2001 From: BohdanMar Date: Mon, 31 Aug 2026 14:35:27 +0200 Subject: [PATCH 5/8] docs: fix schedule rules contradiction and placeholder formatting MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Schedule rules are skipped at staging, not failed — fix the contradiction between the table and the caveats section - Use subs="+quotes" and __placeholder__ formatting to match custom-config.adoc conventions - Fix title casing to match nav entry --- .../ROOT/pages/early-policy-violations.adoc | 23 +++++++++---------- 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/modules/ROOT/pages/early-policy-violations.adoc b/modules/ROOT/pages/early-policy-violations.adoc index 609fff8e..a05475f9 100644 --- a/modules/ROOT/pages/early-policy-violations.adoc +++ b/modules/ROOT/pages/early-policy-violations.adoc @@ -1,4 +1,4 @@ -= Catching policy violations early with Integration Tests += Catching policy violations early with integration tests By default, the Conforma integration test in Konflux runs with `pipeline_intention` set to `staging`. This means some policy rules that are @@ -91,14 +91,14 @@ include::partial$oc_login.adoc[] Create a file called `release-check-its.yaml`: -[,yaml] +[,yaml,subs="+quotes"] ---- apiVersion: appstudio.redhat.com/v1beta2 kind: IntegrationTestScenario metadata: name: release-policy-check spec: - application: + application: ____ resolverRef: resolver: git params: @@ -110,7 +110,7 @@ spec: value: pipelines/enterprise-contract/0.1/enterprise-contract.yaml params: - name: POLICY_CONFIGURATION - value: / + value: __/__ - name: STRICT value: "false" ---- @@ -154,13 +154,12 @@ $ oc get pipelinerun --selector test.appstudio.openshift.io/scenario=release-pol == Considerations -=== Schedule rules will likely fail +=== Schedule rules are skipped The `schedule.weekday_restriction` and `schedule.date_restriction` rules -check whether a release is happening on a permitted day or date. These rules -will likely report violations during integration testing since they are -evaluated against the current time rather than a planned release time. This -is expected and can be safely ignored in the integration test results. +only run when `pipeline_intention` is set to `release`. Since integration +tests use `staging`, these rules are automatically skipped and will not +appear in your results. === OLM rules may not pass until release-ready @@ -205,6 +204,6 @@ Change the `STRICT` parameter: With `STRICT` set to `true`, any policy violation will cause the integration test to fail, preventing the snapshot from being released. -NOTE: If you make this test blocking, be aware that `schedule` rule violations -will block your builds even when the underlying code is fine. Consider -excluding those rules in your integration test ECP if you enable blocking mode. +NOTE: The `schedule` rules are skipped since the integration test uses +`pipeline_intention: staging`. Only rules that run at `staging` can cause +failures in blocking mode. From d52adc0cca06c53ba4df91ee1db6d0a8fd2fd696 Mon Sep 17 00:00:00 2001 From: BohdanMar Date: Mon, 31 Aug 2026 14:57:09 +0200 Subject: [PATCH 6/8] docs: fix review findings from second pass - Add olm.unpinned_snapshot_references to OLM considerations - Fix overclaim about test always passing with STRICT=false - Add namespace reminder before oc create command --- modules/ROOT/pages/early-policy-violations.adoc | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/modules/ROOT/pages/early-policy-violations.adoc b/modules/ROOT/pages/early-policy-violations.adoc index a05475f9..b28580ac 100644 --- a/modules/ROOT/pages/early-policy-violations.adoc +++ b/modules/ROOT/pages/early-policy-violations.adoc @@ -132,6 +132,8 @@ in their own tenant namespace, or point to a policy file in git. Apply it to your namespace: +Ensure you are in the correct namespace, then apply: + [,shell] ---- $ oc create -f release-check-its.yaml @@ -141,8 +143,8 @@ $ oc create -f release-check-its.yaml After your next build completes, the integration test runs and reports any policy violations that would occur at release time. Because `STRICT` is set -to `false`, the test always passes, but the results show which rules would -fail. +to `false`, policy violations do not cause the test to fail. The results +still show which rules would have failed. You can view the results in the Konflux UI under your application's integration tests, or inspect the pipeline run directly: @@ -167,6 +169,8 @@ For OLM (Operator Lifecycle Manager) operators, the following rules may report violations during integration testing that resolve themselves closer to release time: +* `olm.unpinned_snapshot_references` -- snapshot references may not be pinned + until the release process pins them. * `olm.unpinned_related_images` -- related images may not be pinned until the release process pins them. * `olm.inaccessible_related_images` -- images may not be published to their From da18d6255b0d6b2ddda4171200acc0407b54a2fd Mon Sep 17 00:00:00 2001 From: BohdanMar Date: Tue, 1 Sep 2026 10:50:23 +0200 Subject: [PATCH 7/8] docs: fix log viewing command to use taskrun/pod pattern Use the taskrun -> pod -> oc logs pattern matching the existing reproducing-a-konflux-conforma-report.adoc guide. --- modules/ROOT/pages/early-policy-violations.adoc | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/modules/ROOT/pages/early-policy-violations.adoc b/modules/ROOT/pages/early-policy-violations.adoc index b28580ac..de9d6ce4 100644 --- a/modules/ROOT/pages/early-policy-violations.adoc +++ b/modules/ROOT/pages/early-policy-violations.adoc @@ -147,11 +147,13 @@ to `false`, policy violations do not cause the test to fail. The results still show which rules would have failed. You can view the results in the Konflux UI under your application's -integration tests, or inspect the pipeline run directly: +integration tests, or inspect the task run logs directly: -[,shell,subs="+quotes"] +[,shell] ---- -$ oc get pipelinerun --selector test.appstudio.openshift.io/scenario=release-policy-check --sort-by='.status.startTime' -o name | tail -1 | xargs oc logs -c step-report +TR_NAME=$( oc get taskrun --selector tekton.dev/task=verify-enterprise-contract,test.appstudio.openshift.io/scenario=release-policy-check --sort-by='.status.startTime' -o name | tail -1 ) +POD_NAME=$( oc get $TR_NAME -o jsonpath='{.status.podName}' ) +oc logs -c step-report $POD_NAME ---- == Considerations From c4f61148dca0be6b109090844a6b53f74762691c Mon Sep 17 00:00:00 2001 From: BohdanMar Date: Tue, 1 Sep 2026 10:54:30 +0200 Subject: [PATCH 8/8] docs: update custom-config.adoc to v1beta2 and remove stale workaround The v1alpha1 API version and the v1beta1-to-v1alpha1 workaround are outdated. Production Konflux clusters use v1beta2 for IntegrationTestScenario resources. --- modules/ROOT/pages/custom-config.adoc | 18 +----------------- 1 file changed, 1 insertion(+), 17 deletions(-) diff --git a/modules/ROOT/pages/custom-config.adoc b/modules/ROOT/pages/custom-config.adoc index 6ecc309c..fc9ee1b2 100644 --- a/modules/ROOT/pages/custom-config.adoc +++ b/modules/ROOT/pages/custom-config.adoc @@ -39,15 +39,6 @@ spec: The `params` key should be at the same indent level as `application` and `contexts` underneath the `spec` key. -Before saving the YAML file, to workaround a problem with the API versions -present in the cluster, you also need to also modify the API version to change -`v1beta1` to `v1alpha1` in the first line of the YAML file. - -[,yaml] ----- -apiVersion: appstudio.redhat.com/v1alpha1 ----- - Save the file to update the CR in the cluster. NOTE: The config file specified in the above example is @@ -75,7 +66,7 @@ Create a yaml file called `policy.yaml` with the following content: [,yaml] ---- -apiVersion: appstudio.redhat.com/v1alpha1 +apiVersion: appstudio.redhat.com/v1beta2 kind: EnterpriseContractPolicy metadata: name: ec-policy @@ -127,13 +118,6 @@ spec: … ---- -Once again the API version workaround is needed, so modify the `apiVersion` value. - -[,yaml] ----- -apiVersion: appstudio.redhat.com/v1alpha1 ----- - Save the YAML file to update the IntegrationTestScenario CR with the new policy configuration parameter value.