docs: guide for catching policy violations early with ITs #246
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,215 @@ | ||
| = Catching policy violations early with integration tests | ||
|
|
||
| By default, the Conforma integration test in Konflux runs with | ||
| `pipeline_intention` set to `staging`. This means some policy rules that are | ||
| enforced at release time are skipped during integration testing. As a result, | ||
| you may only discover certain violations when you attempt a release. | ||
|
|
||
| This guide explains which rules are skipped, and how to configure an | ||
| additional integration test that surfaces release-time violations earlier in | ||
| your development workflow. | ||
|
|
||
| == What is checked at each stage | ||
|
|
||
| Policy rules use the `pipeline_intention` parameter to determine when they | ||
| should run. The default integration test uses `staging`, while the release | ||
| pipeline uses `release`. | ||
|
|
||
| .Rules by pipeline_intention | ||
| [cols="3,1,1",options="header"] | ||
| |=== | ||
| | Rule | staging | release | ||
|
|
||
| | Most policy rules (signatures, provenance, trusted tasks, etc.) | ||
| | Yes | ||
| | Yes | ||
|
|
||
| | `quay_expiration.expires_label` | ||
| | Yes | ||
| | Yes | ||
|
|
||
|
There was a problem hiding this comment. [low] edge-case The rules table lists olm.unpinned_snapshot_references as running at both staging and release (Yes/Yes), but the Considerations section discusses three OLM rules that may not pass until release-ready and omits olm.unpinned_snapshot_references. If this rule has similar release-timing caveats, it should be mentioned in the considerations. |
||
| | `olm.unpinned_snapshot_references` | ||
| | Yes | ||
| | Yes | ||
|
|
||
| | `olm.unpinned_related_images` | ||
| | Yes | ||
| | Yes | ||
|
|
||
| | `olm.inaccessible_related_images` | ||
| | Yes | ||
| | Yes | ||
|
|
||
| | `olm.unmapped_references` | ||
| | Yes | ||
| | Yes | ||
|
|
||
| | `schedule.weekday_restriction` | ||
| | No | ||
| | Yes | ||
|
|
||
| | `schedule.date_restriction` | ||
| | No | ||
| | Yes | ||
| |=== | ||
|
|
||
| The `schedule` rules are intentionally release-only -- they restrict _when_ a | ||
|
There was a problem hiding this comment. [low] section-heading-style Existing how-to guides do not use numbered step headings (=== Step 1: ...). They use descriptive section headings instead. |
||
| release can happen, which is not relevant during integration testing. | ||
|
|
||
| The majority of policy rules, including signature verification, provenance | ||
| checks, and trusted task validation, run at both `staging` and `release`. | ||
| This means the default integration test already catches most violations. | ||
|
|
||
| == Using POLICY_CONFIGURATION to match your release policy | ||
|
|
||
| The key to catching release-time violations early is the | ||
| `POLICY_CONFIGURATION` parameter. The default integration test uses a | ||
| generic policy, but your release pipeline likely uses a specific | ||
| `EnterpriseContractPolicy` (ECP) tailored to your product. By creating an | ||
| additional `IntegrationTestScenario` that references the same ECP as your | ||
| release pipeline, you can surface violations before you attempt a release. | ||
|
|
||
| You can have multiple enterprise-contract integration tests, each with a | ||
| different `POLICY_CONFIGURATION` value. For example, one for basic validation | ||
| and another matching your release policy. | ||
|
|
||
| === Step 1: Find your release policy configuration | ||
|
|
||
| Your release policy is defined in the `ReleasePlanAdmission` in your managed | ||
| namespace. Ask your release engineering or SRE team for the | ||
| `EnterpriseContractPolicy` (ECP) name or configuration used in your release | ||
| pipeline. The value is typically in the format `namespace/name`, for example | ||
| `rhtap-releng-tenant/registry-rhtap-contract`. | ||
|
|
||
| === Step 2: Create a non-blocking integration test | ||
|
There was a problem hiding this comment. [high] cross-document-consistency The IntegrationTestScenario YAML uses apiVersion: appstudio.redhat.com/v1beta2, while custom-config.adoc uses apiVersion: appstudio.redhat.com/v1alpha1 for the same resource kind (lines 48, 78, 134) and includes an explicit workaround note (lines 42-44) about changing v1beta1 to v1alpha1. Users following both guides encounter contradictory API versions. Suggested fix: Confirm which API version is currently correct for IntegrationTestScenario in the target clusters. Update both documents to use the same version. If v1beta2 is correct, update custom-config.adoc and remove the v1beta1-to-v1alpha1 workaround. If v1alpha1 is still required, update the new page's YAML example. There was a problem hiding this comment. [medium] placeholder-convention Placeholders like and / are bare angle-bracket text inside a YAML block using [,yaml] without subs='+quotes'. The established pattern in custom-config.adoc uses [,yaml,subs="+quotes"] with formatting to visually distinguish replaceable values. Suggested fix: Change the YAML block to use [,yaml,subs='+quotes'] and format placeholders as , /. |
||
|
|
||
| Create a new `IntegrationTestScenario` that references the same policy | ||
| configuration as your release pipeline. Setting `STRICT` to `false` makes | ||
| this test informational -- it reports violations without blocking your builds. | ||
|
|
||
|
There was a problem hiding this comment. [low] edge-case The YAML example for IntegrationTestScenario does not include a namespace in metadata, and the oc create command lacks a -n flag. Since the doc says 'Apply it to your namespace,' users should be reminded to verify their namespace context.
BohdanMar marked this conversation as resolved.
|
||
| include::partial$oc_login.adoc[] | ||
|
|
||
| Create a file called `release-check-its.yaml`: | ||
|
There was a problem hiding this comment. [medium] api-contract The YAML example uses apiVersion: appstudio.redhat.com/v1beta2 for IntegrationTestScenario, while the existing documentation in custom-config.adoc uses v1alpha1 with a documented workaround to downgrade from v1beta1. These two documents now contradict each other on the correct API version for the same resource kind. Suggested fix: Verify the currently correct API version for IntegrationTestScenario and ensure both documents agree. |
||
|
|
||
| [,yaml,subs="+quotes"] | ||
| ---- | ||
| apiVersion: appstudio.redhat.com/v1beta2 | ||
| kind: IntegrationTestScenario | ||
| metadata: | ||
| name: release-policy-check | ||
| spec: | ||
| application: __<your-application-name>__ | ||
| resolverRef: | ||
| resolver: git | ||
| params: | ||
| - name: url | ||
| value: https://github.com/conforma/tekton-catalog | ||
| - name: revision | ||
| value: main | ||
| - name: pathInRepo | ||
| value: pipelines/enterprise-contract/0.1/enterprise-contract.yaml | ||
| params: | ||
| - name: POLICY_CONFIGURATION | ||
| value: __<managed-namespace>/<ecp-name>__ | ||
| - name: STRICT | ||
| value: "false" | ||
| ---- | ||
|
|
||
| Replace `<your-application-name>` with your application name, and set | ||
| `POLICY_CONFIGURATION` to the ECP used by your release pipeline. The value | ||
| can be specified in two ways: | ||
|
|
||
| * **Cluster reference** -- `namespace/name` format pointing to an | ||
| `EnterpriseContractPolicy` CR in the cluster, for example | ||
| `rhtap-releng-tenant/registry-rhtap-contract`. | ||
| * **Git URL** -- `git::github.com/org/repo//path/?ref=branchorsha` format | ||
| pointing to a `policy.yaml` (or `policy.json`) file in a git repository. | ||
| This lets teams manage their ECP in version control without creating | ||
| cluster resources. | ||
|
|
||
| Teams can choose the approach that fits their workflow -- create ECP records | ||
| in their own tenant namespace, or point to a policy file in git. | ||
|
|
||
| Apply it to your namespace: | ||
|
|
||
| Ensure you are in the correct namespace, then apply: | ||
|
|
||
| [,shell] | ||
| ---- | ||
| $ oc create -f release-check-its.yaml | ||
|
There was a problem hiding this comment. [low] edge-case The document states 'the test always passes' with STRICT=false. This is technically inaccurate -- STRICT=false means policy violations do not cause failure, but infrastructure errors could still fail the test. |
||
| ---- | ||
|
|
||
| === Step 3: Review results | ||
|
|
||
| After your next build completes, the integration test runs and reports any | ||
| policy violations that would occur at release time. Because `STRICT` is set | ||
| to `false`, policy violations do not cause the test to fail. The results | ||
| still show which rules would have failed. | ||
|
|
||
|
BohdanMar marked this conversation as resolved.
|
||
| You can view the results in the Konflux UI under your application's | ||
| integration tests, or inspect the task run logs directly: | ||
|
|
||
| [,shell] | ||
| ---- | ||
| TR_NAME=$( oc get taskrun --selector tekton.dev/task=verify-enterprise-contract,test.appstudio.openshift.io/scenario=release-policy-check --sort-by='.status.startTime' -o name | tail -1 ) | ||
| POD_NAME=$( oc get $TR_NAME -o jsonpath='{.status.podName}' ) | ||
| oc logs -c step-report $POD_NAME | ||
| ---- | ||
|
There was a problem hiding this comment. [medium] internal-consistency The 'Schedule rules will likely fail' section and the blocking-mode NOTE (line 210) warn that schedule rules will report violations during integration testing. However, the table at lines 47-49 states that schedule.weekday_restriction and schedule.date_restriction do NOT run at staging pipeline_intention. The guide does not set pipeline_intention to release in the new IntegrationTestScenario YAML. If pipeline_intention defaults to staging, schedule rules would be skipped entirely, contradicting the warning. Suggested fix: Resolve the contradiction: (1) add pipeline_intention=release to the YAML example, (2) revise the schedule-rules caveat if pipeline_intention stays staging, or (3) explain how the ECP overrides pipeline_intention behavior. |
||
|
|
||
| == Considerations | ||
|
|
||
| === Schedule rules are skipped | ||
|
|
||
| The `schedule.weekday_restriction` and `schedule.date_restriction` rules | ||
| only run when `pipeline_intention` is set to `release`. Since integration | ||
| tests use `staging`, these rules are automatically skipped and will not | ||
| appear in your results. | ||
|
|
||
| === OLM rules may not pass until release-ready | ||
|
|
||
| For OLM (Operator Lifecycle Manager) operators, the following rules may | ||
| report violations during integration testing that resolve themselves closer | ||
| to release time: | ||
|
|
||
| * `olm.unpinned_snapshot_references` -- snapshot references may not be pinned | ||
| until the release process pins them. | ||
| * `olm.unpinned_related_images` -- related images may not be pinned until the | ||
| release process pins them. | ||
| * `olm.inaccessible_related_images` -- images may not be published to their | ||
| final registry location until release. | ||
| * `olm.unmapped_references` -- similar to the above, references may not be | ||
| fully mapped until release. | ||
|
|
||
| These are informational during integration testing. If they consistently fail, | ||
| it may indicate an issue worth investigating. | ||
|
|
||
| === Keeping policies in sync | ||
|
|
||
| If the release ECP is updated, your integration test will automatically pick | ||
| up the changes (assuming you reference the same ECP). This ensures your | ||
| early checks stay aligned with what the release pipeline enforces. | ||
|
|
||
| == Making the test blocking | ||
|
|
||
| Once you are confident that your integration test results are clean, you can | ||
| make the test blocking by changing `STRICT` to `true`: | ||
|
|
||
| [,shell,subs="+quotes"] | ||
| ---- | ||
| $ oc edit integrationtestscenario release-policy-check | ||
| ---- | ||
|
|
||
| Change the `STRICT` parameter: | ||
|
|
||
| [,yaml] | ||
| ---- | ||
| - name: STRICT | ||
| value: "true" | ||
| ---- | ||
|
|
||
| With `STRICT` set to `true`, any policy violation will cause the integration | ||
|
There was a problem hiding this comment. [low] internal-consistency The 'Making the test blocking' section advises 'Consider excluding those rules in your integration test ECP if you enable blocking mode.' This creates tension with the core recommendation to reference the same ECP as the release pipeline, and with the 'Keeping policies in sync' section which touts automatic sync from using the same ECP. |
||
| test to fail, preventing the snapshot from being released. | ||
|
|
||
| NOTE: The `schedule` rules are skipped since the integration test uses | ||
| `pipeline_intention: staging`. Only rules that run at `staging` can cause | ||
| failures in blocking mode. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[medium] internal-consistency
The table includes a production column, but the document text only discusses staging and release pipeline_intention values. The production column is never defined or explained anywhere in the prose, leaving the reader without context for what production means, when it is used, or how it differs from release.
Suggested fix: Either add a brief explanation of the production pipeline_intention value in the prose, or remove the production column if it is not relevant to this guide's scope.