docs: guide for catching policy violations early with ITs

[BohdanMar]

Based on the findings from EC-1904, this adds a how-to guide explaining how to
set up an integration test that surfaces release-time policy violations earlier.

Covers:

• Which rules run at staging vs release (only schedule rules are skipped)
• Step-by-step setup of a non-blocking ITS using the same ECP as the release pipeline
• Caveats (schedule rules, OLM rules)
• Option to make it blocking once clean

Fixes: https://redhat.atlassian.net/browse/EC-2007

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: b511bc8a-73d5-45c8-85d9-3d1c047dc30d

---

_{Comment @coderabbitai help to get the list of available commands.}

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 1:08 PM UTC · Ended 1:11 PM UTC

Commit: 87c4a29 · View workflow run →

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 1:12 PM UTC · Completed 1:29 PM UTC

Commit: 87c4a29 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $3.38

[fullsend-ai-review]

Review

Findings

Low

• [scope-creep] modules/ROOT/pages/custom-config.adoc — The changes to custom-config.adoc remove obsolete v1beta1-to-v1alpha1 API version workaround blocks and update the apiVersion from v1alpha1 to v1beta2. While not explicitly mentioned in the PR description, these changes are directly related to the PR's purpose: the new guide uses apiVersion: appstudio.redhat.com/v1beta2 throughout, so the existing page must be updated for consistency.

Previous run

Review

Findings

Medium

• [internal consistency] modules/ROOT/pages/early-policy-violations.adoc:89 — The YAML example uses apiVersion: appstudio.redhat.com/v1beta2 for the IntegrationTestScenario resource, but the existing custom-config.adoc instructs users to use apiVersion: appstudio.redhat.com/v1alpha1 as a workaround and references v1beta1 as the default. Users following both guides encounter three different API versions (v1alpha1, v1beta1, v1beta2) with no explanation of which is correct or whether the workaround from the older doc still applies. If v1beta2 is now the correct version, the older doc should be updated; if v1alpha1 is still required as a workaround, this new guide's YAML may fail when applied.
  Remediation: Either (a) confirm that v1beta2 is the current correct API version and update custom-config.adoc to remove the v1alpha1 workaround, or (b) use the same API version as the existing docs and note the workaround. At minimum, add a note explaining the API version choice.

Low

• [edge case / runtime behavior] modules/ROOT/pages/early-policy-violations.adoc:148 — The command oc get pipelinerun ... -o name | tail -1 | xargs oc logs -c step-report pipes a PipelineRun resource name to oc logs. In standard Kubernetes, kubectl logs only operates on Pods. OpenShift Pipelines (Tekton) may extend oc to support logging PipelineRun resources on Konflux clusters, but this behavior is not universal and may surprise users on environments without those extensions.
  Remediation: Verify that oc logs works with PipelineRun resource names on target Konflux clusters. If not, either switch to tkn pipelinerun logs or add an intermediate step to resolve the pod name from the PipelineRun.

Previous run (2)

Review

Findings

High

• [stale-api-version] modules/ROOT/pages/custom-config.adoc:48 — The new document uses apiVersion: appstudio.redhat.com/v1beta2 for IntegrationTestScenario, but custom-config.adoc instructs users to set apiVersion: appstudio.redhat.com/v1alpha1 (lines 48, 78, 134) and describes a workaround to change v1beta1 to v1alpha1. If the API has moved to v1beta2, these instructions are stale and could cause users to create resources with an outdated API version.
  Remediation: Update the IntegrationTestScenario examples in custom-config.adoc to use appstudio.redhat.com/v1beta2 and remove or revise the v1beta1-to-v1alpha1 workaround instructions.

Medium

• [api-contract] modules/ROOT/pages/early-policy-violations.adoc:92 — The YAML example uses apiVersion: appstudio.redhat.com/v1beta2 for IntegrationTestScenario, while the existing documentation in custom-config.adoc uses v1alpha1 with a documented workaround to downgrade from v1beta1. These two documents now contradict each other on the correct API version for the same resource kind. See also: [stale-api-version] finding above.
  Remediation: Verify the currently correct API version for IntegrationTestScenario and ensure both documents agree.

Low

• [edge-case] modules/ROOT/pages/early-policy-violations.adoc:30 — The rules table lists olm.unpinned_snapshot_references as running at both staging and release (Yes/Yes), but the Considerations section discusses three OLM rules that may not pass until release-ready and omits olm.unpinned_snapshot_references. If this rule has similar release-timing caveats, it should be mentioned in the considerations.

• [edge-case] modules/ROOT/pages/early-policy-violations.adoc:137 — The document states "the test always passes" with STRICT=false. This is technically inaccurate — STRICT=false means policy violations do not cause failure, but infrastructure errors (pipeline failures, resolver issues, network problems) could still fail the test.

• [edge-case] modules/ROOT/pages/early-policy-violations.adoc:89 — The YAML example for IntegrationTestScenario does not include a namespace in metadata, and the oc create command lacks a -n flag. Since the doc says "Apply it to your namespace," consider noting that users should verify their current namespace context or adding -n <namespace> to the command.

---

Next steps:

• /fs-fix — agent addresses review findings automatically
• /fs-fix <your instruction> — agent fixes with your specific guidance
• Push commits directly — review re-runs automatically on push
• /fs-fix-stop — disable automatic fix runs for this PR

Previous run (3)

Review

Findings

High

• [cross-document-consistency] modules/ROOT/pages/early-policy-violations.adoc:84 — The IntegrationTestScenario YAML uses apiVersion: appstudio.redhat.com/v1beta2, while custom-config.adoc uses apiVersion: appstudio.redhat.com/v1alpha1 for the same resource kind (lines 48, 78, 134) and includes an explicit workaround note (lines 42–44) about changing v1beta1 to v1alpha1. Users following both guides encounter contradictory API versions. If v1beta2 is now correct, custom-config.adoc is stale and its workaround should be removed. If v1alpha1 is still required, this new guide has the wrong version.
  Remediation: Confirm which API version is currently correct for IntegrationTestScenario in the target clusters. Update both documents to use the same version. If v1beta2 is correct, update custom-config.adoc and remove the v1beta1-to-v1alpha1 workaround. If v1alpha1 is still required, update this page's YAML example.

Medium

• [internal-consistency] modules/ROOT/pages/early-policy-violations.adoc:153 — The "Schedule rules will likely fail" section and the blocking-mode NOTE (line 210) warn that schedule rules will report violations during integration testing. However, the table at lines 47–49 explicitly states that schedule.weekday_restriction and schedule.date_restriction do NOT run at staging pipeline_intention. The guide does not instruct users to set pipeline_intention to release in the new IntegrationTestScenario — only POLICY_CONFIGURATION and STRICT are set. If pipeline_intention defaults to staging (as stated at lines 3–4), schedule rules would be skipped entirely, contradicting the warning.
  Remediation: Resolve the contradiction: (1) If the new ITS should use pipeline_intention=release, add that parameter to the YAML example. (2) If it stays as staging, revise the schedule-rules caveat and the blocking-mode NOTE. (3) If the ECP can override pipeline_intention behavior, explain that mechanism.

• [placeholder-convention] modules/ROOT/pages/early-policy-violations.adoc:84 — Placeholders like <your-application-name> and <managed-namespace>/<ecp-name> are bare angle-bracket text inside a YAML block using [,yaml] without subs="+quotes". The established pattern in custom-config.adoc uses [,yaml,subs="+quotes"] with __<placeholder>__ formatting to visually distinguish replaceable values.
  Remediation: Change the YAML block to use [,yaml,subs="+quotes"] and format placeholders as __<your-application-name>__, __<managed-namespace>__/__<ecp-name>__.

Low

• [naming-convention] modules/ROOT/pages/early-policy-violations.adoc:1 — The page title capitalizes "Integration Tests" while the nav entry in contents.adoc uses lowercase "integration tests". The casing should be consistent between the two.

• [missing-cross-reference] modules/ROOT/pages/custom-config.adoc — The new page covers POLICY_CONFIGURATION, IntegrationTestScenario, and EnterpriseContractPolicy in detail, overlapping with custom-config.adoc. Neither page cross-references the other.

• [section-heading-style] modules/ROOT/pages/early-policy-violations.adoc:56 — Existing how-to guides do not use numbered step headings (=== Step 1: ...). They use descriptive section headings instead.

• [scope-coherence] modules/ROOT/pages/early-policy-violations.adoc — The PR body references both EC-1904 and EC-2007. The relationship between these two issues is not evident from the PR alone.

---

Next steps:

• /fs-fix — agent addresses review findings automatically
• /fs-fix <your instruction> — agent fixes with your specific guidance
• Push commits directly — review re-runs automatically on push
• /fs-fix-stop — disable automatic fix runs for this PR

Previous run (4)

Review

Findings

High

• [stale-doc] modules/ROOT/pages/custom-config.adoc:48 — custom-config.adoc uses apiVersion appstudio.redhat.com/v1alpha1 for IntegrationTestScenario resources and includes a workaround instructing users to change v1beta1 to v1alpha1. The new guide uses apiVersion appstudio.redhat.com/v1beta2 for the same resource kind without any such workaround. Users following both guides will get conflicting instructions about which API version to use.
  Remediation: Determine which API version is current. If v1beta2 is correct, update custom-config.adoc to use v1beta2 and remove the workaround. If v1alpha1 is still needed, update the new guide to match.

Medium

• [internal-consistency] modules/ROOT/pages/early-policy-violations.adoc:22 — The table includes a production column, but the document text only discusses staging and release pipeline_intention values. The production column is never defined or explained anywhere in the prose, leaving the reader without context for what production means, when it is used, or how it differs from release.
  Remediation: Either add a brief explanation of the production pipeline_intention value in the prose, or remove the production column if it is not relevant to this guide's scope.

Low

• [internal-consistency] modules/ROOT/pages/early-policy-violations.adoc:213 — The "Making the test blocking" section advises "Consider excluding those rules in your integration test ECP if you enable blocking mode." This creates tension with the core recommendation to reference the same ECP as the release pipeline, and with the "Keeping policies in sync" section which touts automatic sync from using the same ECP.

• [edge-case-correctness] modules/ROOT/pages/early-policy-violations.adoc:156 — The section title "Schedule rules will always fail" overclaims. The body text correctly uses softer language ("will likely report violations"). Schedule rules would pass if the integration test happens to run on a permitted day/date.

• [scope-completeness] modules/ROOT/partials/contents.adoc:11 — The navigation entry title "Catching policy violations early with Integration Tests" uses title case for "Integration Tests" while most existing entries use sentence case (e.g., "Using custom configuration", "Using custom data").

• [incomplete-doc] modules/ROOT/pages/custom-config.adoc:16 — custom-config.adoc covers POLICY_CONFIGURATION but does not cross-reference the new early-policy-violations.adoc guide, which covers a related workflow (creating additional IntegrationTestScenarios with STRICT mode).

---

Next steps:

• /fs-fix — agent addresses review findings automatically
• /fs-fix <your instruction> — agent fixes with your specific guidance
• Push commits directly — review re-runs automatically on push
• /fs-fix-stop — disable automatic fix runs for this PR

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 8:41 AM UTC · Completed 8:58 AM UTC

Commit: 87c4a29 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $4.15

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 12:36 PM UTC · Completed 12:54 PM UTC

Commit: 87c4a29 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $4.33

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 12:58 PM UTC · Completed 1:11 PM UTC

Commit: 87c4a29 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $3.80

Superseded by updated review

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 8:51 AM UTC · Ended 8:55 AM UTC

Commit: 87c4a29 · View workflow run →

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 8:56 AM UTC · Completed 9:12 AM UTC

Commit: 87c4a29 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $3.86

[fullsend-ai-review]

Risk Assessment: moderate (2/5)

Details

Documentation-only PR adding a new how-to guide with minor edits to existing files; medium blast radius and elevated code-age on stable doc files nudge the score above minimal, but no security, CI, or dependency risk factors are present.