pipelines: opt-in placeholder mode for schema validation

[raelga]

Problem

sdp-pipelines validates pipeline.yaml after rendering it with a dunder config. That render turns every scalar config value into a string placeholder, which breaks schema validation when the pipeline field is a boolean, integer, or number.

skipSync: {{ .monitoring.skipDatasourceSync }}

# dunder render used by validation:
skipSync: __monitoring.skipDatasourceSync__

# schema expects boolean, got string

Goal

Make the dunder validation pass understand scalar placeholders without weakening normal pipeline validation. Strict validation remains the default for real typed config and all existing callers.

This PR is intentionally scalar-only. Dynamic {{ if }} / {{ range }} pipeline generation and per-region lists are out of scope because Ev2 manifests are generated up front and ARO-HCP config should avoid arrays: https://github.com/Azure/ARO-HCP/blob/main/docs/configuration.md#limitations.

What changes

Adds opt-in placeholder validation with WithAllowPlaceholders(pattern) and ValidatePipelineSchemaWithOptions(...). When enabled, the schema is widened in memory so non-string scalar nodes (boolean, integer, number, and non-string enums, including $ref targets) also accept placeholder strings matching the configured pattern.

No schema file is changed on disk. ValidatePipelineSchema(...), NewPipelineFromFile(...), and NewPipelineFromBytes(...) keep strict behavior unless a caller explicitly passes the new option.

Dunder producer move-in

Moves the scalar dunder placeholder producer from sdp-pipelines into pipelines/types. This gives the producer and validator one shared home and one shared default placeholder contract: DefaultPlaceholderPattern = ^__.+__$.

The moved API is PlaceholderGenerator, NewDunderPlaceholders(), and EV2Mapping(...). Slices are treated as opaque scalar leaves rather than walked element-by-element, matching the array/list policy above.

Tests

The tests cover strict vs placeholder validation for booleans, integers, $ref scalar targets, custom placeholder patterns, and backwards compatibility. They also cover the moved dunder producer and its regex contract with the validator.

The end-to-end test shows the intended flow:

typed scalar config
  -> EV2Mapping(NewDunderPlaceholders())
  -> dunder-rendered pipeline.yaml
  -> ValidatePipelineSchemaWithOptions(..., WithAllowPlaceholders("")) succeeds
  -> strict validation rejects the same dunder-rendered YAML

Follow-ups

After this merges and a new ARO-Tools tag includes it, sdp-pipelines can import the dunder producer from ARO-Tools and pass types.WithAllowPlaceholders("") at the dunder validation site. After that deploys, ARO-HCP can revert the skipSync {{ if }} workaround from Azure/ARO-HCP#5339.

Closed as part of this pivot: #239, sdp-pipelines ADO PR 15824468, and AROSLSRE-914. Those tracked the slice/list path, which is no longer policy-compliant.

[Copilot]

Pull request overview

This PR introduces an opt-in “placeholder mode” for pipeline schema validation, allowing placeholder strings (regex-matched) to be accepted where the schema expects non-string scalars. This is aimed at validating templated pipeline.yaml rendered against a fully “dunderized” config while preserving strict validation as the default for all existing callers.

Changes:

• Added ValidateOption plumbing and WithAllowPlaceholders(pattern) (default ^__.+__$ when pattern is empty).
• Added schema “widening” logic that rewrites decoded JSON Schema in-memory to allow placeholder strings for non-string scalars before compiling.
• Added a new ValidatePipelineSchemaWithOptions entry point and threaded options through NewPipelineFromFile/NewPipelineFromBytes, plus new tests for placeholder behavior.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
pipelines/types/options.go	Adds validation option types and WithAllowPlaceholders defaulting behavior.
pipelines/types/widen.go	Implements in-memory schema traversal and widening to accept placeholder strings for certain scalar schema nodes.
pipelines/types/util.go	Splits schema decode/compile, adds ValidatePipelineSchemaWithOptions, and applies widening when placeholder mode is enabled.
pipelines/types/pipeline.go	Threads validation options through pipeline constructors and uses the new validation entry point.
pipelines/types/util_test.go	Adds test coverage for strict vs placeholder validation scenarios and backwards-compat behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

[raelga]

Update pushed and PR body reframed around the scalar-only path:

• Closed the array/slice path per the ARO-HCP config limitation and Steve/Gerd's feedback that dynamic {{ if }} / {{ range }} pipeline generation is policy-disallowed for Ev2 artifact generation.
• Added pipelines/types/dunder.go so NewDunderPlaceholders / EV2Mapping live next to the placeholder-mode validator and share DefaultPlaceholderPattern.
• Added TestEndToEndPlaceholderValidation to show the full producer + validator contract: typed scalar config -> dunder config -> rendered pipeline.yaml -> placeholder-mode schema validation succeeds, while strict mode still rejects the dunder-rendered YAML.

The follow-up is now smaller: after this merges and an ARO-Tools tag is cut, sdp-pipelines can import the dunder producer from ARO-Tools and opt into WithAllowPlaceholders("") at the dunder validation site.