Skip to content

[Epic]: Validator capability: performance / conformance checks must cover every shipped workload variant #1044

Description

@mchmarny

Goal

Every performance- or conformance-phase validator must actually run against the workload variant the recipe deploys. When a recipe ships a workload the validator currently skippeds on, the gap is closed here — either by extending the validator to cover the new workload kind, or by adding a sibling check that does. A "passing" recipe must not be passing because the gate silently skipped.

Parent initiative: #1041.

Success criteria

  1. For every recipe in recipes/overlays/, the validators its validation.phases.*.checks reference do not return status: "skipped - <reason>" on the intended workload — they either pass or fail with a real finding.
  2. New workload kinds added under an existing intent (e.g., NIM under inference, alongside the existing Dynamo coverage) ship with first-day validator coverage; if the existing validator cannot be extended, a new check is filed and added to the relevant overlay.
  3. Contributor docs (docs/contributor/validations.md) describe the supported workload kinds for each check, so future overlay authors know which check to reference.

Scope

In scope:

  • Extending an existing validator to recognize a new workload kind (e.g., NIM as a peer of Dynamo).
  • Adding a new check when the runtime contract is sufficiently different that extending in place would be awkward.
  • Documenting the supported workload kinds per check.

Out of scope (other epics under #1041):

  • Adding a constraint to a recipe that already references a working validator (performance Epic).
  • Authoring new overlays (overlay-coverage Epic).
  • Validator framework architecture (separate initiative).

New validator capability gaps should be filed as standalone issues and attached here.

Metadata

Metadata

Assignees

Labels

area/validatortheme/validationConstraint evaluation, health checks, and conformance evidence

Fields

No fields configured for Enhancement.

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions