chore(tests): delete every test-suite escape-hatch / allowlist mechanism

[tsouza]

Summary

Structural cleanup. Every test failure must surface as a real bug to fix at the source (cerberus code, seed, dashboard, panel) — no allow-list, tolerance, expected-empty, should_skip, expect.soft, or per-field blank-skip is acceptable anywhere.

Playwright specs

• iterate-all-dashboards.spec.ts: delete EXPECTED_EMPTY_EXPR_SUBSTRINGS (5 entries) + isExpectedEmpty + the conditional in probeTarget. Every empty panel result is now a hard fail.
• iterate-metrics-explorer.spec.ts: delete EXPECTED_EMPTY (6 entries), HISTOGRAM_COMPANION_SUFFIXES, dottedAlias, and the companion-suffix + dotted-alias fallbacks that masked bare-name 0-result failures. Every catalog-published metric must resolve to >= 1 series.
• iterate-drilldown-apps.spec.ts: delete ALERT_ERROR_PATTERNS (banner-substring allow-list), APP_NOT_INSTALLED_BANNER_PATTERNS, and DRILLDOWN_UPSTREAM_GRAFANA_CONSOLE_NOISE. Hard-fail every role=alert banner and every console error. Remove the install-probe early-skip; every catalogue entry must be installed.
• helpers/drilldown.ts: drop grafana-pyroscope-app from the catalogue (cerberus doesn't ship profiling).
• compose_grafana_smoke.spec.ts: flip expect.soft -> hard throw; prune the retired-allowlist documentation blocks.
• helpers/{assertions,dom,sweep}.ts: prune retired-allowlist comments; reword "tolerated" prose.

Compatibility harnesses

• compatibility/loki/cerberus-test-queries.yml: empty the should_skip block (was 17 entries — see commit body for the per-entry rationale capture). File kept as schema placeholder.
• compatibility/loki/cmd/loki-compliance-tester/main.go: delete the Overlay loader, skipKey lookup, SkipReason field on Result, -overlay flag, and every overlay-driven branch in compareAll. Drop the yaml import.
• compatibility/loki/scripts/run-loki-compatibility.sh: drop -overlay arg + DRIVER_OVERLAY env + the skipped jq bucket.
• compatibility/prometheus/expected-failures.json: deleted (was empty; the file itself was the allowlist mechanism).
• compatibility/tempo/expected-failures.json: deleted. Remove the --expected-failures flag, the loader, the docker-compose mount, the run-script DIFF_FLAGS branch.
• compatibility/tempo/driver/differ.go: remove the StartTimeUnixNano blank-skip — an asymmetric blank on either side is now a real divergence (the backend that omitted the field is the bug). Epsilon comparison for parsed numeric values is kept (float-noise absorption, not a case-allowlist).

CI / lefthook gates

• .github/workflows/ci.yml forbid-skip:
  • Replace the "guard new should_skip entries with a tracking ref" step with a hard reject of any non-empty should_skip block. The consumer code is gone; entries would be silently ignored.
  • New step: reject test-suite escape-hatch primitives anywhere in .ts/.tsx/.go (EXPECTED_EMPTY, EXPECTED_TOLERATED, isKnownTolerated, tolerated404, expect.soft, should_tolerate, skipReason/SkipReason, APP_NOT_INSTALLED_BANNER_PATTERNS, DRILLDOWN_UPSTREAM_GRAFANA_CONSOLE_NOISE).
• lefthook.yml: mirror the same forbid-escape-hatch + should_skip guards in the pre-push hook.
• scripts/check-skip-additions.sh: deleted. The "guard untracked entries" policy is replaced by "zero entries".

Docs

• docs/compatibility.md: replace the "Expected-failures allowlist" section with a "No allow-lists" section documenting the new policy.
• compatibility/loki/README.md: align the overlay description with the schema-placeholder reality.
• compatibility/prometheus/{test-cerberus.yml,scripts/run-compatibility.sh}: reword to remove the expected-failures references.

Inventory of compat-harness should_skip entries removed

The 17 entries in compatibility/loki/cerberus-test-queries.yml re-exposed by this PR fall into three buckets — every one is now a real bug to fix at the source. Follow-up tasks should be opened for each:

• Upstream-pinned skip: true rows (10 entries) — fast/structured-metadata.yaml#JSON parsing with structured metadata filter first, regression/metric-queries.yaml#HTTP status code distribution, regression/structured-metadata.yaml#JSON parsing with duration filter and structured metadata, regression/structured-metadata.yaml#Drilldown pattern with case-insensitive error search, plus all six exhaustive/unwrap-aggregations.yaml#Standard variance/deviation/Quantile-over-time/p50/p90/p95/p99/latency/duration entries. The upstream YAML marks these skip: true because Loki's v2 engine doesn't support those shapes — meaning there is no reference baseline. Either the v2-engine compose-config baseline needs to flip, or these cases need to be requested upstream.
• Structured-metadata harness seed-gap cluster (3 entries) — regression/structured-metadata.yaml#Combined line and metadata filter, regression/structured-metadata.yaml#Drilldown with failed search and error level, regression/structured-metadata.yaml#Drilldown with error/warn level and refused search, exhaustive/aggregations.yaml#Count aggregated by detected_level. Reference Loki indexes detected_level from structured-metadata block input; the harness seeder flows OTel records through the upstream ClickHouse exporter shape (SeverityText present, structured-metadata block absent), so the reference returns empty. The fix is in the harness seeder, not in cerberus.
• Plain seed-gap (4 entries) — exhaustive/aggregations.yaml#Count aggregated by {pod,namespace,service_name,cluster and namespace,service_name and container} and exhaustive/unwrap-aggregations.yaml#Without multiple labels. The seeder doesn't write pod / namespace / service_name / cluster / container labels. Fix the seeder.

Inventory of PR #701 entries flagged for rebase

PR #701 (feat/quickstart-rich-observability) adds two more EXPECTED_EMPTY entries on top of main that are NOT in this PR's diff:

• iterate-metrics-explorer.spec.ts — adds system_ prefix and clickhouse_event prefix entries (commit b3a9dad).
• iterate-all-dashboards.spec.ts — broadens the clickhouse_event{name=~"Query match to clickhouse_event{name=~" (commit b4526fd).

Once this PR merges and #701 rebases, those entries become deletion conflicts the rebaser must hand-resolve to "deleted". The forbid-escape-hatch gate will reject #701 outright if any survives the rebase.

Test plan

• CI forbid-skip green on this PR (the new gate doesn't catch any pattern in this PR's own diff).
• CI check green — go vet already clean locally; go test ./compatibility/loki/cmd/loki-compliance-tester/ ./compatibility/tempo/driver/ passes locally (113 tests).
• CI compatibility/{prometheus,loki,tempo} — every previously-tolerated diff now surfaces as a real failure. The compat harnesses are report-only (per task docs(compliance): refresh for v1.0.0-RC1 + per-QL corpora (M5.4) #68), so the CI lane stays green; the compat-score.json shows the new ground truth.
• CI compose-smoke — expect.soft is now a hard throw; any previously-soft failure becomes a hard fail.
• dashboard (push-to-main + nightly) — every previously-tolerated empty panel / 0-series metric / install-skip is now a hard fail.

🤖 Generated with Claude Code