Note: docs/proposals.md step 1 calls for opening a GitHub
Discussion. This repository has Discussions disabled
(has_discussions: false), so this issue is standing in as the
discussion stage — What?/Why? only, no implementation details,
per the documented process. (Separately: enabling Discussions on
this repo would let future proposals follow the process as
written.)
What?
Make end-to-end (E2E) test coverage an explicitly named, required
third tier in the testing requirements — distinct from
"integration" — and define all three tiers by the question each
answers, not by the infrastructure each happens to need:
- Unit validates business logic.
- Integration (IT) validates wiring.
- E2E validates the journey.
A simple classification test for contributors: what would this
test catch that the tier below it wouldn't? A logic/algorithm bug
→ unit. A contract/wiring mismatch between two of our own
components (even if proven against real infra, e.g. a single Kind
cluster) → integration. A bug that only manifests when a real user
or operator journey runs against actually-deployed infrastructure
→ E2E.
Also propose: any repo with runtime/cluster behavior must make its
E2E coverage (or lack of it) visible and tracked, never silent.
Goals
- Give "E2E" its own name and definition in
docs/conventions.md
and CONTRIBUTING.md, instead of folding it into "integration
tests proving end-to-end behavior."
- Define tiers by intent (business logic / wiring / journey), not
by mechanism (mocked client vs. real cluster vs. multi-cluster),
so the guidance is consistent across repos regardless of how each
project's tooling implements a given tier.
- Require that a repo's E2E coverage status — what exists, what
doesn't, and why — is always explicit (e.g. a linked
architecture doc and/or a tracked gap issue), never an unstated
absence.
Why?
Motivation
Auditing praxis-proxy/grid against the current conventions
template surfaced two concrete problems:
-
No distinct E2E tier. docs/conventions.md's testing
section names two tiers (unit, integration), where "integration"
is described as "proving end-to-end behavior" — E2E has no
separate identity or requirement of its own. A downstream repo's
copy of this doc goes further and explicitly downgrades full
cluster runs: "Full kind cluster runs complement these tests but
are not a substitute for focused config-shape assertions" — the
opposite of a mandate.
-
Inconsistent meaning of "integration" across repos. Per
praxis-docs, grid's integration tests use a mocked
kube::Client driving reconcile()/resolve_* directly (no
real cluster), while praxis-operator's integration tests
require a running cluster with the operator deployed. Same
tier name, different infra requirement, different actual
guarantee. Defining tiers by intent (business logic / wiring /
journey) rather than mechanism resolves this: both of the above
are legitimately "wiring" tests regardless of whether the
boundary they cross is mocked or a real single-node cluster.
-
A real gap this ambiguity created. grid separately has a
thought-through E2E CI design
(docs/architecture/ci-kind-e2e.md) that is not linked from
CONTRIBUTING.md or docs/conventions.md at all — a
contributor reading the actual contribution requirements never
encounters it. The practical result:
praxis-proxy/grid#60,
a scoring-order regression, shipped to main undetected and was
only caught because a contributor happened to manually run the
Kind validation while working on an unrelated PR. Even the
person who found it wasn't sure whether the check ran in CI.
None of this assumes every repo can run full multi-cluster E2E on
every PR today — real constraints exist (cross-repo image
publishing, Kind cluster-name isolation for parallel runs, runtime
cost). The point isn't to mandate infrastructure the template can't
guarantee; it's to mandate that the decision about E2E coverage is
never silent, and that the three tiers mean the same thing
everywhere they're used.
User Stories
- As a contributor, I want the conventions to tell me clearly
whether my change needs E2E coverage, and to classify my test by
what it proves rather than guess which tier a given mechanism
belongs to.
- As a maintainer, I want repos with no E2E coverage yet to say so
explicitly (a tracked issue/doc reference), so it shows up in
planning instead of being invisible until something like grid#60
happens.
- As a new repo scaffolded from this template, I want the testing
section to hand me a definition I can apply consistently, not
reinterpret per project.
What?
Make end-to-end (E2E) test coverage an explicitly named, required
third tier in the testing requirements — distinct from
"integration" — and define all three tiers by the question each
answers, not by the infrastructure each happens to need:
A simple classification test for contributors: what would this
test catch that the tier below it wouldn't? A logic/algorithm bug
→ unit. A contract/wiring mismatch between two of our own
components (even if proven against real infra, e.g. a single Kind
cluster) → integration. A bug that only manifests when a real user
or operator journey runs against actually-deployed infrastructure
→ E2E.
Also propose: any repo with runtime/cluster behavior must make its
E2E coverage (or lack of it) visible and tracked, never silent.
Goals
docs/conventions.mdand
CONTRIBUTING.md, instead of folding it into "integrationtests proving end-to-end behavior."
by mechanism (mocked client vs. real cluster vs. multi-cluster),
so the guidance is consistent across repos regardless of how each
project's tooling implements a given tier.
doesn't, and why — is always explicit (e.g. a linked
architecture doc and/or a tracked gap issue), never an unstated
absence.
Why?
Motivation
Auditing
praxis-proxy/gridagainst the current conventionstemplate surfaced two concrete problems:
No distinct E2E tier.
docs/conventions.md's testingsection names two tiers (unit, integration), where "integration"
is described as "proving end-to-end behavior" — E2E has no
separate identity or requirement of its own. A downstream repo's
copy of this doc goes further and explicitly downgrades full
cluster runs: "Full kind cluster runs complement these tests but
are not a substitute for focused config-shape assertions" — the
opposite of a mandate.
Inconsistent meaning of "integration" across repos. Per
praxis-docs,
grid's integration tests use a mockedkube::Clientdrivingreconcile()/resolve_*directly (noreal cluster), while
praxis-operator's integration testsrequire a running cluster with the operator deployed. Same
tier name, different infra requirement, different actual
guarantee. Defining tiers by intent (business logic / wiring /
journey) rather than mechanism resolves this: both of the above
are legitimately "wiring" tests regardless of whether the
boundary they cross is mocked or a real single-node cluster.
A real gap this ambiguity created.
gridseparately has athought-through E2E CI design
(
docs/architecture/ci-kind-e2e.md) that is not linked fromCONTRIBUTING.mdordocs/conventions.mdat all — acontributor reading the actual contribution requirements never
encounters it. The practical result:
praxis-proxy/grid#60,
a scoring-order regression, shipped to
mainundetected and wasonly caught because a contributor happened to manually run the
Kind validation while working on an unrelated PR. Even the
person who found it wasn't sure whether the check ran in CI.
None of this assumes every repo can run full multi-cluster E2E on
every PR today — real constraints exist (cross-repo image
publishing, Kind cluster-name isolation for parallel runs, runtime
cost). The point isn't to mandate infrastructure the template can't
guarantee; it's to mandate that the decision about E2E coverage is
never silent, and that the three tiers mean the same thing
everywhere they're used.
User Stories
whether my change needs E2E coverage, and to classify my test by
what it proves rather than guess which tier a given mechanism
belongs to.
explicitly (a tracked issue/doc reference), so it shows up in
planning instead of being invisible until something like grid#60
happens.
section to hand me a definition I can apply consistently, not
reinterpret per project.