test: pipeline failure scenarios — what CI catches vs. misses - #11
Merged
Conversation
added 3 commits
June 16, 2026 17:15
Adds PIPELINE-TESTS.md summarising 6 injected breakages and which pipeline stage (if any) catches each one. Three of six pass CI undetected — all with the same root cause: no HTTP smoke test is executed through the full ingress→frontend→backend path after deploy.
Adds two new steps to the deploy job that catch the three classes of
breakage that rollout status + tag verification missed:
1. Check Service endpoints — kubectl get endpoints, asserts >= 1 ready
address. Catches label selector mismatches before any HTTP attempt.
2. HTTP smoke test — curl through traefik ingress with Host header,
exercises the full path: ingress → nginx proxy_pass → backend service.
Requests /api/health and /api/items with --retry 10 --retry-all-errors
to absorb the traefik Ingress sync delay.
Previously 3 of 6 injected breakages passed CI undetected (wrong nginx
proxy port, service selector mismatch, wrong ingress port). After this
fix all six scenarios fail the pipeline at the correct step.
jq asserts all Endpoints objects in the namespace (minus the built-in kubernetes one) have at least one ready subset. Zero names hardcoded — picks up new services automatically if the chart grows.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Qué hace este PR
Dos cosas: documenta los gaps del pipeline y los cierra.
Se inyectaron 6 roturas intencionadas (una por rama) para medir qué detecta el pipeline. Resultado: 3 de 6 pasaban CI completamente en verde mientras la app estaba rota en producción. El fix son dos pasos nuevos en el deploy job.
Las 6 ramas de prueba
Cada rama es un commit único sobre `main`. Se pueden abrir como PRs individuales para observar el comportamiento de CI:
El fix: dos pasos nuevos en `zarf-package.yml`
1. Check Service endpoints
```yaml
```
Corre `kubectl get endpoints` para ambos Services y falla si alguno tiene 0 endpoints. Captura los selector mismatches antes de cualquier petición HTTP. Rápido y sin dependencia de red.
2. HTTP smoke test (a través del ingress)
```yaml
```
Hace `curl` a `/api/health` y `/api/items` usando la IP del nodo k3s con `Host: airgapped-demo.local`. Traefik enruta por cabecera Host — no hace falta tocar `/etc/hosts`.
El tráfico recorre el camino completo:
```
traefik → frontend Service (:80) → nginx proxy_pass → backend Service (:3001) → pod
```
Un `--retry 10 --retry-all-errors` absorbe los segundos que tarda traefik en sincronizar el nuevo Ingress.
Por qué los tres gaps silenciosos comparten la misma causa raíz
Los escenarios 1, 2 y 5 fallaban sin detectarse porque el pipeline verificaba que el pod estaba sano y el tag era correcto, pero nunca hacía una petición HTTP real. Los pods pueden estar perfectamente sanos mientras el tráfico se pierde en cualquier capa de routing entre el ingress y el pod.
Un solo `curl` a través del ingress ejerce todas esas capas a la vez.
Archivos en este PR