test: add E2E tests for payload processor with Kind cluster CI by asaadbalum · Pull Request #31 · llm-d/llm-d-inference-payload-processor

asaadbalum · 2026-05-04T11:31:21Z

Summary

Add end-to-end tests that deploy a complete Envoy + Payload Processor + model-server-simulator stack on a Kind cluster and validate core functionality through the actual ext_proc gRPC pipeline.

Base model routing: Verifies that model field extraction from /v1/chat/completions and /v1/completions bodies routes Llama and DeepSeek requests to the correct pools via X-Gateway-Base-Model-Name header.
LoRA adapter routing: Verifies that adapter names are resolved to base models through ConfigMap reconciliation and routed to the correct pool.
Streaming: Verifies that streaming requests ("stream": true) return SSE text/event-stream chunks through the full Envoy → Payload Processor → model-server pipeline.
Metrics: Verifies that ipp_info and ipp_success_total Prometheus metrics are populated after traffic flows.
CI integration: E2E job added to ci-pr-checks.yaml — delegates all orchestration to make test-e2e / hack/test-e2e.sh.

Manifest structure

Kubernetes manifests live under deploy/ following the llm-d-router pattern: shared components (deploy/components/) and environment-specific infrastructure (deploy/environments/dev/e2e-infra/). Test code references these manifests via relative paths with ${VAR} substitution, enabling reuse for both E2E tests and local Kind development.

The E2E Envoy configuration mirrors production:

Envoy distroless-v1.33.2 with request_body_mode: FULL_DUPLEX_STREAMED
allow_mode_override: true, failure_mode_allow: false
request_trailer_mode: SEND, response_trailer_mode: SKIP

Model server simulators use fake model names (ipp-test-llama, ipp-test-deepseek) with --mode=echo, matching the llm-d-router's standard E2E pattern. The sim image is pinned to v0.9.0 for reproducibility.

New files

File	Purpose
`deploy/components/ipp/deployment.yaml`	IPP Deployment (`imagePullPolicy: IfNotPresent`)
`deploy/components/ipp/service.yaml`	IPP Service (NodePort 30090 for metrics)
`deploy/components/ipp/rbac.yaml`	ServiceAccount + ClusterRole + ClusterRoleBinding
`deploy/components/model-server/llama/deployment.yaml`	Llama simulator + Service + adapter ConfigMap
`deploy/components/model-server/deepseek/deployment.yaml`	DeepSeek simulator + Service + adapter ConfigMap
`deploy/environments/dev/e2e-infra/envoy.yaml`	Envoy proxy with ext_proc filter config (NodePort 30080)
`test/e2e/e2e_suite_test.go`	Ginkgo BeforeSuite/AfterSuite — deploys stack, waits for readiness
`test/e2e/e2e_test.go`	7 test cases covering all scenarios above
`test/e2e/README.md`	Developer quickstart guide
`test/e2e/TROUBLESHOOTING.md`	Troubleshooting guide (Kind multi-cluster, image issues)
`hack/test-e2e.sh`	Orchestration script (Kind + image build + test + auto-cleanup)

Modified files

File	Change
`.github/workflows/ci-pr-checks.yaml`	Added `e2e` job (calls `make test-e2e`)
`Makefile`	Added `test-e2e`, `image-build-local`, `image-kind` targets + E2E config vars
`go.mod` / `go.sum`	Added `github.com/onsi/ginkgo/v2`, `github.com/onsi/gomega`, `github.com/openai/openai-go` + transitive deps

What's NOT included

No changes to core IPP logic (pkg/, internal/, cmd/)
No changes to existing tests or CI jobs
No Helm chart modifications
No production deployment manifests (these are test-only)

Test plan

All 7 E2E tests pass on Kind (7 Passed | 0 Failed)
All unit tests pass with race detector
golangci-lint 0 issues (standard + E2E build tags)
CI workflow YAML validated (syntax + job dependency graph)

github-actions · 2026-05-04T11:31:31Z

Unsigned commits detected! Please sign your commits.

For instructions on how to set up GPG/SSH signing and verify your commits, please see GitHub Documentation.

aradhalevy

Looks good, some minor comments (and we will need the new added check to run and pass first)

aradhalevy · 2026-05-05T14:15:02Z

+  name: llama-adapters
+  namespace: $E2E_NS
+  labels:
+    inference.llm-d.io/ipp-managed: "true"


I believe it supposed to be llm-d.ai instead of llm-d.io as per #28.

After this fix the test pass for me locally

aradhalevy · 2026-05-05T14:15:39Z

+  name: deepseek-adapters
+  namespace: $E2E_NS
+  labels:
+    inference.llm-d.io/ipp-managed: "true"


same llm-d.ai instead of llm-d.io

aradhalevy · 2026-05-05T14:41:14Z

+| Streaming routing | SSE chunks returned |
+| Metrics | `bbr_info`, `bbr_success_total` |
+
+## Troubleshooting


When I had a couple of other clusters set up in kind, Envoy tried to route requests to them. Please add a suggestion / troubleshooting to use kind delete clusters --all first to clean your kind environment first.

aradhalevy · 2026-05-05T18:28:08Z

+      containers:
+        - name: payload-processor
+          image: $E2E_IMAGE
+          imagePullPolicy: Never


This should be IfNotPresent if we want to test on a different cluster other than kind. But that requires pushing an image to ghcr.io and might require some more changes, and can be dealt with in another issue / PR if you prefer to keep this PR for kind only

Keeping it for now, will address it in a follow-up pr

Please fix it now

aradhalevy · 2026-05-05T18:32:35Z

+
+      - name: Run E2E tests
+        run: |
+          E2E_IMAGE=ghcr.io/llm-d/llm-d-inference-payload-processor:e2e \


You don't use the Makefile / script here, I think it would be better to use them to have a single source of truth.

github-actions · 2026-05-06T07:57:19Z

⚠️ Large PR detected

Your PR is large. Please consider breaking it into multiple PRs.

The do-not-merge/hold label has been added and can be removed by the reviewers based on their judgement.

aradhalevy

LGTM.
I think this is fine even tough it is a large PR as all the code is needed and relevant to this minimal e2e test suite.

nirrozenbaum · 2026-05-06T12:23:44Z

+
+var (
+	testConfig *testutils.TestConfig
+	ppImage    string


nit: it would be better to align the name on ipp rather than pp (that was the agreed acronym).

nirrozenbaum · 2026-05-06T12:27:21Z

+| Base model routing | Pool routing via header |
+| LoRA adapter routing | ConfigMap adapter lookup |
+| Streaming routing | SSE chunks returned |
+| Metrics | `bbr_info`, `bbr_success_total` |


as a follow up, we should update all metrics to be named ipp instead of bbr.

not a blocker

nirrozenbaum · 2026-05-06T12:28:52Z

+| Streaming routing | SSE chunks returned |
+| Metrics | `bbr_info`, `bbr_success_total` |
+
+## Troubleshooting


this should probably go to a separate troubleshot guide.
quickstart guide should be quick, and simple :)
in other words, the simplest explanation of the green path.

nirrozenbaum · 2026-05-06T12:31:55Z

@@ -0,0 +1,164 @@
+# Llama model server simulator
+apiVersion: apps/v1


can you explain the separation between e2e-deployment and deepseek-model-server?
I see deepseek has deployment + svc.
here I see deployment + svc for a llama plus adapter of deepseek + llama + many other CRs.
not sure I understand the separation.

nirrozenbaum · 2026-05-06T12:36:58Z

+              - '!**/*.md'
+              - '!LICENSE'
+              - '!OWNERS'
+


can you move this logic to the file "ci-pr-checks.yaml" (and on the way to clean from it the lint python and build at the end)?

github-actions · 2026-05-11T07:13:25Z

⚠️ Large PR detected

Your PR is large. Please consider breaking it into multiple PRs.

The do-not-merge/hold label has been added and can be removed by the reviewers based on their judgement.

nirrozenbaum · 2026-05-13T05:57:40Z

@shmuelk can you please review this PR when you have time?
would be good to validate the tests structure is aligned with scheduler.

shmuelk · 2026-05-13T12:22:38Z

@nirrozenbaum I took a very quick look at this PR.

I don't like it's structure. This E2E test looks a lot more like the old IGW E2E test and not like the scheduler's E2E test.

@roytman restructured the End to End test and the development environment on Kind to use the same K8S YAML and config YAML files where possible. Following that idea here will make it easier to put together a development environment on Kind.

nirrozenbaum · 2026-05-13T21:13:06Z

@nirrozenbaum I took a very quick look at this PR.

I don't like it's structure. This E2E test looks a lot more like the old IGW E2E test and not like the scheduler's E2E test.

@roytman restructured the End to End test and the development environment on Kind to use the same K8S YAML and config YAML files where possible. Following that idea here will make it easier to put together a development environment on Kind.

@asaadbalum can you please take a look on @shmuelk's feedback and work towards setting the e2e to work like they do in llm-d scheduler? (or the new name llm-d router).

github-actions · 2026-05-17T07:12:24Z

⚠️ Large PR detected

Your PR is large. Please consider breaking it into multiple PRs.

The do-not-merge/hold label has been added and can be removed by the reviewers based on their judgement.

nirrozenbaum · 2026-05-18T21:18:43Z

cc for another pair of eyes: @noyitz

github-actions · 2026-05-31T12:59:23Z

⚠️ Large PR detected

Your PR is large. Please consider breaking it into multiple PRs.

The do-not-merge/hold label has been added and can be removed by the reviewers based on their judgement.

github-actions · 2026-06-01T08:35:51Z

⚠️ Large PR detected

Your PR is large. Please consider breaking it into multiple PRs.

The do-not-merge/hold label has been added and can be removed by the reviewers based on their judgement.

github-actions · 2026-06-01T08:38:29Z

⚠️ Large PR detected

Your PR is large. Please consider breaking it into multiple PRs.

The do-not-merge/hold label has been added and can be removed by the reviewers based on their judgement.

github-actions · 2026-06-24T01:40:48Z

This PR is marked as stale after 21d of inactivity. After an additional 14d of inactivity (7d to become rotten, then 7d more), it will be closed. To prevent this PR from being closed, add a comment or remove the lifecycle/stale label.

asaadbalum · 2026-07-07T09:02:30Z

@shmuelk Ready for review again, alll of the previous comments addressed and pr rebased.

shmuelk · 2026-07-08T12:08:49Z

+- role: control-plane
+  extraPortMappings:
+  - containerPort: 30080
+    hostPort: 30080


This port here is hard coded. The environment variable E2E_PORT is useless.

Change this to use E2E_PORT when creating the cluster

shmuelk · 2026-07-08T12:09:39Z

+    hostPort: 30080
+    protocol: TCP
+  - containerPort: 30090
+    hostPort: 30090


This port is also hard coded. The environment variable E2E_METRICS_PORT is useless.

Change this to use E2E_METRICS_PORT when creating the cluster

shmuelk · 2026-07-08T12:23:51Z

+## Architecture
+
+```text
+Go test (OpenAI SDK) ──► Envoy NodePort:30080 (ext_proc) ──► Payload Processor


The port numbers are the default values...

shmuelk

Looks much better. I left a few comments

asaadbalum · 2026-07-08T14:08:00Z

Looks much better. I left a few comments

Addressed

shmuelk · 2026-07-09T08:36:13Z

+nodes:
+- role: control-plane
+  extraPortMappings:
+  - containerPort: ${E2E_PORT}


As you didn't change Envoy's Service definition, this line shouldn't have been changed.

Suggested change

- containerPort: ${E2E_PORT}

- containerPort: 30080

shmuelk · 2026-07-09T08:37:09Z

+  - containerPort: ${E2E_PORT}
+    hostPort: ${E2E_PORT}
+    protocol: TCP
+  - containerPort: ${E2E_METRICS_PORT}


As you didn't change the IPPs Service definition this line shouldn't have been changed.

Suggested change

- containerPort: ${E2E_METRICS_PORT}

- containerPort: 30090

shmuelk · 2026-07-09T08:45:45Z

Getting close. I left two more comments. Please run the end to end test locally, with E2E_PORT=31080 and E2E_METRICS_PORT=31090 (i.e. non-default values)

Adds end-to-end tests that deploy a complete stack on a Kind cluster: Envoy proxy (distroless-v1.33.2, FULL_DUPLEX_STREAMED ext_proc), Payload Processor, Llama and DeepSeek model-server simulators, and adapter ConfigMaps. Kubernetes manifests live under deploy/ following the llm-d-router pattern: shared components (deploy/components/) and environment-specific infrastructure (deploy/environments/dev/e2e-infra/). Test code references these manifests via relative paths with ${VAR} substitution. Tests use the OpenAI Go SDK to send requests via Kind NodePort, with net/http for adapter routing and streaming where raw headers are needed. CI delegates all orchestration to hack/test-e2e.sh via make test-e2e. Tests cover base-model routing, LoRA adapter resolution, streaming requests, and ipp_* metrics exposure. Signed-off-by: Asaad Balum <asaad.balum@gmail.com>

asaadbalum · 2026-07-09T14:17:50Z

Getting close. I left two more comments. Please run the end to end test locally, with E2E_PORT=31080 and E2E_METRICS_PORT=31090 (i.e. non-default values)

Done

shmuelk · 2026-07-09T14:55:34Z

/lgtm
/approve

asaadbalum force-pushed the feat/issue-14-add-e2e-tests branch 3 times, most recently from 9da84d3 to 1b2b395 Compare May 4, 2026 11:54

aradhalevy reviewed May 5, 2026

View reviewed changes

asaadbalum force-pushed the feat/issue-14-add-e2e-tests branch from 1b2b395 to 9de9955 Compare May 6, 2026 07:57

github-actions Bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels May 6, 2026

nirrozenbaum mentioned this pull request May 6, 2026

add size label automatically on incoming PRs llm-d/llm-d-router#996

Closed

asaadbalum requested a review from aradhalevy May 6, 2026 08:08

aradhalevy approved these changes May 6, 2026

View reviewed changes

nirrozenbaum reviewed May 6, 2026

View reviewed changes

asaadbalum force-pushed the feat/issue-14-add-e2e-tests branch from 9de9955 to 3c561d1 Compare May 11, 2026 07:13

asaadbalum requested a review from nirrozenbaum May 11, 2026 07:17

asaadbalum force-pushed the feat/issue-14-add-e2e-tests branch from 3c561d1 to 2a0668c Compare May 17, 2026 07:12

shmuelk reviewed May 24, 2026

View reviewed changes

Comment thread test/e2e/e2e_suite_test.go Outdated

shmuelk reviewed May 24, 2026

View reviewed changes

Comment thread deploy/components/model-server/llama/deployment.yaml Outdated

asaadbalum force-pushed the feat/issue-14-add-e2e-tests branch from 2a0668c to 1732c7b Compare May 31, 2026 12:59

asaadbalum force-pushed the feat/issue-14-add-e2e-tests branch from 7fbc4bd to 6942517 Compare June 1, 2026 08:38

asaadbalum requested a review from shmuelk June 2, 2026 10:57

github-actions Bot added the lifecycle/stale label Jun 24, 2026

github-actions Bot added lifecycle/rotten and removed lifecycle/stale labels Jul 1, 2026

asaadbalum force-pushed the feat/issue-14-add-e2e-tests branch from 6942517 to 573a25e Compare July 7, 2026 08:39

github-actions Bot removed the lifecycle/rotten label Jul 7, 2026

nirrozenbaum removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 8, 2026

shmuelk reviewed Jul 8, 2026

View reviewed changes

Comment thread test/e2e/e2e_suite_test.go Outdated

shmuelk reviewed Jul 8, 2026

View reviewed changes

Comment thread test/e2e/e2e_suite_test.go Outdated

shmuelk reviewed Jul 8, 2026

View reviewed changes

asaadbalum force-pushed the feat/issue-14-add-e2e-tests branch from 573a25e to 0975629 Compare July 8, 2026 14:05

shmuelk reviewed Jul 9, 2026

View reviewed changes

asaadbalum force-pushed the feat/issue-14-add-e2e-tests branch from 0975629 to d507609 Compare July 9, 2026 12:19

github-actions Bot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 9, 2026

github-actions Bot approved these changes Jul 9, 2026

View reviewed changes

nirrozenbaum merged commit 9378530 into llm-d:main Jul 9, 2026
11 checks passed

		@@ -0,0 +1,164 @@
		# Llama model server simulator
		apiVersion: apps/v1

Uh oh!

Conversation

asaadbalum commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Manifest structure

New files

Modified files

What's NOT included

Test plan

Uh oh!

github-actions Bot commented May 4, 2026

Uh oh!

aradhalevy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 6, 2026

Uh oh!

aradhalevy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 11, 2026

Uh oh!

nirrozenbaum commented May 13, 2026

Uh oh!

shmuelk commented May 13, 2026

Uh oh!

nirrozenbaum commented May 13, 2026

Uh oh!

github-actions Bot commented May 17, 2026

Uh oh!

nirrozenbaum commented May 18, 2026

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 31, 2026

Uh oh!

github-actions Bot commented Jun 1, 2026

Uh oh!

github-actions Bot commented Jun 1, 2026

Uh oh!

github-actions Bot commented Jun 24, 2026

Uh oh!

asaadbalum commented Jul 7, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shmuelk left a comment

asaadbalum commented May 4, 2026 •

edited

Loading