[BUG] BenchmarkJob sends a hardcoded model name; requests 404 while the job reports Completed

[weetime]

What happened?

BenchmarkJob sends a hardcoded string as the OpenAI model field. Every request 404s, but the job still reports Completed.

BuildInferenceServiceArgs (pkg/controller/v1beta1/benchmark/utils/utils.go:71, main @ 23c97f1) builds the genai-bench arguments for the endpoint.inferenceService path:

args := map[string]string{
    "--api-key":         "sample-key", // TODO: Use actual service account key later
    "--api-model-name":  "vllm-model",
    "--model-tokenizer": *baseModel.Storage.Path,
}

--api-model-name becomes the model field of every request genai-bench sends. vLLM validates it against --served-model-name and returns 404 on a mismatch, so a runtime that does not happen to serve under the literal name vllm-model fails on every single request.

The failure is silent, which is what makes it expensive. The pod exits 0, the BenchmarkJob status goes to Completed, and a complete set of result files, spreadsheets and plots is written to the output location. Only num_error_requests inside the per-run JSON reveals that nothing succeeded. A user reading the dashboard has no signal that the numbers are meaningless.

Scope

This is narrower than it first looks, and the scoping is worth stating so it can be weighed properly:

	count	--served-model-name
config/runtimes/vllm/	17	all vllm-model
config/runtimes/srt/	189	167 distinct HuggingFace ids, none vllm-model

SGLang echoes request.model back without comparing it to the served name, so the SGLang half has no 404 path. None of the bundled runtime definitions are affected — the hardcoded value matches the vLLM catalogue's convention, and SGLang's leniency covers the rest. That is why it has held up.

What breaks is a user-defined vLLM runtime whose --served-model-name is anything else, which is the normal case once you bring your own runtime.

Background: this was one of two problems reported in #778. That issue bundled it with a separate PVC-scheduling problem and has been closed; this one is filed on its own so it can be judged on its own.

What did you expect to happen?

Either the benchmark uses the name the engine actually serves under, or it fails loudly instead of producing a full set of plausible-looking results from 100% failed requests.

How can we reproduce it (as minimally and precisely as possible)?

1. Define a ClusterServingRuntime whose vLLM runner serves under its own name:

apiVersion: ome.io/v1beta1
kind: ClusterServingRuntime
metadata:
  name: vllm-demo
spec:
  protocolVersions: [openAI]
  engineConfig:
    runner:
      name: ome-container
      image: vllm/vllm-openai:v0.8.5.post1
      command:
        - python3
        - -m
        - vllm.entrypoints.openai.api_server
        - --port
        - "8080"
        - --model
        - $(MODEL_PATH)
        - --served-model-name
        - qwen2-5-0-5b-instruct      # anything other than "vllm-model"

2. Deploy an InferenceService using it, and wait for it to become Ready.

3. Run a BenchmarkJob against that InferenceService:

apiVersion: ome.io/v1beta1
kind: BenchmarkJob
metadata:
  name: demo-bench
spec:
  endpoint:
    inferenceService:
      name: <isvc>
      namespace: <ns>
  task: text-to-text
  trafficScenarios: ["D(100,100)"]
  numConcurrency: [1]
  maxTimePerIteration: 1
  maxRequestsPerIteration: 10
  outputLocation:
    storageUri: "pvc://<pvc>/benchmarks"

4. Observe the generated Job's args contain --api-model-name vllm-model, the BenchmarkJob reaches Completed, and the per-run JSON in the output shows every request errored.

That vLLM rejects an unserved name is directly observable. Against a server started with
--served-model-name qwen2-5-0-5b-instruct vllm-model (the alias added deliberately, as a
workaround for exactly this bug), posting /v1/chat/completions with three different model
values:

qwen2-5-0-5b-instruct -> 200 OK
vllm-model            -> 200 OK
not-a-real-name       -> 404 {"object":"error","message":"The model `not-a-real-name` does not exist.","type":"NotFoundError","code":404}

Remove the vllm-model alias — i.e. configure the runtime the way anyone would who has not hit
this bug — and the second line becomes the third.

Anything else we need to know?

One constraint that is easy to miss when fixing this, and that I got wrong on a first attempt (#779, withdrawn):

Reading --served-model-name from inferenceService.Spec.Engine.Runner alone is not sufficient. In the common case the flag lives in the ServingRuntime template, and spec.engine on the InferenceService carries only minReplicas/maxReplicas with no runner override. An InferenceService-only lookup finds nothing there and silently falls through to whatever fallback is chosen. Whatever source is used has to reflect the runtime template and any InferenceService-level override together, i.e. the effective engine arguments rather than either layer on its own.

For the case where a runtime omits --served-model-name entirely, vLLM's own default is worth matching rather than inventing a convention — vllm/entrypoints/openai/api_server.py:

if args.served_model_name is not None:
    served_model_names = args.served_model_name
else:
    served_model_names = [args.model]

so the model path is the name the server will report. SGLang defaults the same way from --model-path.

Happy to send a PR if the direction is agreed.

Environment

• OME version: main @ 23c97f1
• Kubernetes version: v1.29.15
• Cloud provider or hardware configuration: bare-metal GPU node
• OS: Linux
• Runtime and version: vLLM 0.8.5.post1
• Model being served: Qwen2.5-0.5B-Instruct
• Install method: Helm