feat(providers): add vllm provider (self-hosted vLLM via Responses API) - #130
Merged
Conversation
- Added vllm to _VALID_PROVIDER_MODULES in loader.py - Registered vllm in KNOWN_PROVIDERS and PROVIDER_CATALOG - Implemented resolve_credential_detailed branch routing VLLM_BASE_URL to base_url with optional VLLM_API_KEY - Declared provider-vllm in bundle.md install-only stub 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
- Updated README.md: added vllm to provider list - Updated CHANGELOG.md: documented vllm provider addition - Updated CONFIGURATION.md: added vllm configuration section - Updated providers-and-models.md: documented vllm provider specifications - Updated host-config.md: added vllm environment variables - Updated bundle-and-cache.md: noted vllm as install-only provider - Updated LAYERS_AND_RELEASES.md: version reference - Updated INTEGRATION.md: vllm integration notes - Updated architecture.dot: added vllm provider node - Updated SKILL.md: documented vllm in provider catalog 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
main added gemini as the eighth provider (#129) while this branch added vllm as the eighth. Both are now registered, making nine, and every "eight providers" reference has been updated to match. Conflicts resolved by keeping both providers in each list: - provider_sources.py: KNOWN_PROVIDERS and PROVIDER_CATALOG (verified in sync) - config/loader.py: _VALID_PROVIDER_MODULES, reflowed to one name per line - bundle/bundle.md: both install-only provider stubs - CHANGELOG.md: the vllm entry moved from [0.13.0] to [Unreleased], since 0.13.0 was cut on main before this branch's work landed - README.md, LAYERS_AND_RELEASES.md, architecture.dot, SKILL.md, INTEGRATION.md, spec/host-config.md, spec/bundle-and-cache.md, spec/providers-and-models.md: provider counts and lists 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
`amplifier-agent models list --provider vllm` failed against a vLLM server
with no auth -- the common self-hosted deployment -- with:
OpenAIError: Missing credentials. Please pass an `api_key` ... or set
the `OPENAI_API_KEY` environment variable.
`run` was unaffected, so the two paths disagreed about the same server.
Cause: the two paths resolve the key differently. `run` goes through the
provider module's mount(), which reads `os.environ.get("VLLM_API_KEY",
"EMPTY")`, so an absent key correctly yields the module's placeholder.
`models list` never calls mount(); it builds the provider directly from the
resolved credential fields via `_try_instantiate_provider`, which substitutes
`api_key=""` for a field that is not present. VLLMProvider's signature is
`(base_url, *, api_key="EMPTY", config=...)`, so it matches that helper's
base_url+api_key+config attempt and receives the empty string, overriding its
own default. The OpenAI SDK then rejects the empty key with a message naming
`OPENAI_API_KEY`, which is not a variable this provider consults.
Fix: the vllm credential branch now always populates `fields["api_key"]`,
falling back to the same `"EMPTY"` placeholder the module itself uses, so both
paths agree.
That placeholder is not a resolved credential, so it must not win over
configuration: `build_provider_entry` drops it from protected-key
re-assertion when host config supplied a real `api_key`. This preserves the
guarantee the chat-completions branch documents -- a key set in
`provider.config` is never silently replaced by a "no key needed" default --
while a genuine `VLLM_API_KEY` from the environment still takes precedence as
before.
chat-completions is unaffected: its constructor accepts neither `base_url` nor
`api_key`, so it falls through to the config-only attempt and reads its own
environment, never receiving the empty string.
Verified end to end against a live vLLM server in the DTU, and across the
precedence matrix: keyless, keyless + host-config key, env key + host-config
key, and host-config base_url override.
🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)
Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Covers the four things amplifier-agent became responsible for when vllm was
registered: the credential report knows it and resolves its endpoint from the
environment, its models list off that endpoint, a host config naming it
validates, and a session runs on it. Model quality is out of scope -- a
self-hosted vLLM is usually serving a small open-weight model, so the smoke
case asserts one literal token, case-insensitively and as a substring. Anything
stricter would report on the operator's model choice rather than on the wiring.
The suite is opt-in and never affects a default `cli.py run`. It skips when
VLLM_BASE_URL is unset, and skips again when the endpoint is set but not
answering: a server being down is a fact about the operator's machine, not a
defect in amplifier-agent. Both checks run inside the container, which is where
they matter. VLLM_MODEL is optional; when unset the first model id the server
advertises is used, which is correct for a single-model vLLM process.
Endpoint plumbing (dtu_manager._build_varmap plus provisioning/setup-vllm-env.sh):
- The VLLM_* trio travel as DTU `--var` values rather than `passthrough`
entries. Passthrough copies host values verbatim, but the server runs on the
host and inside the container `localhost` is the container. DTU rewrites
localhost to the bridge gateway IP in var values, which is the mechanism
GITEA_URL already depends on, so the endpoint can be written exactly as it is
used on the host.
- An empty value is omitted rather than exported as an empty string. The
provider module resolves its key as `os.environ.get("VLLM_API_KEY", "EMPTY")`,
and that default applies only when the variable is absent; exporting it empty
defeats it and surfaces as an unrelated "set OPENAI_API_KEY" error.
- The vLLM host is appended to no_proxy. DTU exempts only loopback, so without
this the traffic would route through the interception proxy, which buffers
whole response bodies and destroys streaming.
Verified against a live vLLM server: 4 passed keyless with model
auto-discovery, 4 skipped with no endpoint configured, and 16 passed across
adjacent suites confirming the shared profile change is inert for them.
🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)
Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
vllmas a first-class provider in amplifier-agent's provider catalog, wiring up the existing official moduleamplifier-module-provider-vllm— self-hosted or remote vLLM servers (open-weight models like gpt-oss) via vLLM's OpenAI-compatible Responses API, with reasoning-block and tool-calling support. Listed in the ecosystemMODULES.md; completes the last of the catalog gaps (sibling to #127 / #128 / #129).Note it uses the Responses API (
/v1/responses) — distinct fromchat-completions(Chat Completions wire), and endpoint-agnostic unlikeopenai.What changed
Code
config/loader.py: addvllmto_VALID_PROVIDER_MODULES.provider_sources.py: add toKNOWN_PROVIDERSandPROVIDER_CATALOG; add a dedicatedresolve_credential_detailedbranch that readsVLLM_BASE_URL(required) and optionalVLLM_API_KEY, routing base_url intofields["base_url"](mirrorschat-completions; a genericPROVIDER_CREDENTIAL_VARSentry would misroute base_url intoapi_key).bundle.md: declareprovider-vllmin the install-only stub list.Docs: README,
docs/spec/*,docs/CONFIGURATION.md,docs/LAYERS_AND_RELEASES.md,docs/INTEGRATION.md,docs/architecture/architecture.dot,skills/amplifier-agent/SKILL.md,CHANGELOG.md.Also incidentally corrects two pre-existing stale provider counts in
docs/spec/host-config.mdthat had fallen behind earlier catalog merges.Verification (Digital Twin, real vLLM server — no mocks)
Verified end-to-end in an isolated container running the patched build (
amplifier-agent 0.12.0) against a real vLLM server (vllm/vllm-openaion GPU, servingQwen/Qwen2.5-1.5B-Instructwith its Responses API):providers list:vllmappears;resolvable=false/source=nonewithVLLM_BASE_URLunset, flips toresolvable=true/source=envwhen set.provider.moduleis rejected with a valid set that now includesvllm.amplifier-agent runreturned a genuine, non-empty completion, and the vLLM access log shows the request hitPOST /v1/responses(count:/v1/responses=1,/v1/chat/completions=0) — proving it used the Responses API viaprovider-vllm, not a fallback.vllmand onlyVLLM_BASE_URLset (no other provider resolvable), stopping the vLLM server made the same turn fail with a connection error and restarting it made it succeed.bundle.md; the prepared bundle declaresprovider-vllm.