Skip to content

Add vllm backend support#47

Open
Riven-Spell wants to merge 4 commits into
yuhp:mainfrom
Riven-Spell:add-vllm-backend-support
Open

Add vllm backend support#47
Riven-Spell wants to merge 4 commits into
yuhp:mainfrom
Riven-Spell:add-vllm-backend-support

Conversation

@Riven-Spell

@Riven-Spell Riven-Spell commented Jul 17, 2026

Copy link
Copy Markdown

No longer having to manually input my models is a great convenience, but it introduces a pain point over the alternative-- Model context length is not persisted in. This PR comes in as an attempt to add support to the model detail enhancement feature for my local AI setup.

For context-- my local setup is a DGX spark cluster running models, mostly under vLLM and sometimes SGLang. open-webui is hooked up to these instances, and pipes forward the detail about model length. This property is read by this PR, and appended to the model config.

Changes

  • npm audit fix was ran
  • ModelInfoFormat was converted to a enum, and a vllm option was added.
  • MODEL_INFO_ENRICHERS was updated to include a new VLLM model info enricher.
    • The new model info enricher extracts it from the raw model properties struct returned by the backend API
    • modelConfig.limit is set with the new context length
  • Certain properties of the factory were changed to optional to reduce unnecessary noise when creating a new model info enricher.
  • Tests were added, with their logical consistency human verified

AI usage

Local DeepSeek v4 flash instance was used, manually reviewing each code change produced, with a thorough planning session. I admittedly don't know typescript myself, but the output seems logically consistent upon review.

Testing

Automated tests were added to the vitest suite to validate backwards compatibility due to the rotation to an enum. Additionally, tested that the model info enhancer could extract the expected detail. Manually tested by plugging into opencode's config.

@yuhp

yuhp commented Jul 17, 2026

Copy link
Copy Markdown
Owner

Thanks for the PR!

I have one question about the limit values before merging.

My understanding is that vLLM's max_model_len represents the total sequence length available for a request, shared by input tokens and generated output tokens.

If that's the case, setting all three fields to the same value:

{
  "limit": {
    "context": 8192,
    "input": 8192,
    "output": 8192
  }
}

could falsely imply that a request may use 8,192 input tokens and generate another 8,192 output tokens, essentially doubling and exceeding the model's actual 8,192-token context window.

Would it be safer/more accurate to set only:

{
  "limit": {
    "context": 8192
  }
}

unless the backend explicitly exposes a separate, reliable maximum-output-token value (like max_tokens in some configurations)?

Please let me know if there is vLLM behavior or documentation I may be missing here.

@Riven-Spell

Riven-Spell commented Jul 17, 2026

Copy link
Copy Markdown
Author

A output value is required. Otherwise opencode crashes with a failure to parse the config, same as manual configuration.

Unfortunately there's no property like that presented by the back end so far as I can tell.

The input line could be removed and become uncapped. This is how I have it in my local branch currently and manual configuration. Opencode does not super thoroughly document this feature, and I am not certain it was actually intended for public consumption.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants