Add vllm backend support#47
Conversation
|
Thanks for the PR! I have one question about the limit values before merging. My understanding is that vLLM's max_model_len represents the total sequence length available for a request, shared by input tokens and generated output tokens. If that's the case, setting all three fields to the same value: {
"limit": {
"context": 8192,
"input": 8192,
"output": 8192
}
}could falsely imply that a request may use 8,192 input tokens and generate another 8,192 output tokens, essentially doubling and exceeding the model's actual 8,192-token context window. Would it be safer/more accurate to set only: {
"limit": {
"context": 8192
}
}unless the backend explicitly exposes a separate, reliable maximum-output-token value (like max_tokens in some configurations)? Please let me know if there is vLLM behavior or documentation I may be missing here. |
|
A output value is required. Otherwise opencode crashes with a failure to parse the config, same as manual configuration. Unfortunately there's no property like that presented by the back end so far as I can tell. The input line could be removed and become uncapped. This is how I have it in my local branch currently and manual configuration. Opencode does not super thoroughly document this feature, and I am not certain it was actually intended for public consumption. |
No longer having to manually input my models is a great convenience, but it introduces a pain point over the alternative-- Model context length is not persisted in. This PR comes in as an attempt to add support to the model detail enhancement feature for my local AI setup.
For context-- my local setup is a DGX spark cluster running models, mostly under vLLM and sometimes SGLang.
open-webuiis hooked up to these instances, and pipes forward the detail about model length. This property is read by this PR, and appended to the model config.Changes
npm audit fixwas ranModelInfoFormatwas converted to a enum, and avllmoption was added.MODEL_INFO_ENRICHERSwas updated to include a new VLLM model info enricher.I'm not entirely sure settinginputandoutputto these default values is correct--outputis required, however. This could probably be safely set to a multiple of, but I would like feedback from other humans here.max_model_lenis the most vLLM will natively return, so setting the output limit there is reasonable.AI usage
Local DeepSeek v4 flash instance was used, manually reviewing each code change produced, with a thorough planning session. I admittedly don't know typescript myself, but the output seems logically consistent upon review.
Testing
Automated tests were added to the
vitestsuite to validate backwards compatibility due to the rotation to an enum. Additionally, tested that the model info enhancer could extract the expected detail. Manually tested by plugging into opencode's config.