Sometimes it is desirable not to expose source code/etc. to remote services. Paranoid and/or cheap people (like me) sometimes run LLMs on one or more local GPUs using a program like llama.cpp (or vllm on multi-GPU servers) in order to do AI assisted programming locally. Most of the local LLM execution engines present a ("mostly") OpenAI compable API, which is possibly similar to the APIs presented by a few of the commercial AI services (namely OpenAI).
Is it possible to use symposium with the "OpenAI compatible" API presented by any of the popular local LLM execution engines? If so, is it documented anyplace?
I'm excited for some of the features offered by symposium, but I'm not excited about sending my company's source code or my budget (via per tokens fees, or subscriptions) to third parties.
Sometimes it is desirable not to expose source code/etc. to remote services. Paranoid and/or cheap people (like me) sometimes run LLMs on one or more local GPUs using a program like llama.cpp (or vllm on multi-GPU servers) in order to do AI assisted programming locally. Most of the local LLM execution engines present a ("mostly") OpenAI compable API, which is possibly similar to the APIs presented by a few of the commercial AI services (namely OpenAI).
Is it possible to use symposium with the "OpenAI compatible" API presented by any of the popular local LLM execution engines? If so, is it documented anyplace?
I'm excited for some of the features offered by symposium, but I'm not excited about sending my company's source code or my budget (via per tokens fees, or subscriptions) to third parties.