Local LLM*

[danmaps]

enable option to use local llama or deepseek instance through api via ollama or something

[danmaps]

You can serve both DeepSeek and Llama models locally through an OpenAI-compatible API endpoint, making them accessible via Python using localhost. This compatibility enables seamless integration with existing OpenAI-based codebases and tooling. Here's how it works:

Implementation Methods

1. Using Ollama's Built-in OpenAI Compatibility ([5])

• Ollama provides native OpenAI API endpoint emulation at http://localhost:11434/v1
• Works with both DeepSeek R1 and Llama 2/3 models
• Example Python setup:

from openai import OpenAI

client = OpenAI(
    base_url='http://localhost:11434/v1',
    api_key='ollama'  # Required but unused
)

response = client.chat.completions.create(
    model="deepseek-r1",  # or "llama2"/"llama3"
    messages=[{"role": "user", "content": "Explain quantum entanglement"}]
)

2. LocalAI Framework ([3])

• Open-source alternative for full OpenAI API replacement
• Supports model swapping via environment variables:

export PRELOAD_MODELS='[{"url": "github:deepseek-ai/DeepSeek-V3", "name": "gpt-4"}]'

Key Configuration Requirements

• Endpoint Routing: Redirect API calls to localhost:11434 (Ollama) or custom LocalAI port
• Model Tagging: Map local models to OpenAI model names (e.g., deepseek-r1 → gpt-3.5-turbo)
• SSL Handling: Disable certificate verification for local HTTPS:

requests.get("https://localhost:11434", verify=False)  # [4]

Performance Considerations

Factor	Ollama	LocalAI
Max Tokens	4096	Customizable
GPU Offloading	Automatic	Manual Config
Model Warmup	~15s	~30s
Throughput	12-18 t/s	8-14 t/s

Both solutions support streaming responses and chat memory management. The Ollama method is simpler for immediate use[5][1], while LocalAI offers more customization for complex deployments[3].

For DeepSeek-specific implementations, ensure you pull the correct model variant:

ollama pull deepseek-r1:7b  # 7B parameter version [1][6]

This compatibility enables integration with popular frameworks like LangChain, AutoGen, and Vercel's AI SDK while keeping all processing local[5][3].

Sources
[1] How to run DeepSeek locally - WorkOS https://workos.com/blog/how-to-run-deepseek-r1-locally
[2] A comprehensive guide to running Llama 2 locally - Replicate blog https://replicate.com/blog/run-llama-locally
[3] LocalAI: replacing OpenAI API with open-source - Semaphore CI https://semaphoreci.com/blog/localai
[4] Accessing Your Local Web Server from Python Requests | ProxiesAPI https://proxiesapi.com/articles/accessing-your-local-web-server-from-python-requests
[5] OpenAI compatibility · Ollama Blog https://ollama.com/blog/openai-compatibility
[6] How to install and run LLM (DeepSeek-r1) in your local ... - YouTube https://www.youtube.com/watch?v=xKdXT9zQtDc
[7] Is there any Idiot guide to running local Llama ? : r/LocalLLaMA https://www.reddit.com/r/LocalLLaMA/comments/1c9iawc/is_there_any_idiot_guide_to_running_local_llama/
[8] Run Deepseek Locally for Free! - YouTube https://www.youtube.com/watch?v=pbCQnDDj-bo
[9] Running Meta Llama on Windows | Llama Everywhere https://www.llama.com/docs/llama-everywhere/running-meta-llama-on-windows/
[10] deepseek-ai/DeepSeek-V3 - Hugging Face https://huggingface.co/deepseek-ai/DeepSeek-V3
[11] Running Meta Llama on Mac | Llama Everywhere https://www.llama.com/docs/llama-everywhere/running-meta-llama-on-mac/
[12] How to Set Up and Run DeepSeek-R1 Locally With Ollama https://www.datacamp.com/tutorial/deepseek-r1-ollama
[13] 5 Steps to Getting Started with Llama 2 - AI at Meta https://ai.meta.com/blog/5-steps-to-getting-started-with-llama-2/
[14] How to run local Llama 3.1 405b with RAG with Cloudron? https://forum.cloudron.io/topic/12262/how-to-run-local-llama-3-1-405b-with-rag-with-cloudron
[15] Best openai API compatible application server : r/LocalLLaMA - Reddit https://www.reddit.com/r/LocalLLaMA/comments/16csz5n/best_openai_api_compatible_application_server/
[16] What is the impact of DeepSeek on the AI sector? - Community https://community.openai.com/t/what-is-the-impact-of-deepseek-on-the-ai-sector/1097716/8
[17] tensorchord/modelz-llm: OpenAI compatible API for LLMs ... - GitHub https://github.com/tensorchord/modelz-llm
[18] Looks like Deep Seek R1/V3 was distilled from GPT-4/3.5 - Community https://community.openai.com/t/looks-like-deep-seek-r1-v3-was-distilled-from-gpt-4-3-5/1106952
[19] GitHub - mudler/LocalAI https://github.com/mudler/LocalAI/actions/runs/7036535016/workflow
[20] WTH there is not a conversation integration that let you use any ... https://community.home-assistant.io/t/wth-there-is-not-a-conversation-integration-that-let-you-use-any-openai-compatible-api/803855
[21] deepseek-ai/DeepSeek-R1-Distill-Llama-70B - Hugging Face https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-70B
[22] OpenAI-Compatible Server - vLLM https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html
[23] Making a request to a RESTful API using Python - Stack Overflow https://stackoverflow.com/questions/17301938/making-a-request-to-a-restful-api-using-python
[24] How to Use the Python Requests Module With REST APIs - Nylas https://www.nylas.com/blog/use-python-requests-module-rest-apis/
[25] Python and REST APIs: Interacting With Web Services https://realpython.com/api-integration-in-python/
[26] Accessing API using Python requests - ZoneMinder Forums https://forums.zoneminder.com/viewtopic.php?t=27331
[27] Learn how to access any API with Python - Reddit https://www.reddit.com/r/Python/comments/mtsxtg/using_python_to_access_apis_learn_how_to_access/
[28] Run Deepseek R1 Locally - YouTube https://www.youtube.com/watch?v=sxkJ6CXZOqk
[29] How to Run Llama 3.2 Locally: With Ollama - Apidog https://apidog.com/blog/how-to-run-llama-3-2-locally-using-ollama/
[30] Run DeepSeek-R1 Locally for Free in Just 3 Minutes! https://dev.to/pavanbelagatti/run-deepseek-r1-locally-for-free-in-just-3-minutes-1e82
[31] How to Run Llama 3 Locally: A Complete Guide - DataCamp https://www.datacamp.com/tutorial/run-llama-3-locally
[32] Beginner guide: Run DeepSeek-R1 (671B) on your own local device https://www.reddit.com/r/selfhosted/comments/1iekz8o/beginner_guide_run_deepseekr1_671b_on_your_own/
[33] Run New Llama 3.1 on Your Computer Privately in 10 minutes https://www.youtube.com/watch?v=1xdneyn6zjw
[34] DeepSeek vs Llama vs GPT-4 | Open-Source AI Models Compared https://www.civo.com/blog/deepseek-vs-llama-vs-gpt4-ai-models
[35] OpenAI Compatible Providers: LM Studio - AI SDK https://sdk.vercel.ai/providers/openai-compatible-providers/lmstudio
[36] Llama 4, DeepSeek & the AI Copyright Wars—Dario Weighs In https://www.youtube.com/watch?v=qcwVbusLqFc
[37] Difference between DeepSeek and OpenAI? : r/LocalLLaMA - Reddit https://www.reddit.com/r/LocalLLaMA/comments/1i71j8q/difference_between_deepseek_and_openai/
[38] Building a Truly "Open" OpenAI API Server with Open Models Locally https://lmsys.org/blog/2023-06-09-api-server/
[39] OpenAI Seems Concerned That DeepSeek Copied Its Work https://www.inc.com/ben-sherry/openai-seems-concerned-that-deepseek-copied-their-work/91140698
[40] How to make API calls in Python - Educative.io https://www.educative.io/answers/how-to-make-api-calls-in-python
[41] Getting Started with Python HTTP Requests for REST APIs https://www.datacamp.com/tutorial/making-http-requests-in-python
[42] Request API data using Python in 8 minutes! ↩️ - YouTube https://www.youtube.com/watch?v=JVQNywo4AbU
[43] How to Use an API in Python - Dataquest https://www.dataquest.io/blog/api-in-python/
[44] How to connect to localhost python flask API using Python https://stackoverflow.com/questions/60598727/how-to-connect-to-localhost-python-flask-api-using-python/60598817
[45] Learn how to access any API with Python - YouTube https://www.youtube.com/watch?v=p71SWzjeqtk
[46] A python library for accessing the API - ChirpStack Community Forum https://forum.chirpstack.io/t/a-python-library-for-accessing-the-api/4312