AI proxy may be buggy

[donmor]

I'm tring out new AI extensions in this repository. I tried setting endpoint to locally-hosted llama.cpp (without api key), but it doesn't work. I also tried ai-llm-proxy, setting OCIS_URL and LLM_ENDPOINT, and extension endpoint to 127.0.0.1:3030, won't work either. I then proxied 127.0.0.1:3030 to /ai-llm-proxy/ in nginx, nothing changed. The ai-llm-proxy always respond {"error":"Not found"} on v1/model.

Did I miss something? I thought AI extensions can just use any keyless api endpoint and ai-llm-proxy just authorize user and proxy keyed endpoints.

[LukasHirt]

Thanks for the report. This is a config issue, and it does point to a real gap in our docs (packages/ai-llm-proxy has no README).

Why v1/model 404s: ai-llm-proxy only implements one route:

POST /v1/chat/completions

Everything else, including a models-list endpoint, returns 404 by design. It's not a general OpenAI API proxy.

Why it fails even on the right route: the proxy checks that the request's Origin header matches OCIS_URL exactly and rejects anything else with 403. So it has to be reverse-proxied under the same origin as your oCIS instance, not exposed on its own host:port (127.0.0.1:3030 won't work), and your nginx setup only works if it's the same origin serving oCIS itself.

Example config:

llm:
  endpoint: "https://your-ocis.example.com/ai-llm-proxy/v1"
  model: "llama3.1:70b"

with your reverse proxy forwarding /ai-llm-proxy/* to ai-llm-proxy:3030/* (prefix stripped), and the proxy's own LLM_ENDPOINT pointing at your llama.cpp server.

There's also no mode where an extension talks to a keyless LLM directly. Every AI extension goes through ai-llm-proxy, since it validates the oCIS bearer token first.

We'll add a packages/ai-llm-proxy/README.md covering this.

[donmor]

Still fails with nginx configured... This time I found that clientAbortController.signal.aborted is set to true after readBody(req, MAX_BODY_BYTES) and before await fetch(...), this always fails. I have to move req.on(...) below readBody block to solve this.