Add OpenAI-compatible vision backend for photo intake#1
Merged
Conversation
Adds a third `vision_provider` — "openai" — alongside the existing anthropic and ollama backends. It targets any endpoint speaking the OpenAI Chat Completions API (OpenAI, Azure OpenAI, OpenRouter, or a local server such as vLLM / LM Studio / LocalAI) via a configurable base URL, so one branch covers the whole family. - services/vision.py: `_detect_openai` (httpx, no SDK) with base64 data-URI images and json_object structured output; dispatch mirrors the Anthropic path (multi-image request up to MAX_TILES_PER_REQUEST, per-tile fallback beyond it). Extracts the shared JSON_ONLY_SUFFIX. - config.py / tiling.py: OpenAI ingest cap (operator-tunable long edge, default 2048) so high-res tiling kicks in; cost estimate stays None since pricing varies across compatible endpoints. - routers/settings.py + templates: base URL, API key, model, and image size fields; provider allowlist and independent key-clear handling. - crypto.py: openai_api_key added to SENSITIVE_KEYS (encrypted at rest). - Tests for provider dispatch, tiling, custom base URL, auth errors, and sensitive-key masking; README / CHANGELOG / Docker Hub docs updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dgahagan
approved these changes
Jul 13, 2026
dgahagan
left a comment
Owner
There was a problem hiding this comment.
Thanks for this, @mattbasta — and welcome; this is the first outside contribution to Shelf. 🎉
I checked out the branch and ran the full suite locally: 478 unit tests pass, the CSRF lint tripwire is clean, and it merges cleanly against main. Beyond that, it's a genuinely well-crafted change, and it clearly slots into the existing conventions rather than fighting them:
_detect_openaimirrors the Anthropic/Ollama backends exactly — samemax_tokens, timeout, andVisionErrormessaging — and reuses_clean,_prompt_for, andmerge_tile_booksinstead of reimplementing them.- Using raw
httpx(no SDK) keeps it mockable with the existingrespxsetup and lets one branch cover the whole OpenAI-compatible family. - Nice touch extracting
JSON_ONLY_SUFFIXand pointing the Ollama path at it too — that's a real dedup, not a copy-paste. - Security is handled the way I'd want:
openai_api_keyadded toSENSITIVE_KEYS(encrypted at rest + masked), with a test proving the clear-checkbox only touches the OpenAI key. And I appreciated the details —json_objectmode paired with an explicit "JSON" mention in the prompt, the 401/403 check ordered ahead of the generic error, and cost estimate correctly leftNonesince pricing varies across endpoints. - Tests, README, CHANGELOG, Docker Hub docs, and both templates are all updated. Really complete.
Two non-blocking notes, both things the existing backends already do the same way — no changes needed:
- A
finish_reason: "length"truncation falls through to the generic "unreadable response" error. The Anthropic path behaves identically, so this is consistent. max_tokensvsmax_completion_tokens— your comment already calls this out, and it's the right call for the vision models in scope here.
Approving — thanks for keeping it in line with the surrounding code and for the thorough test coverage. Exactly the kind of contribution I was hoping to get.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hey @dgahagan, I hope you're interested in accepting contributions. I have a homelab that I've been building out, and Shelf is the perfect solution for what I'm looking for. I run litellm+vllm with Gemma 4, though, and that exposes an OpenAI-compatible API rather than Anthropic or Ollama.
I leaned on @claude for this, but I tried to keep the PR in line with the conventions of the adjacent code. Please let me know if you have concerns.
Details
Adds a third
vision_provider— "openai" — alongside the existing anthropic and ollama backends. It targets any endpoint speaking the OpenAI Chat Completions API (OpenAI, Azure OpenAI, OpenRouter, or a local server such as vLLM / LM Studio / LocalAI) via a configurable base URL, so one branch covers the whole family._detect_openai(httpx, no SDK) with base64 data-URI images and json_object structured output; dispatch mirrors the Anthropic path (multi-image request up to MAX_TILES_PER_REQUEST, per-tile fallback beyond it). Extracts the shared JSON_ONLY_SUFFIX.