Skip to content

Add OpenAI-compatible vision backend for photo intake#1

Merged
dgahagan merged 1 commit into
dgahagan:mainfrom
mattbasta:openai
Jul 13, 2026
Merged

Add OpenAI-compatible vision backend for photo intake#1
dgahagan merged 1 commit into
dgahagan:mainfrom
mattbasta:openai

Conversation

@mattbasta

Copy link
Copy Markdown
Contributor

Hey @dgahagan, I hope you're interested in accepting contributions. I have a homelab that I've been building out, and Shelf is the perfect solution for what I'm looking for. I run litellm+vllm with Gemma 4, though, and that exposes an OpenAI-compatible API rather than Anthropic or Ollama.

I leaned on @claude for this, but I tried to keep the PR in line with the conventions of the adjacent code. Please let me know if you have concerns.

Details

Adds a third vision_provider — "openai" — alongside the existing anthropic and ollama backends. It targets any endpoint speaking the OpenAI Chat Completions API (OpenAI, Azure OpenAI, OpenRouter, or a local server such as vLLM / LM Studio / LocalAI) via a configurable base URL, so one branch covers the whole family.

  • services/vision.py: _detect_openai (httpx, no SDK) with base64 data-URI images and json_object structured output; dispatch mirrors the Anthropic path (multi-image request up to MAX_TILES_PER_REQUEST, per-tile fallback beyond it). Extracts the shared JSON_ONLY_SUFFIX.
  • config.py / tiling.py: OpenAI ingest cap (operator-tunable long edge, default 2048) so high-res tiling kicks in; cost estimate stays None since pricing varies across compatible endpoints.
  • routers/settings.py + templates: base URL, API key, model, and image size fields; provider allowlist and independent key-clear handling.
  • crypto.py: openai_api_key added to SENSITIVE_KEYS (encrypted at rest).
  • Tests for provider dispatch, tiling, custom base URL, auth errors, and sensitive-key masking; README / CHANGELOG / Docker Hub docs updated.

Adds a third `vision_provider` — "openai" — alongside the existing
anthropic and ollama backends. It targets any endpoint speaking the
OpenAI Chat Completions API (OpenAI, Azure OpenAI, OpenRouter, or a
local server such as vLLM / LM Studio / LocalAI) via a configurable
base URL, so one branch covers the whole family.

- services/vision.py: `_detect_openai` (httpx, no SDK) with base64
  data-URI images and json_object structured output; dispatch mirrors
  the Anthropic path (multi-image request up to MAX_TILES_PER_REQUEST,
  per-tile fallback beyond it). Extracts the shared JSON_ONLY_SUFFIX.
- config.py / tiling.py: OpenAI ingest cap (operator-tunable long edge,
  default 2048) so high-res tiling kicks in; cost estimate stays None
  since pricing varies across compatible endpoints.
- routers/settings.py + templates: base URL, API key, model, and image
  size fields; provider allowlist and independent key-clear handling.
- crypto.py: openai_api_key added to SENSITIVE_KEYS (encrypted at rest).
- Tests for provider dispatch, tiling, custom base URL, auth errors, and
  sensitive-key masking; README / CHANGELOG / Docker Hub docs updated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@dgahagan dgahagan left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, @mattbasta — and welcome; this is the first outside contribution to Shelf. 🎉

I checked out the branch and ran the full suite locally: 478 unit tests pass, the CSRF lint tripwire is clean, and it merges cleanly against main. Beyond that, it's a genuinely well-crafted change, and it clearly slots into the existing conventions rather than fighting them:

  • _detect_openai mirrors the Anthropic/Ollama backends exactly — same max_tokens, timeout, and VisionError messaging — and reuses _clean, _prompt_for, and merge_tile_books instead of reimplementing them.
  • Using raw httpx (no SDK) keeps it mockable with the existing respx setup and lets one branch cover the whole OpenAI-compatible family.
  • Nice touch extracting JSON_ONLY_SUFFIX and pointing the Ollama path at it too — that's a real dedup, not a copy-paste.
  • Security is handled the way I'd want: openai_api_key added to SENSITIVE_KEYS (encrypted at rest + masked), with a test proving the clear-checkbox only touches the OpenAI key. And I appreciated the details — json_object mode paired with an explicit "JSON" mention in the prompt, the 401/403 check ordered ahead of the generic error, and cost estimate correctly left None since pricing varies across endpoints.
  • Tests, README, CHANGELOG, Docker Hub docs, and both templates are all updated. Really complete.

Two non-blocking notes, both things the existing backends already do the same way — no changes needed:

  • A finish_reason: "length" truncation falls through to the generic "unreadable response" error. The Anthropic path behaves identically, so this is consistent.
  • max_tokens vs max_completion_tokens — your comment already calls this out, and it's the right call for the vision models in scope here.

Approving — thanks for keeping it in line with the surrounding code and for the thorough test coverage. Exactly the kind of contribution I was hoping to get.

@dgahagan dgahagan merged commit 6742521 into dgahagan:main Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants