Skip to content

perf: batch provider model refreshes into one providers.json write - #4269

Merged
atomantic merged 2 commits into
mainfrom
claim/issue-4155
Aug 15, 2026
Merged

perf: batch provider model refreshes into one providers.json write#4269
atomantic merged 2 commits into
mainfrom
claim/issue-4155

Conversation

@atomantic

Copy link
Copy Markdown
Owner

Summary

A post-install/delete Ollama fan-out refreshed each matching provider through refreshProviderModels, and every one of those ends in saveProviders — a cache invalidate plus a full rewrite of providers.json. N providers meant N whole-file writes, each immediately superseded by the next, plus N cache invalidate/repopulate cycles for any concurrent getAllProviders()/getProviderById() reader in that window. Commit cba168e already deduped the probes by daemon base URL and 8d10c48 serialized them; the batch write entry point this issue asks for was the piece still missing.

Adds refreshProviderModelsBatch(ids) to the toolkit's provider service (server/lib/aiToolkit/providers.js). Three phases:

  1. Group by ollamaRefreshGroupKey, so providers sharing a daemon and a probe shape are probed once. A null key stays the "not a shared Ollama probe" sentinel — those providers each become a group of one, never a bucket.
  2. Probe one lead per group, sequentially, persisting nothing — a probe that fails or answers late cannot leave a half-written file.
  3. Apply + save once. The providers map is re-read after the probes (they are network-bound and outlive the read cache's TTL), every probed list is applied in one pass, and saveProviders runs exactly once — or not at all when nothing was probed successfully.

It never throws for a per-provider failure: each group carries its own status — updated (probed; [] is a real empty catalog and IS persisted), failed (probe threw, error attached, stored lists untouched), missing (no such provider, or the lead was deleted mid-probe). That lets the host log one line per group instead of one per member.

refreshOllamaBackedProviders in server/services/localLlm.js collapses to the ollama-backed filter plus that logging — the grouping, sentinel juggling and per-provider write loop all move into the toolkit next to the group-key helper they belong to. server/services/providers.js re-exports the new entry point, and server/lib/aiToolkit/CLAUDE.md records the contract ("refreshing more than one provider goes through the batch form, never a loop over refreshProviderModels").

Test plan

  • New server/lib/aiToolkit/providers.batch.test.js (9 tests) runs the real service against a temp providers.json with a counting delegating spy on atomicWrite — the write count is the whole point, so the spy still performs the real write rather than stubbing it:
    • three providers across two daemons → atomicWrite called exactly once, one /api/tags probe per daemon, every member ends up with its group's models
    • a legitimately empty catalog ([]) is persisted, still in one write
    • a failed shared probe writes nothing and leaves the stored lists untouched
    • one failing group does not cost the healthy groups their update (still one write)
    • an unknown id reports missing without probing or blocking the rest
    • an api-type Ollama provider keeps its own (unfiltered) probe rather than joining the tool-filtered bucket for the same daemon
    • parity: the batch lands the same stored state as refreshing each provider one by one, at 1 write instead of 3
    • empty/undefined id list is a no-op; a duplicated id is probed and written once
  • server/services/localLlm.test.js updated to assert the fan-out issues ONE batch call carrying exactly the ollama-backed ids, skips the call entirely when nothing matches or the pull failed, logs one line per failed/missing group, stays silent for updated groups, and survives a rejected batch.
  • cd server && NODE_ENV=test npm test — 1396 files / 29263 tests passing, 26 DB-backed suites skipped as designed (no DB-backed suites run).

Closes #4155

…4155)

A post-install Ollama fan-out refreshed each matching provider through
refreshProviderModels, and every one of those ends in saveProviders — a
cache invalidate plus a full rewrite of providers.json. N providers meant
N whole-file writes, each immediately superseded by the next, and N
cache invalidate/repopulate cycles for any concurrent reader.

Adds refreshProviderModelsBatch(ids) to the toolkit's provider service:
it groups by ollamaRefreshGroupKey (one probe per daemon + probe shape),
probes each group's lead without persisting, then applies every result
and calls saveProviders exactly once. Per-group statuses (updated /
failed / missing) replace the host-side sentinel juggling, so localLlm's
refreshOllamaBackedProviders is now just the ollama-backed filter plus
one log line per skipped group.
@atomantic
atomantic merged commit 2137b20 into main Aug 15, 2026
7 checks passed
@atomantic
atomantic deleted the claim/issue-4155 branch August 15, 2026 05:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Batch provider-refresh writes instead of one full providers.json save per provider

1 participant