feat(deploy): many indexes on one host, each with its own MCP endpoint and UI - #15
Closed
dimittal wants to merge 13 commits into
Closed
feat(deploy): many indexes on one host, each with its own MCP endpoint and UI#15dimittal wants to merge 13 commits into
dimittal wants to merge 13 commits into
Conversation
…t and UI
Adds deploy/fleet — a generator that runs N brains behind one nginx, so each
index is reachable at <host>/<name>/mcp with an explorer at <host>/<name>/ui.
Why one setup rather than one per index: `serve` handles a single brain per
process, so each index needs its own container — but the OpenSearch index name
is already per-brain, so they share one cluster. Measured on a 2-vCPU/3.8GB VM:
the cluster is 983MB once, while an index costs ~255MB (MCP) + ~71MB (UI).
fleet.py generates compose, nginx config and credentials from indexes.yml:
./fleet.py up | add <name> | tokens | status | logs
Two bugs this surfaced, both fixed here:
1. `serve` could not sit behind any reverse proxy. The MCP SDK enables
DNS-rebinding protection with a localhost-only allow-list, so a proxied
request arrives with a foreign Host and is rejected `421 Invalid Host
header`. Passing a non-localhost bind address would have silently disabled
the protection instead, so serve_http now builds an explicit allow-list from
--public-url and the new --allowed-host / OPEN_INDEX_ALLOWED_HOSTS, keeping
the check on. `*` opts out for a trusted proxy.
2. A single shared htpasswd let the `sales` UI password open the `support` UI —
nginx accepts any user present in the file it is given. Credentials are now
per-index files, so the UI isolates the same way the bearer tokens do.
The UI needs a gate at all because Streamlit ships no authentication; published
as-is, anyone reaching the host could read an entire brain. It stays read-only
(writes go through MCP or the CLI) and can be dropped per index with ui: false.
Verified on the VM: both endpoints per index, cross-index credentials rejected
on both doors, data isolation between cluster indexes, Streamlit served under
its path prefix with websockets working.
docs/multi-index.md covers architecture, capacity, LB setup, access and
population. 451 tests pass; mcp_server.py at 100%.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Surfaced by a real dataset: every search against it failed with
Can only use fuzzy queries on keyword and text fields —
not on [fields.reported_date] which is of type [date]
Two causes, both fixed:
- `_search_fields` listed every field whose `search` was not "none",
regardless of declared type. OpenSearch maps `number` to long and
`timestamp` to date, and rejects the *entire* multi_match if either
appears — so one numeric field in a schema broke all search for that
brain. Only string/text fields go into the full-text query now.
- `reported_date` was declared `string`, but its date-shaped values were
mapped as `date` by dynamic detection, which produced the same rejection
for a correctly-declared field. `date_detection` is off; the schema
decides the type, not a guess from the first document indexed.
The synthetic test brains had no numeric or date-like fields, which is why
this survived until real data arrived. Four regression tests cover both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ensure_schema runs once at open. If the index is dropped afterwards — cluster rebuild, manual DELETE, restored snapshot — OpenSearch auto-creates it on the next write with dynamic mapping: doc_type becomes text (every aggregation fails) and date-shaped strings become dates (fuzzy search fails). Both surface long after the write that caused them. One existence check on the write path is cheaper than that debugging. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The container writes with umask 022, so any directory it creates loses group write and the operator can no longer drop a CSV in or rsync data across without sudo — which is exactly what happened loading the first real dataset.
rmtree + mkdir swaps the directory inode. nginx has the old one bind-mounted, so it keeps serving an empty directory: the credential files are plainly present on the host while nginx logs 'No such file or directory' and every UI request 403s. Prune by deleting files instead.
Each of these boots a real Streamlit script run; 60s flaked once under the full suite and would be likelier to on a CI runner.
- The help page moves from rightmost to leftmost and is relabelled "?". Streamlit selects the first tab, so a first-time visitor now lands on the explanation of what this index is and how to connect to it, rather than having to go looking for it. - New Schema tab between "?" and Explore: every doc_type with its fields — type, how it is searched, ranking weight, required — and its relationship vocabulary. Declared and observed edges are merged into one table, so a declared edge with zero uses and an undeclared edge in heavy use are both visible; either is worth noticing before writing to the index. - TAB_GUIDE is now the single source of tab order: the page builds its tabs from it and the "what each tab does" section reads from it, so the two cannot drift. A test asserts the rendered tabs match it. 468 tests pass; ui/view.py at 100%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The map was unreadable: long entity names drawn beside every dot overlapped each other, and anchoring on a few entities is the wrong default for someone who has never seen the index and has no entity in mind. - New build_overview_graph: every entity of the selected doc_types, coloured by type, with the edges that run between them. Only edges with both ends in scope are drawn — a half-edge to a filtered-out type is a line to nowhere. - Doc-type filter at the top; a legend beside the canvas showing each type's colour and how many nodes it contributes. The legend is built from the drawn graph, not the schema, so it describes what is actually on screen. - Node labels are truncated to 22 chars; the full name, type, id and a few fields move to the hover tooltip. Edge labels are dropped past 60 edges — overlapping relationship text on every line was the main offender — and the relationship stays on the edge's tooltip. - Clicking a node focuses on it and its neighbours, with a way back, instead of silently accumulating anchors. - Canvas narrowed to 950px so it fits beside the legend column. Nodes are capped at 250, keeping the highest-degree ones, and the page says so when the cap bites rather than presenting a subset as the whole index. 486 tests pass; ui/view.py at 100%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A bare question mark is a small target and reads as an afterthought. Since this is the tab the page opens on, it should say what it is.
Truncating node labels was not enough. Entity names here are long and arbitrary
("N412NL 2026-04-12 — Crew reported WING A.ICE VLV OPEN L on climb out of PDX"),
and drawn beside every dot they overlap each other and their own edges however
short they are cut. On 156 nodes the canvas was still unreadable.
The canvas now carries shape and colour only: dots coloured by doc_type, the
legend explaining the colours, and the full detail — name, type, id, fields, and
for edges the relationship — on hover. Edge labels go the same way, which also
removes the node-count threshold that used to switch them off.
label="" rather than None: None serialises to null and vis draws that literally.
Node and edge payloads moved into ui/view.py so this is asserted behaviourally
rather than by reading app.py's source — importing app.py runs main().
486 tests pass; ui/view.py at 100%.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Prep for merging to main. Three things were specific to the sandbox host or generated, and neither belongs in a shared repo: - indexes.yml carried this deployment's hostname, its two index names, and auth: false. Anyone cloning would have inherited a config pointing at someone else's box with authentication disabled. It is now indexes.example.yml with safe defaults (auth on, localhost, generic names), and indexes.yml is gitignored per-deployment config like .env. fleet.py says how to create it. - caddy/Caddyfile is generated on every run and was tracked, so it carried the sandbox hostname into git. Gitignored, alongside the compose file already was. - docs and one test referenced the sandbox IP and the azureuser home; the test now uses a TEST-NET-3 documentation address. Also brought docs/multi-index.md up to date with the nginx → Caddy switch: TLS is now obtained automatically rather than terminated at a load balancer, so that section was describing work nobody needs to do any more. 486 tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Superseded by #16, which carries the same product changes with a history that contains no deployment or demo specifics. The deployment tooling stays local for now. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Serves several independent brains from one machine:
Each index has its own doc_types, entities, credentials and read/write policy. Isolation is enforced on both doors — an index's bearer token opens only its
/mcp, its UI password only its/ui.Why one setup, not one per index
servehandles a single brain per process, so each index needs its own container. But the OpenSearch index name is already derived per brain, so they share one cluster — and the cluster is the expensive part.Measured on a 2-vCPU / 3.8GB VM running two indexes:
First index ~1.3GB, each additional ~326MB.
Usage
deploy/fleet/indexes.ymlis the source of truth; compose, nginx config and credentials are generated from it.Two bugs this surfaced, fixed here
1.
servecould not sit behind any reverse proxy. The MCP SDK enables DNS-rebinding protection with a localhost-only allow-list, so a proxied request arrives with a foreignHostand is rejected421 Invalid Host header. Passing a non-localhost bind address to the SDK would have silently disabled the protection instead — soserve_httpnow builds an explicit allow-list from--public-urlplus a new--allowed-host/OPEN_INDEX_ALLOWED_HOSTS, keeping the check on.*opts out for a trusted proxy.2. A single shared htpasswd let the
salesUI password open thesupportUI. nginx accepts any user present in the file it is given, so one file meant no isolation. Credentials are now per-index files.The UI needs a gate at all because Streamlit ships no authentication — published as-is, anyone reaching the host could read an entire brain. It stays read-only (writes go through MCP or the CLI) and can be dropped per index with
ui: false.Verified on a real VM
Not just generated config — deployed and exercised:
supportis absent fromsales, separate cluster indexes_stcore/healthreachable through the proxy (websockets working)Hostheader acceptedNotes for review
deploy/fleet/is deployment tooling, not part of the package; generated artifacts are gitignored.*opt-out.mcp_server.pyis at 100%.docs/multi-index.mdcovers architecture, capacity, LB setup (health check, TLS,Hostforwarding, buffering), access, and populating an index from a spreadsheet.🤖 Generated with Claude Code