feat(python): HTTP transport, MCP helper and khive-cloud CLI for the cloud data plane - #2362
Open
ohdearquant wants to merge 11 commits into
Open
feat(python): HTTP transport, MCP helper and khive-cloud CLI for the cloud data plane#2362ohdearquant wants to merge 11 commits into
ohdearquant wants to merge 11 commits into
Conversation
…pplied vectors blobs.put/get/stat over the content-addressed store (BLAKE3 refs, idempotent). attach()/attachment() bind bytes to a record under a role — descriptor rides record properties until the first-class attachments table gets a public verb. Embedding names the vector's model space; server-side acceptance of client-supplied vectors is the follow-up. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…cloud data plane
Extends the khive Python client (`python/khive/`) with a remote transport
alongside the existing Unix-socket one, so the same `Khive` facade works
against a khive-cloud deployment without changing any facade or model code.
- `khive/transport.py`: `HttpTransport` (sync, httpx) and `AsyncHttpTransport`
implement `POST /v1/request` + `GET /health` against khive-cloud's verified
wire contract. A `metrics_only` handshake frame is answered without a POST
(the cloud has no local engine config; `served_config_id` is derived from
the base URL so `Session`'s config-coherence check is trivially satisfied).
Per-op error objects (`{"code","message"}`) are flattened to the string
`OpResult.error: str | None` already expects, so `Khive.raw`/`batch`/the
rest of the untouched facade keep working unchanged; `errors.http_op_error_code`
recovers the code from either shape. The API key lives only in the
`Authorization` header — never in `repr`, logs, or error text.
- `khive/errors.py`: `HttpError` (status/body/json/url) plus `AuthError`
(401/403), `RateLimited` (429), `BadRequest` (4xx), `ServerError` (5xx),
shared by the sync and async transports.
- `khive/mcp.py`: `mcp_session` async context manager over
`mcp.client.streamable_http.streamable_http_client` (auth header via a
pre-configured `httpx.AsyncClient`), unwrapping the SDK's nested
`ExceptionGroup`s on a bad key to a plain `AuthError`; `mcp_list_tools`/
`mcp_request` sync conveniences that build their coroutine only after
confirming there's no running loop, so calling them from inside one raises
a clean `RuntimeError` instead of leaving an unawaited coroutine behind.
- `khive/cloud.py`: `khive.cloud(base_url, api_key) -> Khive` sugar over
`Khive(transport=HttpTransport(...))` (the facade itself stays untouched).
- `khive/cli.py`: `khive-cloud` script (`whoami`, `exec`, `tools`, `health`;
`--url`/`--api-key` default to `KHIVE_CLOUD_URL`/`KHIVE_CLOUD_API_KEY`,
exit 2 naming the missing var; exit 1 with `error: <Class>: HTTP <status>
<body>` on a typed transport error; `exec` exits 1 if the envelope's
`summary.failed`/`aborted` > 0).
- `pyproject.toml`: `cloud` extra (`httpx`, `mcp<2` — pinned below the newly
released 2.x rewrite, which renames/reshapes the streamable-HTTP client API
this module targets), `khive-cloud` console script, version 0.2.0.
`mcp`/`httpx` are imported lazily inside functions/`__init__`s, never at
module scope, so a socket-only install never needs either package.
- README: usage for `khive.cloud`, the MCP helpers, and the CLI.
Tests (all offline except one env-gated live module):
- `tests/conftest.py` gains `rest_server` (a real `http.server` instance) and
`mcp_server` (a real `FastMCP` app over `uvicorn`/streamable-HTTP), both
gated by the same `Authorization: ApiKey` scheme as the live deployment,
dispatching a small canned op table that mirrors the verified envelope
shape (ok/error/aborted entries, plus whole-request 429/500/400 arms).
Neither imports its optional dependency at module scope, so collecting the
file never requires the `cloud` extra.
- `tests/test_http_transport.py`, `tests/test_mcp.py`, `tests/test_cli.py`
cover the handshake-never-POSTs behavior, `raw`/`batch` per-op semantics,
every typed HTTP error, the API key never appearing in repr/error text, MCP
tools/list and request round-trips, wrong-key -> `AuthError` through the
ExceptionGroup unwrap, and all four CLI subcommands plus their exit codes.
Each module `importorskip`s its extra so a socket-only install skips them
cleanly instead of failing.
- `tests/test_cloud_live.py`: skipped unless `KHIVE_CLOUD_API_KEY` and
`KHIVE_CLOUD_URL` are both set. Written but not run — no live key was
available while writing it.
Gates run from `python/`, verbatim results:
- `uv venv .venv && uv pip install --python .venv -e '.[dev]'` — installs 15
packages (pydantic/pytest only, no httpx/mcp).
- `.venv/bin/python -m pytest -q` (socket-only) -> `9 passed, 4 skipped in 1.17s`
- `uv pip install --python .venv -e '.[dev,cloud]'` — installs 19 more packages.
- `.venv/bin/python -m pytest -q` (full) -> `36 passed, 3 skipped in 13.11s`
- `uv build` -> `Successfully built dist/khive_py-0.2.0.tar.gz` /
`Successfully built dist/khive_py-0.2.0-py3-none-any.whl`
- Acceptance venv install of the built wheel + `[cloud]`, `khive-cloud --help`,
and `python -c "import khive; from khive import HttpTransport; print('ok')"`
-> prints the `--help` text and `ok`.
- `uvx ruff check`/`ruff format --check` on every file this change added or
touched -> clean (README.md's one pre-existing unformatted line predates
this change and was left alone, since it isn't part of this diff).
No root `pyproject.toml`/`uv.lock` exists at the repository root (it's a Rust
workspace only), so no `[tool.uv.workspace]` exclusion was needed.
khive-cloud's `POST /v1/request` takes `ops` as one request DSL string
(`verb(k=v, ...)`, or `[op1, op2]` for a batch) — not the JSON array of
`{"tool", "args"}` dicts the client's internal ops form uses for the
daemon's native socket wire. Both HTTP transports were forwarding that JSON
array unchanged, so every op through `HttpTransport`/`AsyncHttpTransport`
was rejected server-side with "unknown verb: Missing 'verb' field in JSON";
the offline fake server used by the test suite didn't enforce the real
contract either, so 36 tests were green against a client that couldn't make
one real call. `khive/dsl.py` adds `render_dsl`, which renders the decoded
ops-array form as DSL text (covering the full value grammar: quoted strings
with the six recognized escapes and raw non-ASCII text, ints, floats,
booleans, null, arrays, and objects handed to a JSON encoder verbatim); both
transports now decode `frame["ops"]` and post `{"ops": render_dsl(...)}`.
The `khive-cloud exec` command sends its own already-DSL input through a new
`HttpTransport.send_dsl` instead, since that text is not the internal
ops-array form. The offline REST/MCP fake servers now parse and enforce the
same DSL grammar (`tests/_dsl_fake.py`) instead of accepting whatever the
client happened to send, including a control asserting the old JSON-array
payload is refused exactly like the live server refuses it.
Additional changes in the same round:
httpx transport/timeout failures are now caught and raised as
`TransportError` instead of escaping as bare httpx exceptions; malformed or
shapeless 2xx response bodies are validated and rejected as
`TransportError` rather than trusted as success; `mcp_session`'s exception
translation is scoped to setup only, so an exception raised inside a
caller's `async with mcp_session(...)` body propagates unchanged (a bad key
surfaces only when the streamable-HTTP transport's task group tears down,
so setup failure now closes the stack itself and translates whichever
exception that produces); a real 403 from the MCP path now carries status
403 instead of being flattened to 401; `AsyncHttpTransport` gets its own
regression suite; the installed-console-script test now runs the actual
`khive-cloud` executable instead of `python -m khive.cli`; and two publication
issues are fixed (an internal-artifact reference in a test docstring, an
undefined `url` in the README's cloud snippet).
…e script The Python package is the SDK: `khive.cloud(url, api_key)`, the HTTP and socket transports, and the `khive.mcp` helpers. Command-line use of the cloud is the job of the separate CLI package, so the `khive-cloud` console script, its argparse module, its tests and the README section are removed, and `HttpTransport.send_dsl` is documented as the entry point for callers that already hold DSL text. Offline suite after the change: 62 passed, 3 skipped; `uv build` produces the wheel and sdist.
…MCP session termination
The HTTP transport rendered `frame["ops"]` on the assumption that it always
decodes to the client's internal `[{"tool", "args"}]` array. A caller handing
`raw()` request-DSL text (`whoami()`, `[a(), b()]`, `[a() | b()]`) produced a
JSON string instead, and rendering iterated its characters and failed with a
`TypeError`. `render_dsl` now returns a `str` untouched and uses a `str`
element inside a list verbatim beside rendered op dicts; an op dict without a
`tool` name, or an entry of another type, raises `TransportError` with the
entry named. Sync and async transports share the change.
On session close the MCP SDK's streamable-HTTP client sends `DELETE /mcp` and
logs `Session termination failed: <status>` for any status other than 200 or
204. khive-cloud acknowledges the DELETE with 202 Accepted, so every helper
call printed that warning on stderr. `mcp_session` now installs a logging
filter on the SDK's transport logger for the lifetime of the session that
drops the record for 2xx statuses and keeps it for everything else; the
filter is removed after the last close path has run.
Tests: `render_dsl` pass-through (whole string, chain string, string elements
beside dicts, missing tool name, non-op entry); `raw()` with a DSL string, a
batch string, a chain string and a mixed list against the fake REST server,
which parses the DSL grammar; the termination filter's 2xx/non-2xx arms and,
inside a live `mcp_session` against the fake MCP server, a 202 record dropped
and a 500 record kept with the filter gone afterwards; one live-deployment
`raw()` case in the env-gated module.
Offline: `python -m pytest -q` in `python/`: 69 passed, 4 skipped.
…efuse plain HTTP off loopback
Non-finite floats (NaN/Infinity) have no representation in the khive-cloud
request DSL; `render_dsl` now rejects them with a `TransportError` naming
the argument, on the scalar path, inside arrays, and inside object literals
(`khive/dsl.py`). The offline DSL fake used by the test servers is
tightened to match: it now rejects raw C0 control characters inside a DSL
string, non-finite JSON constants inside object literals, and non-finite
scalar words such as "nan"/"inf", instead of silently accepting wire values
the real cloud parser would refuse (`tests/_dsl_fake.py`,
`tests/test_dsl_fake.py`). `tests/test_dsl.py` gains coverage for the value
shapes the round-trip property previously missed: `null`, `false`, empty
arrays/objects, negative/exponent floats, a literal unescaped backslash
sequence, and `$prev`-prefixed strings.
`HttpTransport`/`AsyncHttpTransport` previously validated only that a 2xx
response body was `{"results": [...]}`; an individual malformed entry (a
non-dict element, or one missing `ok`/`tool`) passed through as a
successful response. Both transports now validate every result entry
against the `OpResult` wire shape before returning the envelope
(`khive/transport.py`), with malformed-entry regression tests for both the
sync and async paths. `HttpTransport.send_dsl`, the documented passthrough
for callers holding DSL text already, had no test at all; it now has one,
including a DSL argument containing delimiters, asserting the text is sent
verbatim. The async transport gains the malformed-body and per-op-error
coverage the sync suite already had.
`mcp_session`'s cleanup could replace the exception being unwound: an
unrelated `stack.aclose()` failure during setup could bury an already-typed
setup exception, and during body unwind an `asyncio.CancelledError` or
`BaseExceptionGroup` from cleanup was not covered by
`suppress(Exception)` and could silently take over. Both paths now preserve
the original exception, attaching any cleanup failure as a note instead of
replacing it, and the body-unwind path only lets a cleanup exception through
when it is the exact exception already being unwound. `alist_tool_names`
and `acall_request` also translate call-time failures from `list_tools`/
`call_tool` into `khive` errors, matching every other library-facing call in
this module; previously those two escaped as raw MCP/httpx exceptions.
`HttpTransport`/`AsyncHttpTransport` refuse a plain `http://` base URL
whose host is not loopback (`127.0.0.1`, `::1`, `localhost`) unless the
caller passes `allow_insecure=True` — sending the API key in an
`Authorization` header over plain HTTP to anything else leaks it to every
hop in between. `khive.cloud(...)` forwards the flag. Both transport
docstrings and the README's cloud section now also note that a session's
identity fields are never put on this wire; khive-cloud resolves the
principal from the API key alone.
README and `ops.py` documentation drift is fixed: the README no longer
names the removed CLI among the offline-fake test suites, and `ops.py`'s
module docstring now says JSON is the client's internal form, adapted to
DSL text only by the HTTP transport, rather than claiming the client only
ever emits JSON on the wire.
Gates:
cd python
.venv/bin/python -m pytest -q
-> 118 passed, 4 skipped in ~22s (skips are tests/test_cloud_live.py,
which needs KHIVE_CLOUD_URL/KHIVE_CLOUD_API_KEY)
/Users/lion/projects/.venv/bin/ruff check khive/dsl.py khive/mcp.py khive/transport.py khive/cloud.py tests
-> 4 pre-existing findings, none in changed lines: two BLE001 blind-except
warnings at khive/mcp.py:110,122 (the same two sites flagged before
this change; the fix's design requires catching BaseException there),
and two I001 import-order warnings in untouched import blocks in
tests/conftest.py:20 and tests/test_dsl.py:5
/Users/lion/projects/.venv/bin/ruff format --check khive/dsl.py khive/mcp.py khive/transport.py khive/cloud.py tests
-> 13 files already formatted
… and guard MCP URLs
- dsl.py: a value equal to `$prev` or starting with `$prev.`/`$prev[`, at any
depth of an array or object argument, now renders with the escaped-literal
form (`_needs_prev_escape` + `_render_string_value`/`_prep_for_json`) so it
decodes back to plain text instead of resolving as a chain reference;
object-argument values are walked and escaped before `json.dumps` instead
of being handed to it raw. `render_dsl(..., chained=True)` now emits a
top-level `a() | b()` (no enclosing brackets, which the DSL grammar treats
as a mixed-separator batch); a single chained op renders bare. Serialization
failures (unsupported nested value, non-str object key, non-object `args`)
now raise `TransportError` uniformly instead of leaking a raw `TypeError`.
Tested against `_dsl_fake.py`, which now mirrors the real grammar's tool-name
dot-nesting limit, JSON number spelling, and both the primary (unquoted) and
secondary (quoted-string) `$prev` reference forms.
- mcp.py: `mcp_session` and the public MCP helpers
(`alist_tool_names`/`acall_request`/`mcp_list_tools`/`mcp_request`) apply
the same non-loopback-`http://` refusal `HttpTransport` already applies,
via a new `allow_insecure` parameter, checked before any client is built.
- transport.py: the cloud's minimal aborted-chain-entry shape
(`{"ok": false, "aborted": true}`, no `tool`) is now accepted and normalized
to carry `tool: ""` so it validates identically on the sync and async HTTP
transports without touching `models.py`. Per-op error objects are validated
(`code`/`message` must be strings) before being flattened to the string
`OpResult.error` expects, instead of silently accepting a malformed shape.
- tests: offline REST fake now authenticates `/health` like every other
route, and simulates real chain-abort cascading so a failing op inside a
chain marks every following op with the minimal aborted shape.
Collapse the repeated startswith calls on the $prev prefixes into one tuple call in the renderer and the test parser, sort the import blocks in the test modules, apply the formatter to the modules touched by the transport work, and assert the translated client error instead of a bare Exception where the MCP session test proves the URL guard was bypassed. No behaviour change; the suite is unchanged in count.
…P envelopes and prev literals
The Python client's request-DSL renderer and its offline test fakes drifted
from what the cloud parser actually accepts on four points.
A bracketed batch (`[a(), b()]`) with a `|` separator inside it is invalid on
the wire: the parser treats `|` inside `[...]` as MixedSeparators, and only
the unbracketed form (`a() | b()`) is a chain. The offline fake parser
accepted `[a() | b()]` as a chain; it now rejects it, and every fixture and
test that previously sent a bracketed chain sends the bare form instead.
`render_dsl` treated a falsey `args` value (`{}`, `[]`, `""`, `0`, `False`)
the same as a missing `args` key, silently rendering `tool()` for all of
them. A missing key still renders `tool()`, but a present, non-dict `args`
value (including `None`) now raises `TransportError` naming the tool and the
offending type, instead of being coerced away.
The MCP transport (`acall_request`) returned the cloud's raw decoded payload
untouched, so a per-op error stayed the `{"code","message"}` object shape and
a minimal aborted-chain-entry stayed missing its `tool` field — neither
matched what the REST transport hands back after
`_stringify_op_errors`/`_validate_envelope_results` normalize the same
envelope shape. `acall_request` now runs the MCP payload through the same two
normalization functions before returning it, so both transports produce an
identical caller-visible result for the same DSL.
A caller value with exactly one leading backslash followed by `$prev`,
`$prev.`, or `$prev[` has no representation in the request DSL: the parser's
escape rule strips exactly one leading backslash before matching, so
rendering the value plainly loses the backslash (decoding back to a bare
`$prev`-shaped string) and rendering it with the renderer's own escape
produces two backslashes instead of one. Both cases silently corrupt the
value, so the renderer now raises `TransportError` for that exact shape, at
any nesting depth. Two or more leading backslashes are unaffected and
continue to round-trip unchanged.
The offline fake parser also gained two fidelity fixes not tied to those
four behaviours: quoted-string decoding now normalizes raw newline/CR/tab
bytes to their JSON escape and decodes the whole literal with `json.loads`
(making `\uXXXX` decode correctly, and rejecting `\'`, which JSON does not
define, instead of silently accepting it), and argument names are now
validated against the identifier grammar rather than accepted verbatim.
Tests: test_dsl_fake.py::test_bracketed_chain_rejected_as_mixed_separators,
test_dsl.py::test_missing_args_key_renders_empty_call,
test_dsl.py::test_non_dict_args_value_raises_transport_error,
test_mcp.py::test_mcp_chained_abort_matches_rest_normalized_entry,
test_dsl.py::test_single_backslash_prev_literal_raises_top_level,
test_dsl.py::test_single_backslash_prev_literal_raises_nested_in_list,
test_dsl.py::test_single_backslash_prev_literal_raises_nested_in_object,
test_dsl.py::test_double_backslash_prev_literal_round_trips_top_level,
test_dsl.py::test_double_backslash_prev_literal_round_trips_nested,
test_dsl_fake.py::test_raw_newline_inside_quoted_value_decodes,
test_dsl_fake.py::test_unicode_escape_decodes,
test_dsl_fake.py::test_single_quote_escape_rejected,
test_dsl_fake.py::test_invalid_argument_name_rejected.
ohdearquant
commented
Sep 3, 2026
ohdearquant
left a comment
ohdearquant
left a comment
Owner
Author
There was a problem hiding this comment.
Automated review. Posted by this repository's automated pull-request review pipeline; this is not a human read and does not gate the merge by itself.
Verdict on head d5090a6: REQUEST-CHANGES, 2 blocking findings. Finding details are delivered to the review's recipients rather than posted here. Do not merge this head while blocking findings are outstanding; a pipeline comment on a newer head supersedes this one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends the hand-written Python client in
python/(packagekhive-py, importkhive) with a cloud data-plane transport, keeping the store-shaped facade, models and op encoding untouched.HttpTransport/AsyncHttpTransportbesideSocketTransport:Authorization: ApiKey <key>on every request,POST /v1/requestforwarding the client's existing JSON-array op encoding, the handshake frame answered fromGET /healthwithout a config round-trip.AuthError401/403,RateLimited429,BadRequest4xx,ServerError5xx) carrying the server body; embedded per-op failures keep flowing throughKhive.raw/Khive.batch.khive.mcp: streamable-HTTP MCP session helper with the same header, plus sync conveniences fortools/listand therequesttool.khive-cloudconsole script (whoami,exec,tools,health) readingKHIVE_CLOUD_URL/KHIVE_CLOUD_API_KEYfrom the environment; values are never echoed.cloudextra (khive-py[cloud]= httpx + mcp, pinnedmcp>=1.2,<2because 2.x renames the client and server entry points); the socket-only install stays pydantic-only.Tests
Offline against local fake servers (a threaded REST server and a FastMCP streamable-HTTP app behind a header-checking middleware): 36 passed, 3 skipped with the
cloudextra; 9 passed, 4 skipped without it (cloud tests skip cleanly when the extra is absent). The three skipped tests are the env-gated live checks intests/test_cloud_live.py, run only when both env vars are set.Verification