Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 27 additions & 1 deletion crates/khive-mcp/docs/api/coordinator.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ operation is successful but its operation envelope also carries:
"missing_backends": ["archive"],
"backend_errors": {
"archive": {
"kind": "backend_error",
"kind": "timeout",
"message": "backend search timed out after 5000ms"
}
}
Expand All @@ -69,6 +69,32 @@ through `backend_errors_truncated` and `backend_errors_omitted`. Backend ids and
messages are credential-masked before exposure; changed backend ids carry a
stable hash suffix and `backend_id_masked: true`, ids are capped at 256 Unicode
scalar values, and messages are capped at 1,024 Unicode scalar values.
Each retained cause is typed as `timeout` for coordinator and typed runtime
deadline failures, or `backend_error` otherwise. A degraded-empty
`search_incomplete` error reports `retryable: true` only when every failed leg
is a timeout; any mixed or non-timeout failure keeps it false. This policy is
computed before diagnostic truncation, so an omitted cause cannot change the
classification. Every `retryable: true` error also carries `retry_after_ms`.
The server computes that pace from the full pre-truncation failure set as
2,000ms plus 250ms for each failed backend after the first, capped at 10,000ms.

A conforming client that acts on `retryable: true` uses this published policy:

- At most three total attempts per logical request (the first attempt plus no
more than two reissues).
- Before reissue number _n_ (starting at 1), wait a base of
`max(retry_after_ms, 2000) * 2^(n - 1)` milliseconds plus nonnegative random
jitter of at most half that base. Jitter must never shorten the server-named
delay.
- Key a circuit breaker by the failed backend set. Open it after three
consecutive all-timeout `search_incomplete` outcomes for that same set,
suppressing first attempts as well as retries for 30 seconds. Then admit one
half-open probe: a successful search closes the breaker; another all-timeout
outcome reopens it for 30 seconds.

The server publishes and names this pace but cannot enforce client-side
admission. Clients that do not implement the complete budget, backoff, and
breaker policy must not act on `retryable: true` automatically.

## `t6d` — malformed `tags` must reject, not silently drop the filter

Expand Down
48 changes: 45 additions & 3 deletions crates/khive-mcp/src/coordinator.rs
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,47 @@ impl From<CoordError> for khive_runtime::RuntimeError {
}
}

/// Stable classification for a failed fan-out backend leg.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub enum BackendSearchFailureKind {
/// The backend operation failed for a non-timeout reason.
BackendError,
/// The backend exceeded the coordinator's per-request deadline.
Timeout,
}

impl BackendSearchFailureKind {
pub(crate) fn as_str(self) -> &'static str {
match self {
Self::BackendError => "backend_error",
Self::Timeout => "timeout",
}
}
}

/// Typed failure for one fan-out backend leg.
#[derive(Clone, Debug, Eq, PartialEq)]
pub struct BackendSearchFailure {
pub kind: BackendSearchFailureKind,
pub message: String,
}

impl BackendSearchFailure {
pub fn backend(message: impl Into<String>) -> Self {
Self {
kind: BackendSearchFailureKind::BackendError,
message: message.into(),
}
}

pub fn timeout(message: impl Into<String>) -> Self {
Self {
kind: BackendSearchFailureKind::Timeout,
message: message.into(),
}
}
}

/// Per-backend contribution to a fan-out search.
pub struct BackendSearchResult {
pub backend_id: BackendId,
Expand All @@ -71,8 +112,9 @@ pub struct BackendSearchResult {
pub vector_selected: bool,
/// Populated when this backend errored during the fan-out. A whole-backend
/// failure (e.g. the text arm, or a fatal error before either arm ran) —
/// this backend contributed no hits at all.
pub error: Option<String>,
/// this backend contributed no hits at all. The typed cause is what the
/// classifier reads; it never parses a rendered message.
pub error: Option<BackendSearchFailure>,
/// Populated when only the vector arm failed and the text arm still ran:
/// `entity_hits` still carries the text arm's results, and `error` above
/// stays `None`.
Expand Down Expand Up @@ -379,7 +421,7 @@ pub(crate) mod tests {
entity_hits: vec![],
note_hits: vec![],
vector_selected: true,
error: Some("injected search failure".to_string()),
error: Some(BackendSearchFailure::backend("injected search failure")),
vector_error: None,
}),
)
Expand Down
Loading
Loading