Skip to content

server.md

Latest commit

 

History

History
845 lines (618 loc) · 43.4 KB

server.md

File metadata and controls

845 lines (618 loc) · 43.4 KB
Raw

knowledge-server Design

HTTP API

The server exposes a REST + SSE API.

POST /query

Submit a natural-language question and receive a complete JSON response.

Request body:

{
  "query": "How does authentication work in repo-a v2.1.0?"
}

Response:

{
  "answer": "Authentication in repo-a v2.1.0 uses ...",
  "sources": [
    { "repo": "repo-a", "version": "v2.1.0", "file": "src/auth/middleware.rs", "line": 42 },
    { "repo": "repo-a", "version": "v2.1.0", "file": "src/auth/token.rs",      "line": 17 }
  ],
  "tool_calls_made": 6,
  "provider_used": { "provider_id": "anthropic-0", "kind": "anthropic", "model": "claude-sonnet-4-6" }
}

POST /query/stream

Same request body as /query. The response is a stream of Server-Sent Events.

Event type Payload fields When
intent type, mode After intent classification
phase type, label Before a batch of tool calls
thinking_delta type, text Reasoning-model thinking tokens
text_delta type, text Streamed answer text
tool_call type, name, input The LLM invokes a tool
tool_result type, name, preview The tool returns a result
question type, question, choices The LLM called ask_user
confirm_action type, id, name, input, description A destructive tool awaits approval
title_updated type, title A conversation title was generated
done type, answer, sources, tool_calls_made, provider_used The agentic loop finishes
error type, message An unrecoverable error occurs

GET /repositories

Returns all ingested repositories and their available versions.

[
  { "name": "repo-a", "versions": ["v1.0.0", "v2.1.0"], "url": "https://github.com/org/repo-a" },
  { "name": "repo-b", "versions": ["v0.9.0", "v1.0.0"], "url": null }
]

GET /graph/:repo/:version

Returns the full symbol graph for the given (repo, version) pair as JSON.

{
  "nodes": [
    { "id": "src/auth.rs:AuthManager", "name": "AuthManager", "file": "src/auth.rs",
      "kind": "struct", "start_line": 12, "signature": "struct AuthManager" }
  ],
  "edges": [
    { "id": "calls:src/main.rs:main>src/auth.rs:authenticate",
      "source": "src/main.rs:main", "target": "src/auth.rs:authenticate",
      "relation": "calls" }
  ],
  "truncated": false,
  "total_nodes": 142
}

Edge relation values: calls, contains, inherits, implements, embeds, uses.

Results are served from an in-memory cache pre-warmed at startup. For very large codebases, graphs are capped at 1 500 nodes and 6 000 edges — truncated: true signals that the graph was trimmed. When truncated, class-type nodes are prioritised over function nodes and calls/uses edges are dropped before structural edges.

GET /graph/:repo/:version/source?file=<file>&name=<name>

Returns the source text and metadata for a single symbol.

{
  "name": "authenticate",
  "file": "src/auth.rs",
  "kind": "function",
  "start_line": 24,
  "end_line": 41,
  "signature": "fn authenticate(token: &str) -> Result<User>",
  "source": "fn authenticate(token: &str) -> Result<User> {\n    ...\n}"
}

Returns 404 if the symbol does not exist for the given (repo, version, file, name).

GET /docs/:repo/:version

Returns the documentation index for the given version.

{
  "repo": "repo-a",
  "version": "v2.1.0",
  "generated_at": "2026-05-30T14:22:00Z",
  "sections": {
    "tutorials":    [{ "filename": "getting-started.md", "title": "Getting Started" }],
    "how-to-guides":[{ "filename": "add-a-new-endpoint.md", "title": "Add a New Endpoint" }],
    "explanations": [{ "filename": "auth-flow.md", "title": "Authentication Flow" }],
    "reference":    [{ "filename": "api-reference.md", "title": "API Reference" }]
  }
}

Returns 404 if no documentation has been generated for the given version. Documentation is generated by the harvester's document subcommand (see harvester.md).

GET /docs/:repo/:version/:section/:filename

Serves a single documentation page as text/markdown. Valid sections are tutorials, how-to-guides, explanations, reference.

POST /tool-description

Returns a short plain-English description of a tool call, used by the web UI to label step-timeline entries.

Request body:

{ "name": "search_symbols", "input": { "query": "authenticate", "repo": "repo-a" } }

Response:

{ "description": "Searching for symbols matching "authenticate" in repo-a" }

GET /health

Returns {"status": "ok"}.

GET /llm/providers

Returns the LLM providers visible to the chat page's model picker (those with expose_to_ui = true), with their discovered or allow-listed models. Used to populate the picker.

GET /skills · GET /skills/:id

List or fetch a global skill (markdown with YAML frontmatter). See Skills.

GET /artifacts/:id · GET /artifacts/:id/download

Fetch an artifact's metadata or download its raw content. See Artifacts.

Configuration Reference

[server]
host = "0.0.0.0"    # default
port = 8080         # default

[neo4j]
uri      = "bolt://localhost:7687"
user     = "neo4j"
password = "${NEO4J_PASSWORD}"

[auth]
jwt_secret        = "${JWT_SECRET}"    # required; use a long random string in production
allow_local_login = true               # default; set false to force SSO only

# Optional: Google OAuth 2.0
[auth.google]
client_id     = "${GOOGLE_CLIENT_ID}"
client_secret = "${GOOGLE_CLIENT_SECRET}"
redirect_uri  = "https://harvest.example.com/auth/google/callback"

# Optional: OIDC SSO (Ubuntu One, Dex, Keycloak, …)
[auth.oidc]
issuer_url    = "https://login.example.com"
client_id     = "harvest"
client_secret = "${OIDC_CLIENT_SECRET}"
redirect_uri  = "https://harvest.example.com/auth/oidc/callback"
display_name  = "Single Sign-On"        # shown on the login page (optional)

# Agent behaviour (all optional, defaults shown)
[agent]
max_iterations             = 20        # agentic loop cap (default: 20)
compaction_threshold_chars = 40000     # compact history when it exceeds this (default: 40000)
compaction_keep_last       = 6         # keep this many recent messages after compaction (default: 6)

# One or more LLM providers. Define as many [[llm]] blocks as you need.
# On rate-limit errors they are tried in ascending `priority` order.
[[llm]]
provider       = "anthropic"            # "anthropic" | "gemini" | "openai-compatible"
model          = "claude-sonnet-4-6"
api_key        = "${ANTHROPIC_API_KEY}"
id             = "primary"              # optional; defaults to "<kind>-<priority>"
priority       = 0                      # lower = tried first (default: 0)
timeout_secs   = 120                    # per-request timeout (default: 120)
max_retries    = 3                      # retry attempts on transient errors (default: 3)
expose_to_ui   = true                   # show in the chat-page model picker (default: true)
name           = "Claude (production)"  # optional display name
models         = ["claude-sonnet-4-6"]  # optional allowlist for the picker

# — or — Google Gemini
# [[llm]]
# provider    = "gemini"
# model       = "gemini-2.5-flash"
# api_key     = "${GEMINI_API_KEY}"
# priority    = 1

# — or — OpenAI-compatible (Groq, Ollama, etc.)
# [[llm]]
# provider = "openai-compatible"
# base_url = "https://api.groq.com/openai/v1"
# api_key  = "${GROQ_API_KEY}"
# model    = "llama-3.3-70b-versatile"
# priority = 2

# Optional: serve Diataxis docs generated by the harvester
[documentation]
docs_dir = "/var/harvest/docs"    # must match harvester.toml [documentation].docs_dir

# Optional: control the docs page in the web UI
[ui]
enable_docs = false              # default; set true to show the Document tab

# Optional: serve the harvest-agent binary and generate install scripts
[agents]
binary_path = "/usr/local/bin/harvest-agent"    # path to the compiled agent binary
public_url  = "https://harvest.example.com"     # used in generated install scripts

# Optional: let Harvest provision and manage agents on an LXD cluster
[lxd]
endpoint     = "https://lxd-cluster.example.com:8443"
trust_token  = "eyJjbGllbnRfbmFtZSI6Li4ufQ=="  # one-time; see below
# client_cert / client_key: omit both (as above) to let Harvest generate and
# self-register its own identity. Set both instead to manage the cert yourself
# — see LxdConfig in config.rs for the full field list either way.
ca_cert      = "..."        # optional; PEM of the LXD server's CA if self-signed
insecure     = false        # optional; skip TLS verification (dev only)
project      = "harvest"    # LXD project holding all Harvest-managed containers (default: "default")
image_alias  = "24.04"      # default
image_server = "https://cloud-images.ubuntu.com/releases"  # default
profile      = "default"    # LXD profile applied to every container (storage pool, etc.)

When [documentation].docs_dir is omitted, the /docs routes are not registered. When [ui].enable_docs is false, the Document tab is hidden in the web UI even if docs are available. When [agents] is omitted, the /agents/binary/* and install-script routes still work but return 404 for the binary. When [lxd] is omitted, agents can only be added by installing the daemon on an existing machine — the web UI's "Let Harvest create and manage agent" option is hidden (GET /auth/config reports features.lxd: false).

Multiple LLM providers and the model picker

llm is an array ([[llm]]) — define one block per provider. Providers are sorted by priority (ascending) and wrapped in a FallbackProvider: if a provider returns a rate-limit error (HTTP 429 / RESOURCE_EXHAUSTED), the next one is tried. Non-rate-limit errors propagate immediately.

Each block accepts an optional id (stable identifier used by the chat page's model picker; defaults to <kind>-<priority>), expose_to_ui (set false to keep a provider as failover-only, hidden from the picker), name (display name in the picker), and models (curated allowlist of model names to offer in the picker; defaults to every model the provider advertises). GET /llm/providers returns the picker-visible providers and their models. A client selects a specific provider/model by sending a provider/model field on the query request, which the agent forwards as a ProviderSelection.

Client identity: self-managed vs. manual

By default (client_cert/client_key both omitted), Harvest manages its own LXD client identity end to end:

  1. On startup, if no identity has been persisted yet, lxd::identity::load_or_generate generates a self-signed cert/key pair (rcgen) and stores it on a singleton LxdIdentity Neo4j node — untrusted at this point.
  2. If trust_token is set and the identity isn't yet trusted, lxd::identity::join_with_token submits the cert to LXD's POST /1.0/certificates with that token (an anonymous mTLS request using the untrusted cert itself — this is exactly what LXD's trust-token mechanism is for). On success the identity is marked trusted in Neo4j and the token is never needed again.
  3. If there's no token yet, or the join fails (expired/invalid/already-used token), the server logs a warning and starts anyway with LXD features disabled — this is not a fatal startup error, since the admin may just not have gotten to it yet.

To generate a token, run lxc config trust add --name harvest (no certificate argument) against the LXD cluster with an already-trusted lxc — it prints a short-lived opaque token (core.remote_token_expiry, typically a few hours) to paste into trust_token before restarting the server. Each token is single-use; if a join attempt fails, generate a fresh one.

Setting client_cert/client_key explicitly bypasses all of this — LxdClient::new builds directly from the configured PEM pair, and trust_token/the persisted identity are ignored.

LLM Provider Abstraction

The server defines a LlmProvider trait backed by three implementations, plus a FallbackProvider that wraps them for priority-based failover.

AnthropicProvider

Uses the Anthropic Messages API with native tool use. Configure with provider = "anthropic".

GeminiProvider

Uses the Google Gemini API with native function calling. Configure with provider = "gemini".

[[llm]]
provider = "gemini"
model    = "gemini-2.5-flash"
api_key  = "${GEMINI_API_KEY}"

OpenAiCompatProvider

Uses the OpenAI Chat Completions API with function calling. Works with any compatible endpoint.

# Groq
[[llm]]
provider = "openai-compatible"
base_url = "https://api.groq.com/openai/v1"
api_key  = "gsk_..."
model    = "llama-3.3-70b-versatile"

# Local Ollama (no auth)
[[llm]]
provider = "openai-compatible"
base_url = "http://localhost:11434/v1"
api_key  = "ollama"
model    = "qwen2.5-coder:32b"

FallbackProvider

When more than one [[llm]] block is configured, from_config sorts providers by priority (ascending) and wraps them in a FallbackProvider. On a rate-limit error (HTTP 429 / RESOURCE_EXHAUSTED) the next provider is tried; any other error propagates immediately. A ProviderSelection ({ provider_id, model }) sent by the chat page's model picker causes the matching provider to be tried first, with its model overridden — falling back to the default order only if that provider is rate-limited or the id is unknown.

Every response carries a provider_used block ({ provider_id, kind, model }) identifying which provider and model actually answered, so the UI can show it even after a failover.

Agentic Workflow

1. Classify intent (conversational / research / action / hybrid)
   - conversational turns get only ask_user; everything else gets all tools
2. Optionally compact history if it exceeds compaction_threshold_chars
3. Build messages: system prompt + compacted history + user message (+ attachments)
4. Loop (up to max_iterations):
   a. Stream the LLM response, forwarding thinking_delta / text_delta / tool_call events
   b. If stop_reason is end_turn (or no tool calls) → done, return answer
   c. If the LLM called ask_user → emit a question event and end the turn
   d. Partition tool calls into confirmable (requires_confirmation) and automatic:
      - confirmable: emit confirm_action events and pause until the user resumes
      - automatic:   execute concurrently with join_all, append results
   e. continue loop
5. If max_iterations is hit, make one final synthesis call over gathered tool results
6. Parse [repo:version:file:line] citations from the final answer
7. Return structured response (answer, sources, tool_calls_made, provider_used)

Intent classification

classify_intent returns one of conversational, research, action, or hybrid. On the first turn with no history, simple keyword heuristics detect action verbs (run, deploy, restart, destroy, …) so common commands skip the LLM call entirely. Otherwise a single LLM call classifies the message. conversational turns skip the tool set (only ask_user is wired up) so follow-ups and greetings don't trigger graph queries.

Streaming events

query_streaming emits these AgentEvents over the SSE channel (or an in-memory channel for the non-streaming query wrapper):

Event When
Intent After classification — {mode}
Phase Before a batch of tool calls — derived label like Searching codebase, Reading source, Tracing relationships, Executing on agents
ThinkingDelta Reasoning models emit thinking tokens
TextDelta Streamed answer text
ToolCall The LLM invoked a tool
ToolResult A tool returned a result (with a truncated preview)
Question The LLM called ask_user — {question, choices}
ConfirmAction A confirmable tool awaits approval — {id, name, input, description}
TitleUpdated A conversation title was auto-generated
Done Loop finished — {answer, sources, tool_calls_made, provider_used}
Error Unrecoverable failure — {message}

Confirm-action (paused turns)

Tools flagged requires_confirmation() (e.g. run_terraform_apply, run_terraform_destroy, mutating run_command) don't execute immediately. The loop emits a ConfirmAction event per pending call and returns a PausedTurn (the message list, iteration count, and pending call ids). The client approves by calling POST /projects/:pid/conversations/:cid/confirm-action/resume, which calls resume_after_confirm to push tool results and continue the loop.

System Prompt

The system prompt instructs the LLM to:

  • Act as a code analysis assistant with access to a Neo4j knowledge graph.
  • Understand the graph schema (node labels, relationships, available properties).
  • Cite every factual claim about code in the format [repo:version:file:line].
  • Start broad (listing repositories and versions) and then drill down.
  • Prefer specific graph queries over broad ones to keep context small.
  • Emit Mermaid diagrams and inline harvest-graph snippets where they clarify structure.
  • Never end a response with plain-text questions — use the ask_user tool instead.
  • Reuse conversation context before re-calling tools.

Deployment-scoped agents use specialised system prompts (see prompt.rs) that forbid ask_user and mutating tools as appropriate.

Graph Query Tools

These graph tools are exposed to the LLM (defined in agent/graph_tools.rs). Each maps to one or more Cypher queries. The full tool set also includes machine, skill, infra, terraform, artifact, secret, and interaction tools — see the README "Agent tools" table for the complete list and agent/*_tools.rs for the implementations.

list_repositories

Returns all known repository names and their versions.

MATCH (r:Repository)-[:HAS_VERSION]->(v:Version {ingested: true})
RETURN r.name AS repo, collect(v.tag) AS versions
ORDER BY r.name

search_symbols

Full-text search for functions or classes by name fragment across a repo/version.

Parameters: query: String, repo?: String, version?: String, kind?: "function" | "class" | "any"

CALL db.index.fulltext.queryNodes("symbol_names", $query)
YIELD node, score
WHERE ($repo    IS NULL OR node.repo    = $repo)
  AND ($version IS NULL OR node.version = $version)
RETURN node.repo, node.version, node.file, node.name,
       node.start_line, node.end_line, score
ORDER BY score DESC LIMIT 20

get_symbol_source

Returns the stored source text of a specific function or class.

Parameters: repo: String, version: String, file: String, name: String

get_file_symbols

Lists all symbols defined in a file (without source text).

Parameters: repo: String, version: String, file: String

find_callers

Returns all functions that call the given function within a version.

Parameters: repo: String, version: String, function_name: String

MATCH (caller:Function)-[c:CALLS]->(callee:Function {repo: $repo, version: $version, name: $function_name})
WHERE caller.repo = $repo AND caller.version = $version
RETURN caller.file, caller.name, caller.start_line, c.line AS call_site_line

find_callees

Returns all functions that the given function calls.

Parameters: repo: String, version: String, file: String, function_name: String

get_imports

Returns all import declarations for a file.

Parameters: repo: String, version: String, file: String

compare_symbol_across_versions

Returns the source text of a symbol for two versions side-by-side.

Parameters: repo: String, version_a: String, version_b: String, file: String, name: String

run_cypher (power tool)

Executes an arbitrary read-only Cypher query composed by the LLM. The driver connection uses AccessMode::Read so writes are rejected at the protocol level.

Parameters: query: String, params?: Object

Example use: "find all classes that implement trait X and are imported by module Y", or any traversal that crosses multiple relationship types not covered by the fixed tools.

Source Citation Format

The LLM embeds citations inline using:

[repo-name:v1.2.3:src/path/to/file.rs:42]

The server post-processes the final answer with a regex to extract these into the structured sources array. The raw answer text is returned as-is so clients can render citations as links.

Graph Cache

On startup the server pre-warms the graph cache by querying all Version nodes with ingested: true and fetching graph data for each one. Subsequent requests for GET /graph/:repo/:version are served directly from the in-memory cache with no Neo4j round-trip.

The cache is not invalidated automatically — restart the server after re-ingesting a version to pick up updated graph data.

Authentication

All routes except /health, /auth/*, /agent/events, /agent/results, /agent/output, /agent/ping, /agent/console/*, /agent/tunnel/*, and /agents/binary/* require a valid JWT session cookie.

Endpoints

Method Path Description
GET /auth/config Returns auth capabilities (google, oidc, local_login, features.lxd)
POST /auth/register Register with email + password. First user becomes admin.
POST /auth/login Login and receive a JWT cookie.
POST /auth/logout Clear the JWT cookie.
GET /auth/me Return current user's id, email, name, and role.
PATCH /auth/me Update the current user's name.
GET /auth/google Redirect to Google OAuth consent screen.
GET /auth/google/callback Google OAuth callback — exchanges code for session.
GET /auth/oidc Redirect to the OIDC provider's consent screen.
GET /auth/oidc/callback OIDC callback — exchanges code for session.

Roles

  • admin — full access to all routes including /admin/*.
  • regular — access to routes scoped to groups they belong to.

The first registered user is automatically assigned the admin role.

OIDC endpoints are discovered at startup from the configured issuer_url (via /.well-known/openid-configuration); if discovery fails the server starts anyway with OIDC login unavailable. Google OAuth and OIDC can both be enabled at the same time, and either can be used alongside or instead of local password login (set allow_local_login = false to force SSO).

Projects and Conversations

Projects are workspaces that group conversations, agents, deployments, secrets, skills, and artifacts. Access is controlled by group membership.

Endpoints

GET    /groups                              list groups the user belongs to
GET    /projects                            list accessible projects
POST   /projects                            create a project (requires group membership)
GET    /projects/:pid                       get project details
PUT    /projects/:pid                       update name or description
DELETE /projects/:pid                       delete project and all conversations

GET    /projects/:pid/conversations         list conversations (newest first)
POST   /projects/:pid/conversations         create a conversation
GET    /projects/:pid/conversations/:cid    get conversation with messages array
PUT    /projects/:pid/conversations/:cid    update title and messages
DELETE /projects/:pid/conversations/:cid   delete conversation (owner or admin)
POST   /projects/:pid/conversations/:cid/confirm-action/resume
                                             resume a paused turn after confirm_action approval

POST   /projects/:pid/query                 non-streaming query with history
POST   /projects/:pid/query/stream          streaming query (fire-and-forget; events via project SSE)
GET    /projects/:pid/events                SSE channel for real-time collaboration events

Real-time Events (GET /projects/:pid/events)

The project event stream delivers:

Event type Payload When
presence {users: [{user_id, name, conv_id}]} On connect — snapshot of current users
lock {by: string} A query started; chat is locked
unlock {} Query finished; chat is unlocked
user_join {user_id, name, conv_id} A user opened the events stream
user_leave {user_id, name} A user disconnected
user_message {query, username, attachments} The user's message was accepted
intent {type, mode} After intent classification
phase {type, label} Before a batch of tool calls
thinking_delta {type, text} Reasoning-model thinking tokens
text_delta {type, text} Streamed answer text
tool_call {type, name, input} LLM invoked a tool
tool_result {type, name, preview} Tool returned a result
question {type, question, choices} The LLM called ask_user
confirm_action {type, id, name, input, description} A destructive tool awaits approval
title_updated {type, title} A conversation title was generated
done {type, answer, sources, tool_calls_made, provider_used} The agentic loop finished
error {type, message} An error occurred during the query

Conversation History and Compaction

Conversation messages are stored as a JSON array in Neo4j. When project_query_stream is called, the server:

  1. Loads the existing conversation history from Neo4j.
  2. If the total character count exceeds compaction_threshold_chars, compacts the older portion into a summary message using one LLM call.
  3. Runs the agent with the (possibly compacted) history.
  4. On completion, saves the new turn (user message + assistant reply + sources) back to Neo4j.

Secrets

Each project has a key-value secret store backed by Neo4j. Secrets are scoped to the project and accessible only to project members.

GET    /projects/:pid/secrets               list secret names (not values)
POST   /projects/:pid/secrets               upsert a secret {name, value}
DELETE /projects/:pid/secrets/:name         delete a secret

Secret names are normalised to UPPER_CASE. The LLM can access secrets through three dedicated tools: list_secrets, get_secret, and save_secret. get_secret returns values to the LLM; the preview shown in the UI step timeline is redacted ([secret value retrieved]).

Project Overview

The overview pipeline generates an AI-powered HTML status dashboard for a project.

GET  /projects/:pid/overview            current status HTML + metadata
GET  /projects/:pid/overview/events     SSE stream for in-progress generation steps
POST /projects/:pid/overview/regenerate trigger a new generation

Pipeline steps:

  1. Concatenate all project conversations (up to 20 most recent).
  2. If the combined text exceeds 8 000 characters, call the LLM to produce an env_doc summary of the environment context.
  3. Build a project-scoped agent (graph + machine tools) and call query_streaming with a status-dashboard prompt.
  4. Strip any markdown code fences from the result.
  5. Save env_doc and current_status (the HTML snippet) to the Project node in Neo4j.

Generation steps are broadcast over a broadcast::channel. Multiple clients can subscribe to the same in-progress generation via GET /projects/:pid/overview/events.

Skills

Skills are markdown playbooks (with YAML frontmatter) the agent loads on demand via the list_skills and load_skill tools. There are two scopes:

  • Global skills — seeded at startup from knowledge-server/skills/*.md (juju, lxd, ceph, canonical-k8s, landscape, openstack) and managed by admins.
  • Project skills — scoped to a single project.
GET    /skills                             list global skills
GET    /skills/:id                         fetch a global skill

GET    /projects/:pid/skills               list skills available to the project (global + project)
POST   /projects/:pid/skills               create a project skill
GET    /projects/:pid/skills/:sid          get a project skill
PUT    /projects/:pid/skills/:sid          update a project skill
DELETE /projects/:pid/skills/:sid          delete a project skill

POST   /admin/skills                       create a global skill (admin only)
PUT    /admin/skills/:id                   update a global skill (admin only)
DELETE /admin/skills/:id                   delete a global skill (admin only)

Artifacts

Artifacts are generated or uploaded content — terraform bundles, design docs, guides, or arbitrary context — stored in Neo4j and downloadable as raw text.

GET    /projects/:pid/artifacts            list project artifacts
POST   /projects/:pid/artifacts            create/upload an artifact
GET    /artifacts/:id                      get artifact metadata
PUT    /artifacts/:id                      update artifact content
DELETE /artifacts/:id                      delete an artifact
GET    /artifacts/:id/download             download raw artifact content

Terraform bundles are stored as a JSON map of {path: content}. The artifact_tools::GenerateArtifactTool lets the agent produce bundles; bundle.rs handles serialising/deserialising them for terraform runs.

Deployments

The deployment pipeline is the IaC arm of Harvest. Each project can hold many deployments; each deployment carries an environment description, a design doc, a terraform/terragrunt bundle, an optional guide, context artifacts, and a topologically-sorted execution plan. Deploys/redeploys/destroys run the bundle on a connected agent and record a DeploymentRun with captured stdout/stderr.

GET    /projects/:pid/deployment                        get the project's current deployment
GET    /projects/:pid/deployments                       list deployments
POST   /projects/:pid/deployments                       create a deployment
GET    /projects/:pid/deployments/:did                  get deployment detail
PATCH  /projects/:pid/deployments/:did                  update deployment metadata
DELETE /projects/:pid/deployments/:did                  delete a deployment
POST   /projects/:pid/deployments/:did/deploy           run the bundle (apply)
POST   /projects/:pid/deployments/:did/redeploy         re-apply the bundle
POST   /projects/:pid/deployments/:did/destroy          destroy the infrastructure
GET    /projects/:pid/deployments/:did/runs             list recorded runs
POST   /projects/:pid/deployments/:did/environment/questions   LLM interview for env requirements
POST   /projects/:pid/deployments/:did/design/generate         generate the design doc
POST   /projects/:pid/deployments/:did/design/decisions        capture design decisions
POST   /projects/:pid/deployments/:did/design/revise           revise the design
POST   /projects/:pid/deployments/:did/provision/generate      generate the terraform bundle
POST   /projects/:pid/deployments/:did/provision/propose-change  propose a bundle edit (staged)
POST   /projects/:pid/deployments/:did/provision/apply-change    apply a proposed bundle edit
POST   /projects/:pid/deployments/:did/context-artifacts         add a context artifact
POST   /projects/:pid/deployments/:did/context-artifacts/link    link an existing artifact
DELETE /projects/:pid/deployments/:did/context-artifacts/:aid    remove a context artifact
GET    /projects/:pid/deployments/:did/proposals                list proposed changes
POST   /projects/:pid/deployments/:did/proposals                propose an artifact change
POST   /projects/:pid/deployments/:did/proposals/:propid/approve  approve a proposal
POST   /projects/:pid/deployments/:did/proposals/:propid/discard  discard a proposal
GET    /projects/:pid/deployments/:did/execution-plan          get the execution plan
POST   /projects/:pid/deployments/:did/execution-plan          set the execution plan
POST   /projects/:pid/deployments/:did/run-dag                 execute the plan's DAG

Infrastructure state

Each deployment tracks an infra_state: none → up (successful apply) / broken (failed apply) → destroyed (successful destroy) / destroy_failed (failed destroy). A broken or destroy_failed deployment must be destroyed before it can be re-applied.

Execution plan

The execution plan is a list of steps ({id, action, phase, label, artifact, depends_on}) where action is run/plan/apply/destroy and phase is deploy/destroy. POST /run-dag topologically sorts the steps (deployments::topological_sort) and executes them in order against a chosen agent, streaming each run's output.

Templates

Product templates are group-scoped reusable designs/bundles that pre-seed a new deployment. update_product_template (a deployment tool) saves the current design and bundle as a template; future deployments based on that template get the template content injected into their agent context.

A template can also be uploaded as a .harvest archive via POST /templates/upload — a zip containing metadata.yaml (name/description), design.md, a skills/ directory of .md files with YAML frontmatter, and an artifacts/ directory of example Terraform/Terragrunt/Bash files. Skills and artifacts are discovered by directory listing (every .md under skills/, every file under artifacts/), not by an explicit manifest. design.md becomes the template's design_template: when a deployment uses that template, deployment_system_prompt injects it verbatim as a ## Design Document Template section, with instructions for the agent to follow its structure exactly, fill ${PLACEHOLDER} values, expand $(For each X in Y) { ... } blocks, and turn its ```Diagram fenced blocks into real ```dot diagrams.

GET    /groups/:gid/templates        list templates in a group
POST   /groups/:gid/templates        create a template
GET    /groups/:gid/templates/:tid   get a template
PUT    /groups/:gid/templates/:tid   update a template
DELETE /groups/:gid/templates/:tid   delete a template

Remote Agents (harvest-agent)

GET    /agents/:pid/install.sh                       generate bash install script
GET    /agents/binary/harvest-agent                  serve the agent binary
GET    /projects/:pid/agents                         list agents (online status)
DELETE /projects/:pid/agents/:aid                    remove an agent
POST   /projects/:pid/agents/:aid/execute            run a bash command on an agent
POST   /projects/:pid/agents/:aid/terraform          run a terraform/terragrunt plan|apply|destroy bundle
GET    /projects/:pid/agents/:aid/console            open an interactive shell (WebSocket, xterm.js)
POST   /projects/:pid/agents/:aid/start              start an LXD-managed agent container
POST   /projects/:pid/agents/:aid/stop               stop an LXD-managed agent container
POST   /projects/:pid/agents/:aid/restart            restart an LXD-managed agent container
POST   /projects/:pid/agents/rotate-install-token    rotate the install token
GET    /projects/:pid/agents/flavors                 list LXD container sizes (requires [lxd])
POST   /projects/:pid/agents/lxd                     provision an LXD-managed agent (requires [lxd])
GET    /projects/:pid/agents/:aid/port-forwards      list port forwards for an agent
POST   /projects/:pid/agents/:aid/port-forwards      create a port forward
PUT    /projects/:pid/agents/:aid/port-forwards/:fid update a port forward
DELETE /projects/:pid/agents/:aid/port-forwards/:fid delete a port forward

Agent-facing endpoints (token-authenticated, not JWT):

GET  /agent/events                 SSE stream the agent connects to (Registered, HelloAck, Execute,
                                   RunTerraform, OpenShell, OpenTunnel, Uninstall, Error)
POST /agent/results                agent posts a finished command/terraform result
POST /agent/output                 agent posts a streamed stdout/stderr line (terraform runs)
POST /agent/ping                   agent heartbeat
GET  /agent/console/:session_id    WebSocket upgrade for an interactive console session
GET  /agent/tunnel/:session_id     WebSocket upgrade for a reverse tunnel session

There are two ways an agent joins a project. The web UI's "Add agent" button offers a choice between them whenever [lxd] is configured on the server (see GET /auth/config → features.lxd); otherwise it goes straight to the manual flow.

Manual install — authentication flow:

  1. The admin generates an install token for the project (POST /projects/:pid/agents/rotate-install-token or the initial auto-generated one).
  2. The install script embeds the token and runs harvest-agent.
  3. On first connection to GET /agent/events, the server verifies the install token, creates a Machine node in Neo4j with a new permanent agent_token_hash, and sends a Registered event with the permanent token.
  4. The agent saves the permanent token to /etc/harvest-agent/config.toml.
  5. On subsequent connections, the server recognises the permanent token hash and sends HelloAck.

LXD-managed agents:

POST /projects/:pid/agents/lxd takes {name, description, flavor} (flavor is one of tiny, small, medium, large, extra-large — see GET /projects/:pid/agents/flavors for the exact CPU/RAM each maps to) and provisions the agent end to end:

  1. Ensures a bridge network exists for the Harvest project (lxd/mod.rs::ensure_network) — one LXD network per Harvest project, named deterministically from a hash of the project id (kept ≤ 15 chars for the kernel's IFNAMSIZ interface-name limit), all inside the single LXD project configured in [lxd].project.
  2. Sanitizes the user-supplied name into a valid LXD instance name and appends a random suffix for uniqueness.
  3. Writes an LxdInstance marker node in Neo4j (project_id, hostname, description) — a placeholder consumed once the agent connects.
  4. Creates and starts the container ([lxd].image_alias / image_server / profile, with limits.cpu / limits.memory set from the chosen flavor) and waits for it to reach the Running state.
  5. Runs the same install script a manual install would use (GET /agents/:pid/install.sh) inside the container via the LXD exec API — no separate provisioning path to keep in sync. A container LXD reports as Running hasn't necessarily finished booting (cloud-init, DNS/networking coming up), so this step retries up to EXEC_INSTALL_ATTEMPTS (4) times with a 10s delay (LxdClient::exec_with_retry) before giving up — safe because the install script is idempotent.
  6. If any step fails, the container and the LxdInstance marker are cleaned up and the request returns 502 with the full underlying error chain (both in the response body and server-side logs — see tracing::error! calls throughout lxd_provision.rs and lxd/mod.rs, enabled at debug level for request/response detail via RUST_LOG=knowledge_server::lxd=debug).

The endpoint returns as soon as the install script exits successfully — it does not wait for the agent to open its SSE connection. When agent_events_handler sees a first-time connection whose (project_id, hostname) matches a pending LxdInstance marker, it tags the new Machine node with provider: "lxd", lxd_instance, and description, then deletes the marker. The web UI's existing 15-second agent-list poll picks up the new agent once this happens — the same UX as a manual install appearing after the admin runs the curl command. Manually-installed agents have no provider field (list_agents reports "manual").

Deleting an LXD-managed agent (DELETE /projects/:pid/agents/:aid) additionally calls the LXD API to destroy the backing container before removing the Machine node; if the container deletion fails, the node is left in place and the request returns 502 so the resource isn't silently orphaned.

Command execution:

  1. The LLM calls run_command with an agent_id and command.
  2. The server verifies the agent belongs to the correct project (cross-project isolation).
  3. An Execute SSE event is pushed to the agent.
  4. The agent runs the command and POSTs the result to /agent/results.
  5. A oneshot channel delivers the result back to the waiting run_command tool call.

Terraform execution:

  1. The LLM (or a deployment run) calls run_terraform_plan/apply/destroy with an agent_id, artifact_id (the bundle), and the action.
  2. The server resolves the bundle's file map and pushes a RunTerraform SSE event with the flavor (terraform/terragrunt), action, and files.
  3. The agent writes the files to a temp dir, runs the action, and streams each stdout/stderr line to /agent/output (forwarded to any live SSE subscribers) before posting the final result to /agent/results.
  4. The server records a DeploymentRun (with stdout/stderr previews) and updates the deployment's infra_state.

Interactive console:

GET /projects/:pid/agents/:aid/console upgrades to a WebSocket and sends an OpenShell event to the agent. The agent opens a PTY and bridges it to the WebSocket, so the browser's xterm.js gets a live shell on the agent's machine. The claim handshake (GET /agent/console/:session_id) is authenticated with the agent token and expires after CONSOLE_CLAIM_TIMEOUT_SECS (15s).

Reverse tunnels & port forwards:

GET /projects/:pid/agents/:aid/console-style WebSockets also back port forwards. Creating a port forward opens an OpenTunnel session to the agent for the target port; the server then proxies HTTP traffic under /agents/:agent_id/:route_name/... (and raw WebSocket upgrades) through the tunnel to the agent's service. This lets the server reach services running on an agent's otherwise-private machine.

Admin Routes

All admin routes require the admin role.

GET  /admin/users               list all users with group memberships
PUT  /admin/users/:id/role      set role to "admin" or "regular"
PUT  /admin/users/:id/groups    replace group memberships with a new list
GET  /admin/groups              list all groups
POST /admin/groups              create a group {name, description?}
DELETE /admin/groups/:id        delete a group and its relationships
PUT  /admin/groups/:id/default  set a user's default group
POST /admin/skills              create a global skill
PUT  /admin/skills/:id          update a global skill
DELETE /admin/skills/:id        delete a global skill

Retry Strategy

All LLM providers (Anthropic, Gemini, and OpenAI-compatible) use a shared retry helper (llm/retry.rs) with the following behaviour:

Condition Action
Timeout error, retries remaining Exponential backoff: 2 * 2^min(attempt, 4) seconds
429 rate limited Wait for retry-after header value, or 30 * 2^min(attempt, 4) seconds
529/503 (Anthropic overload) Fixed 5-second wait
503 (Gemini overload) Fixed 5-second wait
502/503 (OpenAI overload) Fixed 5-second wait
Other error Return immediately

max_retries defaults to 3 and is configurable per provider in server.toml. When multiple providers are configured, a rate-limit error that exhausts a provider's retries triggers failover to the next provider in priority order (see FallbackProvider).