kb — comprehensive knowledge boss

kb is a knowledge base over code, docs, tasks, and chats.

Requirements

• Go 1.26+ to build from source (or use a prebuilt release binary).
• An OpenAI-compatible LLM endpoint, exposing chat completions at /v1/chat/completions and embeddings at /v1/embeddings.

Quickstart

Point kb at your OpenAI-compatible endpoint, build, and serve:

git clone https://github.com/alterfo/kb.git
cd kb
make build
export KB_LLM_BASE_URL=http://<your-llm-host>:11434   # chat + completions
export KB_EMBED_BASE_URL=http://<your-llm-host>:11434  # embeddings (same host is fine)
export KB_LLM_MODEL=qwen3.8:latest
export KB_EMBED_MODEL=qwen3-embedding
./bin/kb serve

Or install the latest release directly:

go install github.com/alterfo/kb/cmd/kb@latest

Open http://127.0.0.1:8080. By default serve binds to loopback and has no authentication; binding a non-loopback -addr requires KB_WEB_AUTH_TOKEN (refused otherwise) and enables bearer/X-KB-Token/cookie auth plus an optional KB_WEB_RATE_LIMIT. See Configuration below for the full environment surface and Usage for each CLI command.

Architecture

flowchart TD
    src["External sources<br/>GitHub / GitLab / Wiki / MCP / chats / trackers / files<br/>PDF / XLSX / JSON / SQL DDL"] --> fetch["Connector.Fetch → Document (chan)"]
    fetch --> render["render:<br/>Document → markdown + YAML frontmatter"]
    render --> sink["sink: FileSink / APISink / TeeSink<br/>state: .sync-state.json, tombstones.json"]
    sink --> idx["engine (indexer):<br/>AddOrUpdateDocument / RemoveDocument / Reindex"]

    idx --> chunk["chunk:<br/>sentences + ChatChunker"]
    idx --> graphn["graph:<br/>LLM extraction of entities / relations<br/>→ merge / dedup → communities (Louvain) → summaries"]

    chunk --> vstore[("VectorStore<br/>embeddings BLOB, brute-force cosine")]
    chunk --> bm25[("Lexical index<br/>SQLite FTS5 (default) or in-memory BM25")]
    graphn --> gstore[("GraphStore<br/>entities / relations / communities in SQLite")]

    vstore --> retr["retriever.Retriever:<br/>hybrid + graph-aware fusion<br/>dense multi-query + BM25 + RRF + authority + per-doc cap<br/>→ entity-linking → neighbor expansion → community context"]
    bm25 --> retr
    gstore --> retr

    retr --> rerank["rerank.Reranker:<br/>noop / llm / onnx (optional, fail-open)"]
    rerank --> got["got.Orchestrator:<br/>Graph-of-Thoughts<br/>decompose → DAG → waves → aggregate → gaps → finalize"]

    got --> mcp["internal/mcp:<br/>MCP server, stdio + HTTP"]
    got --> web["internal/web:<br/>dashboard, html/template + htmx + SSE"]

Loading

Persistence is a single file $PERSIST_DIR/kb.db: vector tables (chunks), graph tables (entities/relations/communities), kb_meta/corpus_version, and search_history/ask_runs (dashboard search and ask history) in one SQLite database. The lexical index defaults to a SQLite FTS5 table over chunks (KB_FTS5=false reverts to the legacy in-memory BM25 index, which is rebuilt from chunks whenever corpus_version changes).

See docs/architecture.md for the pipeline in detail.

Configuration

Copy .env.example to .env and adjust as needed (or export the variables directly). Most settings have defaults and live in internal/config/env.go; connector-level options such as KB_SOCKS_PROXY are read directly by the connector that needs them (Discord).

Variable	Default	Meaning
KB_ROOT	./kb_root	Root directory for ingested markdown documents
PERSIST_DIR	./kb_root/.persist	Directory for kb.db, .sync-state.json, tombstones
KB_LLM_BASE_URL	http://127.0.0.1:11434	OpenAI-compatible endpoint base URL for the chat/completions model (entity/relation extraction, answer synthesis, rerank). Point at your model host.
KB_EMBED_BASE_URL	http://127.0.0.1:11434	Query-time embeddings (retrieval) — same host as KB_LLM_BASE_URL is fine.
KB_EMBED_INDEX_BASE_URL	KB_LLM_BASE_URL	Bulk indexing embeddings — defaults to the chat endpoint.
KB_LLM_MODEL	qwen3.8:latest	Chat model
KB_EMBED_MODEL	qwen3-embedding	Embeddings model (must support embeddings)
KB_HYBRID	true	Hybrid retrieval (dense + BM25 + RRF); false = dense-only
KB_RERANK	off	Reranker: off | llm | onnx
KB_AUTHORITY_BONUS	notes/=0.15,notes/approved/=0.30	Authority prior bonuses, prefix=bonus,...
KB_NO_PROXY	127.0.0.1	Comma-separated hosts that bypass HTTP(S)_PROXY (direct connection)
KB_TOP_K	10	Default top-K retrieval results
KB_CHUNK_SIZE	4096	Chunk size (tokens)
KB_CHUNK_OVERLAP	512	Chunk overlap (tokens)
KB_RRF_K	60	Reciprocal Rank Fusion constant
KB_CANDIDATE_K	20	Per-leg candidate window before fusion (bench tuning)
KB_PER_DOC_CAP	2	Max chunks per document in fused results
KB_SET_MAX_ROUNDS	3	Max query-variant rounds for set/count retrieval (ModeSet)
KB_QUALIFIER_FILTER	false	Extract structured metadata qualifiers from the question via one LLM call and filter every retrieval leg
KB_ABSTAIN_THRESHOLD	(off)	Float in (0,1]: answer "not found" when every subgoal is uncovered and average coverage is below the threshold
KB_SUPERSEDE_MODE	soft	soft = rank superseded docs lower; strict = drop a superseded doc from synthesis when its replacement is retrieved
KB_INTRA_DOC_BUDGET	(off)	Approx token budget for pulling sibling sections of winning documents into results (intra-document questions)
KB_STALE_AFTER	24h	Sync staleness threshold for doctor / /integrations
KB_COMMUNITY_ALGO	louvain	Community detection: louvain | leiden (Leiden is hierarchical)
KB_LLM_TIMEOUT	60s	Request timeout for embed + chat calls to the endpoint
KB_LLM_NO_THINK	false	Skip chain-of-thought generation on hybrid-reasoning models (e.g. Qwen3) for non-streaming, non-tool Chat() calls (graph extraction, GoT synthesis) by routing them to Ollama's native /api/chat with think:false instead of /v1/chat/completions — the OpenAI-compat endpoint ignores think:false on Ollama and still pays the full reasoning cost. Only affects the LLM pointed at by KB_LLM_BASE_URL; requires that endpoint to be Ollama, serving a hybrid-reasoning model. Some Ollama versions reject the think field outright for a plain (non-hybrid) model — only enable this when KB_LLM_MODEL actually supports thinking mode. Streaming (kb serve's Ask) and tool-calling requests are unaffected.
KB_MAX_SUBGOALS	5	Max GoT subgoals per question; lower cuts per-question LLM calls at some recall cost
KB_MAX_GAP_QUERIES	3	Max GoT gap-refine queries per question; lower cuts per-question LLM calls
KB_DESCRIBE_MODEL	qwen3.8:latest	Chat model used by kb describe (independent of KB_LLM_MODEL)
KB_DESCRIBE_BATCH	10	Batch size for kb describe summary generation
KB_SOCKS_PROXY	(unset)	SOCKS5 proxy (socks5://host:port) used by connectors that need it (e.g. Discord)
KB_FTS5	true	Lexical index backend: SQLite FTS5 (default) vs. the legacy in-memory BM25 index when false
KB_WEB_AUTH_TOKEN	(unset)	Bearer/X-KB-Token/cookie token required by kb serve; mandatory when -addr is non-loopback
KB_WEB_RATE_LIMIT	0 (off)	Requests per minute per client IP for kb serve; only enforced on non-loopback binds

Connector instances are declared in $KB_ROOT/sources.yaml. The file stores only the names of environment variables that hold secrets; values are read from the process environment at resolve time and never written to disk. Format and per-connector options: docs/sources.md.

Usage

go build -o bin/kb ./cmd/kb

doctor

Health and sync-health report: LLM endpoint reachability (embed dimension + chat round-trip), index version/dimension, a PRAGMA integrity_check on kb.db, and a presence-only per-source report (which secret env vars are set, last sync time, staleness vs KB_STALE_AFTER, last sync error).

./bin/kb doctor

config

Dumps the effective runtime configuration as NAME=VALUE lines, including connector/subsystem variables that are read directly instead of going through internal/config. Secret values are redacted to <set> / (unset).

./bin/kb config show                 # current effective configuration
./bin/kb config show --preset fast   # low-latency DRAGON-tuned preset
./bin/kb config show --preset quality # higher-recall DRAGON-tuned preset

Numeric and enum configuration values are validated at startup, including direct connector/subsystem variables such as KB_SOCKS_PROXY and the benchmark/verification thresholds.

backup

Creates a consistent SQLite backup of $PERSIST_DIR/kb.db using VACUUM INTO. If no destination is given, the backup is written to $PERSIST_DIR/backups/kb-<UTC-timestamp>.db; pass an explicit path to control the location. The command refuses to overwrite an existing destination.

./bin/kb backup
./bin/kb backup /backups/kb-$(date -u +%Y%m%dT%H%M%SZ).db

To recover, stop the running server, replace $PERSIST_DIR/kb.db with the backup file, then run ./bin/kb doctor to verify the restored database. Keep backup files off the same disk as $PERSIST_DIR when possible so a single drive failure cannot take both the live index and the recovery copy.

sync

Runs the configured connectors and writes documents to KB_ROOT via the sink. Cursor is advance-on-success with rollback; tombstones prevent re-import of deleted items; prune happens only on full reconcile.

./bin/kb sync --all            # all sources from sources.yaml
./bin/kb sync --source=NAME    # only the named source
./bin/kb sync --all --api=http://127.0.0.1:8321   # push to a running server (POST /documents) instead of writing files

--api mode indexes documents in the server without writing files; API-fed documents are kept across kb reindex (full reindex garbage-collects only filesystem-backed docs that no longer exist). Chat messages are indexed one message at a time in this mode, so reply chains are chunked per message rather than glued into a single thread chunk — run a file-based sync or a full reindex for thread glueing.

reindex

Rebuilds the vector + graph index from the documents under KB_ROOT (chunking → embeddings → LLM graph extraction → communities → summaries). Optional positional argument restricts reindexing to a subpath. A document whose content is unchanged since the last successful index is skipped (content-hash check) — a repeat reindex only pays the embed/LLM cost for files that actually changed. Output reports indexed/skipped/removed counts.

./bin/kb reindex [subpath]
./bin/kb reindex --reembed   # clear stored embeddings + dimension, re-embed from scratch

--reembed clears stored embeddings and the stored dimension before reindexing — use it when switching KB_EMBED_MODEL or recovering from an ErrDimMismatch.

describe

Walks the corpus for documents without a summary frontmatter key and generates a short description via the LLM (fail-open, in batches), writing it back through the sink + indexer and refreshing BM25. Documents that already have a summary are skipped. If the LLM is unreachable, generation falls back to the first ~200 characters of the first meaningful sentence of the body.

./bin/kb describe [--source NAME]   # only describe documents from one source

Settings: KB_DESCRIBE_MODEL (default qwen3.8:latest) and KB_DESCRIBE_BATCH (default 10).

verify

Runs Q&A evals over a golden set built from closed Leon issues (question = issue title, expected = issue body): retrieval + synthesis answers are judged by an LLM with an offline overlap fallback (fail-open), and a report with pass rate and per-source hit rate is written to PERSIST_DIR/last-qa-report.json.

./bin/kb verify [--pairs testdata/leon-qa/qa_pairs.json] [--limit N]
./bin/kb verify --build-golden [--source leon-ai] [--golden-out testdata/leon-qa/qa_pairs.json]
./bin/kb verify --top-k 8 --report $PERSIST_DIR/last-qa-report.json

Flags: --pairs (golden input path, default testdata/leon-qa/qa_pairs.json), --build-golden (rebuild the golden set from KB_ROOT), --golden-out (golden output path), --report (eval report path, default $PERSIST_DIR/last-qa-report.json), --source (source filter for --build-golden, default leon-ai), --limit (evaluate the first N pairs), --top-k (chunks retrieved per question, default KB_TOP_K).

EnterpriseRAG-Bench (kb bench)

Run the EnterpriseRAG-Bench question set (500 questions, 10 categories) against the kb pipeline and emit a leaderboard-ready submission:

unzip all_documents.zip -d /data/erb/corpus
./bin/kb bench \
  --corpus /data/erb/corpus \
  --questions questions.jsonl \
  --out answers.jsonl \
  --limit 50 --types constrained,conflicting_info,completeness,info_not_found

# one-minute sanity run on the checked-in bilingual subset
./bin/kb bench --smoke

# reuse a persisted index and accumulate metrics history
./bin/kb bench --corpus /data/erb/corpus --questions questions.jsonl \
  --persist-dir /data/erb/persist --history /data/erb/bench-history.json

Outputs: answers.jsonl in the official submission format ({"question_id","answer","document_ids"}) plus a per-type metrics report (*.report.json) with document recall vs gold docs, abstention share and citation coverage. Bench-specific knobs: KB_QUALIFIER_FILTER, KB_SUPERSEDE_MODE, KB_ABSTAIN_THRESHOLD, KB_SET_MAX_ROUNDS, KB_CANDIDATE_K, KB_PER_DOC_CAP, KB_INTRA_DOC_BUDGET (see table above). --smoke uses the checked-in testdata/lang-bench subset (16 docs, 20 questions). --persist-dir reuses a corpus index and skips unchanged docs via doc_hashes; --history (or the default next to the report) appends each run's metrics report. The command needs a live LLM endpoint.

Comparing embedders (RU vs EN)

To check whether swapping the embedding model improves Russian retrieval without regressing English, use the checked-in bilingual dataset (testdata/lang-bench: 16 docs and 20 questions tagged language: "ru" or "en"):

KB_EMBED_MODEL=qwen3-embedding ./bin/kb bench \
  --corpus testdata/lang-bench/corpus \
  --questions testdata/lang-bench/questions.jsonl \
  --out /tmp/baseline.jsonl

# swap the embedder, then re-run
KB_EMBED_MODEL=<new-model> ./bin/kb bench \
  --corpus testdata/lang-bench/corpus \
  --questions testdata/lang-bench/questions.jsonl \
  --out /tmp/candidate.jsonl

./bin/kb bench compare \
  /tmp/baseline.jsonl.report.json \
  /tmp/candidate.jsonl.report.json

bench compare prints signed per-language and per-type deltas (candidate minus baseline) for recall, abstention, and citation coverage; add --out delta.json to also write the comparison as JSON. Each run reindexes the supplied --corpus from scratch into an isolated temporary database using KB_EMBED_MODEL, so no manual reindex is needed; embedding dimension mismatches still fail loudly by design.

DRAGON RU RAG-Bench (kb bench-dragon)

Run kb over the Russian DRAGON RAG benchmark (ai-forever/rag-bench-public-texts — 526 news articles, ai-forever/rag-bench-public-questions — 600 questions), fetched directly from HuggingFace, indexed into an isolated temporary database, and answered through the full GraphRAG + Graph-of-Thoughts pipeline:

./bin/kb bench-dragon --limit 5           # quick smoke run

./bin/kb bench-dragon --smoke --persist-dir /tmp/dragon-persist

# full 526-doc / 600-question run
./bin/kb bench-dragon --out answers.dragon.json --concurrency 3

Output: answers.dragon.json, a {question_id: {found_ids, model_answer}} map in the exact shape DRAGON's official evaluator expects for a leaderboard submission. This is a self-run submission file, not an official DRAGON score — the public question set ships without gold answers, so grading happens only when the file is submitted to the DRAGON maintainers. A committed sample run lives at docs/bench/dragon-answers.json (full 600-question set, indexed with graph extraction on). Flags: --limit (cap the question count), --concurrency (parallel questions), --top-k (chunks per subgoal, default KB_TOP_K), --hf-base-url (override the HuggingFace datasets-server endpoint, mainly for testing), --persist-dir/--force-reindex (reuse or rebuild a persisted index), --doc-limit (keep only the first N fetched texts), and --smoke (a fixed 12-doc/5-question sanity subset). bench-dragon score --history PATH appends each score report to a metrics history. The command needs a live LLM endpoint and network access to datasets-server.huggingface.co.

The verify command needs a live LLM endpoint (retrieval + synthesis); integration tests for the QA harness are gated behind -tags integration + KB_LLM_IT=1.

serve

Starts the web dashboard (internal/web):

• Search — synthesized answers, htmx loading indicator, persistent search history (survives restart) shown on /search; clicking a history entry opens its saved answer via /search?id=<id> without re-running retrieval, with a re-run button.
• Ask (Graph-of-Thoughts, SSE progress) — LogicRAG-style adaptive reasoning: decomposition into a dependency DAG, topological wave scheduling, a greedy forward pass that injects resolved dependency answers, bounded rolling memory (KB_ASK_ROLLING_WINDOW), and one dynamic gap-expansion round; progress renders as a structured list of steps (type/status/stage/answer), not raw JSON; runs persist to SQLite, so /ask/history lists past and in-flight runs and a run started before a restart still shows its last known state instead of an empty page. Completed ask responses are cached in SQLite keyed by hash(query + corpus_version + config fingerprint), so a repeated question against an unchanged corpus and configuration returns the previous answer (including fail-open placeholders) without paying LLM/retrieval cost again; stale entries are pruned on startup.
• Documents — summary list, edit form, htmx delete; /documents/view shows a document's graph relationships (entities/relations whose source chunks overlap the document), not just its raw content.
• Integrations — add/edit/delete source instances in sources.yaml.
• Graph — interactive canvas (Cytoscape.js, vendored, zoom/pan/drag) fed by a paginated/filtered /graph/data JSON endpoint (search, community, type, min-degree filters) instead of dumping the whole graph as static SVG; entity/ relation CRUD unchanged, click a node to open its edit panel.
• MCP — /mcp/info shows the live HTTP endpoint, the full tool list, and copy-paste client config for both the stdio and HTTP transports.
• Reports, cleanup, and trash routes.

./bin/kb serve                 # default 127.0.0.1:8080
./bin/kb serve -addr 127.0.0.1:9000   # custom listen address

serve binds to loopback by default: the dashboard has no authentication there and exposes destructive routes. Binding a non-loopback -addr requires KB_WEB_AUTH_TOKEN to be set (the process refuses to start otherwise), which enables bearer/X-KB-Token/cookie token auth on every non-/healthz route; pair it with KB_WEB_RATE_LIMIT (requests per minute per client IP, non-loopback binds only) if the endpoint is reachable beyond a trusted network. An SSH tunnel or authenticating reverse proxy remains the simpler option for remote access.

mcp

Exposes the knowledge base over the Model Context Protocol (internal/mcp): search, ask, get_document, list_sources, add_note, add_source, graph_query, generate_report, reindex, status.

Two transports:

• stdio — kb mcp, for local process integration (Claude Desktop, Claude Code).
• HTTP — mounted at /mcp inside kb serve (same tool set, streamable HTTP transport), for remote/networked MCP clients. The dashboard's /mcp/info page shows the live endpoint URL, the full tool list, and copy-paste client config for both transports.

./bin/kb mcp

Connectors

Registered types (in internal/connectors/registry, wired in cmd/kb/connectors.go):

Type	Source	Notes
github	GitHub org or explicit repos	issues, PRs, contents, wiki
gitlab	GitLab group or projects	issues, MRs, wikis, files
wiki	MediaWiki or Confluence Cloud	config.variant: mediawiki|confluence
mcp	MCP server	stdio or HTTP transport
telegram	Telegram chats	bot token
slack	Slack channels	bot token
mattermost	Mattermost team/channels	base URL + token
yandex-tracker	Yandex Tracker queues	OAuth token + org id
youtrack	YouTrack projects	base URL + token
kaiten	Kaiten spaces	base URL + token
weeek	Weeek spaces	token
searchapi	Generic search API	configurable fields/pagination/auth
discord	Discord guild channels	bot token; optional SOCKS via KB_SOCKS_PROXY
trello	Trello board	public export, or API key/token for private boards
rss	RSS 2.0 feed	config.feed_url
web	Website via sitemap or explicit pages	config.sitemap_url or config.pages, content_selector
file	Local directory	applies file importers by extension

Connector secrets: GitHub/GitLab token, wiki token/email, chats token, Discord token, trackers token (Trello also key), searchapi per-config auth_* keys — all as env-var names. rss and web need no secrets.

File importers

internal/importer maps file extensions to importers, used by the file connector during kb sync (kb reindex indexes the markdown documents under KB_ROOT instead):

Extension	Importer
.pdf	PDF text extraction (pure-Go, pdftotext fallback)
.xlsx	Excel worksheets → documents
.json	JSON documents (gjson paths)
.sql	SQL DDL schema documents
.md	Legal-codex structural parser (legalru) — non-legal markdown yields no documents
.go	Go source → code-graph documents (code importer)

Memory upgrades: temporal graph, code graph, retrieval modes

Temporal knowledge graph

relations are bi-temporal (internal/store/sqlite): valid_from/valid_to carry when a fact is true in the real world (open-ended when NULL, e.g. a statute redaction date), created_at/expired_at carry when the system learned / stopped considering the fact. On an ingestion conflict (same src + predicate, different dst, still-open valid_to) the old edge is closed (valid_to/expired_at = time of the new fact), not overwritten — the audit trail is preserved.

• GraphStore.RelationsAsOf(ctx, ids, t) — point-in-time query: valid_from <= t AND (valid_to IS NULL OR valid_to > t).
• Neighbors/MatchEntities take an optional time parameter; the default is "now", so existing callers keep the current behavior.
• Legal articles carry their amendment history in frontmatter (redactions: YYYY-MM-DD:FZ,... plus redaction_date/fz_number for the latest revision); deterministic AMENDS edges get valid_from = amendment date (internal/graph/legal.go). Plenum clarifications use INTERPRETS edges and are intentionally non-temporal.

Legal corpus importer (legalru)

internal/importer/legalru is a deterministic structural parser for Russian legal codes: it splits a curated markdown codex (part → section → chapter → article, amendment history, Plenum resolutions) into one Document per article, no LLM involved. Documents carry kind: legal-article (Plenum points: legal-plenum), the ID scheme code/чN/рN/глN/стN, and frontmatter (code, code_title, article_number, article_title, redactions...). The curated gold corpus lives in internal/importer/legalru/testdata/gold/ (ГК РФ, часть первая + Постановление Пленума ВС РФ N 25) with expected_graph.json and qa_pairs.json — see docs/legal-gold-corpus.md.

Code knowledge graph

internal/graph/codegraph extracts a deterministic graph from Go sources with go/ast + go/types, no LLM call: nodes Package/Function/Type/Method, typed edges Calls/Imports/Implements/Declares, discriminated with kind=code. The indexer routes .go documents to this path and skips files with syntax errors (fail-open). Retrieval links symbol names from queries to entities and expands neighbors along Calls/Imports edges exactly like semantic edges.

Retrieval modes

retriever.Options.Mode selects the pipeline (internal/engine/retriever):

• local (default) — the hybrid graph-aware pipeline: dense multi-query + BM25 + RRF + authority prior + per-doc cap → entity-linking → neighbor expansion → community context.
• global — map-reduce over root-level community summaries: parallel partial answers (top 20 communities) + one reduce; degrades to local when no hierarchy exists (fail-open).
• drift — a vector search over community summary embeddings seeds a local refine (top 3 communities, up to 30 seed chunks).

The GoT decompose step picks a mode per sub-question ("main topics / how many" → global; "what exactly about X" → local; otherwise drift). Community detection: KB_COMMUNITY_ALGO=louvain|leiden (default louvain; Leiden produces the multi-level hierarchy, Louvain remains the fallback).

Chat two-phase extraction

Chat documents (kind: message, from the telegram/slack/mattermost connectors) get a thread-scope mini-graph instead of generic extraction: a deterministic phase attributes each message to its speaker (frontmatter user), filters small talk (heuristics, plus an optional LLM classifier that overrides them), and stamps DECIDED/PROPOSED/AGREED edges with the speaker's message timestamp; an LLM phase extracts topic entities and the decision edges themselves. Multi-message threads are glued into one chunk with per-speaker attribution (speakers chunk metadata), so edges are credited to the right participant, not the thread's first author. Edited messages are re-delivered (not skipped) and carry a normalized edit_at frontmatter key (RFC3339, matching updated_at): telegram edited_message/edited_channel_post, slack edited.ts, mattermost EditAt — so edits re-enter the indexing/lineage path.

Verification layer

• internal/verify — DiffGraph golden-graph diff (missing/extra/mismatched report), CheckCitations (every citation in an answer must exist in the retrieved context actually passed to the LLM), ContradictionDetector (LLM pass over retrieved chunks flags explicit contradictions; fail-open, off by default in GoT).
• internal/verify/legaleval — legal faithfulness harness (integration-only, KB_LLM_IT=1): Non-Hallucinated-Statute-Rate, Statute-Relevance-Rate, Legal-Claim-Truthfulness over the gold qa_pairs. Methodology and metric definitions: docs/legal-gold-corpus.md.
• internal/verify/qa — closed-issue Q&A evaluation (kb verify): LLM judge with a token-overlap fallback, writes last-qa-report.json. The fallback path is tested offline with no judge, so QA scoring runs without a live endpoint.
• The deterministic fake-LLM e2e (internal/integration/e2e_fake_test.go) wires DiffGraph and CheckCitations into the import→index→ask path and runs under make check with no network.

Incremental reindexing

• Chunk lineage: the update path soft-closes old chunk versions (valid_to, replaces) instead of physically deleting them (VectorStore.SoftCloseByDoc); retrieval and BM25 see only active chunks (valid_to IS NULL), while ChunksByDoc returns the full version history for lineage links. RemoveDocument still hard-deletes and clears superseded_by marks on other docs.
• Blast-radius supersession: after indexing, chunks of other docs sharing ≥ N entities (default 1) with the new doc's touched set are marked superseded_by = <ref_doc_id> (GraphStore.OverlappingChunks + VectorStore.SetSuperseded). Retrieval applies a soft ×0.9 rank penalty but never excludes them (fail-open — conflicts are still caught by verify.ContradictionDetector).
• Lazy communities: writes mark affected components stale=1 instead of running Leiden per write; GraphUpdater.RefreshStaleCommunities recomputes only stale components in batch at the end of a sync batch and lazily on a query throttle in the retriever. A failed refresh degrades to serving stale summaries as-is (fail-open).
• End-to-end demonstration of this actualization path, with a permanent CI-safe regression test: docs/bench/actualization-report.md. The regression test proves chunk soft-close (VectorStore) and bi-temporal relation-close (GraphStore) both work; live Slack input only reliably drives the chunk-level supersession, since ChatExtractor (the real extraction path for chat-sourced documents) only emits decision-tracking edges, not typed facts, so it doesn't produce a relation to close.

Development

go test ./...     # all offline tests: unit + fake-LLM e2e (no network, no live endpoint)
go vet ./...
gofmt -l .

Integration tests that talk to a real LLM endpoint are gated behind the integration build tag and KB_LLM_IT=1:

KB_LLM_IT=1 go test -tags integration ./...

Testing conventions: DI seams everywhere (EnvLookup, httptest.Server, injected HTTPDoer/clock, in-memory Sink, fake LLM/Embedder/Reranker/GraphStore), golden files in testdata/, fail-open behavior asserted on every stage.

Adding a connector

Step-by-step guide with a checklist of test axes: docs/new-connector.md.

Documentation

Start from the docs index: docs/README.md.

• docs/architecture.md — GraphRAG pipeline diagram and data flow
• docs/sources.md — sources.yaml format and per-connector options
• docs/new-connector.md — how to add a new connector
• docs/legal-gold-corpus.md — legal gold-corpus methodology and eval metrics
• docs/bench/actualization-report.md — chat-actualization demo (temporal updates change answers)
• CONTRIBUTING.md — development and testing conventions
• SECURITY.md — loopback/no-auth design and vulnerability reporting
• CHANGELOG.md — release notes

License

Apache License 2.0. See LICENSE and NOTICE.