Merge remote-tracking branch 'upstream/main' into pr/add-mutation-coordinator

# Conflicts:
#	openkb/cli.py
#	openkb/indexer.py

Author: gwokhou

.github/workflows/ci.yml

@@ -0,0 +1,53 @@
+name: CI
+
+on:
+  push:
+    branches: ["main"]
+  pull_request:
+
+# Least-privilege token: this workflow only reads the repo. Arbitrary
+# dependency build code runs during install, so never expose a writable
+# token to it (see the supply-chain notes in pyproject.toml).
+permissions:
+  contents: read
+
+# Cancel superseded runs on the same ref (rapid PR pushes) instead of
+# letting them pile up and post stale statuses.
+concurrency:
+  group: ci-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  check:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332  # v4.1.7
+        with:
+          persist-credentials: false
+
+      - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39  # v8.2.0
+        with:
+          version: "0.10.2"
+          enable-cache: true
+
+      # `uv sync --locked` installs the exact uv.lock resolution (direct AND
+      # transitive deps) and fails if the lock is stale — a bare `pip install`
+      # would ignore the lockfile and let transitive versions float, defeating
+      # the repo's exact-pin supply-chain policy. Run `uv lock` and commit the
+      # lockfile whenever pyproject dependencies change.
+      - name: Install (locked)
+        run: uv sync --locked --extra dev --python 3.12
+
+      # --no-sync: don't let `uv run` re-sync without the dev extra and
+      # uninstall the tools it is about to run.
+      - name: Ruff lint
+        run: uv run --no-sync ruff check .
+
+      - name: Ruff format check
+        run: uv run --no-sync ruff format --check .
+
+      - name: Mypy
+        run: uv run --no-sync mypy openkb
+
+      - name: Pytest
+        run: uv run --no-sync pytest

.github/workflows/publish.yml

@@ -29,7 +29,7 @@ jobs:
       id-token: write   # OIDC trusted publishing to PyPI
       contents: write   # Create GitHub Release
     steps:
-      - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332  # v4.2.2
+      - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332  # v4.1.7
         with:
           fetch-depth: 0   # hatch-vcs needs full history + tags
 

.gitignore

@@ -16,5 +16,11 @@ wiki/
 output/
 
 # Local only
-docs/
+docs/internal/
 .claude/
+
+# Heavy test-input documents for the examples (the old blanket `docs/` rule
+# used to catch this dir at any depth; the anchored `docs/internal/` above
+# does not). The PDFs already tracked on main stay tracked — this only stops
+# new drops from being swept into commits.
+examples/docs/

AGENTS.md

@@ -0,0 +1,44 @@
+# AGENTS.md — OpenKB map for coding agents
+
+OpenKB compiles raw documents into an interlinked wiki knowledge base using
+LLMs (vectorless retrieval via PageIndex). This repo is developed **agent-first**:
+humans steer, agents execute. Optimize changes for agent legibility.
+
+## Read next
+- `docs/golden-principles.md` — mechanical rules to follow (enforced where possible).
+- `docs/internal/superpowers/{specs,plans}/` — design history & plans *(maintainer-local, not in git)*.
+- `README.md` — user-facing overview and commands.
+
+## Dev commands
+- Install: `pip install -e ".[dev]"`  (or `uv sync --extra dev` — plain `uv sync` skips the dev tools)
+- Run CLI: `openkb <command>`  (entry point: `openkb.cli:cli`)
+- Test: `pytest`
+- Lint/format/types: `ruff check .` · `ruff format .` · `mypy openkb`
+
+## Module map (openkb/)
+- `cli.py` — Click CLI entry point & command wiring *(large; see tech-debt)*.
+- `config.py` — config loading/validation (LiteLLM passthrough, env).
+- `converter.py` — document → markdown conversion (markitdown).
+- `url_ingest.py` — fetch & ingest URLs (trafilatura).
+- `images.py` — figure/image extraction & handling.
+- `indexer.py` — PageIndex tree indexing for long docs.
+- `mutation.py` — crash-safe, serial KB mutations.
+- `locks.py` — atomic writes / file locking (`atomic_write_text`, portalocker).
+- `state.py` — run/session state tracking.
+- `frontmatter.py` — YAML frontmatter round-trip (OKF).
+- `schema.py` — page/content schema constants & helpers.
+- `lint.py` — structural wiki lint (broken links, orphans, index sync).
+- `tree_renderer.py`, `visualize.py`, `watcher.py` — rendering / graph / file watch.
+- `agent/compiler.py` — LLM wiki compiler *(large; see tech-debt)*.
+- `agent/linter.py` — semantic (LLM) wiki lint (contradictions, gaps, staleness).
+- `agent/chat.py`, `agent/chat_session.py` — chat over the wiki *(chat.py large)*.
+- `agent/query.py` — one-off query generator.
+- `agent/tools.py` — shared wiki read/write tool functions used by query/linter (and by chat indirectly via `query.build_chat_agent`).
+- `agent/skills.py`, `agent/skill_runner.py`, `skill/` — Skill Factory.
+- `deck/`, `templates/`, `prompts/` — deck output, templates, prompt assets.
+
+## Hard invariants
+- Deps are pinned **exactly** (supply-chain caution). Vet before bumping.
+- Wiki writes go through `locks.py` / `mutation.py` (never ad-hoc).
+- Modules stay < 800 lines (`tests/test_file_size.py`); grandfathered files are in tech-debt.
+- Keep this file a short map — put depth in `docs/`.

CLAUDE.md

@@ -0,0 +1 @@
+@AGENTS.md

docs/.gitignore

@@ -0,0 +1,8 @@
+# Default-closed: docs/ content stays out of git unless explicitly
+# allowlisted below. This repo publishes code, not design/spec docs — the
+# allowlist restores the safety net the old blanket `docs/` ignore provided,
+# so a doc accidentally written outside docs/internal/ can't be swept into a
+# commit by `git add -A`. To publish a new doc, add a `!its-name.md` line.
+*
+!.gitignore
+!golden-principles.md

docs/golden-principles.md

@@ -0,0 +1,32 @@
+# Golden Principles
+
+Opinionated, mechanical rules that keep this agent-generated codebase legible
+and consistent for future agent runs. Enforced by CI where possible; the rest
+are honored by convention and checked in review. When a rule proves valuable,
+promote it into a lint (see `tests/test_file_size.py` for the pattern).
+
+## Boundaries
+- **Validate data shapes at boundaries.** Parse/validate inputs (frontmatter via
+  `openkb/frontmatter.py`, config via `openkb/config.py`) at the edge. Never build
+  on guessed shapes.
+
+## Reuse
+- **Prefer shared utilities over hand-rolled helpers** so invariants stay
+  centralized. Check `openkb/` for an existing helper before writing a new one.
+
+## I/O and state
+- **All wiki file writes go through `openkb/locks.py` / `openkb/mutation.py`**
+  (atomic, crash-safe). No ad-hoc writes to the wiki tree.
+- **Log through `openkb/log.py`**, not bare `print`, for anything diagnostic.
+
+## Size and shape
+<a id="file-size"></a>
+- **Keep modules focused and under 800 lines** (enforced by
+  `tests/test_file_size.py`). Split large modules into focused units by
+  responsibility. Existing over-limit files are grandfathered (with reasons)
+  in the test's `_GRANDFATHERED` set and additionally tracked in
+  `docs/internal/tech-debt.md` *(maintainer-local, not in git)*.
+
+## Docs
+- **`AGENTS.md` is a map, not a manual.** Keep it short; deep/local docs live
+  under `docs/` (public) and `docs/internal/` (maintainer-local, not in git).

openkb/__init__.py

@@ -1,5 +1,7 @@
 """OpenKB package."""
-from importlib.metadata import PackageNotFoundError, version as _version
+
+from importlib.metadata import PackageNotFoundError
+from importlib.metadata import version as _version
 
 try:
     __version__ = _version("openkb")

openkb/__main__.py

@@ -1,4 +1,5 @@
 """Allow running OpenKB as ``python -m openkb``."""
+
 from openkb.cli import cli
 
 cli()

openkb/agent/_markdown.py

@@ -15,7 +15,6 @@
 from rich.syntax import Syntax
 from rich.text import Text
 
-
 INLINE_CODE_STYLE = "blue"
 BLOCKQUOTE_BAR = "\u258e"