diff --git a/.github/workflows/_ci.yml b/.github/workflows/_ci.yml index e4e1559..9722f36 100644 --- a/.github/workflows/_ci.yml +++ b/.github/workflows/_ci.yml @@ -53,7 +53,7 @@ jobs: if: startsWith(matrix.os, 'ubuntu-') run: | sudo apt-get update - sudo apt-get install -y cmake ninja-build git gcc g++ pkg-config libssl-dev ccache + sudo apt-get install -y cmake ninja-build git gcc g++ pkg-config libssl-dev ccache patchelf # LadybugDB shared libraries are vendored under engine/third_party/ladybug/lib/ # (committed to git, both x86-64 and aarch64). CMake selects the # correct architecture at configure time. Add the lib dir to the @@ -204,10 +204,33 @@ jobs: cp target/release/codescope package/ cp LICENSE package/ [ -f install.sh ] && cp install.sh package/ - # Bundle the parallel indexer script so users/AIs can opt into - # multi-process acceleration for large projects. The script is - # platform-independent bash and auto-discovers the binary. - [ -f codescope-parallel.sh ] && cp codescope-parallel.sh package/ && chmod +x package/codescope-parallel.sh + + # Bundle vendored LadybugDB shared library and set rpath so the + # binary can find it at runtime without Homebrew/system install. + # The binary links against @rpath/liblbug.0.dylib (macOS) or + # liblbug.so.0 (Linux), resolved via the embedded rpath. + case "${{ matrix.artifact }}" in + *-x86_64-linux) + LBUG_LIB="engine/third_party/ladybug/lib/linux/liblbug.so.0" + LBUG_FILE="liblbug.so.0" + ;; + *-aarch64-linux) + LBUG_LIB="engine/third_party/ladybug/lib/linux-aarch64/liblbug.so.0" + LBUG_FILE="liblbug.so.0" + ;; + *-macos) + LBUG_LIB="engine/third_party/ladybug/lib/macos/liblbug.0.dylib" + LBUG_FILE="liblbug.0.dylib" + ;; + esac + cp -L "$LBUG_LIB" "package/$LBUG_FILE" + # Set rpath to $ORIGIN (Linux) / @executable_path (macOS) so the + # dynamic linker finds the bundled library in the same directory. + if [[ "${{ matrix.os }}" == ubuntu-* ]]; then + patchelf --set-rpath '$ORIGIN' package/codescope + elif [[ "${{ matrix.os }}" == macos-* ]]; then + install_name_tool -add_rpath @executable_path/. package/codescope 2>/dev/null || true + fi cd package tar -czf "../${{ matrix.artifact }}.tar.gz" ./* diff --git a/README.md b/README.md index 120e225..f7fd911 100644 --- a/README.md +++ b/README.md @@ -4,13 +4,11 @@ It transforms source code into verifiable facts, understandable models, and inspectable evidence — enabling AI to validate claims against reality instead of hallucinating. ---- - -### What CodeScope is NOT +**Version**: v0.2.1 | **License**: Apache 2.0 -CodeScope is **not** a code explainer, a semantic analyzer, or a replacement for reading code. It does not understand what `Arc` means, why `Rc` is not thread-safe, or how a JWT middleware works. **That is the AI's job.** +--- -### What CodeScope IS +## 1. What Is CodeScope? CodeScope is a **Project Truth Engine** that answers one question: @@ -18,171 +16,35 @@ CodeScope is a **Project Truth Engine** that answers one question: Not "what does this code mean", but "does the code actually do what you claim?" -### By the Numbers - -| Metric | Value | -|--------|-------| -| Languages | 8 (Rust, Go, C/C++, Python, Java, JS/TS) | -| Index speed | 1-10s (100+ files) | -| Query latency | 0.3-1.5 ms | -| Token savings | **98.9%** (260K lines → 40K tokens) | -| MCP tools | 37 (Locate / Understand / Verify / Index) | -| Architecture | Facts → Resolution → Models → Verification | - -### What It Can Do - -| AI says | CodeScope checks | Data source | -|---------|----------------|-------------| -| "登录模块支持 JWT" | JWT library exists? login calls jwt? tests exist? | `entity` + `relation` + `import` | -| "This module is complete" | All functions have callers? coverage adequate? | `relation` + `DeadCodeInspector` | -| "PR fixed memory leak" | Corresponding free exists? error path tested? | `relation` + test file check | -| "Architecture is Controller→Service→Repository" | Does code actually follow this layering? | `architecture_edge` | -| "Module supports 6 languages" | Do the adapters actually exist? | `entity` + `import` | - -### Knowledge Graph (Side Product) - -CodeScope builds a **module-level knowledge graph** as a side product of the verification pipeline. It learns not "code text" but *structured "how this project is organized, what's important, what's redundant, what it promises"* — a four-layer stack: - -| Layer | Table | What it tells you | -|-------|-------|-------------------| -| **Call graph** (who calls who) | `relation` (type=1), `architecture_edge` | Cross-module call dependencies; drives `detect_architecture_drift` | -| **Module health** (who's important / who's dead) | `module_summary` | Per-module `incoming_count` / `outgoing_count` / `dead_entities` / `utilization` / `confidence`; drives `explain_module` | -| **Module dependency** (who impacts whom) | `architecture_edge` (edge weight = call count), `module_edge` | "Change module A → these modules depend on it" | -| **Documented capability** (what it claims it can do) | `capability` + `document` | README-extracted capabilities; drives `detect_capability_drift` / `verify_claim` | - -The knowledge graph is not the product — it is the infrastructure that powers verification. - -#### `module_summary.role`: multi-signal fusion classifier (v0.2.1) - -The `role` column is auto-populated by a **multi-signal fusion CASE** classifier (`engine/src/model/state_builder.cpp`), not a single-source heuristic. It fuses call-graph counts with two signals the call graph cannot give: - -| Signal | Source | What it adds | -|--------|--------|--------------| -| `pub_count` | `entity.visibility=1` (pub/public/export per language Visitor) | Distinguishes external interface layers from internal implementation | -| `entry_reachable` | `graph_nodes.is_entry_point` (main/init/setup/run/handler) | Whether this module is an entry layer | - -Rules match by **priority** (first hit stops): - -| Priority | Role | Rule (multi-signal) | -|----------|------|---------------------| -| 1 | `test` | module name contains `test`/`tests`/`_test`/`mod tests` | -| 2 | `api` | `pub_count > 0 AND incoming ≥ 2×outgoing AND incoming ≥ 3 AND utilization ≥ 0.1` | -| 3 | `entry` | `entry_reachable > 0` | -| 4 | `core` | `incoming ≥ 10 AND outgoing ≤ incoming×1.0 AND utilization ≥ 0.05 AND pub_count > 0` | -| 5 | `utility` | `outgoing ≤ 5 AND pub_count > 0 AND utilization ≥ 0.05` | -| 6 | `business` | `pub_count > 0 AND incoming ≥ 10` (implementation layer — many depend, many deps; outgoing too high for core/api) | -| 7 | `dead` | `incoming=0 AND outgoing=0`, OR `dead_entities = total` | -| 8 | `infra` | true fallback — didn't match any semantic rule | - -Treat `role` as a hint informed by fusion, not a court verdict. Thresholds are `constexpr` in `state_builder.h` — retune against `bun` if your project's role distribution looks off (e.g. `infra > 30%` means thresholds too strict). See `docs/dev_plans/role_classifier_plan.md` for the full design and the v0.2.1 threshold retune log. - -#### What lands in the knowledge graph (memscope-rs, 215 files) - -| Table | Rows | Meaning | -|-------|-----:|---------| -| `entity` | 4,310 | Fine-grained code entities (functions, types, vars) | -| `relation` | 726 | Inter-entity relations (type 3 = containment / definition) | -| `architecture_edge` | 3,351 | Module / directory-level architecture dependencies (edge weight = call count) | -| `module_edge` | 11 | Cross-module dependency edges | -| `capability` | 3 | Capabilities extracted from the README | -| `document` | 1 | README document record | -| `module_summary` | 42 | Per-module knowledge cards | - -What you can learn from it: - -- **Architecture dependencies** — `architecture_edge` tells you which modules depend on which (e.g. `analysis/heap_scanner → unsafe_inference`, weight = number of calls). Drives `detect_architecture_drift`. -- **Capability declarations** — `capability` + `document` structure "what the README claims the project can do". Drives `detect_capability_drift` / `verify_claim(capability_exists)`. -- **Module summaries** — 42 per-module knowledge cards, surfaced by `explain_module`. - -#### Direct query access (v0.2.1) - -The knowledge graph used to be **implicit** — `graph_query` walks the *call* graph (`graph_nodes` / `graph_edges`), not the knowledge-layer tables (`entity` / `relation` / `architecture_edge`). You benefited from it only indirectly via `explain_module`, `detect_capability_drift`, `get_module_tree`. - -v0.2.1 adds `engine_get_knowledge_graph(project_id, table, limit)` + the MCP tool `get_knowledge_graph` so you can now **directly browse** any knowledge-layer table: - -```jsonc -get_knowledge_graph {"table":"architecture_edge","limit":5} -// → {"table":"architecture_edge","rows":[{"id":1,"layer_lower":"analysis/heap_scanner","layer_upper":"unsafe_inference","entity_id":42}],"total":3351,"truncated":false} - -get_knowledge_graph {"table":"capability","limit":10} -// → {"table":"capability","rows":[{"id":1,"name":"borrow_analysis","summary":"scope-aware borrow checking"}],"total":3,"truncated":false} -``` - -Supported tables: `entity`, `relation`, `architecture_edge`, `module_edge`, `capability`, `document`, `module_summary`. Block-level FFI — one call returns the whole result set, never one row per call. - ---- -## Quick Start - -### 60 seconds to your first index - -```bash -# 1. One-command build (auto-detects OS, installs deps, compiles) -bash <(curl -fsSL https://raw.githubusercontent.com/Timwood0x10/CodeScope/main/bootstrap.sh) - -# 2. Index a project -codescope cli index_project '{"project_path":"/path/to/your/project"}' - -# 3. Query -codescope cli get_graph_stats '{}' -# → {"total_nodes":12345,"total_edges":6789,"total_files":99} - -# 4. Start MCP server (for AI clients) -codescope -``` - -📖 详细的中文快速开始指南见 [QUICK_START.md](QUICK_START.md) - -### Prerequisites - -| Platform | Dependencies | One-command | Status | -|----------|-------------|-------------|--------| -| **macOS** | Xcode CLT, cmake, Rust | `bash bootstrap.sh` | ✅ Supported | -| **Linux** | build-essential, cmake, Rust | `bash bootstrap.sh` | ✅ Supported | -| **Windows** | MinGW-w64 (gcc/g++), CMake 3.30+, Rust | `.\install.ps1` | 🚧 Planned | - -> Pre-built binaries are available for **Linux** and **macOS** on the [Releases page](https://github.com/Timwood0x10/CodeScope/releases). -> **Windows** support is planned for a future release. The C++ engine and Rust server build with MinGW-w64; see [#issue] for tracking progress. +It indexes source code into a structured code graph (call graph + reference graph + module knowledge), then exposes **37 MCP tools** that let AI agents locate symbols, trace call paths, verify claims, detect documentation drift, and analyze architecture — all with **~98.9% token savings** vs reading raw source files. -**One-command install (Linux / macOS)**: - -```bash -curl -fsSL https://raw.githubusercontent.com/Timwood0x10/CodeScope/master/install.sh | bash -``` - -After install, `~/.codescope/bin/` contains: - -| File | Purpose | -|------|---------| -| `codescope` | Main binary (CLI + MCP server) — single-process `index` and built-in multi-process `index-parallel` scheduler for large projects | - -Add to PATH and use: - -```bash -export PATH="$PATH:$HOME/.codescope/bin" - -# Small-medium projects: index directly -codescope cli index_project '{"project_path":"/path/to/project"}' - -# Large projects (thousands–tens of thousands of files): use the built-in scheduler -codescope index-parallel /path/to/large/project -``` - -### Build from source manually - -```bash -# macOS: -brew install llvm@21 cmake pkg-config sqlite3 ladybug -cargo build --release +### Supported Languages (8) -# Linux (Ubuntu): -sudo apt-get install -y build-essential cmake llvm-dev libclang-dev libsqlite3-dev -curl -fsSL https://install.ladybugdb.com | sh -cargo build --release -``` +| Language | Parser | IR Translator | Verified | +|----------|--------|---------------|----------| +| Python | ✅ | ✅ | ✅ | +| Go | ✅ | ✅ | ✅ | +| C | ✅ | ✅ | ✅ | +| C++ | ✅ | ✅ | ✅ | +| Rust | ✅ | ✅ | ✅ | +| JavaScript | ✅ | ✅ | ✅ | +| TypeScript | ✅ | ✅ | ✅ | +| Java | ✅ | ✅ | ✅ | + +### Tech Stack + +| Layer | Technology | +|-------|-----------| +| Parser | tree-sitter (unified AST IR, 8 languages) | +| Indexer | C++23 (Clang 17+), SQLite (WAL mode, FTS5) | +| Server | Rust 2024 Edition, MCP Protocol (JSON-RPC 2.0, stdio transport) | +| Graph Storage | SQLite (primary) + optional LadybugDB (Cypher queries) | +| Scheduler | Built-in multi-process parallel indexer (chunk-level work-stealing) | +| Build | CMake 3.30+ (C++), Cargo (Rust) | --- -## Architecture +## 2. Architecture ```mermaid graph TB @@ -191,15 +53,15 @@ graph TB end subgraph "Rust MCP Server" - MCP["MCP Protocol (JSON-RPC 2.0)
37 tools / protocol / transport"] + MCP["MCP Protocol (JSON-RPC 2.0)
37 tools / stdio transport"] DISPATCH["Tool Dispatch
project_id auto-restore"] end subgraph "C++ Core Engine" - PARSER["Parser
tree-sitter → unified IR
6 languages"] + PARSER["Parser
tree-sitter → unified IR
8 languages"] FACTS["Facts Repository
entity / reference / scope / import"] RESOLVER["Resolver Pipeline
Constraint Chain"] - MODEL["Model Engine
Plugin: Workflow / Capability
Architecture / Contract"] + MODEL["Model Engine
Workflow / Capability
Architecture / Contract"] INSPECTOR["Inspector
DeadCodeInspector / verify_integrity"] end @@ -245,32 +107,11 @@ Model Engine ------ workflow / capability / architecture / contract Inspector --------- evidence / finding ``` -### Data Flow - -``` -Facts Layer: 项目里有什么？ 实体、引用、作用域、导入 -Resolution Layer: 谁调了谁？ 调用边、依赖关系、resolve_strategy -Model Layer: 项目怎么工作的？ 工作流、能力、架构、契约 -Verify Layer: 真的吗？证据在哪？ 证据、发现 -``` - -> **v0.2.1 — `resolve_strategy` propagation** -> Each call edge now carries a `resolve_strategy` tag on its way out of the -> Resolver Pipeline: `p1_intra` (resolved in-project), `external` (builtin / -> third-party), `unresolved` (unknown). Frontends can filter `external` / -> `unresolved` out of `find_callees` / `find_callers` results to eliminate -> third-party false positives (e.g. `dropout`, `backward_hook`, `means`, -> `stds` no longer surface as in-project callees). Verified across 8 languages -> on the `bun` project: 100% of `edge_type=1` (call) edges carry a non-empty -> strategy. See `docs/bugs/bug_resolve_strategy.zh.md` for the full fix chain. - -``` - -### Query Flow (Tool Dispatch) +### Query Flow ```mermaid flowchart LR - Q["MCP Client
tool call"] --> Q1["Server receives
project_id auto-restore
from DB (getLatestProjectId)"] + Q["MCP Client
tool call"] --> Q1["Server receives
project_id auto-restore"] Q1 --> Q2{"Tool type?"} Q2 -->|"index_project"| Q3["Spawn worker subprocess
→ memory isolated
→ exits after done"] Q2 -->|"query tools"| Q4["C++ FFI → SQLite query
graph_nodes, graph_edges
search_index, ..."] @@ -281,7 +122,7 @@ flowchart LR Q6 --> R ``` -### Two-Phase Design +### Two-Phase Indexing ```mermaid flowchart LR @@ -303,249 +144,223 @@ flowchart LR A -->|"trigger"| B ``` -## MCP Tools (37 tools) +--- -### Project & Stats +## 3. Quick Start -| Tool | Description | Token Cost | -|------|-------------|:----------:| -| `project_overview` | **Primary** — comprehensive overview: languages, modules, symbols, entry points | **~71** | -| `get_graph_stats` | Quick statistics: nodes, edges, files | **~18** | -| `get_module_tree` | Hierarchical module/directory tree | **~4** | -| `get_entry_points` | Find entry points (main/init/setup/run/handler) | **~5** | -| `get_routes` | Get registered HTTP routes (Gin/Echo/Chi/net/http) | **~50** | -| `get_type_info` | Query type definitions (struct/enum/trait) with reference counts | **~50** | +### Prerequisites -### Symbol Lookup +| Platform | Dependencies | +|----------|-------------| +| **macOS** | Xcode CLT, cmake, Rust (1.85+) | +| **Linux** | build-essential, cmake, Rust (1.85+) | -| Tool | Description | Token Cost | Note | -|------|-------------|:----------:|------| -| `find_symbol` | **Recommended** — find symbol by exact name (kind, file, line/col) | **~30** | | -| `find_definition` | `[DEPRECATED]` Find symbol definition location | **~20** | Use `find_symbol` | -| `find_references` | Find all locations referencing a symbol | **~30** | | -| `explain_symbol` | Get comprehensive symbol info: definition, callers, callees, dependencies | **~50-200** | One-shot deep dive | +### Install Pre-built Binary -### Call Graph +```bash +curl -fsSL https://raw.githubusercontent.com/Timwood0x10/CodeScope/main/install.sh | bash +export PATH="$PATH:$HOME/.codescope/bin" +``` -| Tool | Description | Token Cost | Side Effects | -|------|-------------|:----------:|--------------| -| `find_callers` | Find who calls a function | **~10-50** | Requires CALLS edges | -| `find_callees` | Find what a function calls | **~10-50** | Same | -| `codescope_trace` | Interactive recursive call exploration (depth + direction) | **~50-200** | Large depth = large output | -| `trace_flow` | Recursive execution flow tracing (caller→callee chain) | **~50-200** | Same | -| `shortest_path` | Shortest call path between two functions (BFS) | **~50-100** | | -| `connected_components` | Connected components in the call graph — find independent modules | **~50** | | +### Build from Source -### Search +```bash +git clone https://github.com/Timwood0x10/CodeScope.git +cd CodeScope -| Tool | Description | Token Cost | -|------|-------------|:----------:| -| `search` | **Recommended** — unified search (auto-selects FTS5 or semantic) | **~300-1000** | -| `search_code` | `[DEPRECATED]` Legacy FTS search, use `search` instead | **~300-1000** | +# macOS: +brew install llvm@21 cmake pkg-config +cargo build --release -### Verification Layer +# Linux (Ubuntu): +sudo apt-get install -y build-essential cmake llvm-dev libclang-dev +cargo build --release +``` -| Tool | Description | Token Cost | -|------|-------------|:----------:| -| `verify_integrity` | Check README-promised features actually exist in code | **~100** | -| `verify_claim` | Verify a single claim (capability_exists / contract_holds / architecture_follows) | **~100** | -| `verify_summary` | Parse natural-language summary and verify each claim | **~200-500** | -| `verify_review` | Verify code review comment claims | **~200-500** | -| `verify_reality` | Verify a single AI statement against code evidence | **~200-500** | +### Index and Query -### Drift Detection +```bash +# Index a project +codescope cli index_project '{"project_path":"/path/to/your/project"}' -| Tool | Description | Token Cost | -|------|-------------|:----------:| -| `detect_drift` | Scan all declared capabilities & contracts for doc-vs-code drift | **~200** | -| `detect_documentation_drift` | Check README language claims vs actual code entities | **~150** | -| `detect_capability_drift` | Check declared capabilities have implementing entities | **~150** | -| `detect_architecture_drift` | Check call edges for layer violations (Repository→Controller) | **~150** | +# Quick overview +codescope cli project_overview '{}' -### Change Impact & Module +# Start MCP server (for AI clients) +codescope +``` -| Tool | Description | Token Cost | -|------|-------------|:----------:| -| `detect_changes` | Analyze impact of modified files: direct/indirect callers | **~100-500** | -| `explain_module` | Build module knowledge card: entities, capabilities, integrity score | **~50-200** | +### Large Projects -### Utilities +For projects with thousands of files, use the built-in parallel scheduler: -| Tool | Description | Token Cost | -|------|-------------|:----------:| -| `index_project` | Index entire project directory (parse → IR → graph) | **N/A** | -| `index_file` | Index a single source file | **N/A** | -| `force_index_files` | **Force-index** — bypass default skip rules (`test/`, `docs/`, `vendored/`, `node_modules/`, `.gitignore`, etc.) to index specific files/dirs. Use when the user says "go index xxx/yyy for me" | **N/A** | -| `count_tokens` | Estimate token count (DeepSeek formula) | **~10** | +```bash +codescope index-parallel /path/to/large/project +``` -#### `force_index_files` — user-directed incremental indexing +--- -When the default `FilterPolicy` skips entire directories (test fixtures, vendored deps, generated code, examples) but the user wants to pull those paths in, use `force_index_files`. It **bypasses** the default skip rules but still respects: +## 4. MCP Tools (37 Tools) -- File size limit (`CODESCOPE_MAX_FILE_SIZE`, default 5 MB) -- Language detectability (`detectLanguage` must return non-null) -- Optional language whitelist (`language_filter`) +### Indexing -**MCP call**: +| Tool | Description | Parameters | +|------|-------------|------------| +| `index_project` | Index a project directory: parse all source files, build IR, and construct the code graph. | `{"project_path": "string (required)", "language_filter": "string (optional)"}` | +| `index_file` | Index a single source file. | `{"file_path": "string (required)"}` | +| `force_index_files` | Force-index files/dirs bypassing default skip rules (test/, docs/, node_modules/, .gitignore, etc.). | `{"paths": ["string (required)"], "language_filter": "string (optional)"}` | -```json -{ - "paths": ["/abs/path/to/dir/or/file", "/another/path"], - "language_filter": "java,python" -} -``` +### Project Overview -- `paths`: array of absolute paths; directories are walked recursively -- `language_filter`: optional, comma-separated language whitelist; empty = all detectable languages +| Tool | Description | Parameters | +|------|-------------|------------| +| `project_overview` | **Primary** — comprehensive project overview: languages, modules, symbols, entry points, analysis progress. | `{}` | +| `get_graph_stats` | Quick statistics: nodes, edges, files. | `{}` | +| `get_module_tree` | Hierarchical module/directory tree. | `{}` | +| `get_entry_points` | Find entry points (main/init/setup/run/handler). | `{}` | +| `get_routes` | Get registered HTTP routes (Gin/Echo/Chi/net/http). | `{}` | +| `get_type_info` | Query type definitions (struct/enum/trait) with reference counts. | `{"type_name": "string (optional)"}` | -**CLI call**: +### Symbol Lookup -```bash -codescope force-index [--lang java,python] [--db /path/to/codescope.db] /path/to/xxx /path/to/yyy -``` +| Tool | Description | Parameters | +|------|-------------|------------| +| `find_symbol` | **Recommended** — find symbol by exact name (kind, file, line/col). | `{"symbol_name": "string (required)"}` | +| `find_references` | Find all locations referencing a symbol. | `{"symbol_name": "string (required)", "file_filter": "string (optional)"}` | +| `explain_symbol` | Get comprehensive symbol info: definition, callers, callees, dependencies. | `{"symbol_name": "string (required)"}` | +| `find_definition` | [DEPRECATED — use find_symbol] | `{"symbol_name": "string (required)"}` | -Returns the `engine_index_files` JSON (`files_indexed`/`nodes`/`edges`/`errors`) plus `skipped_files`/`skipped_dirs`/`paths_requested` stats. +### Call Graph +| Tool | Description | Parameters | +|------|-------------|------------| +| `find_callers` | Find who calls a function. | `{"symbol_name": "string (required)", "file_filter": "string (optional)"}` | +| `find_callees` | Find what a function calls. | `{"symbol_name": "string (required)", "file_filter": "string (optional)"}` | +| `codescope_trace` | Interactive recursive call exploration (depth + direction) or shortest path. | `{"function_name": "string", "depth": "integer (default 1, max 5)", "direction": "callers|callees|both", "from": "string", "to": "string"}` | +| `trace_flow` | Recursive execution flow tracing (caller→callee chain). | `{"function_name": "string (required)", "depth": "integer (default 3, max 10)"}` | +| `shortest_path` | Shortest call path between two functions (BFS). | `{"from": "string", "to": "string", "from_id": "integer", "to_id": "integer"}` | +| `connected_components` | Connected components in the call graph. | `{}` | -### Why Java is the (only) exception — a rant +### Graph Query -CodeScope skips `test/`, `tests/`, `docs/`, `examples/`, `samples/`, `bench/`, `vendor/`, ... at **any depth** in the path. This is the correct behavior for every sane project layout: Cargo workspaces nest `crates//tests/`, Lerna monorepos nest `packages//test/`, Gradle multi-module builds nest `subprojects//src/test/`, and users expect all of those to be skipped — they're not the analysis target, and indexing them inflates node counts 3-5x with noise. +| Tool | Description | Parameters | +|------|-------------|------------| +| `graph_query` | Cypher-like DSL query: `MATCH (Function:main)-[Calls]->(Method)`. | `{"dsl": "string (required)"}` | +| `get_graph` | Retrieve the complete code graph in paginated pages. | `{"node_offset": "integer", "node_limit": "integer (max 50000)", "edge_offset": "integer", "edge_limit": "integer (max 200000)", "node_types": "string", "edge_types": "string"}` | +| `get_subgraph` | Fetch a local region centered on a node (1-hop). | `{"node_id": "integer (required)", "radius": "integer", "node_types": "string", "edge_types": "string"}` | +| `get_neighbors` | Fetch direct neighbors (callers + callees) of a graph node. | `{"node_id": "integer (required)", "edge_type": "integer (default -1)", "radius": "integer"}` | -Then there's **Java**. Java, in its infinite wisdom, decided that `org/springframework/samples/petclinic` is a legitimate package name — `samples` is a package component, NOT a docs folder. `src/test/java/...` is the standard Maven test source root, but `src/main/java/.../test/...` can also be a legit package. `examples`, `integration`, `locale` — all fair game as Java package identifiers. Java conflated filesystem layout vocabulary with package naming, and now every tool in the ecosystem has to tiptoe around it. +### Search -This is an **anti-human engineering design**. It forces every static analysis tool to either (a) skip test/ at any depth and break Java, or (b) gate test/ to top-only and leak deep-nested test dirs on every other language's monorepos. The noise on non-Java projects is enormous — rustc alone has `tools/rust-analyzer/crates/*/src/*/tests/` nested 7 deep, all of which were leaking through a depth-3 top-only gate. +| Tool | Description | Parameters | +|------|-------------|------------| +| `search` | **Recommended** — unified search (auto-selects FTS5 or semantic). | `{"query": "string (required)", "limit": "integer (default 20, max 100)"}` | +| `search_code` | [DEPRECATED — use search] | `{"query": "string (required)", "limit": "integer"}` | -CodeScope picks option (c): **Java projects get a carve-out, everyone else gets the correct behavior.** +### Verification -When the indexer detects a `.java` file, it flips `FilterPolicy` into Java mode: `test/`/`docs/`/`samples/`/... collisions are gated to **top-only (depth ≤ 3)** via `java_protected_skip_dirs_`, so `org/.../samples/petclinic` (depth 5+) is NOT skipped but `/test/`, `/src/test/java/`, `/packages//tests/` still are. Every other language (Rust, Go, Python, JS/TS, C/C++, Kotlin, Ruby, Scala, ...) keeps these names skipped at any depth — the way they should be. +| Tool | Description | Parameters | +|------|-------------|------------| +| `verify_integrity` | Check README-promised features actually exist in code. | `{}` | +| `verify_claim` | Verify a single claim (capability_exists / contract_holds / architecture_follows). | `{"claim": "string (required)"}` | +| `verify_summary` | Parse natural-language summary and verify each claim. | `{"text": "string (required)"}` | +| `verify_review` | Verify code review comment claims. | `{"text": "string (required)"}` | +| `verify_reality` | Verify a single AI statement against code evidence. | `{"text": "string (required)"}` | -**If you're indexing a Java project**: nothing to do. The indexer auto-detects `.java` files and flips `FilterPolicy` into Java mode, which routes `test/`/`docs/`/`samples/`/... through `java_protected_skip_dirs_` at top-only (depth ≤ 3) — nested packages like `org/.../samples/petclinic` (depth 5+) are kept, `/test/`/`src/test/java/`/`packages//tests/` are still skipped. This is the only working override mechanism. +### Drift Detection -```bash -# Java project — auto-detection handles it, no env var needed: -codescope index -``` +| Tool | Description | Parameters | +|------|-------------|------------| +| `detect_drift` | Scan all declared capabilities & contracts for doc-vs-code drift. | `{}` | +| `detect_documentation_drift` | Check README language claims vs actual code entities. | `{}` | +| `detect_capability_drift` | Check declared capabilities have implementing entities. | `{}` | +| `detect_architecture_drift` | Check call edges for layer violations (Repository→Controller). | `{}` | -```bash -# If you actually want to skip NESTED test/docs on a Java project -# (i.e. turn OFF the carve-out and apply any-depth skip everywhere), -# CODESCOPE_EXCLUDE_PATHS only ADDS exclude patterns on top of the -# built-in list — it cannot un-skip nested packages on its own. The -# clean path is a project-local .codescopeignore pattern matching the -# specific nested dirs you want dropped. -``` +### Change Impact & Module -The trade-off is documented here in the open. Java's package naming collision is a design flaw in the language, not in CodeScope, and we refuse to let it degrade the experience for the other 99% of projects. +| Tool | Description | Parameters | +|------|-------------|------------| +| `detect_changes` | Analyze impact of modified files: direct/indirect callers. | `{"modified_files": "string (required)"}` | +| `explain_module` | Build module knowledge card: entities, capabilities, integrity score. | `{"module_name": "string (required)"}` | +### Utilities + +| Tool | Description | Parameters | +|------|-------------|------------| +| `detect_ffi_boundaries` | Detect FFI boundaries (extern/C, JNI, WASM, C ABI). | `{}` | +| `count_tokens` | Estimate token count (DeepSeek formula). | `{"text": "string (required)"}` | ### Quick Decision Guide ``` -New project → project_overview (~71 tok) -Module structure → get_module_tree (~4 tok) -Entry points → get_entry_points (~5 tok) -Search code → search (~300-1000 tok) -Call chain → find_callers / find_callees (~10-50 tok) -Deep dive symbol → explain_symbol (~50-200 tok) -HTTP routes → get_routes (~50 tok) -Type info → get_type_info (~50 tok) -Verify claim → verify_claim (~100 tok) -Detect drift → detect_documentation_drift (~150 tok) -Change impact → detect_changes (~100-500 tok) +New project → project_overview +Module structure → get_module_tree +Entry points → get_entry_points +Search code → search +Call chain → find_callers / find_callees +Deep dive symbol → explain_symbol +HTTP routes → get_routes +Type info → get_type_info +Verify claim → verify_claim +Detect drift → detect_documentation_drift +Change impact → detect_changes ``` -## Usage Skill - -### Basic Workflow +--- -``` -1. index_project("/path/to/project") ← 1-30s, index the project -2. project_overview ← 1-2ms, understand project structure -3. find_definition("malloc") ← 10μs, locate a symbol -4. trace_flow("main", "malloc") ← BFS, get execution path -5. verify_claim("login", "supports", "JWT") ← verify a claim -6. verify_summary("已完成登录模块") ← verify AI summary -``` +## 5. Knowledge Graph -### Real-World Execution Path Example +CodeScope builds a **module-level knowledge graph** as a side product of the verification pipeline. It learns structured metadata about how the project is organized, what's important, what's redundant, and what it promises: -```bash -# After scan + enhance of Linux kernel: -codescope_trace(from="copy_process", to="sched_fork") -# → {"path": [ -# {"name":"copy_process","file":"kernel/fork.c","line":1994}, -# {"name":"sched_fork", "file":"kernel/sched/core.c","line":4803} -# ]} - -# Deeper trace: -codescope_trace(from="copy_process", to="dup_mm") -# → {"path": [ -# {"name":"copy_process","file":"kernel/fork.c","line":1994}, -# {"name":"copy_mm", "file":"kernel/fork.c","line":1568}, -# {"name":"dup_mm", "file":"kernel/fork.c","line":1527} -# ]} -``` +| Layer | Table | What it tells you | +|-------|-------|-------------------| +| **Call graph** | `relation`, `architecture_edge` | Cross-module call dependencies; drives `detect_architecture_drift` | +| **Module health** | `module_summary` | Per-module `incoming_count` / `outgoing_count` / `dead_entities` / `utilization` / `role` | +| **Module dependency** | `architecture_edge`, `module_edge` | "Change module A → these modules depend on it" | +| **Documented capability** | `capability` + `document` | README-extracted capabilities; drives `detect_capability_drift` / `verify_claim` | -## Performance Benchmarks +All knowledge-layer tables are directly queryable via `get_knowledge_graph`: -### Real-world index benchmarks (v0.2.1, Apple M3 Max) +```jsonc +get_knowledge_graph {"table":"architecture_edge","limit":5} +// → {"table":"architecture_edge","rows":[...],"total":3351} -| Project | Files | Nodes | Index time | Peak RSS | -|---------|------:|------:|-----------:|---------:| -| **memscope-rs** (Rust) | 215 | 4,344 | ~2 s | ~200 MB | -| **CodeScope** (self, C++/Rust) | 168 | 1,001 | 1.3 s | ~150 MB | -| **ARES_POLIS** | 105 | 1,531 | ~2 s | ~180 MB | -| **rustc** (Rust compiler, monorepo) | 6,029 | 81,033 | 81 s | 5.9 GB | +get_knowledge_graph {"table":"capability","limit":10} +// → {"table":"capability","rows":[...],"total":3} +``` -**What this says:** +Supported tables: `entity`, `relation`, `architecture_edge`, `module_edge`, `capability`, `document`, `module_summary`. -- **Small-medium projects (<300 files):** sub-second to ~2 s index, ~200 MB RSS — excellent ergonomics, daily-driver speed. -- **Very large monorepos (tens of thousands of files):** usable but real resources — rustc took 81 s and 5.9 GB peak. Acceptable for a one-shot index, but plan memory. -- **Query latency (stdio MCP mode, includes process start):** ~60 ms per call; persistent server mode is lower. +--- -### Full Parse & Index (tree-sitter + Graph Builder + Linker) +## 6. Benchmark -| Project | Files | Nodes | Functions | CALLS | ★Cross-File | Time | -|---------|:----:|:-----:|:--------:|:-----:|:----------:|:----:| -| **CodeScope** (self) | 47 | 12K | 3.8K | 23K | 13 | 3s | -| **goagent** (Go) | 2,651 | 155K | 49K | 56K | 49K | 30s | -| **Linux kernel** (full) | **64,694** | **12M** | **3.8M** | **3.7M** | **1.5M** | **3min 07s** | +All benchmarks measured on **Apple M3 Max (36 GB RAM)**. Other hardware will produce different results — expect slower performance on less capable machines. -### Pipeline Architecture (v0.2) +### Index Time -``` -Source Files - │ - ▼ Phase 1: Collect -┌──────────────┐ -│ Translator │ Phase 2: Parallel translate -│ (no resolver)│ Pure: Source → IR, 14 workers -└──────┬───────┘ - │ IR Units - ▼ -┌──────────────┐ Phase 3: Link (serial PassManager) -│ Linker │ -│ ├─ BuildSymbolIndex (scan IR, ~ms) -│ ├─ ResolveCallPass (cross-file CALLS) -│ └─ EmitGraphPass (GraphBuilder → SQLite) -└──────────────┘ -``` +| Project | Files | Nodes | Index Time | Peak RSS | +|---------|------:|------:|-----------:|---------:| +| **CodeScope** (self, C++/Rust) | 168 | 1,001 | **1.3 s** | ~150 MB | +| **memscope-rs** (Rust) | 215 | 4,344 | **~2 s** | ~200 MB | +| **ARES_POLIS** | 105 | 1,531 | **~2 s** | ~180 MB | +| **goagent** (Go) | 2,651 | 155K | **30 s** | — | +| **rustc** (Rust compiler, monorepo) | 6,029 | 81,033 | **81 s** | 5.9 GB | +| **Linux kernel** (full) | 64,694 | 12M | **3 min 07 s** | — | -### Indexing Throughput +### Micro Benchmarks | Metric | Value | |--------|-------| -| **Linux kernel**: 64,694 files | **3 min 07 sec** (~350 files/sec) | -| Functions indexed | **3,840,680** | -| CALLS edges | **3,727,864** | -| Cross-file CALLS (★) | **1,502,432 (40%)** | -| DB size | ~1.2 GB | -| Workers | 14 × 8MB stack | +| Engine init | **14.6 ms** | +| Index throughput | **1,533 KB/s** | +| Symbol definition query | **0.01–0.03 ms** | +| Callers/callees query | **0.01–0.02 ms** | +| 9 queries (total) | **0.17 ms** | +| Query latency (stdio MCP, includes process start) | **~60 ms** | ### Cross-File Resolution -The Linker's `ResolveCallPass` resolves function calls across file boundaries using a global symbol index built from all TranslationUnits. Candidate ranking prefers `.c`/`.cpp` definitions over `.h` prototypes. - | Project | Cross-File CALLS | % of total CALLS | |---------|:---------------:|:----------------:| | CodeScope (C++) | 23 | 0.1% | @@ -558,196 +373,75 @@ The Linker's `ResolveCallPass` resolves function calls across file boundaries us |--------|:----:|:---------:|:-------:| | **CodeScope** (self) | **32 ms** | cpp, rust, c | 2,902 | | **goagent** (Go) | **493 ms** | go, c, cpp, python | 5,172 | -| **Linux kernel/** (core) | **360 ms** | c | 40,335 | - -### C Declaration Detection Accuracy - -| Language | Precision | Recall | Notes | -| ------------------ | --------- | ------ | --------------------------------- | -| **Go** | ~97% | ~96% | `func` pattern is highly specific | -| **Python** | ~98% | ~95% | `def`/`class` nearly zero FP | -| **C/C++ (strict)** | ~85% | ~90% | Requires type keyword in return | -| **C/C++ (old)** | ~65% | ~95% | Permissive, high FP | -| **Rust** | ~90% | ~90% | `fn` is precise | - -### Supported Languages (8) - -| Language | Parser | IR Translator | Verified | -| ---------- | ------ | ------------- | -------- | -| Python | ✅ | ✅ | ✅ | -| Go | ✅ | ✅ | ✅ | -| C | ✅ | ✅ | ✅ | -| C++ | ✅ | ✅ | ✅ | -| Rust | ✅ | ✅ | ✅ | -| JavaScript | ✅ | ✅ | ✅ | -| TypeScript | ✅ | ✅ | ✅ | -| Java | ✅ | ✅ | ✅ | - -## Token Savings +| **Linux kernel** (core) | **360 ms** | c | 40,335 | -Using code graphs instead of raw source files saves **~98.9% tokens** on average across 5 common query scenarios: - -| Scenario | Graph (tokens) | Raw (tokens) | Savings | -| ------------------------ | -------------- | ------------ | --------- | -| Find function definition | ~21 | ~2,265 | **99.1%** | -| Trace callers | ~18 | ~2,000 | **99.1%** | -| Architecture overview | ~32 | ~1,875 | **98.3%** | -| Function analysis | ~43 | ~4,733 | **99.1%** | -| Symbol search | ~23 | ~958 | **97.6%** | +### Token Savings -## Real-World Case Study: Linux Kernel Scheduler +Using code graphs instead of raw source files saves **~98.9% tokens** on average: -Using CodeScope's Fast Scan on the Linux kernel v6.13 — **45 ms** to analyze the scheduler (36 files, 4,913 symbols). Enhanced in **27s** with **45,573 call_edges**. +| Scenario | Raw Source | CodeScope | Savings | +|----------|:----------:|:---------:|:-------:| +| Find function definition | ~2,265 tokens | ~21 tokens | **99.1%** | +| Trace function callers | ~2,000 tokens | ~18 tokens | **99.1%** | +| Project architecture overview | ~1,875 tokens | ~32 tokens | **98.3%** | +| USB subsystem overview | ~24,000 tokens | ~250 tokens | **99.0%** | +| Scheduler analysis | ~15,000 tokens | ~180 tokens | **98.8%** | -### Execution Path Tracing in Action +--- -``` -codescope_trace("copy_process","sched_fork") -→ copy_process(kernel/fork.c:1994) - ↓ sched_fork(kernel/sched/core.c:4803) - -codescope_trace("copy_process","dup_mm") -→ copy_process(kernel/fork.c:1994) - ↓ copy_mm(kernel/fork.c:1568) - ↓ dup_mm(kernel/fork.c:1527) -``` +## 7. Skills (Shell Wrappers) -### Scheduler Code Layout +The `skills/` directory provides shell scripts that wrap common CodeScope queries so you don't need to remember the JSON schema: -``` -kernel/sched/ -├── core.c — __schedule(), schedule() -├── fair.c — CFS Completely Fair Scheduler -├── rt.c — Real-time scheduler -├── deadline.c — Deadline scheduler -├── idle.c — Idle task -├── sched.h — Data structures -└── ext/ — Extensible scheduler API -``` +```bash +cd CodeScope -### Parent-Child Resource Handling → `kernel/fork.c` +# Index a project +./skills/index.sh ~/path/to/project -| Line | Function | Purpose | -| ------ | --------------------- | ------------------------------------------------------ | -| **914** | `dup_task_struct()` | Copy parent's task_struct | -| **1994** | `copy_process()` | **Main entry** — creates new process | -| **2115** | `p = dup_task_struct(current, node)` | Copy kernel stack + thread_info + task_struct | -| **2259** | `sched_fork(clone_flags, p)` | Init child scheduling state, set non-runnable | +# One-shot full analysis report +./skills/analyze.sh ~/path/to/project -**Core mechanism: Copy-On-Write (COW)** — `copy_mm()` shares physical pages between parent and child as read-only. +# Graph statistics +./skills/stats.sh -### Preemption Prevention +# Module tree +./skills/modules.sh -| Location | Mechanism | Description | -| ------------------------------- | ---------------------- | -------------------------------------------------- | -| `include/linux/preempt.h:92` | `preempt_count()` | Per-task counter; >0 disables kernel preemption | -| `kernel/sched/core.c:7061` | `__schedule()` | Main scheduler; only switches when preempt_count==0 | -| `kernel/sched/core.c:7316` | `schedule()` | Voluntary yield | +# Trace call path A → B +./skills/trace.sh func_a func_b -## Configuration +# Top 20 hotspots +./skills/hotspots.sh 20 -### Build from source +# Browse architecture dependencies +./skills/knowledge.sh architecture_edge 20 +``` -```bash -# Build everything — tree-sitter, SQLite, sqlite-vec, grammars -# are all auto-downloaded and compiled into the binary (zero deps) -make build +Each script calls `codescope cli ''` internally. See `skills/skills.md` for the full reference. -# Start MCP server -cargo run --bin codescope -``` +--- -### Environment Variables +## 8. Environment Variables | Variable | Default | Description | |----------|---------|-------------| | `CODESCOPE_DB_PATH` | `.codescope/codescope.db` | SQLite database path | -| `CODESCOPE_LSP` | (unset) | LSP server command for type enhancement (e.g. `pylsp`) | | `CODESCOPE_INDEX_MODE` | `standard` | Index mode: `fast` / `standard` / `strict` | -| `CODESCOPE_EXCLUDE_PATHS` | (unset) | Comma-separated glob patterns to exclude (e.g. `test/*,docs/*`) | -| `CODESCOPE_MMAP_SIZE` | `268435456` (256 MB) | SQLite `mmap_size` pragma value in bytes | -| `CODESCOPE_WORKERS` | `4` | Total parse-worker cores for `codescope index-parallel` (overridable via `--workers N`) | -| `CODESCOPE_MAX_FILE_SIZE` | (unset) | Max source file size to index in bytes; larger files are skipped | +| `CODESCOPE_EXCLUDE_PATHS` | (unset) | Comma-separated glob patterns to exclude | +| `CODESCOPE_WORKERS` | `4` | Total parse-worker cores for `index-parallel` | | `CODESCOPE_WORKER_TIMEOUT` | `300` | Worker subprocess timeout in seconds | -| `CODESCOPE_VERBOSE` | `0` | Set to `1` to enable verbose logging | -| `CODESCOPE_EXPLAIN` | (unset) | Set to `1` to print SQL `EXPLAIN QUERY PLAN` for graph queries | -| `CODESCOPE_DYNAMIC_SCHED` | `auto` | Dynamic CPU scheduling for `index-parallel`: `1` force on, `0` force off, unset = auto (modules > 4 AND total_files > 10000) | -| `CODESCOPE_MEM_LIMIT_MB` | `4096` | Dynamic-scheduler memory ceiling (MB). New worker spawns pause when total child RSS exceeds this; existing workers keep running | -| `CODESCOPE_AGGRESSIVE` | `0` | Set to `1` for 50ms dynamic-scheduler poll interval (default 100ms) | -| `CODESCOPE_SCHED_SHM` | (auto) | Path to the dynamic-scheduler shared-memory file (auto-generated per scheduler PID; propagated to worker subprocesses) | - -> **Note:** `GRAMMARS_DIR` is no longer needed — all tree-sitter grammars are compiled into the binary via CMake FetchContent. - -### Prerequisites - -- Rust 2024 Edition + 1.85+ (`cargo`) -- CMake 3.30+, C++23 compiler (Clang 17+) - -## Data Directory `.codescope/` - -CodeScope automatically creates a `.codescope/` directory in the project root on first run. -All persistent data is stored here — no manual setup needed. - -``` -.codescope/ -├── codescope.db ← SQLite database (WAL mode): all facts, indexes, graphs -├── skills.md ← Quick start guide and command reference -└── *.log ← Analysis run logs with timing + CPU + memory data -``` - -The database contains 40 tables, grouped by purpose (see `engine/src/store/store_schema.cpp`): +| `CODESCOPE_MAX_FILE_SIZE` | (unset) | Max source file size to index in bytes | +| `CODESCOPE_MMAP_SIZE` | 256 MB | SQLite `mmap_size` pragma value | +| `CODESCOPE_MEM_LIMIT_MB` | `4096` | Dynamic-scheduler memory ceiling | +| `CODESCOPE_DYNAMIC_SCHED` | `auto` | Dynamic CPU scheduling: `1` on, `0` off, unset = auto | +| `CODESCOPE_VERBOSE` | `0` | Set to `1` for verbose logging | +| `CODESCOPE_LSP` | (unset) | LSP server command for type enhancement | -| Category | Tables | Purpose | -|----------|--------|---------| -| Core / Project | `projects`, `project_readiness`, `files`, `modules`, `entry_points`, `index_tasks`, `file_scan_state` | Project metadata, file tracking, index phase progress | -| Graph | `graph_nodes`, `graph_edges`, `entity`, `relation`, `semantic_records`, `adjacency`, `adjacency_rev`, `module_edge`, `module_summary` | Code graph nodes/edges, CSR BLOB adjacency, cross-module edges | -| Search | `code_fts` (FTS5), `name_trgm` (FTS5 trigram), `fts_node_map`, `node_vectors` | Full-text + trigram + n-gram vector search | -| Facts / Parser | `reference`, `scope`, `import`, `type_info`, `type_ref`, `route` | Call facts, scope tree, imports, type definitions, HTTP routes | -| Knowledge + Evidence | `capability`, `contract`, `claim`, `evidence`, `evidence_fact`, `finding`, `document` | Verification pipeline: claims, evidence chains, findings | -| Model State | `workflow`, `workflow_step`, `architecture_edge`, `capability_state`, `workflow_state`, `architecture_state` | Workflows, architecture layers, state caches | -| LadybugDB Sync | `lbug_sync_state` | Incremental sync progress to LadybugDB graph store | - -> **Tip**: The database is portable — copy `.codescope/` along with your project to reuse analysis results on another machine. - -## Performance - -Benchmarks measured on **Apple M3 Max (36 GB RAM)**. - -### Micro Benchmarks (test_bench) - -| Metric | Value | -|--------|-------| -| Engine init | **14.6 ms** | -| Index throughput | **1,533 KB/s** | -| Symbol definition query | **0.01–0.03 ms** | -| Callers/callees query | **0.01–0.02 ms** | -| 9 queries (total) | **0.17 ms** | - -### Known Bottleneck (Knowledge Graph Queries) - -The current MCP knowledge graph service has a **~300k-500k node threshold** for fuzzy text search (`CONTAINS`, BM25 full-text, regex name matching) — queries on projects beyond this threshold may **time out at 30 seconds**. - -| Project Scale | Example | Exact-match queries | Fuzzy searches | -|--------------|---------|:------------------:|:--------------:| -| Small-Medium (<50K nodes) | goagent (23K) | ✅ <10ms | ✅ Fast | -| Large (50K-300K nodes) | zigcode (327K) | ✅ <10ms | ⚠️ May time out | -| Very Large (>500K nodes) | JDK (1.36M) | ✅ Exact match works | ❌ Time out | - -> **Root cause**: Full-node-set text scans (`CONTAINS`, `name_pattern` regex) iterate over millions of nodes, exceeding the 30s timeout limit. Index-assisted exact path matching (`ENDS WITH`) works fine. -> -> **Planned fix**: Add a custom exclusion paths parameter to skip `test/`, `doc/`, and other large non-core directories during indexing, keeping effective node count under 300K. - -### Token Savings +--- -| Scenario | Raw Source | CodeScope | Savings | -|----------|:----------:|:---------:|:-------:| -| Find function definition | ~2,265 tokens | ~21 tokens | **99.1%** | -| Trace function callers | ~2,000 tokens | ~18 tokens | **99.1%** | -| Project architecture | ~1,875 tokens | ~32 tokens | **98.3%** | -| USB subsystem overview | ~24,000 tokens | ~250 tokens | **99.0%** | -| Scheduler analysis | ~15,000 tokens | ~180 tokens | **98.8%** | -| **Average** | **~7,416 tokens** | **~81 tokens** | **98.9%** | +## 9. License -## License +Apache 2.0 — see [LICENSE](LICENSE). -Apache 2.0 +**CodeScope v0.2.1** — Built with Rust 2024 + C++23 + tree-sitter + SQLite. \ No newline at end of file diff --git a/README.zh.md b/README.zh.md new file mode 100644 index 0000000..20cead6 --- /dev/null +++ b/README.zh.md @@ -0,0 +1,447 @@ +# CodeScope — 项目真相引擎 + +**CodeScope 不理解代码，它验证代码。** + +它将源代码转化为可验证的事实、可理解的模型和可检查的证据 — 让 AI 能够根据现实验证断言，而非凭空编造。 + +**版本**: v0.2.1 | **许可证**: Apache 2.0 + +--- + +## 1. 什么是 CodeScope？ + +CodeScope 是一个 **项目真相引擎（Project Truth Engine）**，回答一个问题： + +> **"代码真的做了你说的那些事吗？"** + +不是"这段代码什么意思"，而是"代码到底有没有实现你声称的功能？" + +它把源码索引为结构化的代码图（调用图 + 引用图 + 模块知识），然后暴露 **37 个 MCP 工具**，让 AI 代理可以定位符号、追踪调用路径、验证断言、检测文档漂移、分析架构 — 相比读取原始源文件，平均节省 **~98.9% 的 token 消耗**。 + +### 支持的语言（8 种） + +| 语言 | 解析器 | IR 转换 | 已验证 | +|------|--------|---------|--------| +| Python | ✅ | ✅ | ✅ | +| Go | ✅ | ✅ | ✅ | +| C | ✅ | ✅ | ✅ | +| C++ | ✅ | ✅ | ✅ | +| Rust | ✅ | ✅ | ✅ | +| JavaScript | ✅ | ✅ | ✅ | +| TypeScript | ✅ | ✅ | ✅ | +| Java | ✅ | ✅ | ✅ | + +### 技术栈 + +| 层 | 技术 | +|----|------| +| 解析器 | tree-sitter（统一 AST IR，8 种语言） | +| 索引引擎 | C++23（Clang 17+），SQLite（WAL 模式，FTS5） | +| 服务端 | Rust 2024 Edition，MCP 协议（JSON-RPC 2.0，stdio 传输） | +| 图存储 | SQLite（主存储）+ 可选 LadybugDB（Cypher 查询） | +| 调度器 | 内置多进程并行索引器（chunk 级 work-stealing） | +| 构建 | CMake 3.30+（C++），Cargo（Rust） | + +--- + +## 2. 架构 + +```mermaid +graph TB + subgraph "AI 客户端" + Client["Claude Desktop / Cursor / 任意 MCP 客户端"] + end + + subgraph "Rust MCP 服务端" + MCP["MCP 协议 (JSON-RPC 2.0)
37 个工具 / stdio 传输"] + DISPATCH["工具分发
project_id 自动恢复"] + end + + subgraph "C++ 核心引擎" + PARSER["解析器
tree-sitter → 统一 IR
8 种语言"] + FACTS["事实仓库
entity / reference / scope / import"] + RESOLVER["解析器管线
约束链"] + MODEL["模型引擎
Workflow / Capability
Architecture / Contract"] + INSPECTOR["检查器
DeadCodeInspector / verify_integrity"] + end + + subgraph "SQLite (WAL 模式)" + F_STORE["事实存储
entity / reference / scope / import"] + S_STORE["语义存储
resolved_reference / relation"] + M_STORE["模型存储
workflow / capability
architecture / contract"] + E_STORE["证据存储
claim / evidence / finding"] + end + + Client -->|"MCP stdio"| MCP + MCP --> DISPATCH + DISPATCH -->|"FFI"| PARSER + DISPATCH -->|"FFI"| FACTS + DISPATCH -->|"FFI"| RESOLVER + DISPATCH -->|"FFI"| MODEL + DISPATCH -->|"FFI"| INSPECTOR + + PARSER -->|"写入"| F_STORE + F_STORE -->|"读取"| RESOLVER + RESOLVER -->|"写入"| S_STORE + S_STORE -->|"读取"| MODEL + MODEL -->|"写入"| M_STORE + M_STORE -->|"读取"| INSPECTOR + INSPECTOR -->|"写入"| E_STORE +``` + +### 管线 + +``` +源代码 + | + v +Parser ------------ entity / reference / scope / import + | + v +Resolver ---------- resolved_reference / relation + | + v +Model Engine ------ workflow / capability / architecture / contract + | + v +Inspector --------- evidence / finding +``` + +### 查询流程 + +```mermaid +flowchart LR + Q["MCP 客户端
工具调用"] --> Q1["服务端接收
project_id 自动恢复"] + Q1 --> Q2{"工具类型?"} + Q2 -->|"index_project"| Q3["启动 worker 子进程
→ 内存隔离
→ 完成后退出"] + Q2 -->|"查询工具"| Q4["C++ FFI → SQLite 查询
graph_nodes, graph_edges
search_index, ..."] + Q2 -->|"get_communities"| Q5["加载完整图
标签传播
→ JSON (max_communities 限制)"] + Q2 -->|"get_hotspots"| Q6["SQL: COUNT(ge.id) JOIN
graph_edges edge_type=1
ORDER BY caller_count"] + Q4 --> R["结果 JSON
返回 MCP 客户端"] + Q5 --> R + Q6 --> R +``` + +### 双阶段索引 + +```mermaid +flowchart LR + subgraph A["阶段 A: 快速扫描 (毫秒级)"] + S1["scan_project"] + S2["total_symbols"] + S3["module_tree"] + S4["entry_points"] + end + + subgraph B["阶段 B: 后台增强 (异步, 秒级)"] + E1["enhance_project"] + E2["完整 tree-sitter"] + E3["调用图"] + E4["复杂度指标"] + E5["embeddings + FTS"] + end + + A -->|"触发"| B +``` + +--- + +## 3. 快速开始 + +### 前置依赖 + +| 平台 | 依赖 | +|------|------| +| **macOS** | Xcode CLT, cmake, Rust (1.85+) | +| **Linux** | build-essential, cmake, Rust (1.85+) | + +### 安装预编译二进制 + +```bash +curl -fsSL https://raw.githubusercontent.com/Timwood0x10/CodeScope/main/install.sh | bash +export PATH="$PATH:$HOME/.codescope/bin" +``` + +### 源码编译 + +```bash +git clone https://github.com/Timwood0x10/CodeScope.git +cd CodeScope + +# macOS: +brew install llvm@21 cmake pkg-config +cargo build --release + +# Linux (Ubuntu): +sudo apt-get install -y build-essential cmake llvm-dev libclang-dev +cargo build --release +``` + +### 索引与查询 + +```bash +# 索引一个项目 +codescope cli index_project '{"project_path":"/path/to/your/project"}' + +# 快速概览 +codescope cli project_overview '{}' + +# 启动 MCP 服务端（供 AI 客户端使用） +codescope +``` + +### 大型项目 + +对于包含数千个文件的项目，使用内置的并行调度器： + +```bash +codescope index-parallel /path/to/large/project +``` + +--- + +## 4. MCP 工具（37 个工具） + +### 索引 + +| 工具 | 用途 | 参数 | +|------|------|------| +| `index_project` | 索引整个项目目录：解析所有源文件，构建 IR，构建代码图。 | `{"project_path": "string (必填)", "language_filter": "string (可选)"}` | +| `index_file` | 索引单个源文件。 | `{"file_path": "string (必填)"}` | +| `force_index_files` | 强制索引文件/目录，跳过默认排除规则（test/, docs/, node_modules/, .gitignore 等）。 | `{"paths": ["string (必填)"], "language_filter": "string (可选)"}` | + +### 项目概览 + +| 工具 | 用途 | 参数 | +|------|------|------| +| `project_overview` | **主入口** — 全面的项目概览：语言、模块、符号、入口点、分析进度。 | `{}` | +| `get_graph_stats` | 快速统计：节点数、边数、文件数。 | `{}` | +| `get_module_tree` | 分层模块/目录树。 | `{}` | +| `get_entry_points` | 查找入口点（main/init/setup/run/handler）。 | `{}` | +| `get_routes` | 获取注册的 HTTP 路由（Gin/Echo/Chi/net/http）。 | `{}` | +| `get_type_info` | 查询类型定义（struct/enum/trait）及引用计数。 | `{"type_name": "string (可选)"}` | + +### 符号查找 + +| 工具 | 用途 | 参数 | +|------|------|------| +| `find_symbol` | **推荐** — 按精确名称查找符号（类型、文件、行/列）。 | `{"symbol_name": "string (必填)"}` | +| `find_references` | 查找所有引用某个符号的位置。 | `{"symbol_name": "string (必填)", "file_filter": "string (可选)"}` | +| `explain_symbol` | 获取符号的完整信息：定义、调用者、被调用者、依赖关系。 | `{"symbol_name": "string (必填)"}` | +| `find_definition` | [已废弃 — 使用 find_symbol] | `{"symbol_name": "string (必填)"}` | + +### 调用图 + +| 工具 | 用途 | 参数 | +|------|------|------| +| `find_callers` | 查找谁调用了某个函数。 | `{"symbol_name": "string (必填)", "file_filter": "string (可选)"}` | +| `find_callees` | 查找某个函数调用了什么。 | `{"symbol_name": "string (必填)", "file_filter": "string (可选)"}` | +| `codescope_trace` | 交互式递归调用探索（深度 + 方向）或最短路径。 | `{"function_name": "string", "depth": "integer (默认 1, 最大 5)", "direction": "callers|callees|both", "from": "string", "to": "string"}` | +| `trace_flow` | 递归执行流追踪（caller→callee 链）。 | `{"function_name": "string (必填)", "depth": "integer (默认 3, 最大 10)"}` | +| `shortest_path` | 两个函数之间的最短调用路径（BFS）。 | `{"from": "string", "to": "string", "from_id": "integer", "to_id": "integer"}` | +| `connected_components` | 调用图中的连通分量。 | `{}` | + +### 图查询 + +| 工具 | 用途 | 参数 | +|------|------|------| +| `graph_query` | Cypher 风格 DSL 查询：`MATCH (Function:main)-[Calls]->(Method)`。 | `{"dsl": "string (必填)"}` | +| `get_graph` | 分页获取完整代码图。 | `{"node_offset": "integer", "node_limit": "integer (最大 50000)", "edge_offset": "integer", "edge_limit": "integer (最大 200000)", "node_types": "string", "edge_types": "string"}` | +| `get_subgraph` | 获取以某节点为中心的局部区域（1 跳）。 | `{"node_id": "integer (必填)", "radius": "integer", "node_types": "string", "edge_types": "string"}` | +| `get_neighbors` | 获取图节点的直接邻居（调用者 + 被调用者）。 | `{"node_id": "integer (必填)", "edge_type": "integer (默认 -1)", "radius": "integer"}` | + +### 搜索 + +| 工具 | 用途 | 参数 | +|------|------|------| +| `search` | **推荐** — 统一搜索（自动选择 FTS5 或语义搜索）。 | `{"query": "string (必填)", "limit": "integer (默认 20, 最大 100)"}` | +| `search_code` | [已废弃 — 使用 search] | `{"query": "string (必填)", "limit": "integer"}` | + +### 验证 + +| 工具 | 用途 | 参数 | +|------|------|------| +| `verify_integrity` | 检查 README 中承诺的功能是否确实存在于代码中。 | `{}` | +| `verify_claim` | 验证单个断言（capability_exists / contract_holds / architecture_follows）。 | `{"claim": "string (必填)"}` | +| `verify_summary` | 解析自然语言摘要并验证每个断言。 | `{"text": "string (必填)"}` | +| `verify_review` | 验证代码审查评论中的断言。 | `{"text": "string (必填)"}` | +| `verify_reality` | 验证 AI 对项目状态的单个陈述。 | `{"text": "string (必填)"}` | + +### 漂移检测 + +| 工具 | 用途 | 参数 | +|------|------|------| +| `detect_drift` | 扫描所有声明的能力和合约，检测文档与代码之间的漂移。 | `{}` | +| `detect_documentation_drift` | 检查 README 中的语言支持声明与实际代码实体是否一致。 | `{}` | +| `detect_capability_drift` | 检查声明的能力是否有对应的实现实体。 | `{}` | +| `detect_architecture_drift` | 检查调用边是否存在架构层违规（Repository→Controller）。 | `{}` | + +### 变更影响与模块 + +| 工具 | 用途 | 参数 | +|------|------|------| +| `detect_changes` | 分析修改文件的调用图影响范围。 | `{"modified_files": "string (必填)"}` | +| `explain_module` | 构建模块知识卡片：实体、能力、完整性评分。 | `{"module_name": "string (必填)"}` | + +### 实用工具 + +| 工具 | 用途 | 参数 | +|------|------|------| +| `detect_ffi_boundaries` | 检测 FFI 边界（extern/C、JNI、WASM、C ABI）。 | `{}` | +| `count_tokens` | 估算文本的 token 数（DeepSeek 公式）。 | `{"text": "string (必填)"}` | + +### 快速决策指南 + +``` +新项目 → project_overview +模块结构 → get_module_tree +入口点 → get_entry_points +搜索代码 → search +调用链 → find_callers / find_callees +符号深度分析 → explain_symbol +HTTP 路由 → get_routes +类型信息 → get_type_info +验证断言 → verify_claim +检测漂移 → detect_documentation_drift +变更影响分析 → detect_changes +``` + +--- + +## 5. 知识图谱 + +CodeScope 在验证管线的副产品中构建了一个**模块级知识图谱**。它学习的是结构化的元数据 — 项目如何组织、什么重要、什么冗余、承诺了什么： + +| 层级 | 表 | 用途 | +|------|-----|------| +| **调用图** | `relation`, `architecture_edge` | 跨模块调用依赖；驱动 `detect_architecture_drift` | +| **模块健康度** | `module_summary` | 每个模块的 `incoming_count` / `outgoing_count` / `dead_entities` / `utilization` / `role` | +| **模块依赖** | `architecture_edge`, `module_edge` | "修改模块 A → 这些模块依赖于它" | +| **文档化能力** | `capability` + `document` | 从 README 中提取的能力声明；驱动 `detect_capability_drift` / `verify_claim` | + +所有知识层表均可通过 `get_knowledge_graph` 直接查询： + +```jsonc +get_knowledge_graph {"table":"architecture_edge","limit":5} +// → {"table":"architecture_edge","rows":[...],"total":3351} + +get_knowledge_graph {"table":"capability","limit":10} +// → {"table":"capability","rows":[...],"total":3} +``` + +支持的表：`entity`, `relation`, `architecture_edge`, `module_edge`, `capability`, `document`, `module_summary`。 + +--- + +## 6. 性能基准 + +所有基准测试在 **Apple M3 Max（36 GB RAM）** 上测得。其他硬件会产生不同的结果 — 性能较低的机器上预期会慢一些。 + +### 索引时间 + +| 项目 | 文件数 | 节点数 | 索引时间 | 峰值内存 | +|------|------:|------:|---------:|---------:| +| **CodeScope**（自身，C++/Rust） | 168 | 1,001 | **1.3 秒** | ~150 MB | +| **memscope-rs**（Rust） | 215 | 4,344 | **~2 秒** | ~200 MB | +| **ARES_POLIS** | 105 | 1,531 | **~2 秒** | ~180 MB | +| **goagent**（Go） | 2,651 | 155K | **30 秒** | — | +| **rustc**（Rust 编译器，monorepo） | 6,029 | 81,033 | **81 秒** | 5.9 GB | +| **Linux 内核**（完整） | 64,694 | 12M | **3 分 07 秒** | — | + +### 微基准 + +| 指标 | 值 | +|------|----| +| 引擎初始化 | **14.6 ms** | +| 索引吞吐量 | **1,533 KB/s** | +| 符号定义查询 | **0.01–0.03 ms** | +| 调用者/被调用者查询 | **0.01–0.02 ms** | +| 9 次查询（总计） | **0.17 ms** | +| 查询延迟（stdio MCP，含进程启动） | **~60 ms** | + +### 跨文件解析 + +| 项目 | 跨文件 CALLS | 占 CALLS 总数百分比 | +|------|:-----------:|:------------------:| +| CodeScope（C++） | 23 | 0.1% | +| goagent（Go） | 49,258 | 86% | +| Linux 内核（C） | 1,502,432 | 40% | + +### 快速扫描（轻量，毫秒级） + +| 项目 | 时间 | 语言 | 符号数 | +|------|:----:|:----:|:------:| +| **CodeScope**（自身） | **32 ms** | cpp, rust, c | 2,902 | +| **goagent**（Go） | **493 ms** | go, c, cpp, python | 5,172 | +| **Linux 内核**（核心） | **360 ms** | c | 40,335 | + +### Token 节省 + +使用代码图代替原始源文件，平均节省 **~98.9% 的 token**： + +| 场景 | 原始代码 | CodeScope | 节省 | +|------|:--------:|:---------:|:----:| +| 查找函数定义 | ~2,265 tokens | ~21 tokens | **99.1%** | +| 追踪函数调用者 | ~2,000 tokens | ~18 tokens | **99.1%** | +| 项目架构概览 | ~1,875 tokens | ~32 tokens | **98.3%** | +| USB 子系统概览 | ~24,000 tokens | ~250 tokens | **99.0%** | +| 调度器分析 | ~15,000 tokens | ~180 tokens | **98.8%** | + +--- + +## 7. Skills（Shell 封装脚本） + +`skills/` 目录提供了一系列 Shell 脚本，封装了常用的 CodeScope 查询，无需记忆 JSON 参数格式： + +```bash +cd CodeScope + +# 索引项目 +./skills/index.sh ~/path/to/project + +# 一次性完整分析报告 +./skills/analyze.sh ~/path/to/project + +# 图统计 +./skills/stats.sh + +# 模块树 +./skills/modules.sh + +# 追踪调用路径 A → B +./skills/trace.sh func_a func_b + +# 前 20 个热点函数 +./skills/hotspots.sh 20 + +# 浏览架构依赖 +./skills/knowledge.sh architecture_edge 20 +``` + +每个脚本内部调用 `codescope cli ''`。详见 `skills/skills.md`。 + +--- + +## 8. 环境变量 + +| 变量 | 默认值 | 说明 | +|------|--------|------| +| `CODESCOPE_DB_PATH` | `.codescope/codescope.db` | SQLite 数据库路径 | +| `CODESCOPE_INDEX_MODE` | `standard` | 索引模式：`fast` / `standard` / `strict` | +| `CODESCOPE_EXCLUDE_PATHS` | （未设置） | 逗号分隔的 glob 排除模式 | +| `CODESCOPE_WORKERS` | `4` | `index-parallel` 的解析 worker 核心数 | +| `CODESCOPE_WORKER_TIMEOUT` | `300` | Worker 子进程超时时间（秒） | +| `CODESCOPE_MAX_FILE_SIZE` | （未设置） | 允许索引的最大源文件大小（字节） | +| `CODESCOPE_MMAP_SIZE` | 256 MB | SQLite `mmap_size` 参数值 | +| `CODESCOPE_MEM_LIMIT_MB` | `4096` | 动态调度器内存上限（MB） | +| `CODESCOPE_DYNAMIC_SCHED` | `auto` | 动态 CPU 调度：`1` 开启，`0` 关闭，未设置 = 自动 | +| `CODESCOPE_VERBOSE` | `0` | 设为 `1` 开启详细日志 | +| `CODESCOPE_LSP` | （未设置） | 类型增强的 LSP 服务端命令 | + +--- + +## 9. 许可证 + +Apache 2.0 — 详见 [LICENSE](LICENSE)。 + +**CodeScope v0.2.1** — 使用 Rust 2024 + C++23 + tree-sitter + SQLite 构建。 \ No newline at end of file diff --git a/README_CN.md b/README_CN.md deleted file mode 100644 index 81b754d..0000000 --- a/README_CN.md +++ /dev/null @@ -1,437 +0,0 @@ -# CodeScope — Project Truth Engine - -**CodeScope 不解释代码，只验证代码。** - -它把源码变成可验证的事实、可理解的模型、可检查的证据，让 AI 基于项目真相回答问题，而不是幻觉。 - ---- - -### CodeScope 不是什么 - -CodeScope **不是**代码解释器、语义分析器，也不是替代阅读代码的工具。它不理解 `Arc` 是什么意思，不理解 `Rc` 为什么不是线程安全的，也不理解 JWT 中间件怎么工作。**那是 AI 的事。** - -### CodeScope 是什么 - -CodeScope 是一个 **Project Truth Engine**，回答一个问题： - -> **"项目现在到底是什么状态？"** - -不是"这段代码是什么意思"，而是"代码真的像你说的那样工作吗？" - -### 真实数据 - -| 指标 | 值 | -|------|-----| -| 支持语言 | 8 种（Rust, Go, C/C++, Python, Java, JS/TS） | -| 索引速度 | 1-10s（100+ 文件） | -| 查询延迟 | 0.3-1.5 ms | -| Token 节省 | **98.9%**（26 万行 → 4 万 token） | -| MCP 工具 | 37 个（Locate / Understand / Verify / Index） | -| 架构 | Facts → Resolution → Models → Verification | - -### 能验证什么 - -| AI 说 | CodeScope 验证 | 数据来源 | -|-------|---------------|---------| -| "登录模块支持 JWT" | JWT 库存在吗？login 调了 jwt 吗？有测试吗？ | `entity` + `relation` + `import` | -| "这个模块已完成" | 所有功能有调用者吗？覆盖率够吗？ | `relation` + `DeadCodeInspector` | -| "PR 修复了内存泄漏" | 有对应的 free 吗？测试覆盖了 error path 吗？ | `relation` + 测试文件检查 | -| "架构是 Controller→Service→Repository" | 代码真的遵守这个分层吗？ | `architecture_edge` | -| "模块支持 6 种语言" | adapter 真的存在吗？ | `entity` + `import` | - -### 知识图谱（副产品） - -CodeScope 把知识图谱作为验证管线的副产品来构建。它学到的不是「代码文本」，而是*结构化的「这个项目怎么组织、哪重要、哪冗余、承诺了啥」*——四层叠加： - -| 层 | 表 | 告诉你什么 | -|----|----|-----------| -| **调用图**（谁调谁） | `relation`（type=1）、`architecture_edge` | 跨模块调用依赖；驱动 `detect_architecture_drift` | -| **模块健康度**（谁重要/谁有死代码） | `module_summary` | 每模块 `incoming_count`/`outgoing_count`/`dead_entities`/`utilization`/`confidence`；驱动 `explain_module` | -| **模块依赖**（改了影响谁） | `architecture_edge`（边权=调用次数）、`module_edge` | 「改模块 A → 这些模块依赖它」 | -| **文档能力**（声称能干嘛） | `capability` + `document` | 从 README 提取的能力声明；驱动 `detect_capability_drift` / `verify_claim` | - -知识图谱不是产品，是支撑验证的基础设施。 - -#### `module_summary.role`：多信号融合分类器（v0.2.1） - -`role` 列由**多信号融合 CASE** 分类器自动填充（`engine/src/model/state_builder.cpp`），不是单源启发式。它把调用图度数与调用图给不了的两类信号融合： - -| 信号 | 来源 | 增量信息 | -|------|------|----------| -| `pub_count` | `entity.visibility=1`（各语言 Visitor 域 pub/public/export） | 区分「对外接口层」vs「内部实现层」——调用图度数推不出 | -| `entry_reachable` | `graph_nodes.is_entry_point`（main/init/setup/run/handler） | 该模块是否含入口层 | - -规则按**优先级**匹配（命中即停）： - -| 优先级 | Role | 规则（多信号） | -|--------|------|----------------| -| 1 | `test` | 模块名含 `test`/`tests`/`_test`/`mod tests` | -| 2 | `api` | `pub_count > 0 AND incoming ≥ 2×outgoing AND incoming ≥ 3 AND utilization ≥ 0.1` | -| 3 | `entry` | `entry_reachable > 0` | -| 4 | `core` | `incoming ≥ 10 AND outgoing ≤ incoming×1.0 AND utilization ≥ 0.05 AND pub_count > 0` | -| 5 | `utility` | `outgoing ≤ 5 AND pub_count > 0 AND utilization ≥ 0.05` | -| 6 | `business` | `pub_count > 0 AND incoming ≥ 10`（实现层——被多模块调且自己也调多；outgoing 偏高不命中 core/api） | -| 7 | `dead` | `incoming=0 AND outgoing=0`，或 `dead_entities = total` | -| 8 | `infra` | 真兜底——没命中任何语义规则 | - -请把 `role` 当融合后的线索，不当判决。阈值是 `state_builder.h` 里的 `constexpr`——若你项目的 role 分布看着不对（如 `infra > 30%` 说明阈值过严），凭 `bun` 调参。完整设计 + v0.2.1 阈值重调记录见 `docs/dev_plans/role_classifier_plan.md`。 - -#### 知识图谱里落了什么（memscope-rs，215 文件） - -| 表 | 行数 | 含义 | -|----|-----:|------| -| `entity` | 4,310 | 细粒度代码实体（函数、类型、变量） | -| `relation` | 726 | 实体间关系（type 3 = 包含/定义） | -| `architecture_edge` | 3,351 | 模块/目录级架构依赖边（边权 = 调用次数） | -| `module_edge` | 11 | 跨模块依赖边 | -| `capability` | 3 | 从 README 提取的能力声明 | -| `document` | 1 | README 文档记录 | -| `module_summary` | 42 | 每模块知识卡片 | - -能了解到啥： - -- **架构依赖**——`architecture_edge` 告诉你哪些模块依赖哪些（如 `analysis/heap_scanner → unsafe_inference`，边权=调用次数）。驱动 `detect_architecture_drift`。 -- **能力声明**——`capability` + `document` 把 README 的「能干什么」结构化。驱动 `detect_capability_drift` / `verify_claim(capability_exists)`。 -- **模块摘要**——42 个模块的知识卡片，由 `explain_module` 浮出。 - -#### 直接查询入口（v0.2.1） - -过去知识图谱是**隐式**的——`graph_query` 走的是*调用*图（`graph_nodes` / `graph_edges`），不是知识层表（`entity` / `relation` / `architecture_edge`）。你只能通过 `explain_module`、`detect_capability_drift`、`get_module_tree` 间接受益。 - -v0.2.1 新增 `engine_get_knowledge_graph(project_id, table, limit)` + MCP 工具 `get_knowledge_graph`，现在可**直接浏览**任意知识层表： - -```jsonc -get_knowledge_graph {"table":"architecture_edge","limit":5} -// → {"table":"architecture_edge","rows":[{"id":1,"layer_lower":"analysis/heap_scanner","layer_upper":"unsafe_inference","entity_id":42}],"total":3351,"truncated":false} - -get_knowledge_graph {"table":"capability","limit":10} -// → {"table":"capability","rows":[{"id":1,"name":"borrow_analysis","summary":"scope-aware borrow checking"}],"total":3,"truncated":false} -``` - -支持表：`entity`、`relation`、`architecture_edge`、`module_edge`、`capability`、`document`、`module_summary`。块级 FFI——单次调用返回整个结果集，绝不一条边一次调用。 - -**CodeScope 不解释代码，只验证代码。** 解释是 AI 的事，验证才是 CodeScope 的事。 - -## 快速开始 - -### 60 秒完成第一个索引 - -```bash -# 1. 一键构建（自动检测系统、安装依赖、编译） -bash <(curl -fsSL https://raw.githubusercontent.com/Timwood0x10/CodeScope/main/bootstrap.sh) - -# 2. 索引项目 -codescope cli index_project '{"project_path":"/path/to/your/project"}' - -# 3. 查询 -codescope cli get_graph_stats '{}' -# → {"total_nodes":12345,"total_edges":6789,"total_files":99} - -# 4. 启动 MCP 服务（供 AI 客户端使用） -codescope -``` - -📖 详细指南见 [QUICK_START.md](QUICK_START.md) - -### 前置依赖 - -| 平台 | 依赖 | 一键命令 | -|------|------|---------| -| **macOS** | Xcode CLT, cmake, Rust | `bash bootstrap.sh` | -| **Linux** | build-essential, cmake, Rust | `bash bootstrap.sh` | - -> 预编译二进制可在 [Releases 页面](https://github.com/Timwood0x10/CodeScope/releases) 下载。 - -**Linux / macOS 一键安装**： - -```bash -curl -fsSL https://raw.githubusercontent.com/Timwood0x10/CodeScope/master/install.sh | bash -``` - -安装后 `~/.codescope/bin/` 下包含： - -| 文件 | 用途 | -|------|------| -| `codescope` | 主二进制（CLI + MCP server）—— 单进程 `index` + 内置多进程 `index-parallel` 调度器（适合大项目） | - -加入 PATH 后即可使用： - -```bash -export PATH="$PATH:$HOME/.codescope/bin" - -# 中小项目直接索引 -codescope cli index_project '{"project_path":"/path/to/project"}' - -# 大型项目（数千~数万文件）用内置调度器加速 -codescope index-parallel /path/to/large/project -``` - -### 手动构建 - -```bash -# macOS: -brew install llvm@21 cmake pkg-config sqlite3 ladybug -cargo build --release - -# Linux (Ubuntu): -sudo apt-get install -y build-essential cmake llvm-dev libclang-dev libsqlite3-dev -curl -fsSL https://install.ladybugdb.com | sh -cargo build --release -``` - -### 环境变量 - -| 变量 | 默认值 | 说明 | -|------|--------|------| -| `CODESCOPE_DB_PATH` | `.codescope/codescope.db` | SQLite 数据库路径 | -| `GRAMMARS_DIR` | `grammars/` | Grammar .so 文件目录 | -| `CODESCOPE_LSP` | 未设置 | LSP 服务器命令，用于类型增强（如 `pylsp`） | -| `CODESCOPE_INDEX_MODE` | `standard` | 索引模式：`fast` / `standard` / `strict` | -| `CODESCOPE_EXCLUDE_PATHS` | 未设置 | 逗号分隔的排除 glob 模式（如 `test/*,docs/*`） | -| `CODESCOPE_MMAP_SIZE` | `268435456` (256 MB) | SQLite `mmap_size` pragma 值（字节） | -| `CODESCOPE_WORKERS` | `4` | `codescope index-parallel` 的总解析 worker 核心数（可用 `--workers N` 覆盖） | -| `CODESCOPE_MAX_FILE_SIZE` | 未设置 | 索引的最大源文件大小（字节）；超过则跳过 | -| `CODESCOPE_WORKER_TIMEOUT` | `300` | Worker 子进程超时（秒） | -| `CODESCOPE_VERBOSE` | `0` | 设为 `1` 启用详细日志 | -| `CODESCOPE_EXPLAIN` | 未设置 | 设为 `1` 打印图查询的 SQL `EXPLAIN QUERY PLAN` | - -## 数据目录 `.codescope/` - -CodeScope 在首次运行时自动在项目根目录创建 `.codescope/` 目录。 -所有持久化数据都存储在此——无需手动配置。 - -``` -.codescope/ -├── codescope.db ← SQLite 数据库（WAL 模式）：所有事实、索引、图 -├── skills.md ← 快速入门指南和命令参考 -└── *.log ← 分析运行日志（含时间、CPU、内存数据） -``` - -数据库包含 40 张表，按用途分组（见 `engine/src/store/store_schema.cpp`）： - -| 类别 | 表 | 用途 | -|------|------|------| -| 核心 / 项目 | `projects`, `project_readiness`, `files`, `modules`, `entry_points`, `index_tasks`, `file_scan_state` | 项目元数据、文件跟踪、索引阶段进度 | -| 图 | `graph_nodes`, `graph_edges`, `entity`, `relation`, `semantic_records`, `adjacency`, `adjacency_rev`, `module_edge`, `module_summary` | 代码图节点/边、CSR BLOB 邻接表、跨模块边 | -| 搜索 | `code_fts` (FTS5), `name_trgm` (FTS5 trigram), `fts_node_map`, `node_vectors` | 全文 + trigram + n-gram 向量搜索 | -| 事实 / 解析 | `reference`, `scope`, `import`, `type_info`, `type_ref`, `route` | 调用事实、作用域树、导入、类型定义、HTTP 路由 | -| 知识 + 证据 | `capability`, `contract`, `claim`, `evidence`, `evidence_fact`, `finding`, `document` | 验证管线：声明、证据链、发现 | -| 模型状态 | `workflow`, `workflow_step`, `architecture_edge`, `capability_state`, `workflow_state`, `architecture_state` | 工作流、架构层、状态缓存 | -| LadybugDB 同步 | `lbug_sync_state` | 到 LadybugDB 图存储的增量同步进度 | - -> **提示**：数据库是可移植的——将 `.codescope/` 随项目一起复制，即可在其他机器上复用分析结果。 - -## 性能 - -### 真实项目索引实测（v0.2.1，Apple M3 Max） - -| 项目 | 文件 | 节点 | 索引耗时 | 峰值 RSS | -|------|----:|----:|--------:|---------:| -| **memscope-rs**（Rust） | 215 | 4,344 | ~2 s | ~200 MB | -| **CodeScope**（自身，C++/Rust） | 168 | 1,001 | 1.3 s | ~150 MB | -| **ARES_POLIS** | 105 | 1,531 | ~2 s | ~180 MB | -| **rustc**（Rust 编译器，单体库） | 6,029 | 81,033 | 81 s | 5.9 GB | - -**结论：** - -- **中小项目（<300 文件）：** 亚秒~2 秒索引，~200 MB RSS——体验极佳，日常开发速度。 -- **超大单体库（数万文件）：** 能用但需注意资源——rustc 耗 81 秒 + 5.9 GB 峰值内存。一次性索引可接受,需留意内存。 -- **查询延迟（stdio MCP 模式，含进程启动）：** 单次调用 ~60 ms；常驻 server 模式更低。 - -基准测试在 **Apple M3 Max（36 GB 内存）** 上执行。 - -### 微基准测试 (test_bench) - -| 指标 | 数值 | -|------|------| -| 引擎初始化 | **14.6 ms** | -| 索引吞吐量 | **1,533 KB/s** | -| 符号定义查询 | **0.01–0.03 ms** | -| 调用者/被调用者查询 | **0.01–0.02 ms** | -| 9 次查询（总计） | **0.17 ms** | - -### 已知瓶颈（知识图谱查询） - -当前 MCP 知识图谱服务在处理 **30-50 万节点以上** 的大型项目时，模糊搜索（`CONTAINS`、BM25 全文检索、正则匹配）会出现 **30 秒超时**。 - -| 项目 | 节点数 | 精确路径查询 | 模糊搜索 | -|------|--------|:----------:|:--------:| -| 中小型项目（<5 万节点） | eg. goagent(2.3万) | ✅ <10ms | ✅ 流畅 | -| 大型项目（5-30 万节点） | eg. zigcode(33万) | ✅ <10ms | ⚠️ 可能超时 | -| 超大型项目（>50 万节点） | eg. JDK(136万) | ✅ 精确匹配可用 | ❌ 超时 | - -> **原因**：服务端对全节点集的文本扫描（`CONTAINS`、`name_pattern` 正则）需要遍历百万级节点，超过了 30 秒超时限制。精确路径匹配（`ENDS WITH`）利用索引可正常工作。 -> -> **应对**：计划增加自定义排除路径参数，允许索引时主动跳过 `test/`、`doc/` 等大型非核心目录，将有效节点数控制在 30 万以内。 - -### Token 节省 - -| 场景 | 原始源码 | CodeScope | 节省 | -|------|:-------:|:---------:|:----:| -| 查找函数定义 | ~2,265 tokens | ~21 tokens | **99.1%** | -| 追踪函数调用者 | ~2,000 tokens | ~18 tokens | **99.1%** | -| 项目架构概览 | ~1,875 tokens | ~32 tokens | **98.3%** | -| USB 子系统概览 | ~24,000 tokens | ~250 tokens | **99.0%** | -| 调度器分析 | ~15,000 tokens | ~180 tokens | **98.8%** | -| **平均** | **~7,416 tokens** | **~81 tokens** | **98.9%** | - -## License - -Apache 2.0 - -*** - -## 工具使用指南 - -每个 MCP 工具有其适用的场景和副作用（主要是 Token 消耗）。以下指南帮助你在正确的场合选择合适的工具。 - -> **工具总数：37 个**。一些旧工具标记为 `[DEPRECATED]`，建议使用推荐替代。 - -### 核心查询类 - -| 工具 | 适用场景 | Token 消耗 | -|------|---------|:----------:| -| `project_overview` | **首选**——新接手项目的第一步总览（语言、模块、符号、入口点） | **~71** | -| `get_graph_stats` | 快速了解项目规模（文件数、节点数、边数） | **~18** | -| `get_module_tree` | 了解项目的目录/模块层级结构 | **~4** | -| `get_entry_points` | 找项目入口点（main/init/setup/run/handler） | **~5** | -| `get_routes` | 获取项目注册的 HTTP 路由（支持 Gin/Echo/Chi/net/http） | **~50** | -| `get_type_info` | 查询类型定义（struct/enum/trait）及其引用数 | **~50** | - -### 符号查询类 - -| 工具 | 适用场景 | Token 消耗 | 说明 | -|------|---------|:----------:|------| -| `find_symbol` | **推荐**——按精确名称查找符号（类型、文件、行列） | **~30** | | -| `find_definition` | `[DEPRECATED]` 查找符号定义位置 | **~20** | 改用 `find_symbol` | -| `find_references` | 搜索符号被哪些位置引用 | **~30** | | -| `explain_symbol` | 获取符号的完整信息：定义、调用者、被调用者、依赖 | **~50-200** | 一键全方位理解一个函数 | - -### 调用图查询类 - -| 工具 | 适用场景 | Token 消耗 | 副作用 | -|------|---------|:----------:|--------| -| `find_callers` | 调查函数被谁调用——定位 bug 影响范围 | **~10-50** | 依赖 CALLS 边构建 | -| `find_callees` | 调查函数调用了什么——理解函数行为 | **~10-50** | 同上 | -| `codescope_trace` | 交互式递归展开调用者/被调用者（方向+深度控制） | **~50-200** | 深度大时输出膨胀 | -| `trace_flow` | 从函数开始递归追踪执行流（caller→callee 链） | **~50-200** | 同上 | -| `shortest_path` | 两点之间的最短调用路径 | **~50-100** | BFS 近似结果 | -| `connected_components` | 调用图中的连通分量——找独立的功能模块 | **~50** | | - -### 搜索类 - -| 工具 | 适用场景 | Token 消耗 | -|------|---------|:----------:| -| `search` | **推荐**——统一代码搜索（FTS5 + 语义搜索自动选择） | **~300-1000** | -| `search_code` | `[DEPRECATED]` 旧版 FTS 搜索，推荐改用 `search` | **~300-1000** | - -### 验证类（Verification Layer） - -| 工具 | 适用场景 | Token 消耗 | -|------|---------|:----------:| -| `verify_integrity` | 检查 README 承诺的功能是否在代码中真实存在 | **~100** | -| `verify_claim` | 验证单条声明（capability_exists / contract_holds / architecture_follows） | **~100** | -| `verify_summary` | 解析自然语言摘要并逐条验证所有声明 | **~200-500** | -| `verify_review` | 验证 Code Review 评论中的声明 | **~200-500** | -| `verify_reality` | 验证 AI 对项目的单条陈述是否被代码证据支持 | **~200-500** | - -### 漂移检测类（Drift Detection） - -| 工具 | 适用场景 | Token 消耗 | -|------|---------|:----------:| -| `detect_drift` | 扫描所有声明的能力与契约，发现文档与代码之间的偏移 | **~200** | -| `detect_documentation_drift` | 检测 README 声明的语言支持与实际代码的差异 | **~150** | -| `detect_capability_drift` | 检测声明的能力是否有实际实现且有调用者 | **~150** | -| `detect_architecture_drift` | 检测调用边中的架构层违规（如 Repository 调 Controller） | **~150** | - -### 变更影响分析 - -| 工具 | 适用场景 | Token 消耗 | -|------|---------|:----------:| -| `detect_changes` | 修改代码后分析影响范围——返回直接/间接调用者 | **~100-500** | -| `explain_module` | 构建模块知识卡：实体、能力、契约、完整性评分 | **~50-200** | - -### 辅助工具 - -| 工具 | 适用场景 | Token 消耗 | -|------|---------|:----------:| -| `index_project` | 索引整个项目目录（解析+IR+图构建） | **N/A** | -| `index_file` | 索引单个源文件 | **N/A** | -| `force_index_files` | **强制索引**——绕过默认跳过规则（`test/`、`docs/`、`vendored/`、`node_modules/`、`.gitignore` 等）索引指定文件/目录。用户说"去吧 xxx/yyy 给我索引了吧"时使用 | **N/A** | -| `count_tokens` | 估算文本的 token 数（DeepSeek 公式） | **~10** | - -#### `force_index_files` — 用户自定义增量索引 - -当默认 `FilterPolicy` 把某些目录（测试夹具、vendored 依赖、生成代码、示例）整批跳过，而用户又想把这些路径拉进来索引时，用 `force_index_files`。它**绕过**默认 skip 规则，但仍然尊重： - -- 文件大小上限（`CODESCOPE_MAX_FILE_SIZE`，默认 5 MB） -- 语言可识别性（`detectLanguage` 必须返回非 null） -- 可选语言白名单（`language_filter`） - -**MCP 调用**： - -```json -{ - "paths": ["/abs/path/to/dir/or/file", "/another/path"], - "language_filter": "java,python" -} -``` - -- `paths`：绝对路径数组，目录会递归展开 -- `language_filter`：可选，逗号分隔语言白名单；空 = 全部可识别语言 - -**CLI 调用**： - -```bash -codescope force-index [--lang java,python] [--db /path/to/codescope.db] /path/to/xxx /path/to/yyy -``` - -返回 `engine_index_files` 的 JSON（`files_indexed`/`nodes`/`edges`/`errors`），并附 `skipped_files`/`skipped_dirs`/`paths_requested` 统计。 - - -### 为什么 Java 是（唯一的）例外 —— 一段吐槽 - -CodeScope 对 `test/`、`tests/`、`docs/`、`examples/`、`samples/`、`bench/`、`vendor/`…… 这类目录在路径**任意深度**都跳过。这对任何正常的项目布局都是正确行为：Cargo workspace 会嵌 `crates//tests/`，Lerna monorepo 会嵌 `packages//test/`，Gradle 多模块构建会嵌 `subprojects//src/test/`，用户都期望这些被跳过 —— 它们不是分析目标，索引它们只会让节点数膨胀 3-5x 的噪声。 - -然后**Java**来了。Java 凭它无尽的智慧，决定 `org/springframework/samples/petclinic` 是一个合法的包名 —— `samples` 在这里是一个包组件，**不是** docs 文件夹。`src/test/java/...` 是 Maven 的标准测试源根，但 `src/main/java/.../test/...` 也可以是一个合法包。`examples`、`integration`、`locale` —— 全都可以当 Java 包标识符。Java 把文件系统布局词汇跟包命名混为一谈，现在生态里每个工具都得绕着它走。 - -这是一种**反人类的工程设计**。它逼着每个静态分析工具要么 (a) 任意深度跳 test/ 然后坑了 Java，要么 (b) 把 test/ 限到 top-only 然后漏掉其他所有语言 monorepo 里深度嵌套的 test 目录。非 Java 项目上的噪声大得离谱 —— 光 rustc 一个项目就有 `tools/rust-analyzer/crates/*/src/*/tests/` 嵌到第 7 层，全从一个 depth-3 的 top-only 闸门里漏出去。 - -CodeScope 选了方案 (c)：**Java 项目给一个例外，其他所有项目都拿到正确行为。** - -当索引器检测到一个 `.java` 文件时，会把 `FilterPolicy` 切到 Java 模式：`test/`/`docs/`/`samples/`/... 的冲突项被限定到 **top-only（深度 ≤ 3）**，通过 `java_protected_skip_dirs_` 处理，所以 `org/.../samples/petclinic`（深度 5+）**不**跳，但 `/test/`、`/src/test/java/`、`/packages//tests/` 仍然跳。其他所有语言（Rust、Go、Python、JS/TS、C/C++、Kotlin、Ruby、Scala、...）都保持这些名字在任意深度被跳过 —— 本该如此。 - -**如果你在索引 Java 项目**：什么都不用做。索引器自动检测 `.java` 文件并把 `FilterPolicy` 切到 Java 模式，把 `test/`/`docs/`/`samples/`/... 通过 `java_protected_skip_dirs_` 限定到 top-only（深度 ≤ 3）—— 嵌套包如 `org/.../samples/petclinic`（深度 5+）保留，`/test/`/`src/test/java/`/`packages//tests/` 仍然跳。这是唯一能工作的 override 机制。 - -```bash -# Java 项目 —— 自动检测处理，不需要环境变量： -codescope index -``` - -```bash -# 如果你确实想跳掉 Java 项目里嵌套的 test/docs -# （即关掉这个例外，到处都任意深度跳）， -# CODESCOPE_EXCLUDE_PATHS 只能在 built-in 列表之上"加"exclude 模式 -# —— 它不能"un-skip"嵌套包。干净路径是项目级 .codescopeignore -# pattern 精确匹配你想丢的具体嵌套目录。 -``` - -这个 trade-off 在这里公开记录。Java 的包命名冲突是语言本身的设计缺陷，不是 CodeScope 的，我们拒绝让它拖累其他 99% 项目的体验。 - - -### 总结选择策略 - -``` -新接手项目 → project_overview (~71 tok) -查结构 → get_module_tree (~4 tok) -找入口点 → get_entry_points (~5 tok) -搜代码 → search (~300-1000 tok) -查调用链 → find_callers / find_callees (~10-50 tok) -全量理解函数 → explain_symbol (~50-200 tok) -查 HTTP 路由 → get_routes (~50 tok) -查类型定义 → get_type_info (~50 tok) -验证声明 → verify_claim (~100 tok) -查文档偏移 → detect_documentation_drift (~150 tok) -查变更影响 → detect_changes (~100-500 tok) -``` -