Skip to content

Add CodSpeed performance measurement setup - #6

Merged
Ercaner1988 merged 2 commits into
mainfrom
codspeed/setup-benchmarks
Sep 19, 2026
Merged

Ercaner1988 merged 2 commits into
mainfrom
codspeed/setup-benchmarks

Conversation

@codspeed

@codspeed codspeed Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR adds continuous performance measurement to claude-code-setup with CodSpeed, covering the parts of the CLI that do real work per invocation: the memory engine (embedding math, chunking, SQLite FTS5 / semantic / graph search), the security audit secret scanner, and the MCP JSON-RPC server dispatch.

35 benchmarks were built and executed locally in CPU simulation mode before opening this PR, and the same 35 are now reported by the CodSpeed workflow on this PR.

Changes

Library target (src/lib.rs, Cargo.toml)

The crate was binary-only, so nothing outside main.rs could be benchmarked. The modules are now published through a claude_code_setup library target and src/main.rs consumes that library instead of declaring the modules itself. Behaviour is unchanged; cargo test still passes (31 tests) and cargo clippy --all-targets -- -D warnings is clean. The existing Rust workflow passes on Linux, macOS and Windows with this change.

A few already-pure internal helpers were made pub so benchmarks can call them directly: memory_engine::{chunk_content, mean_pool_embeddings, escape_fts5_query, bytes_to_f32_vec, f32_vec_to_bytes} and mcp_server::{handle_request, tool_to_cli_args}.

Benchmarks (benches/, divan via codspeed-divan-compat)

  • memory_engine.rs — cosine similarity (single vector and a 64/512/4096-note linear scan, 384-dim as used by MultilingualE5Small), mean-pooling of chunk embeddings, embedding BLOB encode/decode (including a 1000-note index decode), content chunking, wikilink extraction and FTS5 query escaping, each over realistic markdown notes.
  • memory_search.rs — the SQLite-backed search paths against an in-memory database seeded with 500 synthetic notes, FTS5 rows, embedding blobs and graph edges: keyword search (same query as search_keyword_vec), semantic ranking (the measured part of search_semantic_vec, without the model call), and the 2-hop graph BFS behind get_related_notes.
  • security.rs — secret pattern scanning over clean and secret-bearing MCP configs of 8/64/256 entries, branch description sanitization and protected-branch checks.
  • mcp_protocol.rs — JSON-RPC request parsing and handle_request dispatch for initialize, tools/list (largest response, all tool schemas), resources/list and an unknown method, plus tool-to-CLI argument mapping.

Benchmark data is generated with a small deterministic xorshift PRNG, so no new dependencies were added beyond the CodSpeed divan compatibility layer, and results stay stable between runs.

CI (.github/workflows/codspeed.yml)

New CodSpeed workflow running on ubuntu-latest in simulation mode, on pushes to main, on pull requests and on workflow_dispatch (so CodSpeed can backtest to generate initial data). It authenticates through OIDC (id-token: write), builds with cargo codspeed build and runs with CodSpeedHQ/action@v5. cargo-codspeed is installed and cached by moonrepo/setup-rust. The existing Rust and Release workflows are untouched.

README

CodSpeed badge added to README.md and README.en.md.

Results from the CodSpeed run on this PR

A few representative numbers:

Benchmark Time
similarity::cosine_linear_scan[4096] 16.2 ms
text::wikilinks[512] 6.9 ms
semantic_ranking (500 notes) 4.3 ms
blob_codec::decode_index (1000 notes) 3 ms
keyword_search (FTS5) 2.4 ms
dispatch::tools_list 180.2 µs
graph_related_notes (2-hop BFS) 165.2 µs

Next steps

  • Merge this PR so CodSpeed records a baseline on main; after that every pull request gets a performance report.
  • Two observations already visible in the baseline that are worth a follow-up: find_secrets and extract_wikilinks recompile their regexes on every call, which dominates their cost (the secret scan takes about 1 ms even on a config with no secrets, almost independent of input size). Hoisting the regexes into lazily initialized statics would be a measurable win, and CodSpeed will quantify it.
  • search_semantic_vec scans every stored embedding linearly; the cosine_linear_scan benchmark tracks how that grows with note count (64 -> 4096 notes) if an ANN index is considered later.

- Expose a library target (src/lib.rs) so benchmarks and tests can reach
  the CLI internals; the binary now consumes the library
- Add divan benchmarks (via codspeed-divan-compat) covering the memory
  engine math and text helpers, SQLite FTS5/semantic/graph search paths,
  the security secret scanner and the MCP JSON-RPC dispatch
- Add a CodSpeed GitHub Actions workflow running in simulation mode
- Add the CodSpeed badge to README.md and README.en.md
@codspeed

codspeed Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor Author

Congrats! CodSpeed is installed 🎉

🆕 35 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

Detected benchmarks


ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.


Open in CodSpeed

@codspeed
codspeed Bot marked this pull request as ready for review September 18, 2026 05:36
@codspeed
codspeed Bot requested a review from Ercaner1988 September 18, 2026 05:36
@Ercaner1988
Ercaner1988 merged commit 973722c into main Sep 19, 2026
8 checks passed
@Ercaner1988
Ercaner1988 deleted the codspeed/setup-benchmarks branch September 19, 2026 05:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants