Add CodSpeed performance measurement setup - #6
Merged
Merged
Conversation
- Expose a library target (src/lib.rs) so benchmarks and tests can reach the CLI internals; the binary now consumes the library - Add divan benchmarks (via codspeed-divan-compat) covering the memory engine math and text helpers, SQLite FTS5/semantic/graph search paths, the security secret scanner and the MCP JSON-RPC dispatch - Add a CodSpeed GitHub Actions workflow running in simulation mode - Add the CodSpeed badge to README.md and README.en.md
Contributor
Author
Congrats! CodSpeed is installed 🎉
You will start to see performance impacts in the reports once the benchmarks are run from your default branch.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds continuous performance measurement to
claude-code-setupwith CodSpeed, covering the parts of the CLI that do real work per invocation: the memory engine (embedding math, chunking, SQLite FTS5 / semantic / graph search), the security audit secret scanner, and the MCP JSON-RPC server dispatch.35 benchmarks were built and executed locally in CPU simulation mode before opening this PR, and the same 35 are now reported by the CodSpeed workflow on this PR.
Changes
Library target (
src/lib.rs,Cargo.toml)The crate was binary-only, so nothing outside
main.rscould be benchmarked. The modules are now published through aclaude_code_setuplibrary target andsrc/main.rsconsumes that library instead of declaring the modules itself. Behaviour is unchanged;cargo teststill passes (31 tests) andcargo clippy --all-targets -- -D warningsis clean. The existingRustworkflow passes on Linux, macOS and Windows with this change.A few already-pure internal helpers were made
pubso benchmarks can call them directly:memory_engine::{chunk_content, mean_pool_embeddings, escape_fts5_query, bytes_to_f32_vec, f32_vec_to_bytes}andmcp_server::{handle_request, tool_to_cli_args}.Benchmarks (
benches/, divan viacodspeed-divan-compat)memory_engine.rs— cosine similarity (single vector and a 64/512/4096-note linear scan, 384-dim as used by MultilingualE5Small), mean-pooling of chunk embeddings, embedding BLOB encode/decode (including a 1000-note index decode), content chunking, wikilink extraction and FTS5 query escaping, each over realistic markdown notes.memory_search.rs— the SQLite-backed search paths against an in-memory database seeded with 500 synthetic notes, FTS5 rows, embedding blobs and graph edges: keyword search (same query assearch_keyword_vec), semantic ranking (the measured part ofsearch_semantic_vec, without the model call), and the 2-hop graph BFS behindget_related_notes.security.rs— secret pattern scanning over clean and secret-bearing MCP configs of 8/64/256 entries, branch description sanitization and protected-branch checks.mcp_protocol.rs— JSON-RPC request parsing andhandle_requestdispatch forinitialize,tools/list(largest response, all tool schemas),resources/listand an unknown method, plus tool-to-CLI argument mapping.Benchmark data is generated with a small deterministic xorshift PRNG, so no new dependencies were added beyond the CodSpeed divan compatibility layer, and results stay stable between runs.
CI (
.github/workflows/codspeed.yml)New
CodSpeedworkflow running onubuntu-latestinsimulationmode, on pushes tomain, on pull requests and onworkflow_dispatch(so CodSpeed can backtest to generate initial data). It authenticates through OIDC (id-token: write), builds withcargo codspeed buildand runs withCodSpeedHQ/action@v5.cargo-codspeedis installed and cached bymoonrepo/setup-rust. The existingRustandReleaseworkflows are untouched.README
CodSpeed badge added to
README.mdandREADME.en.md.Results from the CodSpeed run on this PR
A few representative numbers:
similarity::cosine_linear_scan[4096]text::wikilinks[512]semantic_ranking(500 notes)blob_codec::decode_index(1000 notes)keyword_search(FTS5)dispatch::tools_listgraph_related_notes(2-hop BFS)Next steps
main; after that every pull request gets a performance report.find_secretsandextract_wikilinksrecompile their regexes on every call, which dominates their cost (the secret scan takes about 1 ms even on a config with no secrets, almost independent of input size). Hoisting the regexes into lazily initialized statics would be a measurable win, and CodSpeed will quantify it.search_semantic_vecscans every stored embedding linearly; thecosine_linear_scanbenchmark tracks how that grows with note count (64 -> 4096 notes) if an ANN index is considered later.