feat(eval): isolated synthetic smoke fixture and concurrency harness - #113
Merged
Conversation
Add npm run eval:perf:concurrency: a worker-pool harness that measures throughput and tail latency when multiple independent shlog processes hit the same read-only SQLite index concurrently. - eval/concurrency-bench-core.ts: pure arg parsing, shape construction, per-level aggregation and markdown report builder (unit-tested). - eval/concurrency-bench.ts: async runner over a shared job queue. - eval/PERF_BENCH.md: document the harness and a 2026-08-17 local baseline (Apple M4, 6.2k sessions / 318k messages / 420MB index).
…rness defaults - New eval/perf-fixture.ts: deterministic CJK-heavy fixture generator (~60% CJK, ~25% Latin, ~15% paths/commands, mulberry32 seeded) - eval/perf-bench.ts: default flips from real dev data + strict sync to synthetic fixture + --best-effort sync; real data now opt-in - eval/concurrency-bench: integrates --fixture-mb with auto-sync - eval/PERF_BENCH.md: rewrite Safety section for fixture default; update concurrency section for --fixture-mb - docs/ROADMAP.md: move perf baseline from P0 「仍需收口」to done - CONTEXT.md: initial glossary (7 terms: perf baseline, regression gate, e2e perf, concurrency perf, component perf, fixture, dogfood) Verified: npm run check 300/300, cargo test 12/12, smoke with Rust release binary + 4MB fixture: all 7 shapes pass including CJK (豆包输入法 find p50=11.7ms, 部署 health check p50=13.3ms).
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
… TS eval harness)
…ation Default eval:perf and eval:perf:concurrency stay on isolated synthetic smoke. Passing only one path no longer falls back to ~/.codex/sessions or the developer state DB. Docs now call default runs smoke and real-index runs private calibration — not a git-tracked baseline.
catoncat
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
Sherlog Rust rewrite 之后，性能 harness 默认会扫开发者本机
~/.codex/sessions并可能改写真实 index。本 PR 把默认负载改成隔离的合成烟雾，并把真实库测量收成显式本机校准。变更
eval/perf-fixture.ts：确定性 Codex 短句 fixture（消息抽签 ~60/25/15 CJK/Latin/路径）。这是烟雾负载，不是真实体积模型。eval/perf-bench.ts/eval/concurrency-bench.ts：无--root/--db= 合成烟雾；同时传两个路径 = 本机校准；只传一个或与--fixture-mb混用 fail-closed。eval/perf-data-source.ts：共享负载判定（纯函数，单测覆盖）。PERF_BENCH.md/CONTEXT.md/ROADMAP.md）：默认叫合成烟雾；420MB 表标成本机校准观察。没有进 git 的回归基线 JSON，也没有 release perf gate。不在本 PR