Conversation
Maps deepseek-ai/deepseek-harness ('Everything is a Plugin') onto caro's
AgentLoop, CaroML validator/journal layer, and evaluation harness. Tiered
proposals P1-P8 route toward one outcome: rendering the eval verdict the
candidate-ranking default flip has waited on since v1.4.0. Argues one dsh
principle must invert for caro: the safety floor stays privileged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JfmCW2L5d5qkNJ18mDfz6a
…ce bench - Consume HarnessConfig::max_concurrency (was declared but never read) via a tokio Semaphore in run_all_tests; permits span generate+evaluate so a local LLM backend never sees the whole dataset at once. Regression guard: evaluation::harness::tests::test_max_concurrency_bounds_in_flight_generations. - evaluation.yml: reduce the matrix to static_matcher, the only leg that evaluates anything (the runner registers only StaticMatcher and --backend does not filter; embedded-* legs exited 2 behind '|| true'). Raise the regression floor 31.0 -> 75.0 (measured 78.2% on the 101-case dataset). - Declare benches/performance.rs [[bench]] harness=false; cargo auto-discovery ran it under libtest where criterion_main! never executed. Verified via cargo bench --bench performance -- --test. - CLAUDE.md: replace nonexistent 'cargo run --bin caro-eval' with 'cargo test --test evaluation'; version banner 1.4.0 -> 1.5.0. Validation: 81 lib tests pass; eval run 79/101 (78.2%): correctness 26/26, safety 25/25, posix 24/25, multi_backend 4/25 (needs >=2 registered backends); clippy -D warnings clean; fmt clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JfmCW2L5d5qkNJ18mDfz6a
The validated run (79/101, 78.2%) replaces the stale 31% framing: 21 of 22 failures are MultiBackend cases starved of a second backend, and the CI floor was 47 points below measured reality. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JfmCW2L5d5qkNJ18mDfz6a
wildcard
added
documentation
Claude
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
5 Skipped Deployments
|
|
Awesome contribution! 🌟 |
github-actions
Bot
added
rust
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7b58a84613
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 5 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Website Claims Verification ReportRun ID: 34725236965 Platform Resultsclaims-report-macos-latest{
"suite": "website-claims",
"platform": "macos-latest",
"timestamp": "2026-09-12T23:25:14Z",
"passed": 84,
"skipped": 0,
"warnings": 1,
"run_id": "34725236965",
"sha": "7acb4c009e887560a0ab0317da4151771e97974b"
}claims-report-ubuntu-latest{
"suite": "website-claims",
"platform": "ubuntu-latest",
"timestamp": "2026-09-12T23:26:12Z",
"passed": 84,
"skipped": 0,
"warnings": 1,
"run_id": "34725236965",
"sha": "7acb4c009e887560a0ab0317da4151771e97974b"
}DocumentationNext Steps
|
wildcard
added
the
path:refuse-list
|
Agent: Claude Code ( Dispatcher classification: Which glob fired: Rationale (from The other touched paths do not independently refuse: Relaxation path: any of the following unblocks it —
No rebase bead was filed for this PR — refuse-list PRs wait on policy or human review, not on coder-loop work. If the workflow-file constraint should soften for eval-harness CI specifically, reply here and it will be picked up in the next grooming cycle. |
…easured 78.2 Review feedback on #1409: - cubic: the semaphore permit was acquired inside the spawned task, so it bounded execution but not spawn fan-out. Acquire before tokio::spawn so at most max_concurrency tasks (and their captured clones) exist at once; the permit is still held across generate+evaluate. Guard test unchanged and passing. - Codex: BASELINE=75.0 plus the workflow's own -5 rule made 70.0 the blocking line. BASELINE is now the measured 78.2 (block <73.2, warn <78.2). The same 78.2% was reproduced by this PR's CI leg on ubuntu/default features. Validation: 81 lib tests pass; eval 79/101 (78.2%) unchanged; clippy -D warnings clean; fmt clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JfmCW2L5d5qkNJ18mDfz6a
… claim Review feedback on #1409 (cubic): - Re-measured: src/evaluation/ is 5,267 LOC (previous 3,285 omitted evaluators/); tests/evaluation/src/ is 4,812 LOC across 26 modules (not ~6,400 / 20). - BaselineStore::store() has no caller (no baseline is ever written), but load()/compare() are reachable via the --baseline CLI flag, which CI never passes. Section 2.5 and row A6 now say exactly that. - Floor wording follows the workflow: baseline 78.2, block below 73.2. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JfmCW2L5d5qkNJ18mDfz6a
There was a problem hiding this comment.
All reported issues were addressed across 3 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
…ope, gate scope Devil's-advocate pass on the research doc (Gate-4 culture, applied voluntarily to internal-tooling research): - Narrative spine reframed: the candidate-ranking flip has two independent prerequisites — the pipeline was never wired into AgentLoop (P4) AND the eval harness cannot render a verdict (P6a/P6b). Repairing the harness alone flips nothing. - P1: define "replay" as reconstruction, not re-execution (backends are non-deterministic); order the three payoffs; offer a narrower P1a (SFT + verdict record); default-on gated on the privacy review. - §4.0: every Tier 2 item ships default-off; flipping to default-on is a user-facing change that re-enters the validation-discipline gates. - §5: tie each rejection to the proposal it constrains; §5.1 states plainly that dsh's hot-loading is a TypeScript-runtime affordance and the rejection concerns a hypothetical Rust port. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JfmCW2L5d5qkNJ18mDfz6a
Devil's Advocate ReviewAdversarial pass by the
Reviewer's "weakest assumptions to watch" (kept verbatim for the record): the Rust translation of dsh's plugin model; the P1 privacy story beyond the sanitizer sketch; what replayability delivers with non-deterministic backends; whether P3's monotonic-veto rule stays simple to maintain. CI status on the current head
Generated by Claude Code |
There was a problem hiding this comment.
All reported issues were addressed across 1 file (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
…ting claim cubic re-review on #1409: Appendix B rounded the orphan sub-crate to 4,800 LOC while A21/P6b say 4,812 — use one figure. The §4.0 sentence claimed every Tier 2 item ships behind a flag; only P1 does. P2/P3 are behavior-preserving refactors and P6b is consolidation, so the gating claim now matches the proposal text. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JfmCW2L5d5qkNJ18mDfz6a
2 participants
Description
Research into deepseek-ai/deepseek-harness ("Everything is a Plugin"), mapped against caro's three harnesses — the
AgentLoopruntime, the CaroML validator/journal layer, and the evaluation harness — with tiered adoption proposals:docs/research/deepseek-harness-lessons.md. Alongside the doc, this PR ships the smallest repairs the research surfaced (Tier-1/P6a).Motivation
Cargo.toml'scandidate-rankingfeature says: "flip to default in v1.4.0 after the eval harness shows ranking wins." We are at v1.5.0. The flip isn't overdue because ranking lost — the harness couldn't render the verdict: 3 of 4 CI eval matrix legs were silently no-ops, concurrency config was dead, and the one honest leg's regression floor (31.0) was 47 points below measured reality (78.2%). The doc's proposals all route toward making that verdict renderable; the fix commits repair what was unambiguous.Changes Made
docs/research/deepseek-harness-lessons.md(new): dsh case study; caro mapping; 8-row principle gap table; proposals P1–P8 in three tiers; what NOT to adopt (incl. the inversion: dsh's "no privileged core" must stay inverted — caro's safety floor is deliberately privileged); 24-gap eval inventory (Appendix A); orphan-crate salvage list (Appendix B).src/evaluation/harness.rs:HarnessConfig::max_concurrencywas declared but never read —run_all_testsspawned one unbounded task per (case × backend). Now bounded by atokio::sync::Semaphore; the permit is acquired beforetokio::spawn, so at mostmax_concurrencytasks (and their captured clones) exist at once, and it is held across generate+evaluate. New regression guard:evaluation::harness::tests::test_max_concurrency_bounds_in_flight_generations..github/workflows/evaluation.yml: matrix reduced tostatic_matcher— the only leg that evaluates anything (the runner registers onlyStaticMatcher;--backenddoesn't filter;embedded-*legs exited 2 behind|| true). Regression baseline set to the measured 78.2 (the workflow's existing −5 rule then blocks below 73.2 and warns under 78.2; the previous 31.0 was stale). Comments explain what restores the other legs.Cargo.toml: declarebenches/performance.rsas[[bench]] harness = false; cargo auto-discovery ran it under libtest wherecriterion_main!never executed.CLAUDE.md: nonexistentcargo run --bin caro-eval→cargo test --test evaluation; version banner 1.4.0 → 1.5.0.Type of Change
Checklist
Code Quality
cargo fmt --all(cargo fmt --checkclean)cargo clippy -- -D warnings(clean; run with--tests --benches,--no-default-features --features embedded-cpu)cargo test(81 lib tests inevaluation::pass; full eval harness run below)cargo audit(not run locally — no dependency changes; CIcargo-auditanddependency-revieware green on this PR)Testing
Documentation
run_all_tests)release-version-alignment)TDD Workflow
cargo watch -x testduring development (single-shot runs in CI-like environment)Breaking Changes
None. The semaphore bounds concurrency at the existing default (10); results are unchanged (same pass counts), only in-flight parallelism is capped.
API Changes
None.
Migration Guide
n/a
Deprecation Plan
n/a
Related Issues and Specs
thoughts/shared/plans/evaluation-harness-maturity-milestone.mdPerformance Impact
Eval harness runs are now bounded at
max_concurrency(default 10) in-flight generations instead of unbounded (101 × backends at once). For the static matcher this is latency-neutral (sub-second full run); for future LLM backends it prevents resource exhaustion.Benchmarks
cargo bench --bench performance -- --test: all criterion groups now execute (cli_startup, safety validation single/batch/concurrent/sustained, shell types, pattern matching) — previously this target never ran at all, so there is no "before" to compare.Binary Size
Unchanged (no dependency or shipped-code additions; changes are eval/test/CI-side).
Memory Usage
Bounded rather than unbounded task fan-out in eval runs (permit acquired before spawn).
Screenshots / Examples
Before
After
Testing Evidence
Test Output
CI (green): ci.yml run · LLM Evaluation Harness run — the
Evaluate static_matcherjob independently measured 79/101 = 78.2% on ubuntu-latest / default features / Rust 1.97.1, matching the local run on a different feature set and toolchain (static matcher is deterministic).Manual Testing
embedded-cpufeature set) + CI ubuntu-latest default featuresmax_concurrency: 0clamps to 1)Additional Context
Technical Decisions
|| truein the workflow: the runner exits 1 whenever pass rate < 100% (tests/evaluation/main.rs:231), so the real gate is the baseline-compare step; removing|| truewould turn CI permanently red. Fixing the exit-code contract is gap A23 in the doc, deliberately out of scope here.validation-disciplinegates before any spec.BaselineStoreclaim narrowed, semaphore acquired before spawn.## Devil's Advocate Reviewcomment on this PR (Gate-4 culture for AI-drafted proposals, applied voluntarily to internal-tooling research).Future Work
Tier-1: P7 CI doc-verification gates (seed cases: README.md:34 / CLAUDE.md:119 "93.1% pass rate" — untraceable to any harness output); P8 DeepSeek pricing/catalog rows if a DeepSeek backend ever lands. Tier-2: P1 generation journal (unlocks the currently-unreachable
sft_export), P3 validator middleware chain, P2 execution seam, P6b eval consolidation (incl. deleting the never-compiledtests/evaluation/src/orphan). Tier-3: P4 wire the candidate pipeline + flip the flag on evidence; P5 profiles.Questions for Reviewers
Is setting the regression baseline to the measured 78.2 in this PR acceptable, or should the baseline change ship separately with its own bake time? (It is a one-value revert either way.)
Reviewer Checklist
By submitting this PR, I confirm that:
🤖 Generated with Claude Code
https://claude.ai/code/session_01JfmCW2L5d5qkNJ18mDfz6a