Conversation
Research document mapping stanfordnlp/dspy (v3.3.0) concepts onto caro's prompt and evaluation architecture, with a phased adoption plan. Key findings (all verified against main): - Production prompts are 8 hand-edited inline strings, one per backend, while the structured src/prompts/ machinery is dead code. - Three sources of truth disagree on "list all files in the current directory": shipped prompt says `ls`, dead few-shot library says `ls -a`, eval dataset expects `ls -la`. - ~290 eval cases, a chi-square A/B engine, and a versioned prompt registry exist but are not connected into any optimization loop. - Runtime feedback loops are missing where they matter most: the final safety block has no regeneration hook and confidence_score is a per-backend constant. Eight ranked learnings (compiled prompt artifacts, metric-with-feedback, offline GEPA/BootstrapFewShot harness, bounded Refine on safety failure, platform demos as data, shared output adapter, honest confidence, BootstrapFinetune tie-in), a four-phase roadmap, rejected alternatives (dspy-rs runtime dep), risks/guardrails, and 11 proposed follow-up issues. Docs-only; no code changes. Related to #517. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WwYKNGPu4RYoe6dipMkSka
|
The latest updates on your projects. Learn more about Vercel for GitHub.
5 Skipped Deployments
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
Contributor
|
Solid work! 💪 |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
github-actions
Bot
added
documentation
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 1 file
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6fb48548cf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Contributor
Website Claims Verification ReportRun ID: 34723931702 Platform Resultsclaims-report-macos-latest{
"suite": "website-claims",
"platform": "macos-latest",
"timestamp": "2026-09-12T22:55:33Z",
"passed": 84,
"skipped": 0,
"warnings": 1,
"run_id": "34723931702",
"sha": "f9efbd401b36db6972197283c61f29f34de167bc"
}claims-report-ubuntu-latest{
"suite": "website-claims",
"platform": "ubuntu-latest",
"timestamp": "2026-09-12T22:56:19Z",
"passed": 84,
"skipped": 0,
"warnings": 1,
"run_id": "34723931702",
"sha": "f9efbd401b36db6972197283c61f29f34de167bc"
}DocumentationNext Steps
|
wildcard
added
the
path:auto
Corrections from PR #1448 review (cubic, Codex), each verified against main: - Dataset inventory: JSON cases live under tests/evaluation/datasets/** (5 files, 80 cases), not a root datasets/ dir; test_cases.toml has 55 cases, not 56. - Harness backend gap: tests/evaluation/main.rs rejects the CI matrix's embedded-smollm/embedded-qwen names and registers only StaticMatcher for the rest, so no LLM backend has ever been scored by the CI eval workflow. Added to §2.3 and as a Phase 1 prerequisite. - Metric attribution: the CI target imports caro::evaluation, so the gate metric is the boolean one in src/evaluation/evaluators/; the graded ladder in tests/evaluation/src/evaluator.rs belongs to the unlinked caro-evaluation crate. L3 harness must mirror the former. - Paired comparison: prompt_comparison::chi_square_test is unpaired; the Phase 3 exit criterion now requires McNemar's (or paired bootstrap). - Added open umbrella issue #798 (embedded prompt quality) to §2.3 and the follow-up list, now 13 items. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WwYKNGPu4RYoe6dipMkSka
Owner
Author
|
Addressed the review in 9851175 — all five threads resolved:
CI: the only red check, Security Audit, is Filed #1451 for the first follow-up (L2, diagnostic evaluator feedback). Generated by Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Research document: what stanfordnlp/dspy (v3.3.0) teaches about caro's prompt + evaluation architecture, and a phased adoption roadmap. Adds one file:
docs/research/dspy-prompt-optimization.md.Motivation
caro's quality ceiling on small local models (default Qwen2.5-Coder-1.5B Q4) is set by hand-tuned system prompts, iterated via the manual
prompt-tunerloop. DSPy's core idea — the prompt is a compiled, versioned artifact optimized against a metric and a dataset, not hand-edited source — is exactly the discipline caro is missing. Exploration confirmed every ingredient already exists in the repo (~290 eval cases, a 5k-line harness, a chi-square A/B engine, a versioned prompt registry, per-platform few-shot libraries) but none of it is connected into an optimization loop.The motivating example found during research: three sources of truth disagree on "list all files in the current directory" — the shipped embedded prompt teaches
ls(embedded_backend.rs:221), the dead few-shot library saysls -a(smollm_prompt.rs:561), and the eval dataset expectsls -la(correctness-001). Nothing in the build detects this.Changes Made
docs/research/dspy-prompt-optimization.md(docs-only, ~500 lines):program.save()JSON bridge that lets Python-side optimization feed a Rust runtimemainwith file refs: 8 divergent inlinecreate_system_prompt()bodies, deadsrc/prompts/machinery, disconnected eval/A-B/registry, missing runtime feedback loops (no regeneration on final safety block;confidence_scoreis a per-backend constant;_max_iterationsunused), the truncatedprompt-tunerskilloptimize.pysketchdspy-rsruntime dep, Python sidecar, replacing the Rust harness, big-bangsrc/prompts/rewrite)maintodayType of Change
Checklist
Code Quality
cargo fmt --all— N/A, no Rust changescargo clippy -- -D warnings— N/A, no Rust changescargo test— N/A, no Rust changescargo audit— N/A, no dependency changesTesting
mainbefore commitDocumentation
docs/research/following the existing kebab-case convention (gh-dash-tui-fundamentals.md,kavana-ball-interaction.md)TDD Workflow
Breaking Changes
None. No code, config, or public API changes.
Related Issues and Specs
thoughts/shared/plans/evaluation-harness-maturity-milestone.md(Phase 2: Prompt Engineering Framework)tests/evaluation/src/prompt_comparison.rs,tests/evaluation/prompts/v1.0/,src/prompts/minimal.rs,src/evaluation/sft_export.rsPerformance Impact
None (docs-only).
Screenshots / Examples
N/A — no CLI behavior changes.
Testing Evidence
Docs-only PR: no test suite applies. Verification performed:
create_system_prompt()body (src/backends/embedded/embedded_backend.rs:183-268) andtests/evaluation/dataset.yamlcorrectness-001_max_iterations: 2is underscore-prefixed/unused (src/agent/mod.rs:29,77), the final safety pass atsrc/cli/mod.rs:836-838has no regeneration path, andconfidence_scoreconstants per backend (embedded_backend.rs:511,static_matcher.rs:1901, etc.).github/workflows/evaluation.ymlruns the harness under|| trueand skips the threshold check for0.0baselines.claude/skills/prompt-tuner/SKILL.mdis 107 lines ending in an unclosed code fence, anddata/evals/default.yaml(cited bysrc/prompts/minimal.rs) does not existAdditional Context
Technical Decisions
optimize.pywould ship untested. The doc embeds the artifact schema and a clearly-marked sketch; the real harness lands in Phase 2 with a recorded before/after run as evidence.dspy-rsdependency proposed. The optimizer is the valuable part and it runs offline; caro's runtime needs (render + parse + bounded retry) are a few hundred lines of Rust. If revisited, it goes through.claude/rules/external-sdk-integration.mdfirst..claude/rules/validation-discipline.mdgates do not apply; Phase 1–3 feature PRs will carryfeature-evidence.mdartifacts.Future Work
Eleven proposed follow-up issues are listed in §4 of the doc (not filed — owner's call). The two cheapest, highest-leverage next steps are diagnostic
failure_reasonfeedback in the evaluators (L2) and bounded regeneration when the final safety pass blocks (L4). Boy-scout item also listed: repair the truncatedprompt-tunerSKILL.md.Questions for Reviewers
tools/dspy-harness/(besidetools/mlx-finetune/) the right home for Python dev tooling, or is there a preferred location?Reviewer Checklist
main(file refs, counts, CI behavior)By submitting this PR, I confirm that:
🤖 Generated with Claude Code
https://claude.ai/code/session_01WwYKNGPu4RYoe6dipMkSka
Generated by Claude Code
Summary by cubic
Adds a research doc mapping DSPy's prompt-optimization concepts onto caro's prompt and evaluation architecture, with a phased adoption roadmap to close the gap between hand-tuned prompts and an optimization loop.
main: eight divergent inline prompt bodies, dead structured prompt machinery, the CI harness scoring only the static matcher, and three sources of truth disagreeing on a basic query.Written for commit 9851175. Summary will update on new commits.