Govern the solution server hint prompt as a versioned Jinja2 template#930
Govern the solution server hint prompt as a versioned Jinja2 template#930fabianvf wants to merge 3 commits into
Conversation
|
Warning Review limit reached
More reviews will be available in 43 minutes and 59 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughThis PR adds prompt governance documentation, a manifest-driven Jinja2 prompt renderer, a new ChangesPrompt governance and rendering pipeline
Node runtime update
Sequence Diagram(s)sequenceDiagram
participant Server as "server.generate_hint_v3"
participant PromptAPI as render_prompt
participant Env as "Jinja2 Environment"
participant Template as "generate_hint_v3.md.jinja"
Server->>PromptAPI: render_prompt("generate_hint_v3", incidents, ast_diff_str)
PromptAPI->>Env: load template by prompt id
Env->>Template: render with incidents and ast_diff_str
Template-->>Env: rendered prompt text
Env-->>PromptAPI: rendered prompt text
PromptAPI-->>Server: final prompt string
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
fabianvf
force-pushed
the
prompt-template-governance
branch
from
f1957ad to
9e2f469
Compare
Extract the live generate_hint_v3 prompt out of server.py into a versioned Jinja2 asset under src/kai_mcp_solution_server/prompts/, rendered via render_prompt(). The incident loop now lives in the template; the AST-diff computation stays in Python and is passed in as a variable. This brings the solution server's hint prompt under the same governance the editor-extensions repo adopted for its prompts (ISO 42001 A.5.2) — the surface that work explicitly left to be governed here. - prompts/: Jinja2 template + manifest (semver + checksum), render registry; ships as package data (verified present in the built wheel) - byte-exact parity tests prove the template reproduces the previous prompt string exactly across the incident / AST-diff matrix, plus a deterministic mock-model semantic-regression check - scripts/prompts_version.py + prompt-validation.yml validate Jinja syntax, declared variables, and manifest checksums on any prompts/** change - PROMPT_GOVERNANCE.md, a new CODEOWNERS, and an audit-evidence doc - remove the never-called generate_hint_v1/v2 (dead code) and the now-unused get_diff import Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Fabian von Feilitzsch <fabian@fabianism.us>
fabianvf
force-pushed
the
prompt-template-governance
branch
from
9e2f469 to
cba8734
Compare
Trunk Check fails on any PR touching Markdown or TOML because two tools can't
run in CI:
- markdownlint@0.48.0 crashes at startup under the bundled node 18.12.1
("SyntaxError: Invalid regular expression flags" — string-width uses the
unicode-sets `v` regex flag, which needs node 20+).
- taplo@0.10.0's Linux binary fails to install ("Binary not found") with the
old plugins definitions.
Bump the node runtime to 22.16.0, the trunk-io/plugins ref to v1.10.2 (refreshes
tool download definitions; pinned tool versions are unchanged), and markdownlint
to 0.49.0. `trunk check` reports no new issues across the changed files.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Fabian von Feilitzsch <fabian@fabianism.us>
fabianvf
force-pushed
the
prompt-template-governance
branch
from
68e68e4 to
e007b11
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
kai_mcp_solution_server/tests/prompts/test_semantic_regression.py (1)
69-73: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAssert all incident fields that the prompt contract depends on.
The regression currently checks only a subset of incident fields. Add assertions for
code_snip,line_number, andvariablesto catch accidental template regressions in those sections.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kai_mcp_solution_server/tests/prompts/test_semantic_regression.py` around lines 69 - 73, The semantic regression test is only validating a subset of the incident data rendered by the prompt. Update the assertions in test_semantic_regression to also check code_snip, line_number, and variables for each incident, alongside the existing uri, message, and violation_name checks, so the prompt contract covers all fields the template depends on..github/workflows/prompt-validation.yml (1)
25-35: 🔒 Security & Privacy | 🔵 Trivial | 💤 Low valueOptional: align with the repo's action-pinning / credential-hygiene policy.
zizmorflags theseuses:references as unpinned (not pinned to a commit SHA) and notesactions/checkoutdoesn't setpersist-credentials: false. If the repo enforces a blanket pinning policy, pin to immutable SHAs and disable credential persistence on checkout; otherwise this is consistent with existing workflows and can be deferred.🔒 Suggested hardening
- uses: actions/checkout@<commit-sha> # v4 with: persist-credentials: false🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/prompt-validation.yml around lines 25 - 35, The workflow step in the prompt-validation job uses unpinned GitHub Actions and checkout still persists credentials. If this repo follows the pinning/credential-hygiene policy, update the actions referenced by actions/checkout, actions/setup-python, and astral-sh/setup-uv in this job to immutable commit SHAs, and set persist-credentials to false on the actions/checkout step; otherwise leave it as-is if the repo intentionally allows this pattern.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kai_mcp_solution_server/PROMPT_GOVERNANCE.md`:
- Around line 75-76: The required status-check name in PROMPT_GOVERNANCE should
match the exact workflow/job label reported by the CI system. Update the status
check entry from Validate prompt templates to the actual check name used by the
prompt validation workflow, and keep the Code Owners requirement unchanged so
the documented branch protection gate aligns with the status shown by the Prompt
Template Validation (ISO 42001 A.5.2) job.
In `@kai_mcp_solution_server/scripts/prompts_version.py`:
- Around line 46-111: `check()` currently validates template integrity but
misses the manifest↔package version contract, so add a version lockstep
validation between `manifest.yaml` and the package version in `pyproject.toml`.
Update `check()` in prompts_version.py to load/compare both versions, report a
clear problem when they differ, and fail the check alongside the existing
`problems` list; use the existing `load_manifest()` flow and extend it with a
helper or inline read of the package version so the CI gate catches drift.
---
Nitpick comments:
In @.github/workflows/prompt-validation.yml:
- Around line 25-35: The workflow step in the prompt-validation job uses
unpinned GitHub Actions and checkout still persists credentials. If this repo
follows the pinning/credential-hygiene policy, update the actions referenced by
actions/checkout, actions/setup-python, and astral-sh/setup-uv in this job to
immutable commit SHAs, and set persist-credentials to false on the
actions/checkout step; otherwise leave it as-is if the repo intentionally allows
this pattern.
In `@kai_mcp_solution_server/tests/prompts/test_semantic_regression.py`:
- Around line 69-73: The semantic regression test is only validating a subset of
the incident data rendered by the prompt. Update the assertions in
test_semantic_regression to also check code_snip, line_number, and variables for
each incident, alongside the existing uri, message, and violation_name checks,
so the prompt contract covers all fields the template depends on.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 58245fea-789e-4024-9729-d5c539b30002
⛔ Files ignored due to path filters (1)
kai_mcp_solution_server/uv.lockis excluded by!**/*.lock
📒 Files selected for processing (16)
.github/CODEOWNERS.github/workflows/prompt-validation.yml.trunk/trunk.yamlkai_mcp_solution_server/PROMPT_GOVERNANCE.mdkai_mcp_solution_server/docs/ISO-42001-A5.2-evidence.mdkai_mcp_solution_server/pyproject.tomlkai_mcp_solution_server/requirements.txtkai_mcp_solution_server/scripts/prompts_version.pykai_mcp_solution_server/src/kai_mcp_solution_server/prompts/__init__.pykai_mcp_solution_server/src/kai_mcp_solution_server/prompts/manifest.yamlkai_mcp_solution_server/src/kai_mcp_solution_server/prompts/templates/generate_hint_v3.md.jinjakai_mcp_solution_server/src/kai_mcp_solution_server/server.pykai_mcp_solution_server/tests/prompts/__init__.pykai_mcp_solution_server/tests/prompts/oracle.pykai_mcp_solution_server/tests/prompts/test_parity.pykai_mcp_solution_server/tests/prompts/test_semantic_regression.py
| 1. **Require the status check** `Validate prompt templates`. | ||
| 2. **Require review from Code Owners** so `prompts/**` edits need `@konveyor/kai-prompt-reviewers`. |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Align required status-check name with the actual workflow/job check.
Line 75 requires Validate prompt templates, but the workflow context is named Prompt Template Validation (ISO 42001 A.5.2). If these don’t match the actual reported check name, branch protection cannot enforce this gate as documented.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kai_mcp_solution_server/PROMPT_GOVERNANCE.md` around lines 75 - 76, The
required status-check name in PROMPT_GOVERNANCE should match the exact
workflow/job label reported by the CI system. Update the status check entry from
Validate prompt templates to the actual check name used by the prompt validation
workflow, and keep the Code Owners requirement unchanged so the documented
branch protection gate aligns with the status shown by the Prompt Template
Validation (ISO 42001 A.5.2) job.
From CodeRabbit's review: - prompts_version.py: the check now fails if the prompt-set version in manifest.yaml drifts from the package version in pyproject.toml, enforcing the lockstep the governance doc documents. - test_semantic_regression.py: also assert code_snip, line_number, and variables render, so the prompt contract covers every incident field the template depends on, not just uri/message/violation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Fabian von Feilitzsch <fabian@fabianism.us>
1 participant
What
Moves the solution server's live hint prompt (
generate_hint_v3) out of an inline f-string inserver.pyand into a versioned Jinja2 template underkai_mcp_solution_server/src/kai_mcp_solution_server/prompts/, rendered through a smallrender_prompt(...)registry. A checksummedmanifest.yamltracks it.Why
This is the Kai side of the prompt-template governance we just landed in editor-extensions (ISO 42001 A.5.2). That work explicitly scoped out the solution-server prompts, noting they're owned here — this closes that loop. The hint prompt is effectively the instruction set for hint generation, so it gets version control, peer review, and CI validation independent of the application code. The incident loop now lives in the template; the AST-diff computation stays in Python and is passed in as a variable.
Behavior
No behavior change for the live prompt. A parity suite renders the template and asserts it matches a verbatim copy of the old f-string byte-for-byte across the incident / AST-diff matrix (0/1/N incidents, empty/populated diff, multi-line and special-char fields). Jinja renders
None/dicts identically to the old f-strings, so there's no silent drift.generate_hint_v1andgenerate_hint_v2were never called (only v3 is wired up, atserver.py:1098) — they're removed as dead code, along with the now-unusedget_diffimport.What's here
prompts/— the Jinja2 template + manifest and the render registry; shipped as package data (verified present in the built wheel).scripts/prompts_version.py+.github/workflows/prompt-validation.yml— validate Jinja syntax, declared-vs-referenced variables, and manifest checksums on anyprompts/**change, plus the parity and a deterministic (mock-model) semantic-regression test.PROMPT_GOVERNANCE.md, a new.github/CODEOWNERS(scoped to the prompts dir only), and an audit-evidence doc.Verification
uv build(template + manifest in the wheel),pytest tests/prompts(7 pass),python scripts/prompts_version.py check,mypy src, and ruff/Trunk all green.requirements.txtregenerated for the newjinja2dependency.Needs a repo admin (can't be done in code)
@konveyor/kai-prompt-reviewersteam.Validate prompt templatescheck + Code-Owner review on the prompts directory (documented inPROMPT_GOVERNANCE.md).🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes