ci(canon-quality): wire oddkit_audit on PR + push (Phase 3 PR-3.1, soft-block)#149
Open
ci(canon-quality): wire oddkit_audit on PR + push (Phase 3 PR-3.1, soft-block)#149
Conversation
Canon Quality —
|
| Line | Rule | Occurrence | Message |
|---|---|---|---|
| 203 | dead-reference |
klappy://writings/four-questions-that-change-everything |
URI does not resolve |
writings/the-broken-wall-and-the-buried-talent.md — 1 finding(s)
| Line | Rule | Occurrence | Message |
|---|---|---|---|
| 332 | dead-reference |
klappy://draft-zeros/appendix-a-the-biblical-roots |
URI does not resolve |
writings/the-voice-came-first.md — 1 finding(s)
| Line | Rule | Occurrence | Message |
|---|---|---|---|
| 244 | dead-reference |
klappy://writings/four-questions-that-change-everything |
URI does not resolve |
Soft-block mode — this status is informational. The job will not fail. Hard-block ships in PR-3.2 after the observation cycle.
Spec: klappy://docs/oddkit/specs/oddkit-audit · Workflow: .github/workflows/canon-quality.yml · Run: #3
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
…X from hard block - Wrap the SSE/JSON-RPC parsing heredoc with an if-guard so a parse error routes through the same audit_failed=true path used for HTTP failures instead of letting bash -e abort the step (which previously skipped the PR comment and produced a red CI check with zero feedback in soft mode). - Enforcement gate now only fails on STATUS=FINDINGS in hard mode. Per docs/oddkit/specs/oddkit-audit.md, PARTIAL_INDEX is best-effort and must remain non-blocking (warning, retry on next push).
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Hard-mode comment misleads on PARTIAL_INDEX status
- The renderer now mirrors the enforcement gate's predicate (will_fail = mode=='hard' and status=='FINDINGS'), picks the warning icon and a PARTIAL_INDEX-specific footer in hard mode so the comment matches the actual CI outcome.
- ✅ Fixed: Workflow input interpolated unsafely into shell
- Replaced direct ${{ github.event.inputs.scope_paths }} interpolation in the run script with an INPUT_SCOPE_PATHS env var referenced as a shell variable, treating the input as data and removing the quoting/injection hazard.
Preview (81c0aa2b13)
diff --git a/.github/workflows/canon-quality.yml b/.github/workflows/canon-quality.yml
new file mode 100644
--- /dev/null
+++ b/.github/workflows/canon-quality.yml
@@ -1,0 +1,314 @@
+name: Canon Quality
+
+# Calls oddkit_audit against prod and reports/blocks based on findings.
+# Spec: klappy://docs/oddkit/specs/oddkit-audit (DRAFT v2.2)
+# Mode is controlled by repo variable AUDIT_ENFORCEMENT_MODE:
+# - 'soft' (default): comment with findings, never fail the job
+# - 'hard': fail the job when status != OK
+# Set the variable at https://github.com/klappy/klappy.dev/settings/variables/actions
+# PR-3.1 ships in soft. PR-3.2 will flip to hard after the observation cycle.
+
+on:
+ pull_request:
+ branches: [main]
+ paths:
+ - 'writings/**'
+ - 'canon/**'
+ - 'odd/**'
+ - 'docs/**'
+ - '.github/workflows/canon-quality.yml'
+ push:
+ branches: [main]
+ paths:
+ - 'writings/**'
+ - 'canon/**'
+ - 'odd/**'
+ - 'docs/**'
+ - '.github/workflows/canon-quality.yml'
+ workflow_dispatch:
+ inputs:
+ scope_paths:
+ description: 'JSON array of scope paths (default ["writings/"])'
+ required: false
+ default: '["writings/"]'
+
+permissions:
+ contents: read
+ pull-requests: write
+
+env:
+ ODDKIT_MCP_URL: https://oddkit.klappy.dev/mcp
+ ENFORCEMENT_MODE: ${{ vars.AUDIT_ENFORCEMENT_MODE || 'soft' }}
+ USER_AGENT: 'klappy.dev-canon-quality/1.0 (+github-actions; ${{ github.repository }}#${{ github.run_id }})'
+
+jobs:
+ audit:
+ name: Reference integrity audit
+ runs-on: ubuntu-latest
+ timeout-minutes: 5
+ steps:
+ - name: Resolve scope
+ id: scope
+ env:
+ INPUT_SCOPE_PATHS: ${{ github.event.inputs.scope_paths }}
+ run: |
+ if [ -n "$INPUT_SCOPE_PATHS" ]; then
+ PATHS="$INPUT_SCOPE_PATHS"
+ else
+ PATHS='["writings/"]'
+ fi
+ echo "Scope paths: $PATHS"
+ echo "paths=$PATHS" >> "$GITHUB_OUTPUT"
+
+ - name: Call oddkit_audit (with one retry on transient failure)
+ id: audit
+ env:
+ SCOPE_PATHS: ${{ steps.scope.outputs.paths }}
+ run: |
+ set -uo pipefail
+
+ # Build JSON-RPC payload. The 'input' arg is a JSON string per the action's schema.
+ INNER=$(python3 -c "import json,os; print(json.dumps({'scope':{'paths':json.loads(os.environ['SCOPE_PATHS'])}}))")
+ PAYLOAD=$(python3 -c "
+ import json, sys
+ inner = sys.argv[1]
+ print(json.dumps({
+ 'jsonrpc': '2.0',
+ 'id': 1,
+ 'method': 'tools/call',
+ 'params': {'name': 'oddkit_audit', 'arguments': {'input': inner}}
+ }))
+ " "$INNER")
+ echo "$PAYLOAD" > /tmp/req.json
+
+ AUDIT_OK=false
+ HTTP_CODE=000
+ for attempt in 1 2; do
+ HTTP_CODE=$(curl -sS -o /tmp/resp.raw -w '%{http_code}' \
+ -X POST \
+ -H 'Content-Type: application/json' \
+ -H 'Accept: application/json, text/event-stream' \
+ -H "User-Agent: $USER_AGENT" \
+ --max-time 120 \
+ -d @/tmp/req.json \
+ "$ODDKIT_MCP_URL" || echo 000)
+ echo "Attempt $attempt -> HTTP $HTTP_CODE"
+ if [ "$HTTP_CODE" = "200" ]; then
+ AUDIT_OK=true
+ break
+ fi
+ if [ "$attempt" -lt 2 ]; then
+ echo "Transient failure; sleeping 8s before retry"
+ sleep 8
+ fi
+ done
+
+ if [ "$AUDIT_OK" != "true" ]; then
+ echo "Audit could not be completed after 2 attempts."
+ echo "Response (first 500 bytes):"
+ head -c 500 /tmp/resp.raw || true
+ echo ""
+ echo "audit_failed=true" >> "$GITHUB_OUTPUT"
+ echo "http_code=$HTTP_CODE" >> "$GITHUB_OUTPUT"
+ exit 0 # don't fail the step; the report step will surface this
+ fi
+
+ # Parse the SSE response. The MCP server returns event-stream with a final
+ # 'data: { ... }' line containing the JSON-RPC envelope.
+ # Guard with `if !` so any parse failure (sys.exit / unhandled exception)
+ # is routed through the same audit_failed=true path used for HTTP failures,
+ # rather than aborting the step under bash -e and skipping the comment steps.
+ if ! python3 - <<'PY'
+ import json, sys
+ with open('/tmp/resp.raw') as f:
+ raw = f.read()
+ data_lines = [l[6:] for l in raw.splitlines() if l.startswith('data: ')]
+ if not data_lines:
+ sys.stderr.write("No SSE 'data: ' lines in response\n")
+ sys.stderr.write(raw[:500])
+ sys.exit(1)
+ envelope = json.loads(data_lines[-1])
+ if 'error' in envelope:
+ sys.stderr.write(f"JSON-RPC error: {envelope['error']}\n")
+ sys.exit(1)
+ # Extract the action envelope from content[0].text
+ content = envelope.get('result', {}).get('content', [])
+ texts = [c.get('text','') for c in content if c.get('type') == 'text']
+ if not texts:
+ sys.stderr.write("No text content in tool-call result\n")
+ sys.exit(1)
+ inner = json.loads(texts[0])
+ with open('/tmp/audit-result.json','w') as f:
+ json.dump(inner, f, indent=2)
+ # Surface a tiny diagnostic to logs
+ r = inner.get('result', {})
+ print(f"status={r.get('status')} findings={r.get('summary',{}).get('total_findings',0)} files={r.get('summary',{}).get('files_scanned',0)}")
+ PY
+ then
+ echo "Failed to parse audit response."
+ echo "Response (first 500 bytes):"
+ head -c 500 /tmp/resp.raw || true
+ echo ""
+ echo "audit_failed=true" >> "$GITHUB_OUTPUT"
+ echo "http_code=parse_error" >> "$GITHUB_OUTPUT"
+ exit 0 # don't fail the step; the report step will surface this
+ fi
+
+ echo "audit_failed=false" >> "$GITHUB_OUTPUT"
+
+ - name: Upload audit response artifact
+ if: always() && steps.audit.outputs.audit_failed == 'false'
+ uses: actions/upload-artifact@v4
+ with:
+ name: audit-response
+ path: /tmp/audit-result.json
+ retention-days: 14
+ if-no-files-found: warn
+
+ - name: Render comment body
+ id: render
+ if: github.event_name == 'pull_request' && steps.audit.outputs.audit_failed != 'true'
+ env:
+ MODE: ${{ env.ENFORCEMENT_MODE }}
+ run: |
+ python3 - <<'PY'
+ import json, os
+
+ with open('/tmp/audit-result.json') as f:
+ d = json.load(f)
+ r = d.get('result', {})
+ status = r.get('status', '?')
+ summary = r.get('summary', {})
+ findings = r.get('findings', [])
+ suppressed = r.get('suppressed_findings', [])
+ scope = r.get('scope', {})
+ mode = os.environ.get('MODE', 'soft')
+ paths_label = ', '.join(scope.get('paths', []))
+
+ lines = []
+ if status == 'OK':
+ lines.append('### Canon Quality — `oddkit_audit` ✅')
+ lines.append('')
+ lines.append(f'No dead `klappy://` references or legacy link patterns found in `{paths_label}`. {summary.get("files_scanned", 0)} files scanned.')
+ else:
+ # Per the audit spec, PARTIAL_INDEX is non-blocking even in hard mode
+ # (best-effort findings, retry on next push). Only FINDINGS fails in hard.
+ will_fail = mode == 'hard' and status == 'FINDINGS'
+ icon = '❌' if will_fail else '⚠️'
+ lines.append(f'### Canon Quality — `oddkit_audit` {icon}')
+ lines.append('')
+ total = summary.get('total_findings', len(findings))
+ lines.append(f'**{total} finding(s)** in `{paths_label}` ({summary.get("files_scanned", 0)} files scanned). Mode: `{mode}`.')
+ lines.append('')
+
+ # Group by file, cap at 50 rendered to keep PR comment manageable
+ CAP = 50
+ by_file = {}
+ for f in findings[:CAP]:
+ loc = f.get('location', {})
+ by_file.setdefault(loc.get('path', '?'), []).append(
+ (loc.get('line'), f.get('rule_id'), f.get('occurrence'), f.get('message'))
+ )
+
+ for path, items in sorted(by_file.items()):
+ lines.append(f'<details><summary><code>{path}</code> — {len(items)} finding(s)</summary>')
+ lines.append('')
+ lines.append('| Line | Rule | Occurrence | Message |')
+ lines.append('|---:|---|---|---|')
+ for line, rule, occ, msg in items:
+ occ_safe = (occ or '').replace('|', '\\|')
+ msg_safe = (msg or '').replace('|', '\\|')
+ lines.append(f'| {line} | `{rule}` | `{occ_safe}` | {msg_safe} |')
+ lines.append('')
+ lines.append('</details>')
+
+ if len(findings) > CAP:
+ lines.append('')
+ lines.append(f'_… and {len(findings) - CAP} more finding(s). See the `audit-response` workflow artifact for the full list._')
+
+ if suppressed:
+ s_count = len(suppressed) if isinstance(suppressed, list) else suppressed
+ lines.append('')
+ lines.append(f'_{s_count} finding(s) suppressed via `<!-- audit-allow: ... -->` directives._')
+
+ lines.append('')
+ if mode == 'soft':
+ lines.append('> **Soft-block mode** — this status is informational. The job will not fail. Hard-block ships in PR-3.2 after the observation cycle.')
+ elif status == 'PARTIAL_INDEX':
+ lines.append('> **Hard-block mode** — `PARTIAL_INDEX` is non-blocking per the audit spec (best-effort findings, retry on next push). The job will not fail on this status.')
+ else:
+ lines.append('> **Hard-block mode** — this PR will fail until findings are resolved. Fix the dead references or add a line-level allowlist directive (`<!-- audit-allow: dead-reference reason="..." -->`) above the offending link.')
+
+ lines.append('')
+ lines.append('<sub>Spec: `klappy://docs/oddkit/specs/oddkit-audit` · Workflow: `.github/workflows/canon-quality.yml` · Run: [#${{ github.run_number }}](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})</sub>')
+
+ with open('/tmp/comment.md', 'w') as f:
+ f.write('\n'.join(lines))
+ PY
+
+ - name: Sticky comment — audit results
+ if: github.event_name == 'pull_request' && steps.audit.outputs.audit_failed != 'true'
+ uses: marocchino/sticky-pull-request-comment@v2
+ with:
+ header: canon-quality-audit
+ path: /tmp/comment.md
+
+ - name: Sticky comment — audit infrastructure failure
+ if: github.event_name == 'pull_request' && steps.audit.outputs.audit_failed == 'true'
+ uses: marocchino/sticky-pull-request-comment@v2
+ with:
+ header: canon-quality-audit
+ message: |
+ ### Canon Quality — `oddkit_audit` ⚠️ (infrastructure)
+
+ The audit could not be completed (HTTP `${{ steps.audit.outputs.http_code }}` from `${{ env.ODDKIT_MCP_URL }}` after retry). This run produces no signal about reference integrity — re-run the workflow if the failure was transient.
+
+ The job does not fail on infrastructure issues; persistent failures should be tracked separately.
+
+ <sub>Workflow: `.github/workflows/canon-quality.yml` · Run: [#${{ github.run_number }}](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})</sub>
+
+ - name: Workflow step summary
+ if: always()
+ run: |
+ {
+ echo "## Canon Quality — \`oddkit_audit\`"
+ echo ""
+ echo "- **Mode**: \`$ENFORCEMENT_MODE\`"
+ echo "- **Endpoint**: \`$ODDKIT_MCP_URL\`"
+ } >> "$GITHUB_STEP_SUMMARY"
+
+ if [ "${{ steps.audit.outputs.audit_failed }}" = "true" ]; then
+ echo "- **Result**: infrastructure failure (HTTP \`${{ steps.audit.outputs.http_code }}\`)" >> "$GITHUB_STEP_SUMMARY"
+ elif [ -f /tmp/audit-result.json ]; then
+ python3 - <<'PY'
+ import json, os
+ with open('/tmp/audit-result.json') as f:
+ d = json.load(f)
+ r = d.get('result', {})
+ s = r.get('summary', {})
+ out_path = os.environ['GITHUB_STEP_SUMMARY']
+ with open(out_path, 'a') as f:
+ f.write(f"- **Status**: `{r.get('status')}`\n")
+ f.write(f"- **Total findings**: {s.get('total_findings', 0)}\n")
+ f.write(f"- **By severity**: `{s.get('by_severity', {})}`\n")
+ f.write(f"- **Files scanned**: {s.get('files_scanned', 0)}\n")
+ f.write(f"- **Scope**: `{r.get('scope', {})}`\n")
+ if s.get('truncated'):
+ f.write(f"- **Truncated**: yes (response was capped)\n")
+ PY
+ else
+ echo "- **Result**: no audit result file produced" >> "$GITHUB_STEP_SUMMARY"
+ fi
+
+ - name: Enforcement gate
+ if: steps.audit.outputs.audit_failed != 'true'
+ run: |
+ STATUS=$(python3 -c "import json; print(json.load(open('/tmp/audit-result.json'))['result']['status'])")
+ echo "Enforcement mode: $ENFORCEMENT_MODE | Audit status: $STATUS"
+ # Per the audit spec, PARTIAL_INDEX is non-blocking (best-effort findings,
+ # retry on next push). Only FINDINGS should fail the job in hard mode.
+ if [ "$ENFORCEMENT_MODE" = "hard" ] && [ "$STATUS" = "FINDINGS" ]; then
+ echo "::error::Hard-block mode: oddkit_audit returned status=$STATUS. Failing job — fix the dead references or add explicit allowlist directives."
+ exit 1
+ fi
+ echo "Not failing the job (mode=$ENFORCEMENT_MODE, status=$STATUS)."You can send follow-ups to the cloud agent here.
Reviewed by Cursor Bugbot for commit 2d9d09a. Configure here.
… scope input - Render comment now treats PARTIAL_INDEX as non-blocking even in hard mode, matching the enforcement gate which only fails on FINDINGS. Adds a PARTIAL_INDEX-specific footer in hard mode and selects the warning icon when the job will not fail. - Resolve scope step passes github.event.inputs.scope_paths via env (INPUT_SCOPE_PATHS) instead of interpolating into the shell body, eliminating the quoting hazard and shell injection vector under workflow_dispatch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 3 PR-3.1 of the link-rot elimination campaign — wires
oddkit_audit(live in prod since v0.26.0) into klappy.dev's CI as a soft-block quality gate. This is the first GitHub Actions workflow on the repo.## What this PR addsA single new file:.github/workflows/canon-quality.yml(~290 lines, well-commented).Triggers- Pull requests againstmainthat touchwritings/,canon/,odd/,docs/, or the workflow itself- Pushes tomainwith the same path filter- Manualworkflow_dispatchwith optionalscope_pathsinputBehavior1. POST atools/callforoddkit_audittohttps://oddkit.klappy.dev/mcp(default scope["writings/"]per spec v2.2)2. Retry once on transient HTTP failure (handles the DNS/cold-start cases observed during PR #146 verification)3. Parse the SSE response, extract the action envelope, save as a workflow artifact (audit-response, 14-day retention)4. Render a sticky PR comment viamarocchino/sticky-pull-request-comment@v2(header: canon-quality-audit): - Onstatus: OK— green checkmark, files-scanned count - Onstatus: FINDINGS— table grouped by file (max 50 rendered, link to artifact for the rest), columns: line, rule, occurrence, message - On infrastructure failure — explicit "could not be checked" comment so the absence of signal is visible5. Write a workflow run summary with the same metrics6. Enforcement gate: read repo variableAUDIT_ENFORCEMENT_MODE(defaults tosoft) -soft— never fails the job; the comment is the only signal -hard— fails the job whenstatus != OKMode flip mechanismPR-3.1 ships insoft(no repo variable set → default). After the observation cycle (3–5 PRs through the gate), PR-3.2 will setvars.AUDIT_ENFORCEMENT_MODEtohardat https://github.com/klappy/klappy.dev/settings/variables/actions — single repo-config change, no workflow file edit needed.## Self-test on this PRThis PR's path filter includes.github/workflows/canon-quality.yml, so the workflow runs on this PR itself. Expected first-run signal: 3 dead-references inwritings/(the same 3 surfaced by post-deploy verification of v0.26.0 — actual link-rot in canon, real Phase 3 input). The job will not fail because mode issoftby default.## Refs- Spec:klappy://docs/oddkit/specs/oddkit-audit(DRAFT v2.2 — landing in companion canon PR #148)- Promote that made this possible: klappy/oddkit#146 (oddkit v0.26.0)- Session ledger:klappy://odd/ledger/2026-04-27-link-rot-phase-2-shipped(also in klappy.dev#148)- Resume handoff:klappy://odd/handoffs/2026-04-27-link-rot-phase-2-promote-resume(also in klappy.dev#148)- Campaign sequencing:klappy://docs/planning/link-rot-elimination-campaign## After merge1. Observation cycle — let 3–5 PRs run through the soft gate. Watch finding rate, sticky-comment readability, and CI duration.2. Fix surfaced rot — the 3 known dead references inwritings/are the obvious first sweep; line-level allowlist (<!-- audit-allow: dead-reference reason="..." -->) is available for genuinely-deferred targets like upcoming-but-unpublished articles.3. PR-3.2 — setvars.AUDIT_ENFORCEMENT_MODEtohard(no workflow file change required) once observation is satisfactory.## Notes- The workflow usespython3 - <<PYheredocs for JSON manipulation rather than installingjq— keeps the runner setup time near zero.- All comments and step-summaries are read-only; the workflow haspull-requests: writeonly for the sticky comment.- Thesoftdefault is the safe-by-construction state — even if this PR has a typo that makesAUDIT_ENFORCEMENT_MODE=hardaccidentally read incorrectly, the worst case is reporting-only, not surprise PR blocks.Note
Medium Risk
Adds a new CI workflow that calls an external production
oddkitendpoint and posts sticky PR comments, introducing a new dependency that can affect developer workflow (and optionally fail builds when flipped to hard mode). Permissions includepull-requests: write, so misconfiguration could spam or overwrite PR comments.Overview
Introduces a new GitHub Actions workflow (
canon-quality.yml) that runsoddkit_auditon PRs/pushes touching canon content (and via manual dispatch), uploads the JSON audit result as an artifact, and posts a sticky PR comment summarizing findings.Adds a configurable enforcement gate via
AUDIT_ENFORCEMENT_MODE(default soft), with retry/parse handling for audit infrastructure failures and an optional hard mode that fails the job only when the audit status isFINDINGS(notPARTIAL_INDEX).Reviewed by Cursor Bugbot for commit 81c0aa2. Bugbot is set up for automated code reviews on this repo. Configure here.