ci: add agent docs structure linter - #475
Conversation
Port hack/lint-agent-docs from fullsend-ai/fullsend, adapted for this repo's layout (harness/ and docs/ paths, "Configuration" instead of "Configuration and extension"). The linter checks: - Every harness YAML has a doc: field pointing to an existing file - Agent doc H1 ends with " Agent" - Required sections are present (How it helps, Triggers, Commands, Control labels, Configuration, How the agent works, Source) - ### Variables subsection exists under ## Configuration Also adds missing Commands, Control labels sections and renames Environment variables → Variables in docs/scribe.md to pass the linter. Integrated via pre-commit hook, which is already run by CI. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>
PR Summary by QodoCI: add agent docs reference/structure linter via pre-commit
AI Description
Diagram
High-Level Assessment
Files changed (3)
|
|
🤖 Finished Review · ✅ Success · Started 6:50 PM UTC · Completed 7:06 PM UTC |
Code Review by Qodo
Context used✅ Compliance rules (platform):
55 rules 1. Protected paths modified in PR
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
| - id: lint-agent-docs | ||
| name: lint agent doc references and structure | ||
| entry: ./hack/lint-agent-docs | ||
| language: script | ||
| files: ^(harness/|docs/) | ||
| pass_filenames: false |
There was a problem hiding this comment.
1. Protected paths modified in pr 📜 Skill insight § Compliance
This PR modifies protected governance/infrastructure paths (.pre-commit-config.yaml and hack/). Per the compliance checklist, changes under protected paths must be explicitly flagged for required human review and must not be auto-approved.
Agent Prompt
## Issue description
This PR changes protected governance/infrastructure files, which require explicit authorization/justification and mandatory human review signals.
## Issue Context
Protected paths include `.pre-commit-config.yaml` and `hack/`. The PR description contains a summary, but it does not include an explicit linked issue/ADR reference that authorizes the governance/tooling change.
## Fix Focus Areas
- .pre-commit-config.yaml[49-54]
- hack/lint-agent-docs[1-6]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| #!/bin/bash | ||
| # Lint: every harness YAML must have a doc: field pointing to an existing file. | ||
| # Also checks that agent docs follow a consistent structure. | ||
| set -euo pipefail | ||
| shopt -s nullglob | ||
|
|
||
| REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)" | ||
| HARNESS_DIR="$REPO_ROOT/harness" | ||
|
|
||
| errors=0 | ||
|
|
||
| echo "Checking harness files for doc: field..." | ||
| echo "================================================" | ||
|
|
||
| for yaml_file in "$HARNESS_DIR"/*.yaml; do | ||
| name="$(basename "$yaml_file")" | ||
| doc_value="$(grep -E '^doc:' "$yaml_file" | sed 's/^doc:[[:space:]]*//' || true)" | ||
|
|
||
| if [[ -z "$doc_value" ]]; then | ||
| echo " ERROR: $name: missing 'doc:' field" |
There was a problem hiding this comment.
2. No linked issue authorization 📜 Skill insight § Compliance
This is a non-trivial change (new linter script and pre-commit integration) but the PR description does not link to an authorizing issue. The compliance checklist requires an explicit linked issue for non-trivial work.
Agent Prompt
## Issue description
The PR introduces a new CI/pre-commit linter script (non-trivial change) but lacks a linked issue that authorizes the work.
## Issue Context
The compliance rule requires that non-trivial changes (20+ lines / structural changes) include an explicit linked issue in the PR description.
## Fix Focus Areas
- hack/lint-agent-docs[1-139]
- .pre-commit-config.yaml[49-54]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| # Check that the top-level heading ends with " Agent" | ||
| h1="$(awk 'BEGIN{f=0} /^```/{f=1-f; next} f==0 && /^# [^#]/{sub(/^# /,""); print; exit}' "$doc_path")" | ||
| if [[ -n "$h1" ]] && [[ "$h1" != *" Agent" ]]; then | ||
| echo " $doc_basename: top heading \"# $h1\" must end with \" Agent\" (e.g. \"# ${h1} Agent\")" | ||
| errors=$((errors + 1)) | ||
| fi |
There was a problem hiding this comment.
3. Missing h1 not enforced 🐞 Bug ≡ Correctness
hack/lint-agent-docs only validates the H1 suffix when an H1 was found, so a doc with no top-level "# ..." heading silently passes this check. This weakens the linter’s ability to enforce consistent agent doc structure in CI.
Agent Prompt
### Issue description
The H1 validation currently only runs when `h1` is non-empty (`[[ -n "$h1" ]]`), which means documents that omit a top-level `# ...` heading will not fail lint.
### Issue Context
This script is now run via pre-commit/CI, so missing H1s should be treated as lint errors to match the intended documentation structure requirements.
### Fix
Treat an empty `h1` as an error (increment `errors`) and print a clear message (e.g., `missing top-level "# ... Agent" heading`). Then keep the existing suffix check for non-empty H1s.
### Fix Focus Areas
- hack/lint-agent-docs[53-58]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
|
Superseded by #476, which adds the Setup section and strict extra-section enforcement. |
|
🤖 Finished Retro · ✅ Success · Started 7:05 PM UTC · Completed 7:16 PM UTC |
|
Review skipped — this PR is already closed. The Posted by fullsend post-review check |
Retro: PR #475 — ci: add agent docs structure linterPR #475 was a human-authored PR by ralphbean that ported a docs structure linter ( Timeline
Key observationsReview agent findings were never posted. The review agent completed a full analysis with a Qodo caught a correctness bug the review agent missed. Qodo identified that the linter's H1 validation only runs when an H1 heading is found — documents with no H1 silently pass. The review agent's correctness sub-agent (Claude Opus) found 2 LOW-severity items about the shell script but did not flag this validation logic gap. This same bug persists in successor PR #476. See proposal below. Security dimension dispatched with minimal surface. The review agent dispatched a Security sub-agent (Claude Opus) for a CI/linting PR and it returned 0 findings. This provides evidence for #373 (skip irrelevant review dimensions to reduce token cost). Workflow assessmentThis was a lightweight workflow: a human-authored PR received automated reviews, was quickly superseded, and the review agent's findings were never posted. The review agent's behavior was operationally correct (it properly identified protected paths, requested changes, and the post-review script correctly handled the closed state). The gap is in the correctness sub-agent's detection coverage — it missed a real bug that Qodo caught. Proposals filed
|
1 participant
Summary
hack/lint-agent-docsfrom fullsend-ai/fullsend, adapted for this repo'sharness/anddocs/layoutpre-commit run --all-filesstep in lint.yml)docs/scribe.mdto pass: adds missing## Commands/## Control labelsstubs, renames### Environment variables→### VariablesWhat the linter checks
harness/*.yamlhas adoc:field pointing to an existing file# H1ends with " Agent"##sections: How it helps, Triggers, Commands, Control labels, Configuration, How the agent works, Source### Variablessubsection under## ConfigurationTest plan
./hack/lint-agent-docspassespre-commit run lint-agent-docs --all-filespassespre-commit run --all-filespasses (full suite)🤖 Generated with Claude Code