Skip to content

Add comment audit workflow#1031

Open
LasmarKhalifa wants to merge 1 commit into
mainfrom
07-10/doc-comment-audit-workflow
Open

Add comment audit workflow#1031
LasmarKhalifa wants to merge 1 commit into
mainfrom
07-10/doc-comment-audit-workflow

Conversation

@LasmarKhalifa

@LasmarKhalifa LasmarKhalifa commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Part of #1008

In a nutshell: a Roast workflow that guards against doc-comment and inline comment drift. It reads the comments across the codebase (or any passed target), has an agent check each one against the real implementation, and opens a single PR fixing the stale ones. Read-only by default; it opens a PR only when run with -- apply.

How it works (one cog per phase):

  1. Enumerate parses each in-scope file with Prism and extracts atomic units: each contiguous # prose block sitting above a definition (class, module, method, attr, constant), plus its RBS signature, as well as inline comments.
  2. Verify fans out across units in parallel, one agent per unit (Claude Sonnet), judging the comment against the actual code and returning a structured verdict (factual? confidence? suggested fix?).
  3. Plan classifies each verdict: factual, stale-and-auto-fixable (confident with a safe fix), or low-confidence (report only). (Current threshold is set at 0.8, but this should be adjusted over time).
  4. Apply (only with -- apply) splices the fixes in and opens one draft PR on top of the current branch, gated by a check that requires a clean, pushed, even working tree so the PR diff is only the doc fix.
  5. Summary prints the findings.

Scope defaults to lib/roast plus the rbi shims; pass files or directories as targets to narrow it.

The workflow is heavily commented inline, so that a reader can follow the enumeration and classification logic without having to reverse-engineer it.

Example test run scoped to one file (lib/roast/cog/output.rb) here: https://app.graphite.com/github/pr/Shopify/roast/1037

LasmarKhalifa commented Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

@LasmarKhalifa LasmarKhalifa marked this pull request as ready for review July 10, 2026 14:51
#
# Examples:
# # dry-run the full default scope (prints findings, opens nothing, writes nothing):
# bin/roast execute internal/workflows/doc_audit/workflow.rb

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Usage examples reference wrong file path. All four bin/roast execute examples omit the maintenance/ segment — they say internal/workflows/doc_audit/workflow.rb but the file is at internal/workflows/maintenance/doc_audit/workflow.rb. The CLI's resolve_workflow_path will fail with "Workflow file not found" if a user copies any of these commands.

@LasmarKhalifa LasmarKhalifa Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch, I had originally created the workflow in /maintenance, then moved it to /maintenance/doc_audit but forgot to change the examples.

Note: I changed the name of the folder to comment_audit to reflect the fact that it does not just audit doc-comments (I updated it to handle inline comments as well as you had a good point here.

Thanks for the comment!

@LasmarKhalifa LasmarKhalifa force-pushed the 07-10/doc-comment-audit-workflow branch from a5f0059 to 2c10411 Compare July 14, 2026 22:19
@LasmarKhalifa LasmarKhalifa changed the title Add doc-comment audit workflow Add comment audit workflow Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants