feat: scheduled full-repo documentation drift audit - #72
Conversation
A PR diff is a delta; documentation describes end state. Per-PR triggering catches incremental drift but misses accumulated staleness. Add an audit mode for schedule triggers that walks the docs tree using folder indexes, assesses each doc's staleness via LLM, and reports findings as a single updated GitHub Issue (never a new issue per run). Budget cap bounds cost on large repos and reports what was skipped.
|
🤖 Finished Review · ✅ Success · Started 5:55 AM UTC · Completed 6:16 AM UTC Commit: |
ReviewFindingsMedium
Low
Next steps:
|
![fullsend-ai-review[bot]](https://avatars.githubusercontent.com/in/3478501?s=60&v=4){kind=small}
| text = (response.choices[0].message.content or "").strip() | ||
|
|
||
| for severity in ("VERY-STALE", "STALE", "FRESH"): | ||
| if severity in text.upper(): |
There was a problem hiding this comment.
[medium] logic-error
Severity parsing uses substring matching (severity in text.upper()), causing false positives. An LLM response like 'FRESH: docs are not stale' matches STALE before FRESH due to iteration order (VERY-STALE, STALE, FRESH), misclassifying fresh documents as stale.
Suggested fix: Use text.upper().startswith(severity) or a regex to match only the leading severity token.
| return "\n".join(lines) | ||
|
|
||
|
|
||
| def post_audit_issue(findings, repo=None): |
There was a problem hiding this comment.
[medium] api-contract
post_audit_issue() accepts a repo parameter but never passes --repo to gh CLI commands. After setup_docs_environment() changes the working directory to a cloned docs repo, gh infers the repo from the current directory's git remote, creating issues in the wrong repository in separate-docs-repo mode.
Suggested fix: Pass --repo repo to each gh command (gh label create, gh issue list, gh issue edit, gh issue create).
| description: 'Path to a Markdown style configuration file (.md) containing documentation style guidelines. If not set, auto-detects .code-to-docs/style.md in the repository root.' | ||
| required: false | ||
| default: '' | ||
| mode: |
There was a problem hiding this comment.
[medium] scope-coherence
The mode input description advertises three values (comment, detect-only, audit) but detect-only is completely unimplemented. A user setting mode: detect-only gets silent fallthrough to comment behavior, which may fail in a scheduled workflow context.
Suggested fix: Remove detect-only from the description until implemented, or add validation for unrecognized mode values.
| return severity.lower(), reason | ||
|
|
||
| return "stale", text[:200] | ||
| except Exception as e: |
There was a problem hiding this comment.
[medium] error-handling-pattern
The _assess_doc() except block does not call check_context_error(e). Every other LLM API call handler in the codebase (discovery.py, doc_index.py, generation.py, jira_integration.py) calls this function to surface actionable context-window errors.
Suggested fix: Import check_context_error from config and call it before the print statement in the except block.
| if not setup_docs_environment(): | ||
| print("Failed to set up docs environment") | ||
| return | ||
| budget = int(os.environ.get("AUDIT_BUDGET", "20")) |
There was a problem hiding this comment.
[low] error-handling-gap
AUDIT_BUDGET env var parsed with int() without error handling. A non-numeric value causes an unhandled ValueError, inconsistent with how MAX_CONTEXT_CHARS is handled in config.py.
Suggested fix: Wrap in try/except ValueError with fallback to 20.
| return | ||
|
|
||
| # Handle audit mode (scheduled full-repo drift check) | ||
| mode = os.environ.get("MODE", "comment") |
There was a problem hiding this comment.
[low] input-validation
MODE env var read without validation against an allowlist. Unrecognized values silently fall through to comment mode.
Suggested fix: Validate MODE is one of the expected values and warn on unrecognized input.
| from security_utils import run_command_safe, sanitize_output | ||
|
|
||
|
|
||
| def run_audit(max_files=20): |
There was a problem hiding this comment.
[low] input-validation
max_files parameter has no bounds validation. Zero or negative values silently skip all files; very large values could cause excessive LLM API calls.
Suggested fix: Clamp max_files to a reasonable range (e.g. 1-500).
| @@ -0,0 +1,194 @@ | |||
| """Scheduled full-repo documentation drift audit. | |||
There was a problem hiding this comment.
[low] module-docstring-format
Module docstring does not follow the codebase pattern of structured 'This module handles:' inventory used by peer modules.
|
|
||
| import json | ||
| import os | ||
|
|
There was a problem hiding this comment.
[low] import-organization
Imports lack section comments (# Import configuration, etc.) used in peer modules like discovery.py and generation.py.
1 participant
Summary
Adds a scheduled audit mode that walks the entire docs tree and identifies stale documentation.
mode: auditforscheduletriggersaudit-budgetinput bounds cost on large repos (default 20 files); reports what was skippedsrc/audit.pymoduleTest plan
uv run pytest -vpasses (420 tests)