ARM API Reviewer agent: automate review trigger on PR open and on-demand from the author

[ravimeda]

Background

Today the ARM API Reviewer agent is manually triggered by a reviewer from VS Code Copilot Chat against a specific PR. This means:

• Authors get no automatic feedback when they open a PR — they must wait for a human reviewer to run the agent.
• Reviewers must remember to run the agent on every assigned PR, which doesn''t scale.
• There is no self-service way for an author to request a fresh review after pushing fixes without pinging a reviewer.

The repo already has the supporting infrastructure for an automated trigger:

• .github/copilot-review-instructions.md wires GitHub Copilot Code Review for inline PR comments.
• .github/workflows/sdk-generation-agent.md is a precedent for a GitHub Actions–driven coding agent.
• The agent already emits a hidden posted-by: arm-api-reviewer-agent marker on every comment, which enables repeat-run reconciliation under a stable bot identity.

We need to design and ship a server-side trigger so reviews happen automatically on PR open and on demand from the author.

References

• Agent definition — .github/agents/arm-api-reviewer.agent.md
• User docs — documentation/api-reviewer-agent.md
• Copilot Code Review instructions — .github/copilot-review-instructions.md
• Existing GH Actions agent precedent — .github/workflows/sdk-generation-agent.md
• GitHub docs — Copilot code review
• Eval suite that must gate any agent change — .github/skills/evals/arm-api-reviewer/

Requested work

1. Pick a trigger model. Evaluate and choose a combination of:
   • pull_request events (opened, ready_for_review, optionally synchronize with debouncing).
   • issue_comment slash-command (e.g., /arm-review) so authors and reviewers can re-trigger on demand.
   • pull_request labeled event (arm-review-requested / skip-arm-review) as an opt-in/opt-out escape hatch.
2. Add a workflow at .github/workflows/arm-api-review.yaml that:
   • Runs only when specification/** files changed.
   • Skips drafts unless explicitly invoked via /arm-review.
   • Uses pull_request_target safely (no checkout of untrusted code with elevated permissions; read PR files via the GitHub API only).
   • Invokes the agent runtime using the existing arm-api-reviewer.agent.md system prompt and the instruction files under .github/instructions/.
   • Posts findings under a stable bot identity (GitHub App or azure-sdk bot) so the existing reconciliation marker logic (Scenarios A–E) continues to work.
   • Applies label changes (ARMChangesRequested, removes WaitForARMFeedback) per the agent''s Step 8.
3. Decide the model host (GitHub Models / Copilot in Actions, internal Azure OpenAI, or a hosted alternative) and document the secret-management story.
4. Add safety rails:
   • Debounce on synchronize (coalesce to the latest push).
   • Hard cap on changed-file count and on comments-per-run; spill the rest into a single summary comment linking a full report artifact / gist.
   • Permission check on /arm-review (collaborators and the PR author only).
   • skip-arm-review label as an explicit opt-out.
5. Wire the eval suite as a gate on changes to the agent, instruction files, or workflow so rule regressions cannot ship.
6. Document the new triggers in documentation/api-reviewer-agent.md — including the /arm-review command, the labels, and how to opt out.
7. Phased rollout:
   • Phase 1: /arm-review slash-command behind a small service allowlist (low risk, opt-in only).
   • Phase 2: Auto on pull_request: [opened, ready_for_review] for allowlisted services.
   • Phase 3: Full auto including synchronize with debouncing and the label-based opt-in/opt-out.

Acceptance criteria

• Opening a PR that touches specification/** in an allowlisted service triggers the ARM API Reviewer agent automatically and posts findings as PR review comments under a stable bot identity.
• A PR author or collaborator can type /arm-review as a PR comment to (re-)trigger a review on demand.
• Repeat runs do not duplicate comments — the existing posted-by: arm-api-reviewer-agent reconciliation logic (Scenarios A–E) works end-to-end without a human in the loop.
• Drafts, PRs with skip-arm-review, and PRs with no specification/** changes are skipped.
• The eval suite at .github/skills/evals/arm-api-reviewer/ runs as a required check on PRs that modify the agent, instruction files, skills, or the new workflow.
• documentation/api-reviewer-agent.md documents the triggers, the slash command, the labels, and the opt-out path.