Add agent tooling: linter rule scaffold, validate:pr, and skill#4469
Open
markcowl wants to merge 6 commits into
Open
Add agent tooling: linter rule scaffold, validate:pr, and skill#4469markcowl wants to merge 6 commits into
markcowl wants to merge 6 commits into
Conversation
markcowl
added
the
agentic-workflows
Collaborator
|
No changes needing a change description found. |
Contributor
⚡ Benchmark Results
Full details – comparing
|
| Metric | Baseline | Current | Change |
|---|---|---|---|
| total | 🔴 714.1ms | 🔴 719.6ms | +0.8% |
| loader | 🟢 138.4ms | 🟢 138.7ms | +0.2% |
| resolver | 🟢 16.1ms | 🟢 17.3ms | +7.4% 🔴 |
| checker | 🟢 167.8ms | 🟢 167.4ms | -0.2% |
| validation | 🟢 38.9ms | 🟢 39.6ms | +1.7% |
| ↳ validation/@azure-tools/typespec-azure-core | 🟢 5.5ms | 🟢 5.5ms | +1.3% |
| ↳ validation/@typespec/http | 🟢 5.1ms | 🟢 4.8ms | -5.1% |
| ↳ validation/@typespec/rest | 🟢 0.5ms | 🟢 0.5ms | -4.1% |
| ↳ validation/@typespec/versioning | 🔴 26.1ms | 🔴 26.6ms | +2.1% |
| ↳ validation/compiler | 🟢 1.4ms | 🟢 1.5ms | +5.8% |
| linter | 🟢 116.4ms | 🟢 118.0ms | +1.3% |
| ↳ linter/@azure-tools/typespec-azure-core/auth-required | 🟢 0.0ms | 🟢 0.0ms | +3.4% |
| ↳ linter/@azure-tools/typespec-azure-core/bad-record-type | 🟢 0.2ms | 🟢 0.2ms | -5.7% |
| ↳ linter/@azure-tools/typespec-azure-core/byos | 🟢 5.1ms | 🟢 5.7ms | +10.0% |
| ↳ linter/@azure-tools/typespec-azure-core/casing-style | 🟢 0.5ms | 🟢 0.5ms | -0.7% |
| ↳ linter/@azure-tools/typespec-azure-core/composition-over-inheritance | 🟢 0.1ms | 🟢 0.1ms | -2.0% |
| ↳ linter/@azure-tools/typespec-azure-core/documentation-required | 🟢 0.7ms | 🟢 0.8ms | +1.1% |
| ↳ linter/@azure-tools/typespec-azure-core/friendly-name | 🟢 0.5ms | 🟢 0.5ms | -1.3% |
| ↳ linter/@azure-tools/typespec-azure-core/key-visibility-required | 🟢 0.1ms | 🟢 0.1ms | +3.2% |
| ↳ linter/@azure-tools/typespec-azure-core/known-encoding | 🟢 0.2ms | 🟢 0.2ms | -6.2% |
| ↳ linter/@azure-tools/typespec-azure-core/long-running-polling-operation-required | 🟢 0.3ms | 🟢 0.3ms | +2.3% |
| ↳ linter/@azure-tools/typespec-azure-core/no-case-mismatch | 🟢 0.2ms | 🟢 0.2ms | +10.7% |
| ↳ linter/@azure-tools/typespec-azure-core/no-closed-literal-union | 🟢 0.2ms | 🟢 0.2ms | +6.1% |
| ↳ linter/@azure-tools/typespec-azure-core/no-enum | 🟢 0.0ms | 🟢 0.0ms | +3.0% |
| ↳ linter/@azure-tools/typespec-azure-core/no-error-status-codes | 🟢 0.1ms | 🟢 0.1ms | -0.4% |
| ↳ linter/@azure-tools/typespec-azure-core/no-explicit-routes-resource-ops | 🟢 0.1ms | 🟢 0.1ms | +2.5% |
| ↳ linter/@azure-tools/typespec-azure-core/no-format | 🟢 0.4ms | 🟢 0.4ms | -2.0% |
| ↳ linter/@azure-tools/typespec-azure-core/no-generic-numeric | 🟢 0.4ms | 🟢 0.4ms | +9.0% |
| ↳ linter/@azure-tools/typespec-azure-core/no-header-explode | 🟡 16.0ms | 🟡 16.4ms | +3.0% |
| ↳ linter/@azure-tools/typespec-azure-core/no-legacy-usage | 🟢 1.0ms | 🟢 1.0ms | +0.5% |
| ↳ linter/@azure-tools/typespec-azure-core/no-multiple-discriminator | 🟢 0.1ms | 🟢 0.1ms | +1.3% |
| ↳ linter/@azure-tools/typespec-azure-core/no-nullable | 🟢 0.2ms | 🟢 0.2ms | +3.5% |
| ↳ linter/@azure-tools/typespec-azure-core/no-offsetdatetime | 🟢 1.1ms | 🟢 1.2ms | +3.3% |
| ↳ linter/@azure-tools/typespec-azure-core/no-openapi | 🟢 1.7ms | 🟢 1.7ms | +0.6% |
| ↳ linter/@azure-tools/typespec-azure-core/no-private-usage | 🟢 1.7ms | 🟢 1.7ms | +2.4% |
| ↳ linter/@azure-tools/typespec-azure-core/no-query-explode | 🟡 16.8ms | 🟡 16.5ms | -1.8% |
| ↳ linter/@azure-tools/typespec-azure-core/no-response-body | 🔴 21.4ms | 🔴 20.3ms | -4.9% |
| ↳ linter/@azure-tools/typespec-azure-core/no-rest-library-interfaces | 🟢 0.0ms | 🟢 0.0ms | +7.9% |
| ↳ linter/@azure-tools/typespec-azure-core/no-route-parameter-name-mismatch | 🟢 4.4ms | 🟢 4.4ms | +0.3% |
| ↳ linter/@azure-tools/typespec-azure-core/no-rpc-path-params | 🟢 0.2ms | 🟢 0.2ms | +9.4% |
| ↳ linter/@azure-tools/typespec-azure-core/no-string-discriminator | 🟢 0.0ms | 🟢 0.0ms | +1.1% |
| ↳ linter/@azure-tools/typespec-azure-core/no-unknown | 🟢 0.1ms | 🟢 0.2ms | +5.4% |
| ↳ linter/@azure-tools/typespec-azure-core/no-unnamed-union | 🟢 0.3ms | 🟢 0.3ms | +3.6% |
| ↳ linter/@azure-tools/typespec-azure-core/operation-missing-api-version | 🟢 0.2ms | 🟢 0.2ms | +8.4% |
| ↳ linter/@azure-tools/typespec-azure-core/request-body-problem | 🟢 0.2ms | 🟢 0.2ms | +7.0% |
| ↳ linter/@azure-tools/typespec-azure-core/require-versioned | 🟢 0.0ms | 🟢 0.0ms | +8.1% |
| ↳ linter/@azure-tools/typespec-azure-core/response-schema-problem | 🟡 19.0ms | 🟡 19.8ms | +3.9% |
| ↳ linter/@azure-tools/typespec-azure-core/rpc-operation-request-body | 🟢 0.3ms | 🟢 0.3ms | -0.6% |
| ↳ linter/@azure-tools/typespec-azure-core/spread-discriminated-model | 🟢 0.2ms | 🟢 0.2ms | +1.9% |
| ↳ linter/@azure-tools/typespec-azure-core/use-standard-names | 🟢 4.2ms | 🟢 4.3ms | +0.2% |
| ↳ linter/@azure-tools/typespec-azure-core/use-standard-operations | 🟢 0.1ms | 🟢 0.1ms | -4.9% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-common-types-version | 🟢 3.6ms | 🟢 3.6ms | +1.5% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-custom-resource-no-key | 🟢 0.1ms | 🟢 0.1ms | -6.7% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-custom-resource-usage-discourage | 🟢 0.1ms | 🟢 0.1ms | -6.7% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-delete-operation-response-codes | 🟢 4.6ms | 🟢 4.7ms | +1.2% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-no-path-casing-conflicts | 🟢 4.1ms | 🟢 4.1ms | -1.1% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-no-record | 🟢 0.3ms | 🟢 0.4ms | +3.7% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-post-operation-response-codes | 🟢 0.4ms | 🟢 0.4ms | +1.5% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-put-operation-response-codes | 🟢 0.0ms | 🟢 0.0ms | -8.5% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-action-no-segment | 🟢 0.2ms | 🟢 0.2ms | -4.2% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-duplicate-property | 🟢 0.1ms | 🟢 0.1ms | -8.6% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-interface-requires-decorator | 🟢 0.0ms | 🟢 0.0ms | +16.9% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-invalid-action-verb | 🟢 0.1ms | 🟢 0.1ms | +2.0% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-invalid-envelope-property | 🟢 0.1ms | 🟢 0.1ms | +0.1% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-invalid-version-format | 🟢 0.0ms | 🟢 0.0ms | -2.8% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-key-invalid-chars | 🟢 0.2ms | 🟢 0.2ms | +2.6% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-name-pattern | 🟢 0.0ms | 🟢 0.0ms | +3.6% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-operation | 🟢 0.1ms | 🟢 0.1ms | +3.9% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-operation-response | 🟢 4.1ms | 🟢 4.2ms | +1.4% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-patch | 🟢 0.3ms | 🟢 0.3ms | -2.4% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-path-segment-invalid-chars | 🟢 0.2ms | 🟢 0.2ms | +7.2% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-provisioning-state | 🟢 0.1ms | 🟢 0.1ms | -6.9% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/beyond-nesting-levels | 🟢 0.1ms | 🟢 0.1ms | -3.9% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/empty-updateable-properties | 🟢 0.1ms | 🟢 0.1ms | +2.7% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/improper-subscription-list-operation | 🟢 0.0ms | 🟢 0.0ms | +2.4% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/lro-location-header | 🟡 12.7ms | 🟡 12.2ms | -4.0% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/missing-operations-endpoint | 🟢 0.0ms | 🟢 0.0ms | +5.9% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/missing-x-ms-identifiers | 🟢 0.3ms | 🟢 0.3ms | +4.2% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/no-empty-model | 🟢 0.1ms | 🟢 0.1ms | +12.0% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/no-resource-delete-operation | 🟢 0.2ms | 🟢 0.1ms | -3.5% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/no-response-body | 🟡 18.8ms | 🟡 18.9ms | +0.7% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/patch-envelope | 🟢 0.1ms | 🟢 0.1ms | -4.7% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/resource-name | 🟢 0.1ms | 🟢 0.1ms | -0.5% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/secret-prop | 🟢 2.2ms | 🟢 2.4ms | +12.0% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/unsupported-type | 🟢 0.3ms | 🟢 0.4ms | +3.2% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/version-progression | 🟢 0.0ms | 🟢 0.0ms | +7.6% |
| ↳ linter/@azure-tools/typespec-client-generator-core/property-name-conflict | 🟢 0.9ms | 🟢 0.9ms | +0.2% |
| ↳ linter/@azure-tools/typespec-client-generator-core/require-client-suffix | 🟢 0.2ms | 🟢 0.2ms | +8.6% |
| emit | 🟡 232.2ms | 🟡 233.6ms | +0.6% |
| ↳ emit/@azure-tools/typespec-autorest | 🟢 144.4ms | 🟢 146.0ms | +1.1% |
| ↳ emit/@typespec/openapi3 | 🟢 129.9ms | 🟢 131.3ms | +1.1% |
| ↳ emit/@typespec/openapi3/compute | 🟢 115.7ms | 🟢 117.4ms | +1.5% |
| ↳ emit/@typespec/openapi3/write | 🟢 13.6ms | 🟢 14.0ms | +3.1% |
Averaged across 3 specs (azure-arm-resource-manager, azure-core-dataplane, azure-full).
Threshold: changes > ±5% are highlighted.
🟢 Fast · 🟡 Moderate (stages >200ms, rules >10ms) · 🔴 Slow (stages >400ms, rules >20ms)
Collaborator
|
You can try these changes here
|
Member
Author
|
/azp run typespec -azure - PR Tools |
|
No pipelines are associated with this pull request. |
markcowl
commented
May 26, 2026
- eng/scripts/create-linter-rule.ts: scaffolds rule file, test, docs, and linter.ts registration - eng/scripts/validate-pr.ts: general pre-PR validation (build, test, lint, format, cspell, regen-docs, changeset, diff) - .github/skills/create-linter-rule/SKILL.md: 9-step TDD workflow skill for Copilot agents - package.json: add create:linter-rule and validate:pr scripts Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Update SKILL.md: add Step 10 for int:azure-specs External Integration check - Add website/src/content/docs/docs/howtos/contributing/creating-linter-rules.md: comprehensive guide for creating linter rules (humans and AI agents) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add client-generator-core as third valid package for linter rules - Add package selection guidance (TCGC vs ARM vs data-plane) - Update External Integration with spec fix workflow: API-neutral fixes to main, suppressions to main, unreleased fixes to typespec-next - Update scaffold script, skill, and website docs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
New doc at website/src/content/docs/docs/howtos/contributing/ covering: - Detection determinism as the key criterion for rule creation - How linter diagnostics work (warnings, suppression, CI blocking) - Standalone vs hybrid vs AI-only approaches - Decision framework and anti-patterns - Rollout considerations for existing spec violations Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- SKILL.md: add keywords for skill triggering, equivalence class testing guidance, library type exclusion tests, TCGC rules in both rulesets, clarify agent label capability, remove severity option - create-linter-rule.ts: remove --severity option (all rules are warnings) - when-to-write-a-linting-rule.md: fix error description, add 4th table entry (semantic detection + unambiguous resolution) - creating-linter-rules.md: remove severity, add equivalence classes, library type tests, TCGC ruleset guidance, ext integration details - cspell.yaml: add 'incentivizes' to dictionary Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
markcowl
force-pushed
the
markcowl/agent-linter-rule-support
branch
from
210700c to
23ae934
Compare
| Generate all required files using the repo's scaffolding tool: | ||
|
|
||
| ```bash | ||
| pnpm create:linter-rule < rule-name > --package < azure-core | azure-resource-manager | client-generator-core > --description "<description>" |
Member
There was a problem hiding this comment.
just wondering if maintaining this scaffolding script is worth it? Would the AI not just be able to pattern match if we list the files to create?
| ## Step 8: CREATE CHANGESET | ||
|
|
||
| ```bash | ||
| pnpm change add |
Member
There was a problem hiding this comment.
can also use <packges...> --message "abc" and --kind to set the values, might work better than interactive cli
|
|
||
| New linter rules MUST NOT break existing Azure service specs. After pushing your PR: | ||
|
|
||
| 1. Apply the `int:azure-specs` label to the PR to trigger the External Integration check: `gh pr edit --add-label "int:azure-specs"` |
Member
There was a problem hiding this comment.
feel like we should add a policy to add this label automatically when rules are modified
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds three components to improve AI agent performance when creating linter rules:
1.
eng/scripts/create-linter-rule.ts(scaffold script)Generates all required files for a new linter rule:
Usage:
pnpm create:linter-rule <name> --package <azure-core|azure-resource-manager> --severity <warning|error>2.
eng/scripts/validate-pr.ts(pre-PR validation)General validation script for any code change (not linter-rule-specific):
Usage:
pnpm validate:pr [--fix] [--skip-build] [--skip-test]3.
.github/skills/create-linter-rule/SKILL.md9-step TDD workflow skill for Copilot agents creating linter rules. Covers identification, scaffolding, test-first development, implementation, ruleset registration, documentation, changeset creation, and final validation.
Testing
ode --check)create-linter-rule --dry-runproduces expected outputvalidate:pr --helpshows usageFull integration testing requires
pnpm install(not run in this environment).