Skip to content

[TypeSpec Authoring][Benchmark] Add checkExistence(HEAD) operation to existing resource case#15805

Open
JoyerJin wants to merge 2 commits into
mainfrom
joyer-benchmark-checkExistence
Open

[TypeSpec Authoring][Benchmark] Add checkExistence(HEAD) operation to existing resource case#15805
JoyerJin wants to merge 2 commits into
mainfrom
joyer-benchmark-checkExistence

Conversation

@JoyerJin
Copy link
Copy Markdown
Contributor

@JoyerJin JoyerJin commented Jun 1, 2026

No description provided.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new benchmark stimulus (002011-arm-add-check-existence-operation) that exercises the azure-typespec-author skill's ability to add a checkExistence (HEAD) operation to an existing ARM resource, registers it in the armtemplate suite, and provides both a standalone eval file and an entry in the consolidated eval.yaml.

Changes:

  • New standalone eval evals/002011.eval.yaml defining the stimulus, fixtures, graders (tool-calls, skill-invocation, file-matches on ArmResourceCheckExistence<Employee>, prompt) and scoring weights.
  • Added the same stimulus to the aggregated evals/eval.yaml under the armtemplate tag.
  • Wired the new eval into .vally.yaml's armtemplate suite list.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
.github/skills/azure-typespec-author/evaluate/evals/002011.eval.yaml New standalone Vally eval for the checkExistence stimulus.
.github/skills/azure-typespec-author/evaluate/evals/eval.yaml Adds the same stimulus to the aggregated multi-stimulus eval.
.github/skills/azure-typespec-author/evaluate/.vally.yaml Registers the new eval file in the armtemplate suite.

Comment thread .github/skills/azure-typespec-author/evaluate/evals/002011.eval.yaml Outdated
Comment thread .github/skills/azure-typespec-author/evaluate/evals/eval.yaml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants