Skip to content

Add Agents.md evaluation skill, refine service agent guidance, and resolve merge conflicts#1377

Draft
saratpoluri wants to merge 7 commits into
mainfrom
fix/agents-evaluation
Draft

Add Agents.md evaluation skill, refine service agent guidance, and resolve merge conflicts#1377
saratpoluri wants to merge 7 commits into
mainfrom
fix/agents-evaluation

Conversation

@saratpoluri
Copy link
Copy Markdown
Contributor

@saratpoluri saratpoluri commented May 1, 2026

📝 Description

This PR standardizes and improves service-level Agents.md guidance, adds an agent_evaluation skill for scoring and efficacy testing of Agents.md, updates Copilot instruction routing, and resolves merge conflicts with main while preserving the PR’s intent.

What was changed:

  • Reworked multiple service Agents.md files to be more concise and KPI/constraint-driven, including standardized verification guidance.
  • Added .github/skills/agent_evaluation/ with:
    • SKILL.md for evaluation flow and efficacy procedure
    • agents-md-evaluation.md for rubric-based scoring
  • Updated .github/copilot-instructions.md to:
    • rename the skills section to reflect non-language-only skills
    • avoid duplicating rubric details already defined in SKILL.md
    • use updated .github/skills/.../SKILL.md path references consistently
  • Merged latest main and resolved conflicts in affected Agents.md and Copilot instruction files to keep the refined rubric-driven guidance intact.

Why:

  • To improve consistency and quality of service agent instructions.
  • To provide a repeatable evaluation method for Agents.md.
  • To address review feedback and keep instruction sources canonical.
  • To ensure the branch is up to date and conflict-free without changing the intended scope of this PR.

How this adds value:

  • Improves maintainability and usability of service guides.
  • Enables structured, evidence-based evaluation of Agents.md impact.
  • Reduces duplication and ambiguity in Copilot instruction routing.
  • Keeps the PR mergeable with main while retaining all intended guidance refinements.

✨ Type of Change

Select the type of change your PR introduces:

  • 🐞 Bug fix – Non-breaking change which fixes an issue
  • 🚀 New feature – Non-breaking change which adds functionality
  • 🔨 Refactor – Non-breaking change which refactors the code base
  • 💥 Breaking change – Changes that break existing functionality
  • 📚 Documentation update
  • 🔒 Security update
  • 🧪 Tests
  • 🚂 CI

🧪 Testing Scenarios

Describe how the changes were tested and how reviewers can test them too:

  • ✅ Tested manually
  • 🤖 Ran automated end-to-end tests

✅ Checklist

Before submitting the PR, ensure the following:

  • 🔍 PR title is clear and descriptive
  • 📝 For internal contributors: If applicable, include the JIRA ticket number (e.g., ITEP-123456) in the PR title. Do not include full URLs
  • 💬 I have commented my code, especially in hard-to-understand areas
  • 📄 I have made corresponding changes to the documentation
  • ✅ I have added tests that prove my fix is effective or my feature works

@saratpoluri saratpoluri requested review from Copilot and daddo-intel May 1, 2026 20:43
@saratpoluri saratpoluri marked this pull request as draft May 1, 2026 20:46
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR standardizes and streamlines service-level Agents.md guides across multiple SceneScape microservices, adds a new rubric + efficacy-testing skill for evaluating Agents.md quality, and updates Copilot instructions to reference the new evaluation skill.

Changes:

  • Added “Verification Gate (Standardized)” sections with command paths and pass criteria to multiple Agents.md guides.
  • Rewrote several service Agents.md files to be more concise and KPI/constraint-driven.
  • Introduced a new .github/skills/agent_evaluation/ skill (rubric + efficacy test procedure) and linked it from .github/copilot-instructions.md.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
tracker/Agents.md Adds a standardized verification gate table for tracker changes.
tools/tracker/evaluation/Agents.md Adds a standardized verification gate table for evaluation tooling.
mapping/Agents.md Rewrites mapping agent guide into a concise format and adds verification gate commands.
manager/Agents.md Rewrites manager agent guide into a concise format and adds verification gate commands.
controller/Agents.md Rewrites controller agent guide into a concise format and adds verification gate commands.
cluster_analytics/Agents.md Rewrites cluster analytics agent guide into a concise format and adds verification gate commands.
autocalibration/Agents.md Rewrites autocalibration agent guide into a concise format and adds verification gate commands.
.github/skills/agent_evaluation/agents-md-evaluation.md Adds a scoring rubric and required JSON output format for evaluating Agents.md.
.github/skills/agent_evaluation/SKILL.md Adds the evaluation skill entry and an efficacy-testing procedure.
.github/copilot-instructions.md References the new agent evaluation skill and adds an on-demand loading trigger.

Comment thread autocalibration/Agents.md Outdated
Comment thread mapping/Agents.md Outdated
Comment thread manager/Agents.md Outdated
Comment thread controller/Agents.md Outdated
Comment thread cluster_analytics/Agents.md Outdated
saratpoluri and others added 3 commits May 1, 2026 14:42
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Comment thread .github/copilot-instructions.md Outdated
Comment thread .github/copilot-instructions.md
Comment thread .github/copilot-instructions.md Outdated
@saratpoluri
Copy link
Copy Markdown
Contributor Author

@copilot address code review comments

Copy link
Copy Markdown
Contributor

Copilot AI commented Jun 5, 2026

@copilot address code review comments

Addressed in commit 1938651: updated .github/copilot-instructions.md to fix the skills section heading, remove redundant Agents.md evaluation expansion, and correct skill file path references.

Copilot AI changed the title Add agents evaluation and rewrite agents based on the evaluation Add Agents.md evaluation skill and refine service agent guidance Jun 5, 2026
@saratpoluri
Copy link
Copy Markdown
Contributor Author

@copilot resolve the merge conflicts in this pull request. Understand the intent of the changes we made in this PR. We evaluated the Agents.md instructions against the rubric we created, scored it against that and refined it.

Copy link
Copy Markdown
Contributor

Copilot AI commented Jun 5, 2026

@copilot resolve the merge conflicts in this pull request. Understand the intent of the changes we made in this PR. We evaluated the Agents.md instructions against the rubric we created, scored it aga...

Resolved the merge conflicts by merging origin/main into this branch and preserving the intent of this PR’s refined, rubric-driven Agents.md updates. Addressed in commit c9a6ff5.

Copilot AI changed the title Add Agents.md evaluation skill and refine service agent guidance Add Agents.md evaluation skill, refine service agent guidance, and resolve merge conflicts Jun 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants