Add Copilot Studio support to Maester#1441
Add Copilot Studio support to Maester#1441lnfernux wants to merge 16 commits intomaester365:mainfrom
Conversation
There was a problem hiding this comment.
Pull request overview
Adds Copilot Studio (Dataverse-backed) “AI Agent Security” coverage to Maester by introducing new MT.1113–MT.1122 tests, plus connection/docs updates to support authenticating and querying agent configuration from Dataverse.
Changes:
- Introduces
Get-MtAIAgentInfo(Dataverse OData retrieval + caching) and 10 newTest-MtAIAgent*security tests (MT.1113–MT.1122). - Extends
Connect-Maester/Disconnect-Maesterwith a newDataverseservice option and Az-context reuse behavior. - Adds Maester config + documentation updates for Dataverse setup and the new test pages/result details.
Reviewed changes
Copilot reviewed 42 out of 42 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| website/docs/tests/maester/MT.1113.md | New website test page for MT.1113 (broad sharing). |
| website/docs/tests/maester/MT.1114.md | New website test page for MT.1114 (missing/weak auth). |
| website/docs/tests/maester/MT.1115.md | New website test page for MT.1115 (risky HTTP config). |
| website/docs/tests/maester/MT.1116.md | New website test page for MT.1116 (email exfiltration). |
| website/docs/tests/maester/MT.1117.md | New website test page for MT.1117 (dormant agents). |
| website/docs/tests/maester/MT.1118.md | New website test page for MT.1118 (maker auth). |
| website/docs/tests/maester/MT.1119.md | New website test page for MT.1119 (hard-coded creds). |
| website/docs/tests/maester/MT.1120.md | New website test page for MT.1120 (MCP tools). |
| website/docs/tests/maester/MT.1121.md | New website test page for MT.1121 (missing instructions). |
| website/docs/tests/maester/MT.1122.md | New website test page for MT.1122 (orphaned ownership). |
| website/docs/sections/create-entra-app.md | Adds optional Dataverse permission/setup section for app registration. |
| website/docs/connect-maester/readme.md | Adds Dataverse connection guidance to Connect-Maester docs. |
| website/docs/connect-maester/connect-maester-advanced.md | Adds advanced Dataverse (SPN-style) guidance. |
| tests/maester-config.json | Adds DataverseEnvironmentUrl global setting + registers MT.1113–MT.1122 metadata. |
| tests/Maester/AIAgent/Test-AIAgentSecurity.Tests.ps1 | New Pester wrapper to run MT.1113–MT.1122. |
| powershell/public/maester/aiagent/Test-MtAIAgentBroadSharing.ps1 | New MT.1113 implementation. |
| powershell/public/maester/aiagent/Test-MtAIAgentBroadSharing.md | Result detail markdown for MT.1113. |
| powershell/public/maester/aiagent/Test-MtAIAgentNoAuthentication.ps1 | New MT.1114 implementation. |
| powershell/public/maester/aiagent/Test-MtAIAgentNoAuthentication.md | Result detail markdown for MT.1114. |
| powershell/public/maester/aiagent/Test-MtAIAgentRiskyHttpConfig.ps1 | New MT.1115 implementation. |
| powershell/public/maester/aiagent/Test-MtAIAgentRiskyHttpConfig.md | Result detail markdown for MT.1115. |
| powershell/public/maester/aiagent/Test-MtAIAgentEmailExfiltration.ps1 | New MT.1116 implementation. |
| powershell/public/maester/aiagent/Test-MtAIAgentEmailExfiltration.md | Result detail markdown for MT.1116. |
| powershell/public/maester/aiagent/Test-MtAIAgentDormant.ps1 | New MT.1117 implementation. |
| powershell/public/maester/aiagent/Test-MtAIAgentDormant.md | Result detail markdown for MT.1117. |
| powershell/public/maester/aiagent/Test-MtAIAgentAuthorAuthentication.ps1 | New MT.1118 implementation. |
| powershell/public/maester/aiagent/Test-MtAIAgentAuthorAuthentication.md | Result detail markdown for MT.1118. |
| powershell/public/maester/aiagent/Test-MtAIAgentHardCodedCredentials.ps1 | New MT.1119 implementation. |
| powershell/public/maester/aiagent/Test-MtAIAgentHardCodedCredentials.md | Result detail markdown for MT.1119. |
| powershell/public/maester/aiagent/Test-MtAIAgentMcpTools.ps1 | New MT.1120 implementation. |
| powershell/public/maester/aiagent/Test-MtAIAgentMcpTools.md | Result detail markdown for MT.1120. |
| powershell/public/maester/aiagent/Test-MtAIAgentMissingInstructions.ps1 | New MT.1121 implementation. |
| powershell/public/maester/aiagent/Test-MtAIAgentMissingInstructions.md | Result detail markdown for MT.1121. |
| powershell/public/maester/aiagent/Test-MtAIAgentOrphaned.ps1 | New MT.1122 implementation. |
| powershell/public/maester/aiagent/Test-MtAIAgentOrphaned.md | Result detail markdown for MT.1122. |
| powershell/public/Connect-Maester.ps1 | Adds Dataverse service option + Az-context reuse + token preflight attempt. |
| powershell/public/Disconnect-Maester.ps1 | Disconnects Az when Dataverse was used; adds error handling. |
| powershell/internal/Get-MtAIAgentInfo.ps1 | New Dataverse OData retrieval + mapping + caching for agent data. |
| powershell/internal/Clear-ModuleVariable.ps1 | Clears cached AIAgentInfo on reset. |
| powershell/Maester.psm1 | Adds AIAgentInfo to the session object. |
| powershell/Maester.psd1 | Exports the 10 new Test-MtAIAgent* functions. |
| build/aitools/test-metadata/maester-config.json | Adds Dataverse environment URL to test metadata sample config. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Clarified the threshold description for dormant agents.
Updated the description for clarity and corrected a grammatical error.
SamErde
added
enhancement
|
Thanks for the contribution, @lnfernux! I'm not able to test at this point but did do a quick review of the overall structure and it looks like you covered a lot! Something to consider with regards to the verbiage and labeling of this set of tests: The "Dataverse" refers to the enterprise data platform for Microsoft 365 as a whole. The description of/in these tests seems to equate "dataverse" with "Copilot Studio" and related components. What do you think about creating a little more clarity between the general dataverse tests and those that specifically pertain to Copilot Studio or AI-related features? |
Yeah, I agree to a certain degree. The tests are only for Copilot Studio as of now, but this PR also uses the dataverse API (https://learn.microsoft.com/en-us/power-apps/developer/data-platform/webapi/overview) so that's why it refers to that. I can make some adjustments to the test naming and labeling to reflect that the tests are for Copilot Studio, and the connection is via dataverse api. |
|
@SamErde I've done a basic rework, where I've moved all Dataverse references into the latter part. Basically now everything will be referred to as "Copilot Studio" tests, and refer to the dataverse api as the connection method, like "Copilot Studio (via dataverse API)". If you are not connected, error messages should still mention that the dataverse api url isn't configured (because that's still the API we are using). Hopefully that looks and reads a bit better. Quick edit: If you need an SPN with read access to my test environment and I can fix that for you if you want to test. |
Nice, those updates look good. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 42 out of 42 changed files in this pull request and generated 9 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if ([string]::IsNullOrEmpty($failedAgents)) { | ||
| $testResultMarkdown = "Well done. No dormant AI agents found (threshold: $thresholdDays days)." | ||
| } else { | ||
| $testResultMarkdown = "Found $($failedAgents.Count) AI agent(s) that have not been modified in over $thresholdDays days.`n`n%TestResult%" | ||
| $result = "| Agent Name | Environment | Last Modified | Last Published |`n" | ||
| $result += "| --- | --- | --- | --- |`n" | ||
| foreach ($agent in $failedAgents) { | ||
| $lastModified = if ($agent.LastModifiedTime) { ([datetime]$agent.LastModifiedTime).ToString("yyyy-MM-dd") } else { "Unknown" } | ||
| $lastPublished = if ($agent.LastPublishedTime) { ([datetime]$agent.LastPublishedTime).ToString("yyyy-MM-dd") } else { "Unknown" } | ||
| $result += "| $($agent.AIAgentName) | $($agent.EnvironmentId) | $lastModified | $lastPublished |`n" | ||
| } | ||
| $testResultMarkdown = $testResultMarkdown -replace "%TestResult%", $result | ||
| } | ||
|
|
||
| Add-MtTestResultDetail -Result $testResultMarkdown -Severity "Low" | ||
| return [string]::IsNullOrEmpty($failedAgents) | ||
| } |
There was a problem hiding this comment.
$failedAgents is a collection; [string]::IsNullOrEmpty($failedAgents) and returning it will not correctly indicate whether any agents were found. This can flip pass/fail results. Use $failedAgents.Count -eq 0 (or -not $failedAgents) instead, and return ($failedAgents.Count -eq 0).
| Test-MtAIAgentBroadSharing | Should -Be $true -Because "AI agents with 'Any' or 'Any multitenant' access control allow unauthenticated or cross-tenant access, increasing the risk of unauthorized data access and prompt injection." | ||
| } | ||
|
|
||
| # AI agents should require user authentication. | ||
| It "MT.1114: AI agents should require user authentication. See https://maester.dev/docs/tests/MT.1114" -Tag "MT.1114" { | ||
| Test-MtAIAgentNoAuthentication | Should -Be $true -Because "AI agents without authentication allow anonymous access, making them vulnerable to abuse, data exfiltration, and prompt injection attacks." | ||
| } | ||
|
|
||
| # AI agents should not have risky HTTP configurations. | ||
| It "MT.1115: AI agents should not have risky HTTP configurations. See https://maester.dev/docs/tests/MT.1115" -Tag "MT.1115" { | ||
| Test-MtAIAgentRiskyHttpConfig | Should -Be $true -Because "HTTP actions to non-standard ports or plain HTTP endpoints may indicate data exfiltration or misconfigured integrations." | ||
| } | ||
|
|
||
| # AI agents should not send email with AI-controlled inputs. | ||
| It "MT.1116: AI agents should not send email with AI-controlled inputs. See https://maester.dev/docs/tests/MT.1116" -Tag "MT.1116" { | ||
| Test-MtAIAgentEmailExfiltration | Should -Be $true -Because "Email-sending tools with AI-controlled inputs present a risk of data exfiltration to attacker-controlled addresses." | ||
| } | ||
|
|
||
| # Published AI agents should not be dormant. | ||
| It "MT.1117: Published AI agents should not be dormant. See https://maester.dev/docs/tests/MT.1117" -Tag "MT.1117" { | ||
| Test-MtAIAgentDormant | Should -Be $true -Because "Dormant published agents may have outdated configurations and continue to expose functionality without active maintenance." | ||
| } | ||
|
|
||
| # AI agents should not use author (maker) authentication for connections. | ||
| It "MT.1118: AI agents should not use author (maker) authentication for connections. See https://maester.dev/docs/tests/MT.1118" -Tag "MT.1118" { | ||
| Test-MtAIAgentAuthorAuthentication | Should -Be $true -Because "Agents using author (maker) authentication access external services with the maker's credentials, creating privilege escalation and separation of duties risks." | ||
| } | ||
|
|
||
| # AI agents should not have hard-coded credentials in topics. | ||
| It "MT.1119: AI agents should not have hard-coded credentials in topics. See https://maester.dev/docs/tests/MT.1119" -Tag "MT.1119" { | ||
| Test-MtAIAgentHardCodedCredentials | Should -Be $true -Because "Hard-coded credentials in agent topics can be extracted by prompt injection attacks and persist after key rotation." | ||
| } | ||
|
|
||
| # AI agents should not use MCP server tools without review. | ||
| It "MT.1120: AI agents should not use MCP server tools without review. See https://maester.dev/docs/tests/MT.1120" -Tag "MT.1120" { | ||
| Test-MtAIAgentMcpTools | Should -Be $true -Because "MCP tool integrations extend agents with arbitrary external capabilities and may introduce supply chain risks." | ||
| } | ||
|
|
||
| # AI agents with generative orchestration should have custom instructions. | ||
| It "MT.1121: AI agents with generative orchestration should have custom instructions. See https://maester.dev/docs/tests/MT.1121" -Tag "MT.1121" { | ||
| Test-MtAIAgentMissingInstructions | Should -Be $true -Because "Agents using generative orchestration without custom instructions rely on default LLM behavior, increasing prompt injection and off-topic response risk." | ||
| } | ||
|
|
||
| # AI agents should not have orphaned ownership. | ||
| It "MT.1122: AI agents should not have orphaned ownership. See https://maester.dev/docs/tests/MT.1122" -Tag "MT.1122" { | ||
| Test-MtAIAgentOrphaned | Should -Be $true -Because "Agents whose owners are all disabled or deleted cannot be maintained and may continue operating with outdated or insecure configurations." |
There was a problem hiding this comment.
These tests assert the function output directly, but the AI agent test functions return $null when data is unavailable (and record a skipped reason via Add-MtTestResultDetail). As written, $null | Should -Be $true will fail the run instead of producing a skipped test. Capture the result and only assert when it's non-null (consistent with other Maester test wrappers).
| Test-MtAIAgentBroadSharing | Should -Be $true -Because "AI agents with 'Any' or 'Any multitenant' access control allow unauthenticated or cross-tenant access, increasing the risk of unauthorized data access and prompt injection." | |
| } | |
| # AI agents should require user authentication. | |
| It "MT.1114: AI agents should require user authentication. See https://maester.dev/docs/tests/MT.1114" -Tag "MT.1114" { | |
| Test-MtAIAgentNoAuthentication | Should -Be $true -Because "AI agents without authentication allow anonymous access, making them vulnerable to abuse, data exfiltration, and prompt injection attacks." | |
| } | |
| # AI agents should not have risky HTTP configurations. | |
| It "MT.1115: AI agents should not have risky HTTP configurations. See https://maester.dev/docs/tests/MT.1115" -Tag "MT.1115" { | |
| Test-MtAIAgentRiskyHttpConfig | Should -Be $true -Because "HTTP actions to non-standard ports or plain HTTP endpoints may indicate data exfiltration or misconfigured integrations." | |
| } | |
| # AI agents should not send email with AI-controlled inputs. | |
| It "MT.1116: AI agents should not send email with AI-controlled inputs. See https://maester.dev/docs/tests/MT.1116" -Tag "MT.1116" { | |
| Test-MtAIAgentEmailExfiltration | Should -Be $true -Because "Email-sending tools with AI-controlled inputs present a risk of data exfiltration to attacker-controlled addresses." | |
| } | |
| # Published AI agents should not be dormant. | |
| It "MT.1117: Published AI agents should not be dormant. See https://maester.dev/docs/tests/MT.1117" -Tag "MT.1117" { | |
| Test-MtAIAgentDormant | Should -Be $true -Because "Dormant published agents may have outdated configurations and continue to expose functionality without active maintenance." | |
| } | |
| # AI agents should not use author (maker) authentication for connections. | |
| It "MT.1118: AI agents should not use author (maker) authentication for connections. See https://maester.dev/docs/tests/MT.1118" -Tag "MT.1118" { | |
| Test-MtAIAgentAuthorAuthentication | Should -Be $true -Because "Agents using author (maker) authentication access external services with the maker's credentials, creating privilege escalation and separation of duties risks." | |
| } | |
| # AI agents should not have hard-coded credentials in topics. | |
| It "MT.1119: AI agents should not have hard-coded credentials in topics. See https://maester.dev/docs/tests/MT.1119" -Tag "MT.1119" { | |
| Test-MtAIAgentHardCodedCredentials | Should -Be $true -Because "Hard-coded credentials in agent topics can be extracted by prompt injection attacks and persist after key rotation." | |
| } | |
| # AI agents should not use MCP server tools without review. | |
| It "MT.1120: AI agents should not use MCP server tools without review. See https://maester.dev/docs/tests/MT.1120" -Tag "MT.1120" { | |
| Test-MtAIAgentMcpTools | Should -Be $true -Because "MCP tool integrations extend agents with arbitrary external capabilities and may introduce supply chain risks." | |
| } | |
| # AI agents with generative orchestration should have custom instructions. | |
| It "MT.1121: AI agents with generative orchestration should have custom instructions. See https://maester.dev/docs/tests/MT.1121" -Tag "MT.1121" { | |
| Test-MtAIAgentMissingInstructions | Should -Be $true -Because "Agents using generative orchestration without custom instructions rely on default LLM behavior, increasing prompt injection and off-topic response risk." | |
| } | |
| # AI agents should not have orphaned ownership. | |
| It "MT.1122: AI agents should not have orphaned ownership. See https://maester.dev/docs/tests/MT.1122" -Tag "MT.1122" { | |
| Test-MtAIAgentOrphaned | Should -Be $true -Because "Agents whose owners are all disabled or deleted cannot be maintained and may continue operating with outdated or insecure configurations." | |
| $result = Test-MtAIAgentBroadSharing | |
| if ($null -ne $result) { | |
| $result | Should -Be $true -Because "AI agents with 'Any' or 'Any multitenant' access control allow unauthenticated or cross-tenant access, increasing the risk of unauthorized data access and prompt injection." | |
| } | |
| } | |
| # AI agents should require user authentication. | |
| It "MT.1114: AI agents should require user authentication. See https://maester.dev/docs/tests/MT.1114" -Tag "MT.1114" { | |
| $result = Test-MtAIAgentNoAuthentication | |
| if ($null -ne $result) { | |
| $result | Should -Be $true -Because "AI agents without authentication allow anonymous access, making them vulnerable to abuse, data exfiltration, and prompt injection attacks." | |
| } | |
| } | |
| # AI agents should not have risky HTTP configurations. | |
| It "MT.1115: AI agents should not have risky HTTP configurations. See https://maester.dev/docs/tests/MT.1115" -Tag "MT.1115" { | |
| $result = Test-MtAIAgentRiskyHttpConfig | |
| if ($null -ne $result) { | |
| $result | Should -Be $true -Because "HTTP actions to non-standard ports or plain HTTP endpoints may indicate data exfiltration or misconfigured integrations." | |
| } | |
| } | |
| # AI agents should not send email with AI-controlled inputs. | |
| It "MT.1116: AI agents should not send email with AI-controlled inputs. See https://maester.dev/docs/tests/MT.1116" -Tag "MT.1116" { | |
| $result = Test-MtAIAgentEmailExfiltration | |
| if ($null -ne $result) { | |
| $result | Should -Be $true -Because "Email-sending tools with AI-controlled inputs present a risk of data exfiltration to attacker-controlled addresses." | |
| } | |
| } | |
| # Published AI agents should not be dormant. | |
| It "MT.1117: Published AI agents should not be dormant. See https://maester.dev/docs/tests/MT.1117" -Tag "MT.1117" { | |
| $result = Test-MtAIAgentDormant | |
| if ($null -ne $result) { | |
| $result | Should -Be $true -Because "Dormant published agents may have outdated configurations and continue to expose functionality without active maintenance." | |
| } | |
| } | |
| # AI agents should not use author (maker) authentication for connections. | |
| It "MT.1118: AI agents should not use author (maker) authentication for connections. See https://maester.dev/docs/tests/MT.1118" -Tag "MT.1118" { | |
| $result = Test-MtAIAgentAuthorAuthentication | |
| if ($null -ne $result) { | |
| $result | Should -Be $true -Because "Agents using author (maker) authentication access external services with the maker's credentials, creating privilege escalation and separation of duties risks." | |
| } | |
| } | |
| # AI agents should not have hard-coded credentials in topics. | |
| It "MT.1119: AI agents should not have hard-coded credentials in topics. See https://maester.dev/docs/tests/MT.1119" -Tag "MT.1119" { | |
| $result = Test-MtAIAgentHardCodedCredentials | |
| if ($null -ne $result) { | |
| $result | Should -Be $true -Because "Hard-coded credentials in agent topics can be extracted by prompt injection attacks and persist after key rotation." | |
| } | |
| } | |
| # AI agents should not use MCP server tools without review. | |
| It "MT.1120: AI agents should not use MCP server tools without review. See https://maester.dev/docs/tests/MT.1120" -Tag "MT.1120" { | |
| $result = Test-MtAIAgentMcpTools | |
| if ($null -ne $result) { | |
| $result | Should -Be $true -Because "MCP tool integrations extend agents with arbitrary external capabilities and may introduce supply chain risks." | |
| } | |
| } | |
| # AI agents with generative orchestration should have custom instructions. | |
| It "MT.1121: AI agents with generative orchestration should have custom instructions. See https://maester.dev/docs/tests/MT.1121" -Tag "MT.1121" { | |
| $result = Test-MtAIAgentMissingInstructions | |
| if ($null -ne $result) { | |
| $result | Should -Be $true -Because "Agents using generative orchestration without custom instructions rely on default LLM behavior, increasing prompt injection and off-topic response risk." | |
| } | |
| } | |
| # AI agents should not have orphaned ownership. | |
| It "MT.1122: AI agents should not have orphaned ownership. See https://maester.dev/docs/tests/MT.1122" -Tag "MT.1122" { | |
| $result = Test-MtAIAgentOrphaned | |
| if ($null -ne $result) { | |
| $result | Should -Be $true -Because "Agents whose owners are all disabled or deleted cannot be maintained and may continue operating with outdated or insecure configurations." | |
| } |
| if ($dataverseUrl -match '^https?://') { | ||
| $resourceUrl = $dataverseUrl -replace '\.api\.', '.' | ||
| } else { | ||
| $resourceUrl = "https://$dataverseUrl" | ||
| } | ||
|
|
There was a problem hiding this comment.
When DataverseEnvironmentUrl is provided without a scheme and includes '.api.' (as suggested in some docs/examples), $resourceUrl is built without stripping '.api.', so token acquisition will target the API host instead of the environment host. Normalize consistently by always removing '.api.' after ensuring the https:// prefix.
| if ($dataverseUrl -match '^https?://') { | |
| $resourceUrl = $dataverseUrl -replace '\.api\.', '.' | |
| } else { | |
| $resourceUrl = "https://$dataverseUrl" | |
| } | |
| if (-not ($dataverseUrl -match '^https?://')) { | |
| $dataverseUrl = "https://$dataverseUrl" | |
| } | |
| # Resource URL should always point to the environment host (without '.api.') | |
| $resourceUrl = $dataverseUrl -replace '\.api\.', '.' |
| foreach ($upn in $ownerUpns) { | ||
| try { | ||
| $user = Invoke-MtGraphRequest -RelativeUri "users/$upn" -Select 'id,accountEnabled' -ApiVersion 'v1.0' | ||
| if ($user -and $user.accountEnabled -eq $true) { | ||
| $allOwnersInvalid = $false | ||
| break | ||
| } | ||
| } catch { | ||
| # User not found or access denied - treat as invalid | ||
| Write-Verbose "Could not resolve owner $upn for agent $($agent.AIAgentName): $_" | ||
| } |
There was a problem hiding this comment.
The per-owner loop issues a separate Graph call for each UPN (N+1). For tenants with many agents/owners this will be slow and may hit throttling. Consider batching the lookups (Invoke-MtGraphRequest supports -UniqueId arrays in other tests) or resolving owners via a single filtered query before iterating agents.
| $failedAgents = $agents | Where-Object { $_.AccessControlPolicy -eq "Any" -or $_.AccessControlPolicy -eq "Any multitenant" } | ||
|
|
||
| if ([string]::IsNullOrEmpty($failedAgents)) { | ||
| $testResultMarkdown = "Well done. No AI agents are shared broadly." | ||
| } else { | ||
| $testResultMarkdown = "Found $($failedAgents.Count) AI agent(s) with broad sharing configured.`n`n%TestResult%" | ||
| $result = "| Agent Name | Environment | Access Control | Authentication |`n" | ||
| $result += "| --- | --- | --- | --- |`n" | ||
| foreach ($agent in $failedAgents) { | ||
| $result += "| $($agent.AIAgentName) | $($agent.EnvironmentId) | $($agent.AccessControlPolicy) | $($agent.UserAuthenticationType) |`n" | ||
| } | ||
| $testResultMarkdown = $testResultMarkdown -replace "%TestResult%", $result | ||
| } | ||
|
|
||
| Add-MtTestResultDetail -Result $testResultMarkdown -Severity "High" | ||
| return [string]::IsNullOrEmpty($failedAgents) | ||
| } |
There was a problem hiding this comment.
$failedAgents is a collection from the pipeline; using [string]::IsNullOrEmpty($failedAgents) will always evaluate unexpectedly (non-empty collections stringify to 'System.Object[]'), and the function may return incorrect results. Use a proper collection check (e.g., $failedAgents.Count -eq 0 or -not $failedAgents) for both the message branch and the return value.
| --- | ||
| title: MT.1117 - Published AI agents should not be dormant | ||
| description: Checks all published Copilot Studio agents for those that have not been modified or republished within a configurable threshold (default 180 days). |
There was a problem hiding this comment.
This page claims the dormancy threshold is configurable, but the implementation currently hard-codes 180 days and there is no DormantAgentDays setting in the repo. Either implement the configuration setting or update the docs to reflect the fixed threshold.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Description
Adds 10 new Copilot Studio agent security tests (MT.1113-MT.1122) based on the Copilot Studio Agent Security Top 10 Risks. These tests query agent configuration from the Dataverse OData API and check for common security misconfigurations in Copilot Studio agents.
Also adds
Dataverseas a new service option inConnect-Maesterto support authentication to the Dataverse API, and preserves existing Az contexts (federated credentials, managed identity) so the interactive login prompt is skipped when a session already exists.What's been tested
I've tested this locally using both interactive and SPN (clientId + secret). Interactive with
Connect-Maester -Graph,Dataverse, SPN requires you to sign in withConnect-AzAccountandConnect-MgGraphbefore runningConnect-Maester -Dataversesince Graph will be implicit (explicit??). Didn't test in CI/CD.I set up 11 different agents in Copilot Studio plus the default one (this one is excluded from tests):
If you want to test it, here's the guidance on setting up the different agents:
I also had Claude whip up a test plan TEST-VALIDATION-PLAN.md that I went through locally.
Disclaimers
I'm not a good writer, so Claude helped build the skeletons for the .md files, the comments for the scripts and the tests. I've gone over the markdown files to make sure all the links works and has relevant/correct information. The regex for test MT.1119 detects everything I've tested, but man am I bad at regex.
Some things are open for debate and can freely be changed:
Test-MtAIAgent*- can also beTest-MtCopilotStudioor something.Why no Defender XDR table?
This requires Copilot Studio licenses to publish the agents, so I didn't have the option to do this. Might be viable, would love to see someone with access to that test this.
Why no Graph?
I tried using these two endpoints:
GET graph.microsoft.com/copilot/admin/catalog/packages– Retrieves a complete inventory of all agents and apps (Microsoft, External, Shared, and Custom).GET graph.microsoft.com/copilot/admin/catalog/packages/{id}– Fetches detailed metadata for a specific agent or app, including properties and manifest details.Endpoints didn't exist (tried BETA also) and according to https://mc.merill.net/message/MC1173195 it's being added later, so either I'm dumb (I am, but still) or we can try this later.
What's included
New tests (10)
Test-MtAIAgentBroadSharingTest-MtAIAgentNoAuthenticationTest-MtAIAgentRiskyHttpConfigTest-MtAIAgentEmailExfiltrationTest-MtAIAgentDormantTest-MtAIAgentAuthorAuthenticationTest-MtAIAgentHardCodedCredentialsTest-MtAIAgentMcpToolsTest-MtAIAgentMissingInstructionsTest-MtAIAgentOrphanedNew internal function
Get-MtAIAgentInfo- Queries the Dataverse OData API for Copilot Studio agent data, maps option set values, resolves owner UPNs, classifies topics vs tools, and caches results for reuse across tests.Connect-Maester changes
Dataverseto the-ServiceparameterValidateSetGet-AzContextcheck to reuse existing Az sessions from federated credentials, managed identity, or priorConnect-AzAccountcallsDisconnect-Maesterto handle Dataverse serviceDocumentation
website/docs/tests/maester/MT.1113.md-MT.1122.md)website/docs/connect-maester/readme.mdand the advanced connection docs with Dataverse sectionpowershell/public/maester/aiagent/Test-MtAIAgent*.md)Configuration
DataverseEnvironmentUrltotests/maester-config.jsonGlobalSettings (empty default)DormantAgentDaysconfig option (default: 180) for MT.1117New files
Modified files
How to test
Prerequisites
Az.Accountsmodule installedDataverseEnvironmentUrlset intests/Custom/maester-config.json:{ "GlobalSettings": { "DataverseEnvironmentUrl": "https://orgXXXXX.crmNN.dynamics.com" } }bot,botcomponent,systemuser, andconnectionreferencetables.Create an Application User in Power Platform
Maester Security Reader) with Organization-level Read on: Agent (bot), Agent component (botcomponent), User (systemuser), and Connection Reference (connectionreference)*Creating a custom role requires:
Configure Maester
Add the environment URL to
maester-config.json:{ "GlobalSettings": { "DataverseEnvironmentUrl": "https://org12345.crm.dynamics.com" } }Interactive testing
What to verify
Connect-Maester -Service Dataverseconnects and validates tokenDataverseEnvironmentUrlis not configuredReferences
Contribution Checklist
Before submitting this PR, please confirm you have completed the following:
/powershell/tests/pester.ps1on your local system.