Stop requesting Observability API permissions for blueprint agents by default

CHANGELOG.md

@@ -8,9 +8,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ### Upgrade Notes
 
-#### Existing agents: grant Observability API permissions
+#### Agents exporting through the delegated (OBO) route: grant Observability API permissions
 
-Agents provisioned before this release need `Agent365.Observability.OtelWrite` granted as both a **delegated** and an **application** permission on the blueprint app. Requires Global Administrator.
+Agents that export telemetry through the delegated (OBO) route need `Agent365.Observability.OtelWrite` granted as both a **delegated** and an **application** permission on the blueprint app. Requires Global Administrator.
 
 **Option A — Entra portal** (no config files required):
 
@@ -22,6 +22,8 @@ Agents provisioned before this release need `Agent365.Observability.OtelWrite` g
 
 **Option B — CLI** (`a365 setup admin`) has been removed in this release. Use Option A above, or copy the PowerShell instructions printed in the `a365 setup all` summary output.
 
+Blueprint agents that export telemetry through the app-only S2S endpoint don't need these permissions, and `a365 setup all` no longer requests them for blueprint agents (#501).
+
 ### Added
 - Setup and bootstrap now use Microsoft's first-party Agent 365 CLI application when it is present in your tenant, validating it without changing Microsoft's app registration, and fall back to a tenant-owned "Agent 365 CLI" app when it is not (#489).
 - Log separator written at the start of each CLI invocation now redacts values for secret-bearing options (e.g. `--idp-client-secret`) so they are not written to the log file in plain text.
@@ -59,6 +61,7 @@ Agents provisioned before this release need `Agent365.Observability.OtelWrite` g
 - `a365 develop get-token --device-code` — forces device code auth for Microsoft Graph scopes the Windows WAM broker rejects (e.g. Exchange `MailboxSettings.ReadWrite`, `ExchangeMessageTrace.Read.All`).
 
 ### Fixed
+- `a365 setup all` now exits with code 1 when agent registration fails or cannot be verified for blueprint agents or with `--agent-registration-only` (#501).
 - Setup no longer fails to detect the Agent 365 CLI application in tenants where it is not yet provisioned, and reports lookup errors instead of silently switching your configured client app (#489).
 - The first-party Agent 365 CLI app now uses device code authentication when Windows Account Manager is unavailable, avoiding unsupported browser-response errors in WSL, macOS, and Linux (#489).
 - `setup all --authmode s2s` no longer prints spurious "Action Required" PowerShell steps when the agent identity already inherits its app roles from the blueprint, and now retries the grant automatically before falling back to manual steps (#460).
@@ -104,6 +107,7 @@ Agents provisioned before this release need `Agent365.Observability.OtelWrite` g
 
 ### Changed
 
+- `a365 setup all` no longer requests Observability API permissions for blueprint agents, so registered agents export telemetry through the app-only S2S endpoint without admin consent (#501).
 - Hardened token storage: the CLI no longer writes access tokens to a plaintext file — they live only in the OS-protected MSAL cache (DPAPI/Keychain/owner-only file). Any legacy plaintext cache is removed automatically; sign-in prompts are unchanged.
 - `develop-mcp register-external-mcp-server` now sets `exit code 1` on failure paths (validation errors, tenant detection failure, Graph unavailable, Entra app creation failure, MCP-Platform AddMcpServer failure). Previously these paths logged an error and exited `0`, which made the command's success/failure status undetectable from scripts and CI. Successful dry-run and user-initiated cancellation at the y/N prompt continue to exit `0`.
 - Admin consent canary path (when the caller lacks `DelegatedPermissionGrant.Read.All`) no longer prompts for Enter immediately. The CLI now polls every 5 seconds, prints a friendly progress message at 30 seconds, and responds promptly to Enter or Ctrl+C. The previous jargon-heavy message about `oauth2PermissionGrants` was rewritten in plain English; technical details are demoted to `Debug`.

docs/agent365-guided-setup/a365-observability-instructions.md

@@ -775,11 +775,13 @@ This skill is safe to rerun. On subsequent runs:
 
 ### OtelWrite App Role Assignment
 
-`a365 setup all` **attempts** to grant `Agent365.Observability.OtelWrite` to the Agent Identity SP, but this requires **Global Administrator** privileges. If the logged-in user is not a Global Admin, the assignment silently fails with 403 and trace exports will return HTTP 403 from the observability service.
+> **Blueprint agents:** `a365 setup all` does not request Observability API permissions for blueprint agents in any auth mode — the S2S endpoint authorizes registered agent instances without the `OtelWrite` role, so no admin consent is needed. Setup exits with code 1 if registration fails; retry with `a365 setup all --agent-registration-only`. Grant `OtelWrite` manually (steps below) only if the agent still exports through the delegated (OBO) route. Permissions granted by earlier runs are not revoked.
+
+For **AI Teammate** agents, `a365 setup all` still **attempts** to grant `Agent365.Observability.OtelWrite` to the Agent Identity SP, which requires **Global Administrator** privileges. If the logged-in user is not a Global Admin, the assignment fails with 403 and trace exports can return HTTP 403 from the observability service.
 
 **The CLI prints a PowerShell admin consent script** in its output when the assignment fails. When running `a365 setup all`, **always scan the output for this script block** and display it to the user in a fenced code block so they can copy it and hand it to a Global Admin.
 
-If the script was not captured, grant the permission manually via Entra portal (requires Global Admin):
+If the script was not captured — or for a blueprint agent whose SDK still exports through the delegated (OBO) route — grant the permission manually via Entra portal (requires Global Admin):
 1. [Entra portal](https://entra.microsoft.com) > App registrations > select Blueprint app > API permissions
 2. Add a permission > APIs my organization uses > search `9b975845-388f-4429-889e-eab1ef63949c`
 3. Add both **Delegated** and **Application** `Agent365.Observability.OtelWrite` > Grant admin consent

src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/AllSubcommand.cs

@@ -397,13 +397,19 @@ effectiveAuthModeForValidation is not ("obo" or "s2s" or "both"))
                 return;
             }
 
+            // Registered blueprint agents export telemetry app-only over S2S without OtelWrite in every auth
+            // mode, so blueprint setup never requests it; AI Teammate setup (including an AI Teammate config
+            // kept for a dry run) is unchanged.
+            var skipObservabilityPermissions = nonDwConfig is not null
+                && (aiTeammateFlag == false || nonDwConfig.IsBlueprintAgent);
+
             if (nonDwConfig is not null)
             {
                 if (dryRun)
                 {
                     var rawArgs = context.ParseResult.Tokens.Select(t => t.Value).ToArray();
                     var effectiveAuthMode = authMode ?? nonDwConfig.AuthMode;
-                    NonDwBlueprintSetupOrchestrator.PrintDryRunPlan(nonDwConfig, logger, isBootstrap, rawArgs, skipRequirements, isM365, agentRegistrationOnly, effectiveAuthMode, messagingEndpointFlag);
+                    NonDwBlueprintSetupOrchestrator.PrintDryRunPlan(nonDwConfig, logger, isBootstrap, rawArgs, skipRequirements, isM365, agentRegistrationOnly, effectiveAuthMode, messagingEndpointFlag, skipObservabilityPermissions);
                     return;
                 }
 
@@ -442,7 +448,8 @@ effectiveAuthModeForValidation is not ("obo" or "s2s" or "both"))
                     confirmationProvider: confirmationProvider,
                     skipSpProvisioning: skipSpProvisioning,
                     messagingEndpointOverride: messagingEndpointFlag,
-                    nonInteractive: Console.IsInputRedirected);
+                    nonInteractive: Console.IsInputRedirected,
+                    skipObservabilityPermissions: skipObservabilityPermissions);
 
                 context.ExitCode = await NonDwBlueprintSetupOrchestrator.ExecuteAsync(nonDwCtx);
                 return;
@@ -1018,7 +1025,8 @@ await PermissionsSubcommand.RemoveStaleCustomPermissionsAsync(
         // for both DW and non-DW agents; serverNamesByAudience drives the per-server display
         // names so V2 audiences read as e.g. "mcp_MailTools" rather than "Agent 365 Tools".
         var specs = await SetupHelpers.BuildConfiguredPermissionSpecsAsync(
-            ctx.Config, setInheritable: true, isM365: ctx.IsM365, scopesByAudience, serverNamesByAudience);
+            ctx.Config, setInheritable: true, isM365: ctx.IsM365, scopesByAudience, serverNamesByAudience,
+            includeObservability: !ctx.SkipObservabilityPermissions);
 
         // Return the full scopesByAudience map alongside the V1-compat mcpScopes so V2
         // callers (ApplyConsentUrlsIfNeeded) can route per-server audiences to the bare

src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/NonDwBlueprintSetupOrchestrator.cs

@@ -20,8 +20,8 @@ namespace Microsoft.Agents.A365.DevTools.Cli.Commands.SetupSubcommands;
 ///   1. Requirements validation
 ///   2. Blueprint creation (shared with DW)
 ///   3. Batch permissions on the blueprint (shared with DW pipeline; non-DW spec set:
-///      Observability API, Power Platform API, custom). MAC reads from the blueprint,
-///      so stamping here gives the same set visibility there.
+///      Power Platform API and custom; Observability API is not requested). MAC reads
+///      from the blueprint, so stamping here gives the same set visibility there.
 ///   4. Agent Identity creation via POST /beta/servicePrincipals/Microsoft.Graph.AgentIdentity
 ///   5. Agent Identity permission grants (same spec set as step 3) — OBO or S2S
 ///   6. Agent registration via Graph API (copilot/agentRegistrations)
@@ -33,7 +33,7 @@ internal static class NonDwBlueprintSetupOrchestrator
     /// Prints a dry-run plan showing all resources that would be created or configured,
     /// using actual names and values from the loaded config. Makes no API calls.
     /// </summary>
-    public static void PrintDryRunPlan(Agent365Config config, ILogger logger, bool isBootstrap = false, string[]? rawArgs = null, bool skipRequirements = false, bool isM365 = false, bool agentRegistrationOnly = false, string? authMode = null, string? messagingEndpointOverride = null)
+    public static void PrintDryRunPlan(Agent365Config config, ILogger logger, bool isBootstrap = false, string[]? rawArgs = null, bool skipRequirements = false, bool isM365 = false, bool agentRegistrationOnly = false, string? authMode = null, string? messagingEndpointOverride = null, bool skipObservabilityPermissions = false)
     {
         var sub = new string(' ', SetupHelpers.DryRunValCol);
         // --messaging-endpoint flag (if supplied) wins over the init-only config value for the plan.
@@ -117,14 +117,17 @@ public static void PrintDryRunPlan(Agent365Config config, ILogger logger, bool i
             logger.LogInformation(sub + "create managed identity");
         }
 
-        // 3. Inheritable Permissions — non-DW spec set (Observability API, Power Platform API, custom)
-        //    stamped on the blueprint via SetInheritablePermissionsAsync so MAC and other dependent
-        //    systems can see them. The same set is applied to the agent identity SP in step 5.
+        // 3. Inheritable Permissions — non-DW spec set (Power Platform API and custom; Observability API is
+        //    not requested) stamped on the blueprint via SetInheritablePermissionsAsync so MAC and other
+        //    dependent systems can see them. The same set is applied to the agent identity SP in step 5.
         var selectedAuthMode = authMode ?? config.AuthMode;
         var effectiveMode = string.IsNullOrWhiteSpace(selectedAuthMode)
             ? "obo"
             : selectedAuthMode.Trim().ToLowerInvariant();
-        logger.LogInformation(SetupHelpers.DryRunRow(3, "Inheritable Permissions") + "configure for Observability API, Power Platform API, and custom permissions (Global Administrator required; consent URL printed if absent)");
+        logger.LogInformation(SetupHelpers.DryRunRow(3, "Inheritable Permissions") + "configure for {Resources} (Global Administrator required; consent URL printed if absent)",
+            skipObservabilityPermissions ? "Power Platform API and custom permissions" : "Observability API, Power Platform API, and custom permissions");
+        if (skipObservabilityPermissions)
+            logger.LogInformation(sub + "Observability API not requested (registered agents export telemetry with an app-only token)");
 
         // 4. Blueprint Permission Grants — per authMode. The consent URL targets the blueprint
         //    app, and S2S app-role assignments are persisted as grants flowing from the blueprint;
@@ -266,6 +269,7 @@ await ctx.ClientAppValidator.GrantConsentForPermissionsAsync(
     public static async Task<int> ExecuteAsync(SetupContext ctx)
     {
         ctx.Results.IsNonDwBlueprintFlow = true;
+        ctx.Results.ObservabilityPermissionsSkipped = ctx.SkipObservabilityPermissions;
         ctx.Results.TenantId = ctx.Config.TenantId;
         // Bootstrap already printed the "Running..." banner before auth steps; skip here to avoid duplication.
         if (!ctx.IsBootstrap)
@@ -362,8 +366,10 @@ public static async Task<int> ExecuteAsync(SetupContext ctx)
                 // Step 3: Blueprint creation (shared with DW)
                 await AllSubcommand.ExecuteBlueprintStepAsync(ctx);
 
-                // Step 4: Build permission specs — stamps Graph, manifest MCP audiences, Observability,
-                // Power Platform, custom permissions, and Messaging Bot (only when isM365). Mirrors DW.
+                // Step 4: Build permission specs — stamps Graph, manifest MCP audiences, Power Platform,
+                // custom permissions, Messaging Bot (only when isM365), and Observability unless skipped.
+                if (ctx.SkipObservabilityPermissions)
+                    ctx.Logger.LogInformation("Observability API permissions not requested: registered agents export telemetry with an app-only token.");
                 var buildResult = await AllSubcommand.BuildPermissionSpecsAsync(ctx);
                 specs = buildResult.specs;
 
@@ -435,7 +441,7 @@ await AllSubcommand.ExecuteBatchPermissionsStepAsync(
     /// When <paramref name="skipIdentityAndPermissions"/> is true (--agent-registration-only),
     /// identity creation and permission grants are skipped — only registration and project settings run.
     /// </summary>
-    private static async Task ExecuteAgentIdentityAndRegistrationAsync(
+    internal static async Task ExecuteAgentIdentityAndRegistrationAsync(
         SetupContext ctx,
         List<ResourcePermissionSpec> specs,
         bool skipIdentityAndPermissions = false)
@@ -549,22 +555,31 @@ private static async Task ExecuteAgentIdentityAndRegistrationAsync(
         ctx.Logger.LogInformation("");
         ctx.Logger.LogInformation("Registering agent...");
 
+        // Registration is the sole purpose of --agent-registration-only and, with OtelWrite skipped,
+        // the agent's only Observability authorization, so its failure must fail setup.
+        var registrationRequired = skipIdentityAndPermissions || ctx.SkipObservabilityPermissions;
+        void RecordRegistrationFailure(string message)
+        {
+            ctx.Results.AgentRegistrationFailed = true;
+            (registrationRequired ? ctx.Results.Errors : ctx.Results.Warnings).Add(message);
+            ctx.Logger.Log(registrationRequired ? LogLevel.Error : LogLevel.Warning, message);
+        }
+
         if (string.IsNullOrWhiteSpace(ctx.Config.AgenticAppId))
         {
             var registrationSkippedMessage =
                 "Agent registration failed: agent identity ID is not available. " +
                 "Ensure the agent identity was created successfully, then retry with: a365 setup all --agent-registration-only";
-            ctx.Results.Warnings.Add(registrationSkippedMessage);
             using (ctx.Logger.Indent())
-                ctx.Logger.LogWarning(registrationSkippedMessage);
-            ctx.Results.AgentRegistrationFailed = true;
+                RecordRegistrationFailure(registrationSkippedMessage);
         }
         else
         {
 
         // If a registration ID is already stored, verify it still exists before skipping creation.
         string? registrationId = null;
         bool registrationAlreadyExisted = false;
+        bool verificationFailed = false;
 
         if (!string.IsNullOrWhiteSpace(ctx.Config.AgentRegistrationId))
         {
@@ -591,6 +606,16 @@ private static async Task ExecuteAgentIdentityAndRegistrationAsync(
                 // stale value on disk that would cause the same stale-ID check to repeat.
                 await ctx.ConfigService.SaveStateAsync(ctx.Config);
             }
+            else if (registrationRequired)
+            {
+                // An unverifiable registration cannot be the agent's only authorization: keep the stored
+                // ID (no duplicate registration) but fail so the operator retries.
+                using (ctx.Logger.Indent())
+                    RecordRegistrationFailure(
+                        $"Could not verify agent registration {ctx.Config.AgentRegistrationId} (auth or transient error). " +
+                        "Retry with: a365 setup all --agent-registration-only");
+                verificationFailed = true;
+            }
             else
             {
                 // Verification inconclusive (auth or transient error) — preserve the stored ID
@@ -602,7 +627,7 @@ private static async Task ExecuteAgentIdentityAndRegistrationAsync(
             }
         }
 
-        if (string.IsNullOrWhiteSpace(registrationId))
+        if (!verificationFailed && string.IsNullOrWhiteSpace(registrationId))
         {
             var (newId, fromConflict) = await ctx.GraphApiService.RegisterAgentInstanceAsyncV2(
                 ctx.Config.TenantId!,
@@ -631,11 +656,9 @@ private static async Task ExecuteAgentIdentityAndRegistrationAsync(
                 ctx.Logger.LogInformation("");
             }
         }
-        else
+        else if (!verificationFailed)
         {
-            ctx.Results.AgentRegistrationFailed = true;
-            ctx.Results.Warnings.Add("Agent registration failed via Graph copilot/agentRegistrations API.");
-            ctx.Logger.LogWarning("Agent registration failed via Graph copilot/agentRegistrations API.");
+            RecordRegistrationFailure("Agent registration failed via Graph copilot/agentRegistrations API.");
         }
 
         } // end else (AgenticAppId present)

src/Microsoft.Agents.A365.DevTools.Cli/Commands/SetupSubcommands/README.md

@@ -92,6 +92,12 @@ a365 setup all --authmode s2s
 a365 setup all --authmode both
 ```
 
+### Observability permissions
+
+For blueprint agents, `setup all` does not request `Agent365.Observability.OtelWrite` in any auth mode. Registered agents export telemetry with an app-only token through the S2S endpoint, which authorizes them by their agent registration, so no Observability admin consent is needed. Registration is then the agent's only authorization, so a registration failure is reported as an error (exit code 1).
+
+Agents whose SDK still exports through the delegated (OBO) route need `OtelWrite`; grant it manually (see the CHANGELOG upgrade note). AI Teammate setup is unchanged. Re-running setup does not revoke permissions granted earlier.
+
 ---
 
 ### Messaging endpoint (M365 agents)