From 7973bfa47f90dc3e25c1db100bfbb83d79d45f82 Mon Sep 17 00:00:00 2001 From: Krishnadheeraj <12496535+DheerajPannala@users.noreply.github.com> Date: Fri, 25 Sep 2026 18:46:35 +0100 Subject: [PATCH 01/12] Export observability over S2S with app-only tokens in every auth mode Telemetry now always uses the Observability API S2S route (/observabilityService/) with an app-only token for the exporting agent identity. Registered blueprint agent instances therefore need no Agent365.Observability.OtelWrite permission or admin consent; OBO / Agentic User tokens are used only for workload (MCP / Graph) calls. - instrument-observability references: app-only token resolver scaffolds for Node.js, Python and .NET that reuse the hosting connection (FMI assertion for the turn's agent identity, then agent-identity client_credentials); S2S route flag in every auth mode; migration path for existing delegated wiring; .NET options moved to the flattened Microsoft.OpenTelemetry 1.0.3+ API (o.Agent365.*). - Stop hooks and a365-code-validator: require the S2S route flag and flag delegated telemetry tokens (new findings, hook/standalone parity tests). - Provisioning, test-local, shared detection, docs and evals: registration-based authorization; AI Teammates keep the OtelWrite application-role step. Related: microsoft/Agent365-nodejs#290, microsoft/Agent365-Samples#339, microsoft/Agent365-devTools#501 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5cbf5f6b-cc40-4b7e-a591-65848db73a12 --- .github/copilot-instructions.md | 16 +- AGENTS.md | 8 +- CLAUDE.md | 20 +- README.md | 10 +- evals/README.md | 3 + evals/agent365/a365-code-validator/evals.json | 28 +- evals/agent365/a365-setup/evals.json | 2 +- .../instrument-observability/evals.json | 58 ++- .../stop/validate-a365-code-validator.js | 95 ++++- .../stop/validate-instrument-observability.js | 73 +++- plugins/agent365/shared/agent-detection.md | 18 +- .../skills/a365-code-validator/SKILL.md | 100 +++-- .../references/a365-code-validator.js | 54 ++- .../references/validation-checklist.md | 68 ++-- .../agent365/skills/add-workiq-tools/SKILL.md | 4 +- .../skills/instrument-observability/SKILL.md | 301 +++++++-------- .../references/dotnet-observability.md | 343 ++++++++++++------ .../references/nodejs-observability.md | 266 ++++++++++---- .../references/python-observability.md | 229 +++++++++--- .../agent365/skills/make-a365-agent/SKILL.md | 2 +- .../agent365/skills/make-ai-teammate/SKILL.md | 24 +- .../references/deploy-pipeline.md | 2 +- .../references/dotnet-ai-teammate.md | 11 +- .../references/python-ai-teammate.md | 10 +- plugins/agent365/skills/test-local/SKILL.md | 3 +- tests/validate-a365-code-validator.test.js | 259 +++++++++++++ tests/validate-observability.test.js | 332 +++++++++++++++++ 27 files changed, 1813 insertions(+), 526 deletions(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 6de594c..8e4bb2c 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -164,7 +164,7 @@ wrapping. 2. Shows a dry-run preview of all `a365` operations before applying anything 3. Runs `a365 setup all` — creates the Blueprint and Entra ID permissions (add `--m365` for CEA agents; run `a365 setup permissions bot` after for Messaging Bot API grants). Supports `--authmode obo|s2s` for non-AI Teammate agents to control permission grant type; **never passes `--authmode` with `--aiteammate`** (AI Teammate uses the Agentic User identity — agent's own M365 identity, not the caller's token; `--authmode` flag not supported with `--aiteammate`). Skipped entirely when `reuseBlueprint = true`. Handles WAM prompts — if a native sign-in dialog appears, instructs user to complete it without killing the process. Auto-falls back to device code flow if blocked by Conditional Access Policy. 4. After setup, always offers `instrument-observability` as an optional add-on; offers `add-workiq-tools` only when `authMode ≠ s2s` — WorkIQ is silently skipped for S2S agents (requires a user token) -5. `Agent365.Observability.OtelWrite` is auto-granted at provisioning. Other permission grants (Graph, Bot API, custom resources) require Global Administrator consent — `a365 setup all` automatically prints next-steps (typically a PowerShell script) when the developer is not a GA. There is no separate `setup admin` subcommand; the skill displays the printed script verbatim so the user can hand it to a Global Admin +5. No Observability API permission is needed: recent `a365 setup all` versions no longer request `Agent365.Observability.OtelWrite` (or its consent) for blueprint agents. Telemetry is exported over the S2S route with an app-only token, and the route authorizes the registered agent instance; setup exits 1 when registration fails or cannot be verified. Other permission grants (Graph, Bot API, custom resources) require Global Administrator consent — `a365 setup all` automatically prints next-steps (typically a PowerShell script) when the developer is not a GA. There is no separate `setup admin` subcommand; the skill displays the printed script verbatim so the user can hand it to a Global Admin **Normally delegated to from `a365-setup`** after CLI and Azure prerequisites are confirmed. Can also be invoked directly. @@ -249,19 +249,19 @@ wrapping. **Summary of what this skill does:** 1. Loads detection cache; asks a two-stage question (agent kind + auth mode) if not already cached; writes `agentType`+`authMode` back to `.a365-workspace-detection.local.json` so `add-workiq-tools` and future runs skip re-asking 2. Installs the observability packages: unified distros for all paths — `Microsoft.OpenTelemetry` for .NET, `@microsoft/opentelemetry` for Node.js, `microsoft-opentelemetry` for Python. Legacy individual packages (`Microsoft.Agents.A365.Observability.*`, `@microsoft/agents-a365-*`, `microsoft-agents-a365-observability-*`) still accepted by the validator but no longer generated. -3. **OBO path** (`obo` / `agentic-user`, including AI Teammate): calls `useMicrosoftOpenTelemetry()` (Node.js/Python) or `builder.UseMicrosoftOpenTelemetry(o => ...)` (.NET) — the distro **auto-registers `IExporterTokenCache`** in DI for .NET, so no separate `AddAgenticTracingExporter()` or `AddA365Tracing()` call is needed. Token resolver wired to per-turn token refresh via `RegisterObservability(...)` in the message handler. Also requires `.UseOpenTelemetry()` on the `IChatClient` so the AI SDK emits `gen_ai` spans for `InvokeAgentScope` to anchor. Agent id resolved for both Teams agentic turns (via `Activity.GetAgenticInstanceId()`) and Playground/WebChat OBO turns (via `Utility.ResolveAgentIdentity` decoding the OBO token); observability is gracefully skipped when neither path yields a real (agent, tenant) tuple — avoids polluting traces with `Guid.Empty`-grouped orphan spans the exporter cannot authenticate. +3. **Telemetry always uses the S2S route with an app-only token (every auth mode).** The S2S route rejects delegated (`scp`) tokens, so OBO / Agentic User tokens are used only for workload calls (MCP / Graph). No OBS permission or admin consent is needed for registered blueprint agent instances (AI Teammates: complete the `OtelWrite` application-role step `a365 setup all --aiteammate` prints). **OBO path** (`obo` / `agentic-user`, including AI Teammate): writes an app-only token resolver that reuses the hosting connection's blueprint credential (FMI assertion for the turn's agent identity, then agent-identity `client_credentials` for the OBS scope). The files are `Observability/AgentAppTokenResolver.cs` wired as `o.Agent365.TokenResolver`, `observability/app-token-resolver.ts` as `tokenResolver`, or `observability/app_token_resolver.py` whose `OBS_TOKENS.resolve` is `a365_token_resolver`, with a per-turn `OBS_TOKENS.prefetch(...)`. It sets the S2S route flag (`o.Agent365.UseS2SEndpoint = true` on `Microsoft.OpenTelemetry` 1.0.3+, `useS2SEndpoint: true`, `a365_use_s2s_endpoint=True`). There is **no** per-turn `RegisterObservability(..., AgenticTokenStruct)` / `refreshObservabilityToken(..., authorization)` / `exchange_token(...)` for telemetry; legacy wiring like that is migrated. Also requires `.UseOpenTelemetry()` on the `IChatClient` so the AI SDK emits `gen_ai` spans for `InvokeAgentScope` to anchor. .NET agent id: `Activity.GetAgenticInstanceId()` for agentic turns; other turns fall back to the provisioned agent identity in `Agent365Observability:AgentId` (never the blueprint). Observability is gracefully skipped when no real (agent, tenant) tuple exists — avoids polluting traces with `Guid.Empty`-grouped orphan spans the exporter cannot authenticate. 4. **S2S path (all languages)**: Creates a scaffold token-service file that acquires/refreshes the Observability API token (`api://9b975845-388f-4429-889e-eab1ef63949c/.default`) via MSAL with FMI path support every 50 min. - - **.NET**: creates `Observability/ObservabilityServiceExtensions.cs` + `Observability/ObservabilityTokenService.cs`; uses MSAL `ConfidentialClientApplicationBuilder` with `.WithFmiPath()` for FMI 3-hop chain; wires `UseMicrosoftOpenTelemetry()` + `AddAgent365Observability()`; in the message handler uses `new BaggageBuilder().FromTurnContext(turnContext).Build()` (separate `using var`) and `InvokeAgentScope.Start(request, new InvokeAgentScopeDetails(endpoint: new Uri(...)), agentDetails, callerDetails)` (separate `using var`) — **NOT chained; `FromTurnContext()` is a `BaggageBuilder` extension only**; `CallerDetails` with blueprint sponsor identity is **required** for S2S traces to appear + - **.NET**: creates `Observability/ObservabilityServiceExtensions.cs` + `Observability/ObservabilityTokenService.cs`; uses MSAL `ConfidentialClientApplicationBuilder` with `.WithFmiPath()` for FMI 3-hop chain; wires `UseMicrosoftOpenTelemetry()` (with `o.Agent365.UseS2SEndpoint = true` — `o.Agent365.Exporter.UseS2SEndpoint` on 1.0.2 and earlier) + `AddAgent365Observability()`; in the message handler uses `new BaggageBuilder().FromTurnContext(turnContext).Build()` (separate `using var`) and `InvokeAgentScope.Start(request, new InvokeAgentScopeDetails(endpoint: new Uri(...)), agentDetails, callerDetails)` (separate `using var`) — **NOT chained; `FromTurnContext()` is a `BaggageBuilder` extension only**; `CallerDetails` with blueprint sponsor identity is **required** for S2S traces to appear - **Node.js** (`@microsoft/opentelemetry` 1.0 GA): creates `observability/observability-token-service.ts` (exports `startTokenService()`) + `observability/token-cache.ts` (exports `tokenResolver`); for client-secret Hop 1+2 uses direct HTTP POST with `fmi_path` form parameter (MSAL Node.js doesn't serialize `fmiPath`), MSAL for Hop 3; calls `useMicrosoftOpenTelemetry({ a365: { enabled: true, enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver } })` — `useS2SEndpoint` is a first-class option in 1.0+; do NOT use the old hand-rolled `spanProcessors` workaround - **Python** (`microsoft-opentelemetry` 1.1 GA): creates `observability/observability_token_service.py` + `observability/token_cache.py`; for client-secret Hop 1+2 uses direct HTTP POST with `fmi_path` form parameter (MSAL Python doesn't serialize `fmi_path`), MSAL for Hop 3; calls `use_microsoft_opentelemetry(enable_a365=True, a365_enable_observability_exporter=True, a365_use_s2s_endpoint=True, a365_token_resolver=...)` 5. Updates `appsettings.json` (for .NET) with `Agent365Observability` section; S2S adds `ClientId`, `ClientSecret`, and `UseManagedIdentity: true`; creates `appsettings.Development.json` with exporter disabled. Note: `a365 setup all` (CLI 1.1+) auto-writes placeholder sections — skill checks for existing placeholders before creating from scratch. 6. **Stamps verbose-logging env pair** into `.env` (Node.js/Python — commented-out by default with "uncomment to debug" guidance): `OTEL_LOG_LEVEL=INFO` (OTel SDK internal logger) AND `A365_OBSERVABILITY_LOG_LEVEL=info|warn|error` (A365 exporter logger). Both required together — the exporter uses a wrapped logger that defaults to silent, so `OTEL_LOG_LEVEL` alone shows nothing. -7. **Canonical Node.js scope-wrapping pattern (required for spans to reach MAC):** message handler must call `preloadObservabilityToken(turnContext)` FIRST, then build an outer `BaggageBuilderUtils.fromTurnContext(new BaggageBuilder(), turnContext as any).build()` baggage scope, then run `InvokeAgentScope.start(...)` and `InferenceScope.start(...)` INSIDE `baggageScope.run(async () => { ... })`. Without this exact wrapping, spans get filtered as `Partitioned into 0 identity groups (N spans skipped)` and are not exported — the #1 first-run failure mode. Node.js also requires BOTH `a365.enabled: true` AND `a365.enableObservabilityExporter: true` in the `useMicrosoftOpenTelemetry` call (or env `ENABLE_A365_OBSERVABILITY_EXPORTER=true`) — `enabled: true` alone enriches spans but never exports them. +7. **Canonical Node.js scope-wrapping pattern (required for spans to reach MAC):** the message handler builds an outer `BaggageBuilderUtils.fromTurnContext(new BaggageBuilder(), turnContext as any).build()` baggage scope, then runs `InvokeAgentScope.start(...)` and `InferenceScope.start(...)` INSIDE `baggageScope.run(async () => { ... })`. No per-turn token step: the app-only `tokenResolver` acquires and caches the export token on demand. Without this exact wrapping, spans get filtered as `Partitioned into 0 identity groups (N spans skipped)` and are not exported — the #1 first-run failure mode. Node.js also requires BOTH `a365.enabled: true` AND `a365.enableObservabilityExporter: true` in the `useMicrosoftOpenTelemetry` call (or env `ENABLE_A365_OBSERVABILITY_EXPORTER=true`) — `enabled: true` alone enriches spans but never exports them — plus `a365.useS2SEndpoint: true`. 8. **Phase 8.5 — First-run smoke test (Node.js / Python).** After build validation, starts the agent for ~30s in background with verbose envs enabled and greps `.a365-smoketest.log` for `export-group succeeded` / `exported successfully`. Pass/fail visible immediately rather than discovered via MAC 90 min later. .NET skipped — its own boot-time logging covers verification. -9. **Final summary calls out MAC visibility expectations:** 15-90 min indexing lag from first export to spans surfacing in `admin.cloud.microsoft → Advanced Hunting → CloudAppEvents`; instance-approval prerequisite; KQL filter MUST use the runtime AUID (`recipient.agenticAppId`), NOT the blueprint id (which `a365 setup all` stamps into `.env` as `agent365Observability__agentId`). +9. **Final summary calls out MAC visibility expectations:** 15-90 min indexing lag from first export to spans surfacing in `admin.cloud.microsoft → Advanced Hunting → CloudAppEvents`; instance-approval prerequisite; KQL filter MUST use the runtime AUID (`recipient.agenticAppId`), NOT the blueprint id (which `a365 setup all` stamps into `.env` as `agent365Observability__agentId` for AI Teammates); blueprint agents have no OBS permission/consent step — a 403 `insufficient_scope` means the instance isn't registered (`a365 setup all --agent-registration-only`); AI Teammates complete the `OtelWrite` application-role step `a365 setup all --aiteammate` prints. 10. Validates the build passes -**Auth mode note:** All three `authMode` values use an auth handler reference in SDK code — the difference is Azure AD provisioning, not code structure. For .NET OBO: `authHandlerName` comes from config (`AgentApplication:AgenticAuthHandlerName`), not hardcoded. For Node.js OBO: pass `agentApplication.authorization` (the auth object, not a string) to `AgenticTokenCacheInstance.RefreshObservabilityToken`. For Python OBO: use `auth_handler_id=self.auth_handler_name` (from config) in `exchange_token()` — never hardcode `"AGENTIC"`. Agent IDs are always resolved dynamically from TurnContext (`agenticAppId` / `agentic_app_id`), never from config. S2S is supported for .NET, Node.js, and Python. +**Auth mode note:** Telemetry never goes through the auth handler — every `authMode` exports over the S2S route with an app-only token. The auth handler is for workload calls (MCP / Graph): for .NET OBO, `authHandlerName` comes from config (`AgentApplication:AgenticAuthHandlerName`), not hardcoded; for Node.js OBO, `agentApplication.authorization` (the auth object, not a string) is used by the tooling calls; for Python OBO, `auth_handler_id=self.auth_handler_name` (from config) — never hardcode `"AGENTIC"`. Agent IDs are resolved dynamically from TurnContext (`agenticAppId` / `agentic_app_id`); only non-agentic turns of non-AI-Teammate agents fall back to the provisioned agent identity in config. S2S is supported for .NET, Node.js, and Python. **Prerequisite:** `a365-setup` must be run first. Reads `.a365-workspace-detection.local.json` to skip re-detection. @@ -293,7 +293,7 @@ wrapping. 2. Checks exporter activation (`ENABLE_A365_OBSERVABILITY_EXPORTER` / `EnableAgent365Exporter`, Node `enableObservabilityExporter`, Python `a365_enable_observability_exporter`, .NET `EnableAgent365Exporter`). 3. Checks identity binding: runtime Agent Identity / Source Agent ID must be used for `/agents/{agentId}` and `gen_ai.agent.id`; Blueprint ID belongs only in `microsoft.a365.agent.blueprint.id`. 4. Checks semantic span coverage: `invoke_agent`, `chat`, `execute_tool`, and `output_messages` (or framework scopes that produce them). -5. Checks S2S vs OBO endpoint expectations and token resolver signals. +5. Checks that telemetry uses the S2S route with an app-only token in every auth mode and token resolver signals; flags delegated telemetry (missing S2S route flag, or OBO / Agentic User tokens fed to the exporter) and unregistered blueprint agent instances. 6. When an existing target-tenant login is available, runs read-only `a365 query-entra blueprint-scopes` and `a365 query-entra inheritance` checks to verify the live Blueprint ID, actual grants, and effective permission inheritance. 7. Produces a report with blockers, risky findings, and customer-safe runtime verification guidance using SDK logs, direct OTel `partialSuccess`, and public Defender `CloudAppEvents` queries. @@ -393,7 +393,7 @@ All code added by observability instrumentation must be marked with the language - Python install: use `pip3 install ... 2>/dev/null || pip install ...` for cross-platform (pip3 on macOS/Linux, fall back to pip on Windows). No `--pre` flag needed — packages are GA. - .NET S2S: `FromTurnContext()` is only on `BaggageBuilder` — never chain it on `InvokeAgentScope.Start()` - .NET S2S: `InvokeAgentScopeDetails` has no parameterless constructor — always pass `endpoint: new Uri(...)` -- .NET OBO: `RegisterObservability` takes four args: `agentId, tenantId, AgenticTokenStruct, scopes` +- All languages, every auth mode: select the S2S route (`o.Agent365.UseS2SEndpoint = true` on `Microsoft.OpenTelemetry` 1.0.3+ — `o.Agent365.Exporter.*` on 1.0.2 and earlier; `useS2SEndpoint: true`; `a365_use_s2s_endpoint=True`) and pass an app-only token resolver. Never register or refresh a delegated telemetry token (`RegisterObservability(..., AgenticTokenStruct, ...)`, `refreshObservabilityToken(..., authorization)`, `exchange_token(...)` for the observability scope) All code added by WorkIQ wiring must be marked with the language-appropriate comment form: - C# / JavaScript / TypeScript: `// A365 WorkIQ — added by add-workiq-tools skill` diff --git a/AGENTS.md b/AGENTS.md index 2649567..1643d91 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -52,7 +52,7 @@ test-local (no prerequisite) **Phase 9.7.2d** validates environment configuration before either path proceeds. For prod: confirms `a365.generated.config.json` has `completed: true` and non-empty `resourceConsents` (else GA consent handoff is pending); confirms `.env`/`appsettings.json` has agentic-auth + LLM + observability vars; reminds the user that cloud env vars must be set at the cloud platform (`az webapp config appsettings set` / `eb setenv` / `gcloud run services update --set-env-vars`), not just locally; confirms HTTPS messaging endpoint. For local: confirms AgentsPlayground is installed and `.m365agentsplayground.yml` is configured when using agentic auth. Authoritative Microsoft Learn refs: [test-with-devtunnels](https://learn.microsoft.com/en-us/microsoft-agent-365/developer/test-with-devtunnels), [testing](https://learn.microsoft.com/en-us/microsoft-agent-365/developer/testing), [deploy-agent-azure](https://learn.microsoft.com/en-us/microsoft-agent-365/developer/deploy-agent-azure), [deploy-agent-aws](https://learn.microsoft.com/en-us/microsoft-agent-365/developer/deploy-agent-aws), [deploy-agent-gcp](https://learn.microsoft.com/en-us/microsoft-agent-365/developer/deploy-agent-gcp). **Automatically** runs `instrument-observability` (only when `has_obs = false`) and **optionally** offers `add-workiq-tools` (only when `has_workiq = false`). The skill does NOT hand-edit `manifest.json`. Reference: [Create agent instance — Microsoft Learn](https://learn.microsoft.com/en-us/microsoft-agent-365/developer/create-instance). `a365-setup` outputs a mandatory intro message, detects stack/language/CEA/`hasBlueprintConfig`, and the three skill-state flags **`has_aiteammate_structure`**, **`has_obs`**, **`has_workiq`** (the same primary flags that drive `make-ai-teammate` Phase 0C's 8-row matrix). Always updates the a365 CLI to latest (explicit exception to the ✅-skip rule), checks for an existing Azure CLI session before logging in, shows a ✅/❌ prerequisite summary and only processes ❌ missing tools. Asks the blueprint question (reuse vs fresh) then asks **capabilities first** — capability options are auto-filtered: Observability is hidden if `has_obs = true`, WorkIQ is hidden if `has_workiq = true`, the menu collapses to Register + WorkIQ when `(has_aiteammate_structure && has_obs)` (legacy "already an AI Teammate" route, computed inline — the legacy `hasAITeammateChanges` field is **derived, no longer stored**). If AI Teammate is selected, auth mode is skipped (always `agentic-user`); if non-AI Teammate, asks `obo` or `s2s`. Cache fields written: `agentStack`, `programmingLanguage`, `usesTeamsOrCopilot`, `hasBlueprintConfig`, `has_aiteammate_structure`, `has_obs`, `has_workiq`, `agentType`, `authMode`, `reuseBlueprint`, `existingBlueprintId`. Delegates: AI Teammate path → `make-ai-teammate`; all other paths → `make-a365-agent`. -`make-a365-agent` checks for an existing blueprint config before collecting inputs — if found, asks the developer whether to reuse (skips `a365 setup all`) or create fresh. Runs `a365 setup all --authmode obo|s2s` for non-AI Teammate paths; add `--m365` for CEA agents and follow with `a365 setup permissions bot`. `Agent365.Observability.OtelWrite` is auto-granted at provisioning, but other permission grants (Graph, Bot API, custom resources) require Global Administrator consent — when the developer isn't a GA, `a365 setup all` automatically prints next-steps (typically a PowerShell script) for a GA to complete. There is no separate `setup admin` subcommand. Then conditionally invokes `instrument-observability` and `add-workiq-tools`. +`make-a365-agent` checks for an existing blueprint config before collecting inputs — if found, asks the developer whether to reuse (skips `a365 setup all`) or create fresh. Runs `a365 setup all --authmode obo|s2s` for non-AI Teammate paths; add `--m365` for CEA agents and follow with `a365 setup permissions bot`. No Observability API permission is needed: recent CLI versions no longer request `Agent365.Observability.OtelWrite` for blueprint agents, because telemetry is exported over the S2S route with an app-only token and the route authorizes the registered agent instance (setup exits 1 when registration fails or cannot be verified). Other permission grants (Graph, Bot API, custom resources) require Global Administrator consent — when the developer isn't a GA, `a365 setup all` automatically prints next-steps (typically a PowerShell script) for a GA to complete. There is no separate `setup admin` subcommand. Then conditionally invokes `instrument-observability` and `add-workiq-tools`. `add-workiq-tools` and `instrument-observability` read `.a365-workspace-detection.local.json` to skip re-detection and verify prerequisites. `add-workiq-tools` Phase 0B includes a **framework support guard** that hard-stops on unsupported `(programmingLanguage, agentStack)` pairs (Python LangChain / Claude / CrewAI; Node.js Semantic Kernel / Google ADK) before any CLI command runs. Phase 4 branches on the cached `agentStack` into 11 framework-specific sub-sections (§4.1 .NET Agent Framework through §4.11 Python Azure AI Foundry); the stop-hook validator (`validate-add-workiq-tools.js`) is also framework-aware and requires the framework-matching symbol (e.g., `AddToolServersToAgentAsync` for .NET SK, `add_tool_servers_to_agent` for Python). **Phase 4.5 (gated)** offers the Word `@mention` notification handler when *both* gates pass: `programmingLanguage = NodeJS && agentStack = LangChain`, AND `mcp_WordServer` is in `ToolingManifest.json`. Wires `proactive: {}`, per-user conversation index, and a `NotificationType.WpxComment` branch — best-effort because no Microsoft Node.js sample is published yet. Best-effort branches (Python SK, Python/.NET Azure AI Foundry, and the Phase 4.5 @mention handler) mark all generated lines with `// A365 WorkIQ — best-effort wiring (verify against SDK source before production)`. `purview-dlp-integration` is **additive** and independent of observability / WorkIQ. It auto-discovers the app (client) id, display name, blueprint id, and current Graph scopes from `a365.config.json` + `a365.generated.config.json`, then asks only for what's missing (DLP policy choice, sensitive info type, admin UPN, agentic auth handler name). It detects the language and authentication and copies ONE generic env-driven guard (`assets/purview.ts` / `purview.py` / `purview.cs` for delegated agents; `assets/purview-s2s.ts` for Node.js client-secret FMI S2S; .NET is best-effort). Minimal wiring adds an **INPUT gate** before the LLM and optional **output auditing** before the reply; the supplied policy does not filter sensitive responses. Delegated guards use the agent's own token at `/me` with `Content.Process.User` appended by `Grant-DelegatedGraphScope.ps1`. Node.js S2S uses the agent identity FMI token at `/users/{sponsor}/...` with `Content.Process.All` from `Grant-ContentProcessAppRole.ps1`, never a blueprint app-only token or `admin-consent` on the blueprint. Do not switch managed-identity-only agents to client-secret authentication. Manual IDs replace config discovery, not missing authentication; a plain bot without a supported path must stop and route to `a365-setup` before edits. The guards always set `contentEntry.name` and fail closed by default. Policy choice remains new (`New-AiAppDlpPolicy.ps1`), existing (`-ListExisting`), or skip. Verify the `[purview] uploadText -> BLOCKED (… errors=0)` log and that the LLM was not called, not `DistributionStatus`. The stop-hook validator is report-first because disabled/policy-pending bring-up is valid. @@ -250,9 +250,11 @@ Skills reference shared docs via `Read ${CLAUDE_PLUGIN_ROOT}/shared/.md`. - AI Teammate → `obo` (signed-in user OBO) or `agentic-user` (agent's own Azure AD user — persistent M365 identity) - Agent (Non AI Teammate) → `obo` (On-Behalf-Of) or `s2s` (Service Principal, no user token) -All three `authMode` values use an auth handler reference in SDK code — the difference is Azure AD provisioning. For OBO paths: .NET reads `authHandlerName` from config (`AgentApplication:AgenticAuthHandlerName`); Node.js passes `agentApplication.authorization` (the auth object) to `RefreshObservabilityToken`; Python uses `auth_handler_id=self.auth_handler_name` (from config) in `exchange_token()` — never hardcode `"AGENTIC"`. Agent IDs are always resolved dynamically from TurnContext — .NET: `turnContext.Activity.GetAgenticInstanceId()` (service principal object ID); Node.js/Python: `recipient.agenticAppId` / `agentic_app_id`. Results are cached in `.a365-workspace-detection.local.json` under `agentType` and `authMode` fields so subsequent skill invocations skip re-questioning. If `authMode = s2s` and the skill is `add-workiq-tools`, the skill exits immediately — WorkIQ is not available for s2s agents (requires a delegated OBO token). +Telemetry never goes through the auth handler: every `authMode` exports over the S2S route with an app-only token for the exporting agent identity (the S2S route rejects delegated `scp` tokens). The auth handler reference is for workload calls (MCP / Graph): .NET reads `authHandlerName` from config (`AgentApplication:AgenticAuthHandlerName`); Node.js uses `agentApplication.authorization` (the auth object); Python uses `auth_handler_id=self.auth_handler_name` (from config) — never hardcode `"AGENTIC"`. Agent IDs are resolved dynamically from TurnContext — .NET: `turnContext.Activity.GetAgenticInstanceId()` (service principal object ID); Node.js/Python: `recipient.agenticAppId` / `agentic_app_id`. Only non-agentic turns of non-AI-Teammate agents fall back to the provisioned agent identity in config (never the blueprint). Results are cached in `.a365-workspace-detection.local.json` under `agentType` and `authMode` fields so subsequent skill invocations skip re-questioning. If `authMode = s2s` and the skill is `add-workiq-tools`, the skill exits immediately — WorkIQ is not available for s2s agents (requires a delegated OBO token). -**S2S scaffold requirement:** When `authMode = s2s`, `instrument-observability` creates a scaffold token-service file per language: .NET creates `Observability/ObservabilityServiceExtensions.cs` + `Observability/ObservabilityTokenService.cs`, Node.js creates `observability/observability-token-service.ts`, Python creates `observability/observability_token_service.py`. Each acquires and refreshes the Observability API token via MSAL client credentials targeting `api://9b975845-388f-4429-889e-eab1ef63949c/.default`. The .NET files additionally provide the `AddAgent365Observability()` / `Agent365ObservabilityContext` DI extensions that replace `AddAgenticTracingExporter()` and per-turn `RegisterObservability()`. The validator (`validate-instrument-observability.js`) accepts either the OBO signal (`BaggageBuilder` / `BaggageTurnMiddleware`) or the S2S signal to pass the context check. +**App-only telemetry token (every auth mode):** For `obo` / `agentic-user`, `instrument-observability` creates an app-only token resolver that reuses the hosting connection's blueprint credential (FMI assertion for the turn's agent identity → agent-identity `client_credentials` for the OBS scope): `Observability/AgentAppTokenResolver.cs` (.NET), `observability/app-token-resolver.ts` (Node.js), or `observability/app_token_resolver.py` (Python — sync `resolve` plus a per-turn async `prefetch`). No per-turn delegated `RegisterObservability(..., AgenticTokenStruct)` / `refreshObservabilityToken(..., authorization)` / `exchange_token(...)` is generated, and existing delegated wiring is migrated. Every mode sets the S2S route flag (`o.Agent365.UseS2SEndpoint = true` — `o.Agent365.Exporter.*` on `Microsoft.OpenTelemetry` 1.0.2 and earlier; `useS2SEndpoint: true`; `a365_use_s2s_endpoint=True`). Registered blueprint agent instances need no `Agent365.Observability.OtelWrite` permission or admin consent (subject to service policy); AI Teammates complete the `OtelWrite` application-role step `a365 setup all --aiteammate` prints. + +**S2S scaffold requirement:** When `authMode = s2s`, `instrument-observability` creates a scaffold token-service file per language: .NET creates `Observability/ObservabilityServiceExtensions.cs` + `Observability/ObservabilityTokenService.cs`, Node.js creates `observability/observability-token-service.ts`, Python creates `observability/observability_token_service.py`. Each acquires and refreshes the Observability API token via MSAL client credentials targeting `api://9b975845-388f-4429-889e-eab1ef63949c/.default`. The .NET files additionally provide the `AddAgent365Observability()` / `Agent365ObservabilityContext` DI extensions that replace `AddAgenticTracingExporter()` and per-turn `RegisterObservability()`. The validator (`validate-instrument-observability.js`) accepts either the OBO signal (`BaggageBuilder` / `BaggageTurnMiddleware`) or the S2S signal to pass the context check. When the unified distro is used, it also requires the S2S route flag in every auth mode, and it fails the session on delegated telemetry wiring (`refreshObservabilityToken(..., authorization)`, `RegisterObservability(..., AgenticTokenStruct)`, `exchange_token(...)` for the observability scope). --- diff --git a/CLAUDE.md b/CLAUDE.md index 9f3389c..a1a7a9d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -107,8 +107,8 @@ agent365-skills/ them with URL + action and continue. 11. **`a365-code-validator` is report-first.** It diagnoses exporter activation, agent-id - binding, semantic span coverage, S2S/OBO endpoint mismatches, and live Blueprint grants / - effective inheritance through read-only `a365 query-entra` checks when login is already + binding, semantic span coverage, delegated (OBO-route) telemetry or a missing S2S route flag, + and live Blueprint grants / effective inheritance through read-only `a365 query-entra` checks when login is already available. It then asks whether to apply safe fixes, create a fix plan, or stop. It must not provision, install packages, mutate Graph, grant permissions, or run `a365 publish`. @@ -125,6 +125,22 @@ agent365-skills/ (`validate-purview-dlp-integration.js`) is report-first (advisory `findings`, always `ok: true`) because "start disabled / skip policy" is a valid bring-up state. +13. **Telemetry always uses the S2S route with an app-only token, in every `authMode`.** The + S2S route rejects delegated (`scp`) tokens, so OBO / Agentic User tokens are only for workload + calls (MCP / Graph). `obo` / `agentic-user` agents get an app-only resolver that reuses the + hosting connection's blueprint credential (`AgentAppTokenResolver.cs` / + `app-token-resolver.ts` / `app_token_resolver.py`); `s2s` agents get the FMI token-service + scaffold. Always set the S2S route flag (`o.Agent365.UseS2SEndpoint = true`, + `useS2SEndpoint: true`, `a365_use_s2s_endpoint=True`). Never generate a per-turn delegated + telemetry token (`RegisterObservability(..., AgenticTokenStruct)`, + `refreshObservabilityToken(..., authorization)`, `exchange_token(...)` for the observability + scope), and never add the delegated `OtelWrite` scope for telemetry. Registered blueprint agent + instances need no `Agent365.Observability.OtelWrite` permission or admin consent, so never make + an OBS grant a required step for them. For blueprint agents, a 403 `insufficient_scope` means + the instance isn't registered: `a365 setup all --agent-registration-only`. AI Teammates complete + the `OtelWrite` application-role step that `a365 setup all --aiteammate` prints. The application + role is always an accepted fallback on the S2S route. + --- ## Testing diff --git a/README.md b/README.md index 3521e22..4c4bdfb 100644 --- a/README.md +++ b/README.md @@ -248,9 +248,10 @@ wiring any code, asks a two-stage question to determine **agent kind** and **aut - **AI Teammate**: has Agentic User with UPN; then asks whether it uses `obo` (signed-in user) or `agentic-user` (agent's own M365 identity). **Both support Observability and WorkIQ.** - **Agent (Non AI Teammate)**: no Agentic User; then asks whether it is `obo` (On-Behalf-Of) or `s2s` (Service Principal, no user token). **Observability supports both; WorkIQ requires `obo` (delegated user token).** -**Wiring by auth mode:** -- **`obo` / `agentic-user`** (OBO token exchange): the unified `Microsoft.OpenTelemetry` distro (.NET) / `@microsoft/opentelemetry` (Node.js) / `microsoft-opentelemetry` (Python) auto-registers the agentic token cache; the message handler calls per-turn `RegisterObservability` / `RefreshObservabilityToken` / `cache_agentic_token` with the OBO token +**Wiring by auth mode:** every mode exports over the S2S route with an **app-only** token for the exporting agent identity. The S2S route rejects delegated (`scp`) tokens. Registered blueprint agent instances need no `Agent365.Observability.OtelWrite` permission or admin consent; AI Teammates complete the `OtelWrite` application-role step that `a365 setup all --aiteammate` prints. +- **`obo` / `agentic-user`**: adds an app-only token resolver that reuses the agent's hosting connection (blueprint credential). It gets an FMI assertion for the turn's agent identity, then an agent-identity `client_credentials` token for the Observability API. The files are `Observability/AgentAppTokenResolver.cs` (.NET), `observability/app-token-resolver.ts` (Node.js), or `observability/app_token_resolver.py` (Python). OBO / Agentic User tokens are used only for workload calls (MCP / Graph); there is no per-turn delegated `RegisterObservability` / `refreshObservabilityToken` / `exchange_token` for telemetry, and existing delegated wiring is migrated - **`s2s`** (Service Principal, all languages): creates a scaffold token-service file per language (`Observability/ObservabilityTokenService.cs` for .NET, `observability/observability-token-service.ts` for Node.js, `observability/observability_token_service.py` for Python) that acquires the Observability API token (`api://9b975845-388f-4429-889e-eab1ef63949c/.default`) via the MSAL FMI 3-hop chain and refreshes every 50 min — no per-turn token call +- **All modes** set the S2S route flag: `o.Agent365.UseS2SEndpoint = true` (.NET), `useS2SEndpoint: true` (Node.js), or `a365_use_s2s_endpoint=True` (Python) All new code is marked `// A365 Observability — best-effort instrumentation` and changes are non-destructive and idempotent. @@ -269,7 +270,8 @@ All new code is marked `// A365 Observability — best-effort instrumentation` a Report-first diagnostics for agents that should emit MAC Activity / Defender / Purview telemetry. Checks exporter activation, runtime agent ID vs blueprint ID binding, supported A365 semantic -span operations, S2S/OBO endpoint expectations, and (when a target-tenant login is already +span operations, S2S-route selection with an app-only token in every auth mode (flagging +delegated OBO-route telemetry and unregistered blueprint agent instances), and (when a target-tenant login is already available) the live Blueprint grants and effective inheritance through read-only `a365 query-entra` checks. Produces a report with concrete runtime verification commands, then asks whether to apply safe fixes, create a fix plan, or stop. It does not provision resources, @@ -423,7 +425,7 @@ Check which Agent 365 capabilities have already been applied to this agent and t - **6 skills** covering full AI Teammate transformation, Blueprint provisioning for all capability paths, WorkIQ MCP servers, observability instrumentation, and local testing with AgentsPlayground - **Multi-language support** — Node.js (LangChain, OpenAI Agents SDK, Claude SDK, Semantic Kernel, Google ADK), .NET (AgentFramework, Semantic Kernel), and Python (AgentFramework, LangChain, OpenAI, Claude, Semantic Kernel, Google ADK) -- **Auth mode detection** — two-stage question flow determines agent kind (AI Teammate vs Agent (Non AI Teammate)) and auth mode (`obo` / `s2s` / `agentic-user`); drives the correct observability and WorkIQ token path; cached in `.a365-workspace-detection.local.json` across skills +- **Auth mode detection** — two-stage question flow determines agent kind (AI Teammate vs Agent (Non AI Teammate)) and auth mode (`obo` / `s2s` / `agentic-user`); drives the WorkIQ token path and how the app-only observability token is sourced (telemetry always uses the S2S route); cached in `.a365-workspace-detection.local.json` across skills - **Automatic agent detection** — skills detect your LLM framework, programming language, and Custom Engine Agent status, then ask validation questions before any code runs - **Non-destructive and idempotent** — skills wrap existing code without deleting anything; re-running skips what is already configured - **WorkIQ MCP servers** — pre-built M365 integrations for Mail, Calendar, Teams, SharePoint, OneDrive, Word, User profiles, Copilot, and Dataverse/Dynamics 365 diff --git a/evals/README.md b/evals/README.md index 66450fa..d0a6bc0 100644 --- a/evals/README.md +++ b/evals/README.md @@ -143,6 +143,7 @@ To manually test a skill against an eval: | 3 | Already instrumented (.NET) | Idempotency: skip if marker comments present | | 4 | Already instrumented (Node.js) | Idempotency: skip if marker comments present | | 5 | Unknown agent type | Write `.a365setup-unknown-agent` marker and exit with clear error | +| 11 | Delegated telemetry migration (Node.js AI Teammate) | Legacy `refreshObservabilityToken` / no `useS2SEndpoint` wiring is treated as partial and migrated to the S2S route with an app-only token — no OtelWrite / consent step | ### `test-local` Evals @@ -168,6 +169,8 @@ To manually test a skill against an eval: | 8 | Code clean, Activity empty | Branches to tenant-side causes (licensing, enrollment, resource SP, lag) instead of a false code bug | | 9 | No `invoke_agent` root | Catches child-span-only runs / identity-less spans ("0 identity groups") that never land in MAC | | 10 | Guided remediation | Verifies `apply_safe_fixes` applies only the deterministic exporter fix and asks before design changes | +| 11 | Blueprint permissions | Read-only `a365 query-entra` checks; a registered blueprint agent needs no OtelWrite — flags missing registration instead | +| 12 | Delegated telemetry (AI Teammate) | Catches OBO tokens / the delegated route fed to the exporter; recommends the app-only resolver + S2S route, not an OtelWrite grant | --- diff --git a/evals/agent365/a365-code-validator/evals.json b/evals/agent365/a365-code-validator/evals.json index 47c21ed..144ba4f 100644 --- a/evals/agent365/a365-code-validator/evals.json +++ b/evals/agent365/a365-code-validator/evals.json @@ -93,12 +93,12 @@ "id": 5, "prompt": "Validate A365 code for this S2S Python agent", "description": "Python S2S agent mints the observability token with a bare ClientSecretCredential / DefaultAzureCredential against the observability resource instead of the 3-hop FMI exchange", - "expected_output": "The skill flags the token shape: a plain client-credentials or bare managed-identity token has the app/MI as its principal, not the runtime Agent Identity, so the backend rejects it. It explains the failure signatures and recommends the 3-hop FMI exchange (blueprint creds + fmi_path= -> agent-identity assertion -> OtelWrite token). Read-only; prints no internal endpoints.", + "expected_output": "The skill flags the token shape: a plain client-credentials or bare managed-identity token has the app/MI as its principal, not the runtime Agent Identity, so the backend rejects it. It explains the failure signatures and recommends the FMI exchange (blueprint creds + fmi_path= -> agent-identity assertion -> app-only Observability API token without scp). Read-only; prints no internal endpoints.", "files": [], "expectations": [ "Phase 3.5: Flags token shape — a bare ClientSecretCredential/DefaultAzureCredential yields an app/MI principal, not the runtime Agent Identity", "Phase 3.5: Explains the failure signatures: 403 (principal != gen_ai.agent.id) and 400 TenantIdInvalid when no valid token is bound", - "Phase 3.5: Recommends the 3-hop FMI exchange (blueprint creds + fmi_path= -> agent-identity assertion -> Observability API token with roles containing OtelWrite)", + "Phase 3.5: Recommends the FMI exchange (blueprint creds + fmi_path= -> agent-identity assertion -> app-only Observability API token with no scp claim; roles may be empty for a registered agent instance)", "Phase 3: References validation-checklist section 5 (token shape) for the exchange detail", "Phase 5: Does not print internal cluster names, private endpoints, tenant IDs, real agent IDs, or correlation IDs", "Phase 6: Does not generate a full token service silently — offers a plan and asks for a second confirmation" @@ -108,11 +108,11 @@ "id": 6, "prompt": "Check A365 exporter flags for this S2S agent", "description": "S2S agent selects the endpoint only via A365_USE_S2S_ENDPOINT=true in env; the use_microsoft_opentelemetry(...) call does not pass a365_use_s2s_endpoint=True", - "expected_output": "The skill flags reliance on the env var for S2S endpoint selection and recommends setting the transport flag in code (a365_use_s2s_endpoint=True / useS2SEndpoint:true / UseS2SEndpoint), so the S2S token is not posted to the OBO route. It notes this mirrors passing the exporter-enable flag in code rather than depending on env parity.", + "expected_output": "The skill flags reliance on the env var for S2S endpoint selection and recommends setting the transport flag in code (a365_use_s2s_endpoint=True / useS2SEndpoint:true / UseS2SEndpoint), so the app-only token is not posted to the legacy delegated route. It notes that every auth mode uses the S2S route, and that this mirrors passing the exporter-enable flag in code rather than depending on env parity.", "files": [], "expectations": [ "Phase 3.5: Reports that the S2S endpoint is selected via env only, not the a365_use_s2s_endpoint / useS2SEndpoint / UseS2SEndpoint code option", - "Phase 3.5: Explains that an S2S token posted to the OBO endpoint is rejected — token type must match the route", + "Phase 3.5: Explains that an app-only token posted to the legacy delegated route is rejected, and that every auth mode must select the S2S route", "Phase 3.5: Recommends setting the S2S transport flag in code because env-only selection is more fragile", "Phase 5: Concise report that separates export success from Activity eligibility", "Phase 6: Offers the endpoint/exporter flag as a deterministic safe fix after confirmation" @@ -181,18 +181,32 @@ { "id": 11, "prompt": "Validate A365 code and check the Blueprint permissions", - "description": "Project has a local Blueprint ID and an existing target-tenant login; the Blueprint inheritance policy is allAllowed but the Blueprint service principal is missing the required observability grant", - "expected_output": "The skill runs the read-only a365 query-entra Blueprint diagnostics, distinguishes actual service-principal grants from inheritable policy, and reports the missing auth-mode-specific Agent365.Observability.OtelWrite permission as a blocker. It does not run setup permissions, mutate Graph, grant consent, or rewrite IDs.", + "description": "Project has a local Blueprint ID and an existing target-tenant login; the Blueprint inheritance policy is allAllowed but the Blueprint service principal has no Observability API grant, and a365.generated.config.json has no agentRegistrationId", + "expected_output": "The skill runs the read-only a365 query-entra Blueprint diagnostics, distinguishes actual service-principal grants from inheritable policy, and explains that the S2S route authorizes a registered agent instance without Agent365.Observability.OtelWrite. It reports the missing registration (not the missing OtelWrite grant) as the likely 403 cause and recommends a365 setup all --agent-registration-only. It does not run setup permissions, mutate Graph, grant consent, or rewrite IDs.", "files": [], "expectations": [ "Phase 4: Runs `a365 query-entra blueprint-scopes` to inspect delegated and application grants actually present on the Blueprint service principal", "Phase 4: Runs `a365 query-entra inheritance` and interprets `Effective inheritance: OK`, `NONE`, or `BROKEN` plus its exit code", - "Phase 4: For obo/agentic-user requires delegated Agent365.Observability.OtelWrite; for s2s requires the application role", + "Phase 4: Does not treat a missing Agent365.Observability.OtelWrite grant as a blocker for a registered blueprint agent; flags a missing registration (or missing OtelWrite application role) as high and recommends a365 setup all --agent-registration-only", "Phase 4: Compares the live resolved Blueprint with the local agentBlueprintId without printing real IDs in the report", "Phase 4: Treats Graph 401/403 as a skipped authorization gap rather than proof that permissions are absent", "Phase 5: Includes a concise Blueprint permissions section with live-ID, grants, and effective-inheritance status", "Phase 6: Does not run a365 setup permissions, mutate Graph, grant consent, or rewrite the Blueprint ID as a safe fix" ] + }, + { + "id": 12, + "prompt": "Validate A365 code for this AI Teammate Node.js agent", + "description": "AI Teammate agent calls AgenticTokenCacheInstance.refreshObservabilityToken(agentId, tenantId, turnContext, this.authorization) every turn and useMicrosoftOpenTelemetry(...) does not set useS2SEndpoint: true", + "expected_output": "The skill reports delegated telemetry as a high finding. The exporter receives an Agentic User (OBO) token on the legacy delegated route, which needs admin consent, while the S2S route rejects tokens with scp. It recommends the app-only token resolver from the instrument-observability references plus useS2SEndpoint: true, keeping OBO only for workload (MCP / Graph) calls. It does not recommend granting Agent365.Observability.OtelWrite or running a consent flow as the fix.", + "files": [], + "expectations": [ + "Phase 2: The static validator reports node-obs-delegated-token and node-obs-delegated-route", + "Phase 3.5: Explains that every auth mode, including AI Teammate / agentic-user, exports over the S2S route with an app-only token", + "Phase 3.5: Recommends the app-only tokenResolver (hosting connection FMI helper + agent identity client_credentials) and useS2SEndpoint: true", + "Phase 3.5: Does not recommend an OtelWrite grant or admin consent as the fix", + "Phase 6: Treats the resolver swap as a fix that needs a second explicit confirmation, not a silent safe fix" + ] } ] } diff --git a/evals/agent365/a365-setup/evals.json b/evals/agent365/a365-setup/evals.json index 1d2bd3a..24dbe77 100644 --- a/evals/agent365/a365-setup/evals.json +++ b/evals/agent365/a365-setup/evals.json @@ -196,7 +196,7 @@ "make-ai-teammate Phase 9: a365 setup all --aiteammate is run — NOT just a365 setup all (the --aiteammate flag is required for AI Teammate blueprint type)", "make-ai-teammate Phase 9: Setup Summary table shown verbatim from CLI output", "make-ai-teammate Phase 9: Blueprint ID and AGENTIC_APP_ID are read from a365.generated.config.json after setup all completes", - "make-ai-teammate Phase 9: If CLI output includes Permission Grants action item or 403 errors, the printed PowerShell script is displayed verbatim for a Global Administrator to run — required whenever the developer isn't a GA (OtelWrite is auto-granted, but Graph / Bot API / custom permission grants still need GA consent)", + "make-ai-teammate Phase 9: If CLI output includes an Observability API S2S app role item, another Permission Grants action item, or 403 errors, the printed PowerShell script is displayed verbatim for a Global Administrator to run (telemetry uses the S2S route with an app-only token, which the OtelWrite application role authorizes; Graph / Bot API / custom permission grants still need GA consent)", "a365 setup all without --aiteammate is NOT used for the AI Teammate path", "a365 publish is NOT run by a365-setup" ] diff --git a/evals/agent365/instrument-observability/evals.json b/evals/agent365/instrument-observability/evals.json index a2eb827..1d1c6f4 100644 --- a/evals/agent365/instrument-observability/evals.json +++ b/evals/agent365/instrument-observability/evals.json @@ -6,7 +6,7 @@ "id": 1, "prompt": "Instrument observability for this agent", "description": "Test on a .NET AgentFramework project — authMode: obo (OBO path)", - "expected_output": "The skill detects .NET AgentFramework, determines agentType and authMode via two-stage question (AI Teammate, then obo), installs the unified Microsoft.OpenTelemetry distro (which re-exports the legacy A365 Observability Runtime/Hosting types — do NOT install those legacy packages alongside), wires builder.UseMicrosoftOpenTelemetry(o => { o.Exporters = ExportTarget.Agent365; }) in Program.cs (the distro auto-registers IExporterTokenCache in DI — no separate AddAgenticTracingExporter call), chains .AsBuilder().UseFunctionInvocation().UseOpenTelemetry(...) on the IChatClient registration so LLM spans appear, resolves agent identity per-turn (GetAgenticInstanceId for agentic turns; Utility.ResolveAgentIdentity from the OBO token for non-agentic turns — never falls back to Guid.Empty), wraps BaggageBuilder + RegisterObservability calls in `if (hasObservabilityIdentity)`, sets EnableAgent365Exporter: true in appsettings.json (appsettings.Development.json sets false for console-only local dev), and validates the build passes.", + "expected_output": "The skill detects .NET AgentFramework, determines agentType and authMode via two-stage question (AI Teammate, then obo), installs the unified Microsoft.OpenTelemetry distro (which re-exports the legacy A365 Observability Runtime/Hosting types — do NOT install those legacy packages alongside), writes Observability/AgentAppTokenResolver.cs (app-only Observability API token per agent identity from the default blueprint connection: GetAgenticApplicationTokenAsync + agent-identity client_credentials) and registers it as a singleton, wires builder.UseMicrosoftOpenTelemetry(o => { o.Exporters = ExportTarget.Agent365; o.Agent365.UseS2SEndpoint = true; o.Agent365.TokenResolver = ...AgentAppTokenResolver... }) in Program.cs (telemetry uses the S2S route with an app-only token in every auth mode; no AddAgenticTracingExporter and no delegated IExporterTokenCache), chains .AsBuilder().UseFunctionInvocation().UseOpenTelemetry(...) on the IChatClient registration so LLM spans appear, resolves agent identity per-turn (GetAgenticInstanceId for agentic turns; the provisioned agent identity in Agent365Observability:AgentId for non-agentic turns, never the blueprint and never Guid.Empty), wraps BaggageBuilder + InvokeAgentScope in `if (hasObservabilityIdentity)` with no per-turn RegisterObservability call, sets EnableAgent365Exporter: true in appsettings.json (appsettings.Development.json sets false for console-only local dev), and validates the build passes.", "files": [], "expectations": [ "Phase 1: Agent detection identifies .NET AgentFramework correctly", @@ -25,20 +25,20 @@ "Phase 3: Program.cs is read before editing", "Phase 3: `using Microsoft.OpenTelemetry;` is added to Program.cs", "Phase 3: builder.UseMicrosoftOpenTelemetry(o => { ... }) is added with o.Exporters set to ExportTarget.Agent365 (prod) or ExportTarget.Agent365 | ExportTarget.Console (dev)", - "Phase 3: No explicit AddAgenticTracingExporter() call is added — the distro auto-registers IExporterTokenCache in DI", - "Phase 3: o.Agent365.Exporter.UseS2SEndpoint is left at default (false) for OBO — the exporter posts to /observability/ which the OBO token cache authenticates", + "Phase 3: Observability/AgentAppTokenResolver.cs is written and registered with builder.Services.AddSingleton() — no AddAgenticTracingExporter() call and no reliance on the distro's delegated IExporterTokenCache", + "Phase 3: o.Agent365.UseS2SEndpoint = true and o.Agent365.TokenResolver = (agentId, tenantId) => obsTokens?.ResolveAsync(agentId, tenantId) ?? Task.FromResult(null) are set (o.Agent365.Exporter.* only on Microsoft.OpenTelemetry 1.0.2 and earlier); obsTokens is resolved from app.Services after Build()", "Phase 3: IChatClient registration is chained with .AsBuilder().UseFunctionInvocation().UseOpenTelemetry(sourceName: null, cfg => cfg.EnableSensitiveData = true).Build() so gen_ai.inference / gen_ai.tool spans appear (without this, InvokeAgentScope is a hollow parent and no LLM spans show up in MAC)", "Phase 3: EnableAgent365Exporter: true noted — appsettings.json must have this set to activate the backend exporter (a365 setup may write false; this skill corrects it)", "Phase 3: All new lines include marker comment: // A365 Observability — best-effort instrumentation (verify against official sample)", "Phase 3: Existing code is preserved — only additive changes", "Phase 4: Message handler file is read before editing", - "Phase 4: IExporterTokenCache is injected in the agent constructor (auto-registered by the distro)", - "Phase 4: Agent identity resolved per turn — GetAgenticInstanceId() for agentic turns, Utility.ResolveAgentIdentity(turnContext, oboToken) for non-agentic turns (decoded from OBO token via UserAuthorization.GetTurnTokenAsync)", + "Phase 4: IExporterTokenCache is NOT injected and no per-turn RegisterObservability(..., AgenticTokenStruct) call is added — the exporter's app-only AgentAppTokenResolver supplies the token", + "Phase 4: Agent identity resolved per turn — GetAgenticInstanceId() for agentic turns; non-agentic turns fall back to Agent365Observability:AgentId only when it is a GUID different from AgentBlueprintId (never decoded from an OBO token, never the blueprint)", "Phase 4: Falls back cleanly when identity can't be resolved — does NOT use Guid.Empty.ToString() (that creates a synthetic identity the exporter can't authenticate)", - "Phase 4: hasObservabilityIdentity gate wraps both BaggageBuilder construction and RegisterObservability call — skipped cleanly when identity is missing", + "Phase 4: hasObservabilityIdentity gate wraps BaggageBuilder construction and the InvokeAgentScope — skipped cleanly when identity is missing", "Phase 4: BaggageBuilder.TenantId(...).AgentId(...).Build() used (returns IDisposable for using-scope)", - "Phase 4: AgenticTokenStruct constructed with authHandlerName resolved from config (AgentApplication:AgenticAuthHandlerName for agentic, OboAuthHandlerName for OBO) — NOT hardcoded \"AGENTIC\"", - "Phase 4: RegisterObservability call is wrapped in try/catch and only fires when hasObservabilityIdentity is true", + "Phase 4: Telemetry does not use an auth handler; any workload auth handler name still comes from config (AgentApplication:AgenticAuthHandlerName / OboAuthHandlerName) — NOT hardcoded \"AGENTIC\"", + "Phase 4: No OBO token is acquired to resolve the observability identity (no UserAuthorization.GetTurnTokenAsync for telemetry)", "Phase 4: authMode recorded as inline comment in the message handler", "Phase 4: All new lines include marker comment", "Phase 4: Existing handler logic is preserved", @@ -57,7 +57,7 @@ "id": 2, "prompt": "Add A365 observability to this LangChain agent", "description": "Test on a Node.js LangChain project with index.ts/js entry point", - "expected_output": "The skill detects Node.js LangChain, determines agentType and authMode via two-stage question, installs the unified @microsoft/opentelemetry package (single GA 1.0+ distro — legacy @microsoft/agents-a365-observability* and -extensions-{openai,langchain} packages are deprecated and must NOT be installed alongside), wires useMicrosoftOpenTelemetry({ a365: { enabled: true, enableObservabilityExporter: true, tokenResolver } }) with AgenticTokenCacheInstance.getObservabilityToken as the tokenResolver in the entry point BEFORE any LLM/framework imports, calls configureA365Hosting(adapter, { enableBaggage: true }) once at startup to auto-populate baggage from TurnContext (replaces manual BaggageBuilderUtils.fromTurnContext, which was removed in 1.0), wires shutdownMicrosoftOpenTelemetry() in SIGTERM/SIGINT handlers, calls AgenticTokenCacheInstance.RefreshObservabilityToken per turn with agentId/tenantId from TurnContext + agentApplication.authorization, updates .env with observability variables, and validates the build.", + "expected_output": "The skill detects Node.js LangChain, determines agentType and authMode via two-stage question, installs the unified @microsoft/opentelemetry package (single GA 1.0+ distro — legacy @microsoft/agents-a365-observability* and -extensions-{openai,langchain} packages are deprecated and must NOT be installed alongside), writes observability/app-token-resolver.ts (app-only token per agent identity from the hosting connection's getAgenticApplicationToken + agent-identity client_credentials) and wires useMicrosoftOpenTelemetry({ a365: { enabled: true, enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver: appTokenResolver } }) in the entry point BEFORE any LLM/framework imports, binding getObsConnection to adapter.connectionManager.getDefaultConnection() once the adapter exists, calls configureA365Hosting(adapter, { enableBaggage: true }) once at startup to auto-populate baggage from TurnContext (replaces manual BaggageBuilderUtils.fromTurnContext, which was removed in 1.0), wires shutdownMicrosoftOpenTelemetry() in SIGTERM/SIGINT handlers, does NOT call AgenticTokenCacheInstance.refreshObservabilityToken or add a preloadObservabilityToken helper (the S2S route rejects delegated tokens), updates .env with observability variables, and validates the build.", "files": [], "expectations": [ "Phase 1: Agent detection identifies Node.js LangChain correctly", @@ -74,9 +74,9 @@ "Phase 2: Node.js ≥ 20.6.0 confirmed", "Phase 2: Package is verified in package.json dependencies", "Phase 3: Entry point file is read before editing", - "Phase 3: useMicrosoftOpenTelemetry, shutdownMicrosoftOpenTelemetry, configureA365Hosting, and AgenticTokenCacheInstance imported from @microsoft/opentelemetry (single package — NOT from the legacy -observability/-hosting/-runtime packages)", - "Phase 3: useMicrosoftOpenTelemetry({ a365: { enabled: true, enableObservabilityExporter: true, tokenResolver } }) called BEFORE any LLM/framework imports — both a365.enabled and a365.enableObservabilityExporter are required in 1.0+ to actually export spans", - "Phase 3: tokenResolver wired to AgenticTokenCacheInstance.getObservabilityToken(agentId, tenantId) ?? ''", + "Phase 3: useMicrosoftOpenTelemetry, shutdownMicrosoftOpenTelemetry, and configureA365Hosting imported from @microsoft/opentelemetry (single package — NOT from the legacy -observability/-hosting/-runtime packages); AgenticTokenCacheInstance is NOT imported for telemetry", + "Phase 3: useMicrosoftOpenTelemetry({ a365: { enabled: true, enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver } }) called BEFORE any LLM/framework imports — a365.enabled and a365.enableObservabilityExporter are both required in 1.0+ to actually export spans, and useS2SEndpoint: true is required in every auth mode", + "Phase 3: observability/app-token-resolver.ts is written and tokenResolver is wired to createAppTokenResolver(() => getObsConnection()); getObsConnection is assigned to adapter.connectionManager.getDefaultConnection() once the adapter exists", "Phase 3: configureA365Hosting(adapter, { enableBaggage: true }) called once at startup to register BaggageMiddleware", "Phase 3: SIGTERM/SIGINT handlers wired to call await shutdownMicrosoftOpenTelemetry() so pending spans flush on shutdown", "Phase 3: No manual OpenAIAgentsTraceInstrumentor.enable() or LangChainTraceInstrumentor.instrument() call — auto-enabled in 1.0+ and manual calls cause duplicate spans", @@ -85,9 +85,9 @@ "Phase 4: Message handler file is read before editing", "Phase 4: getObservabilityAuthenticationScope is NOT imported — default scope auto-applied (api://9b975845-388f-4429-889e-eab1ef63949c/.default)", "Phase 4: agentId resolved from turnContext.activity?.recipient?.agenticAppId (not from config); tenantId from turnContext.activity?.recipient?.tenantId", - "Phase 4: AgenticTokenCacheInstance.RefreshObservabilityToken called per turn with (agentId, tenantId, turnContext, agentApplication.authorization) — passes the auth object, NOT an auth-handler name string", - "Phase 4 / 5.5: Canonical baggage pattern used — handler calls `preloadObservabilityToken(turnContext)` FIRST, then builds outer `BaggageBuilderUtils.fromTurnContext(new BaggageBuilder(), turnContext as any).build()` and runs `baggageScope.run(async () => { ... })` with `InvokeAgentScope.start(...)` + `InferenceScope.start(...)` INSIDE the callback. Without this exact wrapping order, spans get filtered as 'Partitioned into 0 identity groups (N spans skipped)' and are not exported — the #1 first-run failure mode. `BaggageBuilder` AND `BaggageBuilderUtils` both imported from `@microsoft/opentelemetry`.", - "Phase 4: `preloadObservabilityToken(turnContext)` helper extracts the agentId/tenantId resolution + `AgenticTokenCacheInstance.refreshObservabilityToken(...)` call, called BEFORE entering the baggage scope so cold-turn spans don't see an empty token and silently time out", + "Phase 4: No per-turn token refresh — AgenticTokenCacheInstance.refreshObservabilityToken(..., authorization) is NOT called (it exchanges a delegated token, which the S2S route rejects)", + "Phase 4 / 5.5: Canonical baggage pattern used — handler builds outer `BaggageBuilderUtils.fromTurnContext(new BaggageBuilder(), turnContext as any).build()` and runs `baggageScope.run(async () => { ... })` with `InvokeAgentScope.start(...)` + `InferenceScope.start(...)` INSIDE the callback. Without this exact wrapping, spans get filtered as 'Partitioned into 0 identity groups (N spans skipped)' and are not exported — the #1 first-run failure mode. `BaggageBuilder` AND `BaggageBuilderUtils` both imported from `@microsoft/opentelemetry`.", + "Phase 4: No preloadObservabilityToken helper is added — the app-only resolver acquires and caches tokens on demand, so cold-turn spans export without a preload", "Phase 4: authMode recorded as inline comment in the message handler", "Phase 4: All new lines include marker comment", "Phase 4: Existing handler logic is preserved", @@ -114,6 +114,7 @@ "Phase 2: Packages are already present — installation is skipped or succeeds", "Phase 3: Entry point is checked for existing instrumentation", "Phase 3: Marker comment '// A365 Observability — best-effort instrumentation' is detected", + "Phase 3: Existing wiring counts as complete only when o.Agent365.UseS2SEndpoint = true and AgentAppTokenResolver (or the S2S token-service scaffold) are present and no AgenticTokenStruct / RegisterObservability(..., AgenticTokenStruct) remains; otherwise the delegated wiring is migrated instead of skipped", "Phase 3: No duplicate AddA365Tracing() calls are added", "Phase 4: Message handler is checked for existing BaggageBuilder or BaggageTurnMiddleware code", "Phase 4: No duplicate baggage calls are added", @@ -135,6 +136,7 @@ "Phase 2: Packages are already in package.json — installation succeeds quickly", "Phase 3: Entry point already has ObservabilityManager.configure()", "Phase 3: Marker comment is detected — no duplicate code added", + "Phase 3: Existing wiring counts as complete only when useS2SEndpoint: true and an app-only tokenResolver are present and no refreshObservabilityToken(..., authorization) / AgenticTokenCacheInstance.getObservabilityToken remains; otherwise the delegated wiring is migrated instead of skipped", "Phase 4: Message handler already has baggage context", "Phase 4: No duplicate baggage calls are added", "Phase 6: .env already has ENABLE_A365_OBSERVABILITY_EXPORTER, SERVICE_NAME, A365_OBSERVABILITY_LOG_LEVEL — only missing values updated", @@ -163,7 +165,7 @@ "id": 6, "prompt": "Add A365 observability to this Python agent", "description": "Test on a Python agent project with requirements.txt and a message handler", - "expected_output": "The skill detects Python, determines agentType and authMode via two-stage question, installs the unified microsoft-opentelemetry package (single GA 1.1+ distro — legacy microsoft-agents-a365-observability-core / -hosting / -runtime / -extensions-* are deprecated and must NOT be installed alongside), wires use_microsoft_opentelemetry(enable_a365=True, a365_enable_observability_exporter=True, a365_token_resolver=...) in the entry point with AgenticTokenCache().get_observability_token as the resolver, calls ObservabilityHostingManager.configure(adapter.middleware_set, ObservabilityHostingOptions(enable_baggage=True)) once at startup (enable_baggage defaults to False — must be set explicitly), adds _setup_observability_token helper to the message handler that calls self.agent_app.auth.exchange_token with auth_handler_id from config (not hardcoded 'AGENTIC') and cache_agentic_token on the result, resolves agent_id dynamically from context.activity.recipient.agentic_app_id, updates .env with ENABLE_A365_OBSERVABILITY_EXPORTER, and validates via import check.", + "expected_output": "The skill detects Python, determines agentType and authMode via two-stage question, installs the unified microsoft-opentelemetry package (single GA 1.1+ distro — legacy microsoft-agents-a365-observability-core / -hosting / -runtime / -extensions-* are deprecated and must NOT be installed alongside), writes observability/app_token_resolver.py and wires use_microsoft_opentelemetry(enable_a365=True, a365_enable_observability_exporter=True, a365_use_s2s_endpoint=True, a365_token_resolver=OBS_TOKENS.resolve) in the entry point, calls ObservabilityHostingManager.configure(adapter.middleware_set, ObservabilityHostingOptions(enable_baggage=True)) once at startup (enable_baggage defaults to False — must be set explicitly), adds a _setup_observability_token helper that awaits OBS_TOKENS.prefetch(self.connection_manager, tenant_id, agent_id) (hosting connection's get_agentic_application_token + agent-identity client_credentials) and does NOT call exchange_token for the observability scope, resolves agent_id dynamically from context.activity.recipient.agentic_app_id, updates .env with ENABLE_A365_OBSERVABILITY_EXPORTER, and validates via import check.", "files": [], "expectations": [ "Phase 1: Agent detection identifies Python correctly", @@ -183,15 +185,15 @@ "Phase 2: Package is verified in requirements.txt or pyproject.toml", "Phase 3: Entry point file is read before editing", "Phase 3: use_microsoft_opentelemetry imported from microsoft.opentelemetry (the single unified package — NOT from the legacy microsoft_agents_a365.* namespace) and called at startup with BOTH enable_a365=True AND a365_enable_observability_exporter=True (both required in 1.0+ to actually export spans)", - "Phase 3: a365_token_resolver wired to AgenticTokenCache().get_observability_token (or a local token_cache.get_cached_agentic_token wrapper)", + "Phase 3: observability/app_token_resolver.py is written; OBS_TOKENS = AppTokenResolver() and use_microsoft_opentelemetry(...) receives a365_token_resolver=OBS_TOKENS.resolve and a365_use_s2s_endpoint=True (no AgenticTokenCache or cache of delegated tokens)", "Phase 3: ObservabilityHostingManager.configure(adapter.middleware_set, ObservabilityHostingOptions(enable_baggage=True)) called once at startup — enable_baggage defaults to False, must be set explicitly", "Phase 3: No legacy *Instrumentor().instrument() calls for LangChain / OpenAI / SK / AgentFramework — auto-enabled in 1.0+ and manual calls cause duplicate spans", "Phase 3: All new lines include marker comment: # A365 Observability — best-effort instrumentation (verify against official sample)", "Phase 3: Existing code is preserved", "Phase 4: Message handler file is read before editing", "Phase 4: agent_id resolved from context.activity.recipient.agentic_app_id (not from config); tenant_id from context.activity.recipient.tenant_id", - "Phase 4: _setup_observability_token helper added — calls self.agent_app.auth.exchange_token() with auth_handler_id=self.auth_handler_name (from config, NOT hardcoded 'AGENTIC') and cache_agentic_token() on the result", - "Phase 4: get_observability_authentication_scope is NOT imported unless overriding the default scope — default api://9b975845-388f-4429-889e-eab1ef63949c/.default is auto-applied in 1.1+", + "Phase 4: _setup_observability_token helper added — awaits OBS_TOKENS.prefetch(self.connection_manager, tenant_id, agent_id) inside try/except; does NOT call exchange_token() for the observability scope or cache_agentic_token()", + "Phase 4: get_observability_authentication_scope is NOT imported — the app-only resolver requests api://9b975845-388f-4429-889e-eab1ef63949c/.default itself", "Phase 4: No manual populate(builder, context) baggage call in the handler — ObservabilityHostingManager (registered in Phase 3) auto-populates baggage from TurnContext", "Phase 4: authMode recorded as inline comment in the message handler", "Phase 4: All new lines include marker comment", @@ -243,6 +245,7 @@ "Phase 2: Azure.Identity is required — used for ManagedIdentityCredential in the MSI path of ObservabilityTokenService", "Phase 2: Microsoft.OpenTelemetry installed without --version flag (GA 1.0.2+); no Microsoft.Extensions.Logging 10.0.0 workaround needed (fixed in 1.0.1); no net9.0 TFM upgrade required (net8.0 supported)", "Phase 3: Observability/ObservabilityServiceExtensions.cs is created with Agent365ObservabilityContext and AddAgent365Observability() extension", + "Phase 3: o.Agent365.UseS2SEndpoint = true is set in the UseMicrosoftOpenTelemetry options (o.Agent365.Exporter.UseS2SEndpoint only on Microsoft.OpenTelemetry 1.0.2 and earlier)", "Phase 3: Observability/ObservabilityTokenService.cs is created with MSAL ConfidentialClientApplicationBuilder.WithFmiPath() targeting api://9b975845-388f-4429-889e-eab1ef63949c/.default and 50-minute refresh loop", "Phase 3: Program.cs calls builder.Services.AddAgent365Observability() — NOT AddAgenticTracingExporter()", "Phase 3: Program.cs calls builder.UseMicrosoftOpenTelemetry()", @@ -332,6 +335,23 @@ "Phase 8: Final summary lists agent kind (Agent (Non AI Teammate)), authMode (s2s), scaffold file created, packages installed", "Phase 8: User is reminded to populate AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID in .env" ] + }, + { + "id": 11, + "prompt": "Add A365 observability to this agent", + "description": "Node.js LangChain AI Teammate that already wires observability the legacy delegated way: useMicrosoftOpenTelemetry without useS2SEndpoint, tokenResolver = AgenticTokenCacheInstance.getObservabilityToken, and a per-turn preloadObservabilityToken helper that calls AgenticTokenCacheInstance.refreshObservabilityToken(agentId, tenantId, turnContext, this.authorization)", + "expected_output": "The skill treats the delegated telemetry wiring as has_obs_partial and migrates it to the S2S route with an app-only token. It writes observability/app-token-resolver.ts, switches tokenResolver to the app-only resolver, adds useS2SEndpoint: true, and removes the preloadObservabilityToken helper and refreshObservabilityToken call. The baggage and scope wiring stay intact, and workload (MCP) OBO auth is untouched. It recommends a365.durableDelivery: { enabled: false } during the migration on @microsoft/opentelemetry 1.4.x, and lists every replaced call in the final summary. It does not ask for the delegated OtelWrite scope or a consent flow. If the AI Teammate's OtelWrite application-role step is still pending, it points to the action item that a365 setup all --aiteammate printed.", + "files": [], + "expectations": [ + "Idempotency / Phase 3: marker comments alone do not short-circuit — the missing S2S route flag and the delegated refresh trigger the delegated-telemetry migration instead of a skip", + "Phase 3: observability/app-token-resolver.ts is written and tokenResolver is switched from AgenticTokenCacheInstance.getObservabilityToken to the app-only resolver", + "Phase 3: useS2SEndpoint: true is added to the a365 options", + "Phase 3: durableDelivery: { enabled: false } is recommended or applied during the migration so spooled delegated-route records are not replayed", + "Phase 4: the preloadObservabilityToken helper and the AgenticTokenCacheInstance.refreshObservabilityToken(..., this.authorization) call are removed; BaggageBuilder / InvokeAgentScope wiring is preserved", + "Phase 4: workload OBO calls (MCP tool registration via agentApplication.authorization) are unchanged", + "Phase 7: validate-instrument-observability.js passes (S2S route flag present, no delegated refresh left)", + "Phase 9: final summary lists each replaced delegated call, states that no delegated OBS consent is needed, and (AI Teammate) points to any pending OtelWrite application-role step from a365 setup all --aiteammate" + ] } ] } diff --git a/plugins/agent365/hooks/stop/validate-a365-code-validator.js b/plugins/agent365/hooks/stop/validate-a365-code-validator.js index bdf1de7..e9b42bb 100644 --- a/plugins/agent365/hooks/stop/validate-a365-code-validator.js +++ b/plugins/agent365/hooks/stop/validate-a365-code-validator.js @@ -164,29 +164,56 @@ function validatePython() { ); } - if (hasDistroCall && hasS2SIntent && hasExplicitS2SFalse) { + // Every auth mode exports over the S2S route; the legacy delegated route rejects app-only + // tokens and needs admin consent. Contextual/agent-identity resolvers imply S2S intent too. + const expectsS2S = hasDistroCall && (hasEnableA365 || hasS2SIntent); + if (expectsS2S && hasExplicitS2SFalse) { add( 'critical', 'python-s2s-endpoint-disabled', - 'Python appears to use S2S/Agent Identity export but passes a365_use_s2s_endpoint=False. S2S agents must use service-to-service endpoint mode.', + 'Python passes a365_use_s2s_endpoint=False. A365 export must use the S2S route in every auth mode; the delegated route rejects app-only tokens and needs admin consent.', pyFiles.find(f => /a365_use_s2s_endpoint\s*=\s*False\b/.test(read(f))) ); - } else if (hasDistroCall && hasS2SIntent && !hasExplicitS2S && hasS2SEnv) { + } else if (expectsS2S && !hasExplicitS2S && hasS2SEnv) { add( 'medium', 'python-s2s-endpoint-env-dependent', - 'Python appears to use S2S/Agent Identity export but depends on A365_USE_S2S_ENDPOINT=true in env. Prefer passing a365_use_s2s_endpoint=True in code.', + 'Python depends on A365_USE_S2S_ENDPOINT=true in env to select the S2S route. Prefer passing a365_use_s2s_endpoint=True in code.', pyFiles.find(f => fileContains(f, 'use_microsoft_opentelemetry')) ); - } else if (hasDistroCall && hasS2SIntent && !hasExplicitS2S && !hasS2SEnv) { + } else if (expectsS2S && !hasExplicitS2S && !hasS2SEnv) { add( 'high', 'python-s2s-endpoint-not-set', - 'Python appears to use S2S/Agent Identity export but does not set a365_use_s2s_endpoint=True or A365_USE_S2S_ENDPOINT=true.', + 'Python does not set a365_use_s2s_endpoint=True (or A365_USE_S2S_ENDPOINT=true), so export uses the legacy delegated route. Every auth mode must export over the S2S route with an app-only token.', pyFiles.find(f => fileContains(f, 'use_microsoft_opentelemetry')) ); } + for (const file of pyFiles) { + const content = read(file); + const delegated = findCallBlocks(content, 'exchange_token').some(block => /observability/i.test(block)) || + /(? findCallBlocks(read(f), 'prefetch').some(block => /self\.connection_manager/.test(block))); + if (prefetchFile && !anyFileMatches(pyFiles, /\bself\.connection_manager\s*=/)) { + add( + 'high', + 'python-obs-prefetch-connection-missing', + 'The app-only token prefetch uses self.connection_manager, but no file assigns it. CloudAdapter does not expose its connection manager, so every prefetch fails and no spans export. Store the MsalConnectionManager passed to CloudAdapter on the host.', + prefetchFile + ); + } + for (const file of pyFiles) { const content = read(file); for (const block of findCallBlocks(content, 'a365_request_scope')) { @@ -299,6 +326,30 @@ function validateNode() { ); } + // Every auth mode exports over the S2S route with an app-only token. + if (hasDistroCall && hasA365Enabled && !anyFileMatches(tsFiles, /\buseS2SEndpoint\s*:\s*true\b/)) { + add( + 'high', + 'node-obs-delegated-route', + 'Node code does not set useS2SEndpoint: true, so A365 export uses the legacy delegated route. Every auth mode must export over the S2S route with an app-only tokenResolver.', + tsFiles.find(f => fileContains(f, 'useMicrosoftOpenTelemetry')) + ); + } + for (const file of tsFiles) { + const content = read(file); + const delegatedRefresh = ['refreshObservabilityToken', 'RefreshObservabilityToken'] + .some(name => findCallBlocks(content, name).some(block => /authorization/i.test(block))) || + /AgenticTokenCacheInstance\s*\.\s*getObservabilityToken\s*\(/.test(content); + if (delegatedRefresh) { + add( + 'high', + 'node-obs-delegated-token', + 'Telemetry uses a delegated (OBO) token (refreshObservabilityToken(..., authorization) or AgenticTokenCacheInstance.getObservabilityToken). The S2S route rejects delegated tokens; use an app-only tokenResolver for the agent identity instead.', + file + ); + } + } + const hasSemanticSpans = anyFileContains(tsFiles, 'InvokeAgentScope') || anyFileContains(tsFiles, 'InferenceScope') || anyFileContains(tsFiles, 'ExecuteToolScope') || @@ -357,6 +408,29 @@ function validateDotnet() { 'No InvokeAgentScope found. MAC Activity needs invoke_agent parent spans to anchor chat/tool/inference activity.' ); } + + // Every auth mode exports over the S2S route with an app-only token. + const csFiles = allFiles.filter(f => f.endsWith('.cs')); + if (anyFileContains(csFiles, 'UseMicrosoftOpenTelemetry') && !anyFileMatches(csFiles, /\bUseS2SEndpoint\s*=\s*true\b/)) { + add( + 'high', + 'dotnet-obs-delegated-route', + 'UseMicrosoftOpenTelemetry is wired without UseS2SEndpoint = true, so A365 export uses the legacy delegated route. Set o.Agent365.UseS2SEndpoint = true (o.Agent365.Exporter.UseS2SEndpoint on Microsoft.OpenTelemetry 1.0.2 and earlier) with an app-only TokenResolver in every auth mode.', + csFiles.find(f => fileContains(f, 'UseMicrosoftOpenTelemetry')) + ); + } + for (const file of csFiles) { + const content = read(file); + if (findCallBlocks(content, 'RegisterObservability').some(block => block.includes('AgenticTokenStruct')) || + /\bnew\s+AgenticTokenStruct\s*[({]|IExporterTokenCache\s*<\s*AgenticTokenStruct\s*>\s*\??\s+[A-Za-z_]\w*/.test(content)) { + add( + 'high', + 'dotnet-obs-delegated-token', + 'Telemetry uses a delegated (OBO) token (RegisterObservability with AgenticTokenStruct, new AgenticTokenStruct(...), or an IExporterTokenCache dependency). The S2S route rejects delegated tokens; wire an app-only TokenResolver for the agent identity instead.', + file + ); + } + } } function validateSetupArtifacts() { @@ -378,6 +452,15 @@ function validateSetupArtifacts() { path.join(cwd, 'a365.generated.config.json') ); } + const staticConfig = readJson(path.join(cwd, 'a365.config.json')); + if (staticConfig && staticConfig.aiTeammate === false && generated.agenticAppId && !generated.agentRegistrationId) { + add( + 'medium', + 'agent-registration-not-recorded', + 'a365.generated.config.json has an agent identity but no agentRegistrationId. The S2S route authorizes registered agent instances without an OtelWrite grant; an unregistered instance gets 403 insufficient_scope. Run a365 setup all --agent-registration-only (idempotent).', + path.join(cwd, 'a365.generated.config.json') + ); + } } } diff --git a/plugins/agent365/hooks/stop/validate-instrument-observability.js b/plugins/agent365/hooks/stop/validate-instrument-observability.js index 9ca46a6..299d645 100644 --- a/plugins/agent365/hooks/stop/validate-instrument-observability.js +++ b/plugins/agent365/hooks/stop/validate-instrument-observability.js @@ -30,6 +30,38 @@ const issues = []; const workspaceDetection = readJson(path.join(cwd, '.a365-workspace-detection.local.json')) || {}; const authMode = (workspaceDetection.authMode || '').toLowerCase(); +function read(filePath) { + try { return fs.readFileSync(filePath, 'utf8'); } catch { return ''; } +} + +function anyFileMatches(files, regex) { + return files.some(f => regex.test(read(f))); +} + +// Returns the source text of every `name(...)` call in `content`, with balanced parentheses. +function callBlocks(content, name) { + const blocks = []; + const needle = `${name}(`; + let index = 0; + while ((index = content.indexOf(needle, index)) !== -1) { + let depth = 0; + let end = -1; + for (let i = index + name.length; i < content.length; i++) { + if (content[i] === '(') depth++; + if (content[i] === ')' && --depth === 0) { end = i + 1; break; } + } + if (end === -1) break; + blocks.push(content.slice(index, end)); + index = end; + } + return blocks; +} + +// True when any `name(...)` call in `files` has arguments matching `regex`. +function anyCallMatches(files, name, regex) { + return files.some(f => callBlocks(read(f), name).some(block => regex.test(block))); +} + // ── Detect project type ───────────────────────────────────────────────────── // Walk the project tree once, then bucket by name. @@ -76,7 +108,7 @@ if (isDotnet) { // 2. Program.cs wired // Preferred (Microsoft.OpenTelemetry distro): UseMicrosoftOpenTelemetry covers both OBO and S2S - // - OBO/agentic-user: distro auto-registers IExporterTokenCache; no extra DI calls + // - OBO/agentic-user: UseMicrosoftOpenTelemetry + AgentAppTokenResolver (app-only token, S2S route) // - S2S: UseMicrosoftOpenTelemetry + AddAgent365Observability for the scaffold token service // Legacy (pre-distro, kept for older agents): AddA365Tracing + AddAgenticTracingExporter (OBO) // or AddA365Tracing + AddAgent365Observability (S2S) @@ -132,6 +164,16 @@ if (isDotnet) { } } + // 3b. Telemetry uses the S2S route with an app-only token in every auth mode. + // The S2S route rejects delegated (scp) tokens, and the distro defaults to the delegated route. + if (hasDistroWired && !anyFileMatches(csFiles, /\bUseS2SEndpoint\s*=\s*true\b/)) { + issues.push('Observability export must use the S2S route in every auth mode: set o.Agent365.UseS2SEndpoint = true in UseMicrosoftOpenTelemetry (o.Agent365.Exporter.UseS2SEndpoint on Microsoft.OpenTelemetry 1.0.2 and earlier) and wire an app-only token resolver (AgentAppTokenResolver, or ObservabilityTokenService for s2s)'); + } + if (anyCallMatches(csFiles, 'RegisterObservability', /AgenticTokenStruct/) || + anyFileMatches(csFiles, /\bnew\s+AgenticTokenStruct\s*[({]|IExporterTokenCache\s*<\s*AgenticTokenStruct\s*>\s*\??\s+[A-Za-z_]\w*/)) { + issues.push('RegisterObservability(..., AgenticTokenStruct), new AgenticTokenStruct(...), or an IExporterTokenCache dependency wires a delegated (OBO) telemetry token, which the S2S route rejects — remove the per-turn registration and use AgentAppTokenResolver as o.Agent365.TokenResolver (see dotnet-observability.md)'); + } + // 4. appsettings has observability config const appSettingsFiles = filterByName(allFiles, 'appsettings.json'); const hasAppSettingsConfig = anyFileContains(appSettingsFiles, @@ -219,6 +261,17 @@ if (isNodejs) { } } + // 4b. Telemetry uses the S2S route with an app-only token in every auth mode. + // The S2S route rejects delegated (scp) tokens, and the distro defaults to the delegated route. + if (usesDistro && !anyFileMatches(tsFiles, /\buseS2SEndpoint\s*:\s*true\b/)) { + issues.push('Observability export must use the S2S route in every auth mode: pass useS2SEndpoint: true in the a365 options of useMicrosoftOpenTelemetry() with an app-only tokenResolver (observability/app-token-resolver.ts for obo / agentic-user)'); + } + if (anyCallMatches(tsFiles, 'refreshObservabilityToken', /authorization/i) || + anyCallMatches(tsFiles, 'RefreshObservabilityToken', /authorization/i) || + anyFileMatches(tsFiles, /AgenticTokenCacheInstance\s*\.\s*getObservabilityToken\s*\(/)) { + issues.push('refreshObservabilityToken(..., authorization) or AgenticTokenCacheInstance.getObservabilityToken(...) feeds a delegated (OBO) telemetry token, which the S2S route rejects — remove it (and any preloadObservabilityToken helper) and use the app-only tokenResolver (see nodejs-observability.md)'); + } + // 5. .env has observability vars const hasEnvConfig = envFiles.some(f => fileContains(f, 'ENABLE_A365_OBSERVABILITY_EXPORTER')); @@ -309,6 +362,24 @@ if (isPython) { } } + // 4b. Telemetry uses the S2S route with an app-only token in every auth mode. + // The S2S route rejects delegated (scp) tokens, and the distro defaults to the delegated route. + const pyS2SInCode = anyFileMatches(pyFiles, /\ba365_use_s2s_endpoint\s*=\s*True\b/); + const pyS2SInEnv = anyFileMatches(envFiles, /^\s*A365_USE_S2S_ENDPOINT\s*=\s*(true|1|yes)\s*$/im); + if (usesDistroPy && !pyS2SInCode && !pyS2SInEnv) { + issues.push('Observability export must use the S2S route in every auth mode: pass a365_use_s2s_endpoint=True to use_microsoft_opentelemetry() with an app-only a365_token_resolver (observability/app_token_resolver.py for obo / agentic-user)'); + } + if (anyCallMatches(pyFiles, 'exchange_token', /observability/i) || + anyFileMatches(pyFiles, /(? fileContains(f, 'ENABLE_A365_OBSERVABILITY_EXPORTER')); diff --git a/plugins/agent365/shared/agent-detection.md b/plugins/agent365/shared/agent-detection.md index 2cc735f..f38c4c1 100644 --- a/plugins/agent365/shared/agent-detection.md +++ b/plugins/agent365/shared/agent-detection.md @@ -446,14 +446,14 @@ The cache is written in stages as values become known — always preserve fields - `has_aiteammate_structure`: `1` if any AI Teammate structure signal matches; `0` otherwise: - `AgentApplication` in source files, `CloudAdapter`/`CloudAdapterAiohttp`, `@microsoft/agents-a365-notifications` in `package.json`, `Microsoft.Agents.A365.Notifications` in `.csproj`, or `ToolingManifest.json` exists. -- `has_obs`: `1` ONLY when **all three** observability anchors are present (entry-point + token resolver + handler-side baggage / scope). The entry-point call alone (`UseMicrosoftOpenTelemetry` / `useMicrosoftOpenTelemetry` / `use_microsoft_opentelemetry`) is **not sufficient** — a project can have the call without `tokenResolver` (exports silently fail to auth) or without `BaggageBuilder` / `InvokeAgentScope` (no identity grouping, fails store-publish validation). Anchor sets per language: - - **.NET (`**/*.cs`)**: entry = `UseMicrosoftOpenTelemetry` *(distro auto-registers `IExporterTokenCache` for OBO so the call itself doubles as the token signal)*; handler = `BaggageBuilder` OR `BaggageTurnMiddleware` OR `InvokeAgentScope.Start`. - - **Node.js (`src/**/*.ts`)**: entry = `useMicrosoftOpenTelemetry`; token = `tokenResolver` OR `AgenticTokenCacheInstance` (OBO) OR `getS2SObservabilityToken` / `startTokenService` (S2S); handler = `BaggageBuilder` OR `BaggageBuilderUtils` OR `InvokeAgentScope`. - - **Python (`**/*.py`)**: entry = `use_microsoft_opentelemetry`; token = `token_resolver` OR `AgenticTokenCache` OR `cache_agentic_token` OR S2S: `run_token_service` / `get_s2s_observability_token`; handler = `BaggageBuilder` OR `populate_baggage` OR `InvokeAgentScope`. +- `has_obs`: `1` ONLY when **all four** observability anchors are present (entry-point + app-only token resolver + S2S route flag + handler-side baggage / scope). The entry-point call alone (`UseMicrosoftOpenTelemetry` / `useMicrosoftOpenTelemetry` / `use_microsoft_opentelemetry`) is **not sufficient** — a project can have the call without `tokenResolver` (exports silently fail to auth), without the S2S route flag (telemetry still goes to the legacy delegated route, which rejects app-only tokens), or without `BaggageBuilder` / `InvokeAgentScope` (no identity grouping, fails store-publish validation). Telemetry is exported over the S2S route with an app-only token in **every** auth mode. Anchor sets per language: + - **.NET (`**/*.cs`)**: entry = `UseMicrosoftOpenTelemetry`; token = `AgentAppTokenResolver` (obo / agentic-user) OR `ObservabilityTokenService` / `AddAgent365Observability` (S2S), with no `AgenticTokenStruct` usage (per-turn `RegisterObservability(..., new AgenticTokenStruct(...))`, `new AgenticTokenStruct(...)`, or an `IExporterTokenCache` dependency); route = `UseS2SEndpoint = true`; handler = `BaggageBuilder` OR `BaggageTurnMiddleware` OR `InvokeAgentScope.Start`. + - **Node.js (`src/**/*.ts`)**: entry = `useMicrosoftOpenTelemetry`; token = `tokenResolver` (app-only: `observability/app-token-resolver.ts`, or `getS2SObservabilityToken` / `startTokenService` for S2S), with no `refreshObservabilityToken(..., authorization)` or `AgenticTokenCacheInstance.getObservabilityToken` resolver; route = `useS2SEndpoint: true`; handler = `BaggageBuilder` OR `BaggageBuilderUtils` OR `InvokeAgentScope`. + - **Python (`**/*.py`)**: entry = `use_microsoft_opentelemetry`; token = `token_resolver` (app-only: `AppTokenResolver` / `OBS_TOKENS`, or S2S `run_token_service` / `get_s2s_observability_token`), with no `exchange_token(...)` for the observability scope, `cache_agentic_token(...)`, or `AgenticTokenCache` / `get_cached_agentic_token` resolver; route = `a365_use_s2s_endpoint=True` (or `A365_USE_S2S_ENDPOINT=true` in `.env`); handler = `BaggageBuilder` OR `populate_baggage` OR `InvokeAgentScope`. Package-name matches alone (`Microsoft.Agents.A365.Observability.*`, `@microsoft/agents-a365-observability`, `microsoft-agents-a365-observability-*`, `Microsoft.OpenTelemetry` / `@microsoft/opentelemetry` / `microsoft-opentelemetry`) do NOT count — they reflect installation, not invocation. - **Partial wiring** (entry-point present but not the full anchor set) is treated as `has_obs = 0` *and* surfaces a separate read-time `has_obs_partial = 1` signal so `make-ai-teammate` Phase 9.5 can re-enter `instrument-observability` to complete the wiring rather than silently skip. `has_obs_partial` is computed at read-time from disk and is **not** persisted to the cache. + **Partial wiring** (entry-point present but not the full anchor set) is treated as `has_obs = 0` *and* surfaces a separate read-time `has_obs_partial = 1` signal so `make-ai-teammate` Phase 9.5 can re-enter `instrument-observability` to complete the wiring rather than silently skip. Legacy delegated telemetry (no S2S route flag, or a per-turn delegated token refresh / registration) is partial too, so re-entering migrates it to the S2S route with an app-only token. `has_obs_partial` is computed at read-time from disk and is **not** persisted to the cache. - `has_workiq`: `1` ONLY when **both** signals are true: 1. `ToolingManifest.json` exists AND its top-level `mcpServers` (or legacy `servers`) array is non-empty. 2. The agent code references the framework-appropriate MCP wiring symbol: `addToolServersToAgent` (Node.js any stack), `GetMcpToolsAsync` (.NET AF), `AddToolServersToAgentAsync` (.NET SK), or `add_tool_servers_to_agent` (Python). @@ -482,7 +482,9 @@ Use the **Write** tool to write the merged object back to `.a365-workspace-detec ## AGENTIC_APP_ID Requirement -The A365 observability token resolver requires `AGENTIC_APP_ID` to authenticate. +Agent code such as the WorkIQ / MCP tooling path reads the agent's app ID from `AGENTIC_APP_ID`. +(Observability does not use it: the app-only telemetry token is minted for the turn's agent +identity, or for the configured agent identity on the `s2s` path — never for the blueprint.) Detection order: 1. Check `.env` or `.env.example` for `AGENTIC_APP_ID=` @@ -599,7 +601,9 @@ The `authMode` value (`obo`, `s2s`, or `agentic-user`) drives which code path is - `agentic-user` — agent's own M365 identity (persistent Azure AD user); same OBO wire-up as `obo` but the identity is the agent, not the signed-in human - `s2s` — service principal; no per-turn user token; scaffold token-service file handles credential acquisition. (Note: "autonomous" is a separate axis — an autonomous agent can use either OBO or S2S auth.) -For `obo` and `agentic-user` paths: auth handler name comes from config (`AgentApplication:AgenticAuthHandlerName` in .NET, `agentApplication.authorization` object in Node.js, `auth_handler_id` from config in Python) — never hardcode `"AGENTIC"`. Agent IDs are always resolved dynamically from TurnContext (`agenticAppId` / `agentic_app_id`), never from config. +For `obo` and `agentic-user` paths: auth handler name comes from config (`AgentApplication:AgenticAuthHandlerName` in .NET, `agentApplication.authorization` object in Node.js, `auth_handler_id` from config in Python) — never hardcode `"AGENTIC"`. Agent IDs are resolved dynamically from TurnContext (`agenticAppId` / `agentic_app_id`). The only config fallback is for non-agentic turns of non-AI-Teammate agents, which use the provisioned agent identity that `a365 setup all` writes (`Agent365Observability:AgentId`), never the blueprint ID. + +**Observability is the same in every mode:** export always goes to the S2S route with an app-only token for the exporting agent identity. `authMode` only decides where that token comes from: the hosting connection's blueprint credential (`obo` / `agentic-user`), or the background FMI token-service scaffold (`s2s`). The auth handler (OBO / Agentic User token) is used for workload calls (MCP / Graph) only. The S2S route rejects delegated (`scp`) tokens, so never feed an OBO / Agentic User token to the exporter. Add this inline comment wherever the auth handler is wired: diff --git a/plugins/agent365/skills/a365-code-validator/SKILL.md b/plugins/agent365/skills/a365-code-validator/SKILL.md index 9cc4b10..3346cd6 100644 --- a/plugins/agent365/skills/a365-code-validator/SKILL.md +++ b/plugins/agent365/skills/a365-code-validator/SKILL.md @@ -8,7 +8,8 @@ description: > before shipping. Diagnoses and fixes existing instrumentation; do NOT use it to add observability from scratch — use instrument-observability for that. Checks exporter activation, runtime agent-identity binding (vs blueprint id), - the S2S FMI / OBO token shape, S2S vs OBO endpoint selection, the required semantic spans + the app-only S2S token shape (and any leftover delegated OBO telemetry token), S2S endpoint + selection in every auth mode, the required semantic spans (invoke_agent/chat/execute_tool/output_messages), Activity attributes, and live blueprint grants and inheritance through read-only a365 Microsoft Graph diagnostics. Read-only by default; asks before applying safe fixes. Python, Node.js, and .NET. @@ -35,7 +36,7 @@ hooks: 1. exporter activation status; 2. identity binding status (agent id vs blueprint id); 3. semantic span coverage (invoke_agent/chat/execute_tool/output_messages); - 4. endpoint/token mode (S2S vs OBO); + 4. endpoint/token mode (S2S route with an app-only token in every auth mode, or leftover delegated telemetry); 5. live blueprint grant and effective-inheritance status, or why the check was skipped; 6. concrete next debug commands; 7. if blockers were found, the user was asked whether to fix now, create a fix plan, or stop. @@ -72,9 +73,9 @@ silent or confusing A365 Activity failures: 1. Exporter configured but not actually enabled. 2. `gen_ai.agent.id` set to a Blueprint ID instead of the runtime Agent Identity / Source Agent ID. 3. Generic HTTP/OpenAI spans emitted without A365 semantic operations. -4. S2S/OBO endpoint mismatch. +4. Delegated (OBO-route) telemetry: the S2S route flag is missing, or an OBO / Agentic User token is fed to the exporter. 5. Missing token resolver or wrong S2S token shape. -6. Stale local Blueprint ID or missing Blueprint permissions in the target tenant. +6. Stale local Blueprint ID, missing Blueprint permissions, or an unregistered agent instance in the target tenant. 7. Inheritable permission policy configured without grants on the Blueprint service principal. 8. MAC reporting expectations confused with raw ingest success. @@ -134,7 +135,7 @@ Detect: | `Microsoft.OpenTelemetry` | .NET A365 distro present | | `agentBlueprintId` / `agenticAppId` | Distinguish Blueprint ID from runtime Agent Identity | | `ENABLE_A365_OBSERVABILITY_EXPORTER` / `EnableAgent365Exporter` | Runtime exporter gate | -| `authMode` | Drives S2S vs OBO path expectations | +| `authMode` | Selects how the app-only telemetry token is sourced (export is S2S in every mode) and which workload auth applies | **Mark task complete.** @@ -293,22 +294,46 @@ this agent" reporting. If it is missing or set to the agent identity / agent use of the human caller object ID, export can succeed while caller/user Activity remains blank or incomplete. -### 3.5 S2S vs OBO transport mode +### 3.5 S2S transport mode (every auth mode) Do not expose or require internal service URLs in the report. Validate only the structural -intent: +intent. Telemetry export uses the service-to-service (S2S) route with an **app-only** token for +the runtime Agent Identity in **every** auth mode. `authMode` only changes how that token is +sourced: | Auth mode | Structural expectation | |---|---| -| S2S / application | Uses service-to-service export mode and a token for the runtime Agent Identity | -| OBO / agentic-user | Uses delegated export mode and a token whose delegated scope includes observability write | - -Report whether the code appears to select the correct transport mode (`useS2SEndpoint` / -`a365_use_s2s_endpoint` / `UseS2SEndpoint`) for S2S, without printing backend route templates. -For Python S2S, prefer `a365_use_s2s_endpoint=True` in code rather than relying only on -`A365_USE_S2S_ENDPOINT=true` in runtime environment. Also check the **token shape**: the S2S -observability token must come from the 3-hop FMI exchange (principal == runtime Agent Identity), -not a bare client-credentials call — see checklist §5 for the failure signatures. +| S2S / application | S2S export mode; app-only token from the FMI chain for the configured Agent Identity | +| OBO / agentic-user (incl. AI Teammate) | S2S export mode; app-only token for the turn's Agent Identity from the hosting connection's blueprint credential. OBO / Agentic User tokens are for workload calls (MCP / Graph) only | + +Report whether the code selects the S2S transport mode (`useS2SEndpoint: true` / +`a365_use_s2s_endpoint=True` / `o.Agent365.UseS2SEndpoint = true`) in every auth mode, without +printing backend route templates. For Python, prefer `a365_use_s2s_endpoint=True` in code rather +than relying only on `A365_USE_S2S_ENDPOINT=true` in the runtime environment. + +Flag **delegated telemetry** as `high`: the S2S route rejects any token carrying `scp`, and the +legacy delegated route needs admin consent. Signals are: + +- Node.js `AgenticTokenCacheInstance.refreshObservabilityToken(..., authorization)` (often wrapped + in a `preloadObservabilityToken` helper), or a `tokenResolver` that reads + `AgenticTokenCacheInstance.getObservabilityToken(...)`. +- .NET `RegisterObservability(..., new AgenticTokenStruct(...), ...)`, any `new AgenticTokenStruct(...)` + (for example an `A365OtelWrapper` helper), or an `IExporterTokenCache` dependency. +- Python `exchange_token(..., scopes=get_observability_authentication_scope(), ...)` feeding a + telemetry token cache, `cache_agentic_token(...)`, or an `a365_token_resolver` backed by + `AgenticTokenCache` / `get_cached_agentic_token`. +- A distro call without the S2S transport flag. + +For Python, also flag (`high`) an app-only `prefetch(self.connection_manager, ...)` when nothing +assigns `self.connection_manager`. `CloudAdapter` does not expose its connection manager, so +every prefetch fails. + +The fix is the app-only resolver from the `instrument-observability` references, not an OBS +permission grant. + +Also check the **token shape**: the S2S observability token must come from the FMI exchange +(principal == runtime Agent Identity, no `scp`), not a bare client-credentials call — see +checklist §5 for the failure signatures. ### 3.6 Blueprint permission inheritance wording @@ -372,23 +397,28 @@ do not print either real ID in the support-safe report: | Neither lookup resolves a Blueprint | `high` tenant/setup gap | | Graph returns 401/403 or the caller lacks the required Entra role | `not checked` authentication/authorization gap, not proof that the Blueprint is broken | -Interpret permissions in the context of `authMode`: +Interpret observability authorization for the S2S route, which every auth mode uses: -| Auth mode | Required observability grant | +| Tenant state | Classification | |---|---| -| `obo` / `agentic-user` | Delegated `Agent365.Observability.OtelWrite` | -| `s2s` | Application role `Agent365.Observability.OtelWrite` | -| Unknown / mixed | Report which delegated scopes and app roles exist; do not assume the intended side | +| Agent instance registered (`agentRegistrationId` present in `a365.generated.config.json`, or registration confirmed by `a365 setup all`) | OK — registered instances are authorized without `Agent365.Observability.OtelWrite` (subject to service policy) | +| Application role `Agent365.Observability.OtelWrite` granted on the Blueprint (inherited by agent identities) | OK — the S2S route also accepts it | +| Neither registration nor the application role is evident | `high` — export will likely return 403 `insufficient_scope`. Blueprint agents: `a365 setup all --agent-registration-only`. AI Teammates: complete the `OtelWrite` application-role step `a365 setup all --aiteammate` prints | +| Only the delegated `Agent365.Observability.OtelWrite` scope is granted | Informational — it only matters to legacy delegated-route exporters; flag the code for migration instead (§3.5) | -Treat a missing required OtelWrite grant as `critical`. Treat a non-zero `inheritance` result -for another required resource as `high`. Permission names and resource display names are safe -to summarize, but redact tenant, Blueprint, application, service-principal, and agent IDs. +Do **not** treat a missing `OtelWrite` grant as a blocker for a registered blueprint agent: the +Agent 365 CLI no longer requests Observability API permissions for blueprint agents. Treat a +non-zero `inheritance` result for another required resource as `high`. Permission names and +resource display names are safe to summarize, but redact tenant, Blueprint, application, +service-principal, and agent IDs. For remediation, report the CLI guidance without applying it: - Run `a365 setup requirements` when the output points to a missing `wids` claim or CLI consent. -- Run the relevant `a365 setup permissions ...` flow as a Global Administrator to reconcile - grants and inheritance. +- Run `a365 setup all --agent-registration-only` (idempotent) when a blueprint agent instance is + not registered. For AI Teammates, point to the `OtelWrite` application-role step that + `a365 setup all --aiteammate` prints (the registration-only flag does not apply to them). Run the relevant `a365 setup permissions ...` flow as a Global Administrator to + reconcile other grants and inheritance. - For a stale Blueprint ID, hand off to `a365-setup` so the developer can explicitly choose the correct reuse/re-run/fresh path. Do not rewrite the ID automatically. @@ -460,6 +490,11 @@ When the code checks above pass, have the user confirm, in the *target* tenant: - **Observability resource SP present** — `az ad sp show --id 9b975845-388f-4429-889e-eab1ef63949c` returns a service principal in the tenant. A `404` / `AADSTS500011` ("resource principal … not found") means the observability app isn't provisioned there — an onboarding step, not a code fix. +- **Agent instance registration** — a 403 `insufficient_scope` from the S2S route with a valid + app-only token means the instance is not registered (and has no `OtelWrite` application role). + For blueprint agents, registration is the expected fix, not a permission grant: + `a365 setup all --agent-registration-only`. For AI Teammates, complete the `OtelWrite` + application-role step that `a365 setup all --aiteammate` prints. - **Ingestion lag** — Defender `CloudAppEvents` populates before the admin center; give it ~5 min. **Mark task complete: "Check live blueprint permissions and runtime state".** @@ -581,8 +616,12 @@ Only apply these automatically when the user chooses `apply_safe_fixes`: ```text ENABLE_A365_OBSERVABILITY_EXPORTER=true ``` - - For S2S mode, add the version-appropriate S2S transport setting only if the code/reference for - that project already names it. Do not invent a setting name. + - Add the S2S transport setting (required in every auth mode) only if the code/reference for + that project already names it **and** the scan reported no delegated-token finding + (`*-obs-delegated-token`), i.e. the exporter already gets an app-only token. Moving an + exporter that still sends a delegated token to the S2S route breaks export, because the S2S + route rejects `scp` tokens. In that case, bundle the flag with the resolver swap below, which + needs a second confirmation. Do not invent a setting name. 3. **Config field scaffolding for runtime agent identity** - If the project already has a settings/config class, add a clearly named optional field such as @@ -601,6 +640,11 @@ Ask a focused follow-up before editing: 2. **Token resolver wiring** - If the project already has a token provider with the required exchange flow, ask whether to wire an observability token resolver to it. + - If the exporter is fed a delegated (OBO / Agentic User) token, propose replacing it with the + app-only resolver scaffold from the `instrument-observability` references (hosting-connection + resolver for `obo` / `agentic-user`, FMI token service for `s2s`), together with the S2S + transport flag. Change both at once, never the flag alone. Do not request an OBS + permission grant as the fix. - If no such provider exists, do not generate a full token service silently. Provide a plan and ask the user to confirm a follow-up implementation. diff --git a/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js b/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js index f20f52b..da04cbd 100644 --- a/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js +++ b/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js @@ -136,12 +136,28 @@ function validatePython() { if (envFalse('ENABLE_A365_OBSERVABILITY_EXPORTER') || envFalse('EnableAgent365Exporter')) { add('high', 'exporter-env-disabled', 'An env file explicitly disables A365 export. This is fine for local console-only runs, but production/MAC Activity requires the exporter to be true.'); } - if (hasDistro && s2sIntent && s2sFalse) { - add('critical', 'python-s2s-endpoint-disabled', 'Python appears to use S2S/Agent Identity export but passes a365_use_s2s_endpoint=False.', py.find(f => /a365_use_s2s_endpoint\s*=\s*False\b/.test(read(f)))); - } else if (hasDistro && s2sIntent && !s2sTrue && s2sEnv) { - add('medium', 'python-s2s-endpoint-env-dependent', 'Python appears to use S2S/Agent Identity export but depends on A365_USE_S2S_ENDPOINT=true in env. Prefer a365_use_s2s_endpoint=True in code.', py.find(f => read(f).includes('use_microsoft_opentelemetry'))); - } else if (hasDistro && s2sIntent && !s2sTrue && !s2sEnv) { - add('high', 'python-s2s-endpoint-not-set', 'Python appears to use S2S/Agent Identity export but does not set a365_use_s2s_endpoint=True or A365_USE_S2S_ENDPOINT=true.', py.find(f => read(f).includes('use_microsoft_opentelemetry'))); + // Every auth mode exports over the S2S route; the legacy delegated route rejects app-only + // tokens and needs admin consent. Contextual/agent-identity resolvers imply S2S intent too. + const expectsS2S = hasDistro && (hasEnableA365 || s2sIntent); + if (expectsS2S && s2sFalse) { + add('critical', 'python-s2s-endpoint-disabled', 'Python passes a365_use_s2s_endpoint=False. A365 export must use the S2S route in every auth mode; the delegated route rejects app-only tokens and needs admin consent.', py.find(f => /a365_use_s2s_endpoint\s*=\s*False\b/.test(read(f)))); + } else if (expectsS2S && !s2sTrue && s2sEnv) { + add('medium', 'python-s2s-endpoint-env-dependent', 'Python depends on A365_USE_S2S_ENDPOINT=true in env to select the S2S route. Prefer a365_use_s2s_endpoint=True in code.', py.find(f => read(f).includes('use_microsoft_opentelemetry'))); + } else if (expectsS2S && !s2sTrue && !s2sEnv) { + add('high', 'python-s2s-endpoint-not-set', 'Python does not set a365_use_s2s_endpoint=True (or A365_USE_S2S_ENDPOINT=true), so export uses the legacy delegated route. Every auth mode must export over the S2S route with an app-only token.', py.find(f => read(f).includes('use_microsoft_opentelemetry'))); + } + for (const file of py) { + const content = read(file); + const delegated = callBlocks(content, 'exchange_token').some(block => /observability/i.test(block)) || + /(? callBlocks(read(f), 'prefetch').some(block => /self\.connection_manager/.test(block))); + if (prefetchFile && !anyMatches(py, /\bself\.connection_manager\s*=/)) { + add('high', 'python-obs-prefetch-connection-missing', 'The app-only token prefetch uses self.connection_manager, but no file assigns it. CloudAdapter does not expose its connection manager, so every prefetch fails and no spans export. Store the MsalConnectionManager passed to CloudAdapter on the host.', prefetchFile); } for (const file of py) { @@ -212,6 +228,17 @@ function validateNode() { if (hasDistro && !hasIdentity) { add('high', 'node-missing-identity-scope', 'No BaggageBuilder/InvokeAgentScope/configureA365Hosting usage found; spans may lack agent identity.'); } + // Every auth mode exports over the S2S route with an app-only token. + if (hasDistro && hasEnabled && !anyMatches(ts, /\buseS2SEndpoint\s*:\s*true\b/)) { + add('high', 'node-obs-delegated-route', 'Node code does not set useS2SEndpoint: true, so A365 export uses the legacy delegated route. Every auth mode must export over the S2S route with an app-only tokenResolver.', ts.find(f => read(f).includes('useMicrosoftOpenTelemetry'))); + } + for (const file of ts) { + const content = read(file); + if (['refreshObservabilityToken', 'RefreshObservabilityToken'].some(name => callBlocks(content, name).some(block => /authorization/i.test(block))) || + /AgenticTokenCacheInstance\s*\.\s*getObservabilityToken\s*\(/.test(content)) { + add('high', 'node-obs-delegated-token', 'Telemetry uses a delegated (OBO) token (refreshObservabilityToken(..., authorization) or AgenticTokenCacheInstance.getObservabilityToken). The S2S route rejects delegated tokens; use an app-only tokenResolver for the agent identity instead.', file); + } + } const hasSemantic = anyContains(ts, 'InvokeAgentScope') || anyContains(ts, 'InferenceScope') || anyContains(ts, 'ExecuteToolScope') || @@ -241,6 +268,17 @@ function validateDotnet() { if (hasWiring && !anyContains(cs, 'InvokeAgentScope')) { add('medium', 'dotnet-no-invoke-agent-scope', 'No InvokeAgentScope found; MAC Activity needs an invoke_agent parent span.'); } + // Every auth mode exports over the S2S route with an app-only token. + if (anyContains(cs, 'UseMicrosoftOpenTelemetry') && !anyMatches(cs, /\bUseS2SEndpoint\s*=\s*true\b/)) { + add('high', 'dotnet-obs-delegated-route', 'UseMicrosoftOpenTelemetry is wired without UseS2SEndpoint = true, so A365 export uses the legacy delegated route. Set o.Agent365.UseS2SEndpoint = true (o.Agent365.Exporter.UseS2SEndpoint on Microsoft.OpenTelemetry 1.0.2 and earlier) with an app-only TokenResolver in every auth mode.', cs.find(f => read(f).includes('UseMicrosoftOpenTelemetry'))); + } + for (const file of cs) { + const content = read(file); + if (callBlocks(content, 'RegisterObservability').some(block => block.includes('AgenticTokenStruct')) || + /\bnew\s+AgenticTokenStruct\s*[({]|IExporterTokenCache\s*<\s*AgenticTokenStruct\s*>\s*\??\s+[A-Za-z_]\w*/.test(content)) { + add('high', 'dotnet-obs-delegated-token', 'Telemetry uses a delegated (OBO) token (RegisterObservability with AgenticTokenStruct, new AgenticTokenStruct(...), or an IExporterTokenCache dependency). The S2S route rejects delegated tokens; wire an app-only TokenResolver for the agent identity instead.', file); + } + } } function validateSetupArtifacts() { @@ -252,6 +290,10 @@ function validateSetupArtifacts() { if (generated.agentBlueprintId && generated.agenticAppId && generated.agentBlueprintId === generated.agenticAppId) { add('critical', 'blueprint-id-used-as-agent-id', 'a365.generated.config.json has identical blueprint and agentic app IDs. Verify runtime gen_ai.agent.id uses the agent instance/source agent ID, not the blueprint ID.', path.join(cwd, 'a365.generated.config.json')); } + const staticConfig = readJsonSafe(path.join(cwd, 'a365.config.json')); + if (staticConfig && staticConfig.aiTeammate === false && generated.agenticAppId && !generated.agentRegistrationId) { + add('medium', 'agent-registration-not-recorded', 'a365.generated.config.json has an agent identity but no agentRegistrationId. The S2S route authorizes registered agent instances without an OtelWrite grant; an unregistered instance gets 403 insufficient_scope. Run a365 setup all --agent-registration-only (idempotent).', path.join(cwd, 'a365.generated.config.json')); + } } validatePython(); diff --git a/plugins/agent365/skills/a365-code-validator/references/validation-checklist.md b/plugins/agent365/skills/a365-code-validator/references/validation-checklist.md index ed88a94..785de36 100644 --- a/plugins/agent365/skills/a365-code-validator/references/validation-checklist.md +++ b/plugins/agent365/skills/a365-code-validator/references/validation-checklist.md @@ -38,7 +38,8 @@ useMicrosoftOpenTelemetry({ a365: { enabled: true, enableObservabilityExporter: true, - tokenResolver, + useS2SEndpoint: true, + tokenResolver, // app-only token for the exporting agent identity }, }); ``` @@ -146,37 +147,55 @@ blank or incomplete. ## 5. Endpoint Selection +Every auth mode exports over the service-to-service (S2S) route with an **app-only** token for +the runtime Agent Identity. `authMode` only changes how that token is sourced: + | Auth mode | Structural transport expectation | |---|---| -| S2S / application | Service-to-service export mode; token principal is the runtime Agent Identity | -| OBO / delegated / agentic-user | Delegated export mode; delegated token carries observability write scope | +| S2S / application | S2S export mode; app-only token from the FMI chain for the configured Agent Identity | +| OBO / delegated / agentic-user (incl. AI Teammate) | S2S export mode; app-only token for the turn's Agent Identity from the hosting connection's blueprint credential. OBO / Agentic User tokens are for workload calls only | -S2S also requires an Observability API token with: +The S2S route requires an Observability API token with: ```text -roles contains Agent365.Observability.OtelWrite +no scp claim (any delegated token is rejected) +azp/appid == the exporting Agent Identity == /agents/{agentId} +roles may be absent or empty for a registered agent instance; roles containing +Agent365.Observability.OtelWrite are also accepted ``` -OBO / agentic-user uses: +Select the transport in code: `useS2SEndpoint: true` (Node.js), `a365_use_s2s_endpoint=True` +(Python), `o.Agent365.UseS2SEndpoint = true` (.NET 1.0.3+; `o.Agent365.Exporter.UseS2SEndpoint` +on 1.0.2 and earlier). Depending on `A365_USE_S2S_ENDPOINT=true` in the environment is more +fragile and should be called out. -```text -scp contains Agent365.Observability.OtelWrite -``` +**Delegated telemetry is a finding (`high`).** Look for any of these: -For Python S2S, prefer setting `a365_use_s2s_endpoint=True` in code. Depending on -`A365_USE_S2S_ENDPOINT=true` in environment is more fragile and should be called out. +- Node.js `refreshObservabilityToken(..., authorization)` or a `preloadObservabilityToken` helper, or + a `tokenResolver` reading `AgenticTokenCacheInstance.getObservabilityToken(...)`. +- .NET `RegisterObservability(..., new AgenticTokenStruct(...), ...)`, any `new AgenticTokenStruct(...)`, + or an `IExporterTokenCache` dependency. +- Python `exchange_token(..., scopes=get_observability_authentication_scope(), ...)` feeding a + telemetry token cache, `cache_agentic_token(...)`, or an `AgenticTokenCache` / + `get_cached_agentic_token` resolver. +- A distro call without the S2S transport flag. + +Each sends a delegated token (rejected by the S2S route) or uses the legacy delegated route, which +needs admin consent. Replace it with the app-only resolver from the `instrument-observability` +references. ### How the S2S token is minted (the right shape) A plain client-credentials call for the observability scope does not produce the required -runtime Agent Identity principal. The token must come from the **3-hop FMI exchange**, so its +runtime Agent Identity principal. The token must come from the **FMI exchange**, so its principal equals the runtime Agent Identity: ```text -leg 1: blueprint creds (secret, or MI assertion on Azure) + fmi_path= +leg 1: blueprint creds (secret, certificate, or MI assertion on Azure — for interactive agents, + the hosting connection's getAgenticApplicationToken helper) + fmi_path= -> assertion T1 (scope api://AzureADTokenExchange/.default) -leg 3: authenticate AS the agent identity using T1 as the client assertion - -> Observability API token (scope api://9b975845-.../.default), azp == agent id, roles:[OtelWrite] +leg 2: authenticate AS the agent identity using T1 as the client assertion + -> Observability API token (scope api://9b975845-.../.default), azp == agent id, no scp ``` Flag S2S code that mints the obs token with a bare `ClientSecretCredential` / @@ -228,10 +247,15 @@ Compare the live ID printed by the command with the local `agentBlueprintId`. A Blueprint or a local ID that no longer resolves is a configuration blocker; do not rewrite the ID automatically. -For observability, verify the grant that matches the auth mode: - -- `obo` / `agentic-user`: delegated `Agent365.Observability.OtelWrite` -- `s2s`: application role `Agent365.Observability.OtelWrite` +For observability, the S2S route (every auth mode) authorizes a **registered agent instance** +without `Agent365.Observability.OtelWrite` (subject to service policy); the application role +`Agent365.Observability.OtelWrite` on the Blueprint is accepted as well. The Agent 365 CLI no +longer requests Observability API permissions for blueprint agents, so a missing OtelWrite grant +is **not** a blocker when the instance is registered (`agentRegistrationId` in +`a365.generated.config.json`). If neither registration nor the application role is evident, +report `high` and recommend `a365 setup all --agent-registration-only` for blueprint agents. For AI Teammates, recommend the `OtelWrite` application-role step that `a365 setup all --aiteammate` prints. A delegated +`Agent365.Observability.OtelWrite` scope matters only to legacy delegated-route exporters — flag +that code for migration (§5) instead. If Graph returns 401/403, report the check as unavailable due to caller authorization rather than claiming the Blueprint has no permissions. The list API's least-privileged Graph permission @@ -283,7 +307,9 @@ database, endpoint, or correlation details into the validator report. |---|---| | Export accepted (200/`sent`) but nothing in MAC | Confirm the target tenant and user satisfy current Agent 365 licensing and enrollment prerequisites | | Token fails `AADSTS500011` (resource principal not found) | Observability resource SP isn't in the tenant — Agent 365/observability onboarding, not a code fix | -| S2S token via bare `ClientSecretCredential` → 403 | Mint via the 3-hop FMI exchange so the principal == runtime Agent Identity (see §5) | +| S2S token via bare `ClientSecretCredential` → 403 | Mint via the FMI exchange so the principal == runtime Agent Identity (see §5) | +| S2S export → 403 `insufficient_scope` with a valid app-only token | Instance not registered and no OtelWrite application role. Blueprint agents: run `a365 setup all --agent-registration-only` (a grant is not the expected fix). AI Teammates: complete the application-role step `a365 setup all --aiteammate` prints | +| Exporter fed an OBO / Agentic User token (per-turn refresh or `AgenticTokenStruct` registration) | Replace with the app-only resolver and set the S2S transport flag (see §5) | | Python only passes `enable_a365=True` | Also pass `a365_enable_observability_exporter=True` or set exporter env true | | Queue/background job calls baggage helper with only `blueprint_id` | Pass the runtime `agent_id` explicitly | | Testbench works but app does not | Testbench manually emits supported spans; app may only emit generic spans | @@ -309,7 +335,7 @@ Default report should be short and action-oriented: 3. **Semantic spans missing** — generic HTTP spans may not populate Activity. Fix: add/confirm `invoke_agent`, `chat`, `execute_tool`, `output_messages`. -**Fix order:** token/identity → exporter flag → S2S/OBO mode → semantic spans → backend verification. +**Fix order:** token/identity → exporter flag → S2S route (every auth mode) → semantic spans → backend verification. ``` Only add detailed evidence when needed. Keep concrete file/function names and redact per the diff --git a/plugins/agent365/skills/add-workiq-tools/SKILL.md b/plugins/agent365/skills/add-workiq-tools/SKILL.md index 0b20e36..202794e 100644 --- a/plugins/agent365/skills/add-workiq-tools/SKILL.md +++ b/plugins/agent365/skills/add-workiq-tools/SKILL.md @@ -363,8 +363,8 @@ For every branch: 1. **Grep the target file for the observability anchor symbols:** - **.NET** (`AgentApplication` subclass): `BaggageBuilder`, `InvokeAgentScope`, `InferenceScope`, `Agent365ObservabilityContext` - - **Node.js** (`src/agent.ts` and `src/client.ts`): `BaggageBuilder`, `BaggageBuilderUtils`, `InvokeAgentScope`, `InferenceScope`, `AgenticTokenCacheInstance`, `preloadObservabilityToken` - - **Python** (`agent.py`): `BaggageBuilder`, `populate_baggage`, `InvokeAgentScope`, `with builder.build()`, `AgenticTokenCache` + - **Node.js** (`src/agent.ts` and `src/client.ts`): `BaggageBuilder`, `BaggageBuilderUtils`, `InvokeAgentScope`, `InferenceScope` (and in the entry point: `useS2SEndpoint`, `createAppTokenResolver`) + - **Python** (`agent.py` / `host_agent_server.py`): `BaggageBuilder`, `populate_baggage`, `InvokeAgentScope`, `with builder.build()`, `_setup_observability_token` / `OBS_TOKENS.prefetch` 2. **If any of those symbols are present**, scope your `Edit` `old_string` **as narrowly as possible** — anchor on the **single statement immediately before/after** the new line, never a multi-statement block, never the method signature alone, never the full method body. Examples: - ✅ Good: anchor on the `var response = await chatClient.GetResponseAsync(...)` line and insert `GetMcpToolsAsync` immediately above it. diff --git a/plugins/agent365/skills/instrument-observability/SKILL.md b/plugins/agent365/skills/instrument-observability/SKILL.md index aeb2932..4e5b8df 100644 --- a/plugins/agent365/skills/instrument-observability/SKILL.md +++ b/plugins/agent365/skills/instrument-observability/SKILL.md @@ -5,10 +5,13 @@ description: > Python agents. Adds OTel-based tracing, context propagation, A365 exporter, manual instrumentation scopes (InvokeAgentScope, InferenceScope, ExecuteToolScope — required for store publishing), and updates configuration files. Asks a two-stage question — agent kind - (AI Teammate or Agent (Non AI Teammate)) and auth mode — to determine - the correct token path: OBO (`obo` / `agentic-user`) or Service Principal (`s2s`) - (FMI 3-hop token chain with Power Platform scope supported for .NET, Node.js, and Python — each language - gets a scaffold token-service file that acquires and refreshes the Observability API token via the FMI chain). Non-destructive and idempotent. + (AI Teammate or Agent (Non AI Teammate)) and auth mode — to determine how the telemetry + token is sourced. Every agent exports over the S2S route with an app-only token for the + exporting agent identity: `obo` / `agentic-user` agents get an app-only token resolver that + reuses the agent's hosting connection (blueprint credential), and `s2s` agents get a scaffold + token-service file that acquires and refreshes the Observability API token via the FMI chain + (.NET, Node.js, and Python). No OBS permission or admin consent is needed for registered + blueprint agent instances. Non-destructive and idempotent. compatibility: - claude-code - vscode-copilot @@ -78,7 +81,7 @@ without disrupting the agent's core logic. It: 2. **Installs** the correct A365 observability packages (core + hosting + optional extensions) 3. **Wires** observability in the entry point 4. **Adds** BaggageBuilder context or BaggageMiddleware to message handlers -5. **Implements** the agentic token resolver with caching +5. **Implements** the app-only telemetry token resolver (S2S route) with caching 6. **Adds** manual instrumentation scopes (InvokeAgentScope, InferenceScope, ExecuteToolScope — **required for store publishing**) 7. **Updates** configuration files with observability settings 8. **Validates** the build passes @@ -86,7 +89,18 @@ without disrupting the agent's core logic. It: > **Store publishing requirement:** The Agent 365 store validation requires `InvokeAgentScope`, > `InferenceScope`, and `ExecuteToolScope` to be implemented. This skill wires them. -All changes are **additive** and **idempotent** — rerunning the skill is safe. +All changes are **additive** and **idempotent** — rerunning the skill is safe. The one +exception is legacy delegated telemetry wiring, which this skill replaces (see "Migrating +delegated telemetry" in Phase 3). + +> **Telemetry is always exported over the S2S route with an app-only token** for the exporting +> agent identity, whatever the `authMode`. The S2S route rejects delegated (`scp`) tokens, so +> OBO / Agentic User tokens are used only for workload calls (MCP / Graph), never for the +> exporter. Registered blueprint agent instances need no `Agent365.Observability.OtelWrite` +> permission or admin consent (subject to service policy), and `a365 setup all` no longer requests +> them for blueprint agents. AI Teammate setup (`a365 setup all --aiteammate`) still offers the +> `OtelWrite` application role; complete the app-role action item it prints, because the S2S route +> accepts that role. --- @@ -157,7 +171,7 @@ Store `agentType` (`ai-teammate` = AI Teammate, or `system-agent` = Agent (Non A **Update `.a365-workspace-detection.local.json`** — merge `agentType` and `authMode` into the existing cache file, preserving all other fields (`agentStack`, `programmingLanguage`, `usesTeamsOrCopilot`, `detectedAt`). Use the **Write** tool to write the merged object back. -The `authMode` value drives Phases 3–5: OBO and S2S paths differ in entry point wiring (Phase 3), message handler pattern (Phase 4), and token resolver (Phase 5). **Phases 2, 6, 7, and 8 are identical regardless of `authMode`.** +The `authMode` value drives Phases 3–5: both paths export over the S2S route with an app-only token, but they differ in how that token is sourced — entry point wiring (Phase 3), message handler pattern (Phase 4), and token resolver (Phase 5). **Phases 2, 6, 7, and 8 are identical regardless of `authMode`.** **TaskUpdate** — Mark complete: "Determine agent type and authentication mode" @@ -245,16 +259,36 @@ flag live in the references — see the "Required packages" section of: > **Pre-existing placeholders:** As of CLI 1.1, `a365 setup all` auto-writes `Agent365Observability` placeholder sections to `appsettings.json` (.NET) or `.env` (Node.js/Python). Before creating config from scratch, **check if placeholders already exist** and fill in values rather than duplicating the section. +> **Migrating delegated telemetry (the one non-additive edit).** If the project already wires +> the legacy delegated (OBO-route) telemetry path, replace it. That path is: +> - Node.js: `AgenticTokenCacheInstance.refreshObservabilityToken(..., authorization)`, a +> `preloadObservabilityToken` helper, or a `tokenResolver` reading +> `AgenticTokenCacheInstance.getObservabilityToken(...)`. +> - .NET: `RegisterObservability(..., new AgenticTokenStruct(...), ...)`, any +> `new AgenticTokenStruct(...)` (for example an `A365OtelWrapper` helper), an +> `IExporterTokenCache` dependency, or `AddAgenticTracingExporter()`. +> - Python: `exchange_token(...)` for the observability scope (with or without an explicit +> `scopes=`), `cache_agentic_token(...)`, or an `AgenticTokenCache` / `get_cached_agentic_token` +> resolver. +> - Any language: a distro call without the S2S route flag. +> +> The S2S route rejects those delegated tokens, and the delegated route needs admin consent. Swap the +> resolver for the app-only one, set the S2S route flag, and remove the per-turn delegated +> refresh. Keep the baggage and scope wiring. Durable or offline delivery spools keep each +> record's original route, so for `@microsoft/opentelemetry` 1.4.x set +> `a365.durableDelivery: { enabled: false }` during the migration (as the Agent 365 samples do) +> or clear the spool directory. List every replaced call in the final summary. + ### For .NET AgentFramework 1. **Read** the current entry point (`Program.cs` or detected file). 2. **Edit** — Add observability wiring following the reference pattern in `dotnet-observability.md`: - Add `using Microsoft.OpenTelemetry;` to `Program.cs`. - - **OBO / agentic-user path** (AI Teammate AND Standard .NET agents): Call `builder.UseMicrosoftOpenTelemetry(o => { ... })` with `o.Exporters = ExportTarget.Agent365 | ExportTarget.Console` (Dev) or `ExportTarget.Agent365` (Production). The distro **auto-registers `IExporterTokenCache`** in DI — no `AddAgenticTracingExporter()` call needed. Leave `o.Agent365.Exporter.UseS2SEndpoint` at its default (`false`) — the exporter POSTs to `/observability/` which the OBO token cache authenticates. Also set `"EnableAgent365Exporter": true` in `appsettings.json` to activate the backend exporter — the SDK defaults this to `false` when absent, so without it the exporter is wired but inert. + - **OBO / agentic-user path** (AI Teammate AND Standard .NET agents): First **Write** `Observability/AgentAppTokenResolver.cs` from the reference doc and register it with `builder.Services.AddSingleton();`. It acquires an app-only Observability API token per agent identity from the agent's default (blueprint) connection. Then call `builder.UseMicrosoftOpenTelemetry(o => { ... })` with `o.Exporters = ExportTarget.Agent365 | ExportTarget.Console` (Dev) or `ExportTarget.Agent365` (Production), `o.Agent365.UseS2SEndpoint = true`, and `o.Agent365.TokenResolver = (agentId, tenantId) => obsTokens?.ResolveAsync(agentId, tenantId) ?? Task.FromResult(null);`, where `obsTokens` is resolved from `app.Services` after `builder.Build()`. On `Microsoft.OpenTelemetry` 1.0.2 and earlier these options live under `o.Agent365.Exporter`. Do NOT rely on the distro's auto-registered `IExporterTokenCache` — it serves delegated tokens that the S2S route rejects. Also set `"EnableAgent365Exporter": true` in `appsettings.json` to activate the backend exporter — the SDK defaults this to `false` when absent, so without it the exporter is wired but inert. - **Required: `IChatClient.UseOpenTelemetry()`** — when registering the `IChatClient` (e.g. Azure OpenAI), chain `.AsBuilder().UseFunctionInvocation().UseOpenTelemetry(sourceName: null, cfg => cfg.EnableSensitiveData = true).Build()`. This is what makes the AI SDK emit the `gen_ai.inference` and `gen_ai.tool` spans that `InvokeAgentScope` (Phase 5.5) anchors as children. **Skipping this means no LLM spans appear in MAC, even with everything else wired** — the `InvokeAgent` parent becomes a hollow span. `EnableSensitiveData = true` includes prompts/completions in span attributes (PII consideration — set to `false` for regulated data). - - **S2S path**: First **Write** the two scaffold files from the reference doc — `Observability/ObservabilityServiceExtensions.cs` (DI extension with `AddAgent365Observability()` using `ServiceTokenCache` and conditional `ObservabilityTokenService`) and `Observability/ObservabilityTokenService.cs` (background service that acquires the Observability API token via the MSAL FMI 3-hop chain with `.WithFmiPath()` targeting scope `api://9b975845-388f-4429-889e-eab1ef63949c/.default`, supports MSI with client-secret fallback). Then call `builder.Services.AddAgent365Observability();` and `builder.UseMicrosoftOpenTelemetry(...)` with token resolver reading from the `ServiceTokenCache`. **Critical:** Set `o.Agent365.Exporter.UseS2SEndpoint = true` in the options callback — without this, the exporter posts to the wrong path (`/observability/` instead of `/observabilityService/`) and gets HTTP 401. See "Known Issues" section. - - Optionally register `adapter.Use(new BaggageTurnMiddleware())` (OBO path only) to auto-populate baggage on every request + - **S2S path**: First **Write** the two scaffold files from the reference doc — `Observability/ObservabilityServiceExtensions.cs` (DI extension with `AddAgent365Observability()` using `ServiceTokenCache` and conditional `ObservabilityTokenService`) and `Observability/ObservabilityTokenService.cs` (background service that acquires the Observability API token via the MSAL FMI 3-hop chain with `.WithFmiPath()` targeting scope `api://9b975845-388f-4429-889e-eab1ef63949c/.default`, supports MSI with client-secret fallback). Then call `builder.Services.AddAgent365Observability();` and `builder.UseMicrosoftOpenTelemetry(...)` with token resolver reading from the `ServiceTokenCache`. **Critical:** Set `o.Agent365.UseS2SEndpoint = true` in the options callback (`o.Agent365.Exporter.UseS2SEndpoint` on 1.0.2 and earlier) — without this, the exporter posts to the wrong path (`/observability/` instead of `/observabilityService/`) and gets HTTP 401. See "Known Issues" section. + - Optionally register `adapter.Use(new BaggageTurnMiddleware())` to auto-populate baggage on every request - Mark all new lines with: `// A365 Observability — best-effort instrumentation (verify against official sample)` 3. **Preserve** all existing code — only add new lines, never remove. @@ -264,8 +298,8 @@ flag live in the references — see the "Required packages" section of: 1. **Read** the current entry point (`index.ts`, `app.ts`, or detected file). 2. **Edit** — Add observability initialization following the reference pattern in `nodejs-observability.md`: - - Import `useMicrosoftOpenTelemetry`, `shutdownMicrosoftOpenTelemetry`, `configureA365Hosting`, and `AgenticTokenCacheInstance` — **all from `@microsoft/opentelemetry`** (single package as of GA 1.0; do NOT import from the legacy `-observability`, `-hosting`, or `-runtime` packages). - - **OBO / agentic-user path**: Call `useMicrosoftOpenTelemetry({ a365: { enabled: true, enableObservabilityExporter: true, tokenResolver } })` **before** any LLM/framework imports. Both `enabled: true` AND `enableObservabilityExporter: true` are required in 1.0+ to actually export spans. Wire `tokenResolver` to `AgenticTokenCacheInstance.getObservabilityToken(agentId, tenantId) ?? ''`. + - Import `useMicrosoftOpenTelemetry`, `shutdownMicrosoftOpenTelemetry`, and `configureA365Hosting` — **all from `@microsoft/opentelemetry`** (single package as of GA 1.0; do NOT import from the legacy `-observability`, `-hosting`, or `-runtime` packages). Do NOT import `AgenticTokenCacheInstance` for telemetry — it serves delegated tokens. + - **OBO / agentic-user path**: First **Write** `observability/app-token-resolver.ts` from `nodejs-observability.md` ("App-only Token Resolver Scaffold"). Then call `useMicrosoftOpenTelemetry({ a365: { enabled: true, enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver: appTokenResolver } })` **before** any LLM/framework imports, where `appTokenResolver = createAppTokenResolver(() => getObsConnection())`. Assign `getObsConnection = () => adapter.connectionManager.getDefaultConnection();` once the adapter exists. Both `enabled: true` AND `enableObservabilityExporter: true` are required in 1.0+ to actually export spans; `useS2SEndpoint: true` sends them to the S2S route, which accepts the app-only token. - **S2S path**: First **Write** `observability/token-cache.ts` (in-memory token cache with `cacheToken`/`getCachedToken`/`tokenResolver`) and `observability/observability-token-service.ts` using the scaffold pattern from `nodejs-observability.md` (S2S section). This module acquires the Observability API token via MSAL FMI 3-hop chain (`@azure/msal-node` with `fmiPath` parameter, targeting scope `api://9b975845-388f-4429-889e-eab1ef63949c/.default`, supports MSI with client-secret fallback) and refreshes it every 50 min. Then call `useMicrosoftOpenTelemetry({ a365: { enabled: true, enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver: a365TokenResolver } })`. `useS2SEndpoint: true` is now a first-class option (1.0+); the old workaround with custom `Agent365Exporter` via `spanProcessors` and `ENABLE_A365_OBSERVABILITY_EXPORTER=false` is no longer needed for new instrumentation. If the old workaround is already present in an existing agent, leave it in place — do not delete code as part of this additive skill; flag it in the final summary as a candidate for cleanup if the user explicitly asks to migrate. - **Both paths**: Call `configureA365Hosting(adapter, { enableBaggage: true })` once at startup to register `BaggageMiddleware`. This replaces manual `adapter.use(new BaggageMiddleware())` and removes the need for `BaggageBuilderUtils.fromTurnContext` in handlers. - **Both paths**: Register `SIGTERM`/`SIGINT` handlers calling `await shutdownMicrosoftOpenTelemetry()` to flush pending spans on shutdown. @@ -280,7 +314,7 @@ flag live in the references — see the "Required packages" section of: 2. **Edit** — Add observability configuration following the reference pattern in `python-observability.md`: - Import `use_microsoft_opentelemetry` from `microsoft.opentelemetry` (single unified package; do NOT import from the legacy `microsoft_agents_a365.*` namespace). - - **OBO / agentic-user path**: Call `use_microsoft_opentelemetry(enable_a365=True, a365_enable_observability_exporter=True, a365_token_resolver=...)`. Both `enable_a365=True` AND `a365_enable_observability_exporter=True` are required in 1.0+ to actually export spans. Wire `a365_token_resolver` to `AgenticTokenCache().get_observability_token` from `microsoft.opentelemetry.a365.hosting.token_cache_helpers` (or a custom resolver reading from `token_cache.py`). + - **OBO / agentic-user path**: First **Write** `observability/app_token_resolver.py` from `python-observability.md` ("App-only Token Resolver Scaffold"). Then create `OBS_TOKENS = AppTokenResolver()` and call `use_microsoft_opentelemetry(enable_a365=True, a365_enable_observability_exporter=True, a365_use_s2s_endpoint=True, a365_token_resolver=OBS_TOKENS.resolve)`. Both `enable_a365=True` AND `a365_enable_observability_exporter=True` are required in 1.0+ to actually export spans; `a365_use_s2s_endpoint=True` sends them to the S2S route. Do NOT wire `AgenticTokenCache().get_observability_token` or a cache of `exchange_token()` results — both serve delegated tokens. - **S2S path**: First **Write** `observability/token_cache.py` (in-memory token cache with `cache_token`/`get_cached_token`) and `observability/observability_token_service.py` using the scaffold pattern from `python-observability.md` (S2S section). This module acquires the Observability API token via a 3-hop FMI chain: direct HTTP POST with `fmi_path` for Hops 1+2 (MSAL Python does not properly serialize `fmi_path` — known limitation), then `msal.ConfidentialClientApplication` for Hop 3, targeting scope `api://9b975845-388f-4429-889e-eab1ef63949c/.default`, supports MSI with client-secret fallback, refreshes every 50 min via an `asyncio` background task. Then call `use_microsoft_opentelemetry(enable_a365=True, a365_enable_observability_exporter=True, a365_use_s2s_endpoint=True, a365_token_resolver=...)`. `a365_use_s2s_endpoint=True` is now a first-class kwarg — no workaround needed. Schedule `run_token_service()` as an asyncio task and call `acquire_initial_token()` in your aiohttp lifespan startup. Also install `msal`, `azure-identity`, and `httpx`. - **Both paths**: Call `ObservabilityHostingManager.configure(adapter.middleware_set, ObservabilityHostingOptions(enable_baggage=True))` once at startup to auto-populate baggage from `TurnContext`. **Note:** `enable_baggage` defaults to `False` — must be explicitly set to `True`. - **Auto-instrumentation note**: Do NOT call legacy `*Instrumentor().instrument()` methods for LangChain/OpenAI/SK/AgentFramework — these are auto-enabled in 1.0+ and manual calls cause duplicate spans. @@ -299,9 +333,10 @@ flag live in the references — see the "Required packages" section of: > **Skip this phase** if BaggageMiddleware was registered in Phase 3 — the middleware handles > baggage propagation automatically for every request. -> **Auth mode note:** All three `authMode` values use `authHandlerName: "AGENTIC"` in the -> code — the token exchange call is identical. The identity in traces is determined by Azure AD -> provisioning and the incoming token. Add an inline comment indicating which mode was chosen. +> **Auth mode note:** Telemetry never goes through the auth handler. Every `authMode` exports +> over the S2S route with an app-only token (Phase 3), and the auth handler +> (`AgentApplication:AgenticAuthHandlerName`, etc.) is only for workload calls (MCP / Graph). +> Add an inline comment indicating which mode was chosen. ### For .NET AgentFramework @@ -310,41 +345,29 @@ flag live in the references — see the "Required packages" section of: 2. **Edit** — Follow the reference pattern in `dotnet-observability.md` (full code sample under "Agent Class — Message Handler (OBO Path)"): **OBO path** (`obo` / `agentic-user`) — applies to both **AI Teammate** agents and **Standard .NET agents**: - - Inject `IExporterTokenCache` in the constructor (auto-registered by the distro — no `AddAgenticTracingExporter()` call needed). - - Inject `IConfiguration` (for blueprint/observability config) and `ILogger`. - - **Resolve agent ID for BOTH auth paths** — agentic instance ID from the Activity for agentic turns, **decoded from the auth token** via `Utility.ResolveAgentIdentity(context, authToken)` for non-agentic turns (the SDK names the second parameter generically `authToken` — it accepts both OBO tokens and agentic-path tokens returned by `UserAuthorization.GetTurnTokenAsync`). Do NOT fall back to `Guid.Empty.ToString()` — that creates a synthetic identity the exporter cannot authenticate, polluting traces with `"No token obtained. Skipping export for this identity."` warnings. + - Inject `IConfiguration` (for blueprint/observability config) and `ILogger`. Do **not** inject `IExporterTokenCache` or make a per-turn `RegisterObservability(...)` call: the app-only `AgentAppTokenResolver` wired in Phase 3 supplies the export token. If an existing handler has either, remove it (see "Migrating delegated telemetry" in Phase 3). + - **Resolve the exporting agent identity.** S2S export authenticates *as* this identity, so it must be an agent identity, never the blueprint. Agentic turns carry it on the Activity. Other turns fall back to the provisioned agent identity that `a365 setup all` writes to `Agent365Observability:AgentId` for non-AI-Teammate blueprint agents. Do NOT decode it from an OBO token: `Utility.ResolveAgentIdentity(context, oboToken)` returns the blueprint or bot app ID for OBO tokens. Do NOT fall back to `Guid.Empty.ToString()` either. Both produce an identity the exporter cannot authenticate. ```csharp - string? resolvedAgentId = null; - if (turnContext.Activity.IsAgenticRequest()) - { - resolvedAgentId = turnContext.Activity.GetAgenticInstanceId(); - } - else if (!string.IsNullOrEmpty(authHandlerName)) + var obsConfig = _configuration.GetSection("Agent365Observability"); + + string? resolvedAgentId = turnContext.Activity.IsAgenticRequest() + ? turnContext.Activity.GetAgenticInstanceId() + : obsConfig["AgentId"]; + if (!Guid.TryParse(resolvedAgentId, out _) + || string.Equals(resolvedAgentId, obsConfig["AgentBlueprintId"], StringComparison.OrdinalIgnoreCase)) { - try - { - var authToken = await UserAuthorization - .GetTurnTokenAsync(turnContext, authHandlerName, cancellationToken: cancellationToken) - .ConfigureAwait(false); - if (!string.IsNullOrEmpty(authToken)) - { - resolvedAgentId = Utility.ResolveAgentIdentity(turnContext, authToken); - } - } - catch (Exception ex) - { - _logger.LogDebug(ex, "Could not resolve agent id from auth token; A365 observability skipped for this turn."); - } + resolvedAgentId = null; // not an agent identity; the S2S route would reject it } var resolvedTenantId = turnContext.Activity.Conversation?.TenantId - ?? turnContext.Activity.Recipient?.TenantId; + ?? turnContext.Activity.Recipient?.TenantId + ?? obsConfig["TenantId"]; var hasObservabilityIdentity = !string.IsNullOrEmpty(resolvedAgentId) && !string.IsNullOrEmpty(resolvedTenantId); ``` - `GetAgenticInstanceId()` returns the agent's **service principal object ID** (the instance ID assigned by A365 for the Teams agentic identity). `Utility.ResolveAgentIdentity(context, authToken)` decodes the agent identity from a JWT — works for both OBO tokens and agentic-path tokens (SDK signature names the param generically `authToken`). Both paths produce the same kind of ID — what shows up in MAC Advanced Hunting. - - **Conditional baggage + token registration** — only when `hasObservabilityIdentity == true`. Skip both calls cleanly when the identity can't be resolved: + `GetAgenticInstanceId()` returns the agentic instance ID (`Recipient.AgenticAppId`). That is the identity the app-only token is minted for, and what shows up in MAC Advanced Hunting. + - **Conditional baggage** — only when `hasObservabilityIdentity == true`; skip cleanly when the identity can't be resolved: ```csharp using IDisposable? baggageScope = hasObservabilityIdentity ? new BaggageBuilder() @@ -352,33 +375,12 @@ flag live in the references — see the "Required packages" section of: .AgentId(resolvedAgentId!) .Build() : null; - - if (hasObservabilityIdentity) - { - try - { - _agentTokenCache.RegisterObservability( - resolvedAgentId!, - resolvedTenantId!, - new AgenticTokenStruct( - userAuthorization: UserAuthorization, - turnContext: turnContext, - authHandlerName: authHandlerName ?? string.Empty), - EnvironmentUtils.GetObservabilityAuthenticationScope()); - } - catch (Exception ex) - { - _logger.LogWarning(ex, "Failed to register observability token."); - } - } ``` - Note: Some SDK versions support object-initializer syntax instead. If the constructor form fails to compile, try property-initializer: `new AgenticTokenStruct { UserAuthorization = ..., TurnContext = ..., AuthHandlerName = ... }`. - - The `authHandlerName` should resolve to the agentic auth handler name (from config `AgentApplication:AgenticAuthHandlerName`) when `IsAgenticRequest()` is true, OBO handler name (from `AgentApplication:OboAuthHandlerName`) otherwise. - - **Keep the `Agent365Observability` section in `appsettings.json`** (`EnableAgent365Exporter` and base exporter settings are still required — Phase 6 handles these). For **OBO**, you do **not** need to hardcode per-agent IDs, tenant IDs, or S2S credentials in that section — the agent ID and tenant ID are resolved from the request at runtime on each turn. + - **Keep the `Agent365Observability` section in `appsettings.json`** (`EnableAgent365Exporter` and base exporter settings are still required — Phase 6 handles these). No S2S credentials are needed there: `AgentAppTokenResolver` uses the agent's `Connections` settings, and agentic turns resolve their agent ID at runtime. - **The inline pattern shown above is preferred** for new code (mirrors PR #308 in `microsoft/Agent365-Samples`). The older `A365OtelWrapper.InvokeObservedAgentOperation(...)` static-wrapper pattern at `Agent365-samples/dotnet/agent-framework/sample-agent/telemetry/A365OtelWrapper.cs` is functionally equivalent but uses a separate helper class. **S2S path**: - - Inject `Agent365ObservabilityContext` (singleton registered by `AddAgent365Observability()`) in the constructor — **not** `IExporterTokenCache` + - Inject `Agent365ObservabilityContext` (singleton registered by `AddAgent365Observability()`) in the constructor - **Baggage:** Use `new BaggageBuilder().FromTurnContext(turnContext).Build()` as a separate `using var baggageScope` — `FromTurnContext()` is an extension on `BaggageBuilder` **only**; it does not exist on `InvokeAgentScope` or any scope type - **Scope:** Use `InvokeAgentScope.Start(new Request(...), new InvokeAgentScopeDetails(endpoint: new Uri("...")), _obs.AgentDetails, callerDetails)` as a separate `using var scope` — `InvokeAgentScopeDetails` has **no parameterless constructor**; always pass at least `endpoint`. `CallerDetails` with the blueprint sponsor's identity is **required** for S2S traces to appear in the portal - **No** per-turn `RegisterObservability()` call; **no** `.FromTurnContext()` chaining on the scope @@ -392,57 +394,41 @@ flag live in the references — see the "Required packages" section of: 1. **Read** the detected message handler file. -2. **Edit** — Refresh the per-turn exporter token following the reference pattern in `nodejs-observability.md`: - - Import `AgenticTokenCacheInstance` from `@microsoft/opentelemetry` (single unified package). - - **OBO paths only** (`obo` / `agentic-user`): Resolve `agentId` and `tenantId` dynamically from TurnContext each turn (never from config), then refresh the exporter token (non-fatal, wrap in try/catch): - ``` - const agentId = turnContext.activity?.recipient?.agenticAppId ?? ''; - const tenantId = turnContext.activity?.recipient?.tenantId ?? ''; - await AgenticTokenCacheInstance.RefreshObservabilityToken( - agentId, tenantId, turnContext, - agentApplication.authorization, // ← the AgentApplication auth object, NOT an auth-handler name string - ); - ``` - - `obo` (signed-in user): `agentApplication.authorization` exchanges the token as the **signed-in user** → traces attributed to the user - - `obo` (agentic identity): `agentApplication.authorization` exchanges the token as the **agentic user** provisioned in Azure AD → traces attributed to the agent - - Default observability scope is auto-applied (`api://9b975845-388f-4429-889e-eab1ef63949c/.default`) — no need to import `getObservabilityAuthenticationScope` (removed in 1.0). - - **Recommended pattern:** Extract the agentId/tenantId resolution and token refresh into a `preloadObservabilityToken(turnContext)` helper function to keep the handler clean. See `nodejs-observability.md` for the full helper implementation. - - **S2S path**: Do **NOT** call `AgenticTokenCacheInstance.RefreshObservabilityToken` — there is no user authorization token. The `tokenResolver` passed to `useMicrosoftOpenTelemetry()` (set up in Phase 3) handles authentication via the FMI 3-hop chain token service. - - **Baggage construction is done in Phase 5.5 (canonical pattern), NOT here.** In Phase 5.5 the message handler builds `BaggageBuilderUtils.fromTurnContext(new BaggageBuilder(), turnContext as any).build()` and runs `InvokeAgentScope.start(...)` inside `baggageScope.run(...)`. The `configureA365Hosting(adapter, { enableBaggage: true })` middleware registered in Phase 3 is a fallback that auto-populates baggage outside the handler, but it does NOT cover the scopes you'll add in Phase 5.5 — those need the manual outer wrapping or they get filtered as `Partitioned into 0 identity groups`. In Phase 4 itself, just refresh the token; do NOT call `InvokeAgentScope.start` here. - - Add inline comment: `// A365 auth mode: {authMode} — see: https://learn.microsoft.com/en-us/entra/agent-id/agent-on-behalf-of-oauth-flow` +2. **Edit** — Follow the reference pattern in `nodejs-observability.md` ("Message Handler"): + - **All auth modes:** the handler does **not** touch telemetry tokens. The `tokenResolver` wired in Phase 3 acquires the export token when spans are exported, so there is no per-turn refresh. That resolver is `observability/app-token-resolver.ts` for `obo` / `agentic-user`, and the background token service for `s2s`. + - Do **NOT** call `AgenticTokenCacheInstance.refreshObservabilityToken(...)` or add a `preloadObservabilityToken` helper. That call exchanges a delegated token, needs admin consent, and the S2S route rejects the result. If an existing handler has one, remove it (see "Migrating delegated telemetry" in Phase 3). + - The agent identity comes from TurnContext (`recipient.agenticAppId`) through the Phase 5.5 baggage scope, never from config. The only exception is the optional non-agentic fallback described in `nodejs-observability.md`. + - **Baggage construction is done in Phase 5.5 (canonical pattern), NOT here.** In Phase 5.5 the message handler builds `BaggageBuilderUtils.fromTurnContext(new BaggageBuilder(), turnContext as any).build()` and runs `InvokeAgentScope.start(...)` inside `baggageScope.run(...)`. The `configureA365Hosting(adapter, { enableBaggage: true })` middleware registered in Phase 3 is a fallback that auto-populates baggage outside the handler, but it does NOT cover the scopes you'll add in Phase 5.5 — those need the manual outer wrapping or they get filtered as `Partitioned into 0 identity groups`. Do NOT call `InvokeAgentScope.start` in Phase 4. + - Add inline comment: `// A365 auth mode: {authMode} — telemetry uses an app-only token on the S2S route` - Mark all new lines with: `// A365 Observability — best-effort instrumentation (verify against official sample)` 3. **Preserve** all existing handler logic. ### For Python -1. **Read** the detected message handler file AND `host_agent_server.py` — the helper lives in the HOST file, not the agent class. The verified AF sample places `_setup_observability_token` in `host_agent_server.py:130-156` so it has access to the `AgentApplication` instance and can be called by activity middleware. Per-turn baggage construction also lives in the handler/middleware layer in `host_agent_server.py`, NOT in `agent.py`. +1. **Read** the detected message handler file AND `host_agent_server.py`. The helper lives in the HOST file, not the agent class, so it can reach the host's `connection_manager` and be called by activity middleware. Per-turn baggage construction also lives in the handler/middleware layer in `host_agent_server.py`, NOT in `agent.py`. -2. **Edit `host_agent_server.py`** (the host file) — Refresh the per-turn exporter token following the reference pattern in `python-observability.md`: - - Default observability scope is auto-applied by `microsoft-opentelemetry` 1.1+ — do **not** import `get_observability_authentication_scope` unless you need to override the default. If overriding, pass via `a365_observability_scope_override` to `use_microsoft_opentelemetry`. The `exchange_token()` call below omits `scopes=` and lets the auth handler resolve the default. - - Import `cache_agentic_token` from `token_cache` (the custom module created in Phase 5) — or use `AgenticTokenCache` from the hosting helpers. - - **OBO paths only** (`obo` / `agentic-user`): Resolve `agent_id` and `tenant_id` dynamically from context each turn (never from config), then exchange the OBO token (non-fatal, wrap in try/except): +2. **Edit `host_agent_server.py`** (the host file) — Prefetch the app-only telemetry token per turn following the reference pattern in `python-observability.md`: + - **OBO paths** (`obo` / `agentic-user`): Resolve `agent_id` and `tenant_id` dynamically from context each turn (never from config), then prefetch the app-only token (non-fatal, wrap in try/except). `a365_token_resolver` is synchronous, so the async FMI acquisition happens here and the resolver only reads the cache: ```python agent_id = context.activity.recipient.agentic_app_id tenant_id = context.activity.recipient.tenant_id await self._setup_observability_token(context, tenant_id, agent_id) ``` - The `_setup_observability_token` helper exchanges and caches the token: + The `_setup_observability_token` helper prefetches and caches the app-only token: ```python async def _setup_observability_token(self, context, tenant_id, agent_id): - exaau_token = await self.agent_app.auth.exchange_token( - context, - scopes=get_observability_authentication_scope(), - auth_handler_id=self.auth_handler_name # from config — NOT hardcoded "AGENTIC" - ) - cache_agentic_token(tenant_id, agent_id, exaau_token.token) + try: + await OBS_TOKENS.prefetch(self.connection_manager, tenant_id, agent_id) + except Exception as e: + logger.warning(f"Failed to acquire observability token: {e}") ``` - - `auth_handler_name` must come from config (e.g., `AgentApplication:AgenticAuthHandlerName`) — **never hardcode `"AGENTIC"`**; it is the registered auth handler name in your agent setup. - - `agentic-user` (AI Teammate): the exchange returns a token for the **agent's own Agentic User** identity → traces attribute to the agent - - `obo` (non-AI Teammate): the exchange returns whatever the configured auth handler resolves — typically the **signed-in user**, but it can also be the agent's own identity if the handler is configured that way + - `self.connection_manager` is the host's `MsalConnectionManager` (the object passed to `CloudAdapter(connection_manager=...)`). Its default connection is the blueprint credential, so no auth handler or extra settings are involved. `CloudAdapter` does not expose it publicly. If the host only keeps it in a local variable, store it on the host with `self.connection_manager = ...` and pass that same object to the adapter. Never create a second one. + - Call the helper at the start of **every handler that emits spans**: the message handler and any notification handler (`on_agent_notification`). The sync resolver can only serve tokens a prefetch cached. + - Do **NOT** call `exchange_token(...)` with `get_observability_authentication_scope()`, or cache its result with `cache_agentic_token`. That is a delegated exchange: it needs admin consent, and the S2S route rejects the token. Replace any existing body that does this (see "Migrating delegated telemetry" in Phase 3). - **S2S path**: Do **NOT** call `_setup_observability_token` — token comes from the background token service wired in Phase 3. The handler should NOT touch tokens. - **Baggage:** No manual baggage construction in the handler. Phase 3 registered `ObservabilityHostingManager.configure(adapter.middleware_set, ObservabilityHostingOptions(enable_baggage=True))` which auto-populates baggage from `TurnContext` for every request. (Optional fallback if you skipped that: build manually with `populate(builder, context)` then `with builder.build():`.) - - Add inline comment: `# A365 auth mode: {authMode} — see: https://learn.microsoft.com/en-us/entra/agent-id/agent-on-behalf-of-oauth-flow` + - Add inline comment: `# A365 auth mode: {authMode} — telemetry uses an app-only token on the S2S route` - Mark all new lines with: `# A365 Observability — best-effort instrumentation (verify against official sample)` 3. **Preserve** all existing handler logic. @@ -451,29 +437,27 @@ flag live in the references — see the "Required packages" section of: --- -## Phase 5: Implement Agentic Token Resolver +## Phase 5: Implement App-only Token Resolver -**TaskCreate** — "Implement agentic token resolver with caching" +**TaskCreate** — "Implement app-only token resolver with caching" -For AI Teammate agents and Standard agents on the OBO/agentic-user path, the built-in token cache handles caching automatically — no custom resolver needed. With the **`Microsoft.OpenTelemetry` distro** the cache is auto-registered by `UseMicrosoftOpenTelemetry(...)` (Phase 3). With the **legacy individual packages** it's registered explicitly via `AddAgenticTracingExporter` (.NET), `AgenticTokenCacheInstance` (Node.js), or `AgenticTokenCache` (Python). Skip to step 3 for these agents. +Every path exports with an app-only token for the exporting agent identity. `obo` / `agentic-user` agents use the app-only resolver scaffold, which reuses the hosting connection (blueprint credential). `s2s` agents use the background token-service scaffold. Phase 3 normally writes these files; this phase makes sure they exist on re-runs or partial states. **Never** fall back to the distro's delegated token caches (`IExporterTokenCache`, `AgenticTokenCacheInstance`, `AgenticTokenCache`) or to a legacy `Use_Custom_Resolver` cache of delegated tokens. -### For .NET AgentFramework (hosting path) +### For .NET AgentFramework (OBO / agentic-user path) -1. The distro's `builder.UseMicrosoftOpenTelemetry(...)` call (Phase 3) **auto-registers `IExporterTokenCache`** in DI — no separate `AddAgenticTracingExporter()` call is needed. If you're on the legacy two-package wiring, `AddAgenticTracingExporter()` provides the same DI instance. - -2. In the agent class, inject `IExporterTokenCache` in the constructor and call `RegisterObservability(...)` per turn (already done in Phase 4). +**Check** that `Observability/AgentAppTokenResolver.cs` exists and is registered with `builder.Services.AddSingleton()`. Also check that `o.Agent365.UseS2SEndpoint = true` and `o.Agent365.TokenResolver` are set, with `obsTokens` resolved from `app.Services` after `Build()`. If any piece is missing, add it from `dotnet-observability.md` ("Scaffold: `Observability/AgentAppTokenResolver.cs`"). ### For .NET AgentFramework (S2S path) -The `ObservabilityTokenService` background service (created in Phase 3 via the scaffold) acquires and refreshes the Observability API token automatically via the FMI 3-hop chain (Blueprint → Agent Identity → Power Platform PFAT token) — no manual `TokenResolver` delegate needed. +The `ObservabilityTokenService` background service (created in Phase 3 via the scaffold) acquires and refreshes the Observability API token automatically via the FMI 3-hop chain (Blueprint → Agent Identity → Observability API) — no manual `TokenResolver` delegate needed beyond the cache read wired in Phase 3. 1. **Check** if `Observability/ObservabilityServiceExtensions.cs` and `Observability/ObservabilityTokenService.cs` exist. If yes, **skip** — they were already created in Phase 3. 2. **If absent** (Phase 3 was skipped or re-running the skill on a partial state), create them now following the S2S scaffold patterns in `dotnet-observability.md`. These files provide `AddAgent365Observability()` (DI extension registering `AddServiceTracingExporter`, `ObservabilityTokenService`, and `Agent365ObservabilityContext`) and `ObservabilityTokenService` (background service that acquires the Observability API token via the FMI 3-hop chain and refreshes it every 50 minutes). -### For Node.js (OBO path) +### For Node.js (OBO / agentic-user path) -`AgenticTokenCacheInstance` from `@microsoft/agents-a365-observability-hosting` handles caching automatically. The `useMicrosoftOpenTelemetry()` call in Phase 3 wires it as the `tokenResolver`. No additional token resolver module is needed unless `Use_Custom_Resolver=true` is required (see reference doc for custom resolver pattern). +**Check** that `observability/app-token-resolver.ts` exists. Also check that `useMicrosoftOpenTelemetry()` passes `useS2SEndpoint: true` and `tokenResolver: appTokenResolver`, and that `getObsConnection` is assigned once the adapter exists. If anything is absent, add it from `nodejs-observability.md` ("App-only Token Resolver Scaffold"). ### For Node.js (S2S path) @@ -481,9 +465,9 @@ The `ObservabilityTokenService` background service (created in Phase 3 via the s **If absent** (Phase 3 was skipped or re-running), create `observability/token-cache.ts` and `observability/observability-token-service.ts` now using the scaffold from `nodejs-observability.md` (S2S section). The token service uses MSAL (`@azure/msal-node`) with `fmiPath` to acquire tokens via the FMI 3-hop chain targeting scope `api://9b975845-388f-4429-889e-eab1ef63949c/.default`. Call `startTokenService(config)` at app startup and pass `tokenResolver` from the cache module to `useMicrosoftOpenTelemetry()`. -### For Python (OBO path) +### For Python (OBO / agentic-user path) -The `token_cache.py` custom module (located at project root or `observability/token_cache.py`) provides `cache_agentic_token` and `get_cached_agentic_token`. The `a365_token_resolver` in `use_microsoft_opentelemetry()` (Phase 3) is wired to `get_cached_agentic_token`. The per-turn `_setup_observability_token` helper (Phase 4) calls `cache_agentic_token` after each OBO exchange. If `token_cache.py` is absent (e.g., this phase is reached before Phase 4 ran), create it now following the OBO token cache pattern in `python-observability.md`. +**Check** that `observability/app_token_resolver.py` exists and that `use_microsoft_opentelemetry()` receives `a365_token_resolver=OBS_TOKENS.resolve` and `a365_use_s2s_endpoint=True`. Also check that the Phase 4 `_setup_observability_token` helper awaits `OBS_TOKENS.prefetch(...)`. If anything is absent, add it from `python-observability.md` ("App-only Token Resolver Scaffold"). ### For Python (S2S path) @@ -573,14 +557,12 @@ For LangChain + OpenAI, follow the reference patterns in `nodejs-observability.m **Canonical pattern (generate exactly this shape):** ```typescript -await preloadObservabilityToken(turnContext); // STEP 1 — refresh token (cold-turn fix) - -const baggageScope = BaggageBuilderUtils // STEP 2 — outer baggage scope +const baggageScope = BaggageBuilderUtils // STEP 1 — outer baggage scope .fromTurnContext(new BaggageBuilder(), turnContext as any) .sessionDescription('agent-turn') .build(); -await baggageScope.run(async () => { // STEP 3 — scopes run INSIDE baggage +await baggageScope.run(async () => { // STEP 2 — scopes run INSIDE baggage const scope = InvokeAgentScope.start(request, scopeDetails, agentDetails, callerDetails); try { await scope.withActiveSpanAsync(async () => { @@ -591,7 +573,7 @@ await baggageScope.run(async () => { ``` **Why this exact shape:** -- **Without `preloadObservabilityToken` before `baggageScope.run`**, the first export attempt on a cold turn sees an empty token, retries until timeout, and the span is not exported. +- **No token step in the handler.** The app-only `tokenResolver` (Phase 3) acquires and caches the export token on demand, so a cold turn does not need a preload. Do not reintroduce `preloadObservabilityToken` / `refreshObservabilityToken`. - **Without the outer `baggageScope.run` wrapping `InvokeAgentScope.start`**, the spans have no `microsoft.tenant.id` / `gen_ai.agent.id` baggage attached — the exporter filters them as `Partitioned into 0 identity groups (N spans skipped)` and they never reach MAC. **Additional rules:** @@ -656,13 +638,19 @@ Apply these invariants across all three languages: - Node.js: `agent365Observability__sponsorUserId / __sponsorUserName / __sponsorUserEmail` in `.env`. - Python: same keys exposed via the resolver — see python-observability.md. -5. **S2S-only additions:** - - .NET: add `ClientId`, `ClientSecret`, and `UseManagedIdentity: false` (for +5. **S2S route (all auth modes) and S2S-only additions:** + - All languages: the S2S route flag is set in code in Phase 3 for **every** auth mode: + `o.Agent365.UseS2SEndpoint = true` (.NET), `useS2SEndpoint: true` (Node), or + `a365_use_s2s_endpoint=True` (Python). Keep it in code: Python's `A365_USE_S2S_ENDPOINT` + env fallback is fragile (the code validator flags env-only selection), and the legacy + `AGENT365_USE_S2S_ENDPOINT` env var is ignored. + - `obo` / `agentic-user`: no extra settings — the app-only resolver reads the blueprint + credential from the existing connection settings that `a365 setup all` wrote: `Connections` + (.NET), `connections__service_connection__settings__*` (Node.js), or + `CONNECTIONS__SERVICE_CONNECTION__SETTINGS__*` (Python). + - `s2s`, .NET: add `ClientId`, `ClientSecret`, and `UseManagedIdentity: false` (for local dev — MSI fails off-Azure with `CredentialUnavailableError`) under `Agent365Observability`. - - Node.js / Python: `useS2SEndpoint: true` (Node) / `a365_use_s2s_endpoint=True` - (Python) is set in code in Phase 3 — no env var equivalent in 1.0+. The - legacy `AGENT365_USE_S2S_ENDPOINT` env var is ignored. 6. **Inform the user** when: - `AgentBlueprintId` / `TenantId` are empty → "run `a365 setup` to populate". @@ -797,20 +785,22 @@ Without this phase the skill ends "instrumented successfully" but the user has n **What to expect for MAC visibility (first-run reality check):** - **Indexing lag: 15–90 minutes** between first successful export and spans appearing in `admin.cloud.microsoft → Advanced Hunting → CloudAppEvents`. If you query immediately after instrumenting, you'll see empty results — that's not a bug. - **Instance approval required.** Spans only attribute to a `CloudAppEvents` row when the AI Teammate's agent instance has been approved at `admin.cloud.microsoft/#/agents/all/requested` and an Agentic User UPN has been issued. Without that, exported spans land but don't surface in MAC queries. - - **KQL filter MUST use the AUID, NOT the blueprint id.** The `AgentId` column in `CloudAppEvents` is the runtime AUID resolved from `turnContext.activity.recipient.agenticAppId`. The `agent365Observability__agentId` env var that `a365 setup all` stamps into `.env` is the BLUEPRINT id — filtering by that value returns empty results. Get the AUID from your agent logs (the exporter logs `Obtained token for agent tenant ...`) or from `recipient.agenticAppId` in any inbound activity. + - **KQL filter MUST use the AUID, NOT the blueprint id.** The `AgentId` column in `CloudAppEvents` is the runtime AUID resolved from `turnContext.activity.recipient.agenticAppId`. For AI Teammates, the `agent365Observability__agentId` env var that `a365 setup all` stamps into `.env` is the BLUEPRINT id, and filtering by that value returns empty results. Get the AUID from your agent logs (the exporter logs `Obtained token for agent tenant ...`) or from `recipient.agenticAppId` in any inbound activity. + - **OBS authorization.** Telemetry is exported over the S2S route with an app-only token. Blueprint agents: no OBS permission or consent step — registered instances are authorized without `Agent365.Observability.OtelWrite` (subject to service policy); on 403 `insufficient_scope`, run `a365 setup all --agent-registration-only`. AI Teammates: complete the Observability API S2S app-role action item that `a365 setup all --aiteammate` printed, if it is still pending. **Verbose logging — only enable when actively debugging:** - Node.js / Python: uncomment `OTEL_LOG_LEVEL=INFO` AND `A365_OBSERVABILITY_LOG_LEVEL=info|warn|error` in `.env`. **Both** are required to see `[Agent365Exporter]` activity — the exporter uses a wrapped logger that defaults to silent. - Grep for `exported successfully` / `export-group succeeded` to confirm spans are flowing; `Partitioned into 0 identity groups (N spans skipped)` for spans outside an active baggage scope is **expected** (early framework / middleware / health-ping spans) — not an error. - 3. [If authMode = obo] Confirm the OBO token exchange is working correctly. + 3. [If authMode = obo] Confirm the workload (MCP / Graph) OBO token exchange works. Telemetry does not use it — it uses the app-only resolver. - Signed-in user sub-type: verify the signed-in user's token is passed correctly. → OBO flow docs: https://learn.microsoft.com/en-us/entra/agent-id/agent-on-behalf-of-oauth-flow - Agentic identity sub-type: ensure the agentic user identity has been provisioned in Azure AD. → Identity docs: https://learn.microsoft.com/en-us/microsoft-agent-365/developer/identity 4. [If authMode = agentic-user] Confirm the agentic-user M365 license and identity are provisioned. → Identity docs: https://learn.microsoft.com/en-us/microsoft-agent-365/developer/identity - 5. [If authMode = s2s] No user token required — verify agent blueprint credentials are configured. + 5. [If authMode = s2s] No user token required — verify agent blueprint credentials and the agent identity (`AgentId`) are configured. → Auth flow docs: https://learn.microsoft.com/en-us/microsoft-agent-365/developer/authentication-flow + 6. [If delegated telemetry wiring was migrated] List each replaced call (per-turn refresh / token registration / delegated resolver) so the user can review the diff. All instrumented lines are marked with: // A365 Observability — best-effort instrumentation (verify against official sample) @@ -849,37 +839,21 @@ If expected files are not found: This skill is safe to rerun. On subsequent runs: - Skip package installation if packages already present -- Skip code edits if observability is already wired (detect by marker comments) +- Skip code edits only if observability is already wired **the current way**. Marker comments must be present, the S2S route flag must be set (`useS2SEndpoint: true` / `a365_use_s2s_endpoint=True` / `o.Agent365.UseS2SEndpoint = true`), and no delegated-telemetry signal may remain: `refreshObservabilityToken(..., authorization)`, `AgenticTokenCacheInstance.getObservabilityToken`, `AgenticTokenStruct` / `RegisterObservability(..., AgenticTokenStruct)`, `exchange_token(...)` for the observability scope, `cache_agentic_token`, or an `AgenticTokenCache` resolver. Marker comments alone are not enough — code written by earlier versions of this skill carries them on the delegated wiring. If any delegated signal is present or the route flag is missing, run "Migrating delegated telemetry" (Phase 3) instead of skipping. - Update configuration only if values are missing - Always revalidate the build --- -## S2S Known Issues and Workarounds - -### OtelWrite App Role Assignment - -`a365 setup all` **automatically grants** `Agent365.Observability.OtelWrite` to the Agent Identity SP (both delegated and application) for all newly provisioned agents. No Global Administrator is required for agents set up with this CLI version. - -**Upgrade path — agents provisioned before this CLI release:** OtelWrite must be granted manually. A Global Administrator must do one of the following: +## Observability Known Issues and Workarounds -Option A — Entra portal: -1. [Entra portal](https://entra.microsoft.com) > App registrations > select Blueprint app > API permissions -2. Add a permission > APIs my organization uses > search `9b975845-388f-4429-889e-eab1ef63949c` -3. Add both **Delegated** and **Application** `Agent365.Observability.OtelWrite` > Grant admin consent +### Authorization — registration, not OtelWrite -Option B — Graph API (read `agentIdentityClientId` from `a365.generated.config.json`): +Telemetry is exported over the S2S route with an app-only token for the agent identity. Registered agent instances are authorized on that route **without** `Agent365.Observability.OtelWrite` or admin consent (subject to service policy). `a365 setup all` no longer requests Observability API permissions for blueprint agents, and a failed or unverifiable registration now fails setup. -```bash -# Create a temp JSON body file (required on Windows due to az rest escaping) -echo '{"principalId":"","resourceId":"2a275186-1775-4439-8551-5438df22cdfc","appRoleId":"8f71190c-00c8-461d-a63b-f74abde9ba52"}' > body.json -az rest --method POST --url "https://graph.microsoft.com/v1.0/servicePrincipals//appRoleAssignments" --body @body.json -rm body.json -``` - -- `resourceId` `2a275186-...` is the Observability API SP object ID -- `appRoleId` `8f71190c-...` is the OtelWrite role ID -- For agents provisioned before CLI 1.1, this manual step is still required +- **403 `insufficient_scope` on export** → the agent instance is not registered and has no `OtelWrite` application role. Blueprint agents: run `a365 setup all --agent-registration-only` (idempotent), then retry. AI Teammates: that flag does not apply — complete the Observability API S2S app-role action item `a365 setup all --aiteammate` prints (the application role below). +- **Application role (AI Teammates, and fallback for blueprint agents):** a Global Administrator can grant the `Agent365.Observability.OtelWrite` **application** role on the Blueprint, which agent identities inherit. The S2S route accepts it. Use the PowerShell steps that `a365 setup all` prints, or the Entra portal: App registrations > Blueprint > API permissions > APIs my organization uses > `9b975845-388f-4429-889e-eab1ef63949c` > **Application** `Agent365.Observability.OtelWrite` > Grant admin consent. +- **Do not** add the delegated `OtelWrite` scope or run a delegated consent flow for telemetry. Delegated tokens are rejected by the S2S route. Only agents still on an old SDK that exports over the delegated route need that grant, and they should migrate instead (see "Migrating delegated telemetry" in Phase 3). ### Node.js and .NET SDK `/otlp/` URL Path Bug @@ -887,47 +861,42 @@ The Node.js SDK (`@microsoft/agents-a365-observability@0.2.0-preview.5`) and .NE **Status:** Awaiting SDK fix. No workaround should be applied in generated code — this is an SDK-level issue. -### S2S Endpoint Path — `useS2SEndpoint` Not Passed by Distro - -The `@microsoft/opentelemetry` distro creates `Agent365Exporter` internally but does NOT pass `useS2SEndpoint: true`. For S2S agents, the exporter defaults to the OBO path (`/observability/tenants/{tenantId}/otlp/agents/{agentId}/traces`), but S2S requires `/observabilityService/...`. +### S2S Endpoint Path — Set the Route Flag in Every Auth Mode -**Node.js (`@microsoft/opentelemetry` 1.0+ GA — FIXED):** +The distros default to the delegated route (`/observability/tenants/{tenantId}/otlp/agents/{agentId}/traces`), which rejects app-only tokens. Every agent — `agentic-user`, `obo`, and `s2s` — must select the S2S route (`/observabilityService/...`) explicitly: -`useS2SEndpoint` is now a first-class option in the `a365` options. Generated S2S agent code should pass it directly: +**Node.js (`@microsoft/opentelemetry` 1.0+):** ```typescript useMicrosoftOpenTelemetry({ a365: { enabled: true, enableObservabilityExporter: true, - useS2SEndpoint: true, // first-class option in 1.0+ - tokenResolver: a365TokenResolver, + useS2SEndpoint: true, // required in every auth mode + tokenResolver: appTokenResolver, // or the s2s token-cache resolver }, }); ``` The old `AGENT365_USE_S2S_ENDPOINT` env var workaround and custom `spanProcessors` workarounds are no longer needed. -**.NET (`Microsoft.OpenTelemetry` 1.0.x GA):** +**Python (`microsoft-opentelemetry` 1.1+):** pass `a365_use_s2s_endpoint=True` to `use_microsoft_opentelemetry(...)`. -The `UseMicrosoftOpenTelemetry()` builder extension does NOT set `UseS2SEndpoint = true` on the `Agent365ExporterOptions` when using the unified distro. Without this, the exporter posts to `/observability/` (OBO path) instead of `/observabilityService/` (S2S path), causing HTTP 401. +**.NET (`Microsoft.OpenTelemetry` 1.0.x GA):** -**Fix:** Set `UseS2SEndpoint = true` explicitly in the `UseMicrosoftOpenTelemetry` options callback: +The `UseMicrosoftOpenTelemetry()` builder extension does NOT set `UseS2SEndpoint = true`. Without it, the exporter posts to `/observability/` instead of `/observabilityService/`, causing HTTP 401. Set it explicitly. On 1.0.3+ the options live on `o.Agent365`; on 1.0.2 and earlier, use `o.Agent365.Exporter` instead: ```csharp builder.UseMicrosoftOpenTelemetry(o => { o.Exporters = ExportTarget.Agent365 | ExportTarget.Console; - o.Agent365.Exporter.UseS2SEndpoint = true; // ← Required for S2S agents - o.Agent365.Exporter.TokenResolver = async (agentId, tenantId) => - { - return tokenCache != null - ? await tokenCache.GetObservabilityToken(agentId, tenantId) - : null; - }; + o.Agent365.UseS2SEndpoint = true; // ← Required in every auth mode + // obo / agentic-user: AgentAppTokenResolver; s2s: the ServiceTokenCache read shown in the reference. + o.Agent365.TokenResolver = (agentId, tenantId) => + obsTokens?.ResolveAsync(agentId, tenantId) ?? Task.FromResult(null); }); ``` **URL paths:** -- OBO: `observability/tenants/{tenantId}/otlp/agents/{agentId}/traces` -- S2S: `observabilityService/tenants/{tenantId}/otlp/agents/{agentId}/traces` +- Legacy delegated route (do not use): `observability/tenants/{tenantId}/otlp/agents/{agentId}/traces` +- S2S route (all agents): `observabilityService/tenants/{tenantId}/otlp/agents/{agentId}/traces` ### Node.js MSAL `fmiPath` Not Supported (AADSTS82008) diff --git a/plugins/agent365/skills/instrument-observability/references/dotnet-observability.md b/plugins/agent365/skills/instrument-observability/references/dotnet-observability.md index ccdaf1d..bbcddb1 100644 --- a/plugins/agent365/skills/instrument-observability/references/dotnet-observability.md +++ b/plugins/agent365/skills/instrument-observability/references/dotnet-observability.md @@ -4,6 +4,22 @@ Authoritative package versions and code patterns for instrumenting A365 observab into a .NET AgentFramework agent. All samples mirror the official Microsoft Learn docs (updated 2026-04-30). +> **Telemetry always uses the S2S route.** Every agent (`agentic-user` AI Teammates, `obo`, +> and `s2s`) sets `o.Agent365.UseS2SEndpoint = true` and exports with an **app-only** token for +> the exporting agent identity. `authMode` only selects workload (MCP / Graph) auth and how the +> telemetry token is sourced: `AgentAppTokenResolver` (hosting connection) for `obo` / +> `agentic-user`, and `ObservabilityTokenService` (background FMI chain) for `s2s`. The S2S +> route rejects delegated (`scp`) tokens, so do not register per-turn OBO tokens +> (`RegisterObservability(..., new AgenticTokenStruct(...))`) for telemetry. Registered blueprint +> agent instances need no `Agent365.Observability.OtelWrite` permission or admin consent (subject +> to service policy), and `a365 setup all` no longer requests them for blueprint agents. AI Teammate +> setup (`a365 setup all --aiteammate`) still offers the `OtelWrite` application role; complete the +> app-role action item it prints, because the S2S route accepts that role. +> +> **Option paths by distro version:** `Microsoft.OpenTelemetry` 1.0.3+ exposes the exporter +> options directly on `o.Agent365` (`o.Agent365.UseS2SEndpoint`, `o.Agent365.TokenResolver`). +> Versions 1.0.2 and earlier nested them under `o.Agent365.Exporter`. + --- ## NuGet Packages @@ -11,7 +27,7 @@ into a .NET AgentFramework agent. All samples mirror the official Microsoft Lear | Package | Purpose | |---------|---------| | `Microsoft.Agents.A365.Observability.Runtime` | `AddA365Tracing()`, `BaggageBuilder`, `EnvironmentUtils` — required for all agents | -| `Microsoft.Agents.A365.Observability.Hosting` | `AddAgenticTracingExporter()` — OBO token caching (obo / agentic-user); `AddServiceTracingExporter()` — S2S token cache (`IExporterTokenCache`) | +| `Microsoft.Agents.A365.Observability.Hosting` | `AddAgenticTracingExporter()` — legacy delegated-route OBO token caching (do not use for telemetry); `AddServiceTracingExporter()` — S2S token cache (`IExporterTokenCache`) | | `Microsoft.Agents.A365.Observability.Hosting.Caching` | `IExporterTokenCache`, `AgenticTokenStruct` | | `Microsoft.Agents.A365.Observability.Hosting.Extensions` | `FromTurnContext()` extension on `BaggageBuilder` | | `Microsoft.Agents.A365.Observability.Hosting.Middleware` | `BaggageTurnMiddleware`, `UseObservabilityRequestContext` | @@ -27,9 +43,9 @@ Unified Distro (preferred — single package, GA as of 2026-05-01): | Package | Purpose | |---------|---------| -| `Microsoft.OpenTelemetry` (1.0.3 GA — latest stable) | All-in-one: includes A365 observability types (`BaggageBuilder`, `InvokeAgentScope`, `InferenceScope`, `ExecuteToolScope`, `IExporterTokenCache`, `ServiceTokenCache`, `AgentDetails`, etc.) plus OTel pipeline configuration. Targets `net8.0` and `netstandard2.0`. Auto-instrumentation toggles for SemanticKernel / OpenAI / AgentFramework / AspNetCore / HttpClient / SqlClient / AzureSdk are first-class options on `o.Instrumentation` (all default `true`). | +| `Microsoft.OpenTelemetry` (1.0.3+ GA; snippets verified with 1.1.0) | All-in-one: includes A365 observability types (`BaggageBuilder`, `InvokeAgentScope`, `InferenceScope`, `ExecuteToolScope`, `IExporterTokenCache`, `ServiceTokenCache`, `AgentDetails`, etc.) plus OTel pipeline configuration. Targets `net8.0` and `netstandard2.0`. Auto-instrumentation toggles for SemanticKernel / OpenAI / AgentFramework / AspNetCore / HttpClient / SqlClient / AzureSdk are first-class options on `o.Instrumentation` (all default `true`). | | `Azure.Identity` | `ManagedIdentityCredential` for MSI-based token acquisition | -| `Microsoft.Identity.Client` | MSAL `ConfidentialClientApplicationBuilder` with `.WithFmiPath()` for the FMI token chain | +| `Microsoft.Identity.Client` | MSAL `ConfidentialClientApplicationBuilder`: `.WithFmiPath()` for the S2S FMI token chain, and the agent-identity token exchange in `AgentAppTokenResolver` (already referenced transitively by `Microsoft.Agents.Authentication.Msal`) | Install commands (preferred for **all** paths — OBO / agentic-user / S2S / AI Teammate): ```bash @@ -42,6 +58,9 @@ dotnet add package Microsoft.OpenTelemetry # S2S path only (FMI token chain for ObservabilityTokenService): dotnet add package Azure.Identity dotnet add package Microsoft.Identity.Client + +# obo / agentic-user: no extra packages. AgentAppTokenResolver uses the agent's existing +# Microsoft.Agents.Authentication.Msal connection and its Microsoft.Identity.Client dependency. ``` > **Do NOT also add `Microsoft.Agents.A365.Observability.Runtime` or @@ -71,7 +90,7 @@ Use this pattern for Agent (Non AI Teammate) agents that run without a signed-in Requires two scaffold files in `Observability/` — create these before wiring Program.cs. > **⚠️ Expected configuration (1.0.x GA):** -> - **UseS2SEndpoint:** The distro does NOT set `UseS2SEndpoint = true` on the internal `Agent365Exporter`. You MUST set `o.Agent365.Exporter.UseS2SEndpoint = true` in the `UseMicrosoftOpenTelemetry` options callback, or the exporter posts to `/observability/` (OBO path) instead of `/observabilityService/` (S2S path), causing HTTP 401. +> - **UseS2SEndpoint:** The distro does NOT set `UseS2SEndpoint = true` on the internal `Agent365Exporter` (default `false`). You MUST set `o.Agent365.UseS2SEndpoint = true` in the `UseMicrosoftOpenTelemetry` options callback (`o.Agent365.Exporter.UseS2SEndpoint` on 1.0.2 and earlier). Otherwise the exporter posts to the delegated route (`/observability/`) instead of the S2S route (`/observabilityService/`), causing HTTP 401. This applies to every auth mode, not only S2S. > - **InferenceCallDetails:** The `providerName` parameter is required (not optional). Constructor: `(InferenceOperationType operationName, string model, string providerName, ...)`. > - **ExecuteToolScope.RecordResponse:** Takes `string`, not `Response` object. > - **UseManagedIdentity:** Set `false` for local dev. MSI only works on Azure infrastructure. @@ -182,7 +201,7 @@ public static class ObservabilityServiceExtensions > - `true` (production) — MSI → Blueprint FIC → Agent Identity → API > - `false` (local dev) — Client Secret → Blueprint FIC → Agent Identity → API > -> **Note:** As of CLI 1.1, `a365 setup all` automatically grants `Agent365.Observability.OtelWrite` to the Agent Identity SP (both delegated and application). No manual role assignment is needed for newly provisioned agents. +> **Authorization:** registered agent instances are authorized on the S2S route without the `Agent365.Observability.OtelWrite` permission or admin consent (subject to service policy), and `a365 setup all` no longer requests it for blueprint agents. If export returns 403 `insufficient_scope`, register a blueprint agent instance with `a365 setup all --agent-registration-only`; for AI Teammates, complete the `OtelWrite` application-role step that `a365 setup all --aiteammate` prints. Either way, a Global Administrator can grant the `OtelWrite` application role. ```csharp using Azure.Core; @@ -325,8 +344,9 @@ builder.UseMicrosoftOpenTelemetry(o => ? ExportTarget.Agent365 | ExportTarget.Console : ExportTarget.Agent365; - // ⚠️ Required for S2S: distro does NOT set this automatically (still manual opt-in in 1.0.x GA — defaults to false) - o.Agent365.Exporter.UseS2SEndpoint = true; + // ⚠️ Required in every auth mode: the distro does NOT set this automatically (defaults to false). + // 1.0.3+: o.Agent365.UseS2SEndpoint. 1.0.2 and earlier: o.Agent365.Exporter.UseS2SEndpoint. + o.Agent365.UseS2SEndpoint = true; // Auto-instrumentation toggles (all default `true` in 1.0.x — uncomment to opt out) // o.Instrumentation.EnableSemanticKernelInstrumentation = false; @@ -337,7 +357,7 @@ builder.UseMicrosoftOpenTelemetry(o => // o.Instrumentation.EnableSqlClientInstrumentation = false; // o.Instrumentation.EnableAzureSdkInstrumentation = false; - o.Agent365.Exporter.TokenResolver = async (agentId, tenantId) => + o.Agent365.TokenResolver = async (agentId, tenantId) => { return tokenCache != null ? await tokenCache.GetObservabilityToken(agentId, tenantId) @@ -370,10 +390,12 @@ using Microsoft.OpenTelemetry; var builder = WebApplication.CreateBuilder(args); +// A365 Observability — app-only token resolver for S2S export (see the +// "Scaffold: Observability/AgentAppTokenResolver.cs" section below). +builder.Services.AddSingleton(); + // Microsoft OpenTelemetry distro — configures OTel pipeline + A365 exporter in one call. -// For the OBO / agentic-user path the distro AUTO-REGISTERS -// IExporterTokenCache in DI, so MyAgent can inject it without any -// explicit AddAgenticTracingExporter() / AddA365Tracing() calls. +AgentAppTokenResolver? obsTokens = null; // resolved after Build() builder.UseMicrosoftOpenTelemetry(o => { o.Exporters = builder.Environment.IsDevelopment() @@ -392,9 +414,12 @@ builder.UseMicrosoftOpenTelemetry(o => // o.Instrumentation.EnableAgentFrameworkInstrumentation = false; // o.Instrumentation.EnableSqlClientInstrumentation = false; - // For OBO / agentic-user, leave UseS2SEndpoint at its default (false) — the exporter - // will POST to `/observability/` which the OBO token cache authenticates. Only flip - // this to true on the S2S Path. + // Every auth mode exports over the S2S route with an app-only token for the exporting + // agent identity; the S2S route rejects delegated (OBO / Agentic User) tokens. + // 1.0.3+: o.Agent365.*. 1.0.2 and earlier: o.Agent365.Exporter.*. + o.Agent365.UseS2SEndpoint = true; + o.Agent365.TokenResolver = (agentId, tenantId) => + obsTokens?.ResolveAsync(agentId, tenantId) ?? Task.FromResult(null); }); // Required: IChatClient registration with `.UseOpenTelemetry(...)` — this is what makes @@ -423,8 +448,9 @@ builder.Services.AddSingleton(sp => var app = builder.Build(); -// Token caching is automatic — MyAgent calls `_agentTokenCache.RegisterObservability(...)` -// per turn (see "Agent Class — Message Handler (OBO Path)" section below). +// No per-turn token registration: the exporter calls AgentAppTokenResolver, which acquires and +// caches an app-only token per agent identity from the agent's default (blueprint) connection. +obsTokens = app.Services.GetRequiredService(); ``` > **Two `UseOpenTelemetry()` calls are required** to get a complete trace: @@ -435,10 +461,144 @@ var app = builder.Build(); > wrapper itself emits spans. The `InvokeAgentScope` parent becomes a hollow span with no > `InferenceCall` children. +### Scaffold: `Observability/AgentAppTokenResolver.cs` + +Turns the agent's **existing** default connection (the blueprint credential) into an app-only +Observability API token for whichever agent identity is exporting. Tokens are cached per agent +instance, so multi-instance AI Teammates need no per-instance configuration: + +``` +Blueprint credential (default connection) + → Step 1: FMI assertion for the agent identity IAgenticTokenProvider.GetAgenticApplicationTokenAsync(tenantId, agentId) + → Step 2: agent identity client_credentials scope=api://9b975845-388f-4429-889e-eab1ef63949c/.default +``` + +The exporter passes `(agentId, tenantId)` from span baggage, so no agent ID is configured here. + +```csharp +// A365 Observability — best-effort instrumentation (verify against official sample) +using System.Collections.Concurrent; +using System.Text.Json; +using Microsoft.Agents.Authentication; +using Microsoft.Extensions.Logging; +using Microsoft.Identity.Client; + +namespace ; + +// App-only token for A365 observability export over the S2S route. Workload auth (OBO or +// agentic user for MCP and Graph) is unchanged; only the telemetry credential differs. +// Step 1: the agent's default connection (the blueprint credential) issues an FMI assertion +// for the exporting agent identity: GetAgenticApplicationTokenAsync(tenantId, agentId). +// Step 2: the agent identity exchanges that assertion (client_credentials) for an app-only +// Observability API token. The S2S route rejects delegated (scp) tokens. +public sealed class AgentAppTokenResolver +{ + private static readonly string[] ObservabilityScopes = ["api://9b975845-388f-4429-889e-eab1ef63949c/.default"]; + private static readonly TimeSpan RefreshSkew = TimeSpan.FromMinutes(5); + + private readonly IConnections _connections; + private readonly ILogger _logger; + private readonly ConcurrentDictionary _tokens = new(); + private readonly SemaphoreSlim _refreshGate = new(1, 1); + + public AgentAppTokenResolver(IConnections connections, ILogger logger) + { + _connections = connections; + _logger = logger; + } + + // Wired as o.Agent365.TokenResolver. The exporter calls it for every export batch, + // so tokens are cached per agent identity and refreshed shortly before they expire. + public async Task ResolveAsync(string agentId, string tenantId) + { + if (string.IsNullOrWhiteSpace(agentId) || string.IsNullOrWhiteSpace(tenantId)) + { + return null; + } + + var key = $"{tenantId}:{agentId}".ToLowerInvariant(); + if (TryGetFresh(key, out var token)) + { + return token; + } + + await _refreshGate.WaitAsync().ConfigureAwait(false); + try + { + if (TryGetFresh(key, out token)) + { + return token; + } + + if (_connections.GetDefaultConnection() is not IAgenticTokenProvider blueprint) + { + _logger.LogWarning("The default connection cannot issue agentic tokens; A365 observability export skipped."); + return null; + } + + var assertion = await blueprint.GetAgenticApplicationTokenAsync(tenantId, agentId).ConfigureAwait(false); + var result = await ConfidentialClientApplicationBuilder + .Create(agentId) + .WithClientAssertion((AssertionRequestOptions _) => Task.FromResult(assertion)) + .WithAuthority(new Uri($"https://login.microsoftonline.com/{tenantId}")) + .Build() + .AcquireTokenForClient(ObservabilityScopes) + .ExecuteAsync() + .ConfigureAwait(false); + + if (HasDelegatedScope(result.AccessToken)) + { + _logger.LogWarning("The observability token carries a delegated scp claim, which the S2S route rejects; export skipped."); + return null; + } + + _tokens[key] = result; + return result.AccessToken; + } + catch (Exception ex) + { + _logger.LogWarning(ex, "Could not acquire an app-only A365 observability token for agent {AgentId}; export skipped.", agentId); + return null; + } + finally + { + _refreshGate.Release(); + } + } + + private bool TryGetFresh(string key, out string? token) + { + token = _tokens.TryGetValue(key, out var result) && result.ExpiresOn - DateTimeOffset.UtcNow > RefreshSkew + ? result.AccessToken + : null; + return token is not null; + } + + private static bool HasDelegatedScope(string accessToken) + { + var payload = accessToken.Split('.')[1].Replace('-', '+').Replace('_', '/'); + payload = payload.PadRight(payload.Length + (4 - payload.Length % 4) % 4, '='); + using var claims = JsonDocument.Parse(Convert.FromBase64String(payload)); + return claims.RootElement.TryGetProperty("scp", out _); + } +} +``` + +> `IConnections` is registered by the Agents SDK hosting setup (`AddAgentApplicationOptions` / +> `AddAgent`), and its default connection is the blueprint credential that `a365 setup all` +> writes to `appsettings.json` (`Connections:ServiceConnection`). No extra settings are needed. +> Registered blueprint agent instances are authorized on the S2S route without the +> `Agent365.Observability.OtelWrite` permission or admin consent (subject to service policy). For AI +> Teammates, complete the `OtelWrite` application-role step that `a365 setup all --aiteammate` prints. + ### Required appsettings.json keys (OBO / AI Teammate) `EnableAgent365Exporter` must be `true` — the SDK defaults it to `false` when absent, so -without it the exporter is wired but inert: +without it the exporter is wired but inert. `AgentId` is only a fallback for turns without an +agentic identity: `a365 setup all` writes the provisioned agent identity there for +non-AI-Teammate blueprint agents, and the blueprint ID for AI Teammates, which the handler +ignores. `ClientId` / `ClientSecret` are not read on this path: `AgentAppTokenResolver` uses the +`Connections` settings. ```json { @@ -466,7 +626,9 @@ builder.AddA365Tracing(); // from Microsoft.Agents.A365.O ``` These are subsumed by `UseMicrosoftOpenTelemetry()` and the distro package — mixing the -two causes CS0433 duplicate-type errors. Pick one wiring style per project. +two causes CS0433 duplicate-type errors. Pick one wiring style per project. The legacy +`AddAgenticTracingExporter()` wiring exports over the delegated route with OBO tokens; migrate +it to the distro with `o.Agent365.UseS2SEndpoint = true` and `AgentAppTokenResolver`. --- @@ -531,13 +693,18 @@ branching on `turnContext.IsAgenticRequest()`. For non-AI-Teammate S2S agents, u Path section further down instead — autonomous agents can run on either OBO or S2S, so the S2S Path applies specifically to the S2S auth mode, not to "autonomous" as a whole. -The message handler needs five things: +The message handler needs four things: -1. **Robust agent id resolution** — `Activity.GetAgenticInstanceId()` for agentic requests, `Utility.ResolveAgentIdentity(turnContext, oboToken)` for OBO requests. +1. **Robust agent id resolution** — `Activity.GetAgenticInstanceId()` for agentic requests. For other turns, use the provisioned agent identity in `Agent365Observability:AgentId`, but only when it is a GUID different from `AgentBlueprintId`. S2S export authenticates *as* this identity, so it must never be the blueprint. 2. **Baggage propagation** (`BaggageBuilder`) so the distro's `ActivityProcessor` copies tenant/agent id onto every child `gen_ai` span. Without this, the exporter logs `"spans skipped due to missing tenant or agent ID"`. -3. **Per-turn token registration** (`IExporterTokenCache.RegisterObservability`) so the exporter can OBO-exchange a token to POST traces. -4. **An `InvokeAgentScope` wrapping the LLM call** with `CallerDetails`. This emits the **`InvokeAgent`** event the MAC portal needs as the parent record for the trace UI — without it, Advanced Hunting shows only orphan `InferenceCall` / `ExecuteToolBySDK` rows and the agent-turn view never renders. -5. **A graceful skip when no real (agent, tenant) tuple is available**. Falling back to `Guid.Empty` creates a synthetic identity the exporter cannot authenticate, polluting traces with orphan groups and producing `"No token obtained. Skipping export for this identity."` warnings. +3. **An `InvokeAgentScope` wrapping the LLM call** with `CallerDetails`. This emits the **`InvokeAgent`** event the MAC portal needs as the parent record for the trace UI — without it, Advanced Hunting shows only orphan `InferenceCall` / `ExecuteToolBySDK` rows and the agent-turn view never renders. +4. **A graceful skip when no real (agent, tenant) tuple is available**. Falling back to `Guid.Empty` creates a synthetic identity the exporter cannot authenticate, polluting traces with orphan groups and producing `"No token obtained. Skipping export for this identity."` warnings. + +There is **no per-turn token registration**: `AgentAppTokenResolver` (wired as +`o.Agent365.TokenResolver`) acquires the app-only token when spans are exported. Remove any +existing `IExporterTokenCache` injection and +`RegisterObservability(..., new AgenticTokenStruct(...), ...)` call. That registration feeds +delegated tokens to the exporter, which the S2S route rejects. > **All `Microsoft.Agents.A365.Observability.*` types referenced below flow transitively > through the `Microsoft.OpenTelemetry` distro package.** Do NOT add direct @@ -546,16 +713,15 @@ The message handler needs five things: ```csharp using Microsoft.Agents.Builder; -using Microsoft.Agents.Builder.App.UserAuth; +using Microsoft.Agents.Builder.App; +using Microsoft.Agents.Builder.State; using Microsoft.Extensions.Configuration; using Microsoft.Extensions.Logging; -// The four observability namespaces below are re-exported by Microsoft.OpenTelemetry; +// The observability namespaces below are re-exported by Microsoft.OpenTelemetry; // no separate package install required. -using Microsoft.Agents.A365.Observability.Hosting.Caching; using Microsoft.Agents.A365.Observability.Runtime.Common; using Microsoft.Agents.A365.Observability.Runtime.Tracing.Contracts; using Microsoft.Agents.A365.Observability.Runtime.Tracing.Scopes; -using Microsoft.Agents.A365.Runtime.Utils; // Utility.ResolveAgentIdentity using System; using System.Threading; using System.Threading.Tasks; @@ -564,23 +730,16 @@ using ObsRequest = Microsoft.Agents.A365.Observability.Runtime.Tracing.Contracts public class MyAgent : AgentApplication { - private readonly IExporterTokenCache? _agentTokenCache; private readonly IConfiguration _configuration; private readonly ILogger _logger; - private readonly string? AgenticAuthHandlerName; - private readonly string? OboAuthHandlerName; public MyAgent( AgentApplicationOptions options, - IExporterTokenCache agentTokenCache, IConfiguration configuration, ILogger logger) : base(options) { - _agentTokenCache = agentTokenCache; _configuration = configuration; _logger = logger; - AgenticAuthHandlerName = configuration["AgentApplication:AgenticAuthHandlerName"]; - OboAuthHandlerName = configuration["AgentApplication:OboAuthHandlerName"]; } protected async Task OnMessageAsync( @@ -588,41 +747,35 @@ public class MyAgent : AgentApplication ITurnState turnState, CancellationToken cancellationToken) { - // 1. Select auth handler for this turn (agentic vs OBO) - var authHandlerName = turnContext.IsAgenticRequest() - ? AgenticAuthHandlerName - : OboAuthHandlerName; - - // 2. Resolve agent id — for agentic turns from Activity, for OBO turns from the user token - string? resolvedAgentId = null; - if (turnContext.Activity.IsAgenticRequest()) - { - resolvedAgentId = turnContext.Activity.GetAgenticInstanceId(); - } - else if (!string.IsNullOrEmpty(authHandlerName)) + var obsConfig = _configuration.GetSection("Agent365Observability"); + + // 1. Resolve the exporting agent identity. S2S export authenticates as this identity + // with an app-only token, so it must be an agent identity, never the blueprint. + // Agentic turns (AI Teammate) carry it on the activity; other turns fall back to the + // provisioned agent identity that `a365 setup all` writes to Agent365Observability:AgentId + // for non-AI-Teammate blueprint agents. + string? resolvedAgentId = turnContext.Activity.IsAgenticRequest() + ? turnContext.Activity.GetAgenticInstanceId() + : obsConfig["AgentId"]; + if (!Guid.TryParse(resolvedAgentId, out _) + || string.Equals(resolvedAgentId, obsConfig["AgentBlueprintId"], StringComparison.OrdinalIgnoreCase)) { - try - { - var oboToken = await UserAuthorization.GetTurnTokenAsync( - turnContext, authHandlerName, cancellationToken: cancellationToken).ConfigureAwait(false); - if (!string.IsNullOrEmpty(oboToken)) - { - resolvedAgentId = Utility.ResolveAgentIdentity(turnContext, oboToken); - } - } - catch (Exception ex) - { - _logger.LogDebug(ex, "Could not resolve agent id from OBO token; A365 observability skipped for this turn."); - } + resolvedAgentId = null; } var resolvedTenantId = turnContext.Activity.Conversation?.TenantId - ?? turnContext.Activity.Recipient?.TenantId; + ?? turnContext.Activity.Recipient?.TenantId + ?? obsConfig["TenantId"]; var hasObservabilityIdentity = !string.IsNullOrEmpty(resolvedAgentId) && !string.IsNullOrEmpty(resolvedTenantId); + if (!hasObservabilityIdentity) + { + _logger.LogDebug("No agent identity for this turn; A365 observability skipped."); + } - // 3. Set baggage and register the token ONLY when we have a real identity. + // 2. Set baggage ONLY when we have a real identity. No per-turn token registration: + // the exporter's TokenResolver (AgentAppTokenResolver) acquires the app-only token. // Build() returns IDisposable; `using` accepts null and skips disposal. using IDisposable? baggageScope = hasObservabilityIdentity ? new BaggageBuilder() @@ -631,39 +784,18 @@ public class MyAgent : AgentApplication .Build() : null; - if (hasObservabilityIdentity) - { - try - { - _agentTokenCache?.RegisterObservability( - resolvedAgentId!, - resolvedTenantId!, - new AgenticTokenStruct( - userAuthorization: UserAuthorization, - turnContext: turnContext, - authHandlerName: authHandlerName ?? string.Empty), - EnvironmentUtils.GetObservabilityAuthenticationScope()); - } - catch (Exception ex) - { - _logger.LogWarning(ex, "Failed to register observability token."); - } - } - - // 4. Open an InvokeAgentScope around the LLM call so an "InvokeAgent" event is emitted. + // 3. Open an InvokeAgentScope around the LLM call so an "InvokeAgent" event is emitted. // Required for MAC Advanced Hunting to render the agent turn UI and anchor children. InvokeAgentScope? invokeScope = null; if (hasObservabilityIdentity) { - var obsConfig = _configuration.GetSection("Agent365Observability"); - // Write BOTH identity dimensions so MAC shows per-instance AND blueprint-rolled-up // activity. They become separate span tags: // AgentId → gen_ai.agent.id (this agentic INSTANCE) // AgentBlueprintId → microsoft.a365.agent.blueprint.id (MAC roll-up to the blueprint) // If EITHER is empty MAC loses that grouping dimension. AgentId is resolved live - // (GetAgenticInstanceId() = Recipient.AgenticAppId, or the OBO token); the .NET - // recipient has NO blueprint field, so AgentBlueprintId MUST come from config + // (GetAgenticInstanceId() = Recipient.AgenticAppId, or the configured agent identity); + // the .NET recipient has NO blueprint field, so AgentBlueprintId MUST come from config // (stamped by `a365 setup all`) — guard against it being empty. var blueprintId = obsConfig["AgentBlueprintId"] ?? string.Empty; if (string.IsNullOrEmpty(blueprintId)) @@ -832,7 +964,7 @@ var chatAgent = new ChatClientAgent(chatClient, chatClientOptions) } ``` -The `Microsoft.Agents.A365.Observability: Debug` log level is the key signal — it surfaces exporter activity (`Sending chunk ... to .../observability/tenants/.../traces`, `HTTP 200 exporting spans`, `Partitioned into N identity groups`, etc.). Drop it to `Warning` in production. +The `Microsoft.Agents.A365.Observability: Debug` log level is the key signal — it surfaces exporter activity (`Sending chunk ... to .../observabilityService/tenants/.../traces`, `HTTP 200 exporting spans`, `Partitioned into N identity groups`, etc.). Drop it to `Warning` in production. ### Verifying end-to-end @@ -844,11 +976,13 @@ Agent365Exporter: Exporting batch of N spans. [Agent365Exporter] Partitioned into K identity groups (X spans skipped) Agent365ExporterCore: Obtained token for agent tenant . Agent365ExporterCore: Sending chunk 1 of 1 (J spans, B bytes) - to https://agent365.svc.cloud.microsoft/observability/tenants//otlp/agents//traces?api-version=1. + to https://agent365.svc.cloud.microsoft/observabilityService/tenants//otlp/agents//traces?api-version=1. Agent365ExporterCore: HTTP 200 exporting spans. 'x-ms-correlation-id': ''. ``` -`HTTP 200 exporting spans` confirms the export reached the backend. In **MAC Advanced Hunting** (1–5 min ingestion lag): +`HTTP 200 exporting spans` confirms the export reached the backend. A `/observability/` (not +`/observabilityService/`) URL in that line means `o.Agent365.UseS2SEndpoint = true` is missing. +In **MAC Advanced Hunting** (1–5 min ingestion lag): ```kql CloudAppEvents @@ -864,7 +998,7 @@ CloudAppEvents ## Agent Class — Message Handler (S2S Path, `authMode: s2s`) -Inject `Agent365ObservabilityContext` instead of `IExporterTokenCache`. +Inject `Agent365ObservabilityContext`; there is no `IExporterTokenCache` on any path. `ObservabilityTokenService` holds the token in the background — no per-turn `RegisterObservability` call. ```csharp @@ -1263,18 +1397,19 @@ warn: Agent365ExporterCore: No token obtained for agent {agentId} tenant {tenant | Type | Namespace | Purpose | |------|-----------|---------| | `BaggageBuilder` | `Microsoft.Agents.A365.Observability.Runtime.Common` | Propagates context across spans; `Build()` returns `IDisposable` — use `using var` | -| `EnvironmentUtils` | `Microsoft.Agents.A365.Observability.Runtime.Common` | `GetObservabilityAuthenticationScope()` helper | -| `IExporterTokenCache` | `Microsoft.Agents.A365.Observability.Hosting.Caching` | DI interface for caching and retrieving agentic tokens | +| `EnvironmentUtils` | `Microsoft.Agents.A365.Observability.Runtime.Common` | `GetObservabilityAuthenticationScope()` helper (not needed for telemetry: the token resolvers request the scope themselves) | +| `IExporterTokenCache` | `Microsoft.Agents.A365.Observability.Hosting.Caching` | DI interface for caching exporter tokens. The S2S path uses `IExporterTokenCache`; the `AgenticTokenStruct` form is legacy (delegated route) | | `ServiceTokenCache` | `Microsoft.Agents.A365.Observability.Hosting.Caching` | S2S implementation of `IExporterTokenCache` | -| `AgenticTokenStruct` | `Microsoft.Agents.A365.Observability.Hosting.Caching` | Wraps `TurnContext` + `UserAuthorization` + `AuthHandlerName` for token resolution. Uses **constructor** syntax: `new AgenticTokenStruct(userAuthorization: ..., turnContext: ..., authHandlerName: "AGENTIC")` | -| `Agent365ExporterOptions` | `Microsoft.Agents.A365.Observability.Runtime.Tracing.Exporters` | Exporter config (`TokenResolver`, `MaxQueueSize`, `ScheduledDelayMilliseconds`, etc.) | +| `AgenticTokenStruct` | `Microsoft.Agents.A365.Observability.Hosting.Caching` | Legacy: wraps `TurnContext` + `UserAuthorization` + `AuthHandlerName` so the exporter can exchange a delegated token. Do not use it for telemetry: the S2S route rejects delegated tokens | +| `Agent365ExporterOptions` | `Microsoft.Agents.A365.Observability.Runtime.Tracing.Exporters` | Exporter config (`TokenResolver`, `UseS2SEndpoint`, `MaxQueueSize`, `ScheduledDelayMilliseconds`, etc.) | | `Agent365ExporterType` | `Microsoft.Agents.A365.Observability.Runtime.Tracing.Exporters` | Enum for `AddA365Tracing()` exporter type param | -| `AddAgenticTracingExporter()` | `Microsoft.Agents.A365.Observability.Hosting` | DI extension for OBO token caching (`IExporterTokenCache`) — obo / agentic-user | +| `AddAgenticTracingExporter()` | `Microsoft.Agents.A365.Observability.Hosting` | Legacy DI extension for delegated OBO token caching (`IExporterTokenCache`). Do not use it for telemetry | | `AddServiceTracingExporter()` | `Microsoft.Agents.A365.Observability.Hosting` | Legacy/manual DI extension for S2S token cache (`IExporterTokenCache`) when not using the unified distro | +| `AgentAppTokenResolver` | Scaffold (`Observability/`) | `obo` / `agentic-user`: app-only token per agent identity from the default (blueprint) connection; wire as `o.Agent365.TokenResolver` | | `Agent365ObservabilityContext` | Scaffold (`Observability/`) | Singleton wrapping `AgentDetails` for S2S agents — inject instead of per-turn `RegisterObservability` | | `ObservabilityTokenService` | Scaffold (`Observability/`) | `BackgroundService` — acquires the export token via the FMI 3-hop chain (`.WithFmiPath()` + agent assertion); refreshes every 50 min | | `AddAgent365Observability()` | Scaffold (`Observability/`) | Registers `ServiceTokenCache`, `ObservabilityTokenService` (conditional), and `Agent365ObservabilityContext` | -| `UseMicrosoftOpenTelemetry()` | `Microsoft.OpenTelemetry` | Configures OTel pipeline with A365 exporter (preferred for S2S) | +| `UseMicrosoftOpenTelemetry()` | `Microsoft.OpenTelemetry` | Configures OTel pipeline with A365 exporter (all paths) | | `ExportTarget` | `Microsoft.OpenTelemetry` | Enum: `Agent365`, `Console`, `AzureMonitor` | | `AddA365Tracing()` | `Microsoft.Agents.A365.Observability.Runtime` | Registers OTel TracerProvider with A365 exporter | | `BaggageTurnMiddleware` | `Microsoft.Agents.A365.Observability.Hosting.Middleware` | Adapter middleware — auto-populates baggage from every `ITurnContext` | @@ -1295,7 +1430,7 @@ warn: Agent365ExporterCore: No token obtained for agent {agentId} tenant {tenant | Property | Description | Default | |----------|-------------|---------| -| `UseS2SEndpoint` | Use service-to-service endpoint path | `false` | +| `UseS2SEndpoint` | Use the service-to-service (S2S) route. Set `true` for every agent (`o.Agent365.UseS2SEndpoint` on 1.0.3+) | `false` | | `MaxQueueSize` | Max queue size for batch processor | `2048` | | `ScheduledDelayMilliseconds` | Delay between export batches | `5000` | | `ExporterTimeoutMilliseconds` | Timeout for export operation | `30000` | @@ -1337,22 +1472,22 @@ The `a365 setup` command (as of April 2026) automatically writes the following t | No traces in console | OTel not wired | Call `builder.UseMicrosoftOpenTelemetry()` (or `builder.AddA365Tracing()` for OBO path) | | No logs in Defender | Missing `Logging.LogLevel` config | Add `Microsoft.Agents.A365.Observability: Debug` to appsettings.json | | `AgenticAppId` is null | Missing `AGENTIC_APP_ID` env var | Set it in `.env` or App Service config | -| Token resolver returns null | `AddAgenticTracingExporter()` not called | Add to `Program.cs` DI | -| 401 from A365 exporter | OAuth consent not granted | Run `a365 setup permissions observability`; also check if upgrading past `0.3-beta` (requires new `Agent365.Observability.OtelWrite` permission) | +| Token resolver returns null | `AgentAppTokenResolver` not registered or not wired, or token acquisition failed (see the `Could not acquire an app-only A365 observability token` warning) | Register `builder.Services.AddSingleton()`, wire `o.Agent365.TokenResolver`, and resolve it after `Build()`. Check that the default connection holds the blueprint credential | +| 401 from A365 exporter | The exporter is on the delegated route (`UseS2SEndpoint` not set) or received a delegated token | Set `o.Agent365.UseS2SEndpoint = true` and use `AgentAppTokenResolver` / `ObservabilityTokenService`. The token must have no `scp` claim, and its `azp`/`appid` must equal the exporting agent ID. Do not run a delegated consent flow for telemetry | | Build error on `BaggageBuilder` | Wrong namespace | Use `Microsoft.Agents.A365.Observability.Runtime.Common` | -| Build error on `AgenticTokenStruct` | Object initializer syntax used | Use constructor: `new AgenticTokenStruct(userAuthorization: ..., turnContext: ..., authHandlerName: "AGENTIC")` | -| Build error on `IExporterTokenCache` | Wrong namespace | Use `Microsoft.Agents.A365.Observability.Hosting.Caching` | -| Build error on `AddAgenticTracingExporter` | Wrong namespace | Use `Microsoft.Agents.A365.Observability.Hosting` | +| Build error on `AgenticTokenStruct` / `IExporterTokenCache` in agent code | Legacy delegated-route token registration left in the handler | Remove the `IExporterTokenCache` injection and the per-turn `RegisterObservability(...)` call — telemetry uses `AgentAppTokenResolver` | +| Build error on `AddAgenticTracingExporter` | Wrong namespace (and it is the legacy delegated-route wiring) | The namespace is `Microsoft.Agents.A365.Observability.Hosting`, but prefer removing the call and migrating to `AgentAppTokenResolver` on the S2S route — see "Legacy two-package wiring" above | | Build error on `AddA365Tracing` | Wrong namespace | Use `Microsoft.Agents.A365.Observability.Runtime` | | Spans dropped silently | Missing tenant/agent ID in baggage | Ensure `BaggageBuilder` is set up before creating spans, or register `BaggageTurnMiddleware` | | S2S: token service skipped at startup | Placeholder or missing `Agent365Observability` credentials | Run `a365 setup all` or populate `TenantId`, `AgentId`, `ClientId`, and `ClientSecret` (when `UseManagedIdentity` is `false`) | -| S2S: 401 on export | Token acquired for wrong scope or app | Verify FMI Hop 3 scope is `api://9b975845-388f-4429-889e-eab1ef63949c/.default`. For agents provisioned before CLI 1.1, verify Agent Identity SP has `Agent365.Observability.OtelWrite` app role via Entra portal | +| S2S: 401 on export | Token acquired for wrong scope or app | Verify FMI Hop 3 scope is `api://9b975845-388f-4429-889e-eab1ef63949c/.default` and that the token is app-only and minted for the agent identity | | S2S: FMI Hop 1+2 fails | Blueprint credentials wrong or `.WithFmiPath(agentId)` target incorrect | Check `ClientId` (Blueprint app ID) and `ClientSecret` in appsettings; verify `AgentId` matches the Agent Identity app ID | -| S2S: FMI Hop 3 → 401 on export | Wrong scope or missing role | FMI Hop 3 scope is `api://9b975845-388f-4429-889e-eab1ef63949c/.default`; Agent Identity SP needs `OtelWrite` role assigned via Graph API | +| 403 `insufficient_scope` on export | The agent instance is not registered and has no `OtelWrite` application role. The S2S route authorizes registered blueprint instances without `OtelWrite` (subject to service policy) | Blueprint agents: register the instance with `a365 setup all --agent-registration-only` (idempotent). AI Teammates: complete the `OtelWrite` application-role step that `a365 setup all --aiteammate` prints. Either way, a Global Administrator can grant the `Agent365.Observability.OtelWrite` application role on the Blueprint, which the S2S route accepts | | S2S: MSI fails locally | No Managed Identity available in dev | Set `UseManagedIdentity: false` in appsettings.Development.json, ensure `ClientSecret` is populated | | S2S: `UseMicrosoftOpenTelemetry` not found | Unified distro not installed | Run `dotnet add package Microsoft.OpenTelemetry` (GA 1.0.2+) | | Duplicate spans for SemanticKernel / OpenAI / AgentFramework | Both unified distro auto-instrumentation toggles and the legacy `Microsoft.Agents.A365.Observability.Extensions.*` packages are wired | Uninstall the `Extensions.*` packages — the distro's `o.Instrumentation.Enable*Instrumentation` toggles supersede them | -| S2S: HTTP 401 on span export (correct token) | `UseS2SEndpoint` not set — exporter posts to `/observability/` instead of `/observabilityService/` | Set `o.Agent365.Exporter.UseS2SEndpoint = true` in `UseMicrosoftOpenTelemetry` options | +| HTTP 401 on span export (correct token) | `UseS2SEndpoint` not set — exporter posts to `/observability/` instead of `/observabilityService/` | Set `o.Agent365.UseS2SEndpoint = true` in `UseMicrosoftOpenTelemetry` options (`o.Agent365.Exporter.UseS2SEndpoint` on 1.0.2 and earlier) — required in every auth mode | +| CS1061 `'Agent365Options' does not contain a definition for 'Exporter'` | `Microsoft.OpenTelemetry` 1.0.3+ flattened the exporter options | Use `o.Agent365.UseS2SEndpoint` / `o.Agent365.TokenResolver` instead of `o.Agent365.Exporter.*` | | S2S: CS7036 on `InferenceCallDetails` — missing `providerName` | `providerName` is required (not optional) | Use: `new InferenceCallDetails(operationName: ..., model: ..., providerName: "Azure OpenAI")` | | S2S: CS1503 on `ExecuteToolScope.RecordResponse` | Method takes `string`, not `Response` | Use: `toolScope.RecordResponse(resultString)` | | S2S: `InvokeAgentScopeDetails` constructor error | No parameterless constructor exists | Pass at least `endpoint`: `new InvokeAgentScopeDetails(endpoint: new Uri("..."))` | diff --git a/plugins/agent365/skills/instrument-observability/references/nodejs-observability.md b/plugins/agent365/skills/instrument-observability/references/nodejs-observability.md index 44aa0a4..6f58bad 100644 --- a/plugins/agent365/skills/instrument-observability/references/nodejs-observability.md +++ b/plugins/agent365/skills/instrument-observability/references/nodejs-observability.md @@ -10,6 +10,8 @@ into a Node.js agent. Aligned with `@microsoft/opentelemetry` **GA 1.0.x** (upda > for the authoritative migration guide. > > **Sample-lag note (2026-05):** `Agent365-Samples/nodejs/langchain/sample-agent` has migrated to `@microsoft/opentelemetry` and matches the patterns in this reference. `Agent365-Samples/nodejs/openai/sample-agent` still imports from the legacy `@microsoft/agents-a365-observability*` packages as of this writing — the skill direction (unified `@microsoft/opentelemetry`) is forward-looking. If a user's project already has the legacy imports from following the OpenAI sample literally, the skill should migrate them to `@microsoft/opentelemetry` during the wiring step rather than co-existing. +> +> **Telemetry always uses the S2S route.** Every agent (`agentic-user` AI Teammates, `obo`, and `s2s`) exports with `useS2SEndpoint: true` and an **app-only** token for the exporting agent identity. `authMode` only selects workload (MCP / Graph) auth and how the telemetry token is sourced. The S2S route rejects delegated (`scp`) tokens, so never pass an OBO or Agentic User token to the exporter. Registered blueprint agent instances need no `Agent365.Observability.OtelWrite` permission or admin consent (subject to service policy), and `a365 setup all` no longer requests them for blueprint agents. AI Teammate setup (`a365 setup all --aiteammate`) still offers the `OtelWrite` application role; complete the app-role action item it prints, because the S2S route accepts that role. --- @@ -31,6 +33,8 @@ npm install @microsoft/opentelemetry @opentelemetry/resources npm install @azure/msal-node @azure/identity ``` +> **`obo` / `agentic-user` need no extra packages.** The app-only token resolver reuses the agent's existing `@microsoft/agents-hosting` connection and the built-in `fetch`. + > **No version pin needed.** `@microsoft/opentelemetry` is GA — install latest. `@opentelemetry/resources` is pulled in transitively by `@microsoft/opentelemetry`, but is listed here explicitly because the entry-point examples below import `resourceFromAttributes` from it (TS/module resolution can fail if it's not declared as a direct dep). Minimum Node.js: **20.6.0** (required for ESM `--import` flow). TypeScript: **5.x** recommended. @@ -39,17 +43,17 @@ Minimum Node.js: **20.6.0** (required for ESM `--import` flow). TypeScript: **5. ## Auth Mode Mapping -The agent's `authMode` (read from `.a365-workspace-detection.local.json`) determines which path to wire. The code shape is **identical** for `obo` and `agentic-user` — only the identity the token exchange returns differs. `s2s` uses a completely separate token-service scaffold. +Telemetry export is the same in every mode: the S2S route (`useS2SEndpoint: true`) with an app-only token for the exporting agent identity. The agent's `authMode` (read from `.a365-workspace-detection.local.json`) only decides how that token is sourced. The code shape is **identical** for `obo` and `agentic-user`; `s2s` uses a separate background token service. -| `authMode` | Used by | Token mechanism | Identity in traces | Wiring | Per-turn token refresh | +| `authMode` | Used by | Workload auth (MCP / Graph) | Telemetry token source | Wiring | Per-turn token refresh | |---|---|---|---|---|---| -| `agentic-user` | AI Teammate (always) | OBO exchange | Agent's own M365 identity (Agentic User — UPN, mailbox) | `AgenticTokenCacheInstance` | ✅ Call `refreshObservabilityToken` (camelCase in GA 1.0+) | -| `obo` | Non-AI Teammate | OBO exchange | Whatever the configured auth handler resolves — typically the signed-in user, but can also be the agent's own identity | `AgenticTokenCacheInstance` | ✅ Call `refreshObservabilityToken` | -| `s2s` | Non-AI Teammate | Service principal client credentials (no token exchange) | Agent Identity SP — no user context | Custom `tokenResolver` + background FMI token service | ❌ Do NOT call `refreshObservabilityToken` | +| `agentic-user` | AI Teammate (always) | OBO exchange as the Agentic User | App-only token for the turn's agent identity, from the hosting connection's blueprint credential | `observability/app-token-resolver.ts` | ❌ None — acquired on demand and cached | +| `obo` | Non-AI Teammate | OBO exchange (typically the signed-in user) | Same as `agentic-user` | `observability/app-token-resolver.ts` | ❌ None | +| `s2s` | Non-AI Teammate | None (no user token) | App-only token for the configured agent identity, from a background FMI token service | `observability/token-cache.ts` + `observability/observability-token-service.ts` | ❌ None | > AI Teammate is **always** `agentic-user` — no question is asked. Non-AI Teammate agents are asked at setup whether they want `obo` or `s2s`. > -> Note: "OBO" describes the **token exchange mechanism**, not who the agent acts as. Both `obo` and `agentic-user` use OBO under the hood — they differ only in which identity the configured Azure AD auth handler returns. `s2s` does not use OBO at all. +> Note: "OBO" describes the **workload token exchange** (MCP / Graph), not who the agent acts as, and never the telemetry token. The S2S route rejects delegated (`scp`) tokens, so the exporter must never receive an OBO or Agentic User token. Do not call `AgenticTokenCacheInstance.refreshObservabilityToken(...)`: it performs a delegated exchange that needs admin consent and produces a token the S2S route rejects. --- @@ -58,7 +62,7 @@ The agent's `authMode` (read from `.a365-workspace-detection.local.json`) determ Initialize the unified distro **before** importing the rest of your app so OpenAI Agents and LangChain auto-instrumentation can patch their target libraries. -### OBO / agentic-user (same code; identity decided by the auth handler) +### OBO / agentic-user (same code; app-only telemetry token from the hosting connection) ```typescript // A365 Observability — best-effort instrumentation (verify against official sample) @@ -66,11 +70,16 @@ and LangChain auto-instrumentation can patch their target libraries. import { configDotenv } from 'dotenv'; configDotenv(); -import { - useMicrosoftOpenTelemetry, - AgenticTokenCacheInstance, -} from '@microsoft/opentelemetry'; +import { useMicrosoftOpenTelemetry } from '@microsoft/opentelemetry'; import { resourceFromAttributes } from '@opentelemetry/resources'; +import { createAppTokenResolver, type ConnectionProvider } from './observability/app-token-resolver'; + +// The adapter is created after observability init, so the resolver reads its connection lazily. +let getObsConnection: ConnectionProvider | undefined; +const appTokenResolver = createAppTokenResolver(() => { + if (!getObsConnection) throw new Error('Adapter is not initialised yet.'); + return getObsConnection(); +}); useMicrosoftOpenTelemetry({ // Console exporter floods prod logs — gate on NODE_ENV and Azure App Service's @@ -89,14 +98,27 @@ useMicrosoftOpenTelemetry({ // the code flag — but at least one of the two must be true or no spans reach MAC.) enabled: true, enableObservabilityExporter: true, - tokenResolver: (agentId, tenantId) => - AgenticTokenCacheInstance.getObservabilityToken(agentId, tenantId) ?? '', + // Every auth mode exports over the S2S route with an app-only token for the exporting + // agent identity. The S2S route rejects delegated (OBO / Agentic User) tokens. + useS2SEndpoint: true, + tokenResolver: appTokenResolver, }, // Framework-specific opt-ins. Include for agents that use LangChain. instrumentationOptions: { langchain: {} }, }); + +// ... import app modules AFTER observability init, then hand the adapter's connection to +// the resolver wherever the adapter is created, for example: +// const adapter = agentApplication.adapter as CloudAdapter; +// getObsConnection = () => adapter.connectionManager.getDefaultConnection(); ``` +> **Why a lazy connection getter:** `useMicrosoftOpenTelemetry()` must run before the agent +> modules are imported, but the `CloudAdapter` (and its connection manager) is created by those +> modules. The resolver runs only at export time, after the adapter exists. The default +> connection is the blueprint credential `a365 setup all` writes to `.env`, so no extra +> credentials or settings are needed. + > **Exporter activation (GA 1.0+):** TWO toggles are involved and BOTH must resolve to true: > - `a365.enabled: true` (code) → registers the `A365SpanProcessor` for baggage/attribute enrichment. **Without export.** > - `a365.enableObservabilityExporter: true` (code) **OR** `ENABLE_A365_OBSERVABILITY_EXPORTER=true` (env, auto-stamped by `a365 setup all`) → registers the `Agent365Exporter`. **This is what actually sends spans to MAC.** @@ -222,6 +244,121 @@ process.on('SIGINT', () => shutdown('SIGINT')); --- +## App-only Token Resolver Scaffold (`obo` / `agentic-user`) + +Create `observability/app-token-resolver.ts`. It turns the agent's **existing** hosting +connection (the blueprint credential) into an app-only Observability API token for whichever +agent identity is exporting. Tokens are cached per agent instance, so multi-instance AI +Teammates need no per-instance configuration: + +``` +Blueprint credential (hosting connection) + → Step 1: FMI assertion for the agent identity getAgenticApplicationToken(tenantId, agentId) + → Step 2: agent identity client_credentials scope=api://9b975845-388f-4429-889e-eab1ef63949c/.default +``` + +The exporter passes `(agentId, tenantId)` from span baggage (`recipient.agenticAppId`), so no +agent ID is configured here. + +```typescript +// observability/app-token-resolver.ts +// A365 Observability — best-effort instrumentation (verify against official sample) +// App-only token for A365 observability export over the S2S route. Workload auth +// (OBO / agentic user for MCP and Graph) is unchanged; only the telemetry credential differs. +// Step 1: the agent's hosting connection (the blueprint credential) issues an FMI assertion +// for the exporting agent identity: getAgenticApplicationToken(tenantId, agentId). +// Step 2: the agent identity exchanges that assertion (client_credentials) for an app-only +// Observability API token. The S2S route rejects delegated (scp) tokens. +import type { AuthProvider } from '@microsoft/agents-hosting'; + +const OBS_SCOPE = 'api://9b975845-388f-4429-889e-eab1ef63949c/.default'; +const REFRESH_SKEW_MS = 5 * 60_000; + +/** Returns the agent's hosting connection; called lazily because the adapter is created after observability init. */ +export type ConnectionProvider = () => AuthProvider; + +type CachedToken = { token: string; expiresAt: number }; + +function decodeClaims(token: string): Record { + const payload = token.split('.')[1]; + if (!payload) throw new Error('Observability token is not a JWT.'); + return JSON.parse(Buffer.from(payload, 'base64url').toString('utf8')) as Record; +} + +export function createAppTokenResolver(getProvider: ConnectionProvider) { + const cache = new Map(); + const pending = new Map>(); + + const acquire = async (agentId: string, tenantId: string): Promise => { + const assertion = await getProvider().getAgenticApplicationToken(tenantId, agentId); + const response = await fetch(`https://login.microsoftonline.com/${tenantId}/oauth2/v2.0/token`, { + method: 'POST', + headers: { 'Content-Type': 'application/x-www-form-urlencoded' }, + body: new URLSearchParams({ + grant_type: 'client_credentials', + client_id: agentId, + client_assertion_type: 'urn:ietf:params:oauth:client-assertion-type:jwt-bearer', + client_assertion: assertion, + scope: OBS_SCOPE, + }).toString(), + signal: AbortSignal.timeout(30_000), + }); + // Never log the response body: it can echo request details. + if (!response.ok) throw new Error(`Observability token request failed (HTTP ${response.status}).`); + const body = (await response.json()) as { access_token?: string; expires_in?: number }; + const token = body.access_token; + if (!token) throw new Error('Observability token response had no access_token.'); + + const claims = decodeClaims(token); + if ('scp' in claims) throw new Error('Observability token is delegated (scp claim); the S2S route rejects it.'); + const client = String(claims['azp'] ?? claims['appid'] ?? '').toLowerCase(); + if (client !== agentId.toLowerCase() || String(claims['tid'] ?? '').toLowerCase() !== tenantId.toLowerCase()) { + throw new Error('Observability token client or tenant does not match the exporting agent.'); + } + const exp = claims['exp']; + const expiresAt = typeof exp === 'number' ? exp * 1000 : Date.now() + (body.expires_in ?? 0) * 1000; + cache.set(`${tenantId}:${agentId}`.toLowerCase(), { token, expiresAt }); + return token; + }; + + // Wired as a365.tokenResolver. The exporter calls it for every export batch, so serve from + // the cache and share one in-flight request per agent identity. + return async (agentId: string, tenantId: string): Promise => { + if (!agentId || !tenantId) return ''; + const key = `${tenantId}:${agentId}`.toLowerCase(); + const cached = cache.get(key); + if (cached && Date.now() < cached.expiresAt - REFRESH_SKEW_MS) return cached.token; + let inFlight = pending.get(key); + if (!inFlight) { + inFlight = acquire(agentId, tenantId).finally(() => pending.delete(key)); + pending.set(key, inFlight); + } + try { + return await inFlight; + } catch (err) { + console.warn('[A365 Observability] App-only token acquisition failed:', (err as Error).message); + return ''; + } + }; +} +``` + +> **Turns without an agent identity** (for example a Playground session without agentic auth) +> carry no `gen_ai.agent.id`, so the resolver returns `''` and those spans are not exported. For +> a non-AI-Teammate `obo` agent you can attribute such turns to the provisioned agent identity +> that `a365 setup all` writes to `agent365Observability__agentId` by calling +> `builder.agentId(process.env.agent365Observability__agentId)` on the turn's `BaggageBuilder` +> when `!turnContext.activity.isAgenticRequest()`. Skip this if the value equals +> `agent365Observability__agentBlueprintId`: for AI Teammates the CLI writes the blueprint ID +> there, and a blueprint cannot export over the S2S route. + +> **Authorization:** registered blueprint agent instances are authorized on the S2S route without +> the `Agent365.Observability.OtelWrite` permission or admin consent (subject to service policy). +> For AI Teammates, complete the `OtelWrite` application-role step that `a365 setup all --aiteammate` +> prints. Do not add a delegated observability scope to the auth handler for telemetry. + +--- + ## S2S Token Service Scaffold #### Step 1 — `observability/token-cache.ts` @@ -381,9 +518,10 @@ ENABLE_A365_OBSERVABILITY_EXPORTER=true # `useS2SEndpoint: true` is set in code via the a365 options. ``` -Message handler baggage setup is **identical** to OBO — only the token resolver and credential -source differ. Do **not** call `AgenticTokenCacheInstance.getObservabilityToken` for S2S; the -token comes from your custom `tokenResolver` wired in `useMicrosoftOpenTelemetry`. +Message handler baggage setup is **identical** to OBO; only the token resolver and credential +source differ. Do **not** call `AgenticTokenCacheInstance` in any mode: it serves delegated +tokens, which the S2S route rejects. The token comes from your custom `tokenResolver` wired in +`useMicrosoftOpenTelemetry`. --- @@ -412,7 +550,7 @@ configureA365Hosting(adapter, { | Stack | Pattern | Where the scope lives | |---|---|---| -| **LangChain** | Canonical wrapping: `preloadObservabilityToken` → outer `baggageScope.run` → `InvokeAgentScope.start` + `InferenceScope.start` INSIDE | `src/agent.ts` message handler | +| **LangChain** | Canonical wrapping: outer `baggageScope.run` → `InvokeAgentScope.start` + `InferenceScope.start` INSIDE | `src/agent.ts` message handler | | **OpenAI Agents SDK** | Same as LangChain — canonical wrapping | `src/agent.ts` message handler | | **Claude SDK** | **`InferenceScope.start` only** — no outer baggageScope, no `InvokeAgentScope`. The Claude sample wraps each LLM call individually inside `src/client.ts`; the handler does not open scopes. | `src/client.ts` query wrapper | @@ -422,31 +560,25 @@ Use the matching pattern for the user's stack. Mixing them produces either silen > > The `as any` cast on `turnContext` is needed because the GA `TurnContextLike` shape declares `activity.getAgenticTenantId()` as `string` while `@microsoft/agents-hosting`'s `TurnContext` returns `string | undefined`. -### OBO and agentic-user — refresh exporter token per turn +### OBO and agentic-user — no per-turn token refresh -The handler shape is identical for both `obo` and `agentic-user`. The Azure AD auth handler -configured in your `AgentApplication` decides which identity the token exchange returns: +The handler shape is identical for both `obo` and `agentic-user`, and the handler does NOT +touch telemetry tokens. The app-only resolver wired in the entry point acquires a token for the +exporting agent identity when the spans are exported. The auth handler configured in your +`AgentApplication` is used only for workload calls (MCP / Graph): -- **`agentic-user`** (AI Teammate) — `agentApplication.authorization` exchanges to the - agent's own Azure AD user identity (Agentic User). Traces attribute to the agent. -- **`obo`** (non-AI Teammate) — `agentApplication.authorization` exchanges to whatever - the configured Azure AD auth handler resolves. Typically this is the signed-in user - (traces attribute to that user), but it can also be the agent's own identity if the - handler is configured that way. +- **`agentic-user`** (AI Teammate) — workload calls run as the agent's own Agentic User. + Telemetry is attributed to the agent instance (`recipient.agenticAppId`). +- **`obo`** (non-AI Teammate) — workload calls run as whatever the configured auth handler + resolves, typically the signed-in user. The caller still appears in `CallerDetails`. ```typescript // A365 Observability — best-effort instrumentation (verify against official sample) -// A365 auth mode: agentic-user (or: obo) -import { AgenticTokenCacheInstance, BaggageBuilder, BaggageBuilderUtils } from '@microsoft/opentelemetry'; +// A365 auth mode: agentic-user (or: obo) — telemetry uses an app-only token on the S2S route +import { BaggageBuilder, BaggageBuilderUtils } from '@microsoft/opentelemetry'; async function handleMessage(turnContext: TurnContext, state: ApplicationTurnState) { - // STEP 1 — refresh the exporter token BEFORE entering the baggage scope. Skipping this on - // a cold turn means the first export attempt sees an empty token, retries until timeout, - // and the span is not exported. The token cache is in-memory and lives for the - // process lifetime, so this is a no-op on warm turns. - await preloadObservabilityToken(turnContext); - - // STEP 2 — build outer baggage scope from TurnContext. This populates microsoft.tenant.id + // Build the outer baggage scope from TurnContext. This populates microsoft.tenant.id // and gen_ai.agent.id baggage on every span created inside the run() callback. Without // this wrapping, the exporter filters spans as "Partitioned into 0 identity groups // (N spans skipped)" and they are not exported — the #1 first-run failure mode. @@ -456,30 +588,22 @@ async function handleMessage(turnContext: TurnContext, state: ApplicationTurnSta .build(); await baggageScope.run(async () => { - // STEP 3 — your InvokeAgentScope + InferenceScope + agent invocation go here. + // Your InvokeAgentScope + InferenceScope + agent invocation go here. // Nested scopes inherit the outer baggage automatically. // ... LangChain / OpenAI / agent invocation ... }); } - -async function preloadObservabilityToken(turnContext: TurnContext): Promise { - const agentId = turnContext.activity?.recipient?.agenticAppId ?? ''; - const tenantId = turnContext.activity?.recipient?.tenantId ?? ''; - - // The cache instance handles the OBO token exchange internally. - // Identity returned = whatever the configured auth handler resolves to - // (Agentic User for agentic-user mode; signed-in user for obo mode). - // `as any` casts are required: the GA `TurnContextLike` / `AuthorizationLike` - // interfaces are stricter than the @microsoft/agents-hosting types they were modeled on. - await AgenticTokenCacheInstance.refreshObservabilityToken( - agentId, - tenantId, - turnContext as any, - agentApplication.authorization as any, - ); -} ``` +> **Migrating an existing agent:** remove any `preloadObservabilityToken` helper and +> `AgenticTokenCacheInstance.refreshObservabilityToken(agentId, tenantId, turnContext, authorization)` +> call. That delegated exchange needs admin consent, and the S2S route rejects its token. Also +> replace the old `AgenticTokenCacheInstance.getObservabilityToken` resolver with the app-only +> resolver and set `useS2SEndpoint: true`. If durable delivery is enabled (the 1.4.x default), records spooled +> while the agent used the delegated route replay to that route. Set +> `a365.durableDelivery: { enabled: false }` while migrating, as the Agent 365 samples do, or +> clear the spool directory. + ### S2S — no per-turn refresh For `s2s`, the background token service started in the entry point (see [S2S Token Service Scaffold](#s2s-token-service-scaffold)) populates the in-memory cache every 50 minutes. The custom `tokenResolver` wired in `useMicrosoftOpenTelemetry()` reads from that cache on each export. The handler does NOT touch tokens. @@ -491,7 +615,7 @@ For `s2s`, the background token service started in the entry point (see [S2S Tok async function handleMessage(turnContext: TurnContext, state: ApplicationTurnState) { // BaggageMiddleware (from configureA365Hosting) already populated baggage from TurnContext. // No per-turn token refresh — background token service handles auth. - // Do NOT call AgenticTokenCacheInstance.refreshObservabilityToken for S2S. + // Do NOT call AgenticTokenCacheInstance.refreshObservabilityToken in any auth mode. // ... your agent invocation goes here ... } @@ -864,12 +988,13 @@ SERVICE_NAME=my-agent # A365_OBSERVABILITY_LOG_LEVEL=info|warn|error # ── Runtime agent identity vs blueprint id ────────────────────────────────── -# The `a365 setup all` CLI writes `agent365Observability__agentId=` into -# .env, but that value is the BLUEPRINT id, NOT the runtime AUID. The exporter -# resolves the runtime AUID from `turnContext.activity.recipient.agenticAppId` -# on each turn — that's what shows up in MAC Advanced Hunting (`CloudAppEvents`, -# `AgentId` column). If you write a KQL filter using the blueprint id, you'll get -# empty results. Filter by AUID, not blueprint id. +# For AI Teammates, `a365 setup all` writes `agent365Observability__agentId=` +# into .env — that is the BLUEPRINT id, NOT the runtime AUID. (For other blueprint agents it +# writes the provisioned agent identity.) The exporter resolves the runtime AUID from +# `turnContext.activity.recipient.agenticAppId` on each turn — that's what shows up in MAC +# Advanced Hunting (`CloudAppEvents`, `AgentId` column) and what the app-only token is minted +# for. If you write a KQL filter using the blueprint id, you'll get empty results. Filter by +# AUID, not blueprint id. # Sponsor / CallerDetails for MAC portal trace visibility (S2S agents — no signed-in user). agent365Observability__sponsorUserId=<> @@ -888,10 +1013,11 @@ agent365Observability__sponsorUserEmail=<> > **Removed:** `AGENT365_USE_S2S_ENDPOINT` env var (use `useS2SEndpoint: true` in code instead). > -> **Sample-only switch:** `Use_Custom_Resolver` is a *sample-level* toggle in the -> langchain/openai/claude Agent365 samples that demonstrates swapping between a -> custom in-process token cache and the built-in `AgenticTokenCacheInstance`. It is -> not an SDK contract — you always pass *some* `tokenResolver` to `useMicrosoftOpenTelemetry`. +> **Sample-only switch:** older langchain/openai/claude Agent365 samples used a +> `Use_Custom_Resolver` toggle to swap between a custom in-process cache of delegated tokens +> and the built-in `AgenticTokenCacheInstance`. Both served delegated tokens, which the S2S +> route rejects. It is not an SDK contract; always pass an app-only `tokenResolver` to +> `useMicrosoftOpenTelemetry`. --- @@ -933,8 +1059,8 @@ Key console messages: | `BaggageBuilder` | Fluent builder for tenant/agent/correlation baggage (rarely needed manually) | | `BaggageMiddleware` | Adapter middleware — auto-populates baggage (registered by `configureA365Hosting`) | | `ObservabilityHostingManager` | Lower-level alternative to `configureA365Hosting` | -| `AgenticTokenCacheInstance` | Singleton: `getObservabilityToken`, `refreshObservabilityToken` | -| `AgenticTokenCache` | Class form (advanced; usually the singleton above is enough) | +| `AgenticTokenCacheInstance` | Legacy delegated-token cache (`getObservabilityToken`, `refreshObservabilityToken`). Do not use it for telemetry: the S2S route rejects delegated tokens | +| `AgenticTokenCache` | Class form of the legacy delegated-token cache | | `Agent365Exporter` / `A365SpanProcessor` | Re-exported for advanced custom pipeline scenarios | | `InvokeAgentScope.start(request, scopeDetails, agentDetails, callerDetails)` | Agent invocation scope | | `ExecuteToolScope.start(request, toolDetails, agentDetails, userDetails)` | Tool execution scope | @@ -958,15 +1084,15 @@ Key console messages: | `Partitioned into 0 identity groups (N spans skipped)` in exporter logs | **Expected for spans outside an active baggage scope** — framework / middleware spans created during startup, devtunnel health pings, etc. carry no `microsoft.tenant.id` / `gen_ai.agent.id` baggage and are filtered. NOT an error. | Only worry if the count is non-zero on actual turn spans. The log line you actually want to see for turn spans is `export-group succeeded ... 1 chunk(s) exported successfully` — that confirms a real identity-group reached the backend. | | Duplicate spans for OpenAI/LangChain calls | Manual `.enable()` / `.instrument()` call after migration | Remove manual instrumentor calls; auto-instrumentation is ON by default | | Spans missing baggage | `configureA365Hosting()` not called | Add `configureA365Hosting(adapter, { enableBaggage: true })` once at startup | -| Token resolver always returns `''` | `refreshObservabilityToken` not called per turn (OBO) | Call `AgenticTokenCacheInstance.refreshObservabilityToken(...)` at the start of each handler turn | +| Token resolver always returns `''` | App-only token acquisition failed (see the `[A365 Observability] App-only token acquisition failed` warning), or the turn has no agent identity (`recipient.agenticAppId` is empty) | Check that the hosting connection holds the blueprint credential and that the agent identity belongs to that blueprint. For non-agentic turns see the fallback note under the app-only resolver scaffold | | `Cannot find module '@microsoft/opentelemetry'` | Package not installed | `npm install @microsoft/opentelemetry` | -| 401 on export | Missing `Agent365.Observability.OtelWrite` permission | CLI 1.1+ grants this automatically via `a365 setup all`. For pre-1.1 agents, GA must grant it manually | +| 401 on export | The exporter is on the delegated route (`useS2SEndpoint` not set) or received a delegated token | Set `useS2SEndpoint: true` and use the app-only resolver; the token must have no `scp` claim and its `azp`/`appid` must equal the exporting agent ID | | Spans dropped silently | Missing tenant/agent ID | Ensure `configureA365Hosting({ enableBaggage: true })` is registered before creating spans | | Spans only when `ENABLE_A365_OBSERVABILITY_EXPORTER=true` env, but not via code | The env var is a secondary toggle | Set `enableObservabilityExporter: true` in `a365` options (code is preferred over env var) | | Pending spans lost on shutdown | `shutdownMicrosoftOpenTelemetry()` not called | Add SIGTERM/SIGINT handlers calling `await shutdownMicrosoftOpenTelemetry()` | | TypeScript error on `agentAuid` | Interface field is `agentAUID` (uppercase UID) | Change to `agentAUID: '...'` | | S2S: direct MSAL client credentials rejected | Direct MSAL client credentials not supported for the agent | Use the 3-hop FMI chain: Blueprint → FMI path → Agent Identity → Observability API token | -| S2S: 401 on `observabilityService/` | Token scope mismatch | Ensure Hop 3 scope is `api://9b975845-388f-4429-889e-eab1ef63949c/.default`. Ensure Agent Identity SP has OtelWrite role assigned | -| S2S: 403 on `observabilityService/` | Missing app role on Agent Identity SP | Assign `Agent365.Observability.OtelWrite` to the **Agent Identity** SP (not just the Blueprint) via Graph API | +| S2S: 401 on `observabilityService/` | Token scope or principal mismatch | Ensure the final hop's scope is `api://9b975845-388f-4429-889e-eab1ef63949c/.default` and the token is app-only (no `scp`), minted for the agent identity | +| S2S: 403 `insufficient_scope` on `observabilityService/` | The agent instance is not registered and has no `OtelWrite` application role. The S2S route authorizes registered blueprint instances without `OtelWrite` (subject to service policy) | Blueprint agents: register the instance with `a365 setup all --agent-registration-only` (idempotent). AI Teammates: complete the `OtelWrite` application-role step that `a365 setup all --aiteammate` prints. Either way, a Global Administrator can grant the `Agent365.Observability.OtelWrite` application role on the Blueprint, which the S2S route accepts | | S2S: MSI fails locally | No Managed Identity in dev | Set `AGENT365_USE_MANAGED_IDENTITY=false` and provide `AGENT365_CLIENT_SECRET` | | MSAL `AADSTS82008: fmipath parameter required` | `@azure/msal-node` v3.x does not serialize `fmiPath` to the token endpoint | Use the direct HTTP POST workaround in `acquireT1ViaClientSecret` (MSAL-side limitation; remove once MSAL ships native support) | diff --git a/plugins/agent365/skills/instrument-observability/references/python-observability.md b/plugins/agent365/skills/instrument-observability/references/python-observability.md index edf41c9..8489323 100644 --- a/plugins/agent365/skills/instrument-observability/references/python-observability.md +++ b/plugins/agent365/skills/instrument-observability/references/python-observability.md @@ -11,22 +11,24 @@ into a Python agent. Aligned with `microsoft-opentelemetry` **GA 1.2.x** (releas > See `MIGRATION_A365.md` in the distro repo for the authoritative migration guide. > > **Sample-lag note (2026-05):** `Agent365-Samples/python/agent-framework/sample-agent` is the verified canonical sample — it uses **manual per-turn `BaggageBuilder()` in the handler** (NOT `ObservabilityHostingManager` middleware) and imports `BaggageBuilder` + `get_observability_authentication_scope` from `microsoft.opentelemetry.a365.core.middleware.baggage_builder` and `microsoft_agents_a365.runtime.environment_utils` respectively. The OpenAI sample still uses the legacy `configure(...)` + `OpenAIAgentsTraceInstrumentor().instrument()` pattern — the skill direction (unified `use_microsoft_opentelemetry`) is forward-looking; migrate existing code to it. +> +> **Telemetry always uses the S2S route.** Every agent (`agentic-user` AI Teammates, `obo`, and `s2s`) exports with `a365_use_s2s_endpoint=True` and an **app-only** token for the exporting agent identity. `authMode` only selects workload (MCP / Graph) auth and how the telemetry token is sourced. The S2S route rejects delegated (`scp`) tokens, so never pass an OBO or Agentic User token (for example from `exchange_token(... scopes=get_observability_authentication_scope() ...)`) to the exporter. Registered blueprint agent instances need no `Agent365.Observability.OtelWrite` permission or admin consent (subject to service policy), and `a365 setup all` no longer requests them for blueprint agents. AI Teammate setup (`a365 setup all --aiteammate`) still offers the `OtelWrite` application role; complete the app-role action item it prints, because the S2S route accepts that role. --- ## Auth Mode Mapping -The agent's `authMode` (read from `.a365-workspace-detection.local.json`) determines which path to wire. The code shape is **identical** for `obo` and `agentic-user` — only the identity the token exchange returns differs. `s2s` uses a completely separate token-service scaffold. +Telemetry export is the same in every mode: the S2S route (`a365_use_s2s_endpoint=True`) with an app-only token for the exporting agent identity. The agent's `authMode` (read from `.a365-workspace-detection.local.json`) only decides how that token is sourced. The code shape is **identical** for `obo` and `agentic-user`; `s2s` uses a separate background token service. -| `authMode` | Used by | Token mechanism | Identity in traces | Wiring | Per-turn token refresh | +| `authMode` | Used by | Workload auth (MCP / Graph) | Telemetry token source | Wiring | Per-turn step | |---|---|---|---|---|---| -| `agentic-user` | AI Teammate (always) | OBO exchange | Agent's own M365 identity (Agentic User — UPN, mailbox) | `AgenticTokenCache` or custom resolver | ✅ Call `exchange_token()` and cache the result | -| `obo` | Non-AI Teammate | OBO exchange | Whatever the configured Azure AD auth handler resolves — typically the signed-in user, but can also be the agent's own identity | `AgenticTokenCache` or custom resolver | ✅ Call `exchange_token()` and cache the result | -| `s2s` | Non-AI Teammate | Service principal client credentials (no token exchange) | Agent Identity SP — no user context | Custom `a365_token_resolver` + background FMI token service | ❌ Do NOT call `exchange_token()` | +| `agentic-user` | AI Teammate (always) | OBO exchange as the Agentic User | App-only token for the turn's agent identity, from the hosting connection's blueprint credential | `observability/app_token_resolver.py` | ✅ `await OBS_TOKENS.prefetch(...)` (cached; the sync resolver only reads the cache) | +| `obo` | Non-AI Teammate | OBO exchange (typically the signed-in user) | Same as `agentic-user` | `observability/app_token_resolver.py` | ✅ Same as `agentic-user` | +| `s2s` | Non-AI Teammate | None (no user token) | App-only token for the configured agent identity, from a background FMI token service | `observability/token_cache.py` + `observability/observability_token_service.py` | ❌ None | > AI Teammate is **always** `agentic-user` — no question is asked. Non-AI Teammate agents are asked at setup whether they want `obo` or `s2s`. > -> Note: "OBO" describes the **token exchange mechanism**, not who the agent acts as. Both `obo` and `agentic-user` use OBO under the hood — they differ only in which identity the configured Azure AD auth handler returns. `s2s` does not use OBO at all. +> Note: "OBO" describes the **workload token exchange** (MCP / Graph), not who the agent acts as, and never the telemetry token. Do not call `exchange_token()` for the observability scope in any mode. --- @@ -36,7 +38,7 @@ The agent's `authMode` (read from `.a365-workspace-detection.local.json`) determ |---------|---------| | `microsoft-opentelemetry` (1.2.x GA) | Sole entry point. Re-exports `use_microsoft_opentelemetry`, baggage helpers (`populate`, `BaggageMiddleware`, `ObservabilityHostingManager`), `AgenticTokenCache`, all scope types (`InvokeAgentScope`, `InferenceScope`, `ExecuteToolScope`, `OutputScope`), and all contract types (`AgentDetails`, `CallerDetails`, `UserDetails`, `Request`, `Response`, `InvokeAgentScopeDetails`, `InferenceCallDetails`, `ToolCallDetails`, etc.). **Note:** Unlike Node.js, no `shutdown_microsoft_opentelemetry` helper is exported — see [Graceful Shutdown](#graceful-shutdown) for the OTel SDK-based pattern. | | `microsoft-opentelemetry[langchain]` | Optional extra — adds LangChain instrumentation deps (only if your agent uses LangChain) | -| `msal` (^1.34) | MSAL Python `ConfidentialClientApplication` for Hop 3 token acquisition (S2S only) | +| `msal` (^1.34) | MSAL Python `ConfidentialClientApplication` for the final token hop (all modes; already installed with `microsoft-agents-authentication-msal`) | | `azure-identity` (^1.20) | `ManagedIdentityCredential` for MSI-based token acquisition (S2S only) | | `httpx` (^0.27) | Direct HTTP POST for FMI Hop 1+2 (MSAL `fmi_path` workaround — see Known Issues) | @@ -53,6 +55,8 @@ pip3 install msal azure-identity httpx 2>/dev/null || pip install msal azure-ide ``` > **No `--pre` flag needed.** `microsoft-opentelemetry` is GA — install latest stable. +> +> **`obo` / `agentic-user` need no extra packages.** The app-only token resolver reuses the agent's `microsoft-agents-authentication-msal` connection and its `msal` dependency. Minimum Python: **3.10+** (for `str | None` typing in code samples; the package itself supports 3.9+). @@ -80,7 +84,7 @@ Initialize the unified distro **before** importing the rest of your app so OpenA LangChain, Semantic Kernel, and Agent Framework auto-instrumentation can patch their target libraries. -### OBO / agentic-user (same code; identity decided by the auth handler) +### OBO / agentic-user (same code; app-only telemetry token from the hosting connection) ```python # A365 Observability — best-effort instrumentation (verify against official sample) @@ -89,14 +93,19 @@ from dotenv import load_dotenv load_dotenv() from microsoft.opentelemetry import use_microsoft_opentelemetry -from microsoft.opentelemetry.a365.hosting.token_cache_helpers import AgenticTokenCache +from observability.app_token_resolver import AppTokenResolver -_token_cache = AgenticTokenCache() +# Caches app-only tokens per agent identity. The message handler prefetches the token for the +# turn's agent (see "Message Handler"); the exporter reads it synchronously. +OBS_TOKENS = AppTokenResolver() use_microsoft_opentelemetry( enable_a365=True, a365_enable_observability_exporter=True, # REQUIRED in 1.0+ to actually export spans - a365_token_resolver=_token_cache.get_observability_token, + # Every auth mode exports over the S2S route with an app-only token for the exporting + # agent identity. The S2S route rejects delegated (OBO / Agentic User) tokens. + a365_use_s2s_endpoint=True, + a365_token_resolver=OBS_TOKENS.resolve, ) ``` @@ -218,9 +227,126 @@ async def start_background_tasks(app: web.Application) -> None: ) ``` -> **`a365_use_s2s_endpoint=True` is required for S2S agents.** Without it, the exporter -> posts to `/observability/` (OBO endpoint) instead of `/observabilityService/` (S2S endpoint), -> causing 401 errors. +> **`a365_use_s2s_endpoint=True` is required for every agent.** Without it, the exporter +> posts to the delegated route (`/observability/`) instead of the S2S route +> (`/observabilityService/`), and app-only tokens are rejected there with 401 errors. + +--- + +## App-only Token Resolver Scaffold (`obo` / `agentic-user`) + +Create `observability/app_token_resolver.py`. It turns the agent's **existing** hosting +connection (the blueprint credential) into an app-only Observability API token for whichever +agent identity is exporting. Tokens are cached per agent instance, so multi-instance AI +Teammates need no per-instance configuration: + +``` +Blueprint credential (hosting connection) + → Step 1: FMI assertion for the agent identity get_agentic_application_token(tenant_id, agent_id) + → Step 2: agent identity client_credentials scope=api://9b975845-388f-4429-889e-eab1ef63949c/.default +``` + +`a365_token_resolver` must be a **sync** callable, but the Agents SDK's FMI helper is async. +So each span-emitting handler awaits `prefetch(...)` once per turn, and `resolve(...)` only reads +the cache. `prefetch` replaces a token five minutes before it expires; `resolve` keeps serving +it until one minute before expiry, so a turn that starts just before the refresh point still +exports. + +```python +# observability/app_token_resolver.py +# A365 Observability — best-effort instrumentation (verify against official sample) +"""App-only token for A365 observability export over the S2S route. + +Workload auth (OBO / agentic user for MCP and Graph) is unchanged; only the telemetry +credential differs: + Step 1: the agent's hosting connection (the blueprint credential) issues an FMI assertion + for the exporting agent identity: get_agentic_application_token(tenant_id, agent_id). + Step 2: the agent identity exchanges that assertion (client_credentials) for an app-only + Observability API token. The S2S route rejects delegated (scp) tokens. +""" + +import asyncio +import base64 +import json +import threading +import time + +import msal + +OBSERVABILITY_SCOPE = "api://9b975845-388f-4429-889e-eab1ef63949c/.default" +SERVE_SKEW_SECONDS = 60 # resolve() stops serving a token this close to expiry +REFRESH_SKEW_SECONDS = 300 # prefetch() replaces a token this close to expiry + + +class AppTokenResolver: + """Caches app-only observability tokens per (tenant, agent identity).""" + + def __init__(self): + self._tokens = {} + self._lock = threading.Lock() + + def resolve(self, agent_id, tenant_id): + """Sync a365_token_resolver: returns a cached, unexpired token or None.""" + return self._cached(agent_id, tenant_id, SERVE_SKEW_SECONDS) + + def _cached(self, agent_id, tenant_id, skew_seconds): + if not agent_id or not tenant_id: + return None + with self._lock: + cached = self._tokens.get((tenant_id.lower(), agent_id.lower())) + if cached and time.time() < cached[1] - skew_seconds: + return cached[0] + return None + + async def prefetch(self, connection_manager, tenant_id, agent_id): + """Acquires the token for this turn's agent identity before its spans are exported.""" + if not tenant_id or not agent_id or self._cached(agent_id, tenant_id, REFRESH_SKEW_SECONDS): + return + connection = connection_manager.get_default_connection() + assertion = await connection.get_agentic_application_token(tenant_id, agent_id) + if not assertion: + raise RuntimeError("The hosting connection issued no FMI assertion for this agent identity.") + result = await asyncio.to_thread(_acquire, tenant_id, agent_id, assertion) + token = result.get("access_token") + if not token: + raise RuntimeError(f"Observability token request failed: {result.get('error')}") + claims = _claims(token) + if "scp" in claims: + raise RuntimeError("Observability token is delegated (scp claim); the S2S route rejects it.") + client = str(claims.get("azp") or claims.get("appid") or "").lower() + if client != agent_id.lower() or str(claims.get("tid", "")).lower() != tenant_id.lower(): + raise RuntimeError("Observability token client or tenant does not match the exporting agent.") + expires_at = float(claims.get("exp") or time.time() + float(result.get("expires_in", 0))) + with self._lock: + self._tokens[(tenant_id.lower(), agent_id.lower())] = (token, expires_at) + + +def _acquire(tenant_id, agent_id, assertion): + app = msal.ConfidentialClientApplication( + agent_id, + client_credential={"client_assertion": lambda: assertion}, + authority=f"https://login.microsoftonline.com/{tenant_id}", + ) + return app.acquire_token_for_client(scopes=[OBSERVABILITY_SCOPE]) + + +def _claims(token): + payload = token.split(".")[1] + return json.loads(base64.urlsafe_b64decode(payload + "=" * (-len(payload) % 4))) +``` + +> **Turns without an agent identity** (for example a Playground session without agentic auth) +> have no `recipient.agentic_app_id`. `prefetch` skips them and their spans are not exported. For +> a non-AI-Teammate `obo` agent you can attribute such turns to the provisioned agent identity +> that `a365 setup all` writes to `AGENT365OBSERVABILITY__AGENTID` in the Python `.env`, but only +> when it differs from `AGENT365OBSERVABILITY__AGENTBLUEPRINTID`. Environment variable names are +> case-sensitive on Linux. Use that ID for both `prefetch(...)` and the turn's +> `BaggageBuilder().agent_id(...)`. + +> **Authorization:** registered blueprint agent instances are authorized on the S2S route without +> the `Agent365.Observability.OtelWrite` permission or admin consent (subject to service policy). +> For AI Teammates, complete the `OtelWrite` application-role step that `a365 setup all --aiteammate` +> prints. Do not add a delegated observability scope to the auth handler for telemetry. --- @@ -426,36 +552,34 @@ ObservabilityHostingManager.configure( With `ObservabilityHostingManager.configure(..., enable_baggage=True)` registered at startup, the handler does NOT build baggage manually. Per-turn behavior differs by auth mode. -### OBO and agentic-user — refresh exporter token per turn +### OBO and agentic-user — prefetch the app-only token per turn -The handler shape is identical for both `obo` and `agentic-user`. The Azure AD auth handler -configured in your `AgentApplication` decides which identity the token exchange returns: +The handler shape is identical for both `obo` and `agentic-user`. The auth handler configured in +your `AgentApplication` is used only for workload calls (MCP / Graph); telemetry never goes +through it: -- **`agentic-user`** (AI Teammate) — `agent_app.auth.exchange_token(...)` returns a token for - the agent's own Azure AD user identity (Agentic User). Traces attribute to the agent. -- **`obo`** (non-AI Teammate) — `agent_app.auth.exchange_token(...)` returns a token for - whatever the configured Azure AD auth handler resolves. Typically this is the signed-in user - (traces attribute to that user), but it can also be the agent's own identity if the handler - is configured that way. +- **`agentic-user`** (AI Teammate) — workload calls run as the agent's own Agentic User. + Telemetry is attributed to the agent instance (`recipient.agentic_app_id`). +- **`obo`** (non-AI Teammate) — workload calls run as whatever the configured auth handler + resolves, typically the signed-in user. The caller still appears in `CallerDetails`. + +Because `a365_token_resolver` is synchronous, each turn awaits a cheap, cached `prefetch` for its +agent identity before any spans are exported. Do this in **every handler that emits spans**: the +message handler and any notification handler (`on_agent_notification`). A handler that skips it +can only reuse a token another handler cached, and once that token expires its spans are dropped. ```python # A365 Observability — best-effort instrumentation (verify against official sample) -# A365 auth mode: agentic-user (or: obo) -from microsoft_agents_a365.runtime.environment_utils import get_observability_authentication_scope -from token_cache import cache_agentic_token +# A365 auth mode: agentic-user (or: obo) — telemetry uses an app-only token on the S2S route +# OBS_TOKENS is the AppTokenResolver created next to use_microsoft_opentelemetry(...) at startup. async def _setup_observability_token(self, context, tenant_id, agent_id): - """OBO / agentic-user: exchange token and cache for the exporter to pick up.""" + """Prefetch the app-only observability token for this turn's agent identity.""" try: - exaau_token = await self.agent_app.auth.exchange_token( - context, - scopes=get_observability_authentication_scope(), - auth_handler_id=self.auth_handler_name, # from config — NOT hardcoded "AGENTIC" - ) - cache_agentic_token(tenant_id, agent_id, exaau_token.token) + await OBS_TOKENS.prefetch(self.connection_manager, tenant_id, agent_id) except Exception as e: - logger.warning(f"Failed to cache observability token: {e}") + logger.warning(f"Failed to acquire observability token: {e}") @AGENT_APP.activity("message", auth_handlers=["AGENTIC"]) @@ -463,7 +587,7 @@ async def on_message(context: TurnContext, state: TurnState): tenant_id = context.activity.recipient.tenant_id agent_id = context.activity.recipient.agentic_app_id - # OBO / agentic-user: refresh per-turn token (skip this for S2S). + # Prefetch the app-only telemetry token (cached; a no-op on warm turns). await self._setup_observability_token(context, tenant_id, agent_id) # ObservabilityHostingManager (registered at startup) already populated baggage @@ -472,10 +596,20 @@ async def on_message(context: TurnContext, state: TurnState): await context.send_activity(response) ``` -> **`auth_handler_id`** must come from config (`AgentApplication:AgenticAuthHandlerName`) -> — **never hardcode `"AGENTIC"`**. The handler is the auth handler registered in your -> agent setup; its configured identity (user delegated or agent's own) determines whose -> token gets returned. +> **`self.connection_manager`** is the `MsalConnectionManager` the host passes to its +> `CloudAdapter`. Its default connection holds the blueprint credential that `a365 setup all` +> writes to `.env`, so no extra settings are needed. `CloudAdapter` does not expose it publicly. +> If the host only keeps it in a local variable (for example +> `connection_manager = MsalConnectionManager.from_environment()` inside `start_server()`), store +> it on the host with `self.connection_manager = ...` and pass the same object to `CloudAdapter`. +> Do not create a second connection manager. `auth_handlers=[...]` still comes from +> config (`AgentApplication:AgenticAuthHandlerName`); never hardcode `"AGENTIC"`. + +> **Migrating an existing agent:** replace the body of any existing +> `_setup_observability_token` that calls `exchange_token(..., scopes=get_observability_authentication_scope(), ...)` +> and `cache_agentic_token(...)`. That delegated exchange needs admin consent, and the S2S +> route rejects its token. Also switch `a365_token_resolver` from `AgenticTokenCache` / +> `get_cached_agentic_token` to `OBS_TOKENS.resolve` and set `a365_use_s2s_endpoint=True`. #### Canonical: manual per-turn baggage construction (matches AF sample) @@ -485,7 +619,6 @@ async def on_message(context: TurnContext, state: TurnState): # A365 Observability — best-effort instrumentation (verify against official sample) # Imports use the legacy module paths (still required even with the unified distro entry point): from microsoft.opentelemetry.a365.core.middleware.baggage_builder import BaggageBuilder -from microsoft_agents_a365.runtime.environment_utils import get_observability_authentication_scope # In your message handler: tenant_id = context.activity.recipient.tenant_id @@ -511,7 +644,7 @@ reads from that cache on each export. The handler does NOT touch tokens. async def on_message(context: TurnContext, state: TurnState): # ObservabilityHostingManager (registered at startup) already populated baggage # from TurnContext. No per-turn token refresh — background token service handles auth. - # Do NOT call _setup_observability_token / exchange_token for S2S. + # Do NOT call exchange_token() for the observability scope in any auth mode. response = await self.invoke_llm(context.activity.text) await context.send_activity(response) @@ -866,7 +999,7 @@ python -c "from microsoft.opentelemetry import use_microsoft_opentelemetry; from | `enable_a365` | Enables A365 observability instrumentation (registers span processors) | | `a365_enable_observability_exporter` | **Required in 1.0+** alongside `enable_a365` to actually export spans to A365 | | `a365_token_resolver` | Sync callable `(agent_id, tenant_id) -> str \| None` for export authentication | -| `a365_use_s2s_endpoint` | `True` posts to `/observabilityService/` (S2S endpoint); `False` posts to `/observability/` (OBO endpoint) | +| `a365_use_s2s_endpoint` | Set `True` for every agent: posts to the S2S route (`/observabilityService/`). `False` (the default) posts to the legacy delegated route (`/observability/`), which rejects app-only tokens | | `a365_cluster_category` | Optional cluster label such as `prod` | | `a365_suppress_invoke_agent_input` | Suppresses input messages on `InvokeAgent` spans | | `a365_observability_scope_override` | Overrides the default OAuth scope (default: `api://9b975845-388f-4429-889e-eab1ef63949c/.default`) | @@ -885,8 +1018,8 @@ python -c "from microsoft.opentelemetry import use_microsoft_opentelemetry; from | `ObservabilityHostingManager` | `microsoft.opentelemetry.a365.hosting` | Composite hosting configuration for adapter middleware | | `ObservabilityHostingOptions` | `microsoft.opentelemetry.a365.hosting` | Options for `ObservabilityHostingManager.configure` (defaults: `enable_baggage=False`, `enable_output_logging=False`) | | `BaggageMiddleware` | `microsoft.opentelemetry.a365.hosting` | Adapter middleware — registered by `ObservabilityHostingManager` | -| `AgenticTokenCache` | `microsoft.opentelemetry.a365.hosting.token_cache_helpers` | Hosting token cache for OBO / agentic-user flows | -| `get_observability_authentication_scope()` | `microsoft_agents_a365.runtime.environment_utils` | Returns the default OAuth scope string (legacy module path — still required) | +| `AgenticTokenCache` | `microsoft.opentelemetry.a365.hosting.token_cache_helpers` | Legacy cache of delegated tokens. Do not use it for telemetry: the S2S route rejects delegated tokens | +| `get_observability_authentication_scope()` | `microsoft_agents_a365.runtime.environment_utils` | Returns the default OAuth scope string. Not needed for telemetry: the app-only resolver requests the scope itself | | `InvokeAgentScope.start(request, scope_details, agent_details, caller_details)` | `microsoft.opentelemetry.a365.core` | Agent invocation scope (context manager) | | `ExecuteToolScope.start(request, tool_details, agent_details)` | `microsoft.opentelemetry.a365.core` | Tool execution scope (context manager) | | `InferenceScope.start(request, inference_details, agent_details)` | `microsoft.opentelemetry.a365.core` | LLM inference scope (context manager) | @@ -906,14 +1039,14 @@ python -c "from microsoft.opentelemetry import use_microsoft_opentelemetry; from | Traces not in Admin Center | Missing `a365_enable_observability_exporter=True` (1.0 breaking change) | Set the kwarg in code, or `ENABLE_A365_OBSERVABILITY_EXPORTER=true` in env | | Duplicate spans for OpenAI/LangChain/SK/AgentFramework | Manual `*Instrumentor().instrument()` call after migration | Remove the manual instrumentor calls — auto-instrumentation is ON by default in 1.0+ | | Spans missing baggage | `ObservabilityHostingManager.configure` not called or `enable_baggage` not set to `True` | Default is `False`. Pass `ObservabilityHostingOptions(enable_baggage=True)` explicitly | -| Token resolver returns `None` | Per-turn OBO token cache was never refreshed | Call `exchange_token()` and cache the result at the start of each handler turn (OBO / agentic-user only) | +| Token resolver returns `None` | The turn's app-only token was never prefetched or its acquisition failed (see the `Failed to acquire observability token` warning), or the turn has no agent identity | Await `_setup_observability_token(...)` (which calls `OBS_TOKENS.prefetch(...)`) at the start of each handler turn. Check that the hosting connection holds the blueprint credential | | `ModuleNotFoundError: microsoft.opentelemetry` | Package not installed | `pip install microsoft-opentelemetry` | | `uv sync` runs for minutes / appears to hang on a Google ADK project | OTel resolver backtracking between `google-adk` (`opentelemetry-sdk<1.39.0`) and `microsoft-opentelemetry` 1.1.x (newer transitive OTel SDK) | Add `[tool.uv] override-dependencies` to `pyproject.toml` pinning `opentelemetry-api` and `opentelemetry-sdk` to `>=1.38.0,<1.39.0`. See the "Google ADK projects — pin the OTel stack" section above. | -| 401 on export | Missing `Agent365.Observability.OtelWrite` permission | CLI 1.1+ grants this automatically via `a365 setup all`. For pre-1.1 agents, GA must grant manually | +| 401 on export | The exporter is on the delegated route (`a365_use_s2s_endpoint` not set) or received a delegated token | Set `a365_use_s2s_endpoint=True` and use the app-only resolver. The token must have no `scp` claim, and its `azp`/`appid` must equal the exporting agent ID | | Spans dropped silently | Missing tenant/agent ID in baggage | Ensure `enable_baggage=True` and that `populate(builder, context)` runs before scope creation | -| S2S: OBO token-refresh code still runs in the handler | S2S does not use per-turn OBO token exchange | Remove the OBO handler refresh path; token comes from the background token service via `a365_token_resolver` | +| Delegated (OBO) telemetry token code still runs in the handler | Telemetry no longer uses a per-turn OBO exchange in any mode | Remove the `exchange_token(... get_observability_authentication_scope() ...)` / `cache_agentic_token` path. Interactive agents prefetch an app-only token; `s2s` agents use the background token service | | S2S 401: wrong Hop 3 scope | FMI Hop 3 used `https://api.powerplatform.com/.default` from older samples | Change Hop 3 scope to `api://9b975845-388f-4429-889e-eab1ef63949c/.default` | -| S2S 401 even with correct scope | `OtelWrite` role not on Agent Identity SP | For agents provisioned before CLI 1.1, manually assign `Agent365.Observability.OtelWrite` to the Agent Identity SP via Entra portal | +| 403 `insufficient_scope` even with correct scope | The agent instance is not registered and has no `OtelWrite` application role. The S2S route authorizes registered blueprint instances without `OtelWrite` (subject to service policy) | Blueprint agents: register the instance with `a365 setup all --agent-registration-only` (idempotent). AI Teammates: complete the `OtelWrite` application-role step that `a365 setup all --aiteammate` prints. Either way, a Global Administrator can grant the `Agent365.Observability.OtelWrite` application role on the Blueprint, which the S2S route accepts | | S2S: MSI fails locally | No Managed Identity in dev | Set `AGENT365_USE_MANAGED_IDENTITY=false` and provide `AGENT365_CLIENT_SECRET` | | S2S: FMI Hop 1+2 returns 400 | `fmi_path` missing or wrong `client_id` | Ensure `fmi_path=` (Agent Identity app ID, not Blueprint ID) and `client_id=` | | S2S: `TypeError: Session.request() got an unexpected keyword argument 'fmi_path'` | MSAL Python v1.34.0 limitation | Use direct HTTP POST to `https://login.microsoftonline.com/{tenantId}/oauth2/v2.0/token` with `fmi_path` as form data instead of MSAL `acquire_token_for_client(fmi_path=...)`. MSAL is still used for Hop 3 (no `fmi_path` needed) | diff --git a/plugins/agent365/skills/make-a365-agent/SKILL.md b/plugins/agent365/skills/make-a365-agent/SKILL.md index 0c421ce..37628c4 100644 --- a/plugins/agent365/skills/make-a365-agent/SKILL.md +++ b/plugins/agent365/skills/make-a365-agent/SKILL.md @@ -288,7 +288,7 @@ Monitor output carefully: After `a365 setup all` completes, show the user: 1. **The Setup Summary table** from CLI output — verbatim. -2. **`Agent365.Observability.OtelWrite` is automatically granted** to the agent identity by `a365 setup all` — no GA consent step required for newly provisioned agents. If the CLI output includes a "Permission Grants" action item (upgrade scenario for pre-1.1 agents), display the PowerShell script verbatim so the user can hand it to a Global Admin. +2. **No Observability API permission is needed.** Recent `a365 setup all` versions no longer request `Agent365.Observability.OtelWrite` (or its admin consent) for blueprint agents. Telemetry is exported over the S2S route with an app-only token, and the route authorizes the registered agent instance. These versions also fail setup (exit code 1) when agent registration fails or cannot be verified. If that happens, show the error and have the user re-run `a365 setup all --agent-registration-only` after fixing it. Older CLI versions may still grant OtelWrite, which is harmless. If the CLI output includes an action item for other permissions (Graph, Bot API, custom resources), display the printed PowerShell script verbatim so the user can hand it to a Global Admin. 3. **Skip the client secret action item entirely.** Do not show or mention it. Mark Todo 1 as completed. diff --git a/plugins/agent365/skills/make-ai-teammate/SKILL.md b/plugins/agent365/skills/make-ai-teammate/SKILL.md index 4f6e818..0d03cb8 100644 --- a/plugins/agent365/skills/make-ai-teammate/SKILL.md +++ b/plugins/agent365/skills/make-ai-teammate/SKILL.md @@ -327,11 +327,12 @@ Ask: "What language and framework are you using?" and set `language` and `agentS - `CloudAdapter` in `src/**/*.ts` → `hasHosting` - `onAgentNotification` in `src/**/*.ts` → `hasNotifications` - `ToolingManifest.json` exists → `hasManifest` -- **Observability composite** — compute three sub-signals, then combine: +- **Observability composite** — compute four sub-signals, then combine: - `obs_entry` = `useMicrosoftOpenTelemetry` in any `src/**/*.ts` - - `obs_token` = `tokenResolver` OR `AgenticTokenCacheInstance` in any `src/**/*.ts` (S2S also accepts `getS2SObservabilityToken` / `startTokenService`) + - `obs_token` = `tokenResolver` in any `src/**/*.ts` (the app-only resolver `observability/app-token-resolver.ts` for obo / agentic-user; S2S also accepts `getS2SObservabilityToken` / `startTokenService`), AND no `refreshObservabilityToken(..., authorization)` call or `AgenticTokenCacheInstance.getObservabilityToken` resolver (either means the telemetry token is delegated) + - `obs_route` = `useS2SEndpoint: true` in any `src/**/*.ts` (telemetry uses the S2S route in every auth mode) - `obs_handler` = `BaggageBuilder` OR `BaggageBuilderUtils` OR `InvokeAgentScope` in any `src/**/*.ts` - - `has_obs_complete` = `obs_entry && obs_token && obs_handler` + - `has_obs_complete` = `obs_entry && obs_token && obs_route && obs_handler` - `has_obs_partial` = `obs_entry && !has_obs_complete` - `has_obs` = `has_obs_complete` *(only "true" when the wiring is end-to-end)* @@ -342,9 +343,10 @@ Ask: "What language and framework are you using?" and set `language` and `agentS - `ToolingManifest.json` exists → `hasManifest` - **Observability composite:** - `obs_entry` = `UseMicrosoftOpenTelemetry` in `Program.cs` (or legacy `AddA365Tracing`) - - `obs_token` = OBO: distro auto-registers `IExporterTokenCache` so accept `UseMicrosoftOpenTelemetry` itself; S2S: `ObservabilityTokenService` / `AddAgent365Observability` + - `obs_token` = `AgentAppTokenResolver` (obo / agentic-user) or `ObservabilityTokenService` / `AddAgent365Observability` (S2S), AND no `AgenticTokenStruct` usage (`RegisterObservability(..., new AgenticTokenStruct(...))`, `new AgenticTokenStruct(...)`, or an `IExporterTokenCache` dependency — all register a delegated telemetry token) + - `obs_route` = `UseS2SEndpoint = true` in any `**/*.cs` (telemetry uses the S2S route in every auth mode) - `obs_handler` = `BaggageBuilder` OR `BaggageTurnMiddleware` OR `InvokeAgentScope.Start` in `**/*.cs` - - `has_obs_complete` = `obs_entry && obs_token && obs_handler` + - `has_obs_complete` = `obs_entry && obs_token && obs_route && obs_handler` - `has_obs_partial` = `obs_entry && !has_obs_complete` - `has_obs` = `has_obs_complete` @@ -355,12 +357,18 @@ Ask: "What language and framework are you using?" and set `language` and `agentS - `ToolingManifest.json` exists → `hasManifest` - **Observability composite:** - `obs_entry` = `use_microsoft_opentelemetry` in any `**/*.py` - - `obs_token` = `token_resolver` OR `AgenticTokenCache` OR `cache_agentic_token` OR S2S: `run_token_service` / `get_s2s_observability_token` + - `obs_token` = `token_resolver` (the app-only `AppTokenResolver` / `OBS_TOKENS` for obo / agentic-user; S2S: `run_token_service` / `get_s2s_observability_token`), AND no `exchange_token(...)` for the observability scope, `cache_agentic_token(...)`, or `AgenticTokenCache` / `get_cached_agentic_token` resolver (all are delegated telemetry tokens) + - `obs_route` = `a365_use_s2s_endpoint=True` in any `**/*.py` (or `A365_USE_S2S_ENDPOINT=true` in `.env`) — telemetry uses the S2S route in every auth mode - `obs_handler` = `BaggageBuilder` OR `populate_baggage` OR `InvokeAgentScope` in any `**/*.py` - - `has_obs_complete` = `obs_entry && obs_token && obs_handler` + - `has_obs_complete` = `obs_entry && obs_token && obs_route && obs_handler` - `has_obs_partial` = `obs_entry && !has_obs_complete` - `has_obs` = `has_obs_complete` +> **Legacy delegated telemetry is partial, not complete.** A project whose exporter still uses the +> delegated route or a per-turn delegated token (missing `obs_route`, or a delegated refresh that +> fails `obs_token`) is `has_obs_partial`. Phase 9.5 re-enters `instrument-observability`, which +> migrates it to the S2S route with an app-only token ("Migrating delegated telemetry" in its Phase 3). + **Skill-state signals** (language-agnostic): - **`has_workiq` (composite — replaces the disk-only check):** @@ -829,7 +837,7 @@ Do NOT revert changes on build failure — fix forward. **Three-way gate** — branch on the composite signal computed in Phase 0A.3: - **If `has_obs_complete = true`** (rows 2, 4, 6, 8 *with* full end-to-end wiring): tell the user verbatim *"Observability already wired end-to-end (entry-point + token resolver + handler-side baggage / scopes) — skipping. Run `/agent365:instrument-observability` to reconfigure."* and mark the task complete. Do NOT invoke the sub-skill. -- **If `has_obs_partial = true`** (any signal of obs in the project but at least one of entry / token / handler is missing): **do not skip — recover.** Tell the user verbatim *"Found partial observability wiring (entry-point present, but token resolver and/or message-handler scopes are missing). Completing the wiring now — re-entering `/agent365:instrument-observability` to finish what was left half-done."* Then proceed to the same steps as the `has_obs = false` branch below. The sub-skill is idempotent and additive, so partial-recovery is safe. +- **If `has_obs_partial = true`** (any signal of obs in the project, but the entry, token, route, or handler anchor is missing or delegated): **do not skip — recover.** Tell the user verbatim *"Found partial observability wiring (entry-point present, but token resolver, S2S route, and/or message-handler scopes are missing or still use delegated tokens). Completing the wiring now — re-entering `/agent365:instrument-observability` to finish what was left half-done."* Then proceed to the same steps as the `has_obs = false` branch below. The sub-skill is safe for partial recovery: it adds only missing pieces and migrates delegated telemetry to the app-only S2S path (its Idempotency rule does not skip delegated wiring). - **If `has_obs_complete = false && has_obs_partial = false`** (rows 1, 3, 5, 7 — no obs at all): proceed to the steps below. **Steps (for partial-recovery and fresh-wire paths):** diff --git a/plugins/agent365/skills/make-ai-teammate/references/deploy-pipeline.md b/plugins/agent365/skills/make-ai-teammate/references/deploy-pipeline.md index 288ab86..c0b36cf 100644 --- a/plugins/agent365/skills/make-ai-teammate/references/deploy-pipeline.md +++ b/plugins/agent365/skills/make-ai-teammate/references/deploy-pipeline.md @@ -122,7 +122,7 @@ After completion: node -e "const c=require('./a365.generated.config.json'); console.log('Blueprint ID:', c.agentBlueprintId)" ``` -**If the CLI output includes a "Permission Grants" action item or any 403 errors:** display the PowerShell script printed in the CLI output verbatim so the user can copy it. This is only expected for agents upgrading from a pre-1.1 CLI version where OtelWrite was not yet auto-granted. For newly provisioned agents no admin consent step is required. +**If the CLI output includes an Observability API S2S app role action item, another "Permission Grants" action item, or any 403 errors:** display the PowerShell script printed in the CLI output verbatim so the user can hand it to a Global Administrator. AI Teammate telemetry is exported over the S2S route with an app-only token for the agent instance (never a delegated token). That route accepts the `Agent365.Observability.OtelWrite` **application** role, which the script grants on the Blueprint, and it also accepts the instance's registration where service policy allows. A declined or pending app role grant does not block the rest of setup; continue with the next phase and surface it in the final summary. --- diff --git a/plugins/agent365/skills/make-ai-teammate/references/dotnet-ai-teammate.md b/plugins/agent365/skills/make-ai-teammate/references/dotnet-ai-teammate.md index dc9b08c..1aa86d5 100644 --- a/plugins/agent365/skills/make-ai-teammate/references/dotnet-ai-teammate.md +++ b/plugins/agent365/skills/make-ai-teammate/references/dotnet-ai-teammate.md @@ -272,10 +272,8 @@ namespace YourNamespace.Agent private readonly IChatClient? _chatClient; private readonly IMcpToolRegistrationService _toolService; private readonly IConfiguration? _configuration; - // Auto-registered by the Microsoft.OpenTelemetry distro. Held here so the - // observability skill can wire RegisterObservability(...) per turn without - // having to reopen the constructor. - private readonly IExporterTokenCache? _agentTokenCache; + // A365 observability needs nothing injected here: instrument-observability wires an + // app-only token resolver in Program.cs (S2S route), not a per-turn token registration. private readonly ILogger? _logger; private readonly string? AgenticAuthHandlerName; private readonly string? OboAuthHandlerName; @@ -288,16 +286,11 @@ namespace YourNamespace.Agent AgentApplicationOptions options, IChatClient chatClient, IConfiguration configuration, - IExporterTokenCache agentTokenCache, IMcpToolRegistrationService toolService, ILogger logger) : base(options) { _chatClient = chatClient; _configuration = configuration; - // Auto-registered by the Microsoft.OpenTelemetry distro — used by instrument-observability - // for per-turn RegisterObservability(...) calls. Inject up-front so the constructor doesn't - // need to be reopened when the observability skill runs later. - _agentTokenCache = agentTokenCache; _toolService = toolService; _logger = logger; diff --git a/plugins/agent365/skills/make-ai-teammate/references/python-ai-teammate.md b/plugins/agent365/skills/make-ai-teammate/references/python-ai-teammate.md index 6f28baa..3948709 100644 --- a/plugins/agent365/skills/make-ai-teammate/references/python-ai-teammate.md +++ b/plugins/agent365/skills/make-ai-teammate/references/python-ai-teammate.md @@ -316,6 +316,8 @@ class GenericAgentHost: ChannelId(channel="agents", sub_channel="*") ) async def on_notification(context, state, notification): + # instrument-observability adds its per-turn app-only token prefetch here too + # (await self._setup_observability_token(...)) — notification turns emit spans. notification_type = getattr(context.activity, "name", None) reply = await self._agent.handle_agent_notification_activity( notification_type, @@ -333,9 +335,11 @@ class GenericAgentHost: # MsalConnectionManager reads all CONNECTIONS__* / AGENTAPPLICATION__* env vars # and resolves the right token issuer per service URL. Do NOT pass raw # client_id / client_secret / tenant_id to the adapter — the connection - # manager owns that. - connection_manager = MsalConnectionManager.from_environment() - self._adapter = CloudAdapter(connection_manager=connection_manager) + # manager owns that. Kept on the host (self.connection_manager) because + # instrument-observability mints app-only telemetry tokens from the same + # connection; CloudAdapter does not expose it publicly. + self.connection_manager = MsalConnectionManager.from_environment() + self._adapter = CloudAdapter(connection_manager=self.connection_manager) self._setup_handlers() self._app = web.Application() diff --git a/plugins/agent365/skills/test-local/SKILL.md b/plugins/agent365/skills/test-local/SKILL.md index 47a602d..6ec3b71 100644 --- a/plugins/agent365/skills/test-local/SKILL.md +++ b/plugins/agent365/skills/test-local/SKILL.md @@ -336,10 +336,11 @@ Tell the user: > [Agent365Exporter] Partitioned into K identity groups (X spans skipped) > Agent365ExporterCore: Obtained token for agent tenant . > Agent365ExporterCore: Sending chunk 1 of 1 (J spans, B bytes) -> to https://agent365.svc.cloud.microsoft/observability/tenants//otlp/agents//traces?api-version=1. +> to https://agent365.svc.cloud.microsoft/observabilityService/tenants//otlp/agents//traces?api-version=1. > Agent365ExporterCore: HTTP 200 exporting spans. 'x-ms-correlation-id': ''. > ``` > The `HTTP 200 exporting spans` line is the definitive confirmation that traces reached the A365 backend. +> The URL must contain `/observabilityService/` (the S2S route every auth mode uses). `/observability/` means the S2S route flag is missing (`useS2SEndpoint: true` / `a365_use_s2s_endpoint=True` / `o.Agent365.UseS2SEndpoint = true`). A 403 `insufficient_scope` from the S2S route with an app-only token means the instance isn't registered and has no `OtelWrite` application role. For blueprint agents, run `a365 setup all --agent-registration-only`. For AI Teammates, complete the app-role step that `a365 setup all --aiteammate` prints. > `Partitioned into K identity groups` should show `K >= 1` for at least one batch after a Teams turn — if it's always `0`, the agent/tenant ID is missing from baggage (likely the `Guid.Empty` fallback bug — verify `instrument-observability` was followed correctly). > To see these logs you need `Microsoft.Agents.A365.Observability: Debug` (or lower) in `appsettings.json`'s `Logging:LogLevel`. > diff --git a/tests/validate-a365-code-validator.test.js b/tests/validate-a365-code-validator.test.js index 7498a3f..ef11edb 100644 --- a/tests/validate-a365-code-validator.test.js +++ b/tests/validate-a365-code-validator.test.js @@ -291,3 +291,262 @@ useMicrosoftOpenTelemetry({ } }); }); + +// ── S2S route with an app-only token in every auth mode ───────────────────── + +const STANDALONE = path.join(__dirname, '../plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js'); + +const NODE_PKG = JSON.stringify({ name: 'node-agent', dependencies: { '@microsoft/opentelemetry': '^1.4.0' } }, null, 2); +const NODE_S2S_INDEX = ` +import { useMicrosoftOpenTelemetry } from '@microsoft/opentelemetry'; +useMicrosoftOpenTelemetry({ + a365: { enabled: true, enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver: appTokenResolver }, +}); +`.trim(); + +describe('validate-a365-code-validator — S2S route and delegated telemetry', () => { + test('Node distro without useS2SEndpoint reports delegated route (high)', () => { + const dir = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': NODE_S2S_INDEX.replace(' useS2SEndpoint: true,', ''), + }); + try { + const result = runValidator(VALIDATOR, dir); + assert.equal(result.ok, true); + const finding = result.findings.find(f => f.id === 'node-obs-delegated-route'); + assert.ok(finding, 'expected node-obs-delegated-route'); + assert.equal(finding.severity, 'high'); + } finally { + cleanup(dir); + } + }); + + test('Node refreshObservabilityToken(..., authorization) reports delegated token; app-only S2S wiring does not', () => { + const dir = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': NODE_S2S_INDEX, + 'agent.ts': ` +await AgenticTokenCacheInstance.refreshObservabilityToken( + agentId, tenantId, turnContext as any, this.authorization as any); + `.trim(), + }); + try { + const result = runValidator(VALIDATOR, dir); + const ids = findingIds(result); + assert.ok(ids.includes('node-obs-delegated-token')); + assert.ok(!ids.includes('node-obs-delegated-route')); + } finally { + cleanup(dir); + } + }); + + test('.NET distro without UseS2SEndpoint and with AgenticTokenStruct registration reports both findings', () => { + const dir = createFixture({ + 'Agent.csproj': '', + 'Program.cs': 'builder.UseMicrosoftOpenTelemetry(o => { o.Exporters = ExportTarget.Agent365; });', + 'MyAgent.cs': ` +_agentTokenCache?.RegisterObservability(agentId, tenantId, + new AgenticTokenStruct(userAuthorization: UserAuthorization, turnContext: turnContext, authHandlerName: name), + EnvironmentUtils.GetObservabilityAuthenticationScope()); + `.trim(), + 'appsettings.json': '{ "EnableAgent365Exporter": true }', + }); + try { + const ids = findingIds(runValidator(VALIDATOR, dir)); + assert.ok(ids.includes('dotnet-obs-delegated-route')); + assert.ok(ids.includes('dotnet-obs-delegated-token')); + } finally { + cleanup(dir); + } + }); + + test('.NET distro with UseS2SEndpoint = true and S2S scaffold registration reports no delegated findings', () => { + const dir = createFixture({ + 'Agent.csproj': '', + 'Program.cs': 'builder.UseMicrosoftOpenTelemetry(o => { o.Agent365.UseS2SEndpoint = true; o.Agent365.TokenResolver = (a, t) => r.ResolveAsync(a, t); });', + 'ObservabilityTokenService.cs': '_tokenCache.RegisterObservability(_agentId, _tenantId, obsResult.AccessToken, ObservabilityScopes);', + 'appsettings.json': '{ "EnableAgent365Exporter": true }', + }); + try { + const ids = findingIds(runValidator(VALIDATOR, dir)); + assert.ok(!ids.includes('dotnet-obs-delegated-route')); + assert.ok(!ids.includes('dotnet-obs-delegated-token')); + } finally { + cleanup(dir); + } + }); + + test('Python OBO agent without the S2S flag and with a delegated observability exchange reports both findings', () => { + const dir = createFixture({ + 'requirements.txt': 'microsoft-opentelemetry>=1.1.0\n', + 'app.py': ` +from microsoft.opentelemetry import use_microsoft_opentelemetry +use_microsoft_opentelemetry(enable_a365=True, a365_enable_observability_exporter=True, a365_token_resolver=cache.get) + +async def setup(self, context): + token = await self.agent_app.auth.exchange_token(context, scopes=get_observability_authentication_scope(), auth_handler_id=h) + `.trim(), + }); + try { + const result = runValidator(VALIDATOR, dir); + const ids = findingIds(result); + assert.ok(ids.includes('python-s2s-endpoint-not-set')); + assert.ok(ids.includes('python-obs-delegated-token')); + assert.equal(result.findings.find(f => f.id === 'python-s2s-endpoint-not-set').severity, 'high'); + } finally { + cleanup(dir); + } + }); + + test('Python workload exchange_token for MCP scopes is not reported as delegated telemetry', () => { + const dir = createFixture({ + 'requirements.txt': 'microsoft-opentelemetry>=1.1.0\n', + 'app.py': ` +from microsoft.opentelemetry import use_microsoft_opentelemetry +use_microsoft_opentelemetry(enable_a365=True, a365_enable_observability_exporter=True, a365_use_s2s_endpoint=True, a365_token_resolver=OBS_TOKENS.resolve) +token = await auth.exchange_token(context, scopes=["ea9ffc3e-8a23-4a7d-836d-234d7c7565c1/.default"], auth_handler_id=h) + `.trim(), + }); + try { + const ids = findingIds(runValidator(VALIDATOR, dir)); + assert.ok(!ids.includes('python-obs-delegated-token')); + assert.ok(!ids.includes('python-s2s-endpoint-not-set')); + } finally { + cleanup(dir); + } + }); + + test('Blueprint agent without a recorded registration reports agent-registration-not-recorded', () => { + const agentId = '22222222-2222-2222-2222-222222222222'; + const blueprintId = '33333333-3333-3333-3333-333333333333'; + const unregistered = createFixture({ + 'a365.config.json': JSON.stringify({ aiTeammate: false }), + 'a365.generated.config.json': JSON.stringify({ agentBlueprintId: blueprintId, agenticAppId: agentId }), + }); + const registered = createFixture({ + 'a365.config.json': JSON.stringify({ aiTeammate: false }), + 'a365.generated.config.json': JSON.stringify({ agentBlueprintId: blueprintId, agenticAppId: agentId, agentRegistrationId: 'reg-1' }), + }); + const aiTeammate = createFixture({ + 'a365.config.json': JSON.stringify({ aiTeammate: true }), + 'a365.generated.config.json': JSON.stringify({ agentBlueprintId: blueprintId, agenticAppId: agentId }), + }); + try { + const finding = runValidator(VALIDATOR, unregistered).findings.find(f => f.id === 'agent-registration-not-recorded'); + assert.ok(finding, 'expected agent-registration-not-recorded'); + assert.equal(finding.severity, 'medium'); + assert.ok(!findingIds(runValidator(VALIDATOR, registered)).includes('agent-registration-not-recorded')); + assert.ok(!findingIds(runValidator(VALIDATOR, aiTeammate)).includes('agent-registration-not-recorded')); + } finally { + cleanup(unregistered); + cleanup(registered); + cleanup(aiTeammate); + } + }); + + test('standalone references/a365-code-validator.js reports the same S2S findings as the stop hook', () => { + const dir = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': NODE_S2S_INDEX.replace(' useS2SEndpoint: true,', ''), + 'agent.ts': 'await AgenticTokenCacheInstance.refreshObservabilityToken(agentId, tenantId, turnContext, this.authorization);', + 'Agent.csproj': '', + 'Program.cs': 'builder.UseMicrosoftOpenTelemetry(o => { });', + 'MyAgent.cs': '_c.RegisterObservability(a, t, new AgenticTokenStruct(userAuthorization: u, turnContext: c, authHandlerName: n), s);', + 'requirements.txt': 'microsoft-opentelemetry>=1.1.0\n', + 'app.py': 'use_microsoft_opentelemetry(enable_a365=True)\nt = await auth.exchange_token(ctx, scopes=get_observability_authentication_scope())', + 'a365.config.json': JSON.stringify({ aiTeammate: false }), + 'a365.generated.config.json': JSON.stringify({ agentBlueprintId: 'b', agenticAppId: 'a' }), + }); + const s2sIds = ids => ids.filter(id => /obs-delegated|s2s-endpoint|agent-registration/.test(id)).sort(); + try { + const hook = s2sIds(findingIds(runValidator(VALIDATOR, dir))); + const standalone = s2sIds(findingIds(runValidator(STANDALONE, dir))); + assert.deepEqual(standalone, hook); + assert.deepEqual(hook, [ + 'agent-registration-not-recorded', + 'dotnet-obs-delegated-route', + 'dotnet-obs-delegated-token', + 'node-obs-delegated-route', + 'node-obs-delegated-token', + 'python-obs-delegated-token', + 'python-s2s-endpoint-not-set', + ]); + } finally { + cleanup(dir); + } + }); + + test('standalone and stop-hook scanners flag non-inline delegated telemetry and a missing connection manager identically', () => { + const dir = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': NODE_S2S_INDEX.replace('tokenResolver: appTokenResolver', "tokenResolver: (a, t) => AgenticTokenCacheInstance.getObservabilityToken(a, t) ?? ''"), + 'Agent.csproj': '', + 'Program.cs': 'builder.UseMicrosoftOpenTelemetry(o => { o.Agent365.UseS2SEndpoint = true; });', + 'A365OtelWrapper.cs': 'var agenticToken = new AgenticTokenStruct(userAuthorization: u, turnContext: c, authHandlerName: n);\nagentTokenCache?.RegisterObservability(agentId, tenantId, agenticToken, scopes);', + 'requirements.txt': 'microsoft-opentelemetry>=1.1.0\n', + 'host.py': [ + 'use_microsoft_opentelemetry(enable_a365=True, a365_use_s2s_endpoint=True, a365_token_resolver=get_cached_agentic_token)', + 'token = await self.agent_app.auth.exchange_token(context, auth_handler_id=h)', + 'cache_agentic_token(tenant_id, agent_id, token.token)', + 'await OBS_TOKENS.prefetch(self.connection_manager, tenant_id, agent_id)', + ].join('\n'), + 'token_cache.py': 'def cache_agentic_token(tenant_id, agent_id, token):\n pass\n', + }); + const s2sIds = ids => ids.filter(id => /obs-|s2s-endpoint|agent-registration/.test(id)).sort(); + try { + const hook = s2sIds(findingIds(runValidator(VALIDATOR, dir))); + const standalone = s2sIds(findingIds(runValidator(STANDALONE, dir))); + assert.deepEqual(standalone, hook); + assert.deepEqual(hook, [ + 'dotnet-obs-delegated-token', + 'node-obs-delegated-token', + 'python-obs-delegated-token', + 'python-obs-prefetch-connection-missing', + ]); + } finally { + cleanup(dir); + } + }); + + test('Python lambda resolver over get_cached_agentic_token is flagged by both scanners; a lambda over the app-only resolver is not', () => { + const delegated = createFixture({ + 'requirements.txt': 'microsoft-opentelemetry>=1.1.0\n', + 'host.py': 'use_microsoft_opentelemetry(enable_a365=True, a365_enable_observability_exporter=True, a365_use_s2s_endpoint=True, a365_token_resolver=lambda agent_id, tenant_id: get_cached_agentic_token(tenant_id, agent_id))', + }); + const appOnly = createFixture({ + 'requirements.txt': 'microsoft-opentelemetry>=1.1.0\n', + 'host.py': 'use_microsoft_opentelemetry(enable_a365=True, a365_enable_observability_exporter=True, a365_use_s2s_endpoint=True, a365_token_resolver=lambda agent_id, tenant_id: OBS_TOKENS.resolve(agent_id, tenant_id))', + }); + try { + for (const scanner of [VALIDATOR, STANDALONE]) { + assert.ok(findingIds(runValidator(scanner, delegated)).includes('python-obs-delegated-token'), scanner); + assert.ok(!findingIds(runValidator(scanner, appOnly)).includes('python-obs-delegated-token'), scanner); + } + } finally { + cleanup(delegated); + cleanup(appOnly); + } + }); + + test('app-only S2S wiring in all three languages produces no delegated-telemetry findings in either scanner', () => { + const dir = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': NODE_S2S_INDEX, + 'Agent.csproj': '', + 'Program.cs': 'builder.UseMicrosoftOpenTelemetry(o => { o.Agent365.UseS2SEndpoint = true; o.Agent365.TokenResolver = (a, t) => r.ResolveAsync(a, t); });', + 'requirements.txt': 'microsoft-opentelemetry>=1.1.0\n', + 'host.py': [ + 'use_microsoft_opentelemetry(enable_a365=True, a365_enable_observability_exporter=True, a365_use_s2s_endpoint=True, a365_token_resolver=OBS_TOKENS.resolve)', + 'self.connection_manager = MsalConnectionManager.from_environment()', + 'await OBS_TOKENS.prefetch(self.connection_manager, tenant_id, agent_id)', + ].join('\n'), + }); + const s2sIds = ids => ids.filter(id => /obs-|s2s-endpoint/.test(id)); + try { + assert.deepEqual(s2sIds(findingIds(runValidator(VALIDATOR, dir))), []); + assert.deepEqual(s2sIds(findingIds(runValidator(STANDALONE, dir))), []); + } finally { + cleanup(dir); + } + }); +}); diff --git a/tests/validate-observability.test.js b/tests/validate-observability.test.js index 29572f7..d6a1b09 100644 --- a/tests/validate-observability.test.js +++ b/tests/validate-observability.test.js @@ -347,6 +347,338 @@ baggage = BaggageBuilder().build() }); }); +// ── Distro: telemetry must use the S2S route with an app-only token ───────── + +const DOTNET_DISTRO_VALID = { + '.a365-workspace-detection.local.json': JSON.stringify({ agentType: 'ai-teammate', authMode: 'agentic-user' }), + 'MyAgent.csproj': ``, + 'Program.cs': `builder.Services.AddSingleton(); +builder.UseMicrosoftOpenTelemetry(o => +{ + o.Agent365.UseS2SEndpoint = true; + o.Agent365.TokenResolver = (agentId, tenantId) => obsTokens?.ResolveAsync(agentId, tenantId) ?? Task.FromResult(null); +});`, + 'MyAgent.cs': `using IDisposable? baggageScope = new BaggageBuilder().TenantId(t).AgentId(a).Build(); +invokeScope = InvokeAgentScope.Start(request: r, scopeDetails: d, agentDetails: ad, callerDetails: cd);`, + 'Observability/AgentAppTokenResolver.cs': `public sealed class AgentAppTokenResolver { }`, + 'appsettings.json': DOTNET_VALID['appsettings.json'], +}; + +const NODEJS_DISTRO_VALID = { + '.a365-workspace-detection.local.json': JSON.stringify({ agentType: 'ai-teammate', authMode: 'agentic-user' }), + 'package.json': JSON.stringify({ name: 'my-agent', dependencies: { '@microsoft/opentelemetry': '^1.4.0' } }, null, 2), + 'src/index.ts': ` +import { useMicrosoftOpenTelemetry } from '@microsoft/opentelemetry'; +import { createAppTokenResolver } from './observability/app-token-resolver'; +const appTokenResolver = createAppTokenResolver(() => getObsConnection()); +useMicrosoftOpenTelemetry({ a365: { enabled: true, enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver: appTokenResolver } }); + `.trim(), + 'src/agent.ts': ` +const baggageScope = BaggageBuilderUtils.fromTurnContext(new BaggageBuilder(), turnContext as any).build(); +await baggageScope.run(async () => { const scope = InvokeAgentScope.start(request, details, agentDetails, callerDetails); }); + `.trim(), + 'src/observability/app-token-resolver.ts': `export function createAppTokenResolver(getProvider: any) { return async () => ''; }`, + '.env': 'ENABLE_A365_OBSERVABILITY_EXPORTER=true', +}; + +const PYTHON_DISTRO_VALID = { + '.a365-workspace-detection.local.json': JSON.stringify({ agentType: 'system-agent', authMode: 'obo' }), + 'pyproject.toml': '[project]\ndependencies = ["microsoft-opentelemetry"]\n', + 'host_agent_server.py': ` +from microsoft.opentelemetry import use_microsoft_opentelemetry +from observability.app_token_resolver import AppTokenResolver +OBS_TOKENS = AppTokenResolver() +use_microsoft_opentelemetry(enable_a365=True, a365_enable_observability_exporter=True, a365_use_s2s_endpoint=True, a365_token_resolver=OBS_TOKENS.resolve) + +class GenericAgentHost: + def __init__(self): + self.connection_manager = MsalConnectionManager(**agents_sdk_config) + self.adapter = CloudAdapter(connection_manager=self.connection_manager) + + async def _setup_observability_token(self, context, tenant_id, agent_id): + await OBS_TOKENS.prefetch(self.connection_manager, tenant_id, agent_id) + +with BaggageBuilder().tenant_id(t).agent_id(a).build(): + with InvokeAgentScope.start(request, details, agent_details, caller_details): + pass + `.trim(), + 'observability/app_token_resolver.py': 'class AppTokenResolver:\n pass\n', + '.env': 'ENABLE_A365_OBSERVABILITY_EXPORTER=true', +}; + +describe('validate-observability — S2S route with app-only token (distro, every auth mode)', () => { + test('.NET distro with UseS2SEndpoint and AgentAppTokenResolver → ok', () => { + const dir = createFixture(DOTNET_DISTRO_VALID); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, true, r.reason); + } finally { cleanup(dir); } + }); + + test('.NET distro without UseS2SEndpoint = true → reports the S2S route requirement', () => { + const dir = createFixture({ + ...DOTNET_DISTRO_VALID, + 'Program.cs': DOTNET_DISTRO_VALID['Program.cs'].replace('o.Agent365.UseS2SEndpoint = true;', ''), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /S2S route in every auth mode.*UseS2SEndpoint = true/); + } finally { cleanup(dir); } + }); + + test('.NET per-turn RegisterObservability with AgenticTokenStruct → reports delegated telemetry token', () => { + const dir = createFixture({ + ...DOTNET_DISTRO_VALID, + 'MyAgent.cs': `${DOTNET_DISTRO_VALID['MyAgent.cs']} +_agentTokenCache?.RegisterObservability(agentId, tenantId, + new AgenticTokenStruct(userAuthorization: UserAuthorization, turnContext: turnContext, authHandlerName: name), + EnvironmentUtils.GetObservabilityAuthenticationScope());`, + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /RegisterObservability.*delegated \(OBO\) telemetry token/); + } finally { cleanup(dir); } + }); + + test('.NET S2S scaffold RegisterObservability(agentId, tenantId, token, scopes) is not flagged', () => { + const dir = createFixture({ + ...DOTNET_DISTRO_VALID, + 'Observability/ObservabilityTokenService.cs': `_tokenCache.RegisterObservability(_agentId, _tenantId, obsResult.AccessToken, ObservabilityScopes);`, + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, true, r.reason); + } finally { cleanup(dir); } + }); + + test('Node.js distro with useS2SEndpoint and app-only resolver → ok', () => { + const dir = createFixture(NODEJS_DISTRO_VALID); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, true, r.reason); + } finally { cleanup(dir); } + }); + + test('Node.js distro without useS2SEndpoint: true → reports the S2S route requirement', () => { + const dir = createFixture({ + ...NODEJS_DISTRO_VALID, + 'src/index.ts': NODEJS_DISTRO_VALID['src/index.ts'].replace(' useS2SEndpoint: true,', ''), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /S2S route in every auth mode.*useS2SEndpoint: true/); + } finally { cleanup(dir); } + }); + + test('Node.js refreshObservabilityToken(..., authorization) → reports delegated telemetry token', () => { + const dir = createFixture({ + ...NODEJS_DISTRO_VALID, + 'src/agent.ts': `${NODEJS_DISTRO_VALID['src/agent.ts']} +await AgenticTokenCacheInstance.refreshObservabilityToken( + agentId, tenantId, turnContext as any, this.authorization as any);`, + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /refreshObservabilityToken.*delegated \(OBO\) telemetry token/); + } finally { cleanup(dir); } + }); + + test('Node.js comment mentioning refreshObservabilityToken is not flagged', () => { + const dir = createFixture({ + ...NODEJS_DISTRO_VALID, + 'src/agent.ts': `${NODEJS_DISTRO_VALID['src/agent.ts']} +// Do NOT call AgenticTokenCacheInstance.refreshObservabilityToken in any auth mode (authorization is for workload calls).`, + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, true, r.reason); + } finally { cleanup(dir); } + }); + + test('Python distro with a365_use_s2s_endpoint=True and app-only resolver → ok', () => { + const dir = createFixture(PYTHON_DISTRO_VALID); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, true, r.reason); + } finally { cleanup(dir); } + }); + + test('Python distro without a365_use_s2s_endpoint → reports the S2S route requirement', () => { + const dir = createFixture({ + ...PYTHON_DISTRO_VALID, + 'host_agent_server.py': PYTHON_DISTRO_VALID['host_agent_server.py'].replace(' a365_use_s2s_endpoint=True,', ''), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /S2S route in every auth mode.*a365_use_s2s_endpoint=True/); + } finally { cleanup(dir); } + }); + + test('Python distro relying on A365_USE_S2S_ENDPOINT=true in .env → ok', () => { + const dir = createFixture({ + ...PYTHON_DISTRO_VALID, + 'host_agent_server.py': PYTHON_DISTRO_VALID['host_agent_server.py'].replace(' a365_use_s2s_endpoint=True,', ''), + '.env': 'ENABLE_A365_OBSERVABILITY_EXPORTER=true\nA365_USE_S2S_ENDPOINT=true\n', + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, true, r.reason); + } finally { cleanup(dir); } + }); + + test('Python exchange_token with the observability scope → reports delegated telemetry token', () => { + const dir = createFixture({ + ...PYTHON_DISTRO_VALID, + 'host_agent_server.py': `${PYTHON_DISTRO_VALID['host_agent_server.py']} + +async def _legacy(self, context): + token = await self.agent_app.auth.exchange_token( + context, scopes=get_observability_authentication_scope(), auth_handler_id=self.auth_handler_name)`, + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /exchange_token.*delegated \(OBO\) telemetry token/); + } finally { cleanup(dir); } + }); + + test('Python workload exchange_token for MCP scopes is not flagged', () => { + const dir = createFixture({ + ...PYTHON_DISTRO_VALID, + 'mcp_tools.py': `token = await auth.exchange_token(context, scopes=["ea9ffc3e-8a23-4a7d-836d-234d7c7565c1/.default"], auth_handler_id=handler)`, + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, true, r.reason); + } finally { cleanup(dir); } + }); + + test('.NET AgenticTokenStruct built in a variable (A365OtelWrapper shape) → reports delegated telemetry token', () => { + const dir = createFixture({ + ...DOTNET_DISTRO_VALID, + 'A365OtelWrapper.cs': `var agenticToken = new AgenticTokenStruct(userAuthorization: auth, turnContext: turnContext, authHandlerName: name); +agentTokenCache?.RegisterObservability(agentId, tenantId, agenticToken, scopes);`, + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /RegisterObservability.*delegated \(OBO\) telemetry token/); + } finally { cleanup(dir); } + }); + + test('.NET IExporterTokenCache constructor dependency → reports delegated telemetry token', () => { + const dir = createFixture({ + ...DOTNET_DISTRO_VALID, + 'MyAgent.cs': `${DOTNET_DISTRO_VALID['MyAgent.cs']} +public MyAgent(AgentApplicationOptions options, IExporterTokenCache agentTokenCache) : base(options) { }`, + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /delegated \(OBO\) telemetry token/); + } finally { cleanup(dir); } + }); + + test('Node.js tokenResolver reading AgenticTokenCacheInstance.getObservabilityToken → reports delegated telemetry token', () => { + const dir = createFixture({ + ...NODEJS_DISTRO_VALID, + 'src/index.ts': NODEJS_DISTRO_VALID['src/index.ts'].replace('tokenResolver: appTokenResolver', + "tokenResolver: (agentId, tenantId) => AgenticTokenCacheInstance.getObservabilityToken(agentId, tenantId) ?? ''"), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /getObservabilityToken.*delegated \(OBO\) telemetry token/); + } finally { cleanup(dir); } + }); + + test('Python cache_agentic_token + get_cached_agentic_token resolver (no inline scope) → reports delegated telemetry token', () => { + const dir = createFixture({ + ...PYTHON_DISTRO_VALID, + 'host_agent_server.py': PYTHON_DISTRO_VALID['host_agent_server.py'] + .replace('a365_token_resolver=OBS_TOKENS.resolve', 'a365_token_resolver=get_cached_agentic_token') + .replace('await OBS_TOKENS.prefetch(self.connection_manager, tenant_id, agent_id)', + 'token = await self.agent_app.auth.exchange_token(context, auth_handler_id=self.auth_handler_name)\n cache_agentic_token(tenant_id, agent_id, token.token)'), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /cache_agentic_token.*delegated \(OBO\) telemetry token/); + } finally { cleanup(dir); } + }); + + test('Python leftover legacy token_cache.py definition alone is not flagged', () => { + const dir = createFixture({ + ...PYTHON_DISTRO_VALID, + 'token_cache.py': 'def cache_agentic_token(tenant_id, agent_id, token):\n _cache[(tenant_id, agent_id)] = token\n', + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, true, r.reason); + } finally { cleanup(dir); } + }); + + test('Python leftover per-turn cache_agentic_token(...) beside an app-only resolver → reports delegated telemetry token', () => { + const dir = createFixture({ + ...PYTHON_DISTRO_VALID, + 'host_agent_server.py': `${PYTHON_DISTRO_VALID['host_agent_server.py']} + +async def _legacy_cache(self, context, tenant_id, agent_id): + token = await self.agent_app.auth.exchange_token(context, auth_handler_id=self.auth_handler_name) + cache_agentic_token(tenant_id, agent_id, token.token)`, + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /cache_agentic_token.*delegated \(OBO\) telemetry token/); + } finally { cleanup(dir); } + }); + + test('Python lambda a365_token_resolver over get_cached_agentic_token → reports delegated telemetry token', () => { + const dir = createFixture({ + ...PYTHON_DISTRO_VALID, + 'host_agent_server.py': PYTHON_DISTRO_VALID['host_agent_server.py'].replace('a365_token_resolver=OBS_TOKENS.resolve', + 'a365_token_resolver=lambda agent_id, tenant_id: get_cached_agentic_token(tenant_id, agent_id)'), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /delegated a365_token_resolver.*delegated \(OBO\) telemetry token/); + } finally { cleanup(dir); } + }); + + test('Python lambda a365_token_resolver over the app-only resolver is not flagged', () => { + const dir = createFixture({ + ...PYTHON_DISTRO_VALID, + 'host_agent_server.py': PYTHON_DISTRO_VALID['host_agent_server.py'].replace('a365_token_resolver=OBS_TOKENS.resolve', + 'a365_token_resolver=lambda agent_id, tenant_id: OBS_TOKENS.resolve(agent_id, tenant_id)'), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, true, r.reason); + } finally { cleanup(dir); } + }); + + test('Python prefetch(self.connection_manager) without storing it on the host → reports missing connection manager', () => { + const dir = createFixture({ + ...PYTHON_DISTRO_VALID, + 'host_agent_server.py': PYTHON_DISTRO_VALID['host_agent_server.py'] + .replace('self.connection_manager = MsalConnectionManager(**agents_sdk_config)', 'connection_manager = MsalConnectionManager(**agents_sdk_config)') + .replace('CloudAdapter(connection_manager=self.connection_manager)', 'CloudAdapter(connection_manager=connection_manager)'), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /no file assigns self\.connection_manager/); + } finally { cleanup(dir); } + }); +}); + // ── Unknown project ─────────────────────────────────────────────────────────── describe('validate-observability — unknown project', () => { From 3ff7fba78f84b9204293958ecc740bfb9d6cddf7 Mon Sep 17 00:00:00 2001 From: Krishnadheeraj <12496535+DheerajPannala@users.noreply.github.com> Date: Fri, 25 Sep 2026 19:10:29 +0100 Subject: [PATCH 02/12] Require an app-only token resolver alongside the S2S route flag With the unified distro, the token resolver is the only export credential for the S2S route: without one, .NET falls back to the delegated token cache, Node has no token, and the Python exporter drops every span. The route flag alone therefore passed validation for a configuration that cannot export. - validate-instrument-observability: require o.Agent365.TokenResolver / a365 tokenResolver / a365_token_resolver (or the contextual variants) when the distro is used; legacy token-cache helpers stay accepted without the distro. - a365-code-validator (stop hook and standalone scanner): new high findings dotnet-, node-, and python-obs-token-resolver-missing, kept in parity. - Tests for each language, the kwargs-dict resolver form, and scanner parity; docs list the new check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5cbf5f6b-cc40-4b7e-a591-65848db73a12 --- AGENTS.md | 2 +- .../stop/validate-a365-code-validator.js | 25 ++++++++ .../stop/validate-instrument-observability.js | 57 ++++++++++++------- .../skills/a365-code-validator/SKILL.md | 2 + .../references/a365-code-validator.js | 9 +++ .../references/validation-checklist.md | 2 + tests/validate-a365-code-validator.test.js | 24 ++++++++ tests/validate-observability.test.js | 48 ++++++++++++++++ 8 files changed, 147 insertions(+), 22 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 1643d91..7ff5f24 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -254,7 +254,7 @@ Telemetry never goes through the auth handler: every `authMode` exports over the **App-only telemetry token (every auth mode):** For `obo` / `agentic-user`, `instrument-observability` creates an app-only token resolver that reuses the hosting connection's blueprint credential (FMI assertion for the turn's agent identity → agent-identity `client_credentials` for the OBS scope): `Observability/AgentAppTokenResolver.cs` (.NET), `observability/app-token-resolver.ts` (Node.js), or `observability/app_token_resolver.py` (Python — sync `resolve` plus a per-turn async `prefetch`). No per-turn delegated `RegisterObservability(..., AgenticTokenStruct)` / `refreshObservabilityToken(..., authorization)` / `exchange_token(...)` is generated, and existing delegated wiring is migrated. Every mode sets the S2S route flag (`o.Agent365.UseS2SEndpoint = true` — `o.Agent365.Exporter.*` on `Microsoft.OpenTelemetry` 1.0.2 and earlier; `useS2SEndpoint: true`; `a365_use_s2s_endpoint=True`). Registered blueprint agent instances need no `Agent365.Observability.OtelWrite` permission or admin consent (subject to service policy); AI Teammates complete the `OtelWrite` application-role step `a365 setup all --aiteammate` prints. -**S2S scaffold requirement:** When `authMode = s2s`, `instrument-observability` creates a scaffold token-service file per language: .NET creates `Observability/ObservabilityServiceExtensions.cs` + `Observability/ObservabilityTokenService.cs`, Node.js creates `observability/observability-token-service.ts`, Python creates `observability/observability_token_service.py`. Each acquires and refreshes the Observability API token via MSAL client credentials targeting `api://9b975845-388f-4429-889e-eab1ef63949c/.default`. The .NET files additionally provide the `AddAgent365Observability()` / `Agent365ObservabilityContext` DI extensions that replace `AddAgenticTracingExporter()` and per-turn `RegisterObservability()`. The validator (`validate-instrument-observability.js`) accepts either the OBO signal (`BaggageBuilder` / `BaggageTurnMiddleware`) or the S2S signal to pass the context check. When the unified distro is used, it also requires the S2S route flag in every auth mode, and it fails the session on delegated telemetry wiring (`refreshObservabilityToken(..., authorization)`, `RegisterObservability(..., AgenticTokenStruct)`, `exchange_token(...)` for the observability scope). +**S2S scaffold requirement:** When `authMode = s2s`, `instrument-observability` creates a scaffold token-service file per language: .NET creates `Observability/ObservabilityServiceExtensions.cs` + `Observability/ObservabilityTokenService.cs`, Node.js creates `observability/observability-token-service.ts`, Python creates `observability/observability_token_service.py`. Each acquires and refreshes the Observability API token via MSAL client credentials targeting `api://9b975845-388f-4429-889e-eab1ef63949c/.default`. The .NET files additionally provide the `AddAgent365Observability()` / `Agent365ObservabilityContext` DI extensions that replace `AddAgenticTracingExporter()` and per-turn `RegisterObservability()`. The validator (`validate-instrument-observability.js`) accepts either the OBO signal (`BaggageBuilder` / `BaggageTurnMiddleware`) or the S2S signal to pass the context check. When the unified distro is used, it also requires the S2S route flag and a token resolver (`o.Agent365.TokenResolver` / `tokenResolver` / `a365_token_resolver`) in every auth mode, and it fails the session on delegated telemetry wiring (`refreshObservabilityToken(..., authorization)`, `RegisterObservability(..., AgenticTokenStruct)`, `exchange_token(...)` for the observability scope). --- diff --git a/plugins/agent365/hooks/stop/validate-a365-code-validator.js b/plugins/agent365/hooks/stop/validate-a365-code-validator.js index e9b42bb..604f940 100644 --- a/plugins/agent365/hooks/stop/validate-a365-code-validator.js +++ b/plugins/agent365/hooks/stop/validate-a365-code-validator.js @@ -190,6 +190,15 @@ function validatePython() { ); } + if (expectsS2S && !hasExplicitExporterFalse && !anyFileMatches(pyFiles, /\ba365_(?:contextual_)?token_resolver\b['"]?\s*\]?\s*[=:](?!=)/)) { + add( + 'high', + 'python-obs-token-resolver-missing', + 'use_microsoft_opentelemetry() has no a365_token_resolver (or a365_contextual_token_resolver), so the S2S route gets no app-only token and the exporter drops spans. Pass an app-only resolver (instrument-observability app_token_resolver.py for obo / agentic-user, or the S2S token-service cache).', + pyFiles.find(f => fileContains(f, 'use_microsoft_opentelemetry')) + ); + } + for (const file of pyFiles) { const content = read(file); const delegated = findCallBlocks(content, 'exchange_token').some(block => /observability/i.test(block)) || @@ -335,6 +344,14 @@ function validateNode() { tsFiles.find(f => fileContains(f, 'useMicrosoftOpenTelemetry')) ); } + if (hasDistroCall && hasA365Enabled && !hasExporterFalse && !anyFileMatches(tsFiles, /\btokenResolver\b\s*[:=,}](?!=)/)) { + add( + 'high', + 'node-obs-token-resolver-missing', + 'useMicrosoftOpenTelemetry() has no a365 tokenResolver, so the S2S route gets no app-only token. Pass an app-only tokenResolver (instrument-observability app-token-resolver.ts for obo / agentic-user, or the S2S token service).', + tsFiles.find(f => fileContains(f, 'useMicrosoftOpenTelemetry')) + ); + } for (const file of tsFiles) { const content = read(file); const delegatedRefresh = ['refreshObservabilityToken', 'RefreshObservabilityToken'] @@ -419,6 +436,14 @@ function validateDotnet() { csFiles.find(f => fileContains(f, 'UseMicrosoftOpenTelemetry')) ); } + if (anyFileContains(csFiles, 'UseMicrosoftOpenTelemetry') && !anyFileMatches(csFiles, /\b(?:Contextual)?TokenResolver\s*=(?!=)/)) { + add( + 'high', + 'dotnet-obs-token-resolver-missing', + 'UseMicrosoftOpenTelemetry is wired without o.Agent365.TokenResolver, so the S2S route gets no app-only token (the distro default token cache holds delegated tokens). Set TokenResolver to an app-only resolver (instrument-observability AgentAppTokenResolver for obo / agentic-user, or the ServiceTokenCache from ObservabilityTokenService for s2s).', + csFiles.find(f => fileContains(f, 'UseMicrosoftOpenTelemetry')) + ); + } for (const file of csFiles) { const content = read(file); if (findCallBlocks(content, 'RegisterObservability').some(block => block.includes('AgenticTokenStruct')) || diff --git a/plugins/agent365/hooks/stop/validate-instrument-observability.js b/plugins/agent365/hooks/stop/validate-instrument-observability.js index 299d645..6f25c75 100644 --- a/plugins/agent365/hooks/stop/validate-instrument-observability.js +++ b/plugins/agent365/hooks/stop/validate-instrument-observability.js @@ -169,6 +169,9 @@ if (isDotnet) { if (hasDistroWired && !anyFileMatches(csFiles, /\bUseS2SEndpoint\s*=\s*true\b/)) { issues.push('Observability export must use the S2S route in every auth mode: set o.Agent365.UseS2SEndpoint = true in UseMicrosoftOpenTelemetry (o.Agent365.Exporter.UseS2SEndpoint on Microsoft.OpenTelemetry 1.0.2 and earlier) and wire an app-only token resolver (AgentAppTokenResolver, or ObservabilityTokenService for s2s)'); } + if (hasDistroWired && !anyFileMatches(csFiles, /\b(?:Contextual)?TokenResolver\s*=(?!=)/)) { + issues.push('UseMicrosoftOpenTelemetry is wired without o.Agent365.TokenResolver, so the S2S route gets no app-only token (the distro default token cache holds delegated tokens, which the S2S route rejects) — set o.Agent365.TokenResolver to AgentAppTokenResolver.ResolveAsync (obo / agentic-user) or to the ServiceTokenCache fed by ObservabilityTokenService (s2s) (see dotnet-observability.md)'); + } if (anyCallMatches(csFiles, 'RegisterObservability', /AgenticTokenStruct/) || anyFileMatches(csFiles, /\bnew\s+AgenticTokenStruct\s*[({]|IExporterTokenCache\s*<\s*AgenticTokenStruct\s*>\s*\??\s+[A-Za-z_]\w*/)) { issues.push('RegisterObservability(..., AgenticTokenStruct), new AgenticTokenStruct(...), or an IExporterTokenCache dependency wires a delegated (OBO) telemetry token, which the S2S route rejects — remove the per-turn registration and use AgentAppTokenResolver as o.Agent365.TokenResolver (see dotnet-observability.md)'); @@ -236,14 +239,22 @@ if (isNodejs) { } } - // 4. Token caching wired (tokenResolver, AgenticTokenCacheInstance, preloadObservabilityToken helper, or S2S token service) - const hasTokenCache = anyFileContains(tsFiles, 'tokenResolver') || - anyFileContains(tsFiles, 'AgenticTokenCacheInstance') || - anyFileContains(tsFiles, 'RefreshObservabilityToken') || - anyFileContains(tsFiles, 'preloadObservabilityToken') || - anyFileContains(tsFiles, 'getS2SObservabilityToken'); - if (!hasTokenCache) { - issues.push('No TypeScript/JS file wires a token resolver — observability exports will fail'); + // 4. Token resolver wired. With the distro, the a365 tokenResolver is the only export credential + // for the S2S route, so it must be set. Legacy ObservabilityManager wiring may still use the + // older token-cache helpers. + if (usesDistro) { + if (!anyFileMatches(tsFiles, /\btokenResolver\b\s*[:=,}](?!=)/)) { + issues.push('useMicrosoftOpenTelemetry() has no a365 tokenResolver, so the S2S route gets no app-only token and export fails — pass tokenResolver: appTokenResolver from observability/app-token-resolver.ts (obo / agentic-user) or the observability-token-service resolver (s2s) (see nodejs-observability.md)'); + } + } else { + const hasTokenCache = anyFileContains(tsFiles, 'tokenResolver') || + anyFileContains(tsFiles, 'AgenticTokenCacheInstance') || + anyFileContains(tsFiles, 'RefreshObservabilityToken') || + anyFileContains(tsFiles, 'preloadObservabilityToken') || + anyFileContains(tsFiles, 'getS2SObservabilityToken'); + if (!hasTokenCache) { + issues.push('No TypeScript/JS file wires a token resolver — observability exports will fail'); + } } // 4a. S2S scaffold: token service file must exist when authMode is s2s @@ -332,19 +343,23 @@ if (isPython) { } } - // 4. Token cache wired - // OBO path: cache_agentic_token (new pattern) or AgenticTokenCache (legacy) or exchange_token helper - // S2S path: get_s2s_observability_token or token_resolver - // Distro path: use_microsoft_opentelemetry handles it internally - const hasTokenCache = anyFileContains(pyFiles, 'cache_agentic_token') || - anyFileContains(pyFiles, 'exchange_token') || - anyFileContains(pyFiles, 'AgenticTokenCache') || - anyFileContains(pyFiles, 'token_resolver') || - anyFileContains(pyFiles, 'get_observability_authentication_scope') || - anyFileContains(pyFiles, 'get_s2s_observability_token') || - anyFileContains(pyFiles, 'use_microsoft_opentelemetry'); - if (!hasTokenCache) { - issues.push('No Python file wires a token resolver — observability exports will fail'); + // 4. Token resolver wired. With the distro, a365_token_resolver (or a365_contextual_token_resolver) + // is the only export credential for the S2S route; without it the exporter drops every span. + // Legacy configure() wiring may still use the older token-cache helpers. + if (usesDistroPy) { + if (!anyFileMatches(pyFiles, /\ba365_(?:contextual_)?token_resolver\b['"]?\s*\]?\s*[=:](?!=)/)) { + issues.push('use_microsoft_opentelemetry() has no a365_token_resolver, so the S2S route gets no app-only token and the exporter drops every span — pass a365_token_resolver=OBS_TOKENS.resolve from observability/app_token_resolver.py (obo / agentic-user) or the observability_token_service cache (s2s) (see python-observability.md)'); + } + } else { + const hasTokenCache = anyFileContains(pyFiles, 'cache_agentic_token') || + anyFileContains(pyFiles, 'exchange_token') || + anyFileContains(pyFiles, 'AgenticTokenCache') || + anyFileContains(pyFiles, 'token_resolver') || + anyFileContains(pyFiles, 'get_observability_authentication_scope') || + anyFileContains(pyFiles, 'get_s2s_observability_token'); + if (!hasTokenCache) { + issues.push('No Python file wires a token resolver — observability exports will fail'); + } } // 4a. S2S scaffold: token service file must exist when authMode is s2s diff --git a/plugins/agent365/skills/a365-code-validator/SKILL.md b/plugins/agent365/skills/a365-code-validator/SKILL.md index 3346cd6..e44b960 100644 --- a/plugins/agent365/skills/a365-code-validator/SKILL.md +++ b/plugins/agent365/skills/a365-code-validator/SKILL.md @@ -323,6 +323,8 @@ legacy delegated route needs admin consent. Signals are: telemetry token cache, `cache_agentic_token(...)`, or an `a365_token_resolver` backed by `AgenticTokenCache` / `get_cached_agentic_token`. - A distro call without the S2S transport flag. +- A distro call with no token resolver (`tokenResolver`, `a365_token_resolver`, or + `o.Agent365.TokenResolver`). The S2S route then has no app-only token to export with. For Python, also flag (`high`) an app-only `prefetch(self.connection_manager, ...)` when nothing assigns `self.connection_manager`. `CloudAdapter` does not expose its connection manager, so diff --git a/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js b/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js index da04cbd..d053613 100644 --- a/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js +++ b/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js @@ -146,6 +146,9 @@ function validatePython() { } else if (expectsS2S && !s2sTrue && !s2sEnv) { add('high', 'python-s2s-endpoint-not-set', 'Python does not set a365_use_s2s_endpoint=True (or A365_USE_S2S_ENDPOINT=true), so export uses the legacy delegated route. Every auth mode must export over the S2S route with an app-only token.', py.find(f => read(f).includes('use_microsoft_opentelemetry'))); } + if (expectsS2S && !exporterFalse && !anyMatches(py, /\ba365_(?:contextual_)?token_resolver\b['"]?\s*\]?\s*[=:](?!=)/)) { + add('high', 'python-obs-token-resolver-missing', 'use_microsoft_opentelemetry() has no a365_token_resolver (or a365_contextual_token_resolver), so the S2S route gets no app-only token and the exporter drops spans. Pass an app-only resolver (instrument-observability app_token_resolver.py for obo / agentic-user, or the S2S token-service cache).', py.find(f => read(f).includes('use_microsoft_opentelemetry'))); + } for (const file of py) { const content = read(file); const delegated = callBlocks(content, 'exchange_token').some(block => /observability/i.test(block)) || @@ -232,6 +235,9 @@ function validateNode() { if (hasDistro && hasEnabled && !anyMatches(ts, /\buseS2SEndpoint\s*:\s*true\b/)) { add('high', 'node-obs-delegated-route', 'Node code does not set useS2SEndpoint: true, so A365 export uses the legacy delegated route. Every auth mode must export over the S2S route with an app-only tokenResolver.', ts.find(f => read(f).includes('useMicrosoftOpenTelemetry'))); } + if (hasDistro && hasEnabled && !exporterFalse && !anyMatches(ts, /\btokenResolver\b\s*[:=,}](?!=)/)) { + add('high', 'node-obs-token-resolver-missing', 'useMicrosoftOpenTelemetry() has no a365 tokenResolver, so the S2S route gets no app-only token. Pass an app-only tokenResolver (instrument-observability app-token-resolver.ts for obo / agentic-user, or the S2S token service).', ts.find(f => read(f).includes('useMicrosoftOpenTelemetry'))); + } for (const file of ts) { const content = read(file); if (['refreshObservabilityToken', 'RefreshObservabilityToken'].some(name => callBlocks(content, name).some(block => /authorization/i.test(block))) || @@ -272,6 +278,9 @@ function validateDotnet() { if (anyContains(cs, 'UseMicrosoftOpenTelemetry') && !anyMatches(cs, /\bUseS2SEndpoint\s*=\s*true\b/)) { add('high', 'dotnet-obs-delegated-route', 'UseMicrosoftOpenTelemetry is wired without UseS2SEndpoint = true, so A365 export uses the legacy delegated route. Set o.Agent365.UseS2SEndpoint = true (o.Agent365.Exporter.UseS2SEndpoint on Microsoft.OpenTelemetry 1.0.2 and earlier) with an app-only TokenResolver in every auth mode.', cs.find(f => read(f).includes('UseMicrosoftOpenTelemetry'))); } + if (anyContains(cs, 'UseMicrosoftOpenTelemetry') && !anyMatches(cs, /\b(?:Contextual)?TokenResolver\s*=(?!=)/)) { + add('high', 'dotnet-obs-token-resolver-missing', 'UseMicrosoftOpenTelemetry is wired without o.Agent365.TokenResolver, so the S2S route gets no app-only token (the distro default token cache holds delegated tokens). Set TokenResolver to an app-only resolver (instrument-observability AgentAppTokenResolver for obo / agentic-user, or the ServiceTokenCache from ObservabilityTokenService for s2s).', cs.find(f => read(f).includes('UseMicrosoftOpenTelemetry'))); + } for (const file of cs) { const content = read(file); if (callBlocks(content, 'RegisterObservability').some(block => block.includes('AgenticTokenStruct')) || diff --git a/plugins/agent365/skills/a365-code-validator/references/validation-checklist.md b/plugins/agent365/skills/a365-code-validator/references/validation-checklist.md index 785de36..51d1ec7 100644 --- a/plugins/agent365/skills/a365-code-validator/references/validation-checklist.md +++ b/plugins/agent365/skills/a365-code-validator/references/validation-checklist.md @@ -179,6 +179,8 @@ fragile and should be called out. telemetry token cache, `cache_agentic_token(...)`, or an `AgenticTokenCache` / `get_cached_agentic_token` resolver. - A distro call without the S2S transport flag. +- A distro call with no token resolver (`tokenResolver`, `a365_token_resolver`, or + `o.Agent365.TokenResolver`). The S2S route then has no app-only token to export with. Each sends a delegated token (rejected by the S2S route) or uses the legacy delegated route, which needs admin consent. Replace it with the app-only resolver from the `instrument-observability` diff --git a/tests/validate-a365-code-validator.test.js b/tests/validate-a365-code-validator.test.js index ef11edb..170ec69 100644 --- a/tests/validate-a365-code-validator.test.js +++ b/tests/validate-a365-code-validator.test.js @@ -499,6 +499,7 @@ token = await auth.exchange_token(context, scopes=["ea9ffc3e-8a23-4a7d-836d-234d assert.deepEqual(standalone, hook); assert.deepEqual(hook, [ 'dotnet-obs-delegated-token', + 'dotnet-obs-token-resolver-missing', 'node-obs-delegated-token', 'python-obs-delegated-token', 'python-obs-prefetch-connection-missing', @@ -528,6 +529,29 @@ token = await auth.exchange_token(context, scopes=["ea9ffc3e-8a23-4a7d-836d-234d } }); + test('route-only distro configs without a token resolver are flagged in every language by both scanners', () => { + const dir = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': NODE_S2S_INDEX.replace(', tokenResolver: appTokenResolver', ''), + 'Agent.csproj': '', + 'Program.cs': 'builder.UseMicrosoftOpenTelemetry(o => { o.Agent365.UseS2SEndpoint = true; });', + 'requirements.txt': 'microsoft-opentelemetry>=1.1.0\n', + 'host.py': 'use_microsoft_opentelemetry(enable_a365=True, a365_enable_observability_exporter=True, a365_use_s2s_endpoint=True)', + }); + const resolverIds = ids => ids.filter(id => /token-resolver-missing/.test(id)).sort(); + try { + const hook = resolverIds(findingIds(runValidator(VALIDATOR, dir))); + assert.deepEqual(hook, [ + 'dotnet-obs-token-resolver-missing', + 'node-obs-token-resolver-missing', + 'python-obs-token-resolver-missing', + ]); + assert.deepEqual(resolverIds(findingIds(runValidator(STANDALONE, dir))), hook); + } finally { + cleanup(dir); + } + }); + test('app-only S2S wiring in all three languages produces no delegated-telemetry findings in either scanner', () => { const dir = createFixture({ 'package.json': NODE_PKG, diff --git a/tests/validate-observability.test.js b/tests/validate-observability.test.js index d6a1b09..2327f1d 100644 --- a/tests/validate-observability.test.js +++ b/tests/validate-observability.test.js @@ -677,6 +677,54 @@ async def _legacy_cache(self, context, tenant_id, agent_id): assert.match(r.reason, /no file assigns self\.connection_manager/); } finally { cleanup(dir); } }); + + test('.NET distro without o.Agent365.TokenResolver → reports the missing app-only resolver', () => { + const dir = createFixture({ + ...DOTNET_DISTRO_VALID, + 'Program.cs': DOTNET_DISTRO_VALID['Program.cs'].replace(/\n\s*o\.Agent365\.TokenResolver = [^\n]*/, ''), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /without o\.Agent365\.TokenResolver/); + } finally { cleanup(dir); } + }); + + test('Node.js distro without tokenResolver → reports the missing app-only resolver', () => { + const dir = createFixture({ + ...NODEJS_DISTRO_VALID, + 'src/index.ts': NODEJS_DISTRO_VALID['src/index.ts'].replace(', tokenResolver: appTokenResolver', ''), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /no a365 tokenResolver/); + } finally { cleanup(dir); } + }); + + test('Python distro with a365_use_s2s_endpoint=True but no a365_token_resolver → reports the missing app-only resolver', () => { + const dir = createFixture({ + ...PYTHON_DISTRO_VALID, + 'host_agent_server.py': PYTHON_DISTRO_VALID['host_agent_server.py'].replace(', a365_token_resolver=OBS_TOKENS.resolve', ''), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /no a365_token_resolver/); + } finally { cleanup(dir); } + }); + + test('Python distro passing the resolver through a kwargs dict → ok', () => { + const dir = createFixture({ + ...PYTHON_DISTRO_VALID, + 'host_agent_server.py': PYTHON_DISTRO_VALID['host_agent_server.py'].replace(', a365_token_resolver=OBS_TOKENS.resolve)', + ', **{"a365_token_resolver": OBS_TOKENS.resolve})'), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, true, r.reason); + } finally { cleanup(dir); } + }); }); // ── Unknown project ─────────────────────────────────────────────────────────── From 83a9fe1bd77d66012d1d426e9b9133bb46520391 Mon Sep 17 00:00:00 2001 From: Krishnadheeraj <12496535+DheerajPannala@users.noreply.github.com> Date: Mon, 28 Sep 2026 12:02:46 +0100 Subject: [PATCH 03/12] Match the token resolver at the distro call and gate registration on a blueprint Address the second Copilot review on #84: - The resolver check now looks only at the distro call's own arguments (useMicrosoftOpenTelemetry options, use_microsoft_opentelemetry keywords, UseMicrosoftOpenTelemetry options callback). It accepts explicit and shorthand properties and follows a variable passed to the call to its initializer in the same file, so an unused import or a resolver defined elsewhere no longer counts. One helper is shared verbatim by both stop hooks and the standalone scanner, and a test keeps the copies identical. - agent-registration-not-recorded now also requires a blueprint ID in a365.generated.config.json. - has_obs token anchors in shared/agent-detection.md and make-ai-teammate require the resolver to be wired at the distro call. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5cbf5f6b-cc40-4b7e-a591-65848db73a12 --- AGENTS.md | 2 +- .../stop/validate-a365-code-validator.js | 67 ++++++++++- .../stop/validate-instrument-observability.js | 65 ++++++++++- plugins/agent365/shared/agent-detection.md | 8 +- .../skills/a365-code-validator/SKILL.md | 5 +- .../references/a365-code-validator.js | 67 ++++++++++- .../references/validation-checklist.md | 5 +- .../agent365/skills/make-ai-teammate/SKILL.md | 6 +- tests/validate-a365-code-validator.test.js | 69 +++++++++++ tests/validate-observability.test.js | 108 ++++++++++++++++++ 10 files changed, 380 insertions(+), 22 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 7ff5f24..b52e8a8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -254,7 +254,7 @@ Telemetry never goes through the auth handler: every `authMode` exports over the **App-only telemetry token (every auth mode):** For `obo` / `agentic-user`, `instrument-observability` creates an app-only token resolver that reuses the hosting connection's blueprint credential (FMI assertion for the turn's agent identity → agent-identity `client_credentials` for the OBS scope): `Observability/AgentAppTokenResolver.cs` (.NET), `observability/app-token-resolver.ts` (Node.js), or `observability/app_token_resolver.py` (Python — sync `resolve` plus a per-turn async `prefetch`). No per-turn delegated `RegisterObservability(..., AgenticTokenStruct)` / `refreshObservabilityToken(..., authorization)` / `exchange_token(...)` is generated, and existing delegated wiring is migrated. Every mode sets the S2S route flag (`o.Agent365.UseS2SEndpoint = true` — `o.Agent365.Exporter.*` on `Microsoft.OpenTelemetry` 1.0.2 and earlier; `useS2SEndpoint: true`; `a365_use_s2s_endpoint=True`). Registered blueprint agent instances need no `Agent365.Observability.OtelWrite` permission or admin consent (subject to service policy); AI Teammates complete the `OtelWrite` application-role step `a365 setup all --aiteammate` prints. -**S2S scaffold requirement:** When `authMode = s2s`, `instrument-observability` creates a scaffold token-service file per language: .NET creates `Observability/ObservabilityServiceExtensions.cs` + `Observability/ObservabilityTokenService.cs`, Node.js creates `observability/observability-token-service.ts`, Python creates `observability/observability_token_service.py`. Each acquires and refreshes the Observability API token via MSAL client credentials targeting `api://9b975845-388f-4429-889e-eab1ef63949c/.default`. The .NET files additionally provide the `AddAgent365Observability()` / `Agent365ObservabilityContext` DI extensions that replace `AddAgenticTracingExporter()` and per-turn `RegisterObservability()`. The validator (`validate-instrument-observability.js`) accepts either the OBO signal (`BaggageBuilder` / `BaggageTurnMiddleware`) or the S2S signal to pass the context check. When the unified distro is used, it also requires the S2S route flag and a token resolver (`o.Agent365.TokenResolver` / `tokenResolver` / `a365_token_resolver`) in every auth mode, and it fails the session on delegated telemetry wiring (`refreshObservabilityToken(..., authorization)`, `RegisterObservability(..., AgenticTokenStruct)`, `exchange_token(...)` for the observability scope). +**S2S scaffold requirement:** When `authMode = s2s`, `instrument-observability` creates a scaffold token-service file per language: .NET creates `Observability/ObservabilityServiceExtensions.cs` + `Observability/ObservabilityTokenService.cs`, Node.js creates `observability/observability-token-service.ts`, Python creates `observability/observability_token_service.py`. Each acquires and refreshes the Observability API token via MSAL client credentials targeting `api://9b975845-388f-4429-889e-eab1ef63949c/.default`. The .NET files additionally provide the `AddAgent365Observability()` / `Agent365ObservabilityContext` DI extensions that replace `AddAgenticTracingExporter()` and per-turn `RegisterObservability()`. The validator (`validate-instrument-observability.js`) accepts either the OBO signal (`BaggageBuilder` / `BaggageTurnMiddleware`) or the S2S signal to pass the context check. When the unified distro is used, it also requires the S2S route flag and a token resolver passed in the distro call itself (`o.Agent365.TokenResolver` / `tokenResolver` / `a365_token_resolver`) in every auth mode, and it fails the session on delegated telemetry wiring (`refreshObservabilityToken(..., authorization)`, `RegisterObservability(..., AgenticTokenStruct)`, `exchange_token(...)` for the observability scope). --- diff --git a/plugins/agent365/hooks/stop/validate-a365-code-validator.js b/plugins/agent365/hooks/stop/validate-a365-code-validator.js index 604f940..f353666 100644 --- a/plugins/agent365/hooks/stop/validate-a365-code-validator.js +++ b/plugins/agent365/hooks/stop/validate-a365-code-validator.js @@ -109,6 +109,65 @@ function findCallBlocks(content, functionName) { return blocks; } +// A distro call counts as wired only when its own arguments pass the export token resolver: an +// explicit or shorthand `tokenResolver` property (Node.js), an `a365_token_resolver` keyword or +// `**{...}` key (Python), or a `TokenResolver` assignment in the options callback (.NET). A variable +// passed to the call is followed one level to its initializer in the same file; a resolver symbol +// elsewhere, such as an unused import, does not count. +const RESOLVER_WIRING = { + node: { + call: 'useMicrosoftOpenTelemetry', + wired: /[{,]\s*tokenResolver\s*(?=[,}])|\btokenResolver\s*:/, + comments: /\/\*[\s\S]*?\*\/|(^|[\s,{;(])\/\/[^\n]*/gm, + }, + python: { + call: 'use_microsoft_opentelemetry', + wired: /\ba365_(?:contextual_)?token_resolver\s*=(?!=)|['"]a365_(?:contextual_)?token_resolver['"]\s*:/, + comments: /(^|\s)#[^\n]*/gm, + }, + dotnet: { + call: 'UseMicrosoftOpenTelemetry', + wired: /\b(?:Contextual)?TokenResolver\s*=(?!=)/, + comments: /\/\*[\s\S]*?\*\/|(^|[\s,{;(])\/\/[^\n]*/gm, + }, +}; + +// Returns the text from the bracket at `open` through its matching close bracket. +function bracketBlock(content, open) { + const close = { '(': ')', '{': '}' }[content[open]]; + let depth = 0; + for (let i = open; close && i < content.length; i++) { + if (content[i] === content[open]) depth++; + else if (content[i] === close && --depth === 0) return content.slice(open, i + 1); + } + return ''; +} + +function distroCallHasResolver(files, language) { + const { call, wired, comments } = RESOLVER_WIRING[language]; + return files.some(file => { + const content = read(file).replace(comments, '$1'); + for (let index = content.indexOf(`${call}(`); index !== -1; index = content.indexOf(`${call}(`, index + 1)) { + const args = bracketBlock(content, index + call.length); + if (wired.test(args)) return true; + // .NET: a method group instead of an inline lambda configures the options elsewhere in the file. + if (language === 'dotnet' && !args.includes('=>') && wired.test(content)) return true; + for (const name of new Set(args.match(/[A-Za-z_$][\w$]*/g) || [])) { + const escaped = name.replace(/\$/g, '\\$'); + const initializer = new RegExp(`(?:^|[^\\w$])${escaped}\\s*(?::[^=\\n]*)?=\\s*(?:dict\\s*)?([({])`, 'gm'); + for (let m = initializer.exec(content); m; m = initializer.exec(content)) { + if (wired.test(bracketBlock(content, m.index + m[0].length - 1))) return true; + } + if (language === 'python' && + new RegExp(`\\b${escaped}\\s*\\[\\s*['"]a365_(?:contextual_)?token_resolver['"]\\s*\\]\\s*=(?!=)`).test(content)) { + return true; + } + } + } + return false; + }); +} + function validatePython() { const hasMicrosoftOpenTelemetryPackage = reqFiles.some(f => fileContains(f, 'microsoft-opentelemetry')); const hasDistroCall = anyFileContains(pyFiles, 'use_microsoft_opentelemetry'); @@ -190,7 +249,7 @@ function validatePython() { ); } - if (expectsS2S && !hasExplicitExporterFalse && !anyFileMatches(pyFiles, /\ba365_(?:contextual_)?token_resolver\b['"]?\s*\]?\s*[=:](?!=)/)) { + if (expectsS2S && !hasExplicitExporterFalse && !distroCallHasResolver(pyFiles, 'python')) { add( 'high', 'python-obs-token-resolver-missing', @@ -344,7 +403,7 @@ function validateNode() { tsFiles.find(f => fileContains(f, 'useMicrosoftOpenTelemetry')) ); } - if (hasDistroCall && hasA365Enabled && !hasExporterFalse && !anyFileMatches(tsFiles, /\btokenResolver\b\s*[:=,}](?!=)/)) { + if (hasDistroCall && hasA365Enabled && !hasExporterFalse && !distroCallHasResolver(tsFiles, 'node')) { add( 'high', 'node-obs-token-resolver-missing', @@ -436,7 +495,7 @@ function validateDotnet() { csFiles.find(f => fileContains(f, 'UseMicrosoftOpenTelemetry')) ); } - if (anyFileContains(csFiles, 'UseMicrosoftOpenTelemetry') && !anyFileMatches(csFiles, /\b(?:Contextual)?TokenResolver\s*=(?!=)/)) { + if (anyFileContains(csFiles, 'UseMicrosoftOpenTelemetry') && !distroCallHasResolver(csFiles, 'dotnet')) { add( 'high', 'dotnet-obs-token-resolver-missing', @@ -478,7 +537,7 @@ function validateSetupArtifacts() { ); } const staticConfig = readJson(path.join(cwd, 'a365.config.json')); - if (staticConfig && staticConfig.aiTeammate === false && generated.agenticAppId && !generated.agentRegistrationId) { + if (staticConfig && staticConfig.aiTeammate === false && generated.agentBlueprintId && generated.agenticAppId && !generated.agentRegistrationId) { add( 'medium', 'agent-registration-not-recorded', diff --git a/plugins/agent365/hooks/stop/validate-instrument-observability.js b/plugins/agent365/hooks/stop/validate-instrument-observability.js index 6f25c75..5b2e427 100644 --- a/plugins/agent365/hooks/stop/validate-instrument-observability.js +++ b/plugins/agent365/hooks/stop/validate-instrument-observability.js @@ -62,6 +62,65 @@ function anyCallMatches(files, name, regex) { return files.some(f => callBlocks(read(f), name).some(block => regex.test(block))); } +// A distro call counts as wired only when its own arguments pass the export token resolver: an +// explicit or shorthand `tokenResolver` property (Node.js), an `a365_token_resolver` keyword or +// `**{...}` key (Python), or a `TokenResolver` assignment in the options callback (.NET). A variable +// passed to the call is followed one level to its initializer in the same file; a resolver symbol +// elsewhere, such as an unused import, does not count. +const RESOLVER_WIRING = { + node: { + call: 'useMicrosoftOpenTelemetry', + wired: /[{,]\s*tokenResolver\s*(?=[,}])|\btokenResolver\s*:/, + comments: /\/\*[\s\S]*?\*\/|(^|[\s,{;(])\/\/[^\n]*/gm, + }, + python: { + call: 'use_microsoft_opentelemetry', + wired: /\ba365_(?:contextual_)?token_resolver\s*=(?!=)|['"]a365_(?:contextual_)?token_resolver['"]\s*:/, + comments: /(^|\s)#[^\n]*/gm, + }, + dotnet: { + call: 'UseMicrosoftOpenTelemetry', + wired: /\b(?:Contextual)?TokenResolver\s*=(?!=)/, + comments: /\/\*[\s\S]*?\*\/|(^|[\s,{;(])\/\/[^\n]*/gm, + }, +}; + +// Returns the text from the bracket at `open` through its matching close bracket. +function bracketBlock(content, open) { + const close = { '(': ')', '{': '}' }[content[open]]; + let depth = 0; + for (let i = open; close && i < content.length; i++) { + if (content[i] === content[open]) depth++; + else if (content[i] === close && --depth === 0) return content.slice(open, i + 1); + } + return ''; +} + +function distroCallHasResolver(files, language) { + const { call, wired, comments } = RESOLVER_WIRING[language]; + return files.some(file => { + const content = read(file).replace(comments, '$1'); + for (let index = content.indexOf(`${call}(`); index !== -1; index = content.indexOf(`${call}(`, index + 1)) { + const args = bracketBlock(content, index + call.length); + if (wired.test(args)) return true; + // .NET: a method group instead of an inline lambda configures the options elsewhere in the file. + if (language === 'dotnet' && !args.includes('=>') && wired.test(content)) return true; + for (const name of new Set(args.match(/[A-Za-z_$][\w$]*/g) || [])) { + const escaped = name.replace(/\$/g, '\\$'); + const initializer = new RegExp(`(?:^|[^\\w$])${escaped}\\s*(?::[^=\\n]*)?=\\s*(?:dict\\s*)?([({])`, 'gm'); + for (let m = initializer.exec(content); m; m = initializer.exec(content)) { + if (wired.test(bracketBlock(content, m.index + m[0].length - 1))) return true; + } + if (language === 'python' && + new RegExp(`\\b${escaped}\\s*\\[\\s*['"]a365_(?:contextual_)?token_resolver['"]\\s*\\]\\s*=(?!=)`).test(content)) { + return true; + } + } + } + return false; + }); +} + // ── Detect project type ───────────────────────────────────────────────────── // Walk the project tree once, then bucket by name. @@ -169,7 +228,7 @@ if (isDotnet) { if (hasDistroWired && !anyFileMatches(csFiles, /\bUseS2SEndpoint\s*=\s*true\b/)) { issues.push('Observability export must use the S2S route in every auth mode: set o.Agent365.UseS2SEndpoint = true in UseMicrosoftOpenTelemetry (o.Agent365.Exporter.UseS2SEndpoint on Microsoft.OpenTelemetry 1.0.2 and earlier) and wire an app-only token resolver (AgentAppTokenResolver, or ObservabilityTokenService for s2s)'); } - if (hasDistroWired && !anyFileMatches(csFiles, /\b(?:Contextual)?TokenResolver\s*=(?!=)/)) { + if (hasDistroWired && !distroCallHasResolver(csFiles, 'dotnet')) { issues.push('UseMicrosoftOpenTelemetry is wired without o.Agent365.TokenResolver, so the S2S route gets no app-only token (the distro default token cache holds delegated tokens, which the S2S route rejects) — set o.Agent365.TokenResolver to AgentAppTokenResolver.ResolveAsync (obo / agentic-user) or to the ServiceTokenCache fed by ObservabilityTokenService (s2s) (see dotnet-observability.md)'); } if (anyCallMatches(csFiles, 'RegisterObservability', /AgenticTokenStruct/) || @@ -243,7 +302,7 @@ if (isNodejs) { // for the S2S route, so it must be set. Legacy ObservabilityManager wiring may still use the // older token-cache helpers. if (usesDistro) { - if (!anyFileMatches(tsFiles, /\btokenResolver\b\s*[:=,}](?!=)/)) { + if (!distroCallHasResolver(tsFiles, 'node')) { issues.push('useMicrosoftOpenTelemetry() has no a365 tokenResolver, so the S2S route gets no app-only token and export fails — pass tokenResolver: appTokenResolver from observability/app-token-resolver.ts (obo / agentic-user) or the observability-token-service resolver (s2s) (see nodejs-observability.md)'); } } else { @@ -347,7 +406,7 @@ if (isPython) { // is the only export credential for the S2S route; without it the exporter drops every span. // Legacy configure() wiring may still use the older token-cache helpers. if (usesDistroPy) { - if (!anyFileMatches(pyFiles, /\ba365_(?:contextual_)?token_resolver\b['"]?\s*\]?\s*[=:](?!=)/)) { + if (!distroCallHasResolver(pyFiles, 'python')) { issues.push('use_microsoft_opentelemetry() has no a365_token_resolver, so the S2S route gets no app-only token and the exporter drops every span — pass a365_token_resolver=OBS_TOKENS.resolve from observability/app_token_resolver.py (obo / agentic-user) or the observability_token_service cache (s2s) (see python-observability.md)'); } } else { diff --git a/plugins/agent365/shared/agent-detection.md b/plugins/agent365/shared/agent-detection.md index f38c4c1..cad5774 100644 --- a/plugins/agent365/shared/agent-detection.md +++ b/plugins/agent365/shared/agent-detection.md @@ -447,9 +447,11 @@ The cache is written in stages as values become known — always preserve fields - `has_aiteammate_structure`: `1` if any AI Teammate structure signal matches; `0` otherwise: - `AgentApplication` in source files, `CloudAdapter`/`CloudAdapterAiohttp`, `@microsoft/agents-a365-notifications` in `package.json`, `Microsoft.Agents.A365.Notifications` in `.csproj`, or `ToolingManifest.json` exists. - `has_obs`: `1` ONLY when **all four** observability anchors are present (entry-point + app-only token resolver + S2S route flag + handler-side baggage / scope). The entry-point call alone (`UseMicrosoftOpenTelemetry` / `useMicrosoftOpenTelemetry` / `use_microsoft_opentelemetry`) is **not sufficient** — a project can have the call without `tokenResolver` (exports silently fail to auth), without the S2S route flag (telemetry still goes to the legacy delegated route, which rejects app-only tokens), or without `BaggageBuilder` / `InvokeAgentScope` (no identity grouping, fails store-publish validation). Telemetry is exported over the S2S route with an app-only token in **every** auth mode. Anchor sets per language: - - **.NET (`**/*.cs`)**: entry = `UseMicrosoftOpenTelemetry`; token = `AgentAppTokenResolver` (obo / agentic-user) OR `ObservabilityTokenService` / `AddAgent365Observability` (S2S), with no `AgenticTokenStruct` usage (per-turn `RegisterObservability(..., new AgenticTokenStruct(...))`, `new AgenticTokenStruct(...)`, or an `IExporterTokenCache` dependency); route = `UseS2SEndpoint = true`; handler = `BaggageBuilder` OR `BaggageTurnMiddleware` OR `InvokeAgentScope.Start`. - - **Node.js (`src/**/*.ts`)**: entry = `useMicrosoftOpenTelemetry`; token = `tokenResolver` (app-only: `observability/app-token-resolver.ts`, or `getS2SObservabilityToken` / `startTokenService` for S2S), with no `refreshObservabilityToken(..., authorization)` or `AgenticTokenCacheInstance.getObservabilityToken` resolver; route = `useS2SEndpoint: true`; handler = `BaggageBuilder` OR `BaggageBuilderUtils` OR `InvokeAgentScope`. - - **Python (`**/*.py`)**: entry = `use_microsoft_opentelemetry`; token = `token_resolver` (app-only: `AppTokenResolver` / `OBS_TOKENS`, or S2S `run_token_service` / `get_s2s_observability_token`), with no `exchange_token(...)` for the observability scope, `cache_agentic_token(...)`, or `AgenticTokenCache` / `get_cached_agentic_token` resolver; route = `a365_use_s2s_endpoint=True` (or `A365_USE_S2S_ENDPOINT=true` in `.env`); handler = `BaggageBuilder` OR `populate_baggage` OR `InvokeAgentScope`. + - **.NET (`**/*.cs`)**: entry = `UseMicrosoftOpenTelemetry`; token = the `UseMicrosoftOpenTelemetry(...)` options assign `TokenResolver` (`o.Agent365.TokenResolver`; `o.Agent365.Exporter.TokenResolver` on 1.0.2 and earlier) to the app-only `AgentAppTokenResolver` (obo / agentic-user) or the S2S token cache fed by `ObservabilityTokenService` / `AddAgent365Observability`, with no `AgenticTokenStruct` usage (per-turn `RegisterObservability(..., new AgenticTokenStruct(...))`, `new AgenticTokenStruct(...)`, or an `IExporterTokenCache` dependency); route = `UseS2SEndpoint = true`; handler = `BaggageBuilder` OR `BaggageTurnMiddleware` OR `InvokeAgentScope.Start`. + - **Node.js (`src/**/*.ts`)**: entry = `useMicrosoftOpenTelemetry`; token = the `useMicrosoftOpenTelemetry({...})` call passes `tokenResolver` in its `a365` options (app-only: `observability/app-token-resolver.ts`, or `getS2SObservabilityToken` / `startTokenService` for S2S), with no `refreshObservabilityToken(..., authorization)` or `AgenticTokenCacheInstance.getObservabilityToken` resolver; route = `useS2SEndpoint: true`; handler = `BaggageBuilder` OR `BaggageBuilderUtils` OR `InvokeAgentScope`. + - **Python (`**/*.py`)**: entry = `use_microsoft_opentelemetry`; token = the `use_microsoft_opentelemetry(...)` call passes `a365_token_resolver=` (app-only: `OBS_TOKENS.resolve` from `AppTokenResolver`, or the S2S `run_token_service` / `get_s2s_observability_token` cache), with no `exchange_token(...)` for the observability scope, `cache_agentic_token(...)`, or `AgenticTokenCache` / `get_cached_agentic_token` resolver; route = `a365_use_s2s_endpoint=True` (or `A365_USE_S2S_ENDPOINT=true` in `.env`); handler = `BaggageBuilder` OR `populate_baggage` OR `InvokeAgentScope`. + + The token anchor must be wired at the distro call itself. A variable passed to the call counts when its initializer sets the resolver, but a resolver symbol or scaffold elsewhere does not: an unused import, a `tokenResolver` in another file, or an `ObservabilityTokenService` that is never assigned to `TokenResolver` leaves export without a credential. Package-name matches alone (`Microsoft.Agents.A365.Observability.*`, `@microsoft/agents-a365-observability`, `microsoft-agents-a365-observability-*`, `Microsoft.OpenTelemetry` / `@microsoft/opentelemetry` / `microsoft-opentelemetry`) do NOT count — they reflect installation, not invocation. diff --git a/plugins/agent365/skills/a365-code-validator/SKILL.md b/plugins/agent365/skills/a365-code-validator/SKILL.md index e44b960..c05dbb9 100644 --- a/plugins/agent365/skills/a365-code-validator/SKILL.md +++ b/plugins/agent365/skills/a365-code-validator/SKILL.md @@ -323,8 +323,9 @@ legacy delegated route needs admin consent. Signals are: telemetry token cache, `cache_agentic_token(...)`, or an `a365_token_resolver` backed by `AgenticTokenCache` / `get_cached_agentic_token`. - A distro call without the S2S transport flag. -- A distro call with no token resolver (`tokenResolver`, `a365_token_resolver`, or - `o.Agent365.TokenResolver`). The S2S route then has no app-only token to export with. +- A distro call whose own options pass no token resolver (`tokenResolver`, `a365_token_resolver`, + or `o.Agent365.TokenResolver`). The S2S route then has no app-only token to export with. A + resolver symbol elsewhere, such as an unused import, does not count. For Python, also flag (`high`) an app-only `prefetch(self.connection_manager, ...)` when nothing assigns `self.connection_manager`. `CloudAdapter` does not expose its connection manager, so diff --git a/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js b/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js index d053613..7a1bca4 100644 --- a/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js +++ b/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js @@ -107,6 +107,65 @@ function callBlocks(content, name) { return blocks; } +// A distro call counts as wired only when its own arguments pass the export token resolver: an +// explicit or shorthand `tokenResolver` property (Node.js), an `a365_token_resolver` keyword or +// `**{...}` key (Python), or a `TokenResolver` assignment in the options callback (.NET). A variable +// passed to the call is followed one level to its initializer in the same file; a resolver symbol +// elsewhere, such as an unused import, does not count. +const RESOLVER_WIRING = { + node: { + call: 'useMicrosoftOpenTelemetry', + wired: /[{,]\s*tokenResolver\s*(?=[,}])|\btokenResolver\s*:/, + comments: /\/\*[\s\S]*?\*\/|(^|[\s,{;(])\/\/[^\n]*/gm, + }, + python: { + call: 'use_microsoft_opentelemetry', + wired: /\ba365_(?:contextual_)?token_resolver\s*=(?!=)|['"]a365_(?:contextual_)?token_resolver['"]\s*:/, + comments: /(^|\s)#[^\n]*/gm, + }, + dotnet: { + call: 'UseMicrosoftOpenTelemetry', + wired: /\b(?:Contextual)?TokenResolver\s*=(?!=)/, + comments: /\/\*[\s\S]*?\*\/|(^|[\s,{;(])\/\/[^\n]*/gm, + }, +}; + +// Returns the text from the bracket at `open` through its matching close bracket. +function bracketBlock(content, open) { + const close = { '(': ')', '{': '}' }[content[open]]; + let depth = 0; + for (let i = open; close && i < content.length; i++) { + if (content[i] === content[open]) depth++; + else if (content[i] === close && --depth === 0) return content.slice(open, i + 1); + } + return ''; +} + +function distroCallHasResolver(files, language) { + const { call, wired, comments } = RESOLVER_WIRING[language]; + return files.some(file => { + const content = read(file).replace(comments, '$1'); + for (let index = content.indexOf(`${call}(`); index !== -1; index = content.indexOf(`${call}(`, index + 1)) { + const args = bracketBlock(content, index + call.length); + if (wired.test(args)) return true; + // .NET: a method group instead of an inline lambda configures the options elsewhere in the file. + if (language === 'dotnet' && !args.includes('=>') && wired.test(content)) return true; + for (const name of new Set(args.match(/[A-Za-z_$][\w$]*/g) || [])) { + const escaped = name.replace(/\$/g, '\\$'); + const initializer = new RegExp(`(?:^|[^\\w$])${escaped}\\s*(?::[^=\\n]*)?=\\s*(?:dict\\s*)?([({])`, 'gm'); + for (let m = initializer.exec(content); m; m = initializer.exec(content)) { + if (wired.test(bracketBlock(content, m.index + m[0].length - 1))) return true; + } + if (language === 'python' && + new RegExp(`\\b${escaped}\\s*\\[\\s*['"]a365_(?:contextual_)?token_resolver['"]\\s*\\]\\s*=(?!=)`).test(content)) { + return true; + } + } + } + return false; + }); +} + function validatePython() { const hasPackage = req.some(f => read(f).includes('microsoft-opentelemetry')); const hasDistro = anyContains(py, 'use_microsoft_opentelemetry'); @@ -146,7 +205,7 @@ function validatePython() { } else if (expectsS2S && !s2sTrue && !s2sEnv) { add('high', 'python-s2s-endpoint-not-set', 'Python does not set a365_use_s2s_endpoint=True (or A365_USE_S2S_ENDPOINT=true), so export uses the legacy delegated route. Every auth mode must export over the S2S route with an app-only token.', py.find(f => read(f).includes('use_microsoft_opentelemetry'))); } - if (expectsS2S && !exporterFalse && !anyMatches(py, /\ba365_(?:contextual_)?token_resolver\b['"]?\s*\]?\s*[=:](?!=)/)) { + if (expectsS2S && !exporterFalse && !distroCallHasResolver(py, 'python')) { add('high', 'python-obs-token-resolver-missing', 'use_microsoft_opentelemetry() has no a365_token_resolver (or a365_contextual_token_resolver), so the S2S route gets no app-only token and the exporter drops spans. Pass an app-only resolver (instrument-observability app_token_resolver.py for obo / agentic-user, or the S2S token-service cache).', py.find(f => read(f).includes('use_microsoft_opentelemetry'))); } for (const file of py) { @@ -235,7 +294,7 @@ function validateNode() { if (hasDistro && hasEnabled && !anyMatches(ts, /\buseS2SEndpoint\s*:\s*true\b/)) { add('high', 'node-obs-delegated-route', 'Node code does not set useS2SEndpoint: true, so A365 export uses the legacy delegated route. Every auth mode must export over the S2S route with an app-only tokenResolver.', ts.find(f => read(f).includes('useMicrosoftOpenTelemetry'))); } - if (hasDistro && hasEnabled && !exporterFalse && !anyMatches(ts, /\btokenResolver\b\s*[:=,}](?!=)/)) { + if (hasDistro && hasEnabled && !exporterFalse && !distroCallHasResolver(ts, 'node')) { add('high', 'node-obs-token-resolver-missing', 'useMicrosoftOpenTelemetry() has no a365 tokenResolver, so the S2S route gets no app-only token. Pass an app-only tokenResolver (instrument-observability app-token-resolver.ts for obo / agentic-user, or the S2S token service).', ts.find(f => read(f).includes('useMicrosoftOpenTelemetry'))); } for (const file of ts) { @@ -278,7 +337,7 @@ function validateDotnet() { if (anyContains(cs, 'UseMicrosoftOpenTelemetry') && !anyMatches(cs, /\bUseS2SEndpoint\s*=\s*true\b/)) { add('high', 'dotnet-obs-delegated-route', 'UseMicrosoftOpenTelemetry is wired without UseS2SEndpoint = true, so A365 export uses the legacy delegated route. Set o.Agent365.UseS2SEndpoint = true (o.Agent365.Exporter.UseS2SEndpoint on Microsoft.OpenTelemetry 1.0.2 and earlier) with an app-only TokenResolver in every auth mode.', cs.find(f => read(f).includes('UseMicrosoftOpenTelemetry'))); } - if (anyContains(cs, 'UseMicrosoftOpenTelemetry') && !anyMatches(cs, /\b(?:Contextual)?TokenResolver\s*=(?!=)/)) { + if (anyContains(cs, 'UseMicrosoftOpenTelemetry') && !distroCallHasResolver(cs, 'dotnet')) { add('high', 'dotnet-obs-token-resolver-missing', 'UseMicrosoftOpenTelemetry is wired without o.Agent365.TokenResolver, so the S2S route gets no app-only token (the distro default token cache holds delegated tokens). Set TokenResolver to an app-only resolver (instrument-observability AgentAppTokenResolver for obo / agentic-user, or the ServiceTokenCache from ObservabilityTokenService for s2s).', cs.find(f => read(f).includes('UseMicrosoftOpenTelemetry'))); } for (const file of cs) { @@ -300,7 +359,7 @@ function validateSetupArtifacts() { add('critical', 'blueprint-id-used-as-agent-id', 'a365.generated.config.json has identical blueprint and agentic app IDs. Verify runtime gen_ai.agent.id uses the agent instance/source agent ID, not the blueprint ID.', path.join(cwd, 'a365.generated.config.json')); } const staticConfig = readJsonSafe(path.join(cwd, 'a365.config.json')); - if (staticConfig && staticConfig.aiTeammate === false && generated.agenticAppId && !generated.agentRegistrationId) { + if (staticConfig && staticConfig.aiTeammate === false && generated.agentBlueprintId && generated.agenticAppId && !generated.agentRegistrationId) { add('medium', 'agent-registration-not-recorded', 'a365.generated.config.json has an agent identity but no agentRegistrationId. The S2S route authorizes registered agent instances without an OtelWrite grant; an unregistered instance gets 403 insufficient_scope. Run a365 setup all --agent-registration-only (idempotent).', path.join(cwd, 'a365.generated.config.json')); } } diff --git a/plugins/agent365/skills/a365-code-validator/references/validation-checklist.md b/plugins/agent365/skills/a365-code-validator/references/validation-checklist.md index 51d1ec7..590c5ed 100644 --- a/plugins/agent365/skills/a365-code-validator/references/validation-checklist.md +++ b/plugins/agent365/skills/a365-code-validator/references/validation-checklist.md @@ -179,8 +179,9 @@ fragile and should be called out. telemetry token cache, `cache_agentic_token(...)`, or an `AgenticTokenCache` / `get_cached_agentic_token` resolver. - A distro call without the S2S transport flag. -- A distro call with no token resolver (`tokenResolver`, `a365_token_resolver`, or - `o.Agent365.TokenResolver`). The S2S route then has no app-only token to export with. +- A distro call whose own options pass no token resolver (`tokenResolver`, `a365_token_resolver`, + or `o.Agent365.TokenResolver`). The S2S route then has no app-only token to export with. A + resolver symbol elsewhere, such as an unused import, does not count. Each sends a delegated token (rejected by the S2S route) or uses the legacy delegated route, which needs admin consent. Replace it with the app-only resolver from the `instrument-observability` diff --git a/plugins/agent365/skills/make-ai-teammate/SKILL.md b/plugins/agent365/skills/make-ai-teammate/SKILL.md index 0d03cb8..dba8ec5 100644 --- a/plugins/agent365/skills/make-ai-teammate/SKILL.md +++ b/plugins/agent365/skills/make-ai-teammate/SKILL.md @@ -329,7 +329,7 @@ Ask: "What language and framework are you using?" and set `language` and `agentS - `ToolingManifest.json` exists → `hasManifest` - **Observability composite** — compute four sub-signals, then combine: - `obs_entry` = `useMicrosoftOpenTelemetry` in any `src/**/*.ts` - - `obs_token` = `tokenResolver` in any `src/**/*.ts` (the app-only resolver `observability/app-token-resolver.ts` for obo / agentic-user; S2S also accepts `getS2SObservabilityToken` / `startTokenService`), AND no `refreshObservabilityToken(..., authorization)` call or `AgenticTokenCacheInstance.getObservabilityToken` resolver (either means the telemetry token is delegated) + - `obs_token` = the `useMicrosoftOpenTelemetry({...})` call passes `tokenResolver` in its `a365` options (the app-only resolver `observability/app-token-resolver.ts` for obo / agentic-user; S2S also accepts `getS2SObservabilityToken` / `startTokenService`). A `tokenResolver` symbol elsewhere, such as an unused import, does not count. AND no `refreshObservabilityToken(..., authorization)` call or `AgenticTokenCacheInstance.getObservabilityToken` resolver (either means the telemetry token is delegated) - `obs_route` = `useS2SEndpoint: true` in any `src/**/*.ts` (telemetry uses the S2S route in every auth mode) - `obs_handler` = `BaggageBuilder` OR `BaggageBuilderUtils` OR `InvokeAgentScope` in any `src/**/*.ts` - `has_obs_complete` = `obs_entry && obs_token && obs_route && obs_handler` @@ -343,7 +343,7 @@ Ask: "What language and framework are you using?" and set `language` and `agentS - `ToolingManifest.json` exists → `hasManifest` - **Observability composite:** - `obs_entry` = `UseMicrosoftOpenTelemetry` in `Program.cs` (or legacy `AddA365Tracing`) - - `obs_token` = `AgentAppTokenResolver` (obo / agentic-user) or `ObservabilityTokenService` / `AddAgent365Observability` (S2S), AND no `AgenticTokenStruct` usage (`RegisterObservability(..., new AgenticTokenStruct(...))`, `new AgenticTokenStruct(...)`, or an `IExporterTokenCache` dependency — all register a delegated telemetry token) + - `obs_token` = the `UseMicrosoftOpenTelemetry(...)` options assign `o.Agent365.TokenResolver` (`o.Agent365.Exporter.TokenResolver` on 1.0.2 and earlier) to `AgentAppTokenResolver` (obo / agentic-user) or to the token cache fed by `ObservabilityTokenService` / `AddAgent365Observability` (S2S). A scaffold file that is never assigned does not count. AND no `AgenticTokenStruct` usage (`RegisterObservability(..., new AgenticTokenStruct(...))`, `new AgenticTokenStruct(...)`, or an `IExporterTokenCache` dependency — all register a delegated telemetry token) - `obs_route` = `UseS2SEndpoint = true` in any `**/*.cs` (telemetry uses the S2S route in every auth mode) - `obs_handler` = `BaggageBuilder` OR `BaggageTurnMiddleware` OR `InvokeAgentScope.Start` in `**/*.cs` - `has_obs_complete` = `obs_entry && obs_token && obs_route && obs_handler` @@ -357,7 +357,7 @@ Ask: "What language and framework are you using?" and set `language` and `agentS - `ToolingManifest.json` exists → `hasManifest` - **Observability composite:** - `obs_entry` = `use_microsoft_opentelemetry` in any `**/*.py` - - `obs_token` = `token_resolver` (the app-only `AppTokenResolver` / `OBS_TOKENS` for obo / agentic-user; S2S: `run_token_service` / `get_s2s_observability_token`), AND no `exchange_token(...)` for the observability scope, `cache_agentic_token(...)`, or `AgenticTokenCache` / `get_cached_agentic_token` resolver (all are delegated telemetry tokens) + - `obs_token` = the `use_microsoft_opentelemetry(...)` call passes `a365_token_resolver=` (the app-only `OBS_TOKENS.resolve` for obo / agentic-user; S2S: the `run_token_service` / `get_s2s_observability_token` cache). A resolver defined but not passed does not count. AND no `exchange_token(...)` for the observability scope, `cache_agentic_token(...)`, or `AgenticTokenCache` / `get_cached_agentic_token` resolver (all are delegated telemetry tokens) - `obs_route` = `a365_use_s2s_endpoint=True` in any `**/*.py` (or `A365_USE_S2S_ENDPOINT=true` in `.env`) — telemetry uses the S2S route in every auth mode - `obs_handler` = `BaggageBuilder` OR `populate_baggage` OR `InvokeAgentScope` in any `**/*.py` - `has_obs_complete` = `obs_entry && obs_token && obs_route && obs_handler` diff --git a/tests/validate-a365-code-validator.test.js b/tests/validate-a365-code-validator.test.js index 170ec69..2133ef3 100644 --- a/tests/validate-a365-code-validator.test.js +++ b/tests/validate-a365-code-validator.test.js @@ -4,6 +4,7 @@ const { test, describe } = require('node:test'); const assert = require('node:assert/strict'); +const fs = require('fs'); const path = require('path'); const { createFixture, runValidator, cleanup } = require('./helpers'); @@ -431,16 +432,26 @@ token = await auth.exchange_token(context, scopes=["ea9ffc3e-8a23-4a7d-836d-234d 'a365.config.json': JSON.stringify({ aiTeammate: true }), 'a365.generated.config.json': JSON.stringify({ agentBlueprintId: blueprintId, agenticAppId: agentId }), }); + const noBlueprint = createFixture({ + 'a365.config.json': JSON.stringify({ aiTeammate: false }), + 'a365.generated.config.json': JSON.stringify({ agenticAppId: agentId }), + }); try { const finding = runValidator(VALIDATOR, unregistered).findings.find(f => f.id === 'agent-registration-not-recorded'); assert.ok(finding, 'expected agent-registration-not-recorded'); assert.equal(finding.severity, 'medium'); assert.ok(!findingIds(runValidator(VALIDATOR, registered)).includes('agent-registration-not-recorded')); assert.ok(!findingIds(runValidator(VALIDATOR, aiTeammate)).includes('agent-registration-not-recorded')); + for (const scanner of [VALIDATOR, STANDALONE]) { + assert.ok(findingIds(runValidator(scanner, unregistered)).includes('agent-registration-not-recorded'), scanner); + assert.ok(!findingIds(runValidator(scanner, noBlueprint)).includes('agent-registration-not-recorded'), + `${scanner}: without a blueprint ID there is no blueprint agent instance to register`); + } } finally { cleanup(unregistered); cleanup(registered); cleanup(aiTeammate); + cleanup(noBlueprint); } }); @@ -552,6 +563,64 @@ token = await auth.exchange_token(context, scopes=["ea9ffc3e-8a23-4a7d-836d-234d } }); + test('resolver symbols outside the distro call do not count in either scanner; options passed by variable do', () => { + const csproj = ''; + const unwired = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': `import { tokenResolver } from './workload-token-resolver';\n${NODE_S2S_INDEX.replace(', tokenResolver: appTokenResolver', '')}`, + 'Agent.csproj': csproj, + 'Program.cs': 'builder.UseMicrosoftOpenTelemetry(o => { o.Agent365.UseS2SEndpoint = true; });', + 'WorkloadTokens.cs': 'workloadOptions.TokenResolver = workloadTokens.ResolveAsync;', + 'requirements.txt': 'microsoft-opentelemetry>=1.1.0\n', + 'host.py': 'use_microsoft_opentelemetry(enable_a365=True, a365_enable_observability_exporter=True, a365_use_s2s_endpoint=True)\na365_token_resolver = OBS_TOKENS.resolve', + }); + const byVariable = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': [ + 'const otelOptions = { a365: { enabled: true, enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver: appTokenResolver } };', + 'useMicrosoftOpenTelemetry(otelOptions);', + ].join('\n'), + 'Agent.csproj': csproj, + 'Program.cs': [ + 'builder.UseMicrosoftOpenTelemetry(ConfigureTelemetry);', + 'static void ConfigureTelemetry(MicrosoftOpenTelemetryOptions o) { o.Agent365.UseS2SEndpoint = true; o.Agent365.TokenResolver = (a, t) => r.ResolveAsync(a, t); }', + ].join('\n'), + 'requirements.txt': 'microsoft-opentelemetry>=1.1.0\n', + 'host.py': [ + 'OTEL_KWARGS = dict(enable_a365=True, a365_enable_observability_exporter=True, a365_use_s2s_endpoint=True, a365_token_resolver=OBS_TOKENS.resolve)', + 'use_microsoft_opentelemetry(**OTEL_KWARGS)', + ].join('\n'), + }); + const resolverIds = ids => ids.filter(id => /token-resolver-missing/.test(id)).sort(); + try { + for (const scanner of [VALIDATOR, STANDALONE]) { + assert.deepEqual(resolverIds(findingIds(runValidator(scanner, unwired))), [ + 'dotnet-obs-token-resolver-missing', + 'node-obs-token-resolver-missing', + 'python-obs-token-resolver-missing', + ], scanner); + assert.deepEqual(resolverIds(findingIds(runValidator(scanner, byVariable))), [], scanner); + } + } finally { + cleanup(unwired); + cleanup(byVariable); + } + }); + + test('the stop hooks and the standalone scanner share one distro resolver helper', () => { + const observabilityHook = path.join(__dirname, '../plugins/agent365/hooks/stop/validate-instrument-observability.js'); + const helperText = file => { + const text = fs.readFileSync(file, 'utf8').replace(/\r\n/g, '\n'); + const start = text.indexOf('const RESOLVER_WIRING = {'); + const end = text.indexOf('\n}\n', text.indexOf('function distroCallHasResolver(')); + assert.ok(start >= 0 && end > start, `${file}: resolver helper not found`); + return text.slice(start, end); + }; + const expected = helperText(VALIDATOR); + assert.equal(helperText(STANDALONE), expected, 'standalone scanner must use the same resolver helper'); + assert.equal(helperText(observabilityHook), expected, 'instrument-observability hook must use the same resolver helper'); + }); + test('app-only S2S wiring in all three languages produces no delegated-telemetry findings in either scanner', () => { const dir = createFixture({ 'package.json': NODE_PKG, diff --git a/tests/validate-observability.test.js b/tests/validate-observability.test.js index 2327f1d..9265f59 100644 --- a/tests/validate-observability.test.js +++ b/tests/validate-observability.test.js @@ -714,6 +714,114 @@ async def _legacy_cache(self, context, tenant_id, agent_id): } finally { cleanup(dir); } }); + test('Node.js unused tokenResolver import with no resolver in the distro call → reports the missing app-only resolver', () => { + const dir = createFixture({ + ...NODEJS_DISTRO_VALID, + 'src/index.ts': `import { tokenResolver } from './workload-token-resolver';\n${NODEJS_DISTRO_VALID['src/index.ts'].replace(', tokenResolver: appTokenResolver', '')}`, + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /no a365 tokenResolver/); + } finally { cleanup(dir); } + }); + + test('Node.js options object passed to the distro call by variable → ok', () => { + const dir = createFixture({ + ...NODEJS_DISTRO_VALID, + 'src/index.ts': ` +import { useMicrosoftOpenTelemetry } from '@microsoft/opentelemetry'; +import { createAppTokenResolver } from './observability/app-token-resolver'; +const appTokenResolver = createAppTokenResolver(() => getObsConnection()); +const otelOptions: MicrosoftOpenTelemetryOptions = { + a365: { enabled: true, enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver: appTokenResolver }, +}; +useMicrosoftOpenTelemetry(otelOptions); + `.trim(), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, true, r.reason); + } finally { cleanup(dir); } + }); + + test('Node.js shorthand tokenResolver property in the distro call → ok', () => { + const dir = createFixture({ + ...NODEJS_DISTRO_VALID, + 'src/index.ts': NODEJS_DISTRO_VALID['src/index.ts'] + .replace('const appTokenResolver =', 'const tokenResolver =') + .replace('tokenResolver: appTokenResolver', 'tokenResolver'), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, true, r.reason); + } finally { cleanup(dir); } + }); + + test('Python resolver assigned but not passed to the distro call → reports the missing app-only resolver', () => { + const dir = createFixture({ + ...PYTHON_DISTRO_VALID, + 'host_agent_server.py': PYTHON_DISTRO_VALID['host_agent_server.py'] + .replace(', a365_token_resolver=OBS_TOKENS.resolve)', ')\na365_token_resolver = OBS_TOKENS.resolve'), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /no a365_token_resolver/); + } finally { cleanup(dir); } + }); + + test('Python keyword arguments passed with ** from a dict(...) → ok', () => { + const dir = createFixture({ + ...PYTHON_DISTRO_VALID, + 'host_agent_server.py': PYTHON_DISTRO_VALID['host_agent_server.py'].replace( + 'use_microsoft_opentelemetry(enable_a365=True, a365_enable_observability_exporter=True, a365_use_s2s_endpoint=True, a365_token_resolver=OBS_TOKENS.resolve)', + 'OTEL_KWARGS = dict(enable_a365=True, a365_enable_observability_exporter=True, a365_use_s2s_endpoint=True, a365_token_resolver=OBS_TOKENS.resolve)\nuse_microsoft_opentelemetry(**OTEL_KWARGS)'), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, true, r.reason); + } finally { cleanup(dir); } + }); + + test('Node.js resolver commented out inside the distro call → reports the missing app-only resolver', () => { + const dir = createFixture({ + ...NODEJS_DISTRO_VALID, + 'src/index.ts': NODEJS_DISTRO_VALID['src/index.ts'].replace(', tokenResolver: appTokenResolver', ',\n // tokenResolver: appTokenResolver,\n'), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /no a365 tokenResolver/); + } finally { cleanup(dir); } + }); + + test('Python resolver added to a kwargs dict by key before the distro call → ok', () => { + const dir = createFixture({ + ...PYTHON_DISTRO_VALID, + 'host_agent_server.py': PYTHON_DISTRO_VALID['host_agent_server.py'].replace( + 'use_microsoft_opentelemetry(enable_a365=True, a365_enable_observability_exporter=True, a365_use_s2s_endpoint=True, a365_token_resolver=OBS_TOKENS.resolve)', + 'OTEL_KWARGS = dict(enable_a365=True, a365_enable_observability_exporter=True, a365_use_s2s_endpoint=True)\nOTEL_KWARGS["a365_token_resolver"] = OBS_TOKENS.resolve\nuse_microsoft_opentelemetry(**OTEL_KWARGS)'), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, true, r.reason); + } finally { cleanup(dir); } + }); + + test('.NET TokenResolver assigned outside the UseMicrosoftOpenTelemetry options → reports the missing app-only resolver', () => { + const dir = createFixture({ + ...DOTNET_DISTRO_VALID, + 'Program.cs': DOTNET_DISTRO_VALID['Program.cs'].replace(/\n\s*o\.Agent365\.TokenResolver = [^\n]*/, ''), + 'WorkloadTokens.cs': 'workloadOptions.TokenResolver = workloadTokens.ResolveAsync;', + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /without o\.Agent365\.TokenResolver/); + } finally { cleanup(dir); } + }); + test('Python distro passing the resolver through a kwargs dict → ok', () => { const dir = createFixture({ ...PYTHON_DISTRO_VALID, From b7e93c37216d48588d2336efe5d12d44d6667593 Mon Sep 17 00:00:00 2001 From: Krishnadheeraj <12496535+DheerajPannala@users.noreply.github.com> Date: Mon, 28 Sep 2026 12:26:42 +0100 Subject: [PATCH 04/12] Scope S2S route checks to the distro call and ignore comments Address the "previously missed" items in the third Copilot review on #84: - The S2S route flag now counts only when the distro call itself passes it, using the same call-site matching as the resolver check. An unrelated `useS2SEndpoint: true` object, a `UseS2SEndpoint = true` on other options, or an `a365_use_s2s_endpoint=True` on another call no longer satisfies it. The Python env fallback is unchanged. - The delegated-token and prefetch scans ignore comments, so a comment that documents the legacy call is not reported. - One shared helper (DISTRO_CALLS / distroCallMatches / codeMatches) is kept identical across both stop hooks and the standalone scanner. - Detection docs: the route anchor must also be set at the distro call. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5cbf5f6b-cc40-4b7e-a591-65848db73a12 --- AGENTS.md | 2 +- .../stop/validate-a365-code-validator.js | 119 +++++++++----- .../stop/validate-instrument-observability.js | 147 ++++++++++-------- plugins/agent365/shared/agent-detection.md | 8 +- .../references/a365-code-validator.js | 119 +++++++++----- .../agent365/skills/make-ai-teammate/SKILL.md | 6 +- tests/validate-a365-code-validator.test.js | 54 ++++++- tests/validate-observability.test.js | 70 +++++++++ 8 files changed, 356 insertions(+), 169 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index b52e8a8..2f37f67 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -254,7 +254,7 @@ Telemetry never goes through the auth handler: every `authMode` exports over the **App-only telemetry token (every auth mode):** For `obo` / `agentic-user`, `instrument-observability` creates an app-only token resolver that reuses the hosting connection's blueprint credential (FMI assertion for the turn's agent identity → agent-identity `client_credentials` for the OBS scope): `Observability/AgentAppTokenResolver.cs` (.NET), `observability/app-token-resolver.ts` (Node.js), or `observability/app_token_resolver.py` (Python — sync `resolve` plus a per-turn async `prefetch`). No per-turn delegated `RegisterObservability(..., AgenticTokenStruct)` / `refreshObservabilityToken(..., authorization)` / `exchange_token(...)` is generated, and existing delegated wiring is migrated. Every mode sets the S2S route flag (`o.Agent365.UseS2SEndpoint = true` — `o.Agent365.Exporter.*` on `Microsoft.OpenTelemetry` 1.0.2 and earlier; `useS2SEndpoint: true`; `a365_use_s2s_endpoint=True`). Registered blueprint agent instances need no `Agent365.Observability.OtelWrite` permission or admin consent (subject to service policy); AI Teammates complete the `OtelWrite` application-role step `a365 setup all --aiteammate` prints. -**S2S scaffold requirement:** When `authMode = s2s`, `instrument-observability` creates a scaffold token-service file per language: .NET creates `Observability/ObservabilityServiceExtensions.cs` + `Observability/ObservabilityTokenService.cs`, Node.js creates `observability/observability-token-service.ts`, Python creates `observability/observability_token_service.py`. Each acquires and refreshes the Observability API token via MSAL client credentials targeting `api://9b975845-388f-4429-889e-eab1ef63949c/.default`. The .NET files additionally provide the `AddAgent365Observability()` / `Agent365ObservabilityContext` DI extensions that replace `AddAgenticTracingExporter()` and per-turn `RegisterObservability()`. The validator (`validate-instrument-observability.js`) accepts either the OBO signal (`BaggageBuilder` / `BaggageTurnMiddleware`) or the S2S signal to pass the context check. When the unified distro is used, it also requires the S2S route flag and a token resolver passed in the distro call itself (`o.Agent365.TokenResolver` / `tokenResolver` / `a365_token_resolver`) in every auth mode, and it fails the session on delegated telemetry wiring (`refreshObservabilityToken(..., authorization)`, `RegisterObservability(..., AgenticTokenStruct)`, `exchange_token(...)` for the observability scope). +**S2S scaffold requirement:** When `authMode = s2s`, `instrument-observability` creates a scaffold token-service file per language: .NET creates `Observability/ObservabilityServiceExtensions.cs` + `Observability/ObservabilityTokenService.cs`, Node.js creates `observability/observability-token-service.ts`, Python creates `observability/observability_token_service.py`. Each acquires and refreshes the Observability API token via MSAL client credentials targeting `api://9b975845-388f-4429-889e-eab1ef63949c/.default`. The .NET files additionally provide the `AddAgent365Observability()` / `Agent365ObservabilityContext` DI extensions that replace `AddAgenticTracingExporter()` and per-turn `RegisterObservability()`. The validator (`validate-instrument-observability.js`) accepts either the OBO signal (`BaggageBuilder` / `BaggageTurnMiddleware`) or the S2S signal to pass the context check. When the unified distro is used, it also requires the S2S route flag and a token resolver, both passed in the distro call itself (`o.Agent365.TokenResolver` / `tokenResolver` / `a365_token_resolver`), in every auth mode, and it fails the session on delegated telemetry wiring (`refreshObservabilityToken(..., authorization)`, `RegisterObservability(..., AgenticTokenStruct)`, `exchange_token(...)` for the observability scope). --- diff --git a/plugins/agent365/hooks/stop/validate-a365-code-validator.js b/plugins/agent365/hooks/stop/validate-a365-code-validator.js index f353666..cae567f 100644 --- a/plugins/agent365/hooks/stop/validate-a365-code-validator.js +++ b/plugins/agent365/hooks/stop/validate-a365-code-validator.js @@ -109,29 +109,33 @@ function findCallBlocks(content, functionName) { return blocks; } -// A distro call counts as wired only when its own arguments pass the export token resolver: an -// explicit or shorthand `tokenResolver` property (Node.js), an `a365_token_resolver` keyword or -// `**{...}` key (Python), or a `TokenResolver` assignment in the options callback (.NET). A variable -// passed to the call is followed one level to its initializer in the same file; a resolver symbol -// elsewhere, such as an unused import, does not count. -const RESOLVER_WIRING = { - node: { - call: 'useMicrosoftOpenTelemetry', - wired: /[{,]\s*tokenResolver\s*(?=[,}])|\btokenResolver\s*:/, - comments: /\/\*[\s\S]*?\*\/|(^|[\s,{;(])\/\/[^\n]*/gm, - }, - python: { - call: 'use_microsoft_opentelemetry', - wired: /\ba365_(?:contextual_)?token_resolver\s*=(?!=)|['"]a365_(?:contextual_)?token_resolver['"]\s*:/, - comments: /(^|\s)#[^\n]*/gm, - }, - dotnet: { - call: 'UseMicrosoftOpenTelemetry', - wired: /\b(?:Contextual)?TokenResolver\s*=(?!=)/, - comments: /\/\*[\s\S]*?\*\/|(^|[\s,{;(])\/\/[^\n]*/gm, - }, +// Distro-call checks shared verbatim by both stop hooks and the standalone scanner. An option counts +// only when the distro call's own arguments pass it: an explicit or shorthand property (Node.js), a +// keyword or `**{...}` key (Python), or an assignment in the options callback (.NET). A variable passed +// to the call is followed one level to its initializer in the same file. Comments are ignored, so a +// commented-out option, an unused import, or an unrelated object elsewhere does not count. +const DISTRO_CALLS = { + node: { call: 'useMicrosoftOpenTelemetry', comments: /\/\*[\s\S]*?\*\/|(^|[\s,{;(])\/\/[^\n]*/gm }, + python: { call: 'use_microsoft_opentelemetry', comments: /(^|\s)#[^\n]*/gm }, + dotnet: { call: 'UseMicrosoftOpenTelemetry', comments: /\/\*[\s\S]*?\*\/|(^|[\s,{;(])\/\/[^\n]*/gm }, }; +const TOKEN_RESOLVER_OPTION = { + node: /[{,]\s*tokenResolver\s*(?=[,}])|\btokenResolver\s*:/, + python: /\ba365_(?:contextual_)?token_resolver\s*=(?!=)|['"]a365_(?:contextual_)?token_resolver['"]\s*:/, + dotnet: /\b(?:Contextual)?TokenResolver\s*=(?!=)/, +}; + +const S2S_ROUTE_OPTION = { + node: /\buseS2SEndpoint\s*:\s*true\b/, + python: /\ba365_use_s2s_endpoint\s*=\s*True\b|['"]a365_use_s2s_endpoint['"]\s*:\s*True\b/, + dotnet: /\bUseS2SEndpoint\s*=\s*true\b/, +}; + +function stripComments(content, language) { + return content.replace(DISTRO_CALLS[language].comments, '$1'); +} + // Returns the text from the bracket at `open` through its matching close bracket. function bracketBlock(content, open) { const close = { '(': ')', '{': '}' }[content[open]]; @@ -143,31 +147,60 @@ function bracketBlock(content, open) { return ''; } -function distroCallHasResolver(files, language) { - const { call, wired, comments } = RESOLVER_WIRING[language]; +// Returns the parenthesized arguments of every `name(...)` call in `content`. +function callArguments(content, name) { + const calls = []; + for (let index = content.indexOf(`${name}(`); index !== -1; index = content.indexOf(`${name}(`, index + 1)) { + calls.push(bracketBlock(content, index + name.length)); + } + return calls; +} + +// True when a distro call in `files` passes an option matching `option`. +function distroCallMatches(files, language, option) { + const { call } = DISTRO_CALLS[language]; return files.some(file => { - const content = read(file).replace(comments, '$1'); - for (let index = content.indexOf(`${call}(`); index !== -1; index = content.indexOf(`${call}(`, index + 1)) { - const args = bracketBlock(content, index + call.length); - if (wired.test(args)) return true; + const content = stripComments(read(file), language); + return callArguments(content, call).some(args => { + if (option.test(args)) return true; // .NET: a method group instead of an inline lambda configures the options elsewhere in the file. - if (language === 'dotnet' && !args.includes('=>') && wired.test(content)) return true; + if (language === 'dotnet' && !args.includes('=>')) return option.test(content); for (const name of new Set(args.match(/[A-Za-z_$][\w$]*/g) || [])) { const escaped = name.replace(/\$/g, '\\$'); const initializer = new RegExp(`(?:^|[^\\w$])${escaped}\\s*(?::[^=\\n]*)?=\\s*(?:dict\\s*)?([({])`, 'gm'); for (let m = initializer.exec(content); m; m = initializer.exec(content)) { - if (wired.test(bracketBlock(content, m.index + m[0].length - 1))) return true; + if (option.test(bracketBlock(content, m.index + m[0].length - 1))) return true; } - if (language === 'python' && - new RegExp(`\\b${escaped}\\s*\\[\\s*['"]a365_(?:contextual_)?token_resolver['"]\\s*\\]\\s*=(?!=)`).test(content)) { - return true; + if (language === 'python') { + // `kwargs["key"] = value` is read as the dict entry it adds. + const keyAssignment = new RegExp(`\\b${escaped}\\s*\\[\\s*(['"][^'"\\n]+['"])\\s*\\]\\s*=(?!=)\\s*([^\\n]*)`, 'g'); + for (let m = keyAssignment.exec(content); m; m = keyAssignment.exec(content)) { + if (option.test(`${m[1]}: ${m[2]}`)) return true; + } } } - } - return false; + return false; + }); }); } +function distroCallHasResolver(files, language) { + return distroCallMatches(files, language, TOKEN_RESOLVER_OPTION[language]); +} + +function distroCallUsesS2SRoute(files, language) { + return distroCallMatches(files, language, S2S_ROUTE_OPTION[language]); +} + +// Whole-file scans that ignore comments, so a documented legacy call is not reported. +function codeMatches(files, language, regex) { + return files.some(file => regex.test(stripComments(read(file), language))); +} + +function codeCallMatches(files, language, name, regex) { + return files.some(file => callArguments(stripComments(read(file), language), name).some(args => regex.test(args))); +} + function validatePython() { const hasMicrosoftOpenTelemetryPackage = reqFiles.some(f => fileContains(f, 'microsoft-opentelemetry')); const hasDistroCall = anyFileContains(pyFiles, 'use_microsoft_opentelemetry'); @@ -175,8 +208,8 @@ function validatePython() { const hasExplicitExporter = anyFileMatches(pyFiles, /\ba365_enable_observability_exporter\s*=\s*True\b/); const hasExplicitExporterFalse = anyFileMatches(pyFiles, /\ba365_enable_observability_exporter\s*=\s*False\b/); const hasExporterEnv = envHasTruthy('ENABLE_A365_OBSERVABILITY_EXPORTER') || envHasTruthy('EnableAgent365Exporter'); - const hasExplicitS2S = anyFileMatches(pyFiles, /\ba365_use_s2s_endpoint\s*=\s*True\b/); - const hasExplicitS2SFalse = anyFileMatches(pyFiles, /\ba365_use_s2s_endpoint\s*=\s*False\b/); + const hasExplicitS2S = distroCallUsesS2SRoute(pyFiles, 'python'); + const hasExplicitS2SFalse = distroCallMatches(pyFiles, 'python', /\ba365_use_s2s_endpoint\s*=\s*False\b|['"]a365_use_s2s_endpoint['"]\s*:\s*False\b/); const hasS2SEnv = envHasTruthy('A365_USE_S2S_ENDPOINT'); const hasS2SIntent = anyFileContains(pyFiles, 'a365_contextual_token_resolver') || anyFileContains(pyFiles, 'agent_source_identity') || @@ -259,7 +292,7 @@ function validatePython() { } for (const file of pyFiles) { - const content = read(file); + const content = stripComments(read(file), 'python'); const delegated = findCallBlocks(content, 'exchange_token').some(block => /observability/i.test(block)) || /(? findCallBlocks(read(f), 'prefetch').some(block => /self\.connection_manager/.test(block))); - if (prefetchFile && !anyFileMatches(pyFiles, /\bself\.connection_manager\s*=/)) { + const prefetchFile = pyFiles.find(f => findCallBlocks(stripComments(read(f), 'python'), 'prefetch').some(block => /self\.connection_manager/.test(block))); + if (prefetchFile && !codeMatches(pyFiles, 'python', /\bself\.connection_manager\s*=/)) { add( 'high', 'python-obs-prefetch-connection-missing', @@ -395,7 +428,7 @@ function validateNode() { } // Every auth mode exports over the S2S route with an app-only token. - if (hasDistroCall && hasA365Enabled && !anyFileMatches(tsFiles, /\buseS2SEndpoint\s*:\s*true\b/)) { + if (hasDistroCall && hasA365Enabled && !distroCallUsesS2SRoute(tsFiles, 'node')) { add( 'high', 'node-obs-delegated-route', @@ -412,7 +445,7 @@ function validateNode() { ); } for (const file of tsFiles) { - const content = read(file); + const content = stripComments(read(file), 'node'); const delegatedRefresh = ['refreshObservabilityToken', 'RefreshObservabilityToken'] .some(name => findCallBlocks(content, name).some(block => /authorization/i.test(block))) || /AgenticTokenCacheInstance\s*\.\s*getObservabilityToken\s*\(/.test(content); @@ -487,7 +520,7 @@ function validateDotnet() { // Every auth mode exports over the S2S route with an app-only token. const csFiles = allFiles.filter(f => f.endsWith('.cs')); - if (anyFileContains(csFiles, 'UseMicrosoftOpenTelemetry') && !anyFileMatches(csFiles, /\bUseS2SEndpoint\s*=\s*true\b/)) { + if (anyFileContains(csFiles, 'UseMicrosoftOpenTelemetry') && !distroCallUsesS2SRoute(csFiles, 'dotnet')) { add( 'high', 'dotnet-obs-delegated-route', @@ -504,7 +537,7 @@ function validateDotnet() { ); } for (const file of csFiles) { - const content = read(file); + const content = stripComments(read(file), 'dotnet'); if (findCallBlocks(content, 'RegisterObservability').some(block => block.includes('AgenticTokenStruct')) || /\bnew\s+AgenticTokenStruct\s*[({]|IExporterTokenCache\s*<\s*AgenticTokenStruct\s*>\s*\??\s+[A-Za-z_]\w*/.test(content)) { add( diff --git a/plugins/agent365/hooks/stop/validate-instrument-observability.js b/plugins/agent365/hooks/stop/validate-instrument-observability.js index 5b2e427..01099ac 100644 --- a/plugins/agent365/hooks/stop/validate-instrument-observability.js +++ b/plugins/agent365/hooks/stop/validate-instrument-observability.js @@ -38,53 +38,33 @@ function anyFileMatches(files, regex) { return files.some(f => regex.test(read(f))); } -// Returns the source text of every `name(...)` call in `content`, with balanced parentheses. -function callBlocks(content, name) { - const blocks = []; - const needle = `${name}(`; - let index = 0; - while ((index = content.indexOf(needle, index)) !== -1) { - let depth = 0; - let end = -1; - for (let i = index + name.length; i < content.length; i++) { - if (content[i] === '(') depth++; - if (content[i] === ')' && --depth === 0) { end = i + 1; break; } - } - if (end === -1) break; - blocks.push(content.slice(index, end)); - index = end; - } - return blocks; -} +// Distro-call checks shared verbatim by both stop hooks and the standalone scanner. An option counts +// only when the distro call's own arguments pass it: an explicit or shorthand property (Node.js), a +// keyword or `**{...}` key (Python), or an assignment in the options callback (.NET). A variable passed +// to the call is followed one level to its initializer in the same file. Comments are ignored, so a +// commented-out option, an unused import, or an unrelated object elsewhere does not count. +const DISTRO_CALLS = { + node: { call: 'useMicrosoftOpenTelemetry', comments: /\/\*[\s\S]*?\*\/|(^|[\s,{;(])\/\/[^\n]*/gm }, + python: { call: 'use_microsoft_opentelemetry', comments: /(^|\s)#[^\n]*/gm }, + dotnet: { call: 'UseMicrosoftOpenTelemetry', comments: /\/\*[\s\S]*?\*\/|(^|[\s,{;(])\/\/[^\n]*/gm }, +}; -// True when any `name(...)` call in `files` has arguments matching `regex`. -function anyCallMatches(files, name, regex) { - return files.some(f => callBlocks(read(f), name).some(block => regex.test(block))); -} +const TOKEN_RESOLVER_OPTION = { + node: /[{,]\s*tokenResolver\s*(?=[,}])|\btokenResolver\s*:/, + python: /\ba365_(?:contextual_)?token_resolver\s*=(?!=)|['"]a365_(?:contextual_)?token_resolver['"]\s*:/, + dotnet: /\b(?:Contextual)?TokenResolver\s*=(?!=)/, +}; -// A distro call counts as wired only when its own arguments pass the export token resolver: an -// explicit or shorthand `tokenResolver` property (Node.js), an `a365_token_resolver` keyword or -// `**{...}` key (Python), or a `TokenResolver` assignment in the options callback (.NET). A variable -// passed to the call is followed one level to its initializer in the same file; a resolver symbol -// elsewhere, such as an unused import, does not count. -const RESOLVER_WIRING = { - node: { - call: 'useMicrosoftOpenTelemetry', - wired: /[{,]\s*tokenResolver\s*(?=[,}])|\btokenResolver\s*:/, - comments: /\/\*[\s\S]*?\*\/|(^|[\s,{;(])\/\/[^\n]*/gm, - }, - python: { - call: 'use_microsoft_opentelemetry', - wired: /\ba365_(?:contextual_)?token_resolver\s*=(?!=)|['"]a365_(?:contextual_)?token_resolver['"]\s*:/, - comments: /(^|\s)#[^\n]*/gm, - }, - dotnet: { - call: 'UseMicrosoftOpenTelemetry', - wired: /\b(?:Contextual)?TokenResolver\s*=(?!=)/, - comments: /\/\*[\s\S]*?\*\/|(^|[\s,{;(])\/\/[^\n]*/gm, - }, +const S2S_ROUTE_OPTION = { + node: /\buseS2SEndpoint\s*:\s*true\b/, + python: /\ba365_use_s2s_endpoint\s*=\s*True\b|['"]a365_use_s2s_endpoint['"]\s*:\s*True\b/, + dotnet: /\bUseS2SEndpoint\s*=\s*true\b/, }; +function stripComments(content, language) { + return content.replace(DISTRO_CALLS[language].comments, '$1'); +} + // Returns the text from the bracket at `open` through its matching close bracket. function bracketBlock(content, open) { const close = { '(': ')', '{': '}' }[content[open]]; @@ -96,31 +76,60 @@ function bracketBlock(content, open) { return ''; } -function distroCallHasResolver(files, language) { - const { call, wired, comments } = RESOLVER_WIRING[language]; +// Returns the parenthesized arguments of every `name(...)` call in `content`. +function callArguments(content, name) { + const calls = []; + for (let index = content.indexOf(`${name}(`); index !== -1; index = content.indexOf(`${name}(`, index + 1)) { + calls.push(bracketBlock(content, index + name.length)); + } + return calls; +} + +// True when a distro call in `files` passes an option matching `option`. +function distroCallMatches(files, language, option) { + const { call } = DISTRO_CALLS[language]; return files.some(file => { - const content = read(file).replace(comments, '$1'); - for (let index = content.indexOf(`${call}(`); index !== -1; index = content.indexOf(`${call}(`, index + 1)) { - const args = bracketBlock(content, index + call.length); - if (wired.test(args)) return true; + const content = stripComments(read(file), language); + return callArguments(content, call).some(args => { + if (option.test(args)) return true; // .NET: a method group instead of an inline lambda configures the options elsewhere in the file. - if (language === 'dotnet' && !args.includes('=>') && wired.test(content)) return true; + if (language === 'dotnet' && !args.includes('=>')) return option.test(content); for (const name of new Set(args.match(/[A-Za-z_$][\w$]*/g) || [])) { const escaped = name.replace(/\$/g, '\\$'); const initializer = new RegExp(`(?:^|[^\\w$])${escaped}\\s*(?::[^=\\n]*)?=\\s*(?:dict\\s*)?([({])`, 'gm'); for (let m = initializer.exec(content); m; m = initializer.exec(content)) { - if (wired.test(bracketBlock(content, m.index + m[0].length - 1))) return true; + if (option.test(bracketBlock(content, m.index + m[0].length - 1))) return true; } - if (language === 'python' && - new RegExp(`\\b${escaped}\\s*\\[\\s*['"]a365_(?:contextual_)?token_resolver['"]\\s*\\]\\s*=(?!=)`).test(content)) { - return true; + if (language === 'python') { + // `kwargs["key"] = value` is read as the dict entry it adds. + const keyAssignment = new RegExp(`\\b${escaped}\\s*\\[\\s*(['"][^'"\\n]+['"])\\s*\\]\\s*=(?!=)\\s*([^\\n]*)`, 'g'); + for (let m = keyAssignment.exec(content); m; m = keyAssignment.exec(content)) { + if (option.test(`${m[1]}: ${m[2]}`)) return true; + } } } - } - return false; + return false; + }); }); } +function distroCallHasResolver(files, language) { + return distroCallMatches(files, language, TOKEN_RESOLVER_OPTION[language]); +} + +function distroCallUsesS2SRoute(files, language) { + return distroCallMatches(files, language, S2S_ROUTE_OPTION[language]); +} + +// Whole-file scans that ignore comments, so a documented legacy call is not reported. +function codeMatches(files, language, regex) { + return files.some(file => regex.test(stripComments(read(file), language))); +} + +function codeCallMatches(files, language, name, regex) { + return files.some(file => callArguments(stripComments(read(file), language), name).some(args => regex.test(args))); +} + // ── Detect project type ───────────────────────────────────────────────────── // Walk the project tree once, then bucket by name. @@ -225,14 +234,14 @@ if (isDotnet) { // 3b. Telemetry uses the S2S route with an app-only token in every auth mode. // The S2S route rejects delegated (scp) tokens, and the distro defaults to the delegated route. - if (hasDistroWired && !anyFileMatches(csFiles, /\bUseS2SEndpoint\s*=\s*true\b/)) { + if (hasDistroWired && !distroCallUsesS2SRoute(csFiles, 'dotnet')) { issues.push('Observability export must use the S2S route in every auth mode: set o.Agent365.UseS2SEndpoint = true in UseMicrosoftOpenTelemetry (o.Agent365.Exporter.UseS2SEndpoint on Microsoft.OpenTelemetry 1.0.2 and earlier) and wire an app-only token resolver (AgentAppTokenResolver, or ObservabilityTokenService for s2s)'); } if (hasDistroWired && !distroCallHasResolver(csFiles, 'dotnet')) { issues.push('UseMicrosoftOpenTelemetry is wired without o.Agent365.TokenResolver, so the S2S route gets no app-only token (the distro default token cache holds delegated tokens, which the S2S route rejects) — set o.Agent365.TokenResolver to AgentAppTokenResolver.ResolveAsync (obo / agentic-user) or to the ServiceTokenCache fed by ObservabilityTokenService (s2s) (see dotnet-observability.md)'); } - if (anyCallMatches(csFiles, 'RegisterObservability', /AgenticTokenStruct/) || - anyFileMatches(csFiles, /\bnew\s+AgenticTokenStruct\s*[({]|IExporterTokenCache\s*<\s*AgenticTokenStruct\s*>\s*\??\s+[A-Za-z_]\w*/)) { + if (codeCallMatches(csFiles, 'dotnet', 'RegisterObservability', /AgenticTokenStruct/) || + codeMatches(csFiles, 'dotnet', /\bnew\s+AgenticTokenStruct\s*[({]|IExporterTokenCache\s*<\s*AgenticTokenStruct\s*>\s*\??\s+[A-Za-z_]\w*/)) { issues.push('RegisterObservability(..., AgenticTokenStruct), new AgenticTokenStruct(...), or an IExporterTokenCache dependency wires a delegated (OBO) telemetry token, which the S2S route rejects — remove the per-turn registration and use AgentAppTokenResolver as o.Agent365.TokenResolver (see dotnet-observability.md)'); } @@ -333,12 +342,12 @@ if (isNodejs) { // 4b. Telemetry uses the S2S route with an app-only token in every auth mode. // The S2S route rejects delegated (scp) tokens, and the distro defaults to the delegated route. - if (usesDistro && !anyFileMatches(tsFiles, /\buseS2SEndpoint\s*:\s*true\b/)) { + if (usesDistro && !distroCallUsesS2SRoute(tsFiles, 'node')) { issues.push('Observability export must use the S2S route in every auth mode: pass useS2SEndpoint: true in the a365 options of useMicrosoftOpenTelemetry() with an app-only tokenResolver (observability/app-token-resolver.ts for obo / agentic-user)'); } - if (anyCallMatches(tsFiles, 'refreshObservabilityToken', /authorization/i) || - anyCallMatches(tsFiles, 'RefreshObservabilityToken', /authorization/i) || - anyFileMatches(tsFiles, /AgenticTokenCacheInstance\s*\.\s*getObservabilityToken\s*\(/)) { + if (codeCallMatches(tsFiles, 'node', 'refreshObservabilityToken', /authorization/i) || + codeCallMatches(tsFiles, 'node', 'RefreshObservabilityToken', /authorization/i) || + codeMatches(tsFiles, 'node', /AgenticTokenCacheInstance\s*\.\s*getObservabilityToken\s*\(/)) { issues.push('refreshObservabilityToken(..., authorization) or AgenticTokenCacheInstance.getObservabilityToken(...) feeds a delegated (OBO) telemetry token, which the S2S route rejects — remove it (and any preloadObservabilityToken helper) and use the app-only tokenResolver (see nodejs-observability.md)'); } @@ -438,19 +447,19 @@ if (isPython) { // 4b. Telemetry uses the S2S route with an app-only token in every auth mode. // The S2S route rejects delegated (scp) tokens, and the distro defaults to the delegated route. - const pyS2SInCode = anyFileMatches(pyFiles, /\ba365_use_s2s_endpoint\s*=\s*True\b/); + const pyS2SInCode = distroCallUsesS2SRoute(pyFiles, 'python'); const pyS2SInEnv = anyFileMatches(envFiles, /^\s*A365_USE_S2S_ENDPOINT\s*=\s*(true|1|yes)\s*$/im); if (usesDistroPy && !pyS2SInCode && !pyS2SInEnv) { issues.push('Observability export must use the S2S route in every auth mode: pass a365_use_s2s_endpoint=True to use_microsoft_opentelemetry() with an app-only a365_token_resolver (observability/app_token_resolver.py for obo / agentic-user)'); } - if (anyCallMatches(pyFiles, 'exchange_token', /observability/i) || - anyFileMatches(pyFiles, /(?` dependency); route = `UseS2SEndpoint = true`; handler = `BaggageBuilder` OR `BaggageTurnMiddleware` OR `InvokeAgentScope.Start`. - - **Node.js (`src/**/*.ts`)**: entry = `useMicrosoftOpenTelemetry`; token = the `useMicrosoftOpenTelemetry({...})` call passes `tokenResolver` in its `a365` options (app-only: `observability/app-token-resolver.ts`, or `getS2SObservabilityToken` / `startTokenService` for S2S), with no `refreshObservabilityToken(..., authorization)` or `AgenticTokenCacheInstance.getObservabilityToken` resolver; route = `useS2SEndpoint: true`; handler = `BaggageBuilder` OR `BaggageBuilderUtils` OR `InvokeAgentScope`. - - **Python (`**/*.py`)**: entry = `use_microsoft_opentelemetry`; token = the `use_microsoft_opentelemetry(...)` call passes `a365_token_resolver=` (app-only: `OBS_TOKENS.resolve` from `AppTokenResolver`, or the S2S `run_token_service` / `get_s2s_observability_token` cache), with no `exchange_token(...)` for the observability scope, `cache_agentic_token(...)`, or `AgenticTokenCache` / `get_cached_agentic_token` resolver; route = `a365_use_s2s_endpoint=True` (or `A365_USE_S2S_ENDPOINT=true` in `.env`); handler = `BaggageBuilder` OR `populate_baggage` OR `InvokeAgentScope`. + - **.NET (`**/*.cs`)**: entry = `UseMicrosoftOpenTelemetry`; token = the `UseMicrosoftOpenTelemetry(...)` options assign `TokenResolver` (`o.Agent365.TokenResolver`; `o.Agent365.Exporter.TokenResolver` on 1.0.2 and earlier) to the app-only `AgentAppTokenResolver` (obo / agentic-user) or the S2S token cache fed by `ObservabilityTokenService` / `AddAgent365Observability`, with no `AgenticTokenStruct` usage (per-turn `RegisterObservability(..., new AgenticTokenStruct(...))`, `new AgenticTokenStruct(...)`, or an `IExporterTokenCache` dependency); route = `UseS2SEndpoint = true` in the same options; handler = `BaggageBuilder` OR `BaggageTurnMiddleware` OR `InvokeAgentScope.Start`. + - **Node.js (`src/**/*.ts`)**: entry = `useMicrosoftOpenTelemetry`; token = the `useMicrosoftOpenTelemetry({...})` call passes `tokenResolver` in its `a365` options (app-only: `observability/app-token-resolver.ts`, or `getS2SObservabilityToken` / `startTokenService` for S2S), with no `refreshObservabilityToken(..., authorization)` or `AgenticTokenCacheInstance.getObservabilityToken` resolver; route = `useS2SEndpoint: true` in the same call's `a365` options; handler = `BaggageBuilder` OR `BaggageBuilderUtils` OR `InvokeAgentScope`. + - **Python (`**/*.py`)**: entry = `use_microsoft_opentelemetry`; token = the `use_microsoft_opentelemetry(...)` call passes `a365_token_resolver=` (app-only: `OBS_TOKENS.resolve` from `AppTokenResolver`, or the S2S `run_token_service` / `get_s2s_observability_token` cache), with no `exchange_token(...)` for the observability scope, `cache_agentic_token(...)`, or `AgenticTokenCache` / `get_cached_agentic_token` resolver; route = `a365_use_s2s_endpoint=True` passed in the same call (or `A365_USE_S2S_ENDPOINT=true` in `.env`); handler = `BaggageBuilder` OR `populate_baggage` OR `InvokeAgentScope`. - The token anchor must be wired at the distro call itself. A variable passed to the call counts when its initializer sets the resolver, but a resolver symbol or scaffold elsewhere does not: an unused import, a `tokenResolver` in another file, or an `ObservabilityTokenService` that is never assigned to `TokenResolver` leaves export without a credential. + The token and route anchors must be wired at the distro call itself. A variable passed to the call counts when its initializer sets the option, but a symbol elsewhere does not, and neither do comments: an unused import, a `tokenResolver` in another file, an `ObservabilityTokenService` that is never assigned to `TokenResolver`, or a `useS2SEndpoint: true` on an unrelated object leaves export on the wrong route or without a credential. Package-name matches alone (`Microsoft.Agents.A365.Observability.*`, `@microsoft/agents-a365-observability`, `microsoft-agents-a365-observability-*`, `Microsoft.OpenTelemetry` / `@microsoft/opentelemetry` / `microsoft-opentelemetry`) do NOT count — they reflect installation, not invocation. diff --git a/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js b/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js index 7a1bca4..ec838eb 100644 --- a/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js +++ b/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js @@ -107,29 +107,33 @@ function callBlocks(content, name) { return blocks; } -// A distro call counts as wired only when its own arguments pass the export token resolver: an -// explicit or shorthand `tokenResolver` property (Node.js), an `a365_token_resolver` keyword or -// `**{...}` key (Python), or a `TokenResolver` assignment in the options callback (.NET). A variable -// passed to the call is followed one level to its initializer in the same file; a resolver symbol -// elsewhere, such as an unused import, does not count. -const RESOLVER_WIRING = { - node: { - call: 'useMicrosoftOpenTelemetry', - wired: /[{,]\s*tokenResolver\s*(?=[,}])|\btokenResolver\s*:/, - comments: /\/\*[\s\S]*?\*\/|(^|[\s,{;(])\/\/[^\n]*/gm, - }, - python: { - call: 'use_microsoft_opentelemetry', - wired: /\ba365_(?:contextual_)?token_resolver\s*=(?!=)|['"]a365_(?:contextual_)?token_resolver['"]\s*:/, - comments: /(^|\s)#[^\n]*/gm, - }, - dotnet: { - call: 'UseMicrosoftOpenTelemetry', - wired: /\b(?:Contextual)?TokenResolver\s*=(?!=)/, - comments: /\/\*[\s\S]*?\*\/|(^|[\s,{;(])\/\/[^\n]*/gm, - }, +// Distro-call checks shared verbatim by both stop hooks and the standalone scanner. An option counts +// only when the distro call's own arguments pass it: an explicit or shorthand property (Node.js), a +// keyword or `**{...}` key (Python), or an assignment in the options callback (.NET). A variable passed +// to the call is followed one level to its initializer in the same file. Comments are ignored, so a +// commented-out option, an unused import, or an unrelated object elsewhere does not count. +const DISTRO_CALLS = { + node: { call: 'useMicrosoftOpenTelemetry', comments: /\/\*[\s\S]*?\*\/|(^|[\s,{;(])\/\/[^\n]*/gm }, + python: { call: 'use_microsoft_opentelemetry', comments: /(^|\s)#[^\n]*/gm }, + dotnet: { call: 'UseMicrosoftOpenTelemetry', comments: /\/\*[\s\S]*?\*\/|(^|[\s,{;(])\/\/[^\n]*/gm }, }; +const TOKEN_RESOLVER_OPTION = { + node: /[{,]\s*tokenResolver\s*(?=[,}])|\btokenResolver\s*:/, + python: /\ba365_(?:contextual_)?token_resolver\s*=(?!=)|['"]a365_(?:contextual_)?token_resolver['"]\s*:/, + dotnet: /\b(?:Contextual)?TokenResolver\s*=(?!=)/, +}; + +const S2S_ROUTE_OPTION = { + node: /\buseS2SEndpoint\s*:\s*true\b/, + python: /\ba365_use_s2s_endpoint\s*=\s*True\b|['"]a365_use_s2s_endpoint['"]\s*:\s*True\b/, + dotnet: /\bUseS2SEndpoint\s*=\s*true\b/, +}; + +function stripComments(content, language) { + return content.replace(DISTRO_CALLS[language].comments, '$1'); +} + // Returns the text from the bracket at `open` through its matching close bracket. function bracketBlock(content, open) { const close = { '(': ')', '{': '}' }[content[open]]; @@ -141,31 +145,60 @@ function bracketBlock(content, open) { return ''; } -function distroCallHasResolver(files, language) { - const { call, wired, comments } = RESOLVER_WIRING[language]; +// Returns the parenthesized arguments of every `name(...)` call in `content`. +function callArguments(content, name) { + const calls = []; + for (let index = content.indexOf(`${name}(`); index !== -1; index = content.indexOf(`${name}(`, index + 1)) { + calls.push(bracketBlock(content, index + name.length)); + } + return calls; +} + +// True when a distro call in `files` passes an option matching `option`. +function distroCallMatches(files, language, option) { + const { call } = DISTRO_CALLS[language]; return files.some(file => { - const content = read(file).replace(comments, '$1'); - for (let index = content.indexOf(`${call}(`); index !== -1; index = content.indexOf(`${call}(`, index + 1)) { - const args = bracketBlock(content, index + call.length); - if (wired.test(args)) return true; + const content = stripComments(read(file), language); + return callArguments(content, call).some(args => { + if (option.test(args)) return true; // .NET: a method group instead of an inline lambda configures the options elsewhere in the file. - if (language === 'dotnet' && !args.includes('=>') && wired.test(content)) return true; + if (language === 'dotnet' && !args.includes('=>')) return option.test(content); for (const name of new Set(args.match(/[A-Za-z_$][\w$]*/g) || [])) { const escaped = name.replace(/\$/g, '\\$'); const initializer = new RegExp(`(?:^|[^\\w$])${escaped}\\s*(?::[^=\\n]*)?=\\s*(?:dict\\s*)?([({])`, 'gm'); for (let m = initializer.exec(content); m; m = initializer.exec(content)) { - if (wired.test(bracketBlock(content, m.index + m[0].length - 1))) return true; + if (option.test(bracketBlock(content, m.index + m[0].length - 1))) return true; } - if (language === 'python' && - new RegExp(`\\b${escaped}\\s*\\[\\s*['"]a365_(?:contextual_)?token_resolver['"]\\s*\\]\\s*=(?!=)`).test(content)) { - return true; + if (language === 'python') { + // `kwargs["key"] = value` is read as the dict entry it adds. + const keyAssignment = new RegExp(`\\b${escaped}\\s*\\[\\s*(['"][^'"\\n]+['"])\\s*\\]\\s*=(?!=)\\s*([^\\n]*)`, 'g'); + for (let m = keyAssignment.exec(content); m; m = keyAssignment.exec(content)) { + if (option.test(`${m[1]}: ${m[2]}`)) return true; + } } } - } - return false; + return false; + }); }); } +function distroCallHasResolver(files, language) { + return distroCallMatches(files, language, TOKEN_RESOLVER_OPTION[language]); +} + +function distroCallUsesS2SRoute(files, language) { + return distroCallMatches(files, language, S2S_ROUTE_OPTION[language]); +} + +// Whole-file scans that ignore comments, so a documented legacy call is not reported. +function codeMatches(files, language, regex) { + return files.some(file => regex.test(stripComments(read(file), language))); +} + +function codeCallMatches(files, language, name, regex) { + return files.some(file => callArguments(stripComments(read(file), language), name).some(args => regex.test(args))); +} + function validatePython() { const hasPackage = req.some(f => read(f).includes('microsoft-opentelemetry')); const hasDistro = anyContains(py, 'use_microsoft_opentelemetry'); @@ -173,8 +206,8 @@ function validatePython() { const exporterTrue = anyMatches(py, /\ba365_enable_observability_exporter\s*=\s*True\b/); const exporterFalse = anyMatches(py, /\ba365_enable_observability_exporter\s*=\s*False\b/); const exporterEnv = envTrue('ENABLE_A365_OBSERVABILITY_EXPORTER') || envTrue('EnableAgent365Exporter'); - const s2sTrue = anyMatches(py, /\ba365_use_s2s_endpoint\s*=\s*True\b/); - const s2sFalse = anyMatches(py, /\ba365_use_s2s_endpoint\s*=\s*False\b/); + const s2sTrue = distroCallUsesS2SRoute(py, 'python'); + const s2sFalse = distroCallMatches(py, 'python', /\ba365_use_s2s_endpoint\s*=\s*False\b|['"]a365_use_s2s_endpoint['"]\s*:\s*False\b/); const s2sEnv = envTrue('A365_USE_S2S_ENDPOINT'); const s2sIntent = anyContains(py, 'a365_contextual_token_resolver') || anyContains(py, 'agent_source_identity') || @@ -209,7 +242,7 @@ function validatePython() { add('high', 'python-obs-token-resolver-missing', 'use_microsoft_opentelemetry() has no a365_token_resolver (or a365_contextual_token_resolver), so the S2S route gets no app-only token and the exporter drops spans. Pass an app-only resolver (instrument-observability app_token_resolver.py for obo / agentic-user, or the S2S token-service cache).', py.find(f => read(f).includes('use_microsoft_opentelemetry'))); } for (const file of py) { - const content = read(file); + const content = stripComments(read(file), 'python'); const delegated = callBlocks(content, 'exchange_token').some(block => /observability/i.test(block)) || /(? callBlocks(read(f), 'prefetch').some(block => /self\.connection_manager/.test(block))); - if (prefetchFile && !anyMatches(py, /\bself\.connection_manager\s*=/)) { + const prefetchFile = py.find(f => callBlocks(stripComments(read(f), 'python'), 'prefetch').some(block => /self\.connection_manager/.test(block))); + if (prefetchFile && !codeMatches(py, 'python', /\bself\.connection_manager\s*=/)) { add('high', 'python-obs-prefetch-connection-missing', 'The app-only token prefetch uses self.connection_manager, but no file assigns it. CloudAdapter does not expose its connection manager, so every prefetch fails and no spans export. Store the MsalConnectionManager passed to CloudAdapter on the host.', prefetchFile); } @@ -291,14 +324,14 @@ function validateNode() { add('high', 'node-missing-identity-scope', 'No BaggageBuilder/InvokeAgentScope/configureA365Hosting usage found; spans may lack agent identity.'); } // Every auth mode exports over the S2S route with an app-only token. - if (hasDistro && hasEnabled && !anyMatches(ts, /\buseS2SEndpoint\s*:\s*true\b/)) { + if (hasDistro && hasEnabled && !distroCallUsesS2SRoute(ts, 'node')) { add('high', 'node-obs-delegated-route', 'Node code does not set useS2SEndpoint: true, so A365 export uses the legacy delegated route. Every auth mode must export over the S2S route with an app-only tokenResolver.', ts.find(f => read(f).includes('useMicrosoftOpenTelemetry'))); } if (hasDistro && hasEnabled && !exporterFalse && !distroCallHasResolver(ts, 'node')) { add('high', 'node-obs-token-resolver-missing', 'useMicrosoftOpenTelemetry() has no a365 tokenResolver, so the S2S route gets no app-only token. Pass an app-only tokenResolver (instrument-observability app-token-resolver.ts for obo / agentic-user, or the S2S token service).', ts.find(f => read(f).includes('useMicrosoftOpenTelemetry'))); } for (const file of ts) { - const content = read(file); + const content = stripComments(read(file), 'node'); if (['refreshObservabilityToken', 'RefreshObservabilityToken'].some(name => callBlocks(content, name).some(block => /authorization/i.test(block))) || /AgenticTokenCacheInstance\s*\.\s*getObservabilityToken\s*\(/.test(content)) { add('high', 'node-obs-delegated-token', 'Telemetry uses a delegated (OBO) token (refreshObservabilityToken(..., authorization) or AgenticTokenCacheInstance.getObservabilityToken). The S2S route rejects delegated tokens; use an app-only tokenResolver for the agent identity instead.', file); @@ -334,14 +367,14 @@ function validateDotnet() { add('medium', 'dotnet-no-invoke-agent-scope', 'No InvokeAgentScope found; MAC Activity needs an invoke_agent parent span.'); } // Every auth mode exports over the S2S route with an app-only token. - if (anyContains(cs, 'UseMicrosoftOpenTelemetry') && !anyMatches(cs, /\bUseS2SEndpoint\s*=\s*true\b/)) { + if (anyContains(cs, 'UseMicrosoftOpenTelemetry') && !distroCallUsesS2SRoute(cs, 'dotnet')) { add('high', 'dotnet-obs-delegated-route', 'UseMicrosoftOpenTelemetry is wired without UseS2SEndpoint = true, so A365 export uses the legacy delegated route. Set o.Agent365.UseS2SEndpoint = true (o.Agent365.Exporter.UseS2SEndpoint on Microsoft.OpenTelemetry 1.0.2 and earlier) with an app-only TokenResolver in every auth mode.', cs.find(f => read(f).includes('UseMicrosoftOpenTelemetry'))); } if (anyContains(cs, 'UseMicrosoftOpenTelemetry') && !distroCallHasResolver(cs, 'dotnet')) { add('high', 'dotnet-obs-token-resolver-missing', 'UseMicrosoftOpenTelemetry is wired without o.Agent365.TokenResolver, so the S2S route gets no app-only token (the distro default token cache holds delegated tokens). Set TokenResolver to an app-only resolver (instrument-observability AgentAppTokenResolver for obo / agentic-user, or the ServiceTokenCache from ObservabilityTokenService for s2s).', cs.find(f => read(f).includes('UseMicrosoftOpenTelemetry'))); } for (const file of cs) { - const content = read(file); + const content = stripComments(read(file), 'dotnet'); if (callBlocks(content, 'RegisterObservability').some(block => block.includes('AgenticTokenStruct')) || /\bnew\s+AgenticTokenStruct\s*[({]|IExporterTokenCache\s*<\s*AgenticTokenStruct\s*>\s*\??\s+[A-Za-z_]\w*/.test(content)) { add('high', 'dotnet-obs-delegated-token', 'Telemetry uses a delegated (OBO) token (RegisterObservability with AgenticTokenStruct, new AgenticTokenStruct(...), or an IExporterTokenCache dependency). The S2S route rejects delegated tokens; wire an app-only TokenResolver for the agent identity instead.', file); diff --git a/plugins/agent365/skills/make-ai-teammate/SKILL.md b/plugins/agent365/skills/make-ai-teammate/SKILL.md index dba8ec5..fb24c39 100644 --- a/plugins/agent365/skills/make-ai-teammate/SKILL.md +++ b/plugins/agent365/skills/make-ai-teammate/SKILL.md @@ -330,7 +330,7 @@ Ask: "What language and framework are you using?" and set `language` and `agentS - **Observability composite** — compute four sub-signals, then combine: - `obs_entry` = `useMicrosoftOpenTelemetry` in any `src/**/*.ts` - `obs_token` = the `useMicrosoftOpenTelemetry({...})` call passes `tokenResolver` in its `a365` options (the app-only resolver `observability/app-token-resolver.ts` for obo / agentic-user; S2S also accepts `getS2SObservabilityToken` / `startTokenService`). A `tokenResolver` symbol elsewhere, such as an unused import, does not count. AND no `refreshObservabilityToken(..., authorization)` call or `AgenticTokenCacheInstance.getObservabilityToken` resolver (either means the telemetry token is delegated) - - `obs_route` = `useS2SEndpoint: true` in any `src/**/*.ts` (telemetry uses the S2S route in every auth mode) + - `obs_route` = `useS2SEndpoint: true` in the same `useMicrosoftOpenTelemetry` call's `a365` options (telemetry uses the S2S route in every auth mode) - `obs_handler` = `BaggageBuilder` OR `BaggageBuilderUtils` OR `InvokeAgentScope` in any `src/**/*.ts` - `has_obs_complete` = `obs_entry && obs_token && obs_route && obs_handler` - `has_obs_partial` = `obs_entry && !has_obs_complete` @@ -344,7 +344,7 @@ Ask: "What language and framework are you using?" and set `language` and `agentS - **Observability composite:** - `obs_entry` = `UseMicrosoftOpenTelemetry` in `Program.cs` (or legacy `AddA365Tracing`) - `obs_token` = the `UseMicrosoftOpenTelemetry(...)` options assign `o.Agent365.TokenResolver` (`o.Agent365.Exporter.TokenResolver` on 1.0.2 and earlier) to `AgentAppTokenResolver` (obo / agentic-user) or to the token cache fed by `ObservabilityTokenService` / `AddAgent365Observability` (S2S). A scaffold file that is never assigned does not count. AND no `AgenticTokenStruct` usage (`RegisterObservability(..., new AgenticTokenStruct(...))`, `new AgenticTokenStruct(...)`, or an `IExporterTokenCache` dependency — all register a delegated telemetry token) - - `obs_route` = `UseS2SEndpoint = true` in any `**/*.cs` (telemetry uses the S2S route in every auth mode) + - `obs_route` = `UseS2SEndpoint = true` in the same `UseMicrosoftOpenTelemetry(...)` options (telemetry uses the S2S route in every auth mode) - `obs_handler` = `BaggageBuilder` OR `BaggageTurnMiddleware` OR `InvokeAgentScope.Start` in `**/*.cs` - `has_obs_complete` = `obs_entry && obs_token && obs_route && obs_handler` - `has_obs_partial` = `obs_entry && !has_obs_complete` @@ -358,7 +358,7 @@ Ask: "What language and framework are you using?" and set `language` and `agentS - **Observability composite:** - `obs_entry` = `use_microsoft_opentelemetry` in any `**/*.py` - `obs_token` = the `use_microsoft_opentelemetry(...)` call passes `a365_token_resolver=` (the app-only `OBS_TOKENS.resolve` for obo / agentic-user; S2S: the `run_token_service` / `get_s2s_observability_token` cache). A resolver defined but not passed does not count. AND no `exchange_token(...)` for the observability scope, `cache_agentic_token(...)`, or `AgenticTokenCache` / `get_cached_agentic_token` resolver (all are delegated telemetry tokens) - - `obs_route` = `a365_use_s2s_endpoint=True` in any `**/*.py` (or `A365_USE_S2S_ENDPOINT=true` in `.env`) — telemetry uses the S2S route in every auth mode + - `obs_route` = `a365_use_s2s_endpoint=True` passed to the same `use_microsoft_opentelemetry(...)` call (or `A365_USE_S2S_ENDPOINT=true` in `.env`) — telemetry uses the S2S route in every auth mode - `obs_handler` = `BaggageBuilder` OR `populate_baggage` OR `InvokeAgentScope` in any `**/*.py` - `has_obs_complete` = `obs_entry && obs_token && obs_route && obs_handler` - `has_obs_partial` = `obs_entry && !has_obs_complete` diff --git a/tests/validate-a365-code-validator.test.js b/tests/validate-a365-code-validator.test.js index 2133ef3..cb3f4ad 100644 --- a/tests/validate-a365-code-validator.test.js +++ b/tests/validate-a365-code-validator.test.js @@ -607,18 +607,60 @@ token = await auth.exchange_token(context, scopes=["ea9ffc3e-8a23-4a7d-836d-234d } }); - test('the stop hooks and the standalone scanner share one distro resolver helper', () => { + test('the stop hooks and the standalone scanner share one distro-call helper', () => { const observabilityHook = path.join(__dirname, '../plugins/agent365/hooks/stop/validate-instrument-observability.js'); const helperText = file => { const text = fs.readFileSync(file, 'utf8').replace(/\r\n/g, '\n'); - const start = text.indexOf('const RESOLVER_WIRING = {'); - const end = text.indexOf('\n}\n', text.indexOf('function distroCallHasResolver(')); - assert.ok(start >= 0 && end > start, `${file}: resolver helper not found`); + const start = text.indexOf('const DISTRO_CALLS = {'); + const end = text.indexOf('\n}\n', text.indexOf('function codeCallMatches(')); + assert.ok(start >= 0 && end > start, `${file}: distro-call helper not found`); return text.slice(start, end); }; const expected = helperText(VALIDATOR); - assert.equal(helperText(STANDALONE), expected, 'standalone scanner must use the same resolver helper'); - assert.equal(helperText(observabilityHook), expected, 'instrument-observability hook must use the same resolver helper'); + assert.equal(helperText(STANDALONE), expected, 'standalone scanner must use the same distro-call helper'); + assert.equal(helperText(observabilityHook), expected, 'instrument-observability hook must use the same distro-call helper'); + }); + + test('route flags outside the distro call and legacy calls in comments are handled the same by both scanners', () => { + const csproj = ''; + const routeElsewhere = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': `${NODE_S2S_INDEX.replace(' useS2SEndpoint: true,', '')}\nconst workloadOptions = { useS2SEndpoint: true };`, + 'Agent.csproj': csproj, + 'Program.cs': 'builder.UseMicrosoftOpenTelemetry(o => { o.Agent365.TokenResolver = (a, t) => r.ResolveAsync(a, t); });\nworkloadOptions.UseS2SEndpoint = true;', + 'requirements.txt': 'microsoft-opentelemetry>=1.1.0\n', + 'host.py': 'use_microsoft_opentelemetry(enable_a365=True, a365_enable_observability_exporter=True, a365_token_resolver=OBS_TOKENS.resolve)\nconfigure_workload_client(a365_use_s2s_endpoint=True)', + }); + const commentedLegacy = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': NODE_S2S_INDEX, + 'agent.ts': '// Do NOT call AgenticTokenCacheInstance.refreshObservabilityToken(agentId, tenantId, turnContext, this.authorization).', + 'Agent.csproj': csproj, + 'Program.cs': [ + 'builder.UseMicrosoftOpenTelemetry(o => { o.Agent365.UseS2SEndpoint = true; o.Agent365.TokenResolver = (a, t) => r.ResolveAsync(a, t); });', + '// Legacy: _cache.RegisterObservability(a, t, new AgenticTokenStruct(userAuthorization: u, turnContext: c, authHandlerName: n), s);', + ].join('\n'), + 'requirements.txt': 'microsoft-opentelemetry>=1.1.0\n', + 'host.py': [ + 'use_microsoft_opentelemetry(enable_a365=True, a365_enable_observability_exporter=True, a365_use_s2s_endpoint=True, a365_token_resolver=OBS_TOKENS.resolve)', + '# Legacy: token = await auth.exchange_token(context, scopes=get_observability_authentication_scope())', + ].join('\n'), + }); + const routeIds = ids => ids.filter(id => /obs-delegated-route|s2s-endpoint-not-set/.test(id)).sort(); + const tokenIds = ids => ids.filter(id => /obs-delegated-token/.test(id)); + try { + for (const scanner of [VALIDATOR, STANDALONE]) { + assert.deepEqual(routeIds(findingIds(runValidator(scanner, routeElsewhere))), [ + 'dotnet-obs-delegated-route', + 'node-obs-delegated-route', + 'python-s2s-endpoint-not-set', + ], scanner); + assert.deepEqual(tokenIds(findingIds(runValidator(scanner, commentedLegacy))), [], scanner); + } + } finally { + cleanup(routeElsewhere); + cleanup(commentedLegacy); + } }); test('app-only S2S wiring in all three languages produces no delegated-telemetry findings in either scanner', () => { diff --git a/tests/validate-observability.test.js b/tests/validate-observability.test.js index 9265f59..c3068c5 100644 --- a/tests/validate-observability.test.js +++ b/tests/validate-observability.test.js @@ -822,6 +822,76 @@ useMicrosoftOpenTelemetry(otelOptions); } finally { cleanup(dir); } }); + test('Node.js useS2SEndpoint only on an unrelated object → reports the S2S route requirement', () => { + const dir = createFixture({ + ...NODEJS_DISTRO_VALID, + 'src/index.ts': `${NODEJS_DISTRO_VALID['src/index.ts'].replace(' useS2SEndpoint: true,', '')}\nconst workloadOptions = { useS2SEndpoint: true };`, + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /S2S route in every auth mode.*useS2SEndpoint: true/); + } finally { cleanup(dir); } + }); + + test('Node.js comment documenting the legacy refresh call is not flagged', () => { + const dir = createFixture({ + ...NODEJS_DISTRO_VALID, + 'src/agent.ts': `${NODEJS_DISTRO_VALID['src/agent.ts']}\n// Do NOT call AgenticTokenCacheInstance.refreshObservabilityToken(agentId, tenantId, turnContext, this.authorization).`, + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, true, r.reason); + } finally { cleanup(dir); } + }); + + test('.NET UseS2SEndpoint set outside the UseMicrosoftOpenTelemetry options → reports the S2S route requirement', () => { + const dir = createFixture({ + ...DOTNET_DISTRO_VALID, + 'Program.cs': DOTNET_DISTRO_VALID['Program.cs'].replace('o.Agent365.UseS2SEndpoint = true;', ''), + 'WorkloadClient.cs': 'workloadOptions.UseS2SEndpoint = true;', + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /S2S route in every auth mode.*UseS2SEndpoint = true/); + } finally { cleanup(dir); } + }); + + test('.NET commented-out AgenticTokenStruct registration is not flagged', () => { + const dir = createFixture({ + ...DOTNET_DISTRO_VALID, + 'MyAgent.cs': `${DOTNET_DISTRO_VALID['MyAgent.cs']}\n// Legacy: agentTokenCache?.RegisterObservability(agentId, tenantId, new AgenticTokenStruct(userAuthorization: auth, turnContext: turnContext, authHandlerName: name), scopes);`, + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, true, r.reason); + } finally { cleanup(dir); } + }); + + test('Python a365_use_s2s_endpoint only on an unrelated helper call → reports the S2S route requirement', () => { + const dir = createFixture({ + ...PYTHON_DISTRO_VALID, + 'host_agent_server.py': `${PYTHON_DISTRO_VALID['host_agent_server.py'].replace(' a365_use_s2s_endpoint=True,', '')}\nconfigure_workload_client(a365_use_s2s_endpoint=True)`, + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /S2S route in every auth mode.*a365_use_s2s_endpoint=True/); + } finally { cleanup(dir); } + }); + + test('Python commented-out observability exchange_token is not flagged', () => { + const dir = createFixture({ + ...PYTHON_DISTRO_VALID, + 'host_agent_server.py': `${PYTHON_DISTRO_VALID['host_agent_server.py']}\n# Legacy: token = await auth.exchange_token(context, scopes=get_observability_authentication_scope(), auth_handler_id=handler)`, + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, true, r.reason); + } finally { cleanup(dir); } + }); + test('Python distro passing the resolver through a kwargs dict → ok', () => { const dir = createFixture({ ...PYTHON_DISTRO_VALID, From 305102b92e1e59f74b6bad4258bff2f3d0fc21d8 Mon Sep 17 00:00:00 2001 From: Krishnadheeraj <12496535+DheerajPannala@users.noreply.github.com> Date: Mon, 28 Sep 2026 12:46:13 +0100 Subject: [PATCH 05/12] Read distro options where the SDK reads them Address the fourth Copilot review on #84: options inside the distro call's parentheses but outside the SDK's own options no longer count. - Node.js: only the `a365` options object counts: inline, by variable, shorthand, or a spread one level deep. `instrumentationOptions: { custom: { useS2SEndpoint: true } }` does not. - .NET: only `.Agent365` (or `.Agent365.Exporter`) assignments count, so `workloadOptions.UseS2SEndpoint = true` inside the callback does not. - Python: only the call's own keyword arguments count (plus `**` spreads), not keywords of a nested call. The call itself is masked when initializers are looked up, because Python keyword arguments look like assignments. - When the options cannot be read statically (a method group, a factory call, or a callback that never touches Agent365), the whole file is checked instead, to avoid false positives. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5cbf5f6b-cc40-4b7e-a591-65848db73a12 --- .../stop/validate-a365-code-validator.js | 126 ++++++++++++++---- .../stop/validate-instrument-observability.js | 126 ++++++++++++++---- .../references/a365-code-validator.js | 126 ++++++++++++++---- tests/validate-a365-code-validator.test.js | 27 ++++ tests/validate-observability.test.js | 77 +++++++++++ 5 files changed, 401 insertions(+), 81 deletions(-) diff --git a/plugins/agent365/hooks/stop/validate-a365-code-validator.js b/plugins/agent365/hooks/stop/validate-a365-code-validator.js index cae567f..e4b6712 100644 --- a/plugins/agent365/hooks/stop/validate-a365-code-validator.js +++ b/plugins/agent365/hooks/stop/validate-a365-code-validator.js @@ -110,10 +110,12 @@ function findCallBlocks(content, functionName) { } // Distro-call checks shared verbatim by both stop hooks and the standalone scanner. An option counts -// only when the distro call's own arguments pass it: an explicit or shorthand property (Node.js), a -// keyword or `**{...}` key (Python), or an assignment in the options callback (.NET). A variable passed -// to the call is followed one level to its initializer in the same file. Comments are ignored, so a -// commented-out option, an unused import, or an unrelated object elsewhere does not count. +// only where the SDK reads it: in the `a365` options object passed to useMicrosoftOpenTelemetry +// (Node.js), as a keyword argument of use_microsoft_opentelemetry (Python), or on `.Agent365` in the +// UseMicrosoftOpenTelemetry options callback (.NET). A variable is followed one level to its +// initializer in the same file. Comments are ignored, so a commented-out option, an unused import, or +// an unrelated object does not count. When the options cannot be located statically (a method group, +// a factory call, or a callback that never touches Agent365), the file as a whole is checked instead. const DISTRO_CALLS = { node: { call: 'useMicrosoftOpenTelemetry', comments: /\/\*[\s\S]*?\*\/|(^|[\s,{;(])\/\/[^\n]*/gm }, python: { call: 'use_microsoft_opentelemetry', comments: /(^|\s)#[^\n]*/gm }, @@ -123,13 +125,13 @@ const DISTRO_CALLS = { const TOKEN_RESOLVER_OPTION = { node: /[{,]\s*tokenResolver\s*(?=[,}])|\btokenResolver\s*:/, python: /\ba365_(?:contextual_)?token_resolver\s*=(?!=)|['"]a365_(?:contextual_)?token_resolver['"]\s*:/, - dotnet: /\b(?:Contextual)?TokenResolver\s*=(?!=)/, + dotnet: /\.\s*Agent365\s*\.\s*(?:Exporter\s*\.\s*)?(?:Contextual)?TokenResolver\s*=(?!=)/, }; const S2S_ROUTE_OPTION = { node: /\buseS2SEndpoint\s*:\s*true\b/, python: /\ba365_use_s2s_endpoint\s*=\s*True\b|['"]a365_use_s2s_endpoint['"]\s*:\s*True\b/, - dotnet: /\bUseS2SEndpoint\s*=\s*true\b/, + dotnet: /\.\s*Agent365\s*\.\s*(?:Exporter\s*\.\s*)?UseS2SEndpoint\s*=\s*true\b/, }; function stripComments(content, language) { @@ -138,7 +140,7 @@ function stripComments(content, language) { // Returns the text from the bracket at `open` through its matching close bracket. function bracketBlock(content, open) { - const close = { '(': ')', '{': '}' }[content[open]]; + const close = { '(': ')', '{': '}', '[': ']' }[content[open]]; let depth = 0; for (let i = open; close && i < content.length; i++) { if (content[i] === content[open]) depth++; @@ -156,34 +158,104 @@ function callArguments(content, name) { return calls; } +function escapeRegExp(text) { + return text.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); +} + +// The `{...}` or `(...)` initializers assigned to `name` in `content`. +function initializerBlocks(content, name) { + const initializer = new RegExp(`(?:^|[^\\w$])${escapeRegExp(name)}\\s*(?::[^=\\n]*)?=\\s*(?:dict\\s*)?([({])`, 'gm'); + const blocks = []; + for (let m = initializer.exec(content); m; m = initializer.exec(content)) { + blocks.push(bracketBlock(content, m.index + m[0].length - 1)); + } + return blocks; +} + +// Initializers of the variables `text` references, plus Python `name["key"] = value` entries. +function referencedInitializers(content, text, language) { + const blocks = []; + for (const name of new Set(text.match(/[A-Za-z_$][\w$]*/g) || [])) { + blocks.push(...initializerBlocks(content, name)); + if (language === 'python') { + const keyAssignment = new RegExp(`\\b${escapeRegExp(name)}\\s*\\[\\s*(['"][^'"\\n]+['"])\\s*\\]\\s*=(?!=)\\s*([^\\n]*)`, 'g'); + for (let m = keyAssignment.exec(content); m; m = keyAssignment.exec(content)) blocks.push(`${m[1]}: ${m[2]}`); + } + } + return blocks; +} + +// Python: the call's own keyword arguments. Nested calls and literals are dropped, except `**{...}` +// and `**dict(...)` spreads, whose entries are keyword arguments too. +function pythonKeywordArguments(args) { + let keywords = ''; + for (let i = 1; i < args.length - 1; i++) { + const group = '([{'.includes(args[i]) ? bracketBlock(args, i) : ''; + if (group) { + if (/\*\*\s*(?:dict\s*)?$/.test(args.slice(1, i))) keywords += group; + i += group.length - 1; + } else { + keywords += args[i]; + } + } + return keywords; +} + +// Node.js: the `a365` options objects in `text`, inline (`a365: {...}`), by variable (`a365: options`), +// or shorthand (`{ a365 }`). Null when a value is a call that cannot be read statically. +function a365Objects(content, text) { + const objects = []; + const property = /(?:^|[{,\s])a365\s*:\s*/g; + for (let m = property.exec(text); m; m = property.exec(text)) { + const at = m.index + m[0].length; + if (text[at] === '{') { + objects.push(bracketBlock(text, at)); + continue; + } + const value = text.slice(at).match(/^[A-Za-z_$][\w$.]*(\s*\()?/); + if (!value || value[1]) return null; + objects.push(...initializerBlocks(content, value[0])); + } + if (/[{,]\s*a365\s*(?=[,}])/.test(text)) objects.push(...initializerBlocks(content, 'a365')); + return objects; +} + // True when a distro call in `files` passes an option matching `option`. function distroCallMatches(files, language, option) { const { call } = DISTRO_CALLS[language]; return files.some(file => { const content = stripComments(read(file), language); - return callArguments(content, call).some(args => { - if (option.test(args)) return true; - // .NET: a method group instead of an inline lambda configures the options elsewhere in the file. - if (language === 'dotnet' && !args.includes('=>')) return option.test(content); - for (const name of new Set(args.match(/[A-Za-z_$][\w$]*/g) || [])) { - const escaped = name.replace(/\$/g, '\\$'); - const initializer = new RegExp(`(?:^|[^\\w$])${escaped}\\s*(?::[^=\\n]*)?=\\s*(?:dict\\s*)?([({])`, 'gm'); - for (let m = initializer.exec(content); m; m = initializer.exec(content)) { - if (option.test(bracketBlock(content, m.index + m[0].length - 1))) return true; - } - if (language === 'python') { - // `kwargs["key"] = value` is read as the dict entry it adds. - const keyAssignment = new RegExp(`\\b${escaped}\\s*\\[\\s*(['"][^'"\\n]+['"])\\s*\\]\\s*=(?!=)\\s*([^\\n]*)`, 'g'); - for (let m = keyAssignment.exec(content); m; m = keyAssignment.exec(content)) { - if (option.test(`${m[1]}: ${m[2]}`)) return true; - } - } - } - return false; - }); + for (let index = content.indexOf(`${call}(`); index !== -1; index = content.indexOf(`${call}(`, index + 1)) { + const open = index + call.length; + const args = bracketBlock(content, open); + // Variables are initialized outside the call. Hiding the call keeps Python keyword arguments, + // which look like assignments, from being read as initializers. + const outside = content.slice(0, open) + ' '.repeat(args.length) + content.slice(open + args.length); + if (callPassesOption(content, outside, args, language, option)) return true; + } + return false; }); } +function callPassesOption(content, outside, args, language, option) { + if (language === 'dotnet') { + // A method group, or a callback that never touches Agent365, configures the options elsewhere. + if (!args.includes('=>') || !/\bAgent365\b/.test(args)) return option.test(content); + return [args, ...referencedInitializers(outside, args, language)].some(text => option.test(text)); + } + const texts = [language === 'python' ? pythonKeywordArguments(args) : args, ...referencedInitializers(outside, args, language)]; + if (language === 'python') return texts.some(text => option.test(text)); + const objects = []; + for (const text of texts) { + const found = a365Objects(outside, text); + if (found === null) return option.test(content); + objects.push(...found); + } + if (objects.length === 0) return option.test(content); + return objects.some(object => + option.test(object) || referencedInitializers(outside, object, language).some(init => option.test(init))); +} + function distroCallHasResolver(files, language) { return distroCallMatches(files, language, TOKEN_RESOLVER_OPTION[language]); } diff --git a/plugins/agent365/hooks/stop/validate-instrument-observability.js b/plugins/agent365/hooks/stop/validate-instrument-observability.js index 01099ac..bba234d 100644 --- a/plugins/agent365/hooks/stop/validate-instrument-observability.js +++ b/plugins/agent365/hooks/stop/validate-instrument-observability.js @@ -39,10 +39,12 @@ function anyFileMatches(files, regex) { } // Distro-call checks shared verbatim by both stop hooks and the standalone scanner. An option counts -// only when the distro call's own arguments pass it: an explicit or shorthand property (Node.js), a -// keyword or `**{...}` key (Python), or an assignment in the options callback (.NET). A variable passed -// to the call is followed one level to its initializer in the same file. Comments are ignored, so a -// commented-out option, an unused import, or an unrelated object elsewhere does not count. +// only where the SDK reads it: in the `a365` options object passed to useMicrosoftOpenTelemetry +// (Node.js), as a keyword argument of use_microsoft_opentelemetry (Python), or on `.Agent365` in the +// UseMicrosoftOpenTelemetry options callback (.NET). A variable is followed one level to its +// initializer in the same file. Comments are ignored, so a commented-out option, an unused import, or +// an unrelated object does not count. When the options cannot be located statically (a method group, +// a factory call, or a callback that never touches Agent365), the file as a whole is checked instead. const DISTRO_CALLS = { node: { call: 'useMicrosoftOpenTelemetry', comments: /\/\*[\s\S]*?\*\/|(^|[\s,{;(])\/\/[^\n]*/gm }, python: { call: 'use_microsoft_opentelemetry', comments: /(^|\s)#[^\n]*/gm }, @@ -52,13 +54,13 @@ const DISTRO_CALLS = { const TOKEN_RESOLVER_OPTION = { node: /[{,]\s*tokenResolver\s*(?=[,}])|\btokenResolver\s*:/, python: /\ba365_(?:contextual_)?token_resolver\s*=(?!=)|['"]a365_(?:contextual_)?token_resolver['"]\s*:/, - dotnet: /\b(?:Contextual)?TokenResolver\s*=(?!=)/, + dotnet: /\.\s*Agent365\s*\.\s*(?:Exporter\s*\.\s*)?(?:Contextual)?TokenResolver\s*=(?!=)/, }; const S2S_ROUTE_OPTION = { node: /\buseS2SEndpoint\s*:\s*true\b/, python: /\ba365_use_s2s_endpoint\s*=\s*True\b|['"]a365_use_s2s_endpoint['"]\s*:\s*True\b/, - dotnet: /\bUseS2SEndpoint\s*=\s*true\b/, + dotnet: /\.\s*Agent365\s*\.\s*(?:Exporter\s*\.\s*)?UseS2SEndpoint\s*=\s*true\b/, }; function stripComments(content, language) { @@ -67,7 +69,7 @@ function stripComments(content, language) { // Returns the text from the bracket at `open` through its matching close bracket. function bracketBlock(content, open) { - const close = { '(': ')', '{': '}' }[content[open]]; + const close = { '(': ')', '{': '}', '[': ']' }[content[open]]; let depth = 0; for (let i = open; close && i < content.length; i++) { if (content[i] === content[open]) depth++; @@ -85,34 +87,104 @@ function callArguments(content, name) { return calls; } +function escapeRegExp(text) { + return text.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); +} + +// The `{...}` or `(...)` initializers assigned to `name` in `content`. +function initializerBlocks(content, name) { + const initializer = new RegExp(`(?:^|[^\\w$])${escapeRegExp(name)}\\s*(?::[^=\\n]*)?=\\s*(?:dict\\s*)?([({])`, 'gm'); + const blocks = []; + for (let m = initializer.exec(content); m; m = initializer.exec(content)) { + blocks.push(bracketBlock(content, m.index + m[0].length - 1)); + } + return blocks; +} + +// Initializers of the variables `text` references, plus Python `name["key"] = value` entries. +function referencedInitializers(content, text, language) { + const blocks = []; + for (const name of new Set(text.match(/[A-Za-z_$][\w$]*/g) || [])) { + blocks.push(...initializerBlocks(content, name)); + if (language === 'python') { + const keyAssignment = new RegExp(`\\b${escapeRegExp(name)}\\s*\\[\\s*(['"][^'"\\n]+['"])\\s*\\]\\s*=(?!=)\\s*([^\\n]*)`, 'g'); + for (let m = keyAssignment.exec(content); m; m = keyAssignment.exec(content)) blocks.push(`${m[1]}: ${m[2]}`); + } + } + return blocks; +} + +// Python: the call's own keyword arguments. Nested calls and literals are dropped, except `**{...}` +// and `**dict(...)` spreads, whose entries are keyword arguments too. +function pythonKeywordArguments(args) { + let keywords = ''; + for (let i = 1; i < args.length - 1; i++) { + const group = '([{'.includes(args[i]) ? bracketBlock(args, i) : ''; + if (group) { + if (/\*\*\s*(?:dict\s*)?$/.test(args.slice(1, i))) keywords += group; + i += group.length - 1; + } else { + keywords += args[i]; + } + } + return keywords; +} + +// Node.js: the `a365` options objects in `text`, inline (`a365: {...}`), by variable (`a365: options`), +// or shorthand (`{ a365 }`). Null when a value is a call that cannot be read statically. +function a365Objects(content, text) { + const objects = []; + const property = /(?:^|[{,\s])a365\s*:\s*/g; + for (let m = property.exec(text); m; m = property.exec(text)) { + const at = m.index + m[0].length; + if (text[at] === '{') { + objects.push(bracketBlock(text, at)); + continue; + } + const value = text.slice(at).match(/^[A-Za-z_$][\w$.]*(\s*\()?/); + if (!value || value[1]) return null; + objects.push(...initializerBlocks(content, value[0])); + } + if (/[{,]\s*a365\s*(?=[,}])/.test(text)) objects.push(...initializerBlocks(content, 'a365')); + return objects; +} + // True when a distro call in `files` passes an option matching `option`. function distroCallMatches(files, language, option) { const { call } = DISTRO_CALLS[language]; return files.some(file => { const content = stripComments(read(file), language); - return callArguments(content, call).some(args => { - if (option.test(args)) return true; - // .NET: a method group instead of an inline lambda configures the options elsewhere in the file. - if (language === 'dotnet' && !args.includes('=>')) return option.test(content); - for (const name of new Set(args.match(/[A-Za-z_$][\w$]*/g) || [])) { - const escaped = name.replace(/\$/g, '\\$'); - const initializer = new RegExp(`(?:^|[^\\w$])${escaped}\\s*(?::[^=\\n]*)?=\\s*(?:dict\\s*)?([({])`, 'gm'); - for (let m = initializer.exec(content); m; m = initializer.exec(content)) { - if (option.test(bracketBlock(content, m.index + m[0].length - 1))) return true; - } - if (language === 'python') { - // `kwargs["key"] = value` is read as the dict entry it adds. - const keyAssignment = new RegExp(`\\b${escaped}\\s*\\[\\s*(['"][^'"\\n]+['"])\\s*\\]\\s*=(?!=)\\s*([^\\n]*)`, 'g'); - for (let m = keyAssignment.exec(content); m; m = keyAssignment.exec(content)) { - if (option.test(`${m[1]}: ${m[2]}`)) return true; - } - } - } - return false; - }); + for (let index = content.indexOf(`${call}(`); index !== -1; index = content.indexOf(`${call}(`, index + 1)) { + const open = index + call.length; + const args = bracketBlock(content, open); + // Variables are initialized outside the call. Hiding the call keeps Python keyword arguments, + // which look like assignments, from being read as initializers. + const outside = content.slice(0, open) + ' '.repeat(args.length) + content.slice(open + args.length); + if (callPassesOption(content, outside, args, language, option)) return true; + } + return false; }); } +function callPassesOption(content, outside, args, language, option) { + if (language === 'dotnet') { + // A method group, or a callback that never touches Agent365, configures the options elsewhere. + if (!args.includes('=>') || !/\bAgent365\b/.test(args)) return option.test(content); + return [args, ...referencedInitializers(outside, args, language)].some(text => option.test(text)); + } + const texts = [language === 'python' ? pythonKeywordArguments(args) : args, ...referencedInitializers(outside, args, language)]; + if (language === 'python') return texts.some(text => option.test(text)); + const objects = []; + for (const text of texts) { + const found = a365Objects(outside, text); + if (found === null) return option.test(content); + objects.push(...found); + } + if (objects.length === 0) return option.test(content); + return objects.some(object => + option.test(object) || referencedInitializers(outside, object, language).some(init => option.test(init))); +} + function distroCallHasResolver(files, language) { return distroCallMatches(files, language, TOKEN_RESOLVER_OPTION[language]); } diff --git a/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js b/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js index ec838eb..ed0350c 100644 --- a/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js +++ b/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js @@ -108,10 +108,12 @@ function callBlocks(content, name) { } // Distro-call checks shared verbatim by both stop hooks and the standalone scanner. An option counts -// only when the distro call's own arguments pass it: an explicit or shorthand property (Node.js), a -// keyword or `**{...}` key (Python), or an assignment in the options callback (.NET). A variable passed -// to the call is followed one level to its initializer in the same file. Comments are ignored, so a -// commented-out option, an unused import, or an unrelated object elsewhere does not count. +// only where the SDK reads it: in the `a365` options object passed to useMicrosoftOpenTelemetry +// (Node.js), as a keyword argument of use_microsoft_opentelemetry (Python), or on `.Agent365` in the +// UseMicrosoftOpenTelemetry options callback (.NET). A variable is followed one level to its +// initializer in the same file. Comments are ignored, so a commented-out option, an unused import, or +// an unrelated object does not count. When the options cannot be located statically (a method group, +// a factory call, or a callback that never touches Agent365), the file as a whole is checked instead. const DISTRO_CALLS = { node: { call: 'useMicrosoftOpenTelemetry', comments: /\/\*[\s\S]*?\*\/|(^|[\s,{;(])\/\/[^\n]*/gm }, python: { call: 'use_microsoft_opentelemetry', comments: /(^|\s)#[^\n]*/gm }, @@ -121,13 +123,13 @@ const DISTRO_CALLS = { const TOKEN_RESOLVER_OPTION = { node: /[{,]\s*tokenResolver\s*(?=[,}])|\btokenResolver\s*:/, python: /\ba365_(?:contextual_)?token_resolver\s*=(?!=)|['"]a365_(?:contextual_)?token_resolver['"]\s*:/, - dotnet: /\b(?:Contextual)?TokenResolver\s*=(?!=)/, + dotnet: /\.\s*Agent365\s*\.\s*(?:Exporter\s*\.\s*)?(?:Contextual)?TokenResolver\s*=(?!=)/, }; const S2S_ROUTE_OPTION = { node: /\buseS2SEndpoint\s*:\s*true\b/, python: /\ba365_use_s2s_endpoint\s*=\s*True\b|['"]a365_use_s2s_endpoint['"]\s*:\s*True\b/, - dotnet: /\bUseS2SEndpoint\s*=\s*true\b/, + dotnet: /\.\s*Agent365\s*\.\s*(?:Exporter\s*\.\s*)?UseS2SEndpoint\s*=\s*true\b/, }; function stripComments(content, language) { @@ -136,7 +138,7 @@ function stripComments(content, language) { // Returns the text from the bracket at `open` through its matching close bracket. function bracketBlock(content, open) { - const close = { '(': ')', '{': '}' }[content[open]]; + const close = { '(': ')', '{': '}', '[': ']' }[content[open]]; let depth = 0; for (let i = open; close && i < content.length; i++) { if (content[i] === content[open]) depth++; @@ -154,34 +156,104 @@ function callArguments(content, name) { return calls; } +function escapeRegExp(text) { + return text.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); +} + +// The `{...}` or `(...)` initializers assigned to `name` in `content`. +function initializerBlocks(content, name) { + const initializer = new RegExp(`(?:^|[^\\w$])${escapeRegExp(name)}\\s*(?::[^=\\n]*)?=\\s*(?:dict\\s*)?([({])`, 'gm'); + const blocks = []; + for (let m = initializer.exec(content); m; m = initializer.exec(content)) { + blocks.push(bracketBlock(content, m.index + m[0].length - 1)); + } + return blocks; +} + +// Initializers of the variables `text` references, plus Python `name["key"] = value` entries. +function referencedInitializers(content, text, language) { + const blocks = []; + for (const name of new Set(text.match(/[A-Za-z_$][\w$]*/g) || [])) { + blocks.push(...initializerBlocks(content, name)); + if (language === 'python') { + const keyAssignment = new RegExp(`\\b${escapeRegExp(name)}\\s*\\[\\s*(['"][^'"\\n]+['"])\\s*\\]\\s*=(?!=)\\s*([^\\n]*)`, 'g'); + for (let m = keyAssignment.exec(content); m; m = keyAssignment.exec(content)) blocks.push(`${m[1]}: ${m[2]}`); + } + } + return blocks; +} + +// Python: the call's own keyword arguments. Nested calls and literals are dropped, except `**{...}` +// and `**dict(...)` spreads, whose entries are keyword arguments too. +function pythonKeywordArguments(args) { + let keywords = ''; + for (let i = 1; i < args.length - 1; i++) { + const group = '([{'.includes(args[i]) ? bracketBlock(args, i) : ''; + if (group) { + if (/\*\*\s*(?:dict\s*)?$/.test(args.slice(1, i))) keywords += group; + i += group.length - 1; + } else { + keywords += args[i]; + } + } + return keywords; +} + +// Node.js: the `a365` options objects in `text`, inline (`a365: {...}`), by variable (`a365: options`), +// or shorthand (`{ a365 }`). Null when a value is a call that cannot be read statically. +function a365Objects(content, text) { + const objects = []; + const property = /(?:^|[{,\s])a365\s*:\s*/g; + for (let m = property.exec(text); m; m = property.exec(text)) { + const at = m.index + m[0].length; + if (text[at] === '{') { + objects.push(bracketBlock(text, at)); + continue; + } + const value = text.slice(at).match(/^[A-Za-z_$][\w$.]*(\s*\()?/); + if (!value || value[1]) return null; + objects.push(...initializerBlocks(content, value[0])); + } + if (/[{,]\s*a365\s*(?=[,}])/.test(text)) objects.push(...initializerBlocks(content, 'a365')); + return objects; +} + // True when a distro call in `files` passes an option matching `option`. function distroCallMatches(files, language, option) { const { call } = DISTRO_CALLS[language]; return files.some(file => { const content = stripComments(read(file), language); - return callArguments(content, call).some(args => { - if (option.test(args)) return true; - // .NET: a method group instead of an inline lambda configures the options elsewhere in the file. - if (language === 'dotnet' && !args.includes('=>')) return option.test(content); - for (const name of new Set(args.match(/[A-Za-z_$][\w$]*/g) || [])) { - const escaped = name.replace(/\$/g, '\\$'); - const initializer = new RegExp(`(?:^|[^\\w$])${escaped}\\s*(?::[^=\\n]*)?=\\s*(?:dict\\s*)?([({])`, 'gm'); - for (let m = initializer.exec(content); m; m = initializer.exec(content)) { - if (option.test(bracketBlock(content, m.index + m[0].length - 1))) return true; - } - if (language === 'python') { - // `kwargs["key"] = value` is read as the dict entry it adds. - const keyAssignment = new RegExp(`\\b${escaped}\\s*\\[\\s*(['"][^'"\\n]+['"])\\s*\\]\\s*=(?!=)\\s*([^\\n]*)`, 'g'); - for (let m = keyAssignment.exec(content); m; m = keyAssignment.exec(content)) { - if (option.test(`${m[1]}: ${m[2]}`)) return true; - } - } - } - return false; - }); + for (let index = content.indexOf(`${call}(`); index !== -1; index = content.indexOf(`${call}(`, index + 1)) { + const open = index + call.length; + const args = bracketBlock(content, open); + // Variables are initialized outside the call. Hiding the call keeps Python keyword arguments, + // which look like assignments, from being read as initializers. + const outside = content.slice(0, open) + ' '.repeat(args.length) + content.slice(open + args.length); + if (callPassesOption(content, outside, args, language, option)) return true; + } + return false; }); } +function callPassesOption(content, outside, args, language, option) { + if (language === 'dotnet') { + // A method group, or a callback that never touches Agent365, configures the options elsewhere. + if (!args.includes('=>') || !/\bAgent365\b/.test(args)) return option.test(content); + return [args, ...referencedInitializers(outside, args, language)].some(text => option.test(text)); + } + const texts = [language === 'python' ? pythonKeywordArguments(args) : args, ...referencedInitializers(outside, args, language)]; + if (language === 'python') return texts.some(text => option.test(text)); + const objects = []; + for (const text of texts) { + const found = a365Objects(outside, text); + if (found === null) return option.test(content); + objects.push(...found); + } + if (objects.length === 0) return option.test(content); + return objects.some(object => + option.test(object) || referencedInitializers(outside, object, language).some(init => option.test(init))); +} + function distroCallHasResolver(files, language) { return distroCallMatches(files, language, TOKEN_RESOLVER_OPTION[language]); } diff --git a/tests/validate-a365-code-validator.test.js b/tests/validate-a365-code-validator.test.js index cb3f4ad..cccc2e1 100644 --- a/tests/validate-a365-code-validator.test.js +++ b/tests/validate-a365-code-validator.test.js @@ -663,6 +663,33 @@ token = await auth.exchange_token(context, scopes=["ea9ffc3e-8a23-4a7d-836d-234d } }); + test('route and resolver options count only where the SDK reads them, in both scanners', () => { + const csproj = ''; + const misplaced = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': "useMicrosoftOpenTelemetry({\n a365: { enabled: true, enableObservabilityExporter: true },\n instrumentationOptions: { custom: { useS2SEndpoint: true, tokenResolver: appTokenResolver } },\n});", + 'Agent.csproj': csproj, + 'Program.cs': 'builder.UseMicrosoftOpenTelemetry(o => { workloadOptions.UseS2SEndpoint = true; workloadOptions.TokenResolver = r; });', + 'requirements.txt': 'microsoft-opentelemetry>=1.1.0\n', + 'host.py': 'use_microsoft_opentelemetry(enable_a365=True, a365_enable_observability_exporter=True, instrumentation_options=dict(a365_use_s2s_endpoint=True, a365_token_resolver=OBS_TOKENS.resolve))', + }); + const ids = list => list.filter(id => /obs-delegated-route|s2s-endpoint-not-set|token-resolver-missing/.test(id)).sort(); + try { + for (const scanner of [VALIDATOR, STANDALONE]) { + assert.deepEqual(ids(findingIds(runValidator(scanner, misplaced))), [ + 'dotnet-obs-delegated-route', + 'dotnet-obs-token-resolver-missing', + 'node-obs-delegated-route', + 'node-obs-token-resolver-missing', + 'python-obs-token-resolver-missing', + 'python-s2s-endpoint-not-set', + ], scanner); + } + } finally { + cleanup(misplaced); + } + }); + test('app-only S2S wiring in all three languages produces no delegated-telemetry findings in either scanner', () => { const dir = createFixture({ 'package.json': NODE_PKG, diff --git a/tests/validate-observability.test.js b/tests/validate-observability.test.js index c3068c5..442a1af 100644 --- a/tests/validate-observability.test.js +++ b/tests/validate-observability.test.js @@ -892,6 +892,83 @@ useMicrosoftOpenTelemetry(otelOptions); } finally { cleanup(dir); } }); + test('Node.js route and resolver on another options section → reports both requirements', () => { + const dir = createFixture({ + ...NODEJS_DISTRO_VALID, + 'src/index.ts': NODEJS_DISTRO_VALID['src/index.ts'].replace( + '{ a365: { enabled: true, enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver: appTokenResolver } }', + '{ a365: { enabled: true, enableObservabilityExporter: true }, instrumentationOptions: { custom: { useS2SEndpoint: true, tokenResolver: appTokenResolver } } }'), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /S2S route in every auth mode/); + assert.match(r.reason, /no a365 tokenResolver/); + } finally { cleanup(dir); } + }); + + test('Node.js shorthand a365 options built with a spread → ok', () => { + const dir = createFixture({ + ...NODEJS_DISTRO_VALID, + 'src/index.ts': NODEJS_DISTRO_VALID['src/index.ts'].replace( + 'useMicrosoftOpenTelemetry({ a365: { enabled: true, enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver: appTokenResolver } });', + 'const baseA365 = { useS2SEndpoint: true, tokenResolver: appTokenResolver };\nconst a365 = { ...baseA365, enabled: true, enableObservabilityExporter: true };\nuseMicrosoftOpenTelemetry({ a365 });'), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, true, r.reason); + } finally { cleanup(dir); } + }); + + test('Node.js options built by a factory call → checks the file', () => { + const dir = createFixture({ + ...NODEJS_DISTRO_VALID, + 'src/index.ts': NODEJS_DISTRO_VALID['src/index.ts'].replace( + 'useMicrosoftOpenTelemetry({ a365: { enabled: true, enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver: appTokenResolver } });', + 'useMicrosoftOpenTelemetry(buildOtelOptions());\nfunction buildOtelOptions() {\n return { a365: { enabled: true, enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver: appTokenResolver } };\n}'), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, true, r.reason); + } finally { cleanup(dir); } + }); + + test('.NET route and resolver set on other options inside the callback → reports both requirements', () => { + const dir = createFixture({ + ...DOTNET_DISTRO_VALID, + 'Program.cs': 'builder.Services.AddSingleton();\nbuilder.UseMicrosoftOpenTelemetry(o =>\n{\n workloadOptions.UseS2SEndpoint = true;\n workloadOptions.TokenResolver = workloadTokens.ResolveAsync;\n});', + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /S2S route in every auth mode/); + assert.match(r.reason, /without o\.Agent365\.TokenResolver/); + } finally { cleanup(dir); } + }); + + test('.NET options configured by a method the callback calls → ok', () => { + const dir = createFixture({ + ...DOTNET_DISTRO_VALID, + 'Program.cs': 'builder.UseMicrosoftOpenTelemetry(o => ConfigureTelemetry(o));\nstatic void ConfigureTelemetry(MicrosoftOpenTelemetryOptions o)\n{\n o.Agent365.UseS2SEndpoint = true;\n o.Agent365.TokenResolver = (agentId, tenantId) => obsTokens?.ResolveAsync(agentId, tenantId) ?? Task.FromResult(null);\n}', + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, true, r.reason); + } finally { cleanup(dir); } + }); + + test('Python route keyword inside a nested argument → reports the S2S route requirement', () => { + const dir = createFixture({ + ...PYTHON_DISTRO_VALID, + 'host_agent_server.py': PYTHON_DISTRO_VALID['host_agent_server.py'].replace(' a365_use_s2s_endpoint=True,', ' instrumentation_options=dict(a365_use_s2s_endpoint=True),'), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /S2S route in every auth mode.*a365_use_s2s_endpoint=True/); + } finally { cleanup(dir); } + }); + test('Python distro passing the resolver through a kwargs dict → ok', () => { const dir = createFixture({ ...PYTHON_DISTRO_VALID, From bd7361000ed9c6b4eb177b71ea96c4cb14f1cbad Mon Sep 17 00:00:00 2001 From: Krishnadheeraj <12496535+DheerajPannala@users.noreply.github.com> Date: Mon, 28 Sep 2026 17:20:29 +0100 Subject: [PATCH 06/12] Address S2S observability review feedback - tighten delegated refresh detection and add parity tests - harden app-only resolver scaffolds with token validation and backoff - align migration, CLI, and AI Teammate guidance with review feedback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5cbf5f6b-cc40-4b7e-a591-65848db73a12 --- .github/copilot-instructions.md | 6 +- AGENTS.md | 8 +- CLAUDE.md | 9 +- .../stop/validate-a365-code-validator.js | 40 ++++++- .../stop/validate-instrument-observability.js | 39 ++++++- plugins/agent365/shared/agent-detection.md | 2 +- .../skills/a365-code-validator/SKILL.md | 9 +- .../references/a365-code-validator.js | 39 ++++++- .../references/validation-checklist.md | 3 +- .../skills/instrument-observability/SKILL.md | 25 +++-- .../references/dotnet-observability.md | 79 ++++++++++++-- .../references/nodejs-observability.md | 46 ++++++-- .../references/python-observability.md | 103 ++++++++++++------ .../agent365/skills/make-a365-agent/SKILL.md | 2 +- .../agent365/skills/make-ai-teammate/SKILL.md | 2 +- .../references/deploy-pipeline.md | 5 +- .../references/nodejs-ai-teammate.md | 4 +- .../purview-dlp-integration/assets/purview.py | 4 +- tests/validate-a365-code-validator.test.js | 26 ++++- tests/validate-observability.test.js | 26 ++++- 20 files changed, 368 insertions(+), 109 deletions(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 8e4bb2c..a97ad80 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -164,7 +164,7 @@ wrapping. 2. Shows a dry-run preview of all `a365` operations before applying anything 3. Runs `a365 setup all` — creates the Blueprint and Entra ID permissions (add `--m365` for CEA agents; run `a365 setup permissions bot` after for Messaging Bot API grants). Supports `--authmode obo|s2s` for non-AI Teammate agents to control permission grant type; **never passes `--authmode` with `--aiteammate`** (AI Teammate uses the Agentic User identity — agent's own M365 identity, not the caller's token; `--authmode` flag not supported with `--aiteammate`). Skipped entirely when `reuseBlueprint = true`. Handles WAM prompts — if a native sign-in dialog appears, instructs user to complete it without killing the process. Auto-falls back to device code flow if blocked by Conditional Access Policy. 4. After setup, always offers `instrument-observability` as an optional add-on; offers `add-workiq-tools` only when `authMode ≠ s2s` — WorkIQ is silently skipped for S2S agents (requires a user token) -5. No Observability API permission is needed: recent `a365 setup all` versions no longer request `Agent365.Observability.OtelWrite` (or its consent) for blueprint agents. Telemetry is exported over the S2S route with an app-only token, and the route authorizes the registered agent instance; setup exits 1 when registration fails or cannot be verified. Other permission grants (Graph, Bot API, custom resources) require Global Administrator consent — `a365 setup all` automatically prints next-steps (typically a PowerShell script) when the developer is not a GA. There is no separate `setup admin` subcommand; the skill displays the printed script verbatim so the user can hand it to a Global Admin +5. No Observability API permission is needed: newer `a365 setup all` versions skip `Agent365.Observability.OtelWrite` for blueprint agents and exit 1 when registration fails or cannot be verified; older versions may still grant OtelWrite (harmless) and may exit 0 after a failed registration, so check setup output and rerun `a365 setup all --agent-registration-only` if needed. Telemetry is exported over the S2S route with an app-only token, and the route authorizes the registered agent instance. Other permission grants (Graph, Bot API, custom resources) require Global Administrator consent — `a365 setup all` automatically prints next-steps (typically a PowerShell script) when the developer is not a GA. There is no separate `setup admin` subcommand; the skill displays the printed script verbatim so the user can hand it to a Global Admin **Normally delegated to from `a365-setup`** after CLI and Azure prerequisites are confirmed. Can also be invoked directly. @@ -249,7 +249,7 @@ wrapping. **Summary of what this skill does:** 1. Loads detection cache; asks a two-stage question (agent kind + auth mode) if not already cached; writes `agentType`+`authMode` back to `.a365-workspace-detection.local.json` so `add-workiq-tools` and future runs skip re-asking 2. Installs the observability packages: unified distros for all paths — `Microsoft.OpenTelemetry` for .NET, `@microsoft/opentelemetry` for Node.js, `microsoft-opentelemetry` for Python. Legacy individual packages (`Microsoft.Agents.A365.Observability.*`, `@microsoft/agents-a365-*`, `microsoft-agents-a365-observability-*`) still accepted by the validator but no longer generated. -3. **Telemetry always uses the S2S route with an app-only token (every auth mode).** The S2S route rejects delegated (`scp`) tokens, so OBO / Agentic User tokens are used only for workload calls (MCP / Graph). No OBS permission or admin consent is needed for registered blueprint agent instances (AI Teammates: complete the `OtelWrite` application-role step `a365 setup all --aiteammate` prints). **OBO path** (`obo` / `agentic-user`, including AI Teammate): writes an app-only token resolver that reuses the hosting connection's blueprint credential (FMI assertion for the turn's agent identity, then agent-identity `client_credentials` for the OBS scope). The files are `Observability/AgentAppTokenResolver.cs` wired as `o.Agent365.TokenResolver`, `observability/app-token-resolver.ts` as `tokenResolver`, or `observability/app_token_resolver.py` whose `OBS_TOKENS.resolve` is `a365_token_resolver`, with a per-turn `OBS_TOKENS.prefetch(...)`. It sets the S2S route flag (`o.Agent365.UseS2SEndpoint = true` on `Microsoft.OpenTelemetry` 1.0.3+, `useS2SEndpoint: true`, `a365_use_s2s_endpoint=True`). There is **no** per-turn `RegisterObservability(..., AgenticTokenStruct)` / `refreshObservabilityToken(..., authorization)` / `exchange_token(...)` for telemetry; legacy wiring like that is migrated. Also requires `.UseOpenTelemetry()` on the `IChatClient` so the AI SDK emits `gen_ai` spans for `InvokeAgentScope` to anchor. .NET agent id: `Activity.GetAgenticInstanceId()` for agentic turns; other turns fall back to the provisioned agent identity in `Agent365Observability:AgentId` (never the blueprint). Observability is gracefully skipped when no real (agent, tenant) tuple exists — avoids polluting traces with `Guid.Empty`-grouped orphan spans the exporter cannot authenticate. +3. **Telemetry always uses the S2S route with an app-only token (every auth mode).** The S2S route rejects delegated (`scp`) tokens, so OBO / Agentic User tokens are used only for workload calls (MCP / Graph). No OBS permission or admin consent is needed for registered blueprint agent instances (AI Teammates: complete the `OtelWrite` application-role step `a365 setup all --aiteammate` prints). **OBO path** (`obo` / `agentic-user`, including AI Teammate): writes an app-only token resolver that reuses the hosting connection's blueprint credential (FMI assertion for the turn's agent identity, then agent-identity `client_credentials` for the OBS scope). The files are `Observability/AgentAppTokenResolver.cs` wired as `o.Agent365.TokenResolver`, `observability/app-token-resolver.ts` as `tokenResolver`, or `observability/app_token_resolver.py` whose `OBS_TOKENS.resolve` is `a365_token_resolver`, with a per-turn `OBS_TOKENS.prefetch(...)`. It sets the S2S route flag (`o.Agent365.UseS2SEndpoint = true` on `Microsoft.OpenTelemetry` 1.0.3+, `useS2SEndpoint: true`, `a365_use_s2s_endpoint=True`). There is **no** per-turn `RegisterObservability(..., AgenticTokenStruct)` / `refreshObservabilityToken(...)` / `exchange_token(...)` for telemetry; legacy wiring like that is migrated. Also requires `.UseOpenTelemetry()` on the `IChatClient` so the AI SDK emits `gen_ai` spans for `InvokeAgentScope` to anchor. .NET agent id: `Activity.GetAgenticInstanceId()` for agentic turns; other turns fall back to the provisioned agent identity in `Agent365Observability:AgentId` (never the blueprint). Observability is gracefully skipped when no real (agent, tenant) tuple exists — avoids polluting traces with `Guid.Empty`-grouped orphan spans the exporter cannot authenticate. 4. **S2S path (all languages)**: Creates a scaffold token-service file that acquires/refreshes the Observability API token (`api://9b975845-388f-4429-889e-eab1ef63949c/.default`) via MSAL with FMI path support every 50 min. - **.NET**: creates `Observability/ObservabilityServiceExtensions.cs` + `Observability/ObservabilityTokenService.cs`; uses MSAL `ConfidentialClientApplicationBuilder` with `.WithFmiPath()` for FMI 3-hop chain; wires `UseMicrosoftOpenTelemetry()` (with `o.Agent365.UseS2SEndpoint = true` — `o.Agent365.Exporter.UseS2SEndpoint` on 1.0.2 and earlier) + `AddAgent365Observability()`; in the message handler uses `new BaggageBuilder().FromTurnContext(turnContext).Build()` (separate `using var`) and `InvokeAgentScope.Start(request, new InvokeAgentScopeDetails(endpoint: new Uri(...)), agentDetails, callerDetails)` (separate `using var`) — **NOT chained; `FromTurnContext()` is a `BaggageBuilder` extension only**; `CallerDetails` with blueprint sponsor identity is **required** for S2S traces to appear - **Node.js** (`@microsoft/opentelemetry` 1.0 GA): creates `observability/observability-token-service.ts` (exports `startTokenService()`) + `observability/token-cache.ts` (exports `tokenResolver`); for client-secret Hop 1+2 uses direct HTTP POST with `fmi_path` form parameter (MSAL Node.js doesn't serialize `fmiPath`), MSAL for Hop 3; calls `useMicrosoftOpenTelemetry({ a365: { enabled: true, enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver } })` — `useS2SEndpoint` is a first-class option in 1.0+; do NOT use the old hand-rolled `spanProcessors` workaround @@ -393,7 +393,7 @@ All code added by observability instrumentation must be marked with the language - Python install: use `pip3 install ... 2>/dev/null || pip install ...` for cross-platform (pip3 on macOS/Linux, fall back to pip on Windows). No `--pre` flag needed — packages are GA. - .NET S2S: `FromTurnContext()` is only on `BaggageBuilder` — never chain it on `InvokeAgentScope.Start()` - .NET S2S: `InvokeAgentScopeDetails` has no parameterless constructor — always pass `endpoint: new Uri(...)` -- All languages, every auth mode: select the S2S route (`o.Agent365.UseS2SEndpoint = true` on `Microsoft.OpenTelemetry` 1.0.3+ — `o.Agent365.Exporter.*` on 1.0.2 and earlier; `useS2SEndpoint: true`; `a365_use_s2s_endpoint=True`) and pass an app-only token resolver. Never register or refresh a delegated telemetry token (`RegisterObservability(..., AgenticTokenStruct, ...)`, `refreshObservabilityToken(..., authorization)`, `exchange_token(...)` for the observability scope) +- All languages, every auth mode: select the S2S route (`o.Agent365.UseS2SEndpoint = true` on `Microsoft.OpenTelemetry` 1.0.3+ — `o.Agent365.Exporter.*` on 1.0.2 and earlier; `useS2SEndpoint: true`; `a365_use_s2s_endpoint=True`) and pass an app-only token resolver. Never register or refresh a delegated telemetry token (`RegisterObservability(..., AgenticTokenStruct, ...)`, `refreshObservabilityToken(...)`, `exchange_token(...)` for the observability scope) All code added by WorkIQ wiring must be marked with the language-appropriate comment form: - C# / JavaScript / TypeScript: `// A365 WorkIQ — added by add-workiq-tools skill` diff --git a/AGENTS.md b/AGENTS.md index 2f37f67..19c02cd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -52,7 +52,7 @@ test-local (no prerequisite) **Phase 9.7.2d** validates environment configuration before either path proceeds. For prod: confirms `a365.generated.config.json` has `completed: true` and non-empty `resourceConsents` (else GA consent handoff is pending); confirms `.env`/`appsettings.json` has agentic-auth + LLM + observability vars; reminds the user that cloud env vars must be set at the cloud platform (`az webapp config appsettings set` / `eb setenv` / `gcloud run services update --set-env-vars`), not just locally; confirms HTTPS messaging endpoint. For local: confirms AgentsPlayground is installed and `.m365agentsplayground.yml` is configured when using agentic auth. Authoritative Microsoft Learn refs: [test-with-devtunnels](https://learn.microsoft.com/en-us/microsoft-agent-365/developer/test-with-devtunnels), [testing](https://learn.microsoft.com/en-us/microsoft-agent-365/developer/testing), [deploy-agent-azure](https://learn.microsoft.com/en-us/microsoft-agent-365/developer/deploy-agent-azure), [deploy-agent-aws](https://learn.microsoft.com/en-us/microsoft-agent-365/developer/deploy-agent-aws), [deploy-agent-gcp](https://learn.microsoft.com/en-us/microsoft-agent-365/developer/deploy-agent-gcp). **Automatically** runs `instrument-observability` (only when `has_obs = false`) and **optionally** offers `add-workiq-tools` (only when `has_workiq = false`). The skill does NOT hand-edit `manifest.json`. Reference: [Create agent instance — Microsoft Learn](https://learn.microsoft.com/en-us/microsoft-agent-365/developer/create-instance). `a365-setup` outputs a mandatory intro message, detects stack/language/CEA/`hasBlueprintConfig`, and the three skill-state flags **`has_aiteammate_structure`**, **`has_obs`**, **`has_workiq`** (the same primary flags that drive `make-ai-teammate` Phase 0C's 8-row matrix). Always updates the a365 CLI to latest (explicit exception to the ✅-skip rule), checks for an existing Azure CLI session before logging in, shows a ✅/❌ prerequisite summary and only processes ❌ missing tools. Asks the blueprint question (reuse vs fresh) then asks **capabilities first** — capability options are auto-filtered: Observability is hidden if `has_obs = true`, WorkIQ is hidden if `has_workiq = true`, the menu collapses to Register + WorkIQ when `(has_aiteammate_structure && has_obs)` (legacy "already an AI Teammate" route, computed inline — the legacy `hasAITeammateChanges` field is **derived, no longer stored**). If AI Teammate is selected, auth mode is skipped (always `agentic-user`); if non-AI Teammate, asks `obo` or `s2s`. Cache fields written: `agentStack`, `programmingLanguage`, `usesTeamsOrCopilot`, `hasBlueprintConfig`, `has_aiteammate_structure`, `has_obs`, `has_workiq`, `agentType`, `authMode`, `reuseBlueprint`, `existingBlueprintId`. Delegates: AI Teammate path → `make-ai-teammate`; all other paths → `make-a365-agent`. -`make-a365-agent` checks for an existing blueprint config before collecting inputs — if found, asks the developer whether to reuse (skips `a365 setup all`) or create fresh. Runs `a365 setup all --authmode obo|s2s` for non-AI Teammate paths; add `--m365` for CEA agents and follow with `a365 setup permissions bot`. No Observability API permission is needed: recent CLI versions no longer request `Agent365.Observability.OtelWrite` for blueprint agents, because telemetry is exported over the S2S route with an app-only token and the route authorizes the registered agent instance (setup exits 1 when registration fails or cannot be verified). Other permission grants (Graph, Bot API, custom resources) require Global Administrator consent — when the developer isn't a GA, `a365 setup all` automatically prints next-steps (typically a PowerShell script) for a GA to complete. There is no separate `setup admin` subcommand. Then conditionally invokes `instrument-observability` and `add-workiq-tools`. +`make-a365-agent` checks for an existing blueprint config before collecting inputs — if found, asks the developer whether to reuse (skips `a365 setup all`) or create fresh. Runs `a365 setup all --authmode obo|s2s` for non-AI Teammate paths; add `--m365` for CEA agents and follow with `a365 setup permissions bot` (this still configures the Observability API for the CEA flow; harmless because telemetry uses S2S). No Observability API permission is needed: newer CLI versions skip `Agent365.Observability.OtelWrite` for blueprint agents and exit 1 when registration fails or cannot be verified; older versions may still grant OtelWrite (harmless) and may exit 0 after a failed registration, so check setup output and rerun `a365 setup all --agent-registration-only` if needed. Other permission grants (Graph, Bot API, custom resources) require Global Administrator consent — when the developer isn't a GA, `a365 setup all` automatically prints next-steps (typically a PowerShell script) for a GA to complete. There is no separate `setup admin` subcommand. Then conditionally invokes `instrument-observability` and `add-workiq-tools`. `add-workiq-tools` and `instrument-observability` read `.a365-workspace-detection.local.json` to skip re-detection and verify prerequisites. `add-workiq-tools` Phase 0B includes a **framework support guard** that hard-stops on unsupported `(programmingLanguage, agentStack)` pairs (Python LangChain / Claude / CrewAI; Node.js Semantic Kernel / Google ADK) before any CLI command runs. Phase 4 branches on the cached `agentStack` into 11 framework-specific sub-sections (§4.1 .NET Agent Framework through §4.11 Python Azure AI Foundry); the stop-hook validator (`validate-add-workiq-tools.js`) is also framework-aware and requires the framework-matching symbol (e.g., `AddToolServersToAgentAsync` for .NET SK, `add_tool_servers_to_agent` for Python). **Phase 4.5 (gated)** offers the Word `@mention` notification handler when *both* gates pass: `programmingLanguage = NodeJS && agentStack = LangChain`, AND `mcp_WordServer` is in `ToolingManifest.json`. Wires `proactive: {}`, per-user conversation index, and a `NotificationType.WpxComment` branch — best-effort because no Microsoft Node.js sample is published yet. Best-effort branches (Python SK, Python/.NET Azure AI Foundry, and the Phase 4.5 @mention handler) mark all generated lines with `// A365 WorkIQ — best-effort wiring (verify against SDK source before production)`. `purview-dlp-integration` is **additive** and independent of observability / WorkIQ. It auto-discovers the app (client) id, display name, blueprint id, and current Graph scopes from `a365.config.json` + `a365.generated.config.json`, then asks only for what's missing (DLP policy choice, sensitive info type, admin UPN, agentic auth handler name). It detects the language and authentication and copies ONE generic env-driven guard (`assets/purview.ts` / `purview.py` / `purview.cs` for delegated agents; `assets/purview-s2s.ts` for Node.js client-secret FMI S2S; .NET is best-effort). Minimal wiring adds an **INPUT gate** before the LLM and optional **output auditing** before the reply; the supplied policy does not filter sensitive responses. Delegated guards use the agent's own token at `/me` with `Content.Process.User` appended by `Grant-DelegatedGraphScope.ps1`. Node.js S2S uses the agent identity FMI token at `/users/{sponsor}/...` with `Content.Process.All` from `Grant-ContentProcessAppRole.ps1`, never a blueprint app-only token or `admin-consent` on the blueprint. Do not switch managed-identity-only agents to client-secret authentication. Manual IDs replace config discovery, not missing authentication; a plain bot without a supported path must stop and route to `a365-setup` before edits. The guards always set `contentEntry.name` and fail closed by default. Policy choice remains new (`New-AiAppDlpPolicy.ps1`), existing (`-ListExisting`), or skip. Verify the `[purview] uploadText -> BLOCKED (… errors=0)` log and that the LLM was not called, not `DistributionStatus`. The stop-hook validator is report-first because disabled/policy-pending bring-up is valid. @@ -250,11 +250,11 @@ Skills reference shared docs via `Read ${CLAUDE_PLUGIN_ROOT}/shared/.md`. - AI Teammate → `obo` (signed-in user OBO) or `agentic-user` (agent's own Azure AD user — persistent M365 identity) - Agent (Non AI Teammate) → `obo` (On-Behalf-Of) or `s2s` (Service Principal, no user token) -Telemetry never goes through the auth handler: every `authMode` exports over the S2S route with an app-only token for the exporting agent identity (the S2S route rejects delegated `scp` tokens). The auth handler reference is for workload calls (MCP / Graph): .NET reads `authHandlerName` from config (`AgentApplication:AgenticAuthHandlerName`); Node.js uses `agentApplication.authorization` (the auth object); Python uses `auth_handler_id=self.auth_handler_name` (from config) — never hardcode `"AGENTIC"`. Agent IDs are resolved dynamically from TurnContext — .NET: `turnContext.Activity.GetAgenticInstanceId()` (service principal object ID); Node.js/Python: `recipient.agenticAppId` / `agentic_app_id`. Only non-agentic turns of non-AI-Teammate agents fall back to the provisioned agent identity in config (never the blueprint). Results are cached in `.a365-workspace-detection.local.json` under `agentType` and `authMode` fields so subsequent skill invocations skip re-questioning. If `authMode = s2s` and the skill is `add-workiq-tools`, the skill exits immediately — WorkIQ is not available for s2s agents (requires a delegated OBO token). +Telemetry never goes through the auth handler: every `authMode` exports over the S2S route with an app-only token for the exporting agent identity (the S2S route rejects delegated `scp` tokens). The auth handler reference is for workload calls (MCP / Graph): .NET reads `authHandlerName` from config (`AgentApplication:AgenticAuthHandlerName`); Node.js uses `agentApplication.authorization` (the auth object); Python uses `auth_handler_id=self.auth_handler_name` (from config) — never hardcode `"AGENTIC"`. Agent IDs are resolved dynamically from TurnContext — .NET: `turnContext.Activity.GetAgenticInstanceId()` returns `Recipient.AgenticAppId` (the agent identity's app/client ID); Node.js/Python: `recipient.agenticAppId` / `agentic_app_id`. Only non-agentic turns of non-AI-Teammate agents fall back to the provisioned agent identity in config (never the blueprint). Results are cached in `.a365-workspace-detection.local.json` under `agentType` and `authMode` fields so subsequent skill invocations skip re-questioning. If `authMode = s2s` and the skill is `add-workiq-tools`, the skill exits immediately — WorkIQ is not available for s2s agents (requires a delegated OBO token). -**App-only telemetry token (every auth mode):** For `obo` / `agentic-user`, `instrument-observability` creates an app-only token resolver that reuses the hosting connection's blueprint credential (FMI assertion for the turn's agent identity → agent-identity `client_credentials` for the OBS scope): `Observability/AgentAppTokenResolver.cs` (.NET), `observability/app-token-resolver.ts` (Node.js), or `observability/app_token_resolver.py` (Python — sync `resolve` plus a per-turn async `prefetch`). No per-turn delegated `RegisterObservability(..., AgenticTokenStruct)` / `refreshObservabilityToken(..., authorization)` / `exchange_token(...)` is generated, and existing delegated wiring is migrated. Every mode sets the S2S route flag (`o.Agent365.UseS2SEndpoint = true` — `o.Agent365.Exporter.*` on `Microsoft.OpenTelemetry` 1.0.2 and earlier; `useS2SEndpoint: true`; `a365_use_s2s_endpoint=True`). Registered blueprint agent instances need no `Agent365.Observability.OtelWrite` permission or admin consent (subject to service policy); AI Teammates complete the `OtelWrite` application-role step `a365 setup all --aiteammate` prints. +**App-only telemetry token (every auth mode):** For `obo` / `agentic-user`, `instrument-observability` creates an app-only token resolver that reuses the hosting connection's blueprint credential (FMI assertion for the turn's agent identity → agent-identity `client_credentials` for the OBS scope): `Observability/AgentAppTokenResolver.cs` (.NET), `observability/app-token-resolver.ts` (Node.js), or `observability/app_token_resolver.py` (Python — sync `resolve` plus a per-turn async `prefetch`). No per-turn delegated `RegisterObservability(..., AgenticTokenStruct)` / `refreshObservabilityToken(...)` / `exchange_token(...)` is generated, and existing delegated wiring is migrated. Every mode sets the S2S route flag (`o.Agent365.UseS2SEndpoint = true` — `o.Agent365.Exporter.*` on `Microsoft.OpenTelemetry` 1.0.2 and earlier; `useS2SEndpoint: true`; `a365_use_s2s_endpoint=True`). Registered blueprint agent instances need no `Agent365.Observability.OtelWrite` permission or admin consent (subject to service policy); AI Teammates complete the `OtelWrite` application-role step `a365 setup all --aiteammate` prints. -**S2S scaffold requirement:** When `authMode = s2s`, `instrument-observability` creates a scaffold token-service file per language: .NET creates `Observability/ObservabilityServiceExtensions.cs` + `Observability/ObservabilityTokenService.cs`, Node.js creates `observability/observability-token-service.ts`, Python creates `observability/observability_token_service.py`. Each acquires and refreshes the Observability API token via MSAL client credentials targeting `api://9b975845-388f-4429-889e-eab1ef63949c/.default`. The .NET files additionally provide the `AddAgent365Observability()` / `Agent365ObservabilityContext` DI extensions that replace `AddAgenticTracingExporter()` and per-turn `RegisterObservability()`. The validator (`validate-instrument-observability.js`) accepts either the OBO signal (`BaggageBuilder` / `BaggageTurnMiddleware`) or the S2S signal to pass the context check. When the unified distro is used, it also requires the S2S route flag and a token resolver, both passed in the distro call itself (`o.Agent365.TokenResolver` / `tokenResolver` / `a365_token_resolver`), in every auth mode, and it fails the session on delegated telemetry wiring (`refreshObservabilityToken(..., authorization)`, `RegisterObservability(..., AgenticTokenStruct)`, `exchange_token(...)` for the observability scope). +**S2S scaffold requirement:** When `authMode = s2s`, `instrument-observability` creates a scaffold token-service file per language: .NET creates `Observability/ObservabilityServiceExtensions.cs` + `Observability/ObservabilityTokenService.cs`, Node.js creates `observability/observability-token-service.ts`, Python creates `observability/observability_token_service.py`. Each acquires and refreshes the Observability API token via MSAL client credentials targeting `api://9b975845-388f-4429-889e-eab1ef63949c/.default`. The .NET files additionally provide the `AddAgent365Observability()` / `Agent365ObservabilityContext` DI extensions that replace `AddAgenticTracingExporter()` and per-turn `RegisterObservability()`. The validator (`validate-instrument-observability.js`) accepts either the OBO signal (`BaggageBuilder` / `BaggageTurnMiddleware`) or the S2S signal to pass the context check. When the unified distro is used, it also requires the S2S route flag and a token resolver, both passed in the distro call itself (`o.Agent365.TokenResolver` / `tokenResolver` / `a365_token_resolver`), in every auth mode, and it fails the session on delegated telemetry wiring (`refreshObservabilityToken(...)`, `RegisterObservability(..., AgenticTokenStruct)`, `exchange_token(...)` for the observability scope). --- diff --git a/CLAUDE.md b/CLAUDE.md index a1a7a9d..60dd86b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -133,11 +133,14 @@ agent365-skills/ scaffold. Always set the S2S route flag (`o.Agent365.UseS2SEndpoint = true`, `useS2SEndpoint: true`, `a365_use_s2s_endpoint=True`). Never generate a per-turn delegated telemetry token (`RegisterObservability(..., AgenticTokenStruct)`, - `refreshObservabilityToken(..., authorization)`, `exchange_token(...)` for the observability + `refreshObservabilityToken(...)`, `exchange_token(...)` for the observability scope), and never add the delegated `OtelWrite` scope for telemetry. Registered blueprint agent instances need no `Agent365.Observability.OtelWrite` permission or admin consent, so never make - an OBS grant a required step for them. For blueprint agents, a 403 `insufficient_scope` means - the instance isn't registered: `a365 setup all --agent-registration-only`. AI Teammates complete + an OBS grant a required step for them. Newer `a365 setup all` versions skip OtelWrite for + blueprint agents and fail when registration fails or cannot be verified; older versions may + still grant OtelWrite (harmless) and may exit 0 after a failed registration, so check setup + output and rerun `a365 setup all --agent-registration-only` if needed. For blueprint agents, + a 403 `insufficient_scope` means the instance isn't registered. AI Teammates complete the `OtelWrite` application-role step that `a365 setup all --aiteammate` prints. The application role is always an accepted fallback on the S2S route. diff --git a/plugins/agent365/hooks/stop/validate-a365-code-validator.js b/plugins/agent365/hooks/stop/validate-a365-code-validator.js index e4b6712..c94fb76 100644 --- a/plugins/agent365/hooks/stop/validate-a365-code-validator.js +++ b/plugins/agent365/hooks/stop/validate-a365-code-validator.js @@ -269,10 +269,40 @@ function codeMatches(files, language, regex) { return files.some(file => regex.test(stripComments(read(file), language))); } -function codeCallMatches(files, language, name, regex) { - return files.some(file => callArguments(stripComments(read(file), language), name).some(args => regex.test(args))); +function topLevelArgumentCount(args) { + const text = args.trim().replace(/^\(/, '').replace(/\)$/, ''); + if (!text.trim()) return 0; + let count = 1; + let depth = 0; + let quote = ''; + let escaped = false; + for (let i = 0; i < text.length; i++) { + const ch = text[i]; + if (quote) { + if (escaped) escaped = false; + else if (ch === '\\') escaped = true; + else if (ch === quote) quote = ''; + continue; + } + if (ch === '"' || ch === "'" || ch === '`') { + quote = ch; + } else if (ch === '(' || ch === '[' || ch === '{') { + depth++; + } else if (ch === ')' || ch === ']' || ch === '}') { + depth = Math.max(0, depth - 1); + } else if (ch === ',' && depth === 0) { + count++; + } + } + return count; +} + +function codeCallMatches(files, language, name, matcher) { + return files.some(file => callArguments(stripComments(read(file), language), name) + .some(args => typeof matcher === 'function' ? matcher(args) : matcher.test(args))); } + function validatePython() { const hasMicrosoftOpenTelemetryPackage = reqFiles.some(f => fileContains(f, 'microsoft-opentelemetry')); const hasDistroCall = anyFileContains(pyFiles, 'use_microsoft_opentelemetry'); @@ -518,14 +548,14 @@ function validateNode() { } for (const file of tsFiles) { const content = stripComments(read(file), 'node'); - const delegatedRefresh = ['refreshObservabilityToken', 'RefreshObservabilityToken'] - .some(name => findCallBlocks(content, name).some(block => /authorization/i.test(block))) || + const delegatedRefresh = codeCallMatches([file], 'node', 'refreshObservabilityToken', () => true) || + codeCallMatches([file], 'node', 'RefreshObservabilityToken', args => topLevelArgumentCount(args) >= 4) || /AgenticTokenCacheInstance\s*\.\s*getObservabilityToken\s*\(/.test(content); if (delegatedRefresh) { add( 'high', 'node-obs-delegated-token', - 'Telemetry uses a delegated (OBO) token (refreshObservabilityToken(..., authorization) or AgenticTokenCacheInstance.getObservabilityToken). The S2S route rejects delegated tokens; use an app-only tokenResolver for the agent identity instead.', + 'Telemetry uses a delegated (OBO) token (refreshObservabilityToken(...) or a four-argument RefreshObservabilityToken(...) call, or AgenticTokenCacheInstance.getObservabilityToken). The S2S route rejects delegated tokens; use an app-only tokenResolver for the agent identity instead.', file ); } diff --git a/plugins/agent365/hooks/stop/validate-instrument-observability.js b/plugins/agent365/hooks/stop/validate-instrument-observability.js index bba234d..2abbcec 100644 --- a/plugins/agent365/hooks/stop/validate-instrument-observability.js +++ b/plugins/agent365/hooks/stop/validate-instrument-observability.js @@ -198,8 +198,37 @@ function codeMatches(files, language, regex) { return files.some(file => regex.test(stripComments(read(file), language))); } -function codeCallMatches(files, language, name, regex) { - return files.some(file => callArguments(stripComments(read(file), language), name).some(args => regex.test(args))); +function topLevelArgumentCount(args) { + const text = args.trim().replace(/^\(/, '').replace(/\)$/, ''); + if (!text.trim()) return 0; + let count = 1; + let depth = 0; + let quote = ''; + let escaped = false; + for (let i = 0; i < text.length; i++) { + const ch = text[i]; + if (quote) { + if (escaped) escaped = false; + else if (ch === '\\') escaped = true; + else if (ch === quote) quote = ''; + continue; + } + if (ch === '"' || ch === "'" || ch === '`') { + quote = ch; + } else if (ch === '(' || ch === '[' || ch === '{') { + depth++; + } else if (ch === ')' || ch === ']' || ch === '}') { + depth = Math.max(0, depth - 1); + } else if (ch === ',' && depth === 0) { + count++; + } + } + return count; +} + +function codeCallMatches(files, language, name, matcher) { + return files.some(file => callArguments(stripComments(read(file), language), name) + .some(args => typeof matcher === 'function' ? matcher(args) : matcher.test(args))); } // ── Detect project type ───────────────────────────────────────────────────── @@ -417,10 +446,10 @@ if (isNodejs) { if (usesDistro && !distroCallUsesS2SRoute(tsFiles, 'node')) { issues.push('Observability export must use the S2S route in every auth mode: pass useS2SEndpoint: true in the a365 options of useMicrosoftOpenTelemetry() with an app-only tokenResolver (observability/app-token-resolver.ts for obo / agentic-user)'); } - if (codeCallMatches(tsFiles, 'node', 'refreshObservabilityToken', /authorization/i) || - codeCallMatches(tsFiles, 'node', 'RefreshObservabilityToken', /authorization/i) || + if (codeCallMatches(tsFiles, 'node', 'refreshObservabilityToken', () => true) || + codeCallMatches(tsFiles, 'node', 'RefreshObservabilityToken', args => topLevelArgumentCount(args) >= 4) || codeMatches(tsFiles, 'node', /AgenticTokenCacheInstance\s*\.\s*getObservabilityToken\s*\(/)) { - issues.push('refreshObservabilityToken(..., authorization) or AgenticTokenCacheInstance.getObservabilityToken(...) feeds a delegated (OBO) telemetry token, which the S2S route rejects — remove it (and any preloadObservabilityToken helper) and use the app-only tokenResolver (see nodejs-observability.md)'); + issues.push('refreshObservabilityToken(...), a four-argument RefreshObservabilityToken(...) call, or AgenticTokenCacheInstance.getObservabilityToken(...) feeds a delegated (OBO) telemetry token, which the S2S route rejects — remove it (and any preloadObservabilityToken helper) and use the app-only tokenResolver (see nodejs-observability.md)'); } // 5. .env has observability vars diff --git a/plugins/agent365/shared/agent-detection.md b/plugins/agent365/shared/agent-detection.md index a29027b..98361e9 100644 --- a/plugins/agent365/shared/agent-detection.md +++ b/plugins/agent365/shared/agent-detection.md @@ -448,7 +448,7 @@ The cache is written in stages as values become known — always preserve fields - `AgentApplication` in source files, `CloudAdapter`/`CloudAdapterAiohttp`, `@microsoft/agents-a365-notifications` in `package.json`, `Microsoft.Agents.A365.Notifications` in `.csproj`, or `ToolingManifest.json` exists. - `has_obs`: `1` ONLY when **all four** observability anchors are present (entry-point + app-only token resolver + S2S route flag + handler-side baggage / scope). The entry-point call alone (`UseMicrosoftOpenTelemetry` / `useMicrosoftOpenTelemetry` / `use_microsoft_opentelemetry`) is **not sufficient** — a project can have the call without `tokenResolver` (exports silently fail to auth), without the S2S route flag (telemetry still goes to the legacy delegated route, which rejects app-only tokens), or without `BaggageBuilder` / `InvokeAgentScope` (no identity grouping, fails store-publish validation). Telemetry is exported over the S2S route with an app-only token in **every** auth mode. Anchor sets per language: - **.NET (`**/*.cs`)**: entry = `UseMicrosoftOpenTelemetry`; token = the `UseMicrosoftOpenTelemetry(...)` options assign `TokenResolver` (`o.Agent365.TokenResolver`; `o.Agent365.Exporter.TokenResolver` on 1.0.2 and earlier) to the app-only `AgentAppTokenResolver` (obo / agentic-user) or the S2S token cache fed by `ObservabilityTokenService` / `AddAgent365Observability`, with no `AgenticTokenStruct` usage (per-turn `RegisterObservability(..., new AgenticTokenStruct(...))`, `new AgenticTokenStruct(...)`, or an `IExporterTokenCache` dependency); route = `UseS2SEndpoint = true` in the same options; handler = `BaggageBuilder` OR `BaggageTurnMiddleware` OR `InvokeAgentScope.Start`. - - **Node.js (`src/**/*.ts`)**: entry = `useMicrosoftOpenTelemetry`; token = the `useMicrosoftOpenTelemetry({...})` call passes `tokenResolver` in its `a365` options (app-only: `observability/app-token-resolver.ts`, or `getS2SObservabilityToken` / `startTokenService` for S2S), with no `refreshObservabilityToken(..., authorization)` or `AgenticTokenCacheInstance.getObservabilityToken` resolver; route = `useS2SEndpoint: true` in the same call's `a365` options; handler = `BaggageBuilder` OR `BaggageBuilderUtils` OR `InvokeAgentScope`. + - **Node.js (`src/**/*.ts`)**: entry = `useMicrosoftOpenTelemetry`; token = the `useMicrosoftOpenTelemetry({...})` call passes `tokenResolver` in its `a365` options (app-only: `observability/app-token-resolver.ts`, or `getS2SObservabilityToken` / `startTokenService` for S2S), with no `refreshObservabilityToken(...)` or `AgenticTokenCacheInstance.getObservabilityToken` resolver; route = `useS2SEndpoint: true` in the same call's `a365` options; handler = `BaggageBuilder` OR `BaggageBuilderUtils` OR `InvokeAgentScope`. - **Python (`**/*.py`)**: entry = `use_microsoft_opentelemetry`; token = the `use_microsoft_opentelemetry(...)` call passes `a365_token_resolver=` (app-only: `OBS_TOKENS.resolve` from `AppTokenResolver`, or the S2S `run_token_service` / `get_s2s_observability_token` cache), with no `exchange_token(...)` for the observability scope, `cache_agentic_token(...)`, or `AgenticTokenCache` / `get_cached_agentic_token` resolver; route = `a365_use_s2s_endpoint=True` passed in the same call (or `A365_USE_S2S_ENDPOINT=true` in `.env`); handler = `BaggageBuilder` OR `populate_baggage` OR `InvokeAgentScope`. The token and route anchors must be wired at the distro call itself. A variable passed to the call counts when its initializer sets the option, but a symbol elsewhere does not, and neither do comments: an unused import, a `tokenResolver` in another file, an `ObservabilityTokenService` that is never assigned to `TokenResolver`, or a `useS2SEndpoint: true` on an unrelated object leaves export on the wrong route or without a credential. diff --git a/plugins/agent365/skills/a365-code-validator/SKILL.md b/plugins/agent365/skills/a365-code-validator/SKILL.md index c05dbb9..ea05930 100644 --- a/plugins/agent365/skills/a365-code-validator/SKILL.md +++ b/plugins/agent365/skills/a365-code-validator/SKILL.md @@ -202,6 +202,7 @@ Python must have either explicit code: use_microsoft_opentelemetry( enable_a365=True, a365_enable_observability_exporter=True, + a365_use_s2s_endpoint=True, ) ``` @@ -218,6 +219,7 @@ useMicrosoftOpenTelemetry({ a365: { enabled: true, enableObservabilityExporter: true, + useS2SEndpoint: true, }, }); ``` @@ -314,7 +316,7 @@ than relying only on `A365_USE_S2S_ENDPOINT=true` in the runtime environment. Flag **delegated telemetry** as `high`: the S2S route rejects any token carrying `scp`, and the legacy delegated route needs admin consent. Signals are: -- Node.js `AgenticTokenCacheInstance.refreshObservabilityToken(..., authorization)` (often wrapped +- Node.js `AgenticTokenCacheInstance.refreshObservabilityToken(...)` (often wrapped in a `preloadObservabilityToken` helper), or a `tokenResolver` that reads `AgenticTokenCacheInstance.getObservabilityToken(...)`. - .NET `RegisterObservability(..., new AgenticTokenStruct(...), ...)`, any `new AgenticTokenStruct(...)` @@ -409,8 +411,9 @@ Interpret observability authorization for the S2S route, which every auth mode u | Neither registration nor the application role is evident | `high` — export will likely return 403 `insufficient_scope`. Blueprint agents: `a365 setup all --agent-registration-only`. AI Teammates: complete the `OtelWrite` application-role step `a365 setup all --aiteammate` prints | | Only the delegated `Agent365.Observability.OtelWrite` scope is granted | Informational — it only matters to legacy delegated-route exporters; flag the code for migration instead (§3.5) | -Do **not** treat a missing `OtelWrite` grant as a blocker for a registered blueprint agent: the -Agent 365 CLI no longer requests Observability API permissions for blueprint agents. Treat a +Do **not** treat a missing `OtelWrite` grant as a blocker for a registered blueprint agent: newer +Agent 365 CLI versions skip Observability API permissions for blueprint agents, and older versions +may still grant them harmlessly. Treat a non-zero `inheritance` result for another required resource as `high`. Permission names and resource display names are safe to summarize, but redact tenant, Blueprint, application, service-principal, and agent IDs. diff --git a/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js b/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js index ed0350c..408e3d1 100644 --- a/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js +++ b/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js @@ -267,10 +267,40 @@ function codeMatches(files, language, regex) { return files.some(file => regex.test(stripComments(read(file), language))); } -function codeCallMatches(files, language, name, regex) { - return files.some(file => callArguments(stripComments(read(file), language), name).some(args => regex.test(args))); +function topLevelArgumentCount(args) { + const text = args.trim().replace(/^\(/, '').replace(/\)$/, ''); + if (!text.trim()) return 0; + let count = 1; + let depth = 0; + let quote = ''; + let escaped = false; + for (let i = 0; i < text.length; i++) { + const ch = text[i]; + if (quote) { + if (escaped) escaped = false; + else if (ch === '\\') escaped = true; + else if (ch === quote) quote = ''; + continue; + } + if (ch === '"' || ch === "'" || ch === '`') { + quote = ch; + } else if (ch === '(' || ch === '[' || ch === '{') { + depth++; + } else if (ch === ')' || ch === ']' || ch === '}') { + depth = Math.max(0, depth - 1); + } else if (ch === ',' && depth === 0) { + count++; + } + } + return count; +} + +function codeCallMatches(files, language, name, matcher) { + return files.some(file => callArguments(stripComments(read(file), language), name) + .some(args => typeof matcher === 'function' ? matcher(args) : matcher.test(args))); } + function validatePython() { const hasPackage = req.some(f => read(f).includes('microsoft-opentelemetry')); const hasDistro = anyContains(py, 'use_microsoft_opentelemetry'); @@ -404,9 +434,10 @@ function validateNode() { } for (const file of ts) { const content = stripComments(read(file), 'node'); - if (['refreshObservabilityToken', 'RefreshObservabilityToken'].some(name => callBlocks(content, name).some(block => /authorization/i.test(block))) || + if (codeCallMatches([file], 'node', 'refreshObservabilityToken', () => true) || + codeCallMatches([file], 'node', 'RefreshObservabilityToken', args => topLevelArgumentCount(args) >= 4) || /AgenticTokenCacheInstance\s*\.\s*getObservabilityToken\s*\(/.test(content)) { - add('high', 'node-obs-delegated-token', 'Telemetry uses a delegated (OBO) token (refreshObservabilityToken(..., authorization) or AgenticTokenCacheInstance.getObservabilityToken). The S2S route rejects delegated tokens; use an app-only tokenResolver for the agent identity instead.', file); + add('high', 'node-obs-delegated-token', 'Telemetry uses a delegated (OBO) token (refreshObservabilityToken(...) or a four-argument RefreshObservabilityToken(...) call, or AgenticTokenCacheInstance.getObservabilityToken). The S2S route rejects delegated tokens; use an app-only tokenResolver for the agent identity instead.', file); } } const hasSemantic = anyContains(ts, 'InvokeAgentScope') || diff --git a/plugins/agent365/skills/a365-code-validator/references/validation-checklist.md b/plugins/agent365/skills/a365-code-validator/references/validation-checklist.md index 590c5ed..3fcc290 100644 --- a/plugins/agent365/skills/a365-code-validator/references/validation-checklist.md +++ b/plugins/agent365/skills/a365-code-validator/references/validation-checklist.md @@ -19,6 +19,7 @@ Required code: use_microsoft_opentelemetry( enable_a365=True, a365_enable_observability_exporter=True, + a365_use_s2s_endpoint=True, ) ``` @@ -171,7 +172,7 @@ fragile and should be called out. **Delegated telemetry is a finding (`high`).** Look for any of these: -- Node.js `refreshObservabilityToken(..., authorization)` or a `preloadObservabilityToken` helper, or +- Node.js `refreshObservabilityToken(...)` or a `preloadObservabilityToken` helper, or a `tokenResolver` reading `AgenticTokenCacheInstance.getObservabilityToken(...)`. - .NET `RegisterObservability(..., new AgenticTokenStruct(...), ...)`, any `new AgenticTokenStruct(...)`, or an `IExporterTokenCache` dependency. diff --git a/plugins/agent365/skills/instrument-observability/SKILL.md b/plugins/agent365/skills/instrument-observability/SKILL.md index 4e5b8df..d69449d 100644 --- a/plugins/agent365/skills/instrument-observability/SKILL.md +++ b/plugins/agent365/skills/instrument-observability/SKILL.md @@ -97,8 +97,8 @@ delegated telemetry" in Phase 3). > agent identity, whatever the `authMode`. The S2S route rejects delegated (`scp`) tokens, so > OBO / Agentic User tokens are used only for workload calls (MCP / Graph), never for the > exporter. Registered blueprint agent instances need no `Agent365.Observability.OtelWrite` -> permission or admin consent (subject to service policy), and `a365 setup all` no longer requests -> them for blueprint agents. AI Teammate setup (`a365 setup all --aiteammate`) still offers the +> permission or admin consent (subject to service policy). Newer `a365 setup all` versions skip +> that grant for blueprint agents; older versions may still grant it harmlessly. AI Teammate setup (`a365 setup all --aiteammate`) still offers the > `OtelWrite` application role; complete the app-role action item it prints, because the S2S route > accepts that role. @@ -261,7 +261,7 @@ flag live in the references — see the "Required packages" section of: > **Migrating delegated telemetry (the one non-additive edit).** If the project already wires > the legacy delegated (OBO-route) telemetry path, replace it. That path is: -> - Node.js: `AgenticTokenCacheInstance.refreshObservabilityToken(..., authorization)`, a +> - Node.js: `AgenticTokenCacheInstance.refreshObservabilityToken(...)`, a > `preloadObservabilityToken` helper, or a `tokenResolver` reading > `AgenticTokenCacheInstance.getObservabilityToken(...)`. > - .NET: `RegisterObservability(..., new AgenticTokenStruct(...), ...)`, any @@ -274,10 +274,13 @@ flag live in the references — see the "Required packages" section of: > > The S2S route rejects those delegated tokens, and the delegated route needs admin consent. Swap the > resolver for the app-only one, set the S2S route flag, and remove the per-turn delegated -> refresh. Keep the baggage and scope wiring. Durable or offline delivery spools keep each -> record's original route, so for `@microsoft/opentelemetry` 1.4.x set -> `a365.durableDelivery: { enabled: false }` during the migration (as the Agent 365 samples do) -> or clear the spool directory. List every replaced call in the final summary. +> refresh. Keep the baggage and scope wiring. Durable/offline delivery spools keep each +> record's original route, so disable replay until the installed release enforces S2S for both +> live and replayed exports, or clear the spool directory: +> - Node.js (`@microsoft/opentelemetry` 1.4.0+): `a365: { durableDelivery: { enabled: false } }` +> - .NET (`Microsoft.OpenTelemetry` 1.1.0+): `o.Agent365.DisableOfflineStorage = true` +> - Python (`microsoft-opentelemetry`): `a365_exporter_disable_offline_storage=True` +> List every replaced call in the final summary. ### For .NET AgentFramework @@ -377,7 +380,7 @@ flag live in the references — see the "Required packages" section of: : null; ``` - **Keep the `Agent365Observability` section in `appsettings.json`** (`EnableAgent365Exporter` and base exporter settings are still required — Phase 6 handles these). No S2S credentials are needed there: `AgentAppTokenResolver` uses the agent's `Connections` settings, and agentic turns resolve their agent ID at runtime. - - **The inline pattern shown above is preferred** for new code (mirrors PR #308 in `microsoft/Agent365-Samples`). The older `A365OtelWrapper.InvokeObservedAgentOperation(...)` static-wrapper pattern at `Agent365-samples/dotnet/agent-framework/sample-agent/telemetry/A365OtelWrapper.cs` is functionally equivalent but uses a separate helper class. + - `A365OtelWrapper` is the legacy delegated-token wrapper. Remove it and migrate the handler to the inline app-only S2S pattern above (see "Migrating delegated telemetry" in Phase 3). **S2S path**: - Inject `Agent365ObservabilityContext` (singleton registered by `AddAgent365Observability()`) in the constructor @@ -839,7 +842,7 @@ If expected files are not found: This skill is safe to rerun. On subsequent runs: - Skip package installation if packages already present -- Skip code edits only if observability is already wired **the current way**. Marker comments must be present, the S2S route flag must be set (`useS2SEndpoint: true` / `a365_use_s2s_endpoint=True` / `o.Agent365.UseS2SEndpoint = true`), and no delegated-telemetry signal may remain: `refreshObservabilityToken(..., authorization)`, `AgenticTokenCacheInstance.getObservabilityToken`, `AgenticTokenStruct` / `RegisterObservability(..., AgenticTokenStruct)`, `exchange_token(...)` for the observability scope, `cache_agentic_token`, or an `AgenticTokenCache` resolver. Marker comments alone are not enough — code written by earlier versions of this skill carries them on the delegated wiring. If any delegated signal is present or the route flag is missing, run "Migrating delegated telemetry" (Phase 3) instead of skipping. +- Skip code edits only if observability is already wired **the current way**. Marker comments must be present, the S2S route flag must be set (`useS2SEndpoint: true` / `a365_use_s2s_endpoint=True` / `o.Agent365.UseS2SEndpoint = true`), and no delegated-telemetry signal may remain: `refreshObservabilityToken(...)`, `AgenticTokenCacheInstance.getObservabilityToken`, `AgenticTokenStruct` / `RegisterObservability(..., AgenticTokenStruct)`, `exchange_token(...)` for the observability scope, `cache_agentic_token`, or an `AgenticTokenCache` resolver. Marker comments alone are not enough — code written by earlier versions of this skill carries them on the delegated wiring. If any delegated signal is present or the route flag is missing, run "Migrating delegated telemetry" (Phase 3) instead of skipping. - Update configuration only if values are missing - Always revalidate the build @@ -849,10 +852,10 @@ This skill is safe to rerun. On subsequent runs: ### Authorization — registration, not OtelWrite -Telemetry is exported over the S2S route with an app-only token for the agent identity. Registered agent instances are authorized on that route **without** `Agent365.Observability.OtelWrite` or admin consent (subject to service policy). `a365 setup all` no longer requests Observability API permissions for blueprint agents, and a failed or unverifiable registration now fails setup. +Telemetry is exported over the S2S route with an app-only token for the agent identity. Registered agent instances are authorized on that route **without** `Agent365.Observability.OtelWrite` or admin consent (subject to service policy). Newer `a365 setup all` versions skip Observability API permissions for blueprint agents and exit 1 when registration fails or cannot be verified. Older versions may still grant OtelWrite (harmless) and may exit 0 after a failed registration, so check setup output for registration errors and rerun `a365 setup all --agent-registration-only` if needed. - **403 `insufficient_scope` on export** → the agent instance is not registered and has no `OtelWrite` application role. Blueprint agents: run `a365 setup all --agent-registration-only` (idempotent), then retry. AI Teammates: that flag does not apply — complete the Observability API S2S app-role action item `a365 setup all --aiteammate` prints (the application role below). -- **Application role (AI Teammates, and fallback for blueprint agents):** a Global Administrator can grant the `Agent365.Observability.OtelWrite` **application** role on the Blueprint, which agent identities inherit. The S2S route accepts it. Use the PowerShell steps that `a365 setup all` prints, or the Entra portal: App registrations > Blueprint > API permissions > APIs my organization uses > `9b975845-388f-4429-889e-eab1ef63949c` > **Application** `Agent365.Observability.OtelWrite` > Grant admin consent. +- **Application role (AI Teammates, and fallback for blueprint agents):** a Global Administrator can grant the `Agent365.Observability.OtelWrite` **application** role on the Blueprint, which agent identities inherit. The S2S route accepts it. Use the PowerShell steps that `a365 setup all` prints, or the Entra portal: App registrations > Blueprint > API permissions > APIs my organization uses > `9b975845-388f-4429-889e-eab1ef63949c` > **Application** `Agent365.Observability.OtelWrite` > Grant admin consent. `a365 setup all --aiteammate` still requests OtelWrite as both delegated and application permission; run the printed script. The application role is what the S2S route accepts, and the delegated part is harmless and not used for telemetry. - **Do not** add the delegated `OtelWrite` scope or run a delegated consent flow for telemetry. Delegated tokens are rejected by the S2S route. Only agents still on an old SDK that exports over the delegated route need that grant, and they should migrate instead (see "Migrating delegated telemetry" in Phase 3). ### Node.js and .NET SDK `/otlp/` URL Path Bug diff --git a/plugins/agent365/skills/instrument-observability/references/dotnet-observability.md b/plugins/agent365/skills/instrument-observability/references/dotnet-observability.md index bbcddb1..2711d32 100644 --- a/plugins/agent365/skills/instrument-observability/references/dotnet-observability.md +++ b/plugins/agent365/skills/instrument-observability/references/dotnet-observability.md @@ -12,7 +12,8 @@ into a .NET AgentFramework agent. All samples mirror the official Microsoft Lear > route rejects delegated (`scp`) tokens, so do not register per-turn OBO tokens > (`RegisterObservability(..., new AgenticTokenStruct(...))`) for telemetry. Registered blueprint > agent instances need no `Agent365.Observability.OtelWrite` permission or admin consent (subject -> to service policy), and `a365 setup all` no longer requests them for blueprint agents. AI Teammate +> to service policy). Newer `a365 setup all` versions skip that grant for blueprint agents; older +> versions may still grant it harmlessly. AI Teammate > setup (`a365 setup all --aiteammate`) still offers the `OtelWrite` application role; complete the > app-role action item it prints, because the S2S route accepts that role. > @@ -201,7 +202,7 @@ public static class ObservabilityServiceExtensions > - `true` (production) — MSI → Blueprint FIC → Agent Identity → API > - `false` (local dev) — Client Secret → Blueprint FIC → Agent Identity → API > -> **Authorization:** registered agent instances are authorized on the S2S route without the `Agent365.Observability.OtelWrite` permission or admin consent (subject to service policy), and `a365 setup all` no longer requests it for blueprint agents. If export returns 403 `insufficient_scope`, register a blueprint agent instance with `a365 setup all --agent-registration-only`; for AI Teammates, complete the `OtelWrite` application-role step that `a365 setup all --aiteammate` prints. Either way, a Global Administrator can grant the `OtelWrite` application role. +> **Authorization:** registered agent instances are authorized on the S2S route without the `Agent365.Observability.OtelWrite` permission or admin consent (subject to service policy). Newer `a365 setup all` versions skip that grant for blueprint agents; older versions may still grant it harmlessly. If export returns 403 `insufficient_scope`, register a blueprint agent instance with `a365 setup all --agent-registration-only`; for AI Teammates, complete the `OtelWrite` application-role step that `a365 setup all --aiteammate` prints. Either way, a Global Administrator can grant the `OtelWrite` application role. ```csharp using Azure.Core; @@ -494,12 +495,19 @@ namespace ; public sealed class AgentAppTokenResolver { private static readonly string[] ObservabilityScopes = ["api://9b975845-388f-4429-889e-eab1ef63949c/.default"]; + private static readonly HashSet ObservabilityAudiences = + [ + "api://9b975845-388f-4429-889e-eab1ef63949c", + "9b975845-388f-4429-889e-eab1ef63949c", + ]; private static readonly TimeSpan RefreshSkew = TimeSpan.FromMinutes(5); + private static readonly TimeSpan FailureBackoff = TimeSpan.FromSeconds(60); private readonly IConnections _connections; private readonly ILogger _logger; private readonly ConcurrentDictionary _tokens = new(); - private readonly SemaphoreSlim _refreshGate = new(1, 1); + private readonly ConcurrentDictionary _refreshGates = new(); + private readonly ConcurrentDictionary _failedUntil = new(); public AgentAppTokenResolver(IConnections connections, ILogger logger) { @@ -522,7 +530,13 @@ public sealed class AgentAppTokenResolver return token; } - await _refreshGate.WaitAsync().ConfigureAwait(false); + if (_failedUntil.TryGetValue(key, out var retryAfter) && retryAfter > DateTimeOffset.UtcNow) + { + return null; + } + + var refreshGate = _refreshGates.GetOrAdd(key, _ => new SemaphoreSlim(1, 1)); + await refreshGate.WaitAsync().ConfigureAwait(false); try { if (TryGetFresh(key, out token)) @@ -533,6 +547,7 @@ public sealed class AgentAppTokenResolver if (_connections.GetDefaultConnection() is not IAgenticTokenProvider blueprint) { _logger.LogWarning("The default connection cannot issue agentic tokens; A365 observability export skipped."); + RememberFailure(key); return null; } @@ -546,23 +561,27 @@ public sealed class AgentAppTokenResolver .ExecuteAsync() .ConfigureAwait(false); - if (HasDelegatedScope(result.AccessToken)) + var validationError = ValidateToken(result.AccessToken, agentId, tenantId); + if (validationError is not null) { - _logger.LogWarning("The observability token carries a delegated scp claim, which the S2S route rejects; export skipped."); + _logger.LogWarning("The observability token is not valid for S2S export: {Reason}", validationError); + RememberFailure(key); return null; } _tokens[key] = result; + _failedUntil.TryRemove(key, out _); return result.AccessToken; } catch (Exception ex) { + RememberFailure(key); _logger.LogWarning(ex, "Could not acquire an app-only A365 observability token for agent {AgentId}; export skipped.", agentId); return null; } finally { - _refreshGate.Release(); + refreshGate.Release(); } } @@ -574,19 +593,54 @@ public sealed class AgentAppTokenResolver return token is not null; } - private static bool HasDelegatedScope(string accessToken) + private void RememberFailure(string key) + { + _failedUntil[key] = DateTimeOffset.UtcNow + FailureBackoff; + } + + private static string? ValidateToken(string accessToken, string agentId, string tenantId) { var payload = accessToken.Split('.')[1].Replace('-', '+').Replace('_', '/'); payload = payload.PadRight(payload.Length + (4 - payload.Length % 4) % 4, '='); using var claims = JsonDocument.Parse(Convert.FromBase64String(payload)); - return claims.RootElement.TryGetProperty("scp", out _); + var root = claims.RootElement; + if (root.TryGetProperty("scp", out _)) + { + return "it carries a delegated scp claim"; + } + + var clientId = Claim(root, "azp") ?? Claim(root, "appid"); + if (!string.Equals(clientId, agentId, StringComparison.OrdinalIgnoreCase)) + { + return "azp/appid does not match the exporting agent"; + } + + if (!string.Equals(Claim(root, "tid"), tenantId, StringComparison.OrdinalIgnoreCase)) + { + return "tid does not match the exporting tenant"; + } + + var audience = Claim(root, "aud"); + if (audience is null || !ObservabilityAudiences.Contains(audience)) + { + return "aud does not match the Observability API"; + } + + return null; } + + private static string? Claim(JsonElement claims, string name) => + claims.TryGetProperty(name, out var value) ? value.GetString() : null; } ``` > `IConnections` is registered by the Agents SDK hosting setup (`AddAgentApplicationOptions` / > `AddAgent`), and its default connection is the blueprint credential that `a365 setup all` > writes to `appsettings.json` (`Connections:ServiceConnection`). No extra settings are needed. +> This intentionally mirrors the Node/Python skill scaffolds: the hosting connection lets one +> deployment serve several agent instances/tenants and inherit certificate, federated-identity, +> or managed-identity credentials. The Agent365 samples use a dedicated single-identity OBS +> credential when telemetry auth should be isolated from business auth. > Registered blueprint agent instances are authorized on the S2S route without the > `Agent365.Observability.OtelWrite` permission or admin consent (subject to service policy). For AI > Teammates, complete the `OtelWrite` application-role step that `a365 setup all --aiteammate` prints. @@ -628,7 +682,10 @@ builder.AddA365Tracing(); // from Microsoft.Agents.A365.O These are subsumed by `UseMicrosoftOpenTelemetry()` and the distro package — mixing the two causes CS0433 duplicate-type errors. Pick one wiring style per project. The legacy `AddAgenticTracingExporter()` wiring exports over the delegated route with OBO tokens; migrate -it to the distro with `o.Agent365.UseS2SEndpoint = true` and `AgentAppTokenResolver`. +it to the distro with `o.Agent365.UseS2SEndpoint = true` and `AgentAppTokenResolver`. If offline +storage is enabled, set `o.Agent365.DisableOfflineStorage = true` (Microsoft.OpenTelemetry 1.1.0+) +until the installed release enforces S2S for both live and replayed exports, or clear the storage +directory. --- @@ -1473,7 +1530,7 @@ The `a365 setup` command (as of April 2026) automatically writes the following t | No logs in Defender | Missing `Logging.LogLevel` config | Add `Microsoft.Agents.A365.Observability: Debug` to appsettings.json | | `AgenticAppId` is null | Missing `AGENTIC_APP_ID` env var | Set it in `.env` or App Service config | | Token resolver returns null | `AgentAppTokenResolver` not registered or not wired, or token acquisition failed (see the `Could not acquire an app-only A365 observability token` warning) | Register `builder.Services.AddSingleton()`, wire `o.Agent365.TokenResolver`, and resolve it after `Build()`. Check that the default connection holds the blueprint credential | -| 401 from A365 exporter | The exporter is on the delegated route (`UseS2SEndpoint` not set) or received a delegated token | Set `o.Agent365.UseS2SEndpoint = true` and use `AgentAppTokenResolver` / `ObservabilityTokenService`. The token must have no `scp` claim, and its `azp`/`appid` must equal the exporting agent ID. Do not run a delegated consent flow for telemetry | +| 401 from A365 exporter | The exporter is on the delegated route (`UseS2SEndpoint` not set) or received the wrong token | Set `o.Agent365.UseS2SEndpoint = true` and use `AgentAppTokenResolver` / `ObservabilityTokenService`. The token must have no `scp` claim, `azp`/`appid` equal to the exporting agent ID, matching `tid`, and `aud` equal to the Observability API. Do not run a delegated consent flow for telemetry | | Build error on `BaggageBuilder` | Wrong namespace | Use `Microsoft.Agents.A365.Observability.Runtime.Common` | | Build error on `AgenticTokenStruct` / `IExporterTokenCache` in agent code | Legacy delegated-route token registration left in the handler | Remove the `IExporterTokenCache` injection and the per-turn `RegisterObservability(...)` call — telemetry uses `AgentAppTokenResolver` | | Build error on `AddAgenticTracingExporter` | Wrong namespace (and it is the legacy delegated-route wiring) | The namespace is `Microsoft.Agents.A365.Observability.Hosting`, but prefer removing the call and migrating to `AgentAppTokenResolver` on the S2S route — see "Legacy two-package wiring" above | diff --git a/plugins/agent365/skills/instrument-observability/references/nodejs-observability.md b/plugins/agent365/skills/instrument-observability/references/nodejs-observability.md index 6f58bad..0d4393e 100644 --- a/plugins/agent365/skills/instrument-observability/references/nodejs-observability.md +++ b/plugins/agent365/skills/instrument-observability/references/nodejs-observability.md @@ -11,7 +11,7 @@ into a Node.js agent. Aligned with `@microsoft/opentelemetry` **GA 1.0.x** (upda > > **Sample-lag note (2026-05):** `Agent365-Samples/nodejs/langchain/sample-agent` has migrated to `@microsoft/opentelemetry` and matches the patterns in this reference. `Agent365-Samples/nodejs/openai/sample-agent` still imports from the legacy `@microsoft/agents-a365-observability*` packages as of this writing — the skill direction (unified `@microsoft/opentelemetry`) is forward-looking. If a user's project already has the legacy imports from following the OpenAI sample literally, the skill should migrate them to `@microsoft/opentelemetry` during the wiring step rather than co-existing. > -> **Telemetry always uses the S2S route.** Every agent (`agentic-user` AI Teammates, `obo`, and `s2s`) exports with `useS2SEndpoint: true` and an **app-only** token for the exporting agent identity. `authMode` only selects workload (MCP / Graph) auth and how the telemetry token is sourced. The S2S route rejects delegated (`scp`) tokens, so never pass an OBO or Agentic User token to the exporter. Registered blueprint agent instances need no `Agent365.Observability.OtelWrite` permission or admin consent (subject to service policy), and `a365 setup all` no longer requests them for blueprint agents. AI Teammate setup (`a365 setup all --aiteammate`) still offers the `OtelWrite` application role; complete the app-role action item it prints, because the S2S route accepts that role. +> **Telemetry always uses the S2S route.** Every agent (`agentic-user` AI Teammates, `obo`, and `s2s`) exports with `useS2SEndpoint: true` and an **app-only** token for the exporting agent identity. `authMode` only selects workload (MCP / Graph) auth and how the telemetry token is sourced. The S2S route rejects delegated (`scp`) tokens, so never pass an OBO or Agentic User token to the exporter. Registered blueprint agent instances need no `Agent365.Observability.OtelWrite` permission or admin consent (subject to service policy); newer `a365 setup all` versions skip it for blueprint agents, while older versions may still grant it harmlessly. AI Teammate setup (`a365 setup all --aiteammate`) still offers the `OtelWrite` application role; complete the app-role action item it prints, because the S2S route accepts that role. --- @@ -260,6 +260,12 @@ Blueprint credential (hosting connection) The exporter passes `(agentId, tenantId)` from span baggage (`recipient.agenticAppId`), so no agent ID is configured here. +> **Why this differs from the Agent365-Samples design:** this skill reuses the hosting +> connection (`getAgenticApplicationToken`) so one deployment can serve several agent +> instances/tenants and inherit certificate, federated-identity, or managed-identity +> credentials. The samples use a dedicated single-identity OBS credential to keep telemetry +> auth isolated from business auth. + ```typescript // observability/app-token-resolver.ts // A365 Observability — best-effort instrumentation (verify against official sample) @@ -272,7 +278,9 @@ agent ID is configured here. import type { AuthProvider } from '@microsoft/agents-hosting'; const OBS_SCOPE = 'api://9b975845-388f-4429-889e-eab1ef63949c/.default'; +const OBS_AUDIENCES = new Set(['api://9b975845-388f-4429-889e-eab1ef63949c', '9b975845-388f-4429-889e-eab1ef63949c']); const REFRESH_SKEW_MS = 5 * 60_000; +const FAILURE_BACKOFF_MS = 60_000; /** Returns the agent's hosting connection; called lazily because the adapter is created after observability init. */ export type ConnectionProvider = () => AuthProvider; @@ -288,6 +296,7 @@ function decodeClaims(token: string): Record { export function createAppTokenResolver(getProvider: ConnectionProvider) { const cache = new Map(); const pending = new Map>(); + const failedUntil = new Map(); const acquire = async (agentId: string, tenantId: string): Promise => { const assertion = await getProvider().getAgenticApplicationToken(tenantId, agentId); @@ -315,6 +324,10 @@ export function createAppTokenResolver(getProvider: ConnectionProvider) { if (client !== agentId.toLowerCase() || String(claims['tid'] ?? '').toLowerCase() !== tenantId.toLowerCase()) { throw new Error('Observability token client or tenant does not match the exporting agent.'); } + const audience = String(claims['aud'] ?? ''); + if (!OBS_AUDIENCES.has(audience)) { + throw new Error('Observability token audience does not match the Observability API.'); + } const exp = claims['exp']; const expiresAt = typeof exp === 'number' ? exp * 1000 : Date.now() + (body.expires_in ?? 0) * 1000; cache.set(`${tenantId}:${agentId}`.toLowerCase(), { token, expiresAt }); @@ -328,14 +341,19 @@ export function createAppTokenResolver(getProvider: ConnectionProvider) { const key = `${tenantId}:${agentId}`.toLowerCase(); const cached = cache.get(key); if (cached && Date.now() < cached.expiresAt - REFRESH_SKEW_MS) return cached.token; + const retryAfter = failedUntil.get(key); + if (retryAfter && Date.now() < retryAfter) return ''; let inFlight = pending.get(key); if (!inFlight) { inFlight = acquire(agentId, tenantId).finally(() => pending.delete(key)); pending.set(key, inFlight); } try { - return await inFlight; + const token = await inFlight; + failedUntil.delete(key); + return token; } catch (err) { + failedUntil.set(key, Date.now() + FAILURE_BACKOFF_MS); console.warn('[A365 Observability] App-only token acquisition failed:', (err as Error).message); return ''; } @@ -599,10 +617,11 @@ async function handleMessage(turnContext: TurnContext, state: ApplicationTurnSta > `AgenticTokenCacheInstance.refreshObservabilityToken(agentId, tenantId, turnContext, authorization)` > call. That delegated exchange needs admin consent, and the S2S route rejects its token. Also > replace the old `AgenticTokenCacheInstance.getObservabilityToken` resolver with the app-only -> resolver and set `useS2SEndpoint: true`. If durable delivery is enabled (the 1.4.x default), records spooled -> while the agent used the delegated route replay to that route. Set -> `a365.durableDelivery: { enabled: false }` while migrating, as the Agent 365 samples do, or -> clear the spool directory. +> resolver and set `useS2SEndpoint: true`. If durable delivery is enabled +> (`@microsoft/opentelemetry` 1.4.0+), records spooled while the agent used the delegated +> route replay to that route. Set `a365.durableDelivery: { enabled: false }` until the +> installed release enforces S2S for both live and replayed exports, or clear the spool +> directory. ### S2S — no per-turn refresh @@ -683,8 +702,14 @@ import { // recipient.agenticAppId + recipient.agenticAppBlueprintId on @microsoft/agents-activity // ChannelAccount); env is only a fallback. const recipient = turnContext.activity?.recipient as any; +const configuredAgentId = process.env.agent365Observability__agentId ?? ''; +const configuredBlueprintId = process.env.agent365Observability__agentBlueprintId ?? ''; +const fallbackAgentId = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(configuredAgentId) + && configuredAgentId.toLowerCase() !== configuredBlueprintId.toLowerCase() + ? configuredAgentId + : ''; const agentDetails: AgentDetails = { - agentId: recipient?.agenticAppId ?? process.env.agent365Observability__agentId ?? '', + agentId: recipient?.agenticAppId || fallbackAgentId, agentName: process.env.agent365Observability__agentName ?? 'Email Assistant', agentDescription: process.env.agent365Observability__agentDescription ?? '', agentAUID: recipient?.agenticUserId ?? '', // microsoft.agent.user.id (agentic user) @@ -920,7 +945,7 @@ manual `.enable()` or `.instrument()` calls needed. ```typescript // Default behavior: both auto-enabled when their packages are installed useMicrosoftOpenTelemetry({ - a365: { enabled: true, enableObservabilityExporter: true, tokenResolver: ... }, + a365: { enabled: true, enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver: ... }, }); // Explicit opt-out: @@ -947,6 +972,7 @@ useMicrosoftOpenTelemetry({ a365: { enabled: true, enableObservabilityExporter: true, + useS2SEndpoint: true, tokenResolver: ..., // `logLevel` is the only logger-related option on A365Options — a pipe-separated // list of levels to emit. There is NO `logger: { info, warn, error }` callback hook @@ -1028,7 +1054,7 @@ Set `ENABLE_A365_OBSERVABILITY_EXPORTER=false` — spans go to the console only. For richer local debug, opt into `enableConsoleExporters: true`: ```typescript useMicrosoftOpenTelemetry({ - a365: { enabled: true, enableConsoleExporters: true, tokenResolver: ... }, + a365: { enabled: true, enableConsoleExporters: true, useS2SEndpoint: true, tokenResolver: ... }, }); ``` @@ -1086,7 +1112,7 @@ Key console messages: | Spans missing baggage | `configureA365Hosting()` not called | Add `configureA365Hosting(adapter, { enableBaggage: true })` once at startup | | Token resolver always returns `''` | App-only token acquisition failed (see the `[A365 Observability] App-only token acquisition failed` warning), or the turn has no agent identity (`recipient.agenticAppId` is empty) | Check that the hosting connection holds the blueprint credential and that the agent identity belongs to that blueprint. For non-agentic turns see the fallback note under the app-only resolver scaffold | | `Cannot find module '@microsoft/opentelemetry'` | Package not installed | `npm install @microsoft/opentelemetry` | -| 401 on export | The exporter is on the delegated route (`useS2SEndpoint` not set) or received a delegated token | Set `useS2SEndpoint: true` and use the app-only resolver; the token must have no `scp` claim and its `azp`/`appid` must equal the exporting agent ID | +| 401 on export | The exporter is on the delegated route (`useS2SEndpoint` not set) or received the wrong token | Set `useS2SEndpoint: true` and use the app-only resolver; the token must have no `scp` claim, `azp`/`appid` equal to the exporting agent ID, matching `tid`, and `aud` equal to the Observability API | | Spans dropped silently | Missing tenant/agent ID | Ensure `configureA365Hosting({ enableBaggage: true })` is registered before creating spans | | Spans only when `ENABLE_A365_OBSERVABILITY_EXPORTER=true` env, but not via code | The env var is a secondary toggle | Set `enableObservabilityExporter: true` in `a365` options (code is preferred over env var) | | Pending spans lost on shutdown | `shutdownMicrosoftOpenTelemetry()` not called | Add SIGTERM/SIGINT handlers calling `await shutdownMicrosoftOpenTelemetry()` | diff --git a/plugins/agent365/skills/instrument-observability/references/python-observability.md b/plugins/agent365/skills/instrument-observability/references/python-observability.md index 8489323..98c81e0 100644 --- a/plugins/agent365/skills/instrument-observability/references/python-observability.md +++ b/plugins/agent365/skills/instrument-observability/references/python-observability.md @@ -10,9 +10,9 @@ into a Python agent. Aligned with `microsoft-opentelemetry` **GA 1.2.x** (releas > and the scope types are still importable from their legacy module paths (transitive deps). > See `MIGRATION_A365.md` in the distro repo for the authoritative migration guide. > -> **Sample-lag note (2026-05):** `Agent365-Samples/python/agent-framework/sample-agent` is the verified canonical sample — it uses **manual per-turn `BaggageBuilder()` in the handler** (NOT `ObservabilityHostingManager` middleware) and imports `BaggageBuilder` + `get_observability_authentication_scope` from `microsoft.opentelemetry.a365.core.middleware.baggage_builder` and `microsoft_agents_a365.runtime.environment_utils` respectively. The OpenAI sample still uses the legacy `configure(...)` + `OpenAIAgentsTraceInstrumentor().instrument()` pattern — the skill direction (unified `use_microsoft_opentelemetry`) is forward-looking; migrate existing code to it. +> **Sample-lag note (2026-05):** `Agent365-Samples/python/agent-framework/sample-agent` is the verified canonical sample — it uses **manual per-turn `BaggageBuilder()` in the handler** (NOT `ObservabilityHostingManager` middleware) and imports `BaggageBuilder` from `microsoft.opentelemetry.a365.core.middleware.baggage_builder`. The OpenAI sample still uses the legacy `configure(...)` + `OpenAIAgentsTraceInstrumentor().instrument()` pattern — the skill direction (unified `use_microsoft_opentelemetry`) is forward-looking; migrate existing code to it. > -> **Telemetry always uses the S2S route.** Every agent (`agentic-user` AI Teammates, `obo`, and `s2s`) exports with `a365_use_s2s_endpoint=True` and an **app-only** token for the exporting agent identity. `authMode` only selects workload (MCP / Graph) auth and how the telemetry token is sourced. The S2S route rejects delegated (`scp`) tokens, so never pass an OBO or Agentic User token (for example from `exchange_token(... scopes=get_observability_authentication_scope() ...)`) to the exporter. Registered blueprint agent instances need no `Agent365.Observability.OtelWrite` permission or admin consent (subject to service policy), and `a365 setup all` no longer requests them for blueprint agents. AI Teammate setup (`a365 setup all --aiteammate`) still offers the `OtelWrite` application role; complete the app-role action item it prints, because the S2S route accepts that role. +> **Telemetry always uses the S2S route.** Every agent (`agentic-user` AI Teammates, `obo`, and `s2s`) exports with `a365_use_s2s_endpoint=True` and an **app-only** token for the exporting agent identity. `authMode` only selects workload (MCP / Graph) auth and how the telemetry token is sourced. The S2S route rejects delegated (`scp`) tokens, so never pass an OBO or Agentic User token (for example from `exchange_token(... scopes=get_observability_authentication_scope() ...)`) to the exporter. Registered blueprint agent instances need no `Agent365.Observability.OtelWrite` permission or admin consent (subject to service policy); newer `a365 setup all` versions skip it for blueprint agents, while older versions may still grant it harmlessly. AI Teammate setup (`a365 setup all --aiteammate`) still offers the `OtelWrite` application role; complete the app-role action item it prints, because the S2S route accepts that role. --- @@ -252,6 +252,14 @@ the cache. `prefetch` replaces a token five minutes before it expires; `resolve` it until one minute before expiry, so a turn that starts just before the refresh point still exports. +This differs from the Agent365-Samples design for the same reason as the Node scaffold: the +skill reuses the hosting connection (`get_agentic_application_token`) so one deployment can serve +several agent instances/tenants and inherit certificate, federated-identity, or managed-identity +credentials; the samples use a dedicated single-identity OBS credential to isolate telemetry auth. +After a restart, the sync resolver returns `None` for identities that have not handled a turn and +prefetched yet, so replayed records for idle identities can age out (default max record age: 2 days). +That is acceptable during migration because offline storage is disabled. + ```python # observability/app_token_resolver.py # A365 Observability — best-effort instrumentation (verify against official sample) @@ -274,8 +282,10 @@ import time import msal OBSERVABILITY_SCOPE = "api://9b975845-388f-4429-889e-eab1ef63949c/.default" +OBSERVABILITY_AUDIENCES = {"api://9b975845-388f-4429-889e-eab1ef63949c", "9b975845-388f-4429-889e-eab1ef63949c"} SERVE_SKEW_SECONDS = 60 # resolve() stops serving a token this close to expiry REFRESH_SKEW_SECONDS = 300 # prefetch() replaces a token this close to expiry +FAILURE_BACKOFF_SECONDS = 60 class AppTokenResolver: @@ -283,6 +293,7 @@ class AppTokenResolver: def __init__(self): self._tokens = {} + self._failures = {} self._lock = threading.Lock() def resolve(self, agent_id, tenant_id): @@ -302,23 +313,36 @@ class AppTokenResolver: """Acquires the token for this turn's agent identity before its spans are exported.""" if not tenant_id or not agent_id or self._cached(agent_id, tenant_id, REFRESH_SKEW_SECONDS): return - connection = connection_manager.get_default_connection() - assertion = await connection.get_agentic_application_token(tenant_id, agent_id) - if not assertion: - raise RuntimeError("The hosting connection issued no FMI assertion for this agent identity.") - result = await asyncio.to_thread(_acquire, tenant_id, agent_id, assertion) - token = result.get("access_token") - if not token: - raise RuntimeError(f"Observability token request failed: {result.get('error')}") - claims = _claims(token) - if "scp" in claims: - raise RuntimeError("Observability token is delegated (scp claim); the S2S route rejects it.") - client = str(claims.get("azp") or claims.get("appid") or "").lower() - if client != agent_id.lower() or str(claims.get("tid", "")).lower() != tenant_id.lower(): - raise RuntimeError("Observability token client or tenant does not match the exporting agent.") - expires_at = float(claims.get("exp") or time.time() + float(result.get("expires_in", 0))) + key = (tenant_id.lower(), agent_id.lower()) with self._lock: - self._tokens[(tenant_id.lower(), agent_id.lower())] = (token, expires_at) + retry_after = self._failures.get(key) + if retry_after and time.time() < retry_after: + return + try: + connection = connection_manager.get_default_connection() + assertion = await connection.get_agentic_application_token(tenant_id, agent_id) + if not assertion: + raise RuntimeError("The hosting connection issued no FMI assertion for this agent identity.") + result = await asyncio.to_thread(_acquire, tenant_id, agent_id, assertion) + token = result.get("access_token") + if not token: + raise RuntimeError(f"Observability token request failed: {result.get('error')}") + claims = _claims(token) + if "scp" in claims: + raise RuntimeError("Observability token is delegated (scp claim); the S2S route rejects it.") + client = str(claims.get("azp") or claims.get("appid") or "").lower() + if client != agent_id.lower() or str(claims.get("tid", "")).lower() != tenant_id.lower(): + raise RuntimeError("Observability token client or tenant does not match the exporting agent.") + if str(claims.get("aud", "")) not in OBSERVABILITY_AUDIENCES: + raise RuntimeError("Observability token audience does not match the Observability API.") + expires_at = float(claims.get("exp") or time.time() + float(result.get("expires_in", 0))) + with self._lock: + self._tokens[key] = (token, expires_at) + self._failures.pop(key, None) + except Exception: + with self._lock: + self._failures[key] = time.time() + FAILURE_BACKOFF_SECONDS + raise def _acquire(tenant_id, agent_id, assertion): @@ -574,26 +598,29 @@ can only reuse a token another handler cached, and once that token expires its s # OBS_TOKENS is the AppTokenResolver created next to use_microsoft_opentelemetry(...) at startup. -async def _setup_observability_token(self, context, tenant_id, agent_id): - """Prefetch the app-only observability token for this turn's agent identity.""" - try: - await OBS_TOKENS.prefetch(self.connection_manager, tenant_id, agent_id) - except Exception as e: - logger.warning(f"Failed to acquire observability token: {e}") +class GenericAgentHost: + async def _setup_observability_token(self, context, tenant_id, agent_id): + """Prefetch the app-only observability token for this turn's agent identity.""" + try: + await OBS_TOKENS.prefetch(self.connection_manager, tenant_id, agent_id) + except Exception as e: + logger.warning(f"Failed to acquire observability token: {e}") + def _setup_handlers(self): + """Register activity handlers on the adapter.""" -@AGENT_APP.activity("message", auth_handlers=["AGENTIC"]) -async def on_message(context: TurnContext, state: TurnState): - tenant_id = context.activity.recipient.tenant_id - agent_id = context.activity.recipient.agentic_app_id + @self._adapter.on_activity(ActivityTypes.message) + async def on_message(context, state): + tenant_id = context.activity.recipient.tenant_id + agent_id = context.activity.recipient.agentic_app_id - # Prefetch the app-only telemetry token (cached; a no-op on warm turns). - await self._setup_observability_token(context, tenant_id, agent_id) + # Prefetch the app-only telemetry token (cached; a no-op on warm turns). + await self._setup_observability_token(context, tenant_id, agent_id) - # ObservabilityHostingManager (registered at startup) already populated baggage - # from TurnContext. Your handler logic can run directly: - response = await self.invoke_llm(context.activity.text) - await context.send_activity(response) + # ObservabilityHostingManager (registered at startup) already populated baggage + # from TurnContext. Your handler logic can run directly: + response = await self._agent.process_user_message(context.activity.text or "") + await context.send_activity(response) ``` > **`self.connection_manager`** is the `MsalConnectionManager` the host passes to its @@ -610,6 +637,11 @@ async def on_message(context: TurnContext, state: TurnState): > and `cache_agentic_token(...)`. That delegated exchange needs admin consent, and the S2S > route rejects its token. Also switch `a365_token_resolver` from `AgenticTokenCache` / > `get_cached_agentic_token` to `OBS_TOKENS.resolve` and set `a365_use_s2s_endpoint=True`. +> Also set `a365_exporter_disable_offline_storage=True` until the installed release enforces +> S2S for both live and replayed exports, or clear the offline storage directory. After a +> restart, the sync resolver returns `None` for identities that have not handled a turn and +> prefetched yet, so replayed records for idle identities can age out (default max record age: +> 2 days). This is acceptable during migration because offline storage is disabled. #### Canonical: manual per-turn baggage construction (matches AF sample) @@ -884,6 +916,7 @@ when the distro initializes — no manual `*Instrumentor().instrument()` calls n use_microsoft_opentelemetry( enable_a365=True, a365_enable_observability_exporter=True, + a365_use_s2s_endpoint=True, a365_token_resolver=..., ) ``` @@ -1042,7 +1075,7 @@ python -c "from microsoft.opentelemetry import use_microsoft_opentelemetry; from | Token resolver returns `None` | The turn's app-only token was never prefetched or its acquisition failed (see the `Failed to acquire observability token` warning), or the turn has no agent identity | Await `_setup_observability_token(...)` (which calls `OBS_TOKENS.prefetch(...)`) at the start of each handler turn. Check that the hosting connection holds the blueprint credential | | `ModuleNotFoundError: microsoft.opentelemetry` | Package not installed | `pip install microsoft-opentelemetry` | | `uv sync` runs for minutes / appears to hang on a Google ADK project | OTel resolver backtracking between `google-adk` (`opentelemetry-sdk<1.39.0`) and `microsoft-opentelemetry` 1.1.x (newer transitive OTel SDK) | Add `[tool.uv] override-dependencies` to `pyproject.toml` pinning `opentelemetry-api` and `opentelemetry-sdk` to `>=1.38.0,<1.39.0`. See the "Google ADK projects — pin the OTel stack" section above. | -| 401 on export | The exporter is on the delegated route (`a365_use_s2s_endpoint` not set) or received a delegated token | Set `a365_use_s2s_endpoint=True` and use the app-only resolver. The token must have no `scp` claim, and its `azp`/`appid` must equal the exporting agent ID | +| 401 on export | The exporter is on the delegated route (`a365_use_s2s_endpoint` not set) or received the wrong token | Set `a365_use_s2s_endpoint=True` and use the app-only resolver. The token must have no `scp` claim, `azp`/`appid` equal to the exporting agent ID, matching `tid`, and `aud` equal to the Observability API | | Spans dropped silently | Missing tenant/agent ID in baggage | Ensure `enable_baggage=True` and that `populate(builder, context)` runs before scope creation | | Delegated (OBO) telemetry token code still runs in the handler | Telemetry no longer uses a per-turn OBO exchange in any mode | Remove the `exchange_token(... get_observability_authentication_scope() ...)` / `cache_agentic_token` path. Interactive agents prefetch an app-only token; `s2s` agents use the background token service | | S2S 401: wrong Hop 3 scope | FMI Hop 3 used `https://api.powerplatform.com/.default` from older samples | Change Hop 3 scope to `api://9b975845-388f-4429-889e-eab1ef63949c/.default` | diff --git a/plugins/agent365/skills/make-a365-agent/SKILL.md b/plugins/agent365/skills/make-a365-agent/SKILL.md index 37628c4..e46bdcb 100644 --- a/plugins/agent365/skills/make-a365-agent/SKILL.md +++ b/plugins/agent365/skills/make-a365-agent/SKILL.md @@ -288,7 +288,7 @@ Monitor output carefully: After `a365 setup all` completes, show the user: 1. **The Setup Summary table** from CLI output — verbatim. -2. **No Observability API permission is needed.** Recent `a365 setup all` versions no longer request `Agent365.Observability.OtelWrite` (or its admin consent) for blueprint agents. Telemetry is exported over the S2S route with an app-only token, and the route authorizes the registered agent instance. These versions also fail setup (exit code 1) when agent registration fails or cannot be verified. If that happens, show the error and have the user re-run `a365 setup all --agent-registration-only` after fixing it. Older CLI versions may still grant OtelWrite, which is harmless. If the CLI output includes an action item for other permissions (Graph, Bot API, custom resources), display the printed PowerShell script verbatim so the user can hand it to a Global Admin. +2. **No Observability API permission is needed.** Newer `a365 setup all` versions skip `Agent365.Observability.OtelWrite` for blueprint agents and exit 1 when registration fails or cannot be verified. Older CLI versions may still grant OtelWrite (harmless) and may exit 0 after a failed registration, so check the output for a registration error and rerun `a365 setup all --agent-registration-only` after fixing it. Telemetry is exported over the S2S route with an app-only token, and the route authorizes the registered agent instance. If the CLI output includes an action item for other permissions (Graph, Bot API, custom resources), display the printed PowerShell script verbatim so the user can hand it to a Global Admin. 3. **Skip the client secret action item entirely.** Do not show or mention it. Mark Todo 1 as completed. diff --git a/plugins/agent365/skills/make-ai-teammate/SKILL.md b/plugins/agent365/skills/make-ai-teammate/SKILL.md index fb24c39..ada879d 100644 --- a/plugins/agent365/skills/make-ai-teammate/SKILL.md +++ b/plugins/agent365/skills/make-ai-teammate/SKILL.md @@ -329,7 +329,7 @@ Ask: "What language and framework are you using?" and set `language` and `agentS - `ToolingManifest.json` exists → `hasManifest` - **Observability composite** — compute four sub-signals, then combine: - `obs_entry` = `useMicrosoftOpenTelemetry` in any `src/**/*.ts` - - `obs_token` = the `useMicrosoftOpenTelemetry({...})` call passes `tokenResolver` in its `a365` options (the app-only resolver `observability/app-token-resolver.ts` for obo / agentic-user; S2S also accepts `getS2SObservabilityToken` / `startTokenService`). A `tokenResolver` symbol elsewhere, such as an unused import, does not count. AND no `refreshObservabilityToken(..., authorization)` call or `AgenticTokenCacheInstance.getObservabilityToken` resolver (either means the telemetry token is delegated) + - `obs_token` = the `useMicrosoftOpenTelemetry({...})` call passes `tokenResolver` in its `a365` options (the app-only resolver `observability/app-token-resolver.ts` for obo / agentic-user; S2S also accepts `getS2SObservabilityToken` / `startTokenService`). A `tokenResolver` symbol elsewhere, such as an unused import, does not count. AND no `refreshObservabilityToken(...)` call or `AgenticTokenCacheInstance.getObservabilityToken` resolver (either means the telemetry token is delegated) - `obs_route` = `useS2SEndpoint: true` in the same `useMicrosoftOpenTelemetry` call's `a365` options (telemetry uses the S2S route in every auth mode) - `obs_handler` = `BaggageBuilder` OR `BaggageBuilderUtils` OR `InvokeAgentScope` in any `src/**/*.ts` - `has_obs_complete` = `obs_entry && obs_token && obs_route && obs_handler` diff --git a/plugins/agent365/skills/make-ai-teammate/references/deploy-pipeline.md b/plugins/agent365/skills/make-ai-teammate/references/deploy-pipeline.md index c0b36cf..734afda 100644 --- a/plugins/agent365/skills/make-ai-teammate/references/deploy-pipeline.md +++ b/plugins/agent365/skills/make-ai-teammate/references/deploy-pipeline.md @@ -255,8 +255,8 @@ Pull ``, ``, `` from `a3 > **Not in this table (and why)** — the following Node.js keys appeared in older skill versions but are NOT consumed by `@microsoft/Agents-for-js` or `@microsoft/opentelemetry`, so the skill must NOT write them as required vars: > - `USE_AGENTIC_AUTH` — handler selection is in code (`MyAgent.authHandlerName = 'agentic'`); env var is informational only. > - `agentic_connectionName` — invalid key for the agentic handler (only `type`, `scopes`, `altBlueprintConnectionName` are recognized; `connectionName` is an Azure-Bot-handler legacy alias). -> - `agent365Observability__agentBlueprintId` — CLI writes `__agentId`, not `__agentBlueprintId`. Stray. -> - `agent365Observability__clientId` / `__clientSecret` — never written by the CLI and never read by the distro. Stray. +> - `agent365Observability__agentBlueprintId` — written by the CLI; keep it if present. Observability fallback guards use it to avoid treating a blueprint ID as a runtime agent identity. +> - `agent365Observability__clientId` / `__clientSecret` — written by the CLI for compatibility. Current observability wiring gets tokens from the hosting connection instead; keep these keys if present, but do not hand-author them for new code. > - `agent365Observability__sponsorUserId` / `__sponsorUserName` / `__sponsorUserEmail` — S2S-only. For AI Teammate (always `agentic-user`), `CallerDetails` come from the turn context — these env vars are inert. **For `runTarget = "prod"` — additional verification:** @@ -601,4 +601,3 @@ Connect to `http://localhost:3978/api/messages` (or the dev tunnel URL) and send --- - diff --git a/plugins/agent365/skills/make-ai-teammate/references/nodejs-ai-teammate.md b/plugins/agent365/skills/make-ai-teammate/references/nodejs-ai-teammate.md index 046502c..a2f9567 100644 --- a/plugins/agent365/skills/make-ai-teammate/references/nodejs-ai-teammate.md +++ b/plugins/agent365/skills/make-ai-teammate/references/nodejs-ai-teammate.md @@ -741,8 +741,8 @@ Every key below is consumed by something specific — no dead lines, no duplicat > **What's NOT in this template** (and why): > - `USE_AGENTIC_AUTH` — not read by `@microsoft/Agents-for-js`. Handler selection is driven by `MyAgent.authHandlerName = 'agentic'` in code plus the `agentic_*` env keys above. Including it is harmless but informational only. > - `agentic_connectionName` — invalid key for the agentic handler per [authorizationManager.ts](https://github.com/microsoft/Agents-for-js/blob/main/packages/agents-hosting/src/app/auth/authorizationManager.ts). The agentic handler recognizes only `type`, `scopes`, `altBlueprintConnectionName`. (`connectionName` is a legacy alias for `azureBotOAuthConnectionName` — Azure Bot handler only.) -> - `agent365Observability__agentBlueprintId` — never written by the CLI (it writes `__agentId`) and never read by the distro. Stray. -> - `agent365Observability__clientId/clientSecret` — never written by the CLI and never read by `@microsoft/opentelemetry`. Stray. +> - `agent365Observability__agentBlueprintId` — written by the CLI; keep it if present. The observability fallback guard uses it to avoid treating a blueprint ID as a runtime agent identity. +> - `agent365Observability__clientId/clientSecret` — written by the CLI for compatibility. Current `@microsoft/opentelemetry` wiring gets telemetry tokens from the hosting connection instead; keep these keys if present, but do not hand-author them for new code. > - `agent365Observability__sponsorUserId/Name/Email` — S2S-only per [instrument-observability/SKILL.md](../../instrument-observability/SKILL.md). For `agentic-user` (AI Teammate, always), `CallerDetails` come from the turn context, not env vars. Omit. ### The template diff --git a/plugins/agent365/skills/purview-dlp-integration/assets/purview.py b/plugins/agent365/skills/purview-dlp-integration/assets/purview.py index 9eeb13f..582e10d 100644 --- a/plugins/agent365/skills/purview-dlp-integration/assets/purview.py +++ b/plugins/agent365/skills/purview-dlp-integration/assets/purview.py @@ -226,8 +226,8 @@ async def _post(self, authorization, auth_handler_name, activity, sequence_numbe async def _get_token(self, authorization, auth_handler_name, context) -> str: """Agent-identity (agentic delegated) Graph token — carries the delegated - Content.Process.User scope granted on the agent's agentic consent. Uses the same - exchange_token call the A365 host uses for observability.""" + Content.Process.User scope granted on the agent's agentic consent. This token is + for Purview Graph calls only; A365 observability uses an app-only S2S token.""" handler = auth_handler_name or self.auth_handler_name resp = await authorization.exchange_token(context, scopes=[GRAPH_SCOPE], auth_handler_id=handler) token = getattr(resp, "token", None) or (resp if isinstance(resp, str) else None) diff --git a/tests/validate-a365-code-validator.test.js b/tests/validate-a365-code-validator.test.js index cccc2e1..c752438 100644 --- a/tests/validate-a365-code-validator.test.js +++ b/tests/validate-a365-code-validator.test.js @@ -322,13 +322,13 @@ describe('validate-a365-code-validator — S2S route and delegated telemetry', ( } }); - test('Node refreshObservabilityToken(..., authorization) reports delegated token; app-only S2S wiring does not', () => { + test('Node lowercase refreshObservabilityToken without authorization literal reports delegated token; app-only S2S wiring does not', () => { const dir = createFixture({ 'package.json': NODE_PKG, 'index.ts': NODE_S2S_INDEX, 'agent.ts': ` await AgenticTokenCacheInstance.refreshObservabilityToken( - agentId, tenantId, turnContext as any, this.authorization as any); + agentId, tenantId, turnContext as any, app.auth); `.trim(), }); try { @@ -341,6 +341,28 @@ await AgenticTokenCacheInstance.refreshObservabilityToken( } }); + test('Node PascalCase RefreshObservabilityToken app-only overload is allowed; delegated overload is flagged by both scanners', () => { + const appOnly = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': NODE_S2S_INDEX, + 'agent.ts': 'await tokenCache.RefreshObservabilityToken(agentId, tenantId, resolver);', + }); + const delegated = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': NODE_S2S_INDEX, + 'agent.ts': 'await tokenCache.RefreshObservabilityToken(agentId, tenantId, buildContext({ nested: [1, 2] }), app.auth);', + }); + try { + for (const scanner of [VALIDATOR, STANDALONE]) { + assert.ok(!findingIds(runValidator(scanner, appOnly)).includes('node-obs-delegated-token'), scanner); + assert.ok(findingIds(runValidator(scanner, delegated)).includes('node-obs-delegated-token'), scanner); + } + } finally { + cleanup(appOnly); + cleanup(delegated); + } + }); + test('.NET distro without UseS2SEndpoint and with AgenticTokenStruct registration reports both findings', () => { const dir = createFixture({ 'Agent.csproj': '', diff --git a/tests/validate-observability.test.js b/tests/validate-observability.test.js index 442a1af..bca182d 100644 --- a/tests/validate-observability.test.js +++ b/tests/validate-observability.test.js @@ -473,12 +473,12 @@ _agentTokenCache?.RegisterObservability(agentId, tenantId, } finally { cleanup(dir); } }); - test('Node.js refreshObservabilityToken(..., authorization) → reports delegated telemetry token', () => { + test('Node.js lowercase refreshObservabilityToken without authorization literal → reports delegated telemetry token', () => { const dir = createFixture({ ...NODEJS_DISTRO_VALID, 'src/agent.ts': `${NODEJS_DISTRO_VALID['src/agent.ts']} await AgenticTokenCacheInstance.refreshObservabilityToken( - agentId, tenantId, turnContext as any, this.authorization as any);`, + agentId, tenantId, turnContext as any, app.auth);`, }); try { const r = runValidator(VALIDATOR, dir); @@ -487,6 +487,28 @@ await AgenticTokenCacheInstance.refreshObservabilityToken( } finally { cleanup(dir); } }); + test('Node.js PascalCase app-only RefreshObservabilityToken overload is allowed; delegated overload is flagged', () => { + const appOnly = createFixture({ + ...NODEJS_DISTRO_VALID, + 'src/agent.ts': `${NODEJS_DISTRO_VALID['src/agent.ts']} +await tokenCache.RefreshObservabilityToken(agentId, tenantId, resolver);`, + }); + const delegated = createFixture({ + ...NODEJS_DISTRO_VALID, + 'src/agent.ts': `${NODEJS_DISTRO_VALID['src/agent.ts']} +await tokenCache.RefreshObservabilityToken(agentId, tenantId, buildContext({ nested: [1, 2] }), app.auth);`, + }); + try { + assert.equal(runValidator(VALIDATOR, appOnly).ok, true); + const r = runValidator(VALIDATOR, delegated); + assert.equal(r.ok, false); + assert.match(r.reason, /RefreshObservabilityToken.*delegated \(OBO\) telemetry token/); + } finally { + cleanup(appOnly); + cleanup(delegated); + } + }); + test('Node.js comment mentioning refreshObservabilityToken is not flagged', () => { const dir = createFixture({ ...NODEJS_DISTRO_VALID, From 42f24b7ec334ae6e8a34fa4602b89abc74b2a142 Mon Sep 17 00:00:00 2001 From: Krishnadheeraj <12496535+DheerajPannala@users.noreply.github.com> Date: Mon, 28 Sep 2026 17:52:22 +0100 Subject: [PATCH 07/12] Fix the Python host snippet call and clarify offline-storage guidance - The GenericAgentHost snippet now calls process_user_message with the AgentInterface arguments make-ai-teammate generates. - State the offline-storage replay caveat once, conditioned on storage being enabled, and say .NET offline storage is on by default in 1.1.0+. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5cbf5f6b-cc40-4b7e-a591-65848db73a12 --- .../references/dotnet-observability.md | 8 +++--- .../references/python-observability.md | 26 ++++++++++++------- 2 files changed, 20 insertions(+), 14 deletions(-) diff --git a/plugins/agent365/skills/instrument-observability/references/dotnet-observability.md b/plugins/agent365/skills/instrument-observability/references/dotnet-observability.md index 2711d32..57c2c53 100644 --- a/plugins/agent365/skills/instrument-observability/references/dotnet-observability.md +++ b/plugins/agent365/skills/instrument-observability/references/dotnet-observability.md @@ -682,10 +682,10 @@ builder.AddA365Tracing(); // from Microsoft.Agents.A365.O These are subsumed by `UseMicrosoftOpenTelemetry()` and the distro package — mixing the two causes CS0433 duplicate-type errors. Pick one wiring style per project. The legacy `AddAgenticTracingExporter()` wiring exports over the delegated route with OBO tokens; migrate -it to the distro with `o.Agent365.UseS2SEndpoint = true` and `AgentAppTokenResolver`. If offline -storage is enabled, set `o.Agent365.DisableOfflineStorage = true` (Microsoft.OpenTelemetry 1.1.0+) -until the installed release enforces S2S for both live and replayed exports, or clear the storage -directory. +it to the distro with `o.Agent365.UseS2SEndpoint = true` and `AgentAppTokenResolver`. Offline +storage is on by default in Microsoft.OpenTelemetry 1.1.0+, so set +`o.Agent365.DisableOfflineStorage = true` until the installed release enforces S2S for both live +and replayed exports, or clear the storage directory. --- diff --git a/plugins/agent365/skills/instrument-observability/references/python-observability.md b/plugins/agent365/skills/instrument-observability/references/python-observability.md index 98c81e0..540b56c 100644 --- a/plugins/agent365/skills/instrument-observability/references/python-observability.md +++ b/plugins/agent365/skills/instrument-observability/references/python-observability.md @@ -256,9 +256,11 @@ This differs from the Agent365-Samples design for the same reason as the Node sc skill reuses the hosting connection (`get_agentic_application_token`) so one deployment can serve several agent instances/tenants and inherit certificate, federated-identity, or managed-identity credentials; the samples use a dedicated single-identity OBS credential to isolate telemetry auth. -After a restart, the sync resolver returns `None` for identities that have not handled a turn and -prefetched yet, so replayed records for idle identities can age out (default max record age: 2 days). -That is acceptable during migration because offline storage is disabled. + +Offline storage caveat: after a restart, `resolve` returns `None` for an identity until it handles +a turn and prefetches a token. If offline storage is enabled, records replayed for identities that +stay idle can therefore age out (default maximum record age: 2 days). The migration below disables +offline storage, so this matters only if you turn it back on. ```python # observability/app_token_resolver.py @@ -618,9 +620,15 @@ class GenericAgentHost: await self._setup_observability_token(context, tenant_id, agent_id) # ObservabilityHostingManager (registered at startup) already populated baggage - # from TurnContext. Your handler logic can run directly: - response = await self._agent.process_user_message(context.activity.text or "") - await context.send_activity(response) + # from TurnContext. Continue with the existing handler logic, for example the + # AgentInterface call that make-ai-teammate generates: + reply = await self._agent.process_user_message( + context.activity.text or "", + self._adapter.authorization, + AUTH_HANDLER_NAME or None, + context, + ) + await context.send_activity(reply) ``` > **`self.connection_manager`** is the `MsalConnectionManager` the host passes to its @@ -638,10 +646,8 @@ class GenericAgentHost: > route rejects its token. Also switch `a365_token_resolver` from `AgenticTokenCache` / > `get_cached_agentic_token` to `OBS_TOKENS.resolve` and set `a365_use_s2s_endpoint=True`. > Also set `a365_exporter_disable_offline_storage=True` until the installed release enforces -> S2S for both live and replayed exports, or clear the offline storage directory. After a -> restart, the sync resolver returns `None` for identities that have not handled a turn and -> prefetched yet, so replayed records for idle identities can age out (default max record age: -> 2 days). This is acceptable during migration because offline storage is disabled. +> S2S for both live and replayed exports, or clear the offline storage directory (see the +> offline storage caveat under the app-only resolver scaffold). #### Canonical: manual per-turn baggage construction (matches AF sample) From 687181a85ff98f9555247b1eff514a2c61e73586 Mon Sep 17 00:00:00 2001 From: Krishnadheeraj <12496535+DheerajPannala@users.noreply.github.com> Date: Mon, 28 Sep 2026 18:26:52 +0100 Subject: [PATCH 08/12] Tighten observability option scanning - require direct a365 route and resolver properties in scanner helper - reject nested, sibling, and unresolvable observability options - add regression coverage for direct and identifier-resolved a365 options Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5cbf5f6b-cc40-4b7e-a591-65848db73a12 --- .../stop/validate-a365-code-validator.js | 89 ++++++++++++++++--- .../stop/validate-instrument-observability.js | 89 ++++++++++++++++--- .../references/a365-code-validator.js | 89 ++++++++++++++++--- .../references/nodejs-observability.md | 14 ++- tests/validate-a365-code-validator.test.js | 47 +++++++++- tests/validate-observability.test.js | 55 +++++++++++- 6 files changed, 334 insertions(+), 49 deletions(-) diff --git a/plugins/agent365/hooks/stop/validate-a365-code-validator.js b/plugins/agent365/hooks/stop/validate-a365-code-validator.js index c94fb76..86607e7 100644 --- a/plugins/agent365/hooks/stop/validate-a365-code-validator.js +++ b/plugins/agent365/hooks/stop/validate-a365-code-validator.js @@ -201,8 +201,80 @@ function pythonKeywordArguments(args) { return keywords; } +function stripOuterGroup(text) { + const trimmed = text.trim(); + if (!'({['.includes(trimmed[0] || '')) return trimmed; + const block = bracketBlock(trimmed, 0); + return block && block.length === trimmed.length ? trimmed.slice(1, -1) : trimmed; +} + +function splitTopLevel(text) { + const fields = []; + const inner = stripOuterGroup(text); + let start = 0; + let depth = 0; + let quote = ''; + let escaped = false; + for (let i = 0; i < inner.length; i++) { + const ch = inner[i]; + if (quote) { + if (escaped) escaped = false; + else if (ch === '\\') escaped = true; + else if (ch === quote) quote = ''; + continue; + } + if (ch === '"' || ch === "'" || ch === '`') { + quote = ch; + } else if (ch === '(' || ch === '[' || ch === '{') { + depth++; + } else if (ch === ')' || ch === ']' || ch === '}') { + depth = Math.max(0, depth - 1); + } else if (ch === ',' && depth === 0) { + fields.push(inner.slice(start, i)); + start = i + 1; + } + } + fields.push(inner.slice(start)); + return fields.map(field => field.trim()).filter(Boolean); +} + +function directOptionMatches(field, language, option) { + const direct = field.trim(); + if (language === 'node') { + if (option === S2S_ROUTE_OPTION.node) { + return /^(?:useS2SEndpoint|['"]useS2SEndpoint['"])\s*:\s*true\b/.test(direct); + } + if (option === TOKEN_RESOLVER_OPTION.node) { + return /^(?:tokenResolver|['"]tokenResolver['"])\s*(?::|$)/.test(direct); + } + return false; + } + if (language === 'python') { + const match = direct.match(/^(?:['"])?(a365_(?:contextual_)?token_resolver|a365_use_s2s_endpoint)(?:['"])?\s*(=|:)\s*([\s\S]*)$/); + if (!match) return false; + return option.test(`${match[1]}=${match[3]}`) || option.test(`"${match[1]}": ${match[3]}`); + } + return option.test(direct); +} + +function objectPassesOption(content, text, language, option, seen = new Set()) { + return splitTopLevel(text).some(field => { + const spread = field.match(/^(?:\.\.\.|\*\*)\s*([A-Za-z_$][\w$]*)$/); + if (spread && !seen.has(spread[1])) { + seen.add(spread[1]); + return initializerBlocks(content, spread[1]).some(init => objectPassesOption(content, init, language, option, seen)); + } + const inlineSpread = field.match(/^\*\*\s*(?:dict\s*)?([({])/); + if (inlineSpread) { + const block = bracketBlock(field, inlineSpread.index + inlineSpread[0].length - 1); + return block ? objectPassesOption(content, block, language, option, seen) : false; + } + return directOptionMatches(field, language, option); + }); +} + // Node.js: the `a365` options objects in `text`, inline (`a365: {...}`), by variable (`a365: options`), -// or shorthand (`{ a365 }`). Null when a value is a call that cannot be read statically. +// or shorthand (`{ a365 }`). Unresolvable values are treated as unwired. function a365Objects(content, text) { const objects = []; const property = /(?:^|[{,\s])a365\s*:\s*/g; @@ -213,7 +285,7 @@ function a365Objects(content, text) { continue; } const value = text.slice(at).match(/^[A-Za-z_$][\w$.]*(\s*\()?/); - if (!value || value[1]) return null; + if (!value || value[1]) continue; objects.push(...initializerBlocks(content, value[0])); } if (/[{,]\s*a365\s*(?=[,}])/.test(text)) objects.push(...initializerBlocks(content, 'a365')); @@ -239,21 +311,16 @@ function distroCallMatches(files, language, option) { function callPassesOption(content, outside, args, language, option) { if (language === 'dotnet') { - // A method group, or a callback that never touches Agent365, configures the options elsewhere. - if (!args.includes('=>') || !/\bAgent365\b/.test(args)) return option.test(content); + if (!args.includes('=>') || !/\bAgent365\b/.test(args)) return false; return [args, ...referencedInitializers(outside, args, language)].some(text => option.test(text)); } const texts = [language === 'python' ? pythonKeywordArguments(args) : args, ...referencedInitializers(outside, args, language)]; - if (language === 'python') return texts.some(text => option.test(text)); + if (language === 'python') return texts.some(text => objectPassesOption(outside, text, language, option)); const objects = []; for (const text of texts) { - const found = a365Objects(outside, text); - if (found === null) return option.test(content); - objects.push(...found); + objects.push(...a365Objects(outside, text)); } - if (objects.length === 0) return option.test(content); - return objects.some(object => - option.test(object) || referencedInitializers(outside, object, language).some(init => option.test(init))); + return objects.some(object => objectPassesOption(outside, object, language, option)); } function distroCallHasResolver(files, language) { diff --git a/plugins/agent365/hooks/stop/validate-instrument-observability.js b/plugins/agent365/hooks/stop/validate-instrument-observability.js index 2abbcec..4d836ab 100644 --- a/plugins/agent365/hooks/stop/validate-instrument-observability.js +++ b/plugins/agent365/hooks/stop/validate-instrument-observability.js @@ -130,8 +130,80 @@ function pythonKeywordArguments(args) { return keywords; } +function stripOuterGroup(text) { + const trimmed = text.trim(); + if (!'({['.includes(trimmed[0] || '')) return trimmed; + const block = bracketBlock(trimmed, 0); + return block && block.length === trimmed.length ? trimmed.slice(1, -1) : trimmed; +} + +function splitTopLevel(text) { + const fields = []; + const inner = stripOuterGroup(text); + let start = 0; + let depth = 0; + let quote = ''; + let escaped = false; + for (let i = 0; i < inner.length; i++) { + const ch = inner[i]; + if (quote) { + if (escaped) escaped = false; + else if (ch === '\\') escaped = true; + else if (ch === quote) quote = ''; + continue; + } + if (ch === '"' || ch === "'" || ch === '`') { + quote = ch; + } else if (ch === '(' || ch === '[' || ch === '{') { + depth++; + } else if (ch === ')' || ch === ']' || ch === '}') { + depth = Math.max(0, depth - 1); + } else if (ch === ',' && depth === 0) { + fields.push(inner.slice(start, i)); + start = i + 1; + } + } + fields.push(inner.slice(start)); + return fields.map(field => field.trim()).filter(Boolean); +} + +function directOptionMatches(field, language, option) { + const direct = field.trim(); + if (language === 'node') { + if (option === S2S_ROUTE_OPTION.node) { + return /^(?:useS2SEndpoint|['"]useS2SEndpoint['"])\s*:\s*true\b/.test(direct); + } + if (option === TOKEN_RESOLVER_OPTION.node) { + return /^(?:tokenResolver|['"]tokenResolver['"])\s*(?::|$)/.test(direct); + } + return false; + } + if (language === 'python') { + const match = direct.match(/^(?:['"])?(a365_(?:contextual_)?token_resolver|a365_use_s2s_endpoint)(?:['"])?\s*(=|:)\s*([\s\S]*)$/); + if (!match) return false; + return option.test(`${match[1]}=${match[3]}`) || option.test(`"${match[1]}": ${match[3]}`); + } + return option.test(direct); +} + +function objectPassesOption(content, text, language, option, seen = new Set()) { + return splitTopLevel(text).some(field => { + const spread = field.match(/^(?:\.\.\.|\*\*)\s*([A-Za-z_$][\w$]*)$/); + if (spread && !seen.has(spread[1])) { + seen.add(spread[1]); + return initializerBlocks(content, spread[1]).some(init => objectPassesOption(content, init, language, option, seen)); + } + const inlineSpread = field.match(/^\*\*\s*(?:dict\s*)?([({])/); + if (inlineSpread) { + const block = bracketBlock(field, inlineSpread.index + inlineSpread[0].length - 1); + return block ? objectPassesOption(content, block, language, option, seen) : false; + } + return directOptionMatches(field, language, option); + }); +} + // Node.js: the `a365` options objects in `text`, inline (`a365: {...}`), by variable (`a365: options`), -// or shorthand (`{ a365 }`). Null when a value is a call that cannot be read statically. +// or shorthand (`{ a365 }`). Unresolvable values are treated as unwired. function a365Objects(content, text) { const objects = []; const property = /(?:^|[{,\s])a365\s*:\s*/g; @@ -142,7 +214,7 @@ function a365Objects(content, text) { continue; } const value = text.slice(at).match(/^[A-Za-z_$][\w$.]*(\s*\()?/); - if (!value || value[1]) return null; + if (!value || value[1]) continue; objects.push(...initializerBlocks(content, value[0])); } if (/[{,]\s*a365\s*(?=[,}])/.test(text)) objects.push(...initializerBlocks(content, 'a365')); @@ -168,21 +240,16 @@ function distroCallMatches(files, language, option) { function callPassesOption(content, outside, args, language, option) { if (language === 'dotnet') { - // A method group, or a callback that never touches Agent365, configures the options elsewhere. - if (!args.includes('=>') || !/\bAgent365\b/.test(args)) return option.test(content); + if (!args.includes('=>') || !/\bAgent365\b/.test(args)) return false; return [args, ...referencedInitializers(outside, args, language)].some(text => option.test(text)); } const texts = [language === 'python' ? pythonKeywordArguments(args) : args, ...referencedInitializers(outside, args, language)]; - if (language === 'python') return texts.some(text => option.test(text)); + if (language === 'python') return texts.some(text => objectPassesOption(outside, text, language, option)); const objects = []; for (const text of texts) { - const found = a365Objects(outside, text); - if (found === null) return option.test(content); - objects.push(...found); + objects.push(...a365Objects(outside, text)); } - if (objects.length === 0) return option.test(content); - return objects.some(object => - option.test(object) || referencedInitializers(outside, object, language).some(init => option.test(init))); + return objects.some(object => objectPassesOption(outside, object, language, option)); } function distroCallHasResolver(files, language) { diff --git a/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js b/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js index 408e3d1..a637a50 100644 --- a/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js +++ b/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js @@ -199,8 +199,80 @@ function pythonKeywordArguments(args) { return keywords; } +function stripOuterGroup(text) { + const trimmed = text.trim(); + if (!'({['.includes(trimmed[0] || '')) return trimmed; + const block = bracketBlock(trimmed, 0); + return block && block.length === trimmed.length ? trimmed.slice(1, -1) : trimmed; +} + +function splitTopLevel(text) { + const fields = []; + const inner = stripOuterGroup(text); + let start = 0; + let depth = 0; + let quote = ''; + let escaped = false; + for (let i = 0; i < inner.length; i++) { + const ch = inner[i]; + if (quote) { + if (escaped) escaped = false; + else if (ch === '\\') escaped = true; + else if (ch === quote) quote = ''; + continue; + } + if (ch === '"' || ch === "'" || ch === '`') { + quote = ch; + } else if (ch === '(' || ch === '[' || ch === '{') { + depth++; + } else if (ch === ')' || ch === ']' || ch === '}') { + depth = Math.max(0, depth - 1); + } else if (ch === ',' && depth === 0) { + fields.push(inner.slice(start, i)); + start = i + 1; + } + } + fields.push(inner.slice(start)); + return fields.map(field => field.trim()).filter(Boolean); +} + +function directOptionMatches(field, language, option) { + const direct = field.trim(); + if (language === 'node') { + if (option === S2S_ROUTE_OPTION.node) { + return /^(?:useS2SEndpoint|['"]useS2SEndpoint['"])\s*:\s*true\b/.test(direct); + } + if (option === TOKEN_RESOLVER_OPTION.node) { + return /^(?:tokenResolver|['"]tokenResolver['"])\s*(?::|$)/.test(direct); + } + return false; + } + if (language === 'python') { + const match = direct.match(/^(?:['"])?(a365_(?:contextual_)?token_resolver|a365_use_s2s_endpoint)(?:['"])?\s*(=|:)\s*([\s\S]*)$/); + if (!match) return false; + return option.test(`${match[1]}=${match[3]}`) || option.test(`"${match[1]}": ${match[3]}`); + } + return option.test(direct); +} + +function objectPassesOption(content, text, language, option, seen = new Set()) { + return splitTopLevel(text).some(field => { + const spread = field.match(/^(?:\.\.\.|\*\*)\s*([A-Za-z_$][\w$]*)$/); + if (spread && !seen.has(spread[1])) { + seen.add(spread[1]); + return initializerBlocks(content, spread[1]).some(init => objectPassesOption(content, init, language, option, seen)); + } + const inlineSpread = field.match(/^\*\*\s*(?:dict\s*)?([({])/); + if (inlineSpread) { + const block = bracketBlock(field, inlineSpread.index + inlineSpread[0].length - 1); + return block ? objectPassesOption(content, block, language, option, seen) : false; + } + return directOptionMatches(field, language, option); + }); +} + // Node.js: the `a365` options objects in `text`, inline (`a365: {...}`), by variable (`a365: options`), -// or shorthand (`{ a365 }`). Null when a value is a call that cannot be read statically. +// or shorthand (`{ a365 }`). Unresolvable values are treated as unwired. function a365Objects(content, text) { const objects = []; const property = /(?:^|[{,\s])a365\s*:\s*/g; @@ -211,7 +283,7 @@ function a365Objects(content, text) { continue; } const value = text.slice(at).match(/^[A-Za-z_$][\w$.]*(\s*\()?/); - if (!value || value[1]) return null; + if (!value || value[1]) continue; objects.push(...initializerBlocks(content, value[0])); } if (/[{,]\s*a365\s*(?=[,}])/.test(text)) objects.push(...initializerBlocks(content, 'a365')); @@ -237,21 +309,16 @@ function distroCallMatches(files, language, option) { function callPassesOption(content, outside, args, language, option) { if (language === 'dotnet') { - // A method group, or a callback that never touches Agent365, configures the options elsewhere. - if (!args.includes('=>') || !/\bAgent365\b/.test(args)) return option.test(content); + if (!args.includes('=>') || !/\bAgent365\b/.test(args)) return false; return [args, ...referencedInitializers(outside, args, language)].some(text => option.test(text)); } const texts = [language === 'python' ? pythonKeywordArguments(args) : args, ...referencedInitializers(outside, args, language)]; - if (language === 'python') return texts.some(text => option.test(text)); + if (language === 'python') return texts.some(text => objectPassesOption(outside, text, language, option)); const objects = []; for (const text of texts) { - const found = a365Objects(outside, text); - if (found === null) return option.test(content); - objects.push(...found); + objects.push(...a365Objects(outside, text)); } - if (objects.length === 0) return option.test(content); - return objects.some(object => - option.test(object) || referencedInitializers(outside, object, language).some(init => option.test(init))); + return objects.some(object => objectPassesOption(outside, object, language, option)); } function distroCallHasResolver(files, language) { diff --git a/plugins/agent365/skills/instrument-observability/references/nodejs-observability.md b/plugins/agent365/skills/instrument-observability/references/nodejs-observability.md index 0d4393e..39dba13 100644 --- a/plugins/agent365/skills/instrument-observability/references/nodejs-observability.md +++ b/plugins/agent365/skills/instrument-observability/references/nodejs-observability.md @@ -210,14 +210,12 @@ const a365TokenResolver = (agentId: string, tenantId: string) => useMicrosoftOpenTelemetry({ resource: resourceFromAttributes({ 'service.name': AGENT_NAME }), - a365: A365_ENABLED - ? { - enabled: true, - enableObservabilityExporter: true, - useS2SEndpoint: true, // ← first-class option, no workaround needed - tokenResolver: a365TokenResolver, - } - : undefined, + a365: { + enabled: A365_ENABLED, + enableObservabilityExporter: A365_ENABLED, + useS2SEndpoint: true, // ← first-class option, no workaround needed + tokenResolver: a365TokenResolver, + }, }); // ... import app modules AFTER observability init ... diff --git a/tests/validate-a365-code-validator.test.js b/tests/validate-a365-code-validator.test.js index c752438..5be7a21 100644 --- a/tests/validate-a365-code-validator.test.js +++ b/tests/validate-a365-code-validator.test.js @@ -603,10 +603,7 @@ token = await auth.exchange_token(context, scopes=["ea9ffc3e-8a23-4a7d-836d-234d 'useMicrosoftOpenTelemetry(otelOptions);', ].join('\n'), 'Agent.csproj': csproj, - 'Program.cs': [ - 'builder.UseMicrosoftOpenTelemetry(ConfigureTelemetry);', - 'static void ConfigureTelemetry(MicrosoftOpenTelemetryOptions o) { o.Agent365.UseS2SEndpoint = true; o.Agent365.TokenResolver = (a, t) => r.ResolveAsync(a, t); }', - ].join('\n'), + 'Program.cs': 'builder.UseMicrosoftOpenTelemetry(o => { o.Agent365.UseS2SEndpoint = true; o.Agent365.TokenResolver = (a, t) => r.ResolveAsync(a, t); });', 'requirements.txt': 'microsoft-opentelemetry>=1.1.0\n', 'host.py': [ 'OTEL_KWARGS = dict(enable_a365=True, a365_enable_observability_exporter=True, a365_use_s2s_endpoint=True, a365_token_resolver=OBS_TOKENS.resolve)', @@ -629,6 +626,48 @@ token = await auth.exchange_token(context, scopes=["ea9ffc3e-8a23-4a7d-836d-234d } }); + test('Node a365 direct properties and resolved identifiers pass; nested or unresolvable options fail in both scanners', () => { + const direct = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': NODE_S2S_INDEX, + }); + const resolved = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': [ + 'const a365Options = { enabled: true, enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver: appTokenResolver };', + 'useMicrosoftOpenTelemetry({ a365: a365Options });', + ].join('\n'), + }); + const nested = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': 'useMicrosoftOpenTelemetry({ a365: { enabled: true, enableObservabilityExporter: true, custom: { useS2SEndpoint: true, tokenResolver: appTokenResolver } } });', + }); + const unresolvable = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': 'useMicrosoftOpenTelemetry({ a365: buildA365Options() });\nfunction buildA365Options() { return { enabled: true, enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver: appTokenResolver }; }', + }); + const ids = result => findingIds(result).filter(id => /node-obs-delegated-route|node-obs-token-resolver-missing/.test(id)).sort(); + try { + for (const scanner of [VALIDATOR, STANDALONE]) { + assert.deepEqual(ids(runValidator(scanner, direct)), [], scanner); + assert.deepEqual(ids(runValidator(scanner, resolved)), [], scanner); + assert.deepEqual(ids(runValidator(scanner, nested)), [ + 'node-obs-delegated-route', + 'node-obs-token-resolver-missing', + ], scanner); + assert.deepEqual(ids(runValidator(scanner, unresolvable)), [ + 'node-obs-delegated-route', + 'node-obs-token-resolver-missing', + ], scanner); + } + } finally { + cleanup(direct); + cleanup(resolved); + cleanup(nested); + cleanup(unresolvable); + } + }); + test('the stop hooks and the standalone scanner share one distro-call helper', () => { const observabilityHook = path.join(__dirname, '../plugins/agent365/hooks/stop/validate-instrument-observability.js'); const helperText = file => { diff --git a/tests/validate-observability.test.js b/tests/validate-observability.test.js index bca182d..2c6843e 100644 --- a/tests/validate-observability.test.js +++ b/tests/validate-observability.test.js @@ -929,6 +929,49 @@ useMicrosoftOpenTelemetry(otelOptions); } finally { cleanup(dir); } }); + test('Node.js route and resolver nested under an unrelated a365 child object → reports both requirements', () => { + const dir = createFixture({ + ...NODEJS_DISTRO_VALID, + 'src/index.ts': NODEJS_DISTRO_VALID['src/index.ts'].replace( + '{ a365: { enabled: true, enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver: appTokenResolver } }', + '{ a365: { enabled: true, enableObservabilityExporter: true, custom: { useS2SEndpoint: true, tokenResolver: appTokenResolver } } }'), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /S2S route in every auth mode/); + assert.match(r.reason, /no a365 tokenResolver/); + } finally { cleanup(dir); } + }); + + test('Node.js a365 identifier resolved to an object literal → ok', () => { + const dir = createFixture({ + ...NODEJS_DISTRO_VALID, + 'src/index.ts': NODEJS_DISTRO_VALID['src/index.ts'].replace( + 'useMicrosoftOpenTelemetry({ a365: { enabled: true, enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver: appTokenResolver } });', + 'const a365Options = { enabled: true, enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver: appTokenResolver };\nuseMicrosoftOpenTelemetry({ a365: a365Options });'), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, true, r.reason); + } finally { cleanup(dir); } + }); + + test('Node.js unresolvable a365 identifier → reports both requirements', () => { + const dir = createFixture({ + ...NODEJS_DISTRO_VALID, + 'src/index.ts': NODEJS_DISTRO_VALID['src/index.ts'].replace( + 'useMicrosoftOpenTelemetry({ a365: { enabled: true, enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver: appTokenResolver } });', + 'useMicrosoftOpenTelemetry({ a365: buildA365Options() });'), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, false); + assert.match(r.reason, /S2S route in every auth mode/); + assert.match(r.reason, /no a365 tokenResolver/); + } finally { cleanup(dir); } + }); + test('Node.js shorthand a365 options built with a spread → ok', () => { const dir = createFixture({ ...NODEJS_DISTRO_VALID, @@ -942,7 +985,7 @@ useMicrosoftOpenTelemetry(otelOptions); } finally { cleanup(dir); } }); - test('Node.js options built by a factory call → checks the file', () => { + test('Node.js options built by a factory call → reports both requirements', () => { const dir = createFixture({ ...NODEJS_DISTRO_VALID, 'src/index.ts': NODEJS_DISTRO_VALID['src/index.ts'].replace( @@ -951,7 +994,9 @@ useMicrosoftOpenTelemetry(otelOptions); }); try { const r = runValidator(VALIDATOR, dir); - assert.equal(r.ok, true, r.reason); + assert.equal(r.ok, false); + assert.match(r.reason, /S2S route in every auth mode/); + assert.match(r.reason, /no a365 tokenResolver/); } finally { cleanup(dir); } }); @@ -968,14 +1013,16 @@ useMicrosoftOpenTelemetry(otelOptions); } finally { cleanup(dir); } }); - test('.NET options configured by a method the callback calls → ok', () => { + test('.NET options configured by a method the callback calls → reports both requirements', () => { const dir = createFixture({ ...DOTNET_DISTRO_VALID, 'Program.cs': 'builder.UseMicrosoftOpenTelemetry(o => ConfigureTelemetry(o));\nstatic void ConfigureTelemetry(MicrosoftOpenTelemetryOptions o)\n{\n o.Agent365.UseS2SEndpoint = true;\n o.Agent365.TokenResolver = (agentId, tenantId) => obsTokens?.ResolveAsync(agentId, tenantId) ?? Task.FromResult(null);\n}', }); try { const r = runValidator(VALIDATOR, dir); - assert.equal(r.ok, true, r.reason); + assert.equal(r.ok, false); + assert.match(r.reason, /S2S route in every auth mode/); + assert.match(r.reason, /without o\.Agent365\.TokenResolver/); } finally { cleanup(dir); } }); From e7f9fb7e34d573d8f8f76c30cc6f85071296e23e Mon Sep 17 00:00:00 2001 From: Krishnadheeraj <12496535+DheerajPannala@users.noreply.github.com> Date: Mon, 28 Sep 2026 20:02:32 +0100 Subject: [PATCH 09/12] Fix dynamic A365 option validation - evaluate Node A365 activation/exporter state per distro call - keep route and resolver checks call-scoped with no file fallback - use workspace detection cache for system-agent registration warnings Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5cbf5f6b-cc40-4b7e-a591-65848db73a12 --- .../stop/validate-a365-code-validator.js | 97 ++++++++++++++++--- .../stop/validate-instrument-observability.js | 75 +++++++++++++- .../references/a365-code-validator.js | 97 ++++++++++++++++--- tests/validate-a365-code-validator.test.js | 85 ++++++++++++++-- 4 files changed, 323 insertions(+), 31 deletions(-) diff --git a/plugins/agent365/hooks/stop/validate-a365-code-validator.js b/plugins/agent365/hooks/stop/validate-a365-code-validator.js index 86607e7..10a5d18 100644 --- a/plugins/agent365/hooks/stop/validate-a365-code-validator.js +++ b/plugins/agent365/hooks/stop/validate-a365-code-validator.js @@ -114,8 +114,8 @@ function findCallBlocks(content, functionName) { // (Node.js), as a keyword argument of use_microsoft_opentelemetry (Python), or on `.Agent365` in the // UseMicrosoftOpenTelemetry options callback (.NET). A variable is followed one level to its // initializer in the same file. Comments are ignored, so a commented-out option, an unused import, or -// an unrelated object does not count. When the options cannot be located statically (a method group, -// a factory call, or a callback that never touches Agent365), the file as a whole is checked instead. +// an unrelated object does not count. Unresolvable options are treated as unwired; there is no +// whole-file fallback. const DISTRO_CALLS = { node: { call: 'useMicrosoftOpenTelemetry', comments: /\/\*[\s\S]*?\*\/|(^|[\s,{;(])\/\/[^\n]*/gm }, python: { call: 'use_microsoft_opentelemetry', comments: /(^|\s)#[^\n]*/gm }, @@ -257,6 +257,44 @@ function directOptionMatches(field, language, option) { return option.test(direct); } +function directPropertyValue(field, propertyName) { + const direct = field.trim(); + const quoted = `(?:${escapeRegExp(propertyName)}|['"]${escapeRegExp(propertyName)}['"])`; + const match = direct.match(new RegExp(`^${quoted}\\s*(?::\\s*([\\s\\S]*))?$`)); + if (!match) return { present: false, value: '' }; + return { present: true, value: match[1] === undefined ? null : match[1].trim() }; +} + +function objectPropertyValues(content, text, propertyName, seen = new Set()) { + const values = []; + for (const field of splitTopLevel(text)) { + const spread = field.match(/^(?:\.\.\.|\*\*)\s*([A-Za-z_$][\w$]*)$/); + if (spread && !seen.has(spread[1])) { + seen.add(spread[1]); + for (const init of initializerBlocks(content, spread[1])) { + values.push(...objectPropertyValues(content, init, propertyName, seen)); + } + continue; + } + const inlineSpread = field.match(/^\*\*\s*(?:dict\s*)?([({])/); + if (inlineSpread) { + const block = bracketBlock(field, inlineSpread.index + inlineSpread[0].length - 1); + if (block) values.push(...objectPropertyValues(content, block, propertyName, seen)); + continue; + } + const property = directPropertyValue(field, propertyName); + if (property.present) values.push(property.value); + } + return values; +} + +function booleanLiteral(value) { + if (value === null) return 'dynamic'; + if (/^true\b/.test(value)) return 'true'; + if (/^false\b/.test(value)) return 'false'; + return 'dynamic'; +} + function objectPassesOption(content, text, language, option, seen = new Set()) { return splitTopLevel(text).some(field => { const spread = field.match(/^(?:\.\.\.|\*\*)\s*([A-Za-z_$][\w$]*)$/); @@ -323,6 +361,39 @@ function callPassesOption(content, outside, args, language, option) { return objects.some(object => objectPassesOption(outside, object, language, option)); } +function nodeDistroCallStates(files) { + const states = []; + const { call } = DISTRO_CALLS.node; + for (const file of files) { + const content = stripComments(read(file), 'node'); + for (let index = content.indexOf(`${call}(`); index !== -1; index = content.indexOf(`${call}(`, index + 1)) { + const open = index + call.length; + const args = bracketBlock(content, open); + const outside = content.slice(0, open) + ' '.repeat(args.length) + content.slice(open + args.length); + const texts = [args, ...referencedInitializers(outside, args, 'node')]; + const objects = texts.flatMap(text => a365Objects(outside, text)); + if (objects.length === 0) { + states.push({ file, active: true, exporter: 'dynamic', hasRoute: false, hasResolver: false }); + continue; + } + for (const object of objects) { + const enabledValues = objectPropertyValues(outside, object, 'enabled'); + const enabled = enabledValues.length ? booleanLiteral(enabledValues.at(-1)) : 'false'; + const exporterValues = objectPropertyValues(outside, object, 'enableObservabilityExporter'); + const exporter = exporterValues.length ? booleanLiteral(exporterValues.at(-1)) : 'absent'; + states.push({ + file, + active: enabled !== 'false', + exporter, + hasRoute: objectPassesOption(outside, object, 'node', S2S_ROUTE_OPTION.node), + hasResolver: objectPassesOption(outside, object, 'node', TOKEN_RESOLVER_OPTION.node), + }); + } + } + } + return states; +} + function distroCallHasResolver(files, language) { return distroCallMatches(files, language, TOKEN_RESOLVER_OPTION[language]); } @@ -558,9 +629,10 @@ function validatePython() { function validateNode() { const hasMicrosoftOtelPackage = packageJsonFiles.some(f => fileContains(f, '@microsoft/opentelemetry')); const hasDistroCall = anyFileContains(tsFiles, 'useMicrosoftOpenTelemetry'); - const hasA365Enabled = anyFileContains(tsFiles, 'enabled: true') || anyFileContains(tsFiles, 'enabled:true'); - const hasExporterFlag = anyFileMatches(tsFiles, /\benableObservabilityExporter\s*:\s*true\b/); - const hasExporterFalse = anyFileMatches(tsFiles, /\benableObservabilityExporter\s*:\s*false\b/); + const nodeStates = nodeDistroCallStates(tsFiles); + const hasA365Enabled = nodeStates.some(state => state.active); + const hasExporterFlag = nodeStates.some(state => state.active && (state.exporter === 'true' || state.exporter === 'dynamic')); + const hasExporterFalse = nodeStates.some(state => state.active && state.exporter === 'false'); const hasExporterEnv = envHasTruthy('ENABLE_A365_OBSERVABILITY_EXPORTER'); if (hasMicrosoftOtelPackage && !hasDistroCall) { @@ -577,7 +649,7 @@ function validateNode() { 'critical', 'node-exporter-explicitly-disabled', 'Node code sets enableObservabilityExporter:false. This disables A365 backend export.', - tsFiles.find(f => /enableObservabilityExporter\s*:\s*false\b/.test(read(f))) + (nodeStates.find(state => state.active && state.exporter === 'false') || {}).file ); } else if (hasDistroCall && hasA365Enabled && !hasExporterFlag && !hasExporterEnv) { add( @@ -588,7 +660,7 @@ function validateNode() { ); } - if (hasDistroCall && !anyFileContains(tsFiles, 'BaggageBuilder') && !anyFileContains(tsFiles, 'InvokeAgentScope') && !anyFileContains(tsFiles, 'configureA365Hosting')) { + if (hasDistroCall && hasA365Enabled && !anyFileContains(tsFiles, 'BaggageBuilder') && !anyFileContains(tsFiles, 'InvokeAgentScope') && !anyFileContains(tsFiles, 'configureA365Hosting')) { add( 'high', 'node-missing-identity-scope', @@ -597,7 +669,7 @@ function validateNode() { } // Every auth mode exports over the S2S route with an app-only token. - if (hasDistroCall && hasA365Enabled && !distroCallUsesS2SRoute(tsFiles, 'node')) { + if (hasDistroCall && hasA365Enabled && !nodeStates.some(state => state.active && state.hasRoute)) { add( 'high', 'node-obs-delegated-route', @@ -605,7 +677,7 @@ function validateNode() { tsFiles.find(f => fileContains(f, 'useMicrosoftOpenTelemetry')) ); } - if (hasDistroCall && hasA365Enabled && !hasExporterFalse && !distroCallHasResolver(tsFiles, 'node')) { + if (hasDistroCall && hasA365Enabled && !hasExporterFalse && !nodeStates.some(state => state.active && state.hasResolver)) { add( 'high', 'node-obs-token-resolver-missing', @@ -636,7 +708,7 @@ function validateNode() { anyFileContains(tsFiles, 'execute_tool') || anyFileContains(tsFiles, 'output_messages'); - if (hasDistroCall && !hasSemanticSpans) { + if (hasDistroCall && hasA365Enabled && !hasSemanticSpans) { add( 'medium', 'node-no-explicit-a365-semantic-spans', @@ -739,7 +811,10 @@ function validateSetupArtifacts() { ); } const staticConfig = readJson(path.join(cwd, 'a365.config.json')); - if (staticConfig && staticConfig.aiTeammate === false && generated.agentBlueprintId && generated.agenticAppId && !generated.agentRegistrationId) { + const detection = readJson(path.join(cwd, '.a365-workspace-detection.local.json')) || {}; + const agentType = String(detection.agentType || '').toLowerCase(); + const isSystemAgent = agentType === 'system-agent' || (!agentType && staticConfig && staticConfig.aiTeammate === false); + if (isSystemAgent && generated.agentBlueprintId && generated.agenticAppId && !generated.agentRegistrationId) { add( 'medium', 'agent-registration-not-recorded', diff --git a/plugins/agent365/hooks/stop/validate-instrument-observability.js b/plugins/agent365/hooks/stop/validate-instrument-observability.js index 4d836ab..98fb0eb 100644 --- a/plugins/agent365/hooks/stop/validate-instrument-observability.js +++ b/plugins/agent365/hooks/stop/validate-instrument-observability.js @@ -43,8 +43,8 @@ function anyFileMatches(files, regex) { // (Node.js), as a keyword argument of use_microsoft_opentelemetry (Python), or on `.Agent365` in the // UseMicrosoftOpenTelemetry options callback (.NET). A variable is followed one level to its // initializer in the same file. Comments are ignored, so a commented-out option, an unused import, or -// an unrelated object does not count. When the options cannot be located statically (a method group, -// a factory call, or a callback that never touches Agent365), the file as a whole is checked instead. +// an unrelated object does not count. Unresolvable options are treated as unwired; there is no +// whole-file fallback. const DISTRO_CALLS = { node: { call: 'useMicrosoftOpenTelemetry', comments: /\/\*[\s\S]*?\*\/|(^|[\s,{;(])\/\/[^\n]*/gm }, python: { call: 'use_microsoft_opentelemetry', comments: /(^|\s)#[^\n]*/gm }, @@ -186,6 +186,44 @@ function directOptionMatches(field, language, option) { return option.test(direct); } +function directPropertyValue(field, propertyName) { + const direct = field.trim(); + const quoted = `(?:${escapeRegExp(propertyName)}|['"]${escapeRegExp(propertyName)}['"])`; + const match = direct.match(new RegExp(`^${quoted}\\s*(?::\\s*([\\s\\S]*))?$`)); + if (!match) return { present: false, value: '' }; + return { present: true, value: match[1] === undefined ? null : match[1].trim() }; +} + +function objectPropertyValues(content, text, propertyName, seen = new Set()) { + const values = []; + for (const field of splitTopLevel(text)) { + const spread = field.match(/^(?:\.\.\.|\*\*)\s*([A-Za-z_$][\w$]*)$/); + if (spread && !seen.has(spread[1])) { + seen.add(spread[1]); + for (const init of initializerBlocks(content, spread[1])) { + values.push(...objectPropertyValues(content, init, propertyName, seen)); + } + continue; + } + const inlineSpread = field.match(/^\*\*\s*(?:dict\s*)?([({])/); + if (inlineSpread) { + const block = bracketBlock(field, inlineSpread.index + inlineSpread[0].length - 1); + if (block) values.push(...objectPropertyValues(content, block, propertyName, seen)); + continue; + } + const property = directPropertyValue(field, propertyName); + if (property.present) values.push(property.value); + } + return values; +} + +function booleanLiteral(value) { + if (value === null) return 'dynamic'; + if (/^true\b/.test(value)) return 'true'; + if (/^false\b/.test(value)) return 'false'; + return 'dynamic'; +} + function objectPassesOption(content, text, language, option, seen = new Set()) { return splitTopLevel(text).some(field => { const spread = field.match(/^(?:\.\.\.|\*\*)\s*([A-Za-z_$][\w$]*)$/); @@ -252,6 +290,39 @@ function callPassesOption(content, outside, args, language, option) { return objects.some(object => objectPassesOption(outside, object, language, option)); } +function nodeDistroCallStates(files) { + const states = []; + const { call } = DISTRO_CALLS.node; + for (const file of files) { + const content = stripComments(read(file), 'node'); + for (let index = content.indexOf(`${call}(`); index !== -1; index = content.indexOf(`${call}(`, index + 1)) { + const open = index + call.length; + const args = bracketBlock(content, open); + const outside = content.slice(0, open) + ' '.repeat(args.length) + content.slice(open + args.length); + const texts = [args, ...referencedInitializers(outside, args, 'node')]; + const objects = texts.flatMap(text => a365Objects(outside, text)); + if (objects.length === 0) { + states.push({ file, active: true, exporter: 'dynamic', hasRoute: false, hasResolver: false }); + continue; + } + for (const object of objects) { + const enabledValues = objectPropertyValues(outside, object, 'enabled'); + const enabled = enabledValues.length ? booleanLiteral(enabledValues.at(-1)) : 'false'; + const exporterValues = objectPropertyValues(outside, object, 'enableObservabilityExporter'); + const exporter = exporterValues.length ? booleanLiteral(exporterValues.at(-1)) : 'absent'; + states.push({ + file, + active: enabled !== 'false', + exporter, + hasRoute: objectPassesOption(outside, object, 'node', S2S_ROUTE_OPTION.node), + hasResolver: objectPassesOption(outside, object, 'node', TOKEN_RESOLVER_OPTION.node), + }); + } + } + } + return states; +} + function distroCallHasResolver(files, language) { return distroCallMatches(files, language, TOKEN_RESOLVER_OPTION[language]); } diff --git a/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js b/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js index a637a50..34fdf7d 100644 --- a/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js +++ b/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js @@ -112,8 +112,8 @@ function callBlocks(content, name) { // (Node.js), as a keyword argument of use_microsoft_opentelemetry (Python), or on `.Agent365` in the // UseMicrosoftOpenTelemetry options callback (.NET). A variable is followed one level to its // initializer in the same file. Comments are ignored, so a commented-out option, an unused import, or -// an unrelated object does not count. When the options cannot be located statically (a method group, -// a factory call, or a callback that never touches Agent365), the file as a whole is checked instead. +// an unrelated object does not count. Unresolvable options are treated as unwired; there is no +// whole-file fallback. const DISTRO_CALLS = { node: { call: 'useMicrosoftOpenTelemetry', comments: /\/\*[\s\S]*?\*\/|(^|[\s,{;(])\/\/[^\n]*/gm }, python: { call: 'use_microsoft_opentelemetry', comments: /(^|\s)#[^\n]*/gm }, @@ -255,6 +255,44 @@ function directOptionMatches(field, language, option) { return option.test(direct); } +function directPropertyValue(field, propertyName) { + const direct = field.trim(); + const quoted = `(?:${escapeRegExp(propertyName)}|['"]${escapeRegExp(propertyName)}['"])`; + const match = direct.match(new RegExp(`^${quoted}\\s*(?::\\s*([\\s\\S]*))?$`)); + if (!match) return { present: false, value: '' }; + return { present: true, value: match[1] === undefined ? null : match[1].trim() }; +} + +function objectPropertyValues(content, text, propertyName, seen = new Set()) { + const values = []; + for (const field of splitTopLevel(text)) { + const spread = field.match(/^(?:\.\.\.|\*\*)\s*([A-Za-z_$][\w$]*)$/); + if (spread && !seen.has(spread[1])) { + seen.add(spread[1]); + for (const init of initializerBlocks(content, spread[1])) { + values.push(...objectPropertyValues(content, init, propertyName, seen)); + } + continue; + } + const inlineSpread = field.match(/^\*\*\s*(?:dict\s*)?([({])/); + if (inlineSpread) { + const block = bracketBlock(field, inlineSpread.index + inlineSpread[0].length - 1); + if (block) values.push(...objectPropertyValues(content, block, propertyName, seen)); + continue; + } + const property = directPropertyValue(field, propertyName); + if (property.present) values.push(property.value); + } + return values; +} + +function booleanLiteral(value) { + if (value === null) return 'dynamic'; + if (/^true\b/.test(value)) return 'true'; + if (/^false\b/.test(value)) return 'false'; + return 'dynamic'; +} + function objectPassesOption(content, text, language, option, seen = new Set()) { return splitTopLevel(text).some(field => { const spread = field.match(/^(?:\.\.\.|\*\*)\s*([A-Za-z_$][\w$]*)$/); @@ -321,6 +359,39 @@ function callPassesOption(content, outside, args, language, option) { return objects.some(object => objectPassesOption(outside, object, language, option)); } +function nodeDistroCallStates(files) { + const states = []; + const { call } = DISTRO_CALLS.node; + for (const file of files) { + const content = stripComments(read(file), 'node'); + for (let index = content.indexOf(`${call}(`); index !== -1; index = content.indexOf(`${call}(`, index + 1)) { + const open = index + call.length; + const args = bracketBlock(content, open); + const outside = content.slice(0, open) + ' '.repeat(args.length) + content.slice(open + args.length); + const texts = [args, ...referencedInitializers(outside, args, 'node')]; + const objects = texts.flatMap(text => a365Objects(outside, text)); + if (objects.length === 0) { + states.push({ file, active: true, exporter: 'dynamic', hasRoute: false, hasResolver: false }); + continue; + } + for (const object of objects) { + const enabledValues = objectPropertyValues(outside, object, 'enabled'); + const enabled = enabledValues.length ? booleanLiteral(enabledValues.at(-1)) : 'false'; + const exporterValues = objectPropertyValues(outside, object, 'enableObservabilityExporter'); + const exporter = exporterValues.length ? booleanLiteral(exporterValues.at(-1)) : 'absent'; + states.push({ + file, + active: enabled !== 'false', + exporter, + hasRoute: objectPassesOption(outside, object, 'node', S2S_ROUTE_OPTION.node), + hasResolver: objectPassesOption(outside, object, 'node', TOKEN_RESOLVER_OPTION.node), + }); + } + } + } + return states; +} + function distroCallHasResolver(files, language) { return distroCallMatches(files, language, TOKEN_RESOLVER_OPTION[language]); } @@ -475,28 +546,29 @@ function validatePython() { function validateNode() { const hasPackage = pkg.some(f => read(f).includes('@microsoft/opentelemetry')); const hasDistro = anyContains(ts, 'useMicrosoftOpenTelemetry'); - const hasEnabled = anyMatches(ts, /\benabled\s*:\s*true\b/); - const exporterTrue = anyMatches(ts, /\benableObservabilityExporter\s*:\s*true\b/); - const exporterFalse = anyMatches(ts, /\benableObservabilityExporter\s*:\s*false\b/); + const nodeStates = nodeDistroCallStates(ts); + const hasEnabled = nodeStates.some(state => state.active); + const exporterTrue = nodeStates.some(state => state.active && (state.exporter === 'true' || state.exporter === 'dynamic')); + const exporterFalse = nodeStates.some(state => state.active && state.exporter === 'false'); const exporterEnv = envTrue('ENABLE_A365_OBSERVABILITY_EXPORTER'); if (hasPackage && !hasDistro) { add('high', 'node-missing-distro-init', '@microsoft/opentelemetry is installed but no useMicrosoftOpenTelemetry() call was found.', pkg.find(f => read(f).includes('@microsoft/opentelemetry'))); } if (hasDistro && hasEnabled && exporterFalse) { - add('critical', 'node-exporter-explicitly-disabled', 'Node code sets enableObservabilityExporter:false; A365 backend export is disabled.', ts.find(f => /enableObservabilityExporter\s*:\s*false\b/.test(read(f)))); + add('critical', 'node-exporter-explicitly-disabled', 'Node code sets enableObservabilityExporter:false; A365 backend export is disabled.', (nodeStates.find(state => state.active && state.exporter === 'false') || {}).file); } else if (hasDistro && hasEnabled && !exporterTrue && !exporterEnv) { add('critical', 'node-exporter-not-enabled', 'Node code enables A365 but does not enable the exporter and no truthy exporter env was found.', ts.find(f => read(f).includes('useMicrosoftOpenTelemetry'))); } const hasIdentity = anyContains(ts, 'BaggageBuilder') || anyContains(ts, 'InvokeAgentScope') || anyContains(ts, 'configureA365Hosting'); - if (hasDistro && !hasIdentity) { + if (hasDistro && hasEnabled && !hasIdentity) { add('high', 'node-missing-identity-scope', 'No BaggageBuilder/InvokeAgentScope/configureA365Hosting usage found; spans may lack agent identity.'); } // Every auth mode exports over the S2S route with an app-only token. - if (hasDistro && hasEnabled && !distroCallUsesS2SRoute(ts, 'node')) { + if (hasDistro && hasEnabled && !nodeStates.some(state => state.active && state.hasRoute)) { add('high', 'node-obs-delegated-route', 'Node code does not set useS2SEndpoint: true, so A365 export uses the legacy delegated route. Every auth mode must export over the S2S route with an app-only tokenResolver.', ts.find(f => read(f).includes('useMicrosoftOpenTelemetry'))); } - if (hasDistro && hasEnabled && !exporterFalse && !distroCallHasResolver(ts, 'node')) { + if (hasDistro && hasEnabled && !exporterFalse && !nodeStates.some(state => state.active && state.hasResolver)) { add('high', 'node-obs-token-resolver-missing', 'useMicrosoftOpenTelemetry() has no a365 tokenResolver, so the S2S route gets no app-only token. Pass an app-only tokenResolver (instrument-observability app-token-resolver.ts for obo / agentic-user, or the S2S token service).', ts.find(f => read(f).includes('useMicrosoftOpenTelemetry'))); } for (const file of ts) { @@ -514,7 +586,7 @@ function validateNode() { anyContains(ts, 'invoke_agent') || anyContains(ts, 'execute_tool') || anyContains(ts, 'output_messages'); - if (hasDistro && !hasSemantic) { + if (hasDistro && hasEnabled && !hasSemantic) { add('medium', 'node-no-explicit-a365-semantic-spans', 'No explicit supported A365 semantic spans found; baggage alone is not enough for MAC Activity.'); } } @@ -562,7 +634,10 @@ function validateSetupArtifacts() { add('critical', 'blueprint-id-used-as-agent-id', 'a365.generated.config.json has identical blueprint and agentic app IDs. Verify runtime gen_ai.agent.id uses the agent instance/source agent ID, not the blueprint ID.', path.join(cwd, 'a365.generated.config.json')); } const staticConfig = readJsonSafe(path.join(cwd, 'a365.config.json')); - if (staticConfig && staticConfig.aiTeammate === false && generated.agentBlueprintId && generated.agenticAppId && !generated.agentRegistrationId) { + const detection = readJsonSafe(path.join(cwd, '.a365-workspace-detection.local.json')) || {}; + const agentType = String(detection.agentType || '').toLowerCase(); + const isSystemAgent = agentType === 'system-agent' || (!agentType && staticConfig && staticConfig.aiTeammate === false); + if (isSystemAgent && generated.agentBlueprintId && generated.agenticAppId && !generated.agentRegistrationId) { add('medium', 'agent-registration-not-recorded', 'a365.generated.config.json has an agent identity but no agentRegistrationId. The S2S route authorizes registered agent instances without an OtelWrite grant; an unregistered instance gets 403 insufficient_scope. Run a365 setup all --agent-registration-only (idempotent).', path.join(cwd, 'a365.generated.config.json')); } } diff --git a/tests/validate-a365-code-validator.test.js b/tests/validate-a365-code-validator.test.js index 5be7a21..880b3df 100644 --- a/tests/validate-a365-code-validator.test.js +++ b/tests/validate-a365-code-validator.test.js @@ -363,6 +363,68 @@ await AgenticTokenCacheInstance.refreshObservabilityToken( } }); + test('Node dynamic enabled is active; dynamic exporter with route and resolver is accepted by both scanners', () => { + const missingRoute = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': [ + 'const enabled = process.env.A365_ENABLED === "true";', + 'useMicrosoftOpenTelemetry({ a365: { enabled, enableObservabilityExporter: true, tokenResolver: appTokenResolver } });', + ].join('\n'), + }); + const docShape = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': [ + 'const A365_ENABLED = process.env.ENABLE_A365_OBSERVABILITY_EXPORTER === "true";', + 'useMicrosoftOpenTelemetry({', + ' a365: {', + ' enabled: A365_ENABLED,', + ' enableObservabilityExporter: A365_ENABLED,', + ' useS2SEndpoint: true,', + ' tokenResolver: appTokenResolver,', + ' },', + '});', + 'InvokeAgentScope.start(request, details, agentDetails, callerDetails);', + ].join('\n'), + }); + const ids = result => findingIds(result).filter(id => /node-(obs-delegated-route|obs-token-resolver-missing|exporter)/.test(id)).sort(); + try { + for (const scanner of [VALIDATOR, STANDALONE]) { + assert.ok(ids(runValidator(scanner, missingRoute)).includes('node-obs-delegated-route'), scanner); + assert.deepEqual(ids(runValidator(scanner, docShape)), [], scanner); + } + } finally { + cleanup(missingRoute); + cleanup(docShape); + } + }); + + test('Node literal false or absent enabled stays inactive; shorthand enabled variable is active', () => { + const disabled = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': 'useMicrosoftOpenTelemetry({ a365: { enabled: false, enableObservabilityExporter: true } });', + }); + const absent = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': 'useMicrosoftOpenTelemetry({ a365: { enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver: appTokenResolver } });', + }); + const shorthandActive = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': 'const enabled = true;\nuseMicrosoftOpenTelemetry({ a365: { enabled, enableObservabilityExporter: true, tokenResolver: appTokenResolver } });', + }); + const nodeA365Ids = result => findingIds(result).filter(id => /^node-(?:obs-|exporter|missing-identity|no-explicit)/.test(id)).sort(); + try { + for (const scanner of [VALIDATOR, STANDALONE]) { + assert.deepEqual(nodeA365Ids(runValidator(scanner, disabled)), [], scanner); + assert.deepEqual(nodeA365Ids(runValidator(scanner, absent)), [], scanner); + assert.ok(nodeA365Ids(runValidator(scanner, shorthandActive)).includes('node-obs-delegated-route'), scanner); + } + } finally { + cleanup(disabled); + cleanup(absent); + cleanup(shorthandActive); + } + }); + test('.NET distro without UseS2SEndpoint and with AgenticTokenStruct registration reports both findings', () => { const dir = createFixture({ 'Agent.csproj': '', @@ -439,19 +501,26 @@ token = await auth.exchange_token(context, scopes=["ea9ffc3e-8a23-4a7d-836d-234d } }); - test('Blueprint agent without a recorded registration reports agent-registration-not-recorded', () => { + test('Blueprint agent without a recorded registration uses detection cache before aiTeammate fallback', () => { const agentId = '22222222-2222-2222-2222-222222222222'; const blueprintId = '33333333-3333-3333-3333-333333333333'; - const unregistered = createFixture({ - 'a365.config.json': JSON.stringify({ aiTeammate: false }), + const systemAgent = createFixture({ + '.a365-workspace-detection.local.json': JSON.stringify({ agentType: 'system-agent' }), + 'a365.config.json': JSON.stringify({}), 'a365.generated.config.json': JSON.stringify({ agentBlueprintId: blueprintId, agenticAppId: agentId }), }); const registered = createFixture({ + '.a365-workspace-detection.local.json': JSON.stringify({ agentType: 'system-agent' }), 'a365.config.json': JSON.stringify({ aiTeammate: false }), 'a365.generated.config.json': JSON.stringify({ agentBlueprintId: blueprintId, agenticAppId: agentId, agentRegistrationId: 'reg-1' }), }); const aiTeammate = createFixture({ - 'a365.config.json': JSON.stringify({ aiTeammate: true }), + '.a365-workspace-detection.local.json': JSON.stringify({ agentType: 'ai-teammate' }), + 'a365.config.json': JSON.stringify({ aiTeammate: false }), + 'a365.generated.config.json': JSON.stringify({ agentBlueprintId: blueprintId, agenticAppId: agentId }), + }); + const fallback = createFixture({ + 'a365.config.json': JSON.stringify({ aiTeammate: false }), 'a365.generated.config.json': JSON.stringify({ agentBlueprintId: blueprintId, agenticAppId: agentId }), }); const noBlueprint = createFixture({ @@ -459,20 +528,22 @@ token = await auth.exchange_token(context, scopes=["ea9ffc3e-8a23-4a7d-836d-234d 'a365.generated.config.json': JSON.stringify({ agenticAppId: agentId }), }); try { - const finding = runValidator(VALIDATOR, unregistered).findings.find(f => f.id === 'agent-registration-not-recorded'); + const finding = runValidator(VALIDATOR, systemAgent).findings.find(f => f.id === 'agent-registration-not-recorded'); assert.ok(finding, 'expected agent-registration-not-recorded'); assert.equal(finding.severity, 'medium'); assert.ok(!findingIds(runValidator(VALIDATOR, registered)).includes('agent-registration-not-recorded')); assert.ok(!findingIds(runValidator(VALIDATOR, aiTeammate)).includes('agent-registration-not-recorded')); for (const scanner of [VALIDATOR, STANDALONE]) { - assert.ok(findingIds(runValidator(scanner, unregistered)).includes('agent-registration-not-recorded'), scanner); + assert.ok(findingIds(runValidator(scanner, systemAgent)).includes('agent-registration-not-recorded'), scanner); + assert.ok(findingIds(runValidator(scanner, fallback)).includes('agent-registration-not-recorded'), scanner); assert.ok(!findingIds(runValidator(scanner, noBlueprint)).includes('agent-registration-not-recorded'), `${scanner}: without a blueprint ID there is no blueprint agent instance to register`); } } finally { - cleanup(unregistered); + cleanup(systemAgent); cleanup(registered); cleanup(aiTeammate); + cleanup(fallback); cleanup(noBlueprint); } }); From 7c83702d94dc02bdcef98b0fb7dcd092a9b54dbb Mon Sep 17 00:00:00 2001 From: Krishnadheeraj <12496535+DheerajPannala@users.noreply.github.com> Date: Mon, 28 Sep 2026 20:11:27 +0100 Subject: [PATCH 10/12] Refine Node A365 activation checks - keep visible non-A365 useMicrosoftOpenTelemetry calls inactive - require exact boolean literals for dynamic activation handling - use workspace detection for registration warning parity Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5cbf5f6b-cc40-4b7e-a591-65848db73a12 --- .../stop/validate-a365-code-validator.js | 52 ++++++++++++++----- .../stop/validate-instrument-observability.js | 52 ++++++++++++++----- .../references/a365-code-validator.js | 52 ++++++++++++++----- tests/validate-a365-code-validator.test.js | 37 ++++++++++++- 4 files changed, 155 insertions(+), 38 deletions(-) diff --git a/plugins/agent365/hooks/stop/validate-a365-code-validator.js b/plugins/agent365/hooks/stop/validate-a365-code-validator.js index 10a5d18..c721ee6 100644 --- a/plugins/agent365/hooks/stop/validate-a365-code-validator.js +++ b/plugins/agent365/hooks/stop/validate-a365-code-validator.js @@ -290,8 +290,9 @@ function objectPropertyValues(content, text, propertyName, seen = new Set()) { function booleanLiteral(value) { if (value === null) return 'dynamic'; - if (/^true\b/.test(value)) return 'true'; - if (/^false\b/.test(value)) return 'false'; + const trimmed = value.trim(); + if (/^true$/.test(trimmed)) return 'true'; + if (/^false$/.test(trimmed)) return 'false'; return 'dynamic'; } @@ -312,9 +313,11 @@ function objectPassesOption(content, text, language, option, seen = new Set()) { } // Node.js: the `a365` options objects in `text`, inline (`a365: {...}`), by variable (`a365: options`), -// or shorthand (`{ a365 }`). Unresolvable values are treated as unwired. +// or shorthand (`{ a365 }`). `readable` means the options object was visible; unreadable values +// are conservatively active, while readable options with no `a365` key are inactive. function a365Objects(content, text) { const objects = []; + let unreadable = false; const property = /(?:^|[{,\s])a365\s*:\s*/g; for (let m = property.exec(text); m; m = property.exec(text)) { const at = m.index + m[0].length; @@ -323,11 +326,34 @@ function a365Objects(content, text) { continue; } const value = text.slice(at).match(/^[A-Za-z_$][\w$.]*(\s*\()?/); - if (!value || value[1]) continue; - objects.push(...initializerBlocks(content, value[0])); + if (!value || value[1]) { + unreadable = true; + continue; + } + const initializers = initializerBlocks(content, value[0]); + if (initializers.length === 0) unreadable = true; + objects.push(...initializers); + } + if (/[{,]\s*a365\s*(?=[,}])/.test(text)) { + const initializers = initializerBlocks(content, 'a365'); + if (initializers.length === 0) unreadable = true; + objects.push(...initializers); + } + return { objects, readable: !unreadable }; +} + +function nodeOptionsTexts(outside, args) { + const values = splitTopLevel(args); + if (values.length === 0) return { texts: [], unreadable: false }; + const first = values[0].trim(); + if (!first) return { texts: [], unreadable: false }; + if (first[0] === '{') return { texts: [first], unreadable: false }; + const identifier = first.match(/^[A-Za-z_$][\w$]*$/); + if (identifier) { + const initializers = initializerBlocks(outside, identifier[0]); + return initializers.length ? { texts: initializers, unreadable: false } : { texts: [], unreadable: true }; } - if (/[{,]\s*a365\s*(?=[,}])/.test(text)) objects.push(...initializerBlocks(content, 'a365')); - return objects; + return { texts: [], unreadable: true }; } // True when a distro call in `files` passes an option matching `option`. @@ -355,8 +381,8 @@ function callPassesOption(content, outside, args, language, option) { const texts = [language === 'python' ? pythonKeywordArguments(args) : args, ...referencedInitializers(outside, args, language)]; if (language === 'python') return texts.some(text => objectPassesOption(outside, text, language, option)); const objects = []; - for (const text of texts) { - objects.push(...a365Objects(outside, text)); + for (const text of nodeOptionsTexts(outside, args).texts) { + objects.push(...a365Objects(outside, text).objects); } return objects.some(object => objectPassesOption(outside, object, language, option)); } @@ -370,10 +396,12 @@ function nodeDistroCallStates(files) { const open = index + call.length; const args = bracketBlock(content, open); const outside = content.slice(0, open) + ' '.repeat(args.length) + content.slice(open + args.length); - const texts = [args, ...referencedInitializers(outside, args, 'node')]; - const objects = texts.flatMap(text => a365Objects(outside, text)); + const options = nodeOptionsTexts(outside, args); + const texts = options.texts; + const resolved = texts.map(text => a365Objects(outside, text)); + const objects = resolved.flatMap(result => result.objects); if (objects.length === 0) { - states.push({ file, active: true, exporter: 'dynamic', hasRoute: false, hasResolver: false }); + states.push({ file, active: options.unreadable || resolved.some(result => !result.readable), exporter: 'dynamic', hasRoute: false, hasResolver: false }); continue; } for (const object of objects) { diff --git a/plugins/agent365/hooks/stop/validate-instrument-observability.js b/plugins/agent365/hooks/stop/validate-instrument-observability.js index 98fb0eb..2d5801a 100644 --- a/plugins/agent365/hooks/stop/validate-instrument-observability.js +++ b/plugins/agent365/hooks/stop/validate-instrument-observability.js @@ -219,8 +219,9 @@ function objectPropertyValues(content, text, propertyName, seen = new Set()) { function booleanLiteral(value) { if (value === null) return 'dynamic'; - if (/^true\b/.test(value)) return 'true'; - if (/^false\b/.test(value)) return 'false'; + const trimmed = value.trim(); + if (/^true$/.test(trimmed)) return 'true'; + if (/^false$/.test(trimmed)) return 'false'; return 'dynamic'; } @@ -241,9 +242,11 @@ function objectPassesOption(content, text, language, option, seen = new Set()) { } // Node.js: the `a365` options objects in `text`, inline (`a365: {...}`), by variable (`a365: options`), -// or shorthand (`{ a365 }`). Unresolvable values are treated as unwired. +// or shorthand (`{ a365 }`). `readable` means the options object was visible; unreadable values +// are conservatively active, while readable options with no `a365` key are inactive. function a365Objects(content, text) { const objects = []; + let unreadable = false; const property = /(?:^|[{,\s])a365\s*:\s*/g; for (let m = property.exec(text); m; m = property.exec(text)) { const at = m.index + m[0].length; @@ -252,11 +255,34 @@ function a365Objects(content, text) { continue; } const value = text.slice(at).match(/^[A-Za-z_$][\w$.]*(\s*\()?/); - if (!value || value[1]) continue; - objects.push(...initializerBlocks(content, value[0])); + if (!value || value[1]) { + unreadable = true; + continue; + } + const initializers = initializerBlocks(content, value[0]); + if (initializers.length === 0) unreadable = true; + objects.push(...initializers); + } + if (/[{,]\s*a365\s*(?=[,}])/.test(text)) { + const initializers = initializerBlocks(content, 'a365'); + if (initializers.length === 0) unreadable = true; + objects.push(...initializers); + } + return { objects, readable: !unreadable }; +} + +function nodeOptionsTexts(outside, args) { + const values = splitTopLevel(args); + if (values.length === 0) return { texts: [], unreadable: false }; + const first = values[0].trim(); + if (!first) return { texts: [], unreadable: false }; + if (first[0] === '{') return { texts: [first], unreadable: false }; + const identifier = first.match(/^[A-Za-z_$][\w$]*$/); + if (identifier) { + const initializers = initializerBlocks(outside, identifier[0]); + return initializers.length ? { texts: initializers, unreadable: false } : { texts: [], unreadable: true }; } - if (/[{,]\s*a365\s*(?=[,}])/.test(text)) objects.push(...initializerBlocks(content, 'a365')); - return objects; + return { texts: [], unreadable: true }; } // True when a distro call in `files` passes an option matching `option`. @@ -284,8 +310,8 @@ function callPassesOption(content, outside, args, language, option) { const texts = [language === 'python' ? pythonKeywordArguments(args) : args, ...referencedInitializers(outside, args, language)]; if (language === 'python') return texts.some(text => objectPassesOption(outside, text, language, option)); const objects = []; - for (const text of texts) { - objects.push(...a365Objects(outside, text)); + for (const text of nodeOptionsTexts(outside, args).texts) { + objects.push(...a365Objects(outside, text).objects); } return objects.some(object => objectPassesOption(outside, object, language, option)); } @@ -299,10 +325,12 @@ function nodeDistroCallStates(files) { const open = index + call.length; const args = bracketBlock(content, open); const outside = content.slice(0, open) + ' '.repeat(args.length) + content.slice(open + args.length); - const texts = [args, ...referencedInitializers(outside, args, 'node')]; - const objects = texts.flatMap(text => a365Objects(outside, text)); + const options = nodeOptionsTexts(outside, args); + const texts = options.texts; + const resolved = texts.map(text => a365Objects(outside, text)); + const objects = resolved.flatMap(result => result.objects); if (objects.length === 0) { - states.push({ file, active: true, exporter: 'dynamic', hasRoute: false, hasResolver: false }); + states.push({ file, active: options.unreadable || resolved.some(result => !result.readable), exporter: 'dynamic', hasRoute: false, hasResolver: false }); continue; } for (const object of objects) { diff --git a/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js b/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js index 34fdf7d..4268de7 100644 --- a/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js +++ b/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js @@ -288,8 +288,9 @@ function objectPropertyValues(content, text, propertyName, seen = new Set()) { function booleanLiteral(value) { if (value === null) return 'dynamic'; - if (/^true\b/.test(value)) return 'true'; - if (/^false\b/.test(value)) return 'false'; + const trimmed = value.trim(); + if (/^true$/.test(trimmed)) return 'true'; + if (/^false$/.test(trimmed)) return 'false'; return 'dynamic'; } @@ -310,9 +311,11 @@ function objectPassesOption(content, text, language, option, seen = new Set()) { } // Node.js: the `a365` options objects in `text`, inline (`a365: {...}`), by variable (`a365: options`), -// or shorthand (`{ a365 }`). Unresolvable values are treated as unwired. +// or shorthand (`{ a365 }`). `readable` means the options object was visible; unreadable values +// are conservatively active, while readable options with no `a365` key are inactive. function a365Objects(content, text) { const objects = []; + let unreadable = false; const property = /(?:^|[{,\s])a365\s*:\s*/g; for (let m = property.exec(text); m; m = property.exec(text)) { const at = m.index + m[0].length; @@ -321,11 +324,34 @@ function a365Objects(content, text) { continue; } const value = text.slice(at).match(/^[A-Za-z_$][\w$.]*(\s*\()?/); - if (!value || value[1]) continue; - objects.push(...initializerBlocks(content, value[0])); + if (!value || value[1]) { + unreadable = true; + continue; + } + const initializers = initializerBlocks(content, value[0]); + if (initializers.length === 0) unreadable = true; + objects.push(...initializers); + } + if (/[{,]\s*a365\s*(?=[,}])/.test(text)) { + const initializers = initializerBlocks(content, 'a365'); + if (initializers.length === 0) unreadable = true; + objects.push(...initializers); + } + return { objects, readable: !unreadable }; +} + +function nodeOptionsTexts(outside, args) { + const values = splitTopLevel(args); + if (values.length === 0) return { texts: [], unreadable: false }; + const first = values[0].trim(); + if (!first) return { texts: [], unreadable: false }; + if (first[0] === '{') return { texts: [first], unreadable: false }; + const identifier = first.match(/^[A-Za-z_$][\w$]*$/); + if (identifier) { + const initializers = initializerBlocks(outside, identifier[0]); + return initializers.length ? { texts: initializers, unreadable: false } : { texts: [], unreadable: true }; } - if (/[{,]\s*a365\s*(?=[,}])/.test(text)) objects.push(...initializerBlocks(content, 'a365')); - return objects; + return { texts: [], unreadable: true }; } // True when a distro call in `files` passes an option matching `option`. @@ -353,8 +379,8 @@ function callPassesOption(content, outside, args, language, option) { const texts = [language === 'python' ? pythonKeywordArguments(args) : args, ...referencedInitializers(outside, args, language)]; if (language === 'python') return texts.some(text => objectPassesOption(outside, text, language, option)); const objects = []; - for (const text of texts) { - objects.push(...a365Objects(outside, text)); + for (const text of nodeOptionsTexts(outside, args).texts) { + objects.push(...a365Objects(outside, text).objects); } return objects.some(object => objectPassesOption(outside, object, language, option)); } @@ -368,10 +394,12 @@ function nodeDistroCallStates(files) { const open = index + call.length; const args = bracketBlock(content, open); const outside = content.slice(0, open) + ' '.repeat(args.length) + content.slice(open + args.length); - const texts = [args, ...referencedInitializers(outside, args, 'node')]; - const objects = texts.flatMap(text => a365Objects(outside, text)); + const options = nodeOptionsTexts(outside, args); + const texts = options.texts; + const resolved = texts.map(text => a365Objects(outside, text)); + const objects = resolved.flatMap(result => result.objects); if (objects.length === 0) { - states.push({ file, active: true, exporter: 'dynamic', hasRoute: false, hasResolver: false }); + states.push({ file, active: options.unreadable || resolved.some(result => !result.readable), exporter: 'dynamic', hasRoute: false, hasResolver: false }); continue; } for (const object of objects) { diff --git a/tests/validate-a365-code-validator.test.js b/tests/validate-a365-code-validator.test.js index 880b3df..7464e19 100644 --- a/tests/validate-a365-code-validator.test.js +++ b/tests/validate-a365-code-validator.test.js @@ -411,17 +411,50 @@ await AgenticTokenCacheInstance.refreshObservabilityToken( 'package.json': NODE_PKG, 'index.ts': 'const enabled = true;\nuseMicrosoftOpenTelemetry({ a365: { enabled, enableObservabilityExporter: true, tokenResolver: appTokenResolver } });', }); + const expressionActive = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': "useMicrosoftOpenTelemetry({ a365: { enabled: false || process.env.X === 'true', enableObservabilityExporter: true, tokenResolver: appTokenResolver } });", + }); const nodeA365Ids = result => findingIds(result).filter(id => /^node-(?:obs-|exporter|missing-identity|no-explicit)/.test(id)).sort(); try { for (const scanner of [VALIDATOR, STANDALONE]) { assert.deepEqual(nodeA365Ids(runValidator(scanner, disabled)), [], scanner); assert.deepEqual(nodeA365Ids(runValidator(scanner, absent)), [], scanner); assert.ok(nodeA365Ids(runValidator(scanner, shorthandActive)).includes('node-obs-delegated-route'), scanner); + assert.ok(nodeA365Ids(runValidator(scanner, expressionActive)).includes('node-obs-delegated-route'), scanner); } } finally { cleanup(disabled); cleanup(absent); cleanup(shorthandActive); + cleanup(expressionActive); + } + }); + + test('Node calls with no visible A365 options stay inactive, but unreadable options are active', () => { + const emptyCall = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': 'useMicrosoftOpenTelemetry();', + }); + const azureMonitorOnly = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': 'useMicrosoftOpenTelemetry({ azureMonitor: { connectionString } });', + }); + const factory = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': 'useMicrosoftOpenTelemetry(buildOtelOptions());\nfunction buildOtelOptions() { return { a365: { enabled: true, enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver: appTokenResolver } }; }', + }); + const nodeA365Ids = result => findingIds(result).filter(id => /^node-(?:obs-|exporter|missing-identity|no-explicit)/.test(id)).sort(); + try { + for (const scanner of [VALIDATOR, STANDALONE]) { + assert.deepEqual(nodeA365Ids(runValidator(scanner, emptyCall)), [], scanner); + assert.deepEqual(nodeA365Ids(runValidator(scanner, azureMonitorOnly)), [], scanner); + assert.ok(nodeA365Ids(runValidator(scanner, factory)).includes('node-obs-delegated-route'), scanner); + } + } finally { + cleanup(emptyCall); + cleanup(azureMonitorOnly); + cleanup(factory); } }); @@ -531,11 +564,11 @@ token = await auth.exchange_token(context, scopes=["ea9ffc3e-8a23-4a7d-836d-234d const finding = runValidator(VALIDATOR, systemAgent).findings.find(f => f.id === 'agent-registration-not-recorded'); assert.ok(finding, 'expected agent-registration-not-recorded'); assert.equal(finding.severity, 'medium'); - assert.ok(!findingIds(runValidator(VALIDATOR, registered)).includes('agent-registration-not-recorded')); - assert.ok(!findingIds(runValidator(VALIDATOR, aiTeammate)).includes('agent-registration-not-recorded')); for (const scanner of [VALIDATOR, STANDALONE]) { assert.ok(findingIds(runValidator(scanner, systemAgent)).includes('agent-registration-not-recorded'), scanner); assert.ok(findingIds(runValidator(scanner, fallback)).includes('agent-registration-not-recorded'), scanner); + assert.ok(!findingIds(runValidator(scanner, registered)).includes('agent-registration-not-recorded'), scanner); + assert.ok(!findingIds(runValidator(scanner, aiTeammate)).includes('agent-registration-not-recorded'), scanner); assert.ok(!findingIds(runValidator(scanner, noBlueprint)).includes('agent-registration-not-recorded'), `${scanner}: without a blueprint ID there is no blueprint agent instance to register`); } From f6ced83169d4b25bd31ff514e56809f438309611 Mon Sep 17 00:00:00 2001 From: Krishnadheeraj <12496535+DheerajPannala@users.noreply.github.com> Date: Mon, 28 Sep 2026 20:15:36 +0100 Subject: [PATCH 11/12] Follow options spreads when reading distro calls nodeOptionsTexts read only the literal options object, so a route, resolver or a365 block inside a same-file spread ({ ...baseOptions }) was missed, and an unresolvable spread made the call look inactive. It now follows top-level spreads to same-file object literals; any other spread keeps the call active. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5cbf5f6b-cc40-4b7e-a591-65848db73a12 --- .../stop/validate-a365-code-validator.js | 35 ++++++++++++++--- .../stop/validate-instrument-observability.js | 35 ++++++++++++++--- .../references/a365-code-validator.js | 35 ++++++++++++++--- tests/validate-a365-code-validator.test.js | 38 +++++++++++++++++++ tests/validate-observability.test.js | 13 +++++++ 5 files changed, 141 insertions(+), 15 deletions(-) diff --git a/plugins/agent365/hooks/stop/validate-a365-code-validator.js b/plugins/agent365/hooks/stop/validate-a365-code-validator.js index c721ee6..09cf929 100644 --- a/plugins/agent365/hooks/stop/validate-a365-code-validator.js +++ b/plugins/agent365/hooks/stop/validate-a365-code-validator.js @@ -347,15 +347,40 @@ function nodeOptionsTexts(outside, args) { if (values.length === 0) return { texts: [], unreadable: false }; const first = values[0].trim(); if (!first) return { texts: [], unreadable: false }; - if (first[0] === '{') return { texts: [first], unreadable: false }; + if (first[0] === '{') return nodeObjectTexts(outside, first, new Set()); const identifier = first.match(/^[A-Za-z_$][\w$]*$/); - if (identifier) { - const initializers = initializerBlocks(outside, identifier[0]); - return initializers.length ? { texts: initializers, unreadable: false } : { texts: [], unreadable: true }; - } + if (identifier) return nodeIdentifierTexts(outside, identifier[0], new Set()); return { texts: [], unreadable: true }; } +// An options object plus the same-file objects its top-level spreads resolve to. Any other spread +// (a call, a member access, an import) makes the options unreadable. +function nodeObjectTexts(outside, text, seen) { + const result = { texts: [text], unreadable: false }; + for (const field of splitTopLevel(text)) { + if (!field.startsWith('...')) continue; + const name = field.slice(3).trim().match(/^[A-Za-z_$][\w$]*$/); + const spread = name ? nodeIdentifierTexts(outside, name[0], seen) : { texts: [], unreadable: true }; + result.texts.push(...spread.texts); + result.unreadable = result.unreadable || spread.unreadable; + } + return result; +} + +function nodeIdentifierTexts(outside, name, seen) { + if (seen.has(name)) return { texts: [], unreadable: false }; + seen.add(name); + const objects = initializerBlocks(outside, name).filter(block => block.startsWith('{')); + if (objects.length === 0) return { texts: [], unreadable: true }; + const result = { texts: [], unreadable: false }; + for (const object of objects) { + const nested = nodeObjectTexts(outside, object, seen); + result.texts.push(...nested.texts); + result.unreadable = result.unreadable || nested.unreadable; + } + return result; +} + // True when a distro call in `files` passes an option matching `option`. function distroCallMatches(files, language, option) { const { call } = DISTRO_CALLS[language]; diff --git a/plugins/agent365/hooks/stop/validate-instrument-observability.js b/plugins/agent365/hooks/stop/validate-instrument-observability.js index 2d5801a..bc2c2b3 100644 --- a/plugins/agent365/hooks/stop/validate-instrument-observability.js +++ b/plugins/agent365/hooks/stop/validate-instrument-observability.js @@ -276,15 +276,40 @@ function nodeOptionsTexts(outside, args) { if (values.length === 0) return { texts: [], unreadable: false }; const first = values[0].trim(); if (!first) return { texts: [], unreadable: false }; - if (first[0] === '{') return { texts: [first], unreadable: false }; + if (first[0] === '{') return nodeObjectTexts(outside, first, new Set()); const identifier = first.match(/^[A-Za-z_$][\w$]*$/); - if (identifier) { - const initializers = initializerBlocks(outside, identifier[0]); - return initializers.length ? { texts: initializers, unreadable: false } : { texts: [], unreadable: true }; - } + if (identifier) return nodeIdentifierTexts(outside, identifier[0], new Set()); return { texts: [], unreadable: true }; } +// An options object plus the same-file objects its top-level spreads resolve to. Any other spread +// (a call, a member access, an import) makes the options unreadable. +function nodeObjectTexts(outside, text, seen) { + const result = { texts: [text], unreadable: false }; + for (const field of splitTopLevel(text)) { + if (!field.startsWith('...')) continue; + const name = field.slice(3).trim().match(/^[A-Za-z_$][\w$]*$/); + const spread = name ? nodeIdentifierTexts(outside, name[0], seen) : { texts: [], unreadable: true }; + result.texts.push(...spread.texts); + result.unreadable = result.unreadable || spread.unreadable; + } + return result; +} + +function nodeIdentifierTexts(outside, name, seen) { + if (seen.has(name)) return { texts: [], unreadable: false }; + seen.add(name); + const objects = initializerBlocks(outside, name).filter(block => block.startsWith('{')); + if (objects.length === 0) return { texts: [], unreadable: true }; + const result = { texts: [], unreadable: false }; + for (const object of objects) { + const nested = nodeObjectTexts(outside, object, seen); + result.texts.push(...nested.texts); + result.unreadable = result.unreadable || nested.unreadable; + } + return result; +} + // True when a distro call in `files` passes an option matching `option`. function distroCallMatches(files, language, option) { const { call } = DISTRO_CALLS[language]; diff --git a/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js b/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js index 4268de7..7e53733 100644 --- a/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js +++ b/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js @@ -345,15 +345,40 @@ function nodeOptionsTexts(outside, args) { if (values.length === 0) return { texts: [], unreadable: false }; const first = values[0].trim(); if (!first) return { texts: [], unreadable: false }; - if (first[0] === '{') return { texts: [first], unreadable: false }; + if (first[0] === '{') return nodeObjectTexts(outside, first, new Set()); const identifier = first.match(/^[A-Za-z_$][\w$]*$/); - if (identifier) { - const initializers = initializerBlocks(outside, identifier[0]); - return initializers.length ? { texts: initializers, unreadable: false } : { texts: [], unreadable: true }; - } + if (identifier) return nodeIdentifierTexts(outside, identifier[0], new Set()); return { texts: [], unreadable: true }; } +// An options object plus the same-file objects its top-level spreads resolve to. Any other spread +// (a call, a member access, an import) makes the options unreadable. +function nodeObjectTexts(outside, text, seen) { + const result = { texts: [text], unreadable: false }; + for (const field of splitTopLevel(text)) { + if (!field.startsWith('...')) continue; + const name = field.slice(3).trim().match(/^[A-Za-z_$][\w$]*$/); + const spread = name ? nodeIdentifierTexts(outside, name[0], seen) : { texts: [], unreadable: true }; + result.texts.push(...spread.texts); + result.unreadable = result.unreadable || spread.unreadable; + } + return result; +} + +function nodeIdentifierTexts(outside, name, seen) { + if (seen.has(name)) return { texts: [], unreadable: false }; + seen.add(name); + const objects = initializerBlocks(outside, name).filter(block => block.startsWith('{')); + if (objects.length === 0) return { texts: [], unreadable: true }; + const result = { texts: [], unreadable: false }; + for (const object of objects) { + const nested = nodeObjectTexts(outside, object, seen); + result.texts.push(...nested.texts); + result.unreadable = result.unreadable || nested.unreadable; + } + return result; +} + // True when a distro call in `files` passes an option matching `option`. function distroCallMatches(files, language, option) { const { call } = DISTRO_CALLS[language]; diff --git a/tests/validate-a365-code-validator.test.js b/tests/validate-a365-code-validator.test.js index 7464e19..dce2ba8 100644 --- a/tests/validate-a365-code-validator.test.js +++ b/tests/validate-a365-code-validator.test.js @@ -458,6 +458,44 @@ await AgenticTokenCacheInstance.refreshObservabilityToken( } }); + test('Node options spreads are followed, and unresolvable spreads keep the call active in both scanners', () => { + const spreadNoRoute = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': 'const baseOptions = { a365: { enabled: true, enableObservabilityExporter: true, tokenResolver: appTokenResolver } };\nuseMicrosoftOpenTelemetry({ ...baseOptions });', + }); + const spreadWired = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': 'const baseOptions = { a365: { enabled: true, enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver: appTokenResolver } };\nuseMicrosoftOpenTelemetry({ ...baseOptions });', + }); + const spreadCall = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': 'useMicrosoftOpenTelemetry({ ...buildBaseOptions() });', + }); + const spreadImport = createFixture({ + 'package.json': NODE_PKG, + 'index.ts': "import { baseOptions } from './otel-options';\nuseMicrosoftOpenTelemetry({ ...baseOptions });", + }); + const s2sIds = result => findingIds(result).filter(id => /^node-obs-(?:delegated-route|token-resolver-missing)$/.test(id)).sort(); + const unwired = ['node-obs-delegated-route', 'node-obs-token-resolver-missing']; + try { + for (const scanner of [VALIDATOR, STANDALONE]) { + assert.deepEqual(s2sIds(runValidator(scanner, spreadNoRoute)), ['node-obs-delegated-route'], + `${scanner}: a same-file options spread is read, so its missing route is reported`); + assert.deepEqual(s2sIds(runValidator(scanner, spreadWired)), [], + `${scanner}: route and resolver inside a same-file options spread count as wired`); + assert.deepEqual(s2sIds(runValidator(scanner, spreadCall)), unwired, + `${scanner}: a spread built by a call cannot be read, so the call stays active and unwired`); + assert.deepEqual(s2sIds(runValidator(scanner, spreadImport)), unwired, + `${scanner}: an imported spread cannot be read, so the call stays active and unwired`); + } + } finally { + cleanup(spreadNoRoute); + cleanup(spreadWired); + cleanup(spreadCall); + cleanup(spreadImport); + } + }); + test('.NET distro without UseS2SEndpoint and with AgenticTokenStruct registration reports both findings', () => { const dir = createFixture({ 'Agent.csproj': '', diff --git a/tests/validate-observability.test.js b/tests/validate-observability.test.js index 2c6843e..30e1f07 100644 --- a/tests/validate-observability.test.js +++ b/tests/validate-observability.test.js @@ -957,6 +957,19 @@ useMicrosoftOpenTelemetry(otelOptions); } finally { cleanup(dir); } }); + test('Node.js options spread resolved to an object literal → ok', () => { + const dir = createFixture({ + ...NODEJS_DISTRO_VALID, + 'src/index.ts': NODEJS_DISTRO_VALID['src/index.ts'].replace( + 'useMicrosoftOpenTelemetry({ a365: { enabled: true, enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver: appTokenResolver } });', + 'const baseOptions = { a365: { enabled: true, enableObservabilityExporter: true, useS2SEndpoint: true, tokenResolver: appTokenResolver } };\nuseMicrosoftOpenTelemetry({ ...baseOptions });'), + }); + try { + const r = runValidator(VALIDATOR, dir); + assert.equal(r.ok, true, r.reason); + } finally { cleanup(dir); } + }); + test('Node.js unresolvable a365 identifier → reports both requirements', () => { const dir = createFixture({ ...NODEJS_DISTRO_VALID, From bb4da5dd4586f4c1ba13f9d03babf9282f4d9f93 Mon Sep 17 00:00:00 2001 From: Krishnadheeraj <12496535+DheerajPannala@users.noreply.github.com> Date: Mon, 28 Sep 2026 20:35:06 +0100 Subject: [PATCH 12/12] Recognize blueprint agents without an aiTeammate:false field Without the detection cache, the registration warning required an explicit aiTeammate:false, so config-free blueprint setup (no a365.config.json) and configs without the field never got it. The generated agenticAppId is written only by blueprint-agent setup, so only an explicit aiTeammate:true now marks an AI Teammate; the cached agentType stays authoritative. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5cbf5f6b-cc40-4b7e-a591-65848db73a12 --- .../stop/validate-a365-code-validator.js | 4 +++- .../references/a365-code-validator.js | 4 +++- tests/validate-a365-code-validator.test.js | 20 +++++++++++++++++++ 3 files changed, 26 insertions(+), 2 deletions(-) diff --git a/plugins/agent365/hooks/stop/validate-a365-code-validator.js b/plugins/agent365/hooks/stop/validate-a365-code-validator.js index 09cf929..98195f3 100644 --- a/plugins/agent365/hooks/stop/validate-a365-code-validator.js +++ b/plugins/agent365/hooks/stop/validate-a365-code-validator.js @@ -866,7 +866,9 @@ function validateSetupArtifacts() { const staticConfig = readJson(path.join(cwd, 'a365.config.json')); const detection = readJson(path.join(cwd, '.a365-workspace-detection.local.json')) || {}; const agentType = String(detection.agentType || '').toLowerCase(); - const isSystemAgent = agentType === 'system-agent' || (!agentType && staticConfig && staticConfig.aiTeammate === false); + // The cached agentType is authoritative. Without it, only an explicit aiTeammate: true marks an AI Teammate: + // the generated agenticAppId is written only by blueprint-agent setup, which can run without a365.config.json. + const isSystemAgent = agentType === 'system-agent' || (!agentType && !(staticConfig && staticConfig.aiTeammate === true)); if (isSystemAgent && generated.agentBlueprintId && generated.agenticAppId && !generated.agentRegistrationId) { add( 'medium', diff --git a/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js b/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js index 7e53733..eeab502 100644 --- a/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js +++ b/plugins/agent365/skills/a365-code-validator/references/a365-code-validator.js @@ -689,7 +689,9 @@ function validateSetupArtifacts() { const staticConfig = readJsonSafe(path.join(cwd, 'a365.config.json')); const detection = readJsonSafe(path.join(cwd, '.a365-workspace-detection.local.json')) || {}; const agentType = String(detection.agentType || '').toLowerCase(); - const isSystemAgent = agentType === 'system-agent' || (!agentType && staticConfig && staticConfig.aiTeammate === false); + // The cached agentType is authoritative. Without it, only an explicit aiTeammate: true marks an AI Teammate: + // the generated agenticAppId is written only by blueprint-agent setup, which can run without a365.config.json. + const isSystemAgent = agentType === 'system-agent' || (!agentType && !(staticConfig && staticConfig.aiTeammate === true)); if (isSystemAgent && generated.agentBlueprintId && generated.agenticAppId && !generated.agentRegistrationId) { add('medium', 'agent-registration-not-recorded', 'a365.generated.config.json has an agent identity but no agentRegistrationId. The S2S route authorizes registered agent instances without an OtelWrite grant; an unregistered instance gets 403 insufficient_scope. Run a365 setup all --agent-registration-only (idempotent).', path.join(cwd, 'a365.generated.config.json')); } diff --git a/tests/validate-a365-code-validator.test.js b/tests/validate-a365-code-validator.test.js index dce2ba8..63d82da 100644 --- a/tests/validate-a365-code-validator.test.js +++ b/tests/validate-a365-code-validator.test.js @@ -598,6 +598,17 @@ token = await auth.exchange_token(context, scopes=["ea9ffc3e-8a23-4a7d-836d-234d 'a365.config.json': JSON.stringify({ aiTeammate: false }), 'a365.generated.config.json': JSON.stringify({ agenticAppId: agentId }), }); + const configFree = createFixture({ + 'a365.generated.config.json': JSON.stringify({ agentBlueprintId: blueprintId, agenticAppId: agentId }), + }); + const noAiTeammateField = createFixture({ + 'a365.config.json': JSON.stringify({ agentBlueprintId: blueprintId }), + 'a365.generated.config.json': JSON.stringify({ agentBlueprintId: blueprintId, agenticAppId: agentId }), + }); + const explicitAiTeammate = createFixture({ + 'a365.config.json': JSON.stringify({ aiTeammate: true }), + 'a365.generated.config.json': JSON.stringify({ agentBlueprintId: blueprintId, agenticAppId: agentId }), + }); try { const finding = runValidator(VALIDATOR, systemAgent).findings.find(f => f.id === 'agent-registration-not-recorded'); assert.ok(finding, 'expected agent-registration-not-recorded'); @@ -609,6 +620,12 @@ token = await auth.exchange_token(context, scopes=["ea9ffc3e-8a23-4a7d-836d-234d assert.ok(!findingIds(runValidator(scanner, aiTeammate)).includes('agent-registration-not-recorded'), scanner); assert.ok(!findingIds(runValidator(scanner, noBlueprint)).includes('agent-registration-not-recorded'), `${scanner}: without a blueprint ID there is no blueprint agent instance to register`); + assert.ok(findingIds(runValidator(scanner, configFree)).includes('agent-registration-not-recorded'), + `${scanner}: config-free blueprint setup writes no a365.config.json, and only blueprint setup writes agenticAppId`); + assert.ok(findingIds(runValidator(scanner, noAiTeammateField)).includes('agent-registration-not-recorded'), + `${scanner}: a blueprint config does not need an explicit aiTeammate:false field`); + assert.ok(!findingIds(runValidator(scanner, explicitAiTeammate)).includes('agent-registration-not-recorded'), + `${scanner}: an explicit aiTeammate:true config is an AI Teammate, which --agent-registration-only does not apply to`); } } finally { cleanup(systemAgent); @@ -616,6 +633,9 @@ token = await auth.exchange_token(context, scopes=["ea9ffc3e-8a23-4a7d-836d-234d cleanup(aiTeammate); cleanup(fallback); cleanup(noBlueprint); + cleanup(configFree); + cleanup(noAiTeammateField); + cleanup(explicitAiTeammate); } });