From 4e61df1d001c84b4b22034cf76bd5a8679904d91 Mon Sep 17 00:00:00 2001 From: Shamir Abdul Aziz Date: Wed, 13 May 2026 09:55:18 -0700 Subject: [PATCH 1/4] Reorganize labs under unified Zava naming + add master README Renames the lab directories under labs/ to a consistent Zava-themed naming scheme so all labs share branding, and rewrites labs/README.md as a true master index. Renames: - starter-lab -> zava-eats (Grubify food-ordering on ACA) - azure-friday -> zava-cafe (App Service + Azure SQL e-commerce) - zava-aks-postgres -> zava-athletic (AKS + private Postgres) - vm-cosmosdb -> zava-infra/scenarios/perf-drift - deployment-compliance -> zava-infra/scenarios/compliance - terraform-drift... -> zava-infra/scenarios/tf-drift - (split out) -> zava-itsupport (ServiceNow laptop-replacement) The IT-support lab is split out of zava-cafe and azure-friday into its own standalone lab (zava-itsupport). zava-power (PowerGrid ZeroOps) is unchanged. Adds: - labs/README.md - rewritten as master index with per-lab synopses, decision aids, and shared prereqs - labs/LAUNCHER.md - multi-lab dispatcher docs - labs/AGENTS.md - lab authoring guide - labs/lab.sh / lab.ps1 - launcher scripts - labs/sim.sh / sim.ps1 - simulator dispatcher - labs/_platform/ - shared platform helpers - labs/recipes/ - 3 portable agent-config bundles - sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/ Misc: - .gitignore: exclude labs/**/.deployed/ launcher state and *.legacy Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .gitignore | 5 + labs/AGENTS.md | 127 + labs/LAUNCHER.md | 118 + labs/README.md | 327 +- labs/_platform/check-prereqs.ps1 | 117 + labs/_platform/helpers/manifest.py | 149 + labs/_platform/http_trigger.py | 208 + labs/_platform/schema/lab.example.yaml | 56 + labs/_platform/schema/lab.schema.json | 93 + labs/_platform/template/README.md.tmpl | 33 + labs/_platform/template/azure.yaml.tmpl | 34 + labs/_platform/template/infra/main.bicep.tmpl | 26 + labs/_platform/template/lab.yaml.tmpl | 35 + .../scripts/check-environment.ps1.tmpl | 48 + .../template/scripts/post-provision.ps1.tmpl | 52 + .../scripts/scenarios/example.ps1.tmpl | 13 + labs/deployment-compliance/azure.yaml | 11 - labs/lab.ps1 | 223 + labs/lab.sh | 10 + labs/recipes/README.md | 89 + labs/recipes/_convert_ops.py | 223 + .../.gitignore | 8 + .../README.md | 152 + .../agent.json | 107 + .../auto-investigate-azmon.yaml | 21 + .../incident-platforms/azure-monitor.yaml | 3 + .../incident-platforms/servicenow.yaml | 3 + .../scheduled-tasks/weekly-cost-report.yaml | 15 + .../config/hooks/change-risk-assessor.yaml | 40 + .../config/hooks/sql-write-guard.yaml | 30 + .../config/skills/sql-blocking-diagnosis.md | 29 + .../config/skills/sql-blocking-diagnosis.yaml | 11 + .../config/skills/sql-blocking-fix.md | 31 + .../config/skills/sql-blocking-fix.yaml | 12 + .../config/skills/sql-performance-fix.md | 38 + .../config/skills/sql-performance-fix.yaml | 13 + .../config/skills/sql-query-diagnosis.md | 31 + .../config/skills/sql-query-diagnosis.yaml | 11 + .../deployment-validator-gh.instructions.md | 32 + .../subagents/deployment-validator-gh.yaml | 13 + .../deployment-validator.instructions.md | 18 + .../subagents/deployment-validator.yaml | 14 + ...l-performance-investigator.instructions.md | 52 + .../sql-performance-investigator.yaml | 23 + .../config/tools/AssessChangeRisk.yaml | 110 + .../connectors.json | 12 + .../expected-config.json | 44 + .../.gitignore | 8 + .../README.md | 114 + .../agent.json | 86 + .../snow-laptop-replacement.yaml | 20 + .../incident-platforms/servicenow.yaml | 3 + .../it-support-handler.instructions.md | 48 + .../config/subagents/it-support-handler.yaml | 18 + .../config/tools/CheckWarranty.yaml | 45 + .../tools/LookupServiceNowIncident.yaml | 60 + .../connectors.json | 12 + .../expected-config.json | 30 + .../.gitignore | 3 + .../README.md | 121 + .../agent.json | 103 + .../auto-investigate-azmon.yaml | 21 + .../incident-platforms/azure-monitor.yaml | 3 + .../incident-platforms/servicenow.yaml | 3 + .../pod-fleet-audit-daily.yaml | 10 + .../skills/config-regression-diagnosis.md | 105 + .../skills/config-regression-diagnosis.yaml | 9 + .../skills/crash-regression-diagnosis.md | 93 + .../skills/crash-regression-diagnosis.yaml | 15 + .../config/skills/deployment-rollback.md | 352 ++ .../config/skills/deployment-rollback.yaml | 8 + .../config/skills/deployment-validation.md | 173 + .../config/skills/deployment-validation.yaml | 17 + .../config/skills/disk-pressure-diagnosis.md | 135 + .../skills/disk-pressure-diagnosis.yaml | 9 + .../config/skills/grid-status-diagnosis.md | 268 ++ .../config/skills/grid-status-diagnosis.yaml | 8 + .../config/skills/meter-api-diagnosis.md | 291 ++ .../config/skills/meter-api-diagnosis.yaml | 8 + .../skills/notification-svc-diagnosis.md | 282 ++ .../skills/notification-svc-diagnosis.yaml | 8 + .../config/skills/outage-api-diagnosis.md | 254 ++ .../config/skills/outage-api-diagnosis.yaml | 8 + .../skills/perf-regression-diagnosis.md | 130 + .../skills/perf-regression-diagnosis.yaml | 10 + .../config/skills/plot-incident-metrics.md | 97 + .../config/skills/plot-incident-metrics.yaml | 9 + .../config/skills/pod-fleet-audit-deck.md | 366 ++ .../config/skills/pod-fleet-audit-deck.yaml | 15 + .../config/skills/release-on-sre-fix.md | 92 + .../config/skills/release-on-sre-fix.yaml | 10 + .../config/skills/repo-routing.md | 101 + .../config/skills/repo-routing.yaml | 13 + .../config/skills/sre-agent-customizer.md | 27 + .../config/skills/sre-agent-customizer.yaml | 21 + .../deployment-validator.instructions.md | 212 + .../subagents/deployment-validator.yaml | 33 + .../incident-handler.instructions.md | 70 + .../config/subagents/incident-handler.yaml | 31 + ...eline-failure-investigator.instructions.md | 101 + .../pipeline-failure-investigator.yaml | 30 + .../pod-incident-remediator.instructions.md | 84 + .../subagents/pod-incident-remediator.yaml | 32 + .../release-orchestrator.instructions.md | 49 + .../subagents/release-orchestrator.yaml | 26 + .../utility-ops-agent.instructions.md | 159 + .../config/subagents/utility-ops-agent.yaml | 33 + .../subagents/vm-ops-agent.instructions.md | 34 + .../config/subagents/vm-ops-agent.yaml | 31 + .../web-app-troubleshooter.instructions.md | 106 + .../subagents/web-app-troubleshooter.yaml | 32 + .../connectors.json | 40 + .../expected-config.json | 54 + labs/sim.ps1 | 165 + labs/sim.sh | 9 + labs/starter-lab/azure.yaml | 15 - labs/vm-cosmosdb/azure.yaml | 13 - .../.github/skills/deploying-demo/SKILL.md | 0 .../skills/managing-sre-agent/SKILL.md | 0 .../.github/skills/running-demo/SKILL.md | 0 .../running-demo/scripts/break-db-perf.ps1 | 0 .../running-demo/scripts/break-network.ps1 | 0 .../skills/running-demo/scripts/break-sql.ps1 | 0 .../running-demo/scripts/fix-db-perf.ps1 | 0 .../running-demo/scripts/fix-network.ps1 | 0 .../skills/running-demo/scripts/fix-sql.ps1 | 0 .../.gitignore | 0 .../AGENTS.md | 0 .../README.md | 18 +- .../azure.yaml | 4 +- .../docs/images/storefront-broken.png | Bin .../docs/images/storefront-healthy.png | Bin .../infra/main.bicep | 2 +- .../infra/main.bicepparam | 2 +- .../infra/main.json | 6 +- .../infra/modules/acr.bicep | 0 .../infra/modules/aks.bicep | 0 .../infra/modules/identity.bicep | 0 .../infra/modules/monitoring.bicep | 0 .../infra/modules/pg-admin.bicep | 0 .../infra/modules/postgresql.bicep | 0 .../infra/modules/sre-agent.bicep | 4 +- .../infra/modules/vnet.bicep | 0 .../k8s/README.md | 0 .../k8s/api-deployment.yaml | 0 .../k8s/api-service.yaml | 0 .../k8s/configmap.yaml | 0 .../k8s/ingress.yaml | 0 .../k8s/jobs/load-categories.yaml | 0 .../k8s/secret.yaml | 0 .../k8s/service-account.yaml | 0 .../k8s/storefront-deployment.yaml | 0 .../k8s/storefront-service.yaml | 0 labs/zava-athletic/lab.yaml | 49 + .../scripts/_aks-helpers.ps1 | 0 .../scripts/check-environment.ps1 | 0 .../scripts/post-provision.ps1 | 28 + .../scripts/setup-sre-agent.ps1 | 0 .../scripts/watch-agent.ps1 | 0 .../src/api/.dockerignore | 0 .../src/api/Dockerfile | 0 .../src/api/bin/run-sql.js | 0 .../src/api/db/client.js | 0 .../src/api/db/seed.js | 0 .../src/api/logging/logger.js | 0 .../src/api/package-lock.json | 0 .../src/api/package.json | 0 .../src/api/routes/diagnostics.js | 0 .../src/api/routes/health.js | 0 .../src/api/routes/orders.js | 0 .../src/api/routes/products.js | 0 .../src/api/server.js | 0 .../src/storefront/.dockerignore | 0 .../src/storefront/Dockerfile | 0 .../src/storefront/package-lock.json | 0 .../src/storefront/package.json | 0 .../src/storefront/server.js | 0 .../knowledge-base/zava-architecture.md | 0 labs/zava-cafe/.gitignore | 12 + labs/zava-cafe/README.md | 112 + labs/zava-cafe/azure.yaml | 43 + labs/zava-cafe/dashboard.json | 857 ++++ labs/zava-cafe/infra/main.bicep | 68 + labs/zava-cafe/infra/main.bicepparam | 8 + .../infra/modules/identity.bicep | 0 .../infra/modules/monitoring.bicep | 0 .../infra/modules/sre-agent.bicep | 0 .../infra/modules/subscription-rbac.bicep | 0 labs/zava-cafe/infra/resources.bicep | 444 ++ labs/zava-cafe/infra/seed-database.sql | 130 + labs/zava-cafe/lab.yaml | 30 + labs/zava-cafe/scripts/invoke-thread.sh | 44 + labs/zava-cafe/scripts/post-provision.sh | 402 ++ labs/zava-cafe/scripts/prereqs.sh | 127 + labs/zava-cafe/scripts/sql_entra.py | 105 + labs/zava-cafe/simulator/demo.py | 1635 +++++++ labs/zava-cafe/simulator/expand_data.py | 57 + labs/zava-cafe/simulator/requirements.txt | 3 + labs/zava-cafe/src/.gitignore | 5 + labs/zava-cafe/src/Program.cs | 140 + .../src/Properties/launchSettings.json | 41 + labs/zava-cafe/src/ZavaCafeApp.csproj | 16 + labs/zava-cafe/src/ZavaCafeApp.http | 6 + .../src/appsettings.Development.json | 14 + labs/zava-cafe/src/appsettings.json | 15 + .../sre-config/agent1/.github/instructions.md | 1478 +++++++ .../deployment-validator-gh.yaml | 53 + .../deployment-validator.yaml | 34 + .../agent1/agents/example_agent.yaml | 19 + .../sql-performance-investigator.yaml | 121 + .../agent1/hooks/change-risk-assessor.yaml | 40 + .../agent1/hooks/sql-write-guard.yaml | 30 + .../weekly-cost-report.yaml | 20 + .../skills/sql-blocking-diagnosis/SKILL.md | 40 + .../agent1/skills/sql-blocking-fix/SKILL.md | 43 + .../skills/sql-performance-fix/SKILL.md | 50 + .../skills/sql-query-diagnosis/SKILL.md | 44 + .../AssessChangeRisk/AssessChangeRisk.yaml | 113 + .../sre-config/agent1/tools/example_tool.yaml | 19 + .../sre-config/simulate-dtu-spike.ps1 | 39 + .../sre-config/simulate-slow-queries.ps1 | 53 + labs/zava-eats/.github/instructions.md | 1478 +++++++ labs/{starter-lab => zava-eats}/.gitignore | 0 labs/{starter-lab => zava-eats}/README.md | 20 +- labs/zava-eats/agents/example_agent.yaml | 19 + labs/zava-eats/azure.yaml | 42 + .../docs/architecture.svg | 0 .../infra/main.bicep | 0 .../infra/main.bicepparam | 0 .../infra/modules/alert-rules.bicep | 0 .../infra/modules/container-app.bicep | 0 labs/zava-eats/infra/modules/identity.bicep | 19 + labs/zava-eats/infra/modules/monitoring.bicep | 40 + labs/zava-eats/infra/modules/sre-agent.bicep | 81 + .../infra/modules/subscription-rbac.bicep | 39 + .../infra/resources.bicep | 0 .../knowledge-base/github-issue-triage.md | 0 .../knowledge-base/grubify-architecture.md | 0 .../knowledge-base/http-500-errors.md | 0 .../incident-report-template.md | 0 labs/zava-eats/lab.yaml | 28 + .../lab/skillable-instructions.md | 4 +- .../scripts/break-app.sh | 0 .../scripts/create-sample-issues.sh | 0 .../scripts/generate-pptx.py | 0 labs/zava-eats/scripts/invoke-thread.sh | 54 + .../scripts/post-provision-srectl.sh | 0 .../scripts/post-provision.sh | 82 + .../scripts/prereqs.sh | 0 .../scripts/setup-github-srectl.sh | 0 .../scripts/setup-github.sh | 0 .../scripts/setup.sh | 0 .../scripts/yaml-to-api-json.py | 0 .../sre-config/.github/instructions.md | 1478 +++++++ .../sre-config/agents/code-analyzer.yaml | 0 .../sre-config/agents/example_agent.yaml | 19 + .../agents/incident-handler-core.yaml | 0 .../agents/incident-handler-full.yaml | 0 .../sre-config/agents/issue-triager.yaml | 0 .../sre-config/connectors/github-oauth.yaml | 0 .../skills/grubify-diagnosis/SKILL.md | 38 + .../CheckGrubifyHealth.yaml | 60 + .../sre-config/tools/example_tool.yaml | 19 + labs/zava-eats/tools/example_tool.yaml | 19 + labs/zava-infra/README.md | 52 + labs/zava-infra/lab.yaml | 28 + .../workflows/deploy-container-app.yml | 0 .../scenarios/compliance}/README.md | 16 + .../scenarios/compliance/azure.yaml | 41 + .../compliance}/docs/architecture.svg | 0 .../hooks/deployment-compliance-approval.yaml | 0 .../containerapp-deployment-alert.yaml | 0 .../scenarios/compliance}/infra/main.bicep | 0 .../scenarios/compliance}/infra/main.json | 0 .../infra/modules/monitoring.bicep | 0 .../compliance}/infra/modules/roles.bicep | 0 .../compliance}/infra/modules/sre-agent.bicep | 0 .../compliance}/infra/modules/workload.bicep | 0 labs/zava-infra/scenarios/compliance/lab.yaml | 23 + .../scheduled-tasks/compliance-scan.yaml | 0 .../compliance}/scripts/break-compliance.sh | 0 .../scenarios/compliance}/scripts/deploy.sh | 0 .../compliance}/scripts/post-deploy.sh | 28 + .../scenarios/compliance}/scripts/prereqs.sh | 0 .../deployment-compliance-check/SKILL.md | 0 .../compliance_detection.md | 0 .../scenarios/compliance}/src/api/Dockerfile | 0 .../scenarios/compliance}/src/api/init.sql | 0 .../compliance}/src/api/package.json | 0 .../scenarios/compliance}/src/api/server.js | 0 .../zava-infra/scenarios/perf-drift/README.md | 54 + .../scenarios/perf-drift/azure.yaml | 27 + .../perf-drift}/docs/architecture.svg | 0 .../hooks/vm-remediation-approval.yaml | 0 .../scenarios/perf-drift}/infra/main.bicep | 0 .../perf-drift}/infra/main.bicepparam | 0 .../perf-drift}/infra/modules/cosmosdb.bicep | 0 .../infra/modules/monitoring.bicep | 0 .../perf-drift}/infra/modules/network.bicep | 0 .../perf-drift}/infra/modules/roles.bicep | 0 .../perf-drift}/infra/modules/sre-agent.bicep | 0 .../perf-drift}/infra/modules/vm.bicep | 0 labs/zava-infra/scenarios/perf-drift/lab.yaml | 30 + .../compliance-drift-scan.yaml | 0 .../scenarios/perf-drift}/scripts/break-db.sh | 0 .../scenarios/perf-drift}/scripts/break-vm.sh | 14 +- .../perf-drift}/scripts/install-app.sh | 0 .../perf-drift}/scripts/post-deploy.sh | 23 + .../skills/compliance-drift-detection.md | 0 .../compliance-drift-detection/SKILL.md | 0 .../skills/vm-performance-diagnostics.md | 0 .../vm-performance-diagnostics/SKILL.md | 0 .../scenarios/perf-drift}/src/app.js | 0 .../scenarios/perf-drift}/src/package.json | 0 .../scenarios/perf-drift}/src/setup.sh | 0 .../scenarios/tf-drift}/README.md | 18 +- .../scenarios/tf-drift}/app/package.json | 0 .../scenarios/tf-drift}/app/server.js | 0 labs/zava-infra/scenarios/tf-drift/lab.yaml | 44 + .../tf-drift}/scripts/deploy-app.ps1 | 0 .../tf-drift}/scripts/generate-load.ps1 | 0 .../tf-drift}/scripts/induce-drift.ps1 | 0 .../tf-drift}/scripts/revert-drift.ps1 | 0 .../scripts/simulate-tfc-notification.ps1 | 0 .../skills/terraform-drift-analysis.md | 0 .../tf-drift}/terraform/logic-app.tf | 0 .../scenarios/tf-drift}/terraform/main.tf | 0 .../scenarios/tf-drift}/terraform/outputs.tf | 0 .../tf-drift}/terraform/providers.tf | 0 .../terraform/terraform.tfvars.example | 0 .../tf-drift}/terraform/variables.tf | 0 labs/zava-itsupport/.gitignore | 9 + labs/zava-itsupport/README.md | 66 + labs/zava-itsupport/azure.yaml | 29 + labs/zava-itsupport/infra/abbreviations.json | 10 + labs/zava-itsupport/infra/main.bicep | 58 + labs/zava-itsupport/infra/main.bicepparam | 5 + .../infra/modules/identity.bicep | 17 + .../infra/modules/monitoring.bicep | 34 + .../infra/modules/sre-agent.bicep | 79 + .../infra/modules/subscription-rbac.bicep | 33 + labs/zava-itsupport/infra/resources.bicep | 224 + labs/zava-itsupport/lab.yaml | 23 + .../laptop-request-site/.dockerignore | 3 + .../laptop-request-site/Dockerfile | 9 + .../laptop-request-site/index.html | 278 ++ .../laptop-request-site/package.json | 7 + .../laptop-request-site/server.js | 18 + .../laptop-request-site/style.css | 444 ++ .../scripts/laptop-request-demo.sh | 37 + labs/zava-itsupport/scripts/post-provision.sh | 247 ++ .../sre-config/.github/instructions.md | 1478 +++++++ .../sre-config/agents/example_agent.yaml | 19 + .../it-support-handler.yaml | 69 + .../tools/CheckWarranty/CheckWarranty.yaml | 45 + .../LookupServiceNowIncident.yaml | 60 + .../sre-config/tools/example_tool.yaml | 19 + .../warranty-tool/.dockerignore | 4 + labs/zava-itsupport/warranty-tool/Dockerfile | 8 + labs/zava-itsupport/warranty-tool/app.py | 119 + .../warranty-tool/check_warranty.py | 36 + .../warranty-tool/requirements.txt | 4 + labs/zava-itsupport/warranty-tool/startup.sh | 2 + labs/zava-power/.gitignore | 9 + .../deployment-validator.yaml | 244 ++ .../incident-handler/incident-handler.yaml | 100 + .../pipeline-failure-investigator.yaml | 129 + .../pod-incident-remediator.yaml | 115 + .../release-orchestrator.yaml | 74 + .../utility-ops-agent/utility-ops-agent.yaml | 191 + .../agents/vm-ops-agent/vm-ops-agent.yaml | 64 + .../web-app-troubleshooter.yaml | 136 + .../sre-config/connectors/datadog-mcp.yaml | 40 + .../sre-config/connectors/dynatrace-mcp.yaml | 39 + .../sre-config/connectors/servicenow-mcp.yaml | 46 + .../response-plans/auto-investigate.yaml | 43 + .../scheduled-tasks/pod-fleet-audit.yaml | 33 + .../skills/SRE Agent customizer/SKILL.md | 49 + .../config-regression-diagnosis/SKILL.md | 117 + .../crash-regression-diagnosis/SKILL.md | 106 + .../skills/deployment-rollback/SKILL.md | 360 ++ .../skills/deployment-validation/SKILL.md | 187 + .../skills/disk-pressure-diagnosis/SKILL.md | 143 + .../skills/grid-status-diagnosis/SKILL.md | 276 ++ .../skills/meter-api-diagnosis/SKILL.md | 299 ++ .../notification-svc-diagnosis/SKILL.md | 290 ++ .../skills/outage-api-diagnosis/SKILL.md | 262 ++ .../skills/perf-regression-diagnosis/SKILL.md | 144 + .../skills/plot-incident-metrics/SKILL.md | 109 + .../skills/pod-fleet-audit-deck/SKILL.md | 379 ++ .../skills/release-on-sre-fix/SKILL.md | 107 + .../sre-config/skills/repo-routing/SKILL.md | 113 + .../skills/servicenow-incident-mgmt/SKILL.md | 256 ++ .../tools/BurstLoadTest/BurstLoadTest.yaml | 117 + .../CreateServiceNowIncident.yaml | 71 + .../GetActiveRevision/GetActiveRevision.yaml | 86 + .../LookupServiceNowIncident.yaml | 60 + .../ProbeServiceLatency.yaml | 88 + .../.rendered/sre-config/tools/README.md | 129 + .../RemediateContainerApp.yaml | 204 + .../ResolveServiceNowIncident.yaml | 67 + .../RollbackContainerAppRevision.yaml | 122 + .../UpdateServiceNowWorkNotes.yaml | 59 + .../tools/UploadChartToServiceNow.yaml | 216 + labs/zava-power/AGENTS.md | 58 + labs/zava-power/README.md | 105 + labs/zava-power/azure.yaml | 37 + labs/zava-power/bugs/build-failure/app.py | 50 + .../bugs/build-failure/requirements.txt | 3 + labs/zava-power/bugs/config/main.go | 203 + labs/zava-power/bugs/crash/app.py | 215 + labs/zava-power/bugs/perf/server.js | 146 + labs/zava-power/infra/bicepconfig.json | 17 + labs/zava-power/infra/main.bicep | 153 + labs/zava-power/infra/main.bicepparam | 7 + labs/zava-power/infra/modules/aks.bicep | 60 + labs/zava-power/infra/modules/alerts.bicep | 88 + labs/zava-power/infra/modules/arc-vm.bicep | 112 + .../infra/modules/container-apps.bicep | 230 + .../infra/modules/container-registry.bicep | 19 + .../infra/modules/observability.bicep | 58 + labs/zava-power/infra/modules/sre-agent.bicep | 134 + .../infra/modules/sre-identity.bicep | 19 + .../roles/powergrid-sre-agent-operator.json | 24 + .../deployment-rollback-runbook.md | 267 ++ .../knowledge-base/grid-status-runbook.md | 273 ++ .../incident-report-template.md | 187 + .../knowledge-base/meter-api-runbook.md | 299 ++ .../notification-svc-runbook.md | 287 ++ .../knowledge-base/outage-api-runbook.md | 261 ++ .../knowledge-base/powergrid-architecture.md | 220 + .../knowledge-base/sre-agent-architecture.md | 138 + labs/zava-power/lab.yaml | 62 + labs/zava-power/pipelines/build.yml | 167 + labs/zava-power/pipelines/release.yml | 185 + labs/zava-power/scripts/check-environment.ps1 | 64 + labs/zava-power/scripts/post-provision.ps1 | 241 ++ labs/zava-power/scripts/render-config.py | 48 + labs/zava-power/simulator/demo.py | 3749 ++++++++++++++++ labs/zava-power/simulator/requirements.txt | 3 + .../src/grid-status-api/.dockerignore | 2 + .../zava-power/src/grid-status-api/Dockerfile | 12 + .../src/grid-status-api/package-lock.json | 3818 +++++++++++++++++ .../src/grid-status-api/package.json | 13 + labs/zava-power/src/grid-status-api/server.js | 213 + labs/zava-power/src/meter-api/Dockerfile | 13 + labs/zava-power/src/meter-api/MeterApi.csproj | 14 + labs/zava-power/src/meter-api/Program.cs | 88 + .../meter-api/Properties/launchSettings.json | 13 + .../bin/Debug/net8.0/MeterApi.deps.json | 23 + .../meter-api/bin/Debug/net8.0/MeterApi.dll | Bin 0 -> 15872 bytes .../meter-api/bin/Debug/net8.0/MeterApi.exe | Bin 0 -> 151552 bytes .../meter-api/bin/Debug/net8.0/MeterApi.pdb | Bin 0 -> 21908 bytes .../Debug/net8.0/MeterApi.runtimeconfig.json | 19 + .../MeterApi.staticwebassets.endpoints.json | 1 + ...CoreApp,Version=v8.0.AssemblyAttributes.cs | 4 + .../obj/Debug/net8.0/MeterApi.AssemblyInfo.cs | 22 + .../net8.0/MeterApi.AssemblyInfoInputs.cache | 1 + ....GeneratedMSBuildEditorConfig.editorconfig | 23 + .../Debug/net8.0/MeterApi.GlobalUsings.g.cs | 17 + ...rApi.MvcApplicationPartsAssemblyInfo.cache | 0 .../obj/Debug/net8.0/MeterApi.assets.cache | Bin 0 -> 205 bytes .../MeterApi.csproj.CoreCompileInputs.cache | 1 + .../MeterApi.csproj.FileListAbsolute.txt | 26 + .../meter-api/obj/Debug/net8.0/MeterApi.dll | Bin 0 -> 15872 bytes .../net8.0/MeterApi.genruntimeconfig.cache | 1 + .../meter-api/obj/Debug/net8.0/MeterApi.pdb | Bin 0 -> 21908 bytes .../meter-api/obj/Debug/net8.0/apphost.exe | Bin 0 -> 151552 bytes .../obj/Debug/net8.0/ref/MeterApi.dll | Bin 0 -> 6144 bytes .../obj/Debug/net8.0/refint/MeterApi.dll | Bin 0 -> 6144 bytes .../Debug/net8.0/rjsmcshtml.dswa.cache.json | 1 + .../Debug/net8.0/rjsmrazor.dswa.cache.json | 1 + .../staticwebassets.build.endpoints.json | 1 + .../Debug/net8.0/staticwebassets.build.json | 1 + .../net8.0/staticwebassets.build.json.cache | 1 + .../obj/Debug/net8.0/swae.build.ex.cache | 0 .../obj/MeterApi.csproj.nuget.dgspec.json | 77 + .../obj/MeterApi.csproj.nuget.g.props | 16 + .../obj/MeterApi.csproj.nuget.g.targets | 2 + .../src/meter-api/obj/project.assets.json | 83 + .../src/meter-api/obj/project.nuget.cache | 8 + .../src/notification-svc/Dockerfile | 23 + labs/zava-power/src/notification-svc/go.mod | 3 + labs/zava-power/src/notification-svc/main.go | 227 + labs/zava-power/src/outage-api/Dockerfile | 12 + labs/zava-power/src/outage-api/app.py | 225 + .../src/outage-api/requirements.txt | 3 + labs/zava-power/src/portal-web/.dockerignore | 3 + labs/zava-power/src/portal-web/Dockerfile | 18 + labs/zava-power/src/portal-web/index.html | 13 + labs/zava-power/src/portal-web/nginx.conf | 36 + .../src/portal-web/package-lock.json | 1724 ++++++++ labs/zava-power/src/portal-web/package.json | 21 + labs/zava-power/src/portal-web/src/App.css | 462 ++ labs/zava-power/src/portal-web/src/App.jsx | 45 + .../src/components/StatusBanner.jsx | 63 + labs/zava-power/src/portal-web/src/main.jsx | 13 + .../src/portal-web/src/pages/Billing.jsx | 70 + .../src/portal-web/src/pages/Dashboard.jsx | 90 + .../src/portal-web/src/pages/Outages.jsx | 103 + .../src/portal-web/src/pages/Usage.jsx | 69 + labs/zava-power/src/portal-web/vite.config.js | 9 + .../deployment-validator.yaml | 244 ++ .../incident-handler/incident-handler.yaml | 100 + .../pipeline-failure-investigator.yaml | 129 + .../pod-incident-remediator.yaml | 115 + .../release-orchestrator.yaml | 74 + .../utility-ops-agent/utility-ops-agent.yaml | 191 + .../agents/vm-ops-agent/vm-ops-agent.yaml | 64 + .../web-app-troubleshooter.yaml | 136 + .../sre-config/connectors/datadog-mcp.yaml | 40 + .../sre-config/connectors/dynatrace-mcp.yaml | 39 + .../sre-config/connectors/servicenow-mcp.yaml | 46 + .../response-plans/auto-investigate.yaml | 43 + .../scheduled-tasks/pod-fleet-audit.yaml | 33 + .../skills/SRE Agent customizer/SKILL.md | 49 + .../config-regression-diagnosis/SKILL.md | 117 + .../crash-regression-diagnosis/SKILL.md | 106 + .../skills/deployment-rollback/SKILL.md | 360 ++ .../skills/deployment-validation/SKILL.md | 187 + .../skills/disk-pressure-diagnosis/SKILL.md | 143 + .../skills/grid-status-diagnosis/SKILL.md | 276 ++ .../skills/meter-api-diagnosis/SKILL.md | 299 ++ .../notification-svc-diagnosis/SKILL.md | 290 ++ .../skills/outage-api-diagnosis/SKILL.md | 262 ++ .../skills/perf-regression-diagnosis/SKILL.md | 144 + .../skills/plot-incident-metrics/SKILL.md | 109 + .../skills/pod-fleet-audit-deck/SKILL.md | 379 ++ .../skills/release-on-sre-fix/SKILL.md | 107 + .../sre-config/skills/repo-routing/SKILL.md | 113 + .../skills/servicenow-incident-mgmt/SKILL.md | 256 ++ .../tools/BurstLoadTest/BurstLoadTest.yaml | 117 + .../CreateServiceNowIncident.yaml | 71 + .../GetActiveRevision/GetActiveRevision.yaml | 86 + .../LookupServiceNowIncident.yaml | 60 + .../ProbeServiceLatency.yaml | 88 + labs/zava-power/sre-config/tools/README.md | 129 + .../RemediateContainerApp.yaml | 204 + .../ResolveServiceNowIncident.yaml | 67 + .../RollbackContainerAppRevision.yaml | 122 + .../UpdateServiceNowWorkNotes.yaml | 59 + .../tools/UploadChartToServiceNow.yaml | 216 + .../.gitignore | 2 + .../README.md | 35 + .../agent.json | 93 + .../incident-filters/powergrid-sev01.yaml | 12 + .../incident-platforms/azure-monitor.yaml | 5 + .../incident-platforms/servicenow.yaml | 5 + .../scheduled-tasks/pod-fleet-audit.yaml | 13 + .../config/common-prompts/safety-rules.yaml | 9 + .../config/hooks/deny-prod-deletes.yaml | 10 + .../hooks/require-approval-for-restarts.yaml | 10 + .../skills/config-regression-diagnosis.md | 105 + .../skills/config-regression-diagnosis.yaml | 13 + .../skills/crash-regression-diagnosis.md | 93 + .../skills/crash-regression-diagnosis.yaml | 15 + .../config/skills/deployment-rollback.md | 352 ++ .../config/skills/deployment-rollback.yaml | 9 + .../config/skills/deployment-validation.md | 173 + .../config/skills/deployment-validation.yaml | 18 + .../config/skills/disk-pressure-diagnosis.md | 135 + .../skills/disk-pressure-diagnosis.yaml | 10 + .../config/skills/grid-status-diagnosis.md | 268 ++ .../config/skills/grid-status-diagnosis.yaml | 9 + .../config/skills/meter-api-diagnosis.md | 291 ++ .../config/skills/meter-api-diagnosis.yaml | 9 + .../skills/notification-svc-diagnosis.md | 282 ++ .../skills/notification-svc-diagnosis.yaml | 9 + .../config/skills/outage-api-diagnosis.md | 254 ++ .../config/skills/outage-api-diagnosis.yaml | 9 + .../skills/perf-regression-diagnosis.md | 130 + .../skills/perf-regression-diagnosis.yaml | 17 + .../config/skills/plot-incident-metrics.md | 97 + .../config/skills/plot-incident-metrics.yaml | 13 + .../config/skills/pod-fleet-audit-deck.md | 366 ++ .../config/skills/pod-fleet-audit-deck.yaml | 15 + .../config/skills/release-on-sre-fix.md | 92 + .../config/skills/release-on-sre-fix.yaml | 19 + .../config/skills/repo-routing.md | 101 + .../config/skills/repo-routing.yaml | 13 + .../config/skills/servicenow-incident-mgmt.md | 242 ++ .../skills/servicenow-incident-mgmt.yaml | 7 + .../deployment-validator.instructions.md | 212 + .../subagents/deployment-validator.yaml | 18 + .../incident-handler.instructions.md | 70 + .../config/subagents/incident-handler.yaml | 16 + ...eline-failure-investigator.instructions.md | 101 + .../pipeline-failure-investigator.yaml | 15 + .../pod-incident-remediator.instructions.md | 84 + .../subagents/pod-incident-remediator.yaml | 18 + .../release-orchestrator.instructions.md | 49 + .../subagents/release-orchestrator.yaml | 12 + .../utility-ops-agent.instructions.md | 159 + .../config/subagents/utility-ops-agent.yaml | 19 + .../subagents/vm-ops-agent.instructions.md | 34 + .../config/subagents/vm-ops-agent.yaml | 16 + .../web-app-troubleshooter.instructions.md | 106 + .../subagents/web-app-troubleshooter.yaml | 18 + .../connectors.json | 28 + .../expected-config.json | 69 + 600 files changed, 53320 insertions(+), 293 deletions(-) create mode 100644 labs/AGENTS.md create mode 100644 labs/LAUNCHER.md create mode 100644 labs/_platform/check-prereqs.ps1 create mode 100644 labs/_platform/helpers/manifest.py create mode 100644 labs/_platform/http_trigger.py create mode 100644 labs/_platform/schema/lab.example.yaml create mode 100644 labs/_platform/schema/lab.schema.json create mode 100644 labs/_platform/template/README.md.tmpl create mode 100644 labs/_platform/template/azure.yaml.tmpl create mode 100644 labs/_platform/template/infra/main.bicep.tmpl create mode 100644 labs/_platform/template/lab.yaml.tmpl create mode 100644 labs/_platform/template/scripts/check-environment.ps1.tmpl create mode 100644 labs/_platform/template/scripts/post-provision.ps1.tmpl create mode 100644 labs/_platform/template/scripts/scenarios/example.ps1.tmpl delete mode 100644 labs/deployment-compliance/azure.yaml create mode 100644 labs/lab.ps1 create mode 100644 labs/lab.sh create mode 100644 labs/recipes/README.md create mode 100644 labs/recipes/_convert_ops.py create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/.gitignore create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/README.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/agent.json create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/automations/incident-filters/auto-investigate-azmon.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/automations/incident-platforms/azure-monitor.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/automations/incident-platforms/servicenow.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/automations/scheduled-tasks/weekly-cost-report.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/hooks/change-risk-assessor.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/hooks/sql-write-guard.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-blocking-diagnosis.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-blocking-diagnosis.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-blocking-fix.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-blocking-fix.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-performance-fix.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-performance-fix.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-query-diagnosis.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-query-diagnosis.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/subagents/deployment-validator-gh.instructions.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/subagents/deployment-validator-gh.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/subagents/deployment-validator.instructions.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/subagents/deployment-validator.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/subagents/sql-performance-investigator.instructions.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/subagents/sql-performance-investigator.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/tools/AssessChangeRisk.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/connectors.json create mode 100644 labs/recipes/azmon-aca-servicenow-zavacafe-ops/expected-config.json create mode 100644 labs/recipes/azmon-aca-servicenow-zavaitsupport/.gitignore create mode 100644 labs/recipes/azmon-aca-servicenow-zavaitsupport/README.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavaitsupport/agent.json create mode 100644 labs/recipes/azmon-aca-servicenow-zavaitsupport/automations/incident-filters/snow-laptop-replacement.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavaitsupport/automations/incident-platforms/servicenow.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavaitsupport/config/subagents/it-support-handler.instructions.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavaitsupport/config/subagents/it-support-handler.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavaitsupport/config/tools/CheckWarranty.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavaitsupport/config/tools/LookupServiceNowIncident.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavaitsupport/connectors.json create mode 100644 labs/recipes/azmon-aca-servicenow-zavaitsupport/expected-config.json create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/.gitignore create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/README.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/agent.json create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/automations/incident-filters/auto-investigate-azmon.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/automations/incident-platforms/azure-monitor.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/automations/incident-platforms/servicenow.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/automations/scheduled-tasks/pod-fleet-audit-daily.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/config-regression-diagnosis.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/config-regression-diagnosis.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/crash-regression-diagnosis.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/crash-regression-diagnosis.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/deployment-rollback.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/deployment-rollback.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/deployment-validation.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/deployment-validation.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/disk-pressure-diagnosis.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/disk-pressure-diagnosis.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/grid-status-diagnosis.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/grid-status-diagnosis.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/meter-api-diagnosis.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/meter-api-diagnosis.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/notification-svc-diagnosis.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/notification-svc-diagnosis.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/outage-api-diagnosis.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/outage-api-diagnosis.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/perf-regression-diagnosis.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/perf-regression-diagnosis.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/plot-incident-metrics.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/plot-incident-metrics.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/pod-fleet-audit-deck.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/pod-fleet-audit-deck.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/release-on-sre-fix.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/release-on-sre-fix.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/repo-routing.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/repo-routing.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/sre-agent-customizer.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/sre-agent-customizer.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/deployment-validator.instructions.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/deployment-validator.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/incident-handler.instructions.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/incident-handler.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/pipeline-failure-investigator.instructions.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/pipeline-failure-investigator.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/pod-incident-remediator.instructions.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/pod-incident-remediator.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/release-orchestrator.instructions.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/release-orchestrator.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/utility-ops-agent.instructions.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/utility-ops-agent.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/vm-ops-agent.instructions.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/vm-ops-agent.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/web-app-troubleshooter.instructions.md create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/web-app-troubleshooter.yaml create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/connectors.json create mode 100644 labs/recipes/azmon-aca-servicenow-zavapower-ops/expected-config.json create mode 100644 labs/sim.ps1 create mode 100644 labs/sim.sh delete mode 100644 labs/starter-lab/azure.yaml delete mode 100644 labs/vm-cosmosdb/azure.yaml rename labs/{zava-aks-postgres => zava-athletic}/.github/skills/deploying-demo/SKILL.md (100%) rename labs/{zava-aks-postgres => zava-athletic}/.github/skills/managing-sre-agent/SKILL.md (100%) rename labs/{zava-aks-postgres => zava-athletic}/.github/skills/running-demo/SKILL.md (100%) rename labs/{zava-aks-postgres => zava-athletic}/.github/skills/running-demo/scripts/break-db-perf.ps1 (100%) rename labs/{zava-aks-postgres => zava-athletic}/.github/skills/running-demo/scripts/break-network.ps1 (100%) rename labs/{zava-aks-postgres => zava-athletic}/.github/skills/running-demo/scripts/break-sql.ps1 (100%) rename labs/{zava-aks-postgres => zava-athletic}/.github/skills/running-demo/scripts/fix-db-perf.ps1 (100%) rename labs/{zava-aks-postgres => zava-athletic}/.github/skills/running-demo/scripts/fix-network.ps1 (100%) rename labs/{zava-aks-postgres => zava-athletic}/.github/skills/running-demo/scripts/fix-sql.ps1 (100%) rename labs/{zava-aks-postgres => zava-athletic}/.gitignore (100%) rename labs/{zava-aks-postgres => zava-athletic}/AGENTS.md (100%) rename labs/{zava-aks-postgres => zava-athletic}/README.md (83%) rename labs/{zava-aks-postgres => zava-athletic}/azure.yaml (94%) rename labs/{zava-aks-postgres => zava-athletic}/docs/images/storefront-broken.png (100%) rename labs/{zava-aks-postgres => zava-athletic}/docs/images/storefront-healthy.png (100%) rename labs/{zava-aks-postgres => zava-athletic}/infra/main.bicep (99%) rename labs/{zava-aks-postgres => zava-athletic}/infra/main.bicepparam (90%) rename labs/{zava-aks-postgres => zava-athletic}/infra/main.json (99%) rename labs/{zava-aks-postgres => zava-athletic}/infra/modules/acr.bicep (100%) rename labs/{zava-aks-postgres => zava-athletic}/infra/modules/aks.bicep (100%) rename labs/{zava-aks-postgres => zava-athletic}/infra/modules/identity.bicep (100%) rename labs/{zava-aks-postgres => zava-athletic}/infra/modules/monitoring.bicep (100%) rename labs/{zava-aks-postgres => zava-athletic}/infra/modules/pg-admin.bicep (100%) rename labs/{zava-aks-postgres => zava-athletic}/infra/modules/postgresql.bicep (100%) rename labs/{zava-aks-postgres => zava-athletic}/infra/modules/sre-agent.bicep (99%) rename labs/{zava-aks-postgres => zava-athletic}/infra/modules/vnet.bicep (100%) rename labs/{zava-aks-postgres => zava-athletic}/k8s/README.md (100%) rename labs/{zava-aks-postgres => zava-athletic}/k8s/api-deployment.yaml (100%) rename labs/{zava-aks-postgres => zava-athletic}/k8s/api-service.yaml (100%) rename labs/{zava-aks-postgres => zava-athletic}/k8s/configmap.yaml (100%) rename labs/{zava-aks-postgres => zava-athletic}/k8s/ingress.yaml (100%) rename labs/{zava-aks-postgres => zava-athletic}/k8s/jobs/load-categories.yaml (100%) rename labs/{zava-aks-postgres => zava-athletic}/k8s/secret.yaml (100%) rename labs/{zava-aks-postgres => zava-athletic}/k8s/service-account.yaml (100%) rename labs/{zava-aks-postgres => zava-athletic}/k8s/storefront-deployment.yaml (100%) rename labs/{zava-aks-postgres => zava-athletic}/k8s/storefront-service.yaml (100%) create mode 100644 labs/zava-athletic/lab.yaml rename labs/{zava-aks-postgres => zava-athletic}/scripts/_aks-helpers.ps1 (100%) rename labs/{zava-aks-postgres => zava-athletic}/scripts/check-environment.ps1 (100%) rename labs/{zava-aks-postgres => zava-athletic}/scripts/post-provision.ps1 (91%) rename labs/{zava-aks-postgres => zava-athletic}/scripts/setup-sre-agent.ps1 (100%) rename labs/{zava-aks-postgres => zava-athletic}/scripts/watch-agent.ps1 (100%) rename labs/{zava-aks-postgres => zava-athletic}/src/api/.dockerignore (100%) rename labs/{zava-aks-postgres => zava-athletic}/src/api/Dockerfile (100%) rename labs/{zava-aks-postgres => zava-athletic}/src/api/bin/run-sql.js (100%) rename labs/{zava-aks-postgres => zava-athletic}/src/api/db/client.js (100%) rename labs/{zava-aks-postgres => zava-athletic}/src/api/db/seed.js (100%) rename labs/{zava-aks-postgres => zava-athletic}/src/api/logging/logger.js (100%) rename labs/{zava-aks-postgres => zava-athletic}/src/api/package-lock.json (100%) rename labs/{zava-aks-postgres => zava-athletic}/src/api/package.json (100%) rename labs/{zava-aks-postgres => zava-athletic}/src/api/routes/diagnostics.js (100%) rename labs/{zava-aks-postgres => zava-athletic}/src/api/routes/health.js (100%) rename labs/{zava-aks-postgres => zava-athletic}/src/api/routes/orders.js (100%) rename labs/{zava-aks-postgres => zava-athletic}/src/api/routes/products.js (100%) rename labs/{zava-aks-postgres => zava-athletic}/src/api/server.js (100%) rename labs/{zava-aks-postgres => zava-athletic}/src/storefront/.dockerignore (100%) rename labs/{zava-aks-postgres => zava-athletic}/src/storefront/Dockerfile (100%) rename labs/{zava-aks-postgres => zava-athletic}/src/storefront/package-lock.json (100%) rename labs/{zava-aks-postgres => zava-athletic}/src/storefront/package.json (100%) rename labs/{zava-aks-postgres => zava-athletic}/src/storefront/server.js (100%) rename labs/{zava-aks-postgres => zava-athletic}/sre-config/knowledge-base/zava-architecture.md (100%) create mode 100644 labs/zava-cafe/.gitignore create mode 100644 labs/zava-cafe/README.md create mode 100644 labs/zava-cafe/azure.yaml create mode 100644 labs/zava-cafe/dashboard.json create mode 100644 labs/zava-cafe/infra/main.bicep create mode 100644 labs/zava-cafe/infra/main.bicepparam rename labs/{starter-lab => zava-cafe}/infra/modules/identity.bicep (100%) rename labs/{starter-lab => zava-cafe}/infra/modules/monitoring.bicep (100%) rename labs/{starter-lab => zava-cafe}/infra/modules/sre-agent.bicep (100%) rename labs/{starter-lab => zava-cafe}/infra/modules/subscription-rbac.bicep (100%) create mode 100644 labs/zava-cafe/infra/resources.bicep create mode 100644 labs/zava-cafe/infra/seed-database.sql create mode 100644 labs/zava-cafe/lab.yaml create mode 100644 labs/zava-cafe/scripts/invoke-thread.sh create mode 100644 labs/zava-cafe/scripts/post-provision.sh create mode 100644 labs/zava-cafe/scripts/prereqs.sh create mode 100644 labs/zava-cafe/scripts/sql_entra.py create mode 100644 labs/zava-cafe/simulator/demo.py create mode 100644 labs/zava-cafe/simulator/expand_data.py create mode 100644 labs/zava-cafe/simulator/requirements.txt create mode 100644 labs/zava-cafe/src/.gitignore create mode 100644 labs/zava-cafe/src/Program.cs create mode 100644 labs/zava-cafe/src/Properties/launchSettings.json create mode 100644 labs/zava-cafe/src/ZavaCafeApp.csproj create mode 100644 labs/zava-cafe/src/ZavaCafeApp.http create mode 100644 labs/zava-cafe/src/appsettings.Development.json create mode 100644 labs/zava-cafe/src/appsettings.json create mode 100644 labs/zava-cafe/sre-config/agent1/.github/instructions.md create mode 100644 labs/zava-cafe/sre-config/agent1/agents/deployment-validator-gh/deployment-validator-gh.yaml create mode 100644 labs/zava-cafe/sre-config/agent1/agents/deployment-validator/deployment-validator.yaml create mode 100644 labs/zava-cafe/sre-config/agent1/agents/example_agent.yaml create mode 100644 labs/zava-cafe/sre-config/agent1/agents/sql-performance-investigator/sql-performance-investigator.yaml create mode 100644 labs/zava-cafe/sre-config/agent1/hooks/change-risk-assessor.yaml create mode 100644 labs/zava-cafe/sre-config/agent1/hooks/sql-write-guard.yaml create mode 100644 labs/zava-cafe/sre-config/agent1/scheduledtasks/weekly-cost-report/weekly-cost-report.yaml create mode 100644 labs/zava-cafe/sre-config/agent1/skills/sql-blocking-diagnosis/SKILL.md create mode 100644 labs/zava-cafe/sre-config/agent1/skills/sql-blocking-fix/SKILL.md create mode 100644 labs/zava-cafe/sre-config/agent1/skills/sql-performance-fix/SKILL.md create mode 100644 labs/zava-cafe/sre-config/agent1/skills/sql-query-diagnosis/SKILL.md create mode 100644 labs/zava-cafe/sre-config/agent1/tools/AssessChangeRisk/AssessChangeRisk.yaml create mode 100644 labs/zava-cafe/sre-config/agent1/tools/example_tool.yaml create mode 100644 labs/zava-cafe/sre-config/simulate-dtu-spike.ps1 create mode 100644 labs/zava-cafe/sre-config/simulate-slow-queries.ps1 create mode 100644 labs/zava-eats/.github/instructions.md rename labs/{starter-lab => zava-eats}/.gitignore (100%) rename labs/{starter-lab => zava-eats}/README.md (77%) create mode 100644 labs/zava-eats/agents/example_agent.yaml create mode 100644 labs/zava-eats/azure.yaml rename labs/{starter-lab => zava-eats}/docs/architecture.svg (100%) rename labs/{starter-lab => zava-eats}/infra/main.bicep (100%) rename labs/{starter-lab => zava-eats}/infra/main.bicepparam (100%) rename labs/{starter-lab => zava-eats}/infra/modules/alert-rules.bicep (100%) rename labs/{starter-lab => zava-eats}/infra/modules/container-app.bicep (100%) create mode 100644 labs/zava-eats/infra/modules/identity.bicep create mode 100644 labs/zava-eats/infra/modules/monitoring.bicep create mode 100644 labs/zava-eats/infra/modules/sre-agent.bicep create mode 100644 labs/zava-eats/infra/modules/subscription-rbac.bicep rename labs/{starter-lab => zava-eats}/infra/resources.bicep (100%) rename labs/{starter-lab => zava-eats}/knowledge-base/github-issue-triage.md (100%) rename labs/{starter-lab => zava-eats}/knowledge-base/grubify-architecture.md (100%) rename labs/{starter-lab => zava-eats}/knowledge-base/http-500-errors.md (100%) rename labs/{starter-lab => zava-eats}/knowledge-base/incident-report-template.md (100%) create mode 100644 labs/zava-eats/lab.yaml rename labs/{starter-lab => zava-eats}/lab/skillable-instructions.md (99%) rename labs/{starter-lab => zava-eats}/scripts/break-app.sh (100%) mode change 100755 => 100644 rename labs/{starter-lab => zava-eats}/scripts/create-sample-issues.sh (100%) mode change 100755 => 100644 rename labs/{starter-lab => zava-eats}/scripts/generate-pptx.py (100%) create mode 100644 labs/zava-eats/scripts/invoke-thread.sh rename labs/{starter-lab => zava-eats}/scripts/post-provision-srectl.sh (100%) mode change 100755 => 100644 rename labs/{starter-lab => zava-eats}/scripts/post-provision.sh (87%) mode change 100755 => 100644 rename labs/{starter-lab => zava-eats}/scripts/prereqs.sh (100%) mode change 100755 => 100644 rename labs/{starter-lab => zava-eats}/scripts/setup-github-srectl.sh (100%) mode change 100755 => 100644 rename labs/{starter-lab => zava-eats}/scripts/setup-github.sh (100%) mode change 100755 => 100644 rename labs/{starter-lab => zava-eats}/scripts/setup.sh (100%) rename labs/{starter-lab => zava-eats}/scripts/yaml-to-api-json.py (100%) create mode 100644 labs/zava-eats/sre-config/.github/instructions.md rename labs/{starter-lab => zava-eats}/sre-config/agents/code-analyzer.yaml (100%) create mode 100644 labs/zava-eats/sre-config/agents/example_agent.yaml rename labs/{starter-lab => zava-eats}/sre-config/agents/incident-handler-core.yaml (100%) rename labs/{starter-lab => zava-eats}/sre-config/agents/incident-handler-full.yaml (100%) rename labs/{starter-lab => zava-eats}/sre-config/agents/issue-triager.yaml (100%) rename labs/{starter-lab => zava-eats}/sre-config/connectors/github-oauth.yaml (100%) create mode 100644 labs/zava-eats/sre-config/skills/grubify-diagnosis/SKILL.md create mode 100644 labs/zava-eats/sre-config/tools/CheckGrubifyHealth/CheckGrubifyHealth.yaml create mode 100644 labs/zava-eats/sre-config/tools/example_tool.yaml create mode 100644 labs/zava-eats/tools/example_tool.yaml create mode 100644 labs/zava-infra/README.md create mode 100644 labs/zava-infra/lab.yaml rename labs/{deployment-compliance => zava-infra/scenarios/compliance}/.github/workflows/deploy-container-app.yml (100%) rename labs/{deployment-compliance => zava-infra/scenarios/compliance}/README.md (83%) create mode 100644 labs/zava-infra/scenarios/compliance/azure.yaml rename labs/{deployment-compliance => zava-infra/scenarios/compliance}/docs/architecture.svg (100%) rename labs/{deployment-compliance => zava-infra/scenarios/compliance}/hooks/deployment-compliance-approval.yaml (100%) rename labs/{deployment-compliance => zava-infra/scenarios/compliance}/incident-filters/containerapp-deployment-alert.yaml (100%) rename labs/{deployment-compliance => zava-infra/scenarios/compliance}/infra/main.bicep (100%) rename labs/{deployment-compliance => zava-infra/scenarios/compliance}/infra/main.json (100%) rename labs/{deployment-compliance => zava-infra/scenarios/compliance}/infra/modules/monitoring.bicep (100%) rename labs/{deployment-compliance => zava-infra/scenarios/compliance}/infra/modules/roles.bicep (100%) rename labs/{deployment-compliance => zava-infra/scenarios/compliance}/infra/modules/sre-agent.bicep (100%) rename labs/{deployment-compliance => zava-infra/scenarios/compliance}/infra/modules/workload.bicep (100%) create mode 100644 labs/zava-infra/scenarios/compliance/lab.yaml rename labs/{deployment-compliance => zava-infra/scenarios/compliance}/scheduled-tasks/compliance-scan.yaml (100%) rename labs/{deployment-compliance => zava-infra/scenarios/compliance}/scripts/break-compliance.sh (100%) rename labs/{deployment-compliance => zava-infra/scenarios/compliance}/scripts/deploy.sh (100%) mode change 100755 => 100644 rename labs/{deployment-compliance => zava-infra/scenarios/compliance}/scripts/post-deploy.sh (96%) mode change 100755 => 100644 rename labs/{deployment-compliance => zava-infra/scenarios/compliance}/scripts/prereqs.sh (100%) rename labs/{deployment-compliance => zava-infra/scenarios/compliance}/skills/deployment-compliance-check/SKILL.md (100%) rename labs/{deployment-compliance => zava-infra/scenarios/compliance}/skills/deployment-compliance-check/compliance_detection.md (100%) rename labs/{deployment-compliance => zava-infra/scenarios/compliance}/src/api/Dockerfile (100%) rename labs/{deployment-compliance => zava-infra/scenarios/compliance}/src/api/init.sql (100%) rename labs/{deployment-compliance => zava-infra/scenarios/compliance}/src/api/package.json (100%) rename labs/{deployment-compliance => zava-infra/scenarios/compliance}/src/api/server.js (100%) create mode 100644 labs/zava-infra/scenarios/perf-drift/README.md create mode 100644 labs/zava-infra/scenarios/perf-drift/azure.yaml rename labs/{vm-cosmosdb => zava-infra/scenarios/perf-drift}/docs/architecture.svg (100%) rename labs/{vm-cosmosdb => zava-infra/scenarios/perf-drift}/hooks/vm-remediation-approval.yaml (100%) rename labs/{vm-cosmosdb => zava-infra/scenarios/perf-drift}/infra/main.bicep (100%) rename labs/{vm-cosmosdb => zava-infra/scenarios/perf-drift}/infra/main.bicepparam (100%) rename labs/{vm-cosmosdb => zava-infra/scenarios/perf-drift}/infra/modules/cosmosdb.bicep (100%) rename labs/{vm-cosmosdb => zava-infra/scenarios/perf-drift}/infra/modules/monitoring.bicep (100%) rename labs/{vm-cosmosdb => zava-infra/scenarios/perf-drift}/infra/modules/network.bicep (100%) rename labs/{vm-cosmosdb => zava-infra/scenarios/perf-drift}/infra/modules/roles.bicep (100%) rename labs/{vm-cosmosdb => zava-infra/scenarios/perf-drift}/infra/modules/sre-agent.bicep (100%) rename labs/{vm-cosmosdb => zava-infra/scenarios/perf-drift}/infra/modules/vm.bicep (100%) create mode 100644 labs/zava-infra/scenarios/perf-drift/lab.yaml rename labs/{vm-cosmosdb => zava-infra/scenarios/perf-drift}/scheduled-tasks/compliance-drift-scan.yaml (100%) rename labs/{vm-cosmosdb => zava-infra/scenarios/perf-drift}/scripts/break-db.sh (100%) rename labs/{vm-cosmosdb => zava-infra/scenarios/perf-drift}/scripts/break-vm.sh (88%) rename labs/{vm-cosmosdb => zava-infra/scenarios/perf-drift}/scripts/install-app.sh (100%) rename labs/{vm-cosmosdb => zava-infra/scenarios/perf-drift}/scripts/post-deploy.sh (95%) rename labs/{vm-cosmosdb => zava-infra/scenarios/perf-drift}/skills/compliance-drift-detection.md (100%) rename labs/{vm-cosmosdb => zava-infra/scenarios/perf-drift}/skills/compliance-drift-detection/SKILL.md (100%) rename labs/{vm-cosmosdb => zava-infra/scenarios/perf-drift}/skills/vm-performance-diagnostics.md (100%) rename labs/{vm-cosmosdb => zava-infra/scenarios/perf-drift}/skills/vm-performance-diagnostics/SKILL.md (100%) rename labs/{vm-cosmosdb => zava-infra/scenarios/perf-drift}/src/app.js (100%) rename labs/{vm-cosmosdb => zava-infra/scenarios/perf-drift}/src/package.json (100%) rename labs/{vm-cosmosdb => zava-infra/scenarios/perf-drift}/src/setup.sh (100%) rename labs/{terraform-drift-detection => zava-infra/scenarios/tf-drift}/README.md (70%) rename labs/{terraform-drift-detection => zava-infra/scenarios/tf-drift}/app/package.json (100%) rename labs/{terraform-drift-detection => zava-infra/scenarios/tf-drift}/app/server.js (100%) create mode 100644 labs/zava-infra/scenarios/tf-drift/lab.yaml rename labs/{terraform-drift-detection => zava-infra/scenarios/tf-drift}/scripts/deploy-app.ps1 (100%) rename labs/{terraform-drift-detection => zava-infra/scenarios/tf-drift}/scripts/generate-load.ps1 (100%) rename labs/{terraform-drift-detection => zava-infra/scenarios/tf-drift}/scripts/induce-drift.ps1 (100%) rename labs/{terraform-drift-detection => zava-infra/scenarios/tf-drift}/scripts/revert-drift.ps1 (100%) rename labs/{terraform-drift-detection => zava-infra/scenarios/tf-drift}/scripts/simulate-tfc-notification.ps1 (100%) rename labs/{terraform-drift-detection => zava-infra/scenarios/tf-drift}/skills/terraform-drift-analysis.md (100%) rename labs/{terraform-drift-detection => zava-infra/scenarios/tf-drift}/terraform/logic-app.tf (100%) rename labs/{terraform-drift-detection => zava-infra/scenarios/tf-drift}/terraform/main.tf (100%) rename labs/{terraform-drift-detection => zava-infra/scenarios/tf-drift}/terraform/outputs.tf (100%) rename labs/{terraform-drift-detection => zava-infra/scenarios/tf-drift}/terraform/providers.tf (100%) rename labs/{terraform-drift-detection => zava-infra/scenarios/tf-drift}/terraform/terraform.tfvars.example (100%) rename labs/{terraform-drift-detection => zava-infra/scenarios/tf-drift}/terraform/variables.tf (100%) create mode 100644 labs/zava-itsupport/.gitignore create mode 100644 labs/zava-itsupport/README.md create mode 100644 labs/zava-itsupport/azure.yaml create mode 100644 labs/zava-itsupport/infra/abbreviations.json create mode 100644 labs/zava-itsupport/infra/main.bicep create mode 100644 labs/zava-itsupport/infra/main.bicepparam create mode 100644 labs/zava-itsupport/infra/modules/identity.bicep create mode 100644 labs/zava-itsupport/infra/modules/monitoring.bicep create mode 100644 labs/zava-itsupport/infra/modules/sre-agent.bicep create mode 100644 labs/zava-itsupport/infra/modules/subscription-rbac.bicep create mode 100644 labs/zava-itsupport/infra/resources.bicep create mode 100644 labs/zava-itsupport/lab.yaml create mode 100644 labs/zava-itsupport/laptop-request-site/.dockerignore create mode 100644 labs/zava-itsupport/laptop-request-site/Dockerfile create mode 100644 labs/zava-itsupport/laptop-request-site/index.html create mode 100644 labs/zava-itsupport/laptop-request-site/package.json create mode 100644 labs/zava-itsupport/laptop-request-site/server.js create mode 100644 labs/zava-itsupport/laptop-request-site/style.css create mode 100644 labs/zava-itsupport/scripts/laptop-request-demo.sh create mode 100644 labs/zava-itsupport/scripts/post-provision.sh create mode 100644 labs/zava-itsupport/sre-config/.github/instructions.md create mode 100644 labs/zava-itsupport/sre-config/agents/example_agent.yaml create mode 100644 labs/zava-itsupport/sre-config/agents/it-support-handler/it-support-handler.yaml create mode 100644 labs/zava-itsupport/sre-config/tools/CheckWarranty/CheckWarranty.yaml create mode 100644 labs/zava-itsupport/sre-config/tools/LookupServiceNowIncident/LookupServiceNowIncident.yaml create mode 100644 labs/zava-itsupport/sre-config/tools/example_tool.yaml create mode 100644 labs/zava-itsupport/warranty-tool/.dockerignore create mode 100644 labs/zava-itsupport/warranty-tool/Dockerfile create mode 100644 labs/zava-itsupport/warranty-tool/app.py create mode 100644 labs/zava-itsupport/warranty-tool/check_warranty.py create mode 100644 labs/zava-itsupport/warranty-tool/requirements.txt create mode 100644 labs/zava-itsupport/warranty-tool/startup.sh create mode 100644 labs/zava-power/.gitignore create mode 100644 labs/zava-power/.rendered/sre-config/agents/deployment-validator/deployment-validator.yaml create mode 100644 labs/zava-power/.rendered/sre-config/agents/incident-handler/incident-handler.yaml create mode 100644 labs/zava-power/.rendered/sre-config/agents/pipeline-failure-investigator/pipeline-failure-investigator.yaml create mode 100644 labs/zava-power/.rendered/sre-config/agents/pod-incident-remediator/pod-incident-remediator.yaml create mode 100644 labs/zava-power/.rendered/sre-config/agents/release-orchestrator/release-orchestrator.yaml create mode 100644 labs/zava-power/.rendered/sre-config/agents/utility-ops-agent/utility-ops-agent.yaml create mode 100644 labs/zava-power/.rendered/sre-config/agents/vm-ops-agent/vm-ops-agent.yaml create mode 100644 labs/zava-power/.rendered/sre-config/agents/web-app-troubleshooter/web-app-troubleshooter.yaml create mode 100644 labs/zava-power/.rendered/sre-config/connectors/datadog-mcp.yaml create mode 100644 labs/zava-power/.rendered/sre-config/connectors/dynatrace-mcp.yaml create mode 100644 labs/zava-power/.rendered/sre-config/connectors/servicenow-mcp.yaml create mode 100644 labs/zava-power/.rendered/sre-config/response-plans/auto-investigate.yaml create mode 100644 labs/zava-power/.rendered/sre-config/scheduled-tasks/pod-fleet-audit.yaml create mode 100644 labs/zava-power/.rendered/sre-config/skills/SRE Agent customizer/SKILL.md create mode 100644 labs/zava-power/.rendered/sre-config/skills/config-regression-diagnosis/SKILL.md create mode 100644 labs/zava-power/.rendered/sre-config/skills/crash-regression-diagnosis/SKILL.md create mode 100644 labs/zava-power/.rendered/sre-config/skills/deployment-rollback/SKILL.md create mode 100644 labs/zava-power/.rendered/sre-config/skills/deployment-validation/SKILL.md create mode 100644 labs/zava-power/.rendered/sre-config/skills/disk-pressure-diagnosis/SKILL.md create mode 100644 labs/zava-power/.rendered/sre-config/skills/grid-status-diagnosis/SKILL.md create mode 100644 labs/zava-power/.rendered/sre-config/skills/meter-api-diagnosis/SKILL.md create mode 100644 labs/zava-power/.rendered/sre-config/skills/notification-svc-diagnosis/SKILL.md create mode 100644 labs/zava-power/.rendered/sre-config/skills/outage-api-diagnosis/SKILL.md create mode 100644 labs/zava-power/.rendered/sre-config/skills/perf-regression-diagnosis/SKILL.md create mode 100644 labs/zava-power/.rendered/sre-config/skills/plot-incident-metrics/SKILL.md create mode 100644 labs/zava-power/.rendered/sre-config/skills/pod-fleet-audit-deck/SKILL.md create mode 100644 labs/zava-power/.rendered/sre-config/skills/release-on-sre-fix/SKILL.md create mode 100644 labs/zava-power/.rendered/sre-config/skills/repo-routing/SKILL.md create mode 100644 labs/zava-power/.rendered/sre-config/skills/servicenow-incident-mgmt/SKILL.md create mode 100644 labs/zava-power/.rendered/sre-config/tools/BurstLoadTest/BurstLoadTest.yaml create mode 100644 labs/zava-power/.rendered/sre-config/tools/CreateServiceNowIncident/CreateServiceNowIncident.yaml create mode 100644 labs/zava-power/.rendered/sre-config/tools/GetActiveRevision/GetActiveRevision.yaml create mode 100644 labs/zava-power/.rendered/sre-config/tools/LookupServiceNowIncident/LookupServiceNowIncident.yaml create mode 100644 labs/zava-power/.rendered/sre-config/tools/ProbeServiceLatency/ProbeServiceLatency.yaml create mode 100644 labs/zava-power/.rendered/sre-config/tools/README.md create mode 100644 labs/zava-power/.rendered/sre-config/tools/RemediateContainerApp/RemediateContainerApp.yaml create mode 100644 labs/zava-power/.rendered/sre-config/tools/ResolveServiceNowIncident/ResolveServiceNowIncident.yaml create mode 100644 labs/zava-power/.rendered/sre-config/tools/RollbackContainerAppRevision/RollbackContainerAppRevision.yaml create mode 100644 labs/zava-power/.rendered/sre-config/tools/UpdateServiceNowWorkNotes/UpdateServiceNowWorkNotes.yaml create mode 100644 labs/zava-power/.rendered/sre-config/tools/UploadChartToServiceNow.yaml create mode 100644 labs/zava-power/AGENTS.md create mode 100644 labs/zava-power/README.md create mode 100644 labs/zava-power/azure.yaml create mode 100644 labs/zava-power/bugs/build-failure/app.py create mode 100644 labs/zava-power/bugs/build-failure/requirements.txt create mode 100644 labs/zava-power/bugs/config/main.go create mode 100644 labs/zava-power/bugs/crash/app.py create mode 100644 labs/zava-power/bugs/perf/server.js create mode 100644 labs/zava-power/infra/bicepconfig.json create mode 100644 labs/zava-power/infra/main.bicep create mode 100644 labs/zava-power/infra/main.bicepparam create mode 100644 labs/zava-power/infra/modules/aks.bicep create mode 100644 labs/zava-power/infra/modules/alerts.bicep create mode 100644 labs/zava-power/infra/modules/arc-vm.bicep create mode 100644 labs/zava-power/infra/modules/container-apps.bicep create mode 100644 labs/zava-power/infra/modules/container-registry.bicep create mode 100644 labs/zava-power/infra/modules/observability.bicep create mode 100644 labs/zava-power/infra/modules/sre-agent.bicep create mode 100644 labs/zava-power/infra/modules/sre-identity.bicep create mode 100644 labs/zava-power/infra/roles/powergrid-sre-agent-operator.json create mode 100644 labs/zava-power/knowledge-base/deployment-rollback-runbook.md create mode 100644 labs/zava-power/knowledge-base/grid-status-runbook.md create mode 100644 labs/zava-power/knowledge-base/incident-report-template.md create mode 100644 labs/zava-power/knowledge-base/meter-api-runbook.md create mode 100644 labs/zava-power/knowledge-base/notification-svc-runbook.md create mode 100644 labs/zava-power/knowledge-base/outage-api-runbook.md create mode 100644 labs/zava-power/knowledge-base/powergrid-architecture.md create mode 100644 labs/zava-power/knowledge-base/sre-agent-architecture.md create mode 100644 labs/zava-power/lab.yaml create mode 100644 labs/zava-power/pipelines/build.yml create mode 100644 labs/zava-power/pipelines/release.yml create mode 100644 labs/zava-power/scripts/check-environment.ps1 create mode 100644 labs/zava-power/scripts/post-provision.ps1 create mode 100644 labs/zava-power/scripts/render-config.py create mode 100644 labs/zava-power/simulator/demo.py create mode 100644 labs/zava-power/simulator/requirements.txt create mode 100644 labs/zava-power/src/grid-status-api/.dockerignore create mode 100644 labs/zava-power/src/grid-status-api/Dockerfile create mode 100644 labs/zava-power/src/grid-status-api/package-lock.json create mode 100644 labs/zava-power/src/grid-status-api/package.json create mode 100644 labs/zava-power/src/grid-status-api/server.js create mode 100644 labs/zava-power/src/meter-api/Dockerfile create mode 100644 labs/zava-power/src/meter-api/MeterApi.csproj create mode 100644 labs/zava-power/src/meter-api/Program.cs create mode 100644 labs/zava-power/src/meter-api/Properties/launchSettings.json create mode 100644 labs/zava-power/src/meter-api/bin/Debug/net8.0/MeterApi.deps.json create mode 100644 labs/zava-power/src/meter-api/bin/Debug/net8.0/MeterApi.dll create mode 100644 labs/zava-power/src/meter-api/bin/Debug/net8.0/MeterApi.exe create mode 100644 labs/zava-power/src/meter-api/bin/Debug/net8.0/MeterApi.pdb create mode 100644 labs/zava-power/src/meter-api/bin/Debug/net8.0/MeterApi.runtimeconfig.json create mode 100644 labs/zava-power/src/meter-api/bin/Debug/net8.0/MeterApi.staticwebassets.endpoints.json create mode 100644 labs/zava-power/src/meter-api/obj/Debug/net8.0/.NETCoreApp,Version=v8.0.AssemblyAttributes.cs create mode 100644 labs/zava-power/src/meter-api/obj/Debug/net8.0/MeterApi.AssemblyInfo.cs create mode 100644 labs/zava-power/src/meter-api/obj/Debug/net8.0/MeterApi.AssemblyInfoInputs.cache create mode 100644 labs/zava-power/src/meter-api/obj/Debug/net8.0/MeterApi.GeneratedMSBuildEditorConfig.editorconfig create mode 100644 labs/zava-power/src/meter-api/obj/Debug/net8.0/MeterApi.GlobalUsings.g.cs create mode 100644 labs/zava-power/src/meter-api/obj/Debug/net8.0/MeterApi.MvcApplicationPartsAssemblyInfo.cache create mode 100644 labs/zava-power/src/meter-api/obj/Debug/net8.0/MeterApi.assets.cache create mode 100644 labs/zava-power/src/meter-api/obj/Debug/net8.0/MeterApi.csproj.CoreCompileInputs.cache create mode 100644 labs/zava-power/src/meter-api/obj/Debug/net8.0/MeterApi.csproj.FileListAbsolute.txt create mode 100644 labs/zava-power/src/meter-api/obj/Debug/net8.0/MeterApi.dll create mode 100644 labs/zava-power/src/meter-api/obj/Debug/net8.0/MeterApi.genruntimeconfig.cache create mode 100644 labs/zava-power/src/meter-api/obj/Debug/net8.0/MeterApi.pdb create mode 100644 labs/zava-power/src/meter-api/obj/Debug/net8.0/apphost.exe create mode 100644 labs/zava-power/src/meter-api/obj/Debug/net8.0/ref/MeterApi.dll create mode 100644 labs/zava-power/src/meter-api/obj/Debug/net8.0/refint/MeterApi.dll create mode 100644 labs/zava-power/src/meter-api/obj/Debug/net8.0/rjsmcshtml.dswa.cache.json create mode 100644 labs/zava-power/src/meter-api/obj/Debug/net8.0/rjsmrazor.dswa.cache.json create mode 100644 labs/zava-power/src/meter-api/obj/Debug/net8.0/staticwebassets.build.endpoints.json create mode 100644 labs/zava-power/src/meter-api/obj/Debug/net8.0/staticwebassets.build.json create mode 100644 labs/zava-power/src/meter-api/obj/Debug/net8.0/staticwebassets.build.json.cache create mode 100644 labs/zava-power/src/meter-api/obj/Debug/net8.0/swae.build.ex.cache create mode 100644 labs/zava-power/src/meter-api/obj/MeterApi.csproj.nuget.dgspec.json create mode 100644 labs/zava-power/src/meter-api/obj/MeterApi.csproj.nuget.g.props create mode 100644 labs/zava-power/src/meter-api/obj/MeterApi.csproj.nuget.g.targets create mode 100644 labs/zava-power/src/meter-api/obj/project.assets.json create mode 100644 labs/zava-power/src/meter-api/obj/project.nuget.cache create mode 100644 labs/zava-power/src/notification-svc/Dockerfile create mode 100644 labs/zava-power/src/notification-svc/go.mod create mode 100644 labs/zava-power/src/notification-svc/main.go create mode 100644 labs/zava-power/src/outage-api/Dockerfile create mode 100644 labs/zava-power/src/outage-api/app.py create mode 100644 labs/zava-power/src/outage-api/requirements.txt create mode 100644 labs/zava-power/src/portal-web/.dockerignore create mode 100644 labs/zava-power/src/portal-web/Dockerfile create mode 100644 labs/zava-power/src/portal-web/index.html create mode 100644 labs/zava-power/src/portal-web/nginx.conf create mode 100644 labs/zava-power/src/portal-web/package-lock.json create mode 100644 labs/zava-power/src/portal-web/package.json create mode 100644 labs/zava-power/src/portal-web/src/App.css create mode 100644 labs/zava-power/src/portal-web/src/App.jsx create mode 100644 labs/zava-power/src/portal-web/src/components/StatusBanner.jsx create mode 100644 labs/zava-power/src/portal-web/src/main.jsx create mode 100644 labs/zava-power/src/portal-web/src/pages/Billing.jsx create mode 100644 labs/zava-power/src/portal-web/src/pages/Dashboard.jsx create mode 100644 labs/zava-power/src/portal-web/src/pages/Outages.jsx create mode 100644 labs/zava-power/src/portal-web/src/pages/Usage.jsx create mode 100644 labs/zava-power/src/portal-web/vite.config.js create mode 100644 labs/zava-power/sre-config/agents/deployment-validator/deployment-validator.yaml create mode 100644 labs/zava-power/sre-config/agents/incident-handler/incident-handler.yaml create mode 100644 labs/zava-power/sre-config/agents/pipeline-failure-investigator/pipeline-failure-investigator.yaml create mode 100644 labs/zava-power/sre-config/agents/pod-incident-remediator/pod-incident-remediator.yaml create mode 100644 labs/zava-power/sre-config/agents/release-orchestrator/release-orchestrator.yaml create mode 100644 labs/zava-power/sre-config/agents/utility-ops-agent/utility-ops-agent.yaml create mode 100644 labs/zava-power/sre-config/agents/vm-ops-agent/vm-ops-agent.yaml create mode 100644 labs/zava-power/sre-config/agents/web-app-troubleshooter/web-app-troubleshooter.yaml create mode 100644 labs/zava-power/sre-config/connectors/datadog-mcp.yaml create mode 100644 labs/zava-power/sre-config/connectors/dynatrace-mcp.yaml create mode 100644 labs/zava-power/sre-config/connectors/servicenow-mcp.yaml create mode 100644 labs/zava-power/sre-config/response-plans/auto-investigate.yaml create mode 100644 labs/zava-power/sre-config/scheduled-tasks/pod-fleet-audit.yaml create mode 100644 labs/zava-power/sre-config/skills/SRE Agent customizer/SKILL.md create mode 100644 labs/zava-power/sre-config/skills/config-regression-diagnosis/SKILL.md create mode 100644 labs/zava-power/sre-config/skills/crash-regression-diagnosis/SKILL.md create mode 100644 labs/zava-power/sre-config/skills/deployment-rollback/SKILL.md create mode 100644 labs/zava-power/sre-config/skills/deployment-validation/SKILL.md create mode 100644 labs/zava-power/sre-config/skills/disk-pressure-diagnosis/SKILL.md create mode 100644 labs/zava-power/sre-config/skills/grid-status-diagnosis/SKILL.md create mode 100644 labs/zava-power/sre-config/skills/meter-api-diagnosis/SKILL.md create mode 100644 labs/zava-power/sre-config/skills/notification-svc-diagnosis/SKILL.md create mode 100644 labs/zava-power/sre-config/skills/outage-api-diagnosis/SKILL.md create mode 100644 labs/zava-power/sre-config/skills/perf-regression-diagnosis/SKILL.md create mode 100644 labs/zava-power/sre-config/skills/plot-incident-metrics/SKILL.md create mode 100644 labs/zava-power/sre-config/skills/pod-fleet-audit-deck/SKILL.md create mode 100644 labs/zava-power/sre-config/skills/release-on-sre-fix/SKILL.md create mode 100644 labs/zava-power/sre-config/skills/repo-routing/SKILL.md create mode 100644 labs/zava-power/sre-config/skills/servicenow-incident-mgmt/SKILL.md create mode 100644 labs/zava-power/sre-config/tools/BurstLoadTest/BurstLoadTest.yaml create mode 100644 labs/zava-power/sre-config/tools/CreateServiceNowIncident/CreateServiceNowIncident.yaml create mode 100644 labs/zava-power/sre-config/tools/GetActiveRevision/GetActiveRevision.yaml create mode 100644 labs/zava-power/sre-config/tools/LookupServiceNowIncident/LookupServiceNowIncident.yaml create mode 100644 labs/zava-power/sre-config/tools/ProbeServiceLatency/ProbeServiceLatency.yaml create mode 100644 labs/zava-power/sre-config/tools/README.md create mode 100644 labs/zava-power/sre-config/tools/RemediateContainerApp/RemediateContainerApp.yaml create mode 100644 labs/zava-power/sre-config/tools/ResolveServiceNowIncident/ResolveServiceNowIncident.yaml create mode 100644 labs/zava-power/sre-config/tools/RollbackContainerAppRevision/RollbackContainerAppRevision.yaml create mode 100644 labs/zava-power/sre-config/tools/UpdateServiceNowWorkNotes/UpdateServiceNowWorkNotes.yaml create mode 100644 labs/zava-power/sre-config/tools/UploadChartToServiceNow.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/.gitignore create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/README.md create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/agent.json create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/automations/incident-filters/powergrid-sev01.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/automations/incident-platforms/azure-monitor.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/automations/incident-platforms/servicenow.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/automations/scheduled-tasks/pod-fleet-audit.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/common-prompts/safety-rules.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/hooks/deny-prod-deletes.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/hooks/require-approval-for-restarts.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/config-regression-diagnosis.md create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/config-regression-diagnosis.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/crash-regression-diagnosis.md create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/crash-regression-diagnosis.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/deployment-rollback.md create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/deployment-rollback.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/deployment-validation.md create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/deployment-validation.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/disk-pressure-diagnosis.md create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/disk-pressure-diagnosis.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/grid-status-diagnosis.md create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/grid-status-diagnosis.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/meter-api-diagnosis.md create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/meter-api-diagnosis.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/notification-svc-diagnosis.md create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/notification-svc-diagnosis.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/outage-api-diagnosis.md create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/outage-api-diagnosis.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/perf-regression-diagnosis.md create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/perf-regression-diagnosis.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/plot-incident-metrics.md create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/plot-incident-metrics.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/pod-fleet-audit-deck.md create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/pod-fleet-audit-deck.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/release-on-sre-fix.md create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/release-on-sre-fix.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/repo-routing.md create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/repo-routing.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/servicenow-incident-mgmt.md create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/skills/servicenow-incident-mgmt.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/subagents/deployment-validator.instructions.md create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/subagents/deployment-validator.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/subagents/incident-handler.instructions.md create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/subagents/incident-handler.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/subagents/pipeline-failure-investigator.instructions.md create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/subagents/pipeline-failure-investigator.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/subagents/pod-incident-remediator.instructions.md create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/subagents/pod-incident-remediator.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/subagents/release-orchestrator.instructions.md create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/subagents/release-orchestrator.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/subagents/utility-ops-agent.instructions.md create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/subagents/utility-ops-agent.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/subagents/vm-ops-agent.instructions.md create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/subagents/vm-ops-agent.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/subagents/web-app-troubleshooter.instructions.md create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/config/subagents/web-app-troubleshooter.yaml create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/connectors.json create mode 100644 sreagent-templates/recipes/azmon-aca-servicenow-powergrid-ops/expected-config.json diff --git a/.gitignore b/.gitignore index 300f7ca36..8ac19f6fb 100644 --- a/.gitignore +++ b/.gitignore @@ -9,3 +9,8 @@ Thumbs.db Desktop.ini samples/deployment-compliance/skills/.DS_Store + +# Lab launcher state +labs/**/.deployed/ +labs/**/*.legacy + diff --git a/labs/AGENTS.md b/labs/AGENTS.md new file mode 100644 index 000000000..a1ac1f23e --- /dev/null +++ b/labs/AGENTS.md @@ -0,0 +1,127 @@ +# AGENTS.md — Authoring a Zava Unlimited lab + +> Audience: AI assistants (Copilot CLI, Claude Code, Cursor, VS Code Copilot, +> GitHub Copilot Workspace) helping a human contributor add a new lab to the +> Zava Unlimited SRE Agent demo platform. + +This file is the universal contract. Read it end-to-end before generating any +files. The human contributor will paste a prompt like: + +> "Help me add a new lab for Azure SQL connection-pool exhaustion." + +Your job: interview them, then scaffold a working lab they can `azd up`. + +## Platform shape + +Every lab lives in `labs//` and provides: + +| File | Required | Purpose | +|---|---|---| +| `lab.yaml` | ✅ | Manifest — see `_platform/schema/lab.schema.json` | +| `azure.yaml` | ✅ | azd entrypoint with pre/postprovision hooks | +| `infra/main.bicep` | ✅ | Subscription-scoped bicep that creates RG + resources | +| `scripts/check-environment.ps1` | ✅ | preprovision: prereq + prompt collection (reads `lab.yaml`) | +| `scripts/post-provision.ps1` | ✅ | postprovision: image build, srectl apply, write `.deployed/.json`, optionally launch sim | +| `scripts/scenarios/*.ps1` (or `.py`/`.sh`) | ⚪ | One file per break/fix scenario declared in `lab.yaml` | +| `simulator/` | ⚪ | Lab's own rich sim UI (the meta-sim's `sim.command` points here) | +| `README.md` | ✅ | First non-blank, non-heading line is the launcher's description | + +## Contract: lab.yaml + +Read the full schema at `_platform/schema/lab.schema.json`. Annotated example +at `_platform/schema/lab.example.yaml`. Key points: + +- `name` must equal the directory name (kebab-case) +- `prereqs` lists CLI tools that must be on PATH (e.g. `az`, `azd`, `srectl`) +- `prompts` declares values the launcher collects interactively and stashes in + azd env (use SCREAMING_SNAKE for `name`) +- `scenarios[].runner` is a path relative to the lab root; the meta-sim shells + out to it. `.ps1` runs in pwsh, `.py` in python, `.sh` in bash. +- `sim.command` + `args` is how the meta-sim launches the lab's own rich UI + +## Contract: post-provision.ps1 + +The one **mandatory** thing this script must do at the end of a successful +deploy is write `.deployed/.json` with at minimum: + +```json +{ + "name": "", + "deployedAt": "", + "subscriptionId": "", + "resourceGroup": "", + "region": "" +} +``` + +Add any extra fields the lab's sim or scenarios need (e.g. `sreAgentName`, +`portalUrl`, `containerRegistryName`). The meta-sim and scenario runners read +this file to know what's deployed and where. + +If `$env:LAB_NO_AUTOLAUNCH` is set, do NOT launch the sim at the end (the +multi-lab launcher sets this to avoid blocking). + +## Authoring flow — what to ask the contributor + +Don't ask everything at once. Ask in this order: + +1. **What does this lab demonstrate?** (one sentence) +2. **Lab name** (kebab-case, e.g. `zava-fintech`) and **subsidiary** (e.g. + `Zava Fintech` for the displayName/branding) +3. **What Azure compute?** (AKS, ACA, VM, App Service, Functions, …) — this + shapes `infra/main.bicep` +4. **What sample app?** (existing image? new code in `src/`? none / infra-only?) +5. **What integrations?** (ServiceNow, GitHub, Datadog, …) — each adds a + prompt and likely a connector + secret +6. **What scenarios?** Get 3-8 break/fix scenarios. For each: + - id (kebab-case) + - what breaks + - what the agent should do + - approximate runtime +7. **Any non-Azure prereqs?** (`docker`, `srectl`, `kubectl`, `helm`, …) + +Then: + +1. Run `pwsh ./labs/lab.ps1 -New ` — this drops the skeleton +2. Edit `lab.yaml` to fill in prereqs, prompts, scenarios collected above +3. Edit `infra/main.bicep` for the resources implied by step 3-5 +4. For each scenario, create `scripts/scenarios/.ps1` from the example + template (it shows the polling-for-thread-URL pattern) +5. Validate: `python _platform/helpers/manifest.py validate /lab.yaml` +6. Test discovery: `pwsh ./labs/lab.ps1 -List` should show the new lab +7. (Optional) Deploy: `./lab.sh -Labs ` + +## Reference labs to mimic + +- `zava-power/` — full ACA + ServiceNow + 8 scenarios. Best reference for + complex labs with rich integrations. +- `zava-athletic/` — simpler AKS+Postgres lab with 3 scenarios. Best + reference for single-domain labs. + +## Hard rules — do not violate + +- **Never modify `_platform/`** without an explicit ask from the human. That's + the platform itself; lab-author flow only adds new lab dirs. +- **Never overwrite an existing lab dir.** If `` exists, ask the + human to pick a different name or explicitly confirm they want to delete. +- **Never commit secrets.** Prompts with `secret: true` go to azd env at + deploy time; they must NOT be hardcoded into bicep, scripts, or yaml. +- **Bicep must be subscription-scoped** (`targetScope = 'subscription'`) and + create its own RG. azd assumes this. +- **`.deployed/` is gitignored.** Don't reference it from code that runs + before deploy. It only exists post-provision. + +## When you finish + +Tell the human: + +``` +Lab '' scaffolded. Next steps: +1. Review labs//lab.yaml and labs//infra/main.bicep +2. Implement scenario runners in labs//scripts/scenarios/ +3. Validate: python labs/_platform/helpers/manifest.py validate labs//lab.yaml +4. Deploy: cd labs && ./lab.sh -Labs +``` + +If you're a Copilot CLI user, the `lab-author` skill (in `.github/extensions/`) +wraps this whole flow with adaptive Q&A — use it instead of doing this manually. diff --git a/labs/LAUNCHER.md b/labs/LAUNCHER.md new file mode 100644 index 000000000..1983acc89 --- /dev/null +++ b/labs/LAUNCHER.md @@ -0,0 +1,118 @@ +# Zava Unlimited + +A growing collection of Azure SRE Agent demo labs, all deployable through one +launcher and breakable through one meta-simulator. + +> Zava is a fictional retail conglomerate. Each subsidiary (Zava Power, Zava +> Athletic, Zava Cafe, Zava Eats, Zava IT Support, Zava Infra) is a +> self-contained lab that demonstrates a different Azure workload + SRE Agent +> autonomy story. + +## TL;DR + +```bash +./lab.sh # POSIX — pick one or more labs to deploy +pwsh ./lab.ps1 # Windows / cross-platform +./sim.sh # POSIX — pick a deployed lab + scenario to break/fix +pwsh ./sim.ps1 +``` + +## Two top-level commands + +| Command | Purpose | +|---|---| +| `lab.ps1` / `lab.sh` | Discover labs, prompt for inputs, run `azd up` | +| `sim.ps1` / `sim.sh` | Discover **deployed** labs, run break/fix scenarios | + +## Currently shipping + +| Lab | Subsidiary | Workload | +|---|---|---| +| `zava-power/` | Zava Power | ACA + ServiceNow utility ops, 8 scenarios | +| `zava-athletic/` | Zava Athletic | AKS + PostgreSQL e-commerce, 3 scenarios | +| `zava-cafe/` | Zava Cafe | App Service + Azure SQL specialty-coffee e-commerce | +| `zava-eats/` | Zava Eats | Starter lab — Grubify food-ordering sample, first break/fix | +| `zava-itsupport/` | Zava IT Support | ACA — IT helpdesk + ServiceNow MCP | +| `zava-infra/` | Zava Infra | 3 scenarios — tf-drift, perf-drift, compliance | + +Run `pwsh ./lab.ps1 -List` for the live list. + +## Authoring a new lab + +Two paths: + +1. **Conversational, in Copilot CLI:** install the `lab-author` skill (under + `.copilot/extensions/lab-author/`) and just say "Help me add a new lab to + Zava Unlimited". The skill will interview you and call the scaffolder. +2. **Manual / any AI assistant:** read `AGENTS.md` for the contract, then run + `pwsh ./lab.ps1 -New ` for the skeleton. + +The platform is in `_platform/` — schema, helpers, template. Don't modify it +when adding a lab; just drop a new sibling directory. + +## Multi-lab launcher (`lab.ps1`) + +Interactive picker by default: + +``` +Which lab(s) do you want to deploy? + [1] zava-power ACA + ServiceNow utility-platform demo with 8 break/fix scenarios. + [2] zava-athletic AI-first AKS + PostgreSQL e-commerce demo with 3 break/fix scenarios. + [3] zava-eats Starter lab — Grubify food-ordering sample, first break/fix. + [a] all + [q] quit +``` + +Pick one, several (comma-separated), or `a` for all. Each lab gets its own +azd environment so they coexist cleanly. + +### Non-interactive + +```bash +./lab.sh -Labs zava-power # deploy one +./lab.sh -Labs zava-power,zava-athletic # deploy multiple +./lab.sh -List # list available labs +./lab.sh -Down zava-power # tear down +./lab.sh -New my-new-lab # scaffold a new lab +``` + +### Behavior + +- **Single-lab deploy** auto-launches the simulator at the end of postprovision. +- **Multi-lab deploy** sets `LAB_NO_AUTOLAUNCH=1` so postprovision finishes + cleanly; launch sims manually after via `./sim.sh -Lab `. +- Deploys run sequentially (azd serializes resource state anyway). + +## Meta-simulator (`sim.ps1`) + +After one or more labs are deployed (each writes +`.deployed/.json` from its post-provision), `sim` discovers them: + +```bash +./sim.sh # interactive: pick a deployed lab +./sim.sh -List # list deployed labs + their scenarios +./sim.sh -Lab zava-power # run that lab's full sim UI +./sim.sh -Scenario zava-power db-outage # run one scenario directly +``` + +If only one lab is deployed, `sim` enters its UI directly. If multiple are +deployed, you get a picker that includes a unified "scenarios across all +labs" view. + +## Anatomy of a lab + +``` +labs// +├── lab.yaml # manifest (schema in _platform/schema/) +├── azure.yaml # azd entrypoint w/ pre+postprovision hooks +├── infra/main.bicep # subscription-scoped IaC +├── scripts/ +│ ├── check-environment.ps1 # preprovision: prereqs + prompts → azd env +│ ├── post-provision.ps1 # image build, srectl apply, write .deployed/ +│ └── scenarios/.ps1 # one runner per scenario in lab.yaml +├── simulator/ # (optional) lab's own rich UI +└── README.md +``` + +See `AGENTS.md` for the full contract and `_platform/schema/lab.example.yaml` +for an annotated manifest. diff --git a/labs/README.md b/labs/README.md index f3b28da55..5dce80f2c 100644 --- a/labs/README.md +++ b/labs/README.md @@ -1,284 +1,139 @@ -# Azure SRE Agent Hands-On Lab +# Azure SRE Agent — Labs -Deploy an Azure SRE Agent connected to a sample application with a single `azd up` command. Watch it diagnose and remediate issues autonomously. +A collection of self-contained, end-to-end **Azure SRE Agent** demo labs. Each lab is a single `azd up` package — Bicep infra + sample app + agent config + break/fix scenarios — built to demo investigation, diagnosis, and autonomous remediation in 20–40 minutes. -**Learn more:** [What is Azure SRE Agent?](https://sre.azure.com/docs/overview) - -## Architecture - -

- Lab Architecture -

- -## Prerequisites - -### Required Tools - -| Tool | macOS | Windows | -|------|-------|---------| -| [Azure CLI](https://learn.microsoft.com/cli/azure/install-azure-cli) 2.60+ | `brew install azure-cli` | `winget install Microsoft.AzureCLI` | -| [Azure Developer CLI](https://learn.microsoft.com/azure/developer/azure-developer-cli/install-azd) 1.9+ | `brew install azd` | `winget install Microsoft.Azd` | -| [Git](https://git-scm.com/) 2.x | `brew install git` | `winget install Git.Git` (includes Git Bash) | -| [Python](https://python.org) 3.10+ | `brew install python3` | `winget install Python.Python.3.12` | - -> **Windows note:** After installing Python, disable the Windows Store app aliases: -> **Settings → Apps → Advanced app settings → App execution aliases** → turn OFF `python.exe` and `python3.exe` - -### Azure Requirements - -- Active Azure subscription -- **Owner** role on the subscription (needed for RBAC role assignments) -- Register the resource provider: - ```bash - az provider register -n Microsoft.App --wait - ``` - -### Optional - -- GitHub account (for code search and issue triage scenarios — uses OAuth sign-in, or a [fine-grained PAT](https://github.com/settings/personal-access-tokens/new) scoped to your fork with `Contents:Read`, `Issues:Read+Write`, `Metadata:Read` for least-privilege access) - -## Quick Start - -### Check prerequisites - -Run the prereqs script to verify everything is installed: - -```bash -# macOS/Linux -bash scripts/prereqs.sh - -# Windows (Git Bash or CMD) -"C:\Program Files\Git\bin\bash.exe" scripts/prereqs.sh -``` - -### macOS / Linux - -```bash -# 1. Clone the repo -git clone https://github.com/dm-chelupati/sre-agent-lab.git -cd sre-agent-lab -git submodule update --init --recursive - -# 2. Sign in to Azure -az login -azd auth login - -# 3. Create environment and deploy -azd env new sre-lab -azd up -# Select your subscription and eastus2 as the region -``` - -### Windows - -```cmd -REM 1. Clone the repo (in CMD or PowerShell) -git clone https://github.com/dm-chelupati/sre-agent-lab.git -cd sre-agent-lab -git submodule update --init --recursive - -REM 2. Sign in to Azure -az login -azd auth login - -REM 3. Create environment and deploy -azd env new sre-lab -azd up - -REM If post-provision fails with 'bash not found' or 'Python not found': -set PATH=%PATH%;C:\Users\%USERNAME%\AppData\Local\Programs\Python\Python312 -"C:\Program Files\Git\bin\bash.exe" scripts/post-provision.sh -``` - -Deployment takes ~8-12 minutes. - -## What Gets Deployed - -### Azure Infrastructure (via Bicep) - -| Resource | Service | Purpose | Docs | -|----------|---------|---------|------| -| SRE Agent | `Microsoft.App/agents` | AI agent for incident investigation | [Overview](https://sre.azure.com/docs/overview) | -| Grubify API | Azure Container Apps | Sample app to monitor | | -| Grubify Frontend | Azure Container Apps | Sample app UI | | -| Log Analytics | `Microsoft.OperationalInsights` | Log storage for KQL queries | [Azure Observability](https://sre.azure.com/docs/capabilities/diagnose-azure-observability) | -| App Insights | `Microsoft.Insights` | Request tracing and exceptions | | -| Alert Rules | `Microsoft.Insights/metricAlerts` | HTTP 5xx and error log alerts | | -| Managed Identity | `Microsoft.ManagedIdentity` | Agent identity for Azure access | [Permissions](https://sre.azure.com/docs/tutorials/agent-config/manage-permissions) | -| Container Registry | `Microsoft.ContainerRegistry` | Grubify container images | | - -### RBAC Roles Assigned - -| Role | Scope | Purpose | -|------|-------|---------| -| SRE Agent Administrator | Agent resource | User can manage agent via data plane APIs | -| Reader | Resource group | Agent can read all resources | -| Monitoring Reader | Resource group | Agent can read metrics and alerts | -| Log Analytics Reader | Log Analytics workspace | Agent can query logs via KQL | - -See: [Manage Permissions](https://sre.azure.com/docs/tutorials/agent-config/manage-permissions) +> **Multiple labs?** See [`LAUNCHER.md`](LAUNCHER.md) — `./lab.sh` picks any combination of labs to deploy. +> +> **Authoring a new lab?** See [`AGENTS.md`](AGENTS.md) and [`_platform/`](_platform/). +> +> **Want just the agent config (no infra)?** See [Recipes](#recipes) below. -### SRE Agent Configuration (via post-provision script) +--- -| Component | Purpose | Docs | -|-----------|---------|------| -| Knowledge Base | HTTP error runbook, app architecture, incident template | [Memory & Knowledge](https://sre.azure.com/docs/concepts/memory) | -| incident-handler subagent | Investigates alerts using logs, metrics, runbooks | [Custom Agents](https://sre.azure.com/docs/concepts/subagents) | -| Response Plan | Routes HTTP 500 alerts to incident-handler | [Response Plans](https://sre.azure.com/docs/capabilities/incident-response-plans) | -| Azure Monitor | Incident platform — alerts flow to the agent | [Incident Platforms](https://sre.azure.com/docs/concepts/incident-platforms) | -| GitHub OAuth connector | Code search and issue management (optional) | [Connectors](https://sre.azure.com/docs/concepts/connectors) | -| code-analyzer subagent | Source code root cause analysis | [Custom Agents](https://sre.azure.com/docs/concepts/subagents) | -| issue-triager subagent | Automated issue triage from runbook | [Custom Agents](https://sre.azure.com/docs/concepts/subagents) | +## Labs at a glance -> **Note on GitHub tools:** GitHub OAuth tools (code search, issue management) are **built-in native tools**, not MCP tools. Once the GitHub OAuth connector is set up, all agents — including subagents — get access to GitHub tools automatically through global settings. No explicit `mcp_tools` assignment is needed in subagent YAML. This is different from MCP connector tools (Datadog, Splunk, etc.) which require explicit `mcp_tools` assignment. -| Scheduled Task | Triage customer issues every 12 hours | [Scheduled Tasks](https://sre.azure.com/docs/capabilities/scheduled-tasks) | -| Code Repo | Agent indexes the Grubify source code | [Deep Context](https://sre.azure.com/docs/concepts/workspace-tools) | +| # | Lab | What it demos | Stack | Compute | Difficulty | +|---|---|---|---|---|---| +| 1 | [`zava-eats`](zava-eats/) | **Starter lab** — break a Node.js food-ordering app, watch the agent diagnose HTTP 5xx and remediate. GitHub OAuth + 3 subagents. | Node.js / Express (Grubify) | Azure Container Apps | ★ | +| 2 | [`zava-cafe`](zava-cafe/) | Azure SQL DTU spikes, missing indexes, blocking chains, post-deploy regression validation with rollback. Includes safety hooks (write-guard + change-risk assessor). | .NET 8 / ASP.NET Core (specialty coffee e-commerce) | Azure App Service + Azure SQL DB | ★★ | +| 3 | [`zava-itsupport`](zava-itsupport/) | IT helpdesk laptop-replacement workflow — ServiceNow ticket → warranty lookup → Browser Operator drives procurement portal. | Node.js 20 portal + Python 3.11 / FastAPI warranty API | Azure Container Apps + ServiceNow MCP | ★★ | +| 4 | [`zava-power`](zava-power/) | **Microservice ops at scale** — utility platform with 5 services, 8 subagents, 15 skills, full incident lifecycle (detect → investigate → remediate → resolve in ServiceNow). | Python/Flask + .NET 8 + Node.js 20 + Go 1.22 + React (5 microservices) | Azure Container Apps (+ optional Arc-VM, AKS) | ★★★ | +| 5 | [`zava-athletic`](zava-athletic/) | **AKS + private Postgres** scenarios: PG stop, NetworkPolicy egress block, missing-index slow-query. Anthropic-backed agent with 8 AzMon alerts. | Node.js / Express e-commerce | AKS (private cluster) + PostgreSQL Flexible Server | ★★★ | +| 6 | [`zava-infra`](zava-infra/) | **Infrastructure governance** umbrella — see 3 sub-scenarios below. | Mixed | Mixed (ACA, App Service, VM, Cosmos DB) | ★★ | -## Post-Deployment +### `zava-infra` sub-scenarios -### Re-run the setup script +| Sub-scenario | What it demos | +|---|---| +| [`zava-infra/scenarios/perf-drift`](zava-infra/scenarios/perf-drift/) | VM CPU/memory pressure + Cosmos DB RU drift; Azure Monitor alerts → agent investigates SAP-style workload on Windows VMs. | +| [`zava-infra/scenarios/compliance`](zava-infra/scenarios/compliance/) | Container App revision compliance — Activity Log alert when an out-of-policy image is deployed; agent rolls back via approval hook. | +| [`zava-infra/scenarios/tf-drift`](zava-infra/scenarios/tf-drift/) | Terraform Cloud drift detection — webhook → agent diagnoses drift, opens PR with `terraform plan` summary. (Manual deploy.) | -```bash -# Full re-run (rebuilds container images + re-uploads everything) -./scripts/post-provision.sh +--- -# Skip container image builds (just update KB, subagents, response plan) -./scripts/post-provision.sh --retry +## Recipes -# Windows: run from CMD with Python in PATH -set PATH=%PATH%;C:\Users\%USERNAME%\AppData\Local\Programs\Python\Python312 -"C:\Program Files\Git\bin\bash.exe" scripts/post-provision.sh --retry -``` +Portable, lab-agnostic SRE Agent config bundles — agent + subagents + skills + hooks + tools — that you can apply to **your own** workload (no infra, no app code). -### Manual container deploy (Windows fallback) +| Recipe | Source lab | What you get | +|---|---|---| +| [`recipes/azmon-aca-servicenow-zavacafe-ops`](recipes/azmon-aca-servicenow-zavacafe-ops/) | [`zava-cafe`](zava-cafe/) | SQL ops + deployment validation: 3 subagents, 4 skills, 2 hooks. App Insights, AzMon, ServiceNow, Azure SQL MCP, ADO. | +| [`recipes/azmon-aca-servicenow-zavapower-ops`](recipes/azmon-aca-servicenow-zavapower-ops/) | [`zava-power`](zava-power/) | Microservice ops: 8 subagents, 15 skills. AzMon, ServiceNow, optional Datadog & Dynatrace MCP. | +| [`recipes/azmon-aca-servicenow-zavaitsupport`](recipes/azmon-aca-servicenow-zavaitsupport/) | [`zava-itsupport`](zava-itsupport/) | IT helpdesk laptop replacement: 1 subagent, ServiceNow Incident Platform, `CheckWarranty` + `LookupServiceNowIncident` tools, Browser Operator. | -If the script deploys images but the app still shows the default page: +See [`recipes/README.md`](recipes/README.md) for the recipe authoring + upstream contribution flow. -```cmd -for /f "tokens=*" %a in ('azd env get-value AZURE_CONTAINER_REGISTRY_NAME') do set ACR=%a -for /f "tokens=*" %a in ('azd env get-value CONTAINER_APP_NAME') do set APP=%a -for /f "tokens=*" %a in ('azd env get-value FRONTEND_APP_NAME') do set FE=%a -az containerapp update --name %APP% --resource-group rg-sre-lab --image %ACR%.azurecr.io/grubify-api:latest -az containerapp update --name %FE% --resource-group rg-sre-lab --image %ACR%.azurecr.io/grubify-frontend:latest -``` +--- -## Verify Setup +## Pick a lab -After deployment completes, open your agent at [sre.azure.com](https://sre.azure.com) and click **Full setup**. You should see green checkmarks on: +**By experience level:** -| Card | Expected Status | -|------|----------------| -| **Code** | ✅ 1 repository | -| **Incidents** | ✅ Connected to Azure Monitor | -| **Azure resources** | ✅ 1 resource group added | -| **Knowledge files** | ✅ 1 file | +- **First time?** → [`zava-eats`](zava-eats/) (★, ~40 min, no GitHub required) +- **Want SQL incidents?** → [`zava-cafe`](zava-cafe/) or [`zava-athletic`](zava-athletic/) +- **Want microservices?** → [`zava-power`](zava-power/) +- **Want IT helpdesk / ServiceNow flows?** → [`zava-itsupport`](zava-itsupport/) +- **Want AKS + private network?** → [`zava-athletic`](zava-athletic/) +- **Want infra governance / drift / compliance?** → [`zava-infra`](zava-infra/) -> **Checkpoint:** If any card is missing a checkmark, re-run the post-provision script: `bash scripts/post-provision.sh --retry` +**By compute platform:** -Once verified, click **"Done and go to agent"** to open the agent chat and start the team onboarding conversation. +| Platform | Labs | +|---|---| +| Azure Container Apps | `zava-eats`, `zava-itsupport`, `zava-power`, `zava-infra/compliance` | +| Azure App Service | `zava-cafe` | +| AKS | `zava-athletic` | +| Azure VMs (Windows) | `zava-infra/perf-drift` | -### Team Onboarding +**By data tier:** -The agent opens a **"Team onboarding"** thread automatically. It will: +| Data | Labs | +|---|---| +| Azure SQL DB | `zava-cafe` | +| PostgreSQL Flexible Server | `zava-athletic` | +| Cosmos DB | `zava-infra/perf-drift` | +| In-memory only | `zava-eats`, `zava-itsupport`, `zava-power` | -1. **Explore your connected context** — reads the code repository, Azure resources, and knowledge files you connected during setup -2. **Interview you about your team** — ask about your team structure, on-call rotation, services you own, and escalation paths +--- -Since the agent already has context from setup, try asking it questions: +## Prerequisites (shared across all labs) -> *"What do you know about the Grubify app architecture?"* -> -> *"Summarize the HTTP errors runbook"* -> -> *"What Azure resources are in my resource group?"* +| Tool | macOS | Windows | +|---|---|---| +| [Azure CLI](https://learn.microsoft.com/cli/azure/install-azure-cli) 2.60+ | `brew install azure-cli` | `winget install Microsoft.AzureCLI` | +| [Azure Developer CLI](https://learn.microsoft.com/azure/developer/azure-developer-cli/install-azd) 1.9+ | `brew install azd` | `winget install Microsoft.Azd` | +| [Git](https://git-scm.com/) 2.x | `brew install git` | `winget install Git.Git` (includes Git Bash) | +| [Python](https://python.org) 3.10+ | `brew install python3` | `winget install Python.Python.3.12` | -The agent saves your team information to persistent memory and references it in every future investigation. +Plus per-lab tools listed in each lab's README (e.g., `kubectl` not required for `zava-athletic`; `pwsh` for `zava-cafe`). -> **Tip:** Ask *"What should I do next?"* for personalized recommendations based on what's connected. +**Azure requirements (every lab):** +- Active Azure subscription +- **Owner** role on the subscription (needed for RBAC role assignments) +- `az provider register -n Microsoft.App --wait` +- SRE Agent regions: `eastus2`, `swedencentral`, `australiaeast` -## Lab Scenarios +Run [`scripts/prereqs.sh`](scripts/prereqs.sh) to verify your environment. -### Scenario 1: IT Operations (No GitHub required) +--- -Break the app and watch the agent investigate: +## Quick start (any lab) ```bash -./scripts/break-app.sh # macOS/Linux -# Windows: "C:\Program Files\Git\bin\bash.exe" scripts/break-app.sh -``` - -Then open [sre.azure.com](https://sre.azure.com) → Incidents to watch the agent: -1. Detect the Azure Monitor alert -2. Query Log Analytics for error patterns -3. Reference the HTTP errors runbook -4. Apply remediation (restart/scale) -5. Summarize with root cause and evidence - -### Scenario 2: Developer (Requires GitHub) - -Ask the agent to search source code for root causes: -- File:line references to problematic code -- Correlation of production errors to code changes -- Suggested fixes with before/after examples - -### Scenario 3: Workflow Automation (Requires GitHub) +git clone https://github.com/dm-chelupati/sre-agent-lab.git +cd sre-agent-lab +git submodule update --init --recursive -Create sample support issues and let the agent triage them: +az login && azd auth login -```bash -./scripts/create-sample-issues.sh +cd labs/ # e.g. labs/zava-eats +azd env new +azd up # pick subscription + region (eastus2 recommended) ``` -The agent classifies issues (Documentation, Bug, Feature Request), applies labels, and posts triage comments following the runbook. - -## Adding GitHub Later - -After initial setup, add GitHub by signing in via the OAuth URL: - -```bash -./scripts/setup-github.sh # macOS/Linux -# Windows: "C:\Program Files\Git\bin\bash.exe" scripts/setup-github.sh -``` +Each lab's README has the exact post-deploy steps — open the agent at [sre.azure.com](https://sre.azure.com), then run the lab's break script. -> **Security tip:** The OAuth flow requests broad repo access. For least-privilege, -> use a [fine-grained PAT](https://github.com/settings/personal-access-tokens/new) -> scoped to your grubify fork only with permissions: `Contents:Read`, `Issues:Read+Write`, `Metadata:Read`. -> ```bash -> export GITHUB_PAT=github_pat_xxxx -> ./scripts/setup-github.sh -> ``` +--- -## Cleanup +## Cleanup (any lab) ```bash +cd labs/ azd down --purge ``` -## Troubleshooting - -| Issue | Fix | -|-------|-----| -| `'bash' is not recognized` (Windows) | Run via: `"C:\Program Files\Git\bin\bash.exe" scripts/post-provision.sh` | -| `Python was not found` (Windows) | Install: `winget install Python.Python.3.12`, disable App execution aliases | -| `curl: error encountered when reading a file` | Python isn't in Git Bash PATH: `export PATH="$PATH:/c/Users/$USER/AppData/Local/Programs/Python/Python312"` | -| `roleAssignments/write` denied | Need Owner role on subscription. Check: `az role assignment list --assignee $(az ad signed-in-user show --query id -o tsv)` | -| `Microsoft.App not registered` | Run: `az provider register -n Microsoft.App --wait` | -| Grubify shows default page after deploy | Run manual deploy commands (see Post-Deployment section above) | -| Post-provision 405 on response plan | Wait 30s and run: `./scripts/post-provision.sh --retry` | -| Agent can't create issues on forked repo | Forks have Issues disabled by default. Enable: repo Settings → Features → Issues ✅, or run `gh api -X PATCH repos/OWNER/REPO -f has_issues=true` | - -## Regions - -SRE Agent is available in: `eastus2`, `swedencentral`, `australiaeast` +--- ## Links -- [Azure SRE Agent Documentation](https://sre.azure.com/docs) -- [Getting Started Guide](https://sre.azure.com/docs/get-started/create-and-setup) +- [Azure SRE Agent docs](https://sre.azure.com/docs) +- [Getting started](https://sre.azure.com/docs/get-started/create-and-setup) - [Connectors](https://sre.azure.com/docs/concepts/connectors) -- [Custom Agents](https://sre.azure.com/docs/concepts/subagents) -- [Incident Response](https://sre.azure.com/docs/capabilities/incident-response) -- [Azure Observability](https://sre.azure.com/docs/capabilities/diagnose-azure-observability) +- [Custom subagents](https://sre.azure.com/docs/concepts/subagents) +- [Incident response](https://sre.azure.com/docs/capabilities/incident-response) +- [Multi-lab launcher (`LAUNCHER.md`)](LAUNCHER.md) +- [Recipes (portable agent configs)](recipes/) +- [Lab authoring guide (`AGENTS.md`)](AGENTS.md) ## License diff --git a/labs/_platform/check-prereqs.ps1 b/labs/_platform/check-prereqs.ps1 new file mode 100644 index 000000000..b3a5d4c50 --- /dev/null +++ b/labs/_platform/check-prereqs.ps1 @@ -0,0 +1,117 @@ +<# +.SYNOPSIS + Unified prereq gate for every lab. Run BEFORE `azd up` so users don't burn + 20-30 minutes provisioning Azure resources only to fail at post-provision. + +.PARAMETER Lab + The lab name (only used in the printed banner). + +.PARAMETER Strict + If set, any missing/private prereq aborts. Default: warn but allow continue. + +.NOTES + srectl + the SRE Agent MCP server are currently in *Microsoft private preview*. + Public users cannot pull them from the web. The honest UX: + 1. Detect them. + 2. If missing, point at the onboarding contact + offer a "infra-only" mode + (Bicep deploys, but skip the post-provision srectl steps). +#> +[CmdletBinding()] +param( + [string]$Lab = "(unknown)", + [switch]$Strict +) + +$ErrorActionPreference = 'Continue' +$missing = @() +$private = @() + +function Test-Cmd { + param([string]$Name, [string]$Hint, [switch]$Private) + if (Get-Command $Name -ErrorAction SilentlyContinue) { + Write-Host " ✓ $Name" -ForegroundColor Green + return $true + } + if ($Private) { + Write-Host " ⚠ $Name (Microsoft private preview — $Hint)" -ForegroundColor Yellow + $script:private += $Name + } else { + Write-Host " ✗ $Name (install: $Hint)" -ForegroundColor Red + $script:missing += $Name + } + return $false +} + +Write-Host "`n═══ Prereq check for '$Lab' ═══`n" -ForegroundColor Cyan + +# --- public, required for every lab --- +Test-Cmd 'azd' 'https://aka.ms/azd-install' | Out-Null +Test-Cmd 'az' 'https://aka.ms/install-azure-cli' | Out-Null +Test-Cmd 'python' 'https://www.python.org/downloads/' | Out-Null +Test-Cmd 'pwsh' 'https://aka.ms/powershell' | Out-Null + +# --- private preview --- +$srectlPresent = Test-Cmd 'srectl' 'request access via aka.ms/sreagent-onboarding' -Private + +# --- az login state --- +$acct = az account show 2>$null | ConvertFrom-Json +if ($acct) { + Write-Host " ✓ az login: $($acct.user.name) ($($acct.name))" -ForegroundColor Green +} else { + Write-Host " ✗ not logged in to Azure — run: az login" -ForegroundColor Red + $missing += 'az login' +} + +# --- azd login state (best-effort) --- +$azdAuth = azd auth login --check-status 2>&1 | Out-String +if ($azdAuth -match 'Logged in') { + Write-Host " ✓ azd login" -ForegroundColor Green +} else { + Write-Host " ⚠ azd not logged in — run: azd auth login" -ForegroundColor Yellow +} + +# --- summary --- +Write-Host "" +if ($missing.Count -gt 0) { + Write-Host "✗ Missing required tools: $($missing -join ', ')" -ForegroundColor Red + Write-Host " Install them and re-run. Aborting.`n" -ForegroundColor Red + exit 2 +} + +if ($private.Count -gt 0) { + Write-Host "⚠ Private-preview tools missing: $($private -join ', ')" -ForegroundColor Yellow + Write-Host @" + + These tools are required for the SRE Agent configuration step that runs + AFTER the Bicep infrastructure deploy: + + • srectl — applies subagents/skills/tools/scheduled-tasks to the agent + • SRE Agent MCP — optional; surfaces srectl as MCP tools to your IDE + + If you don't have access yet, the lab's Bicep stage will still deploy the + Azure infrastructure successfully, but post-provision will fail when it + tries to call ``srectl init`` / ``srectl apply-yaml``. + + To get access: + 1. Microsoft FTEs: see https://aka.ms/sreagent-onboarding (internal) + 2. Customers: contact your Microsoft account team for SRE Agent preview + +"@ -ForegroundColor DarkYellow + + if ($Strict) { + Write-Host " -Strict set — aborting.`n" -ForegroundColor Red + exit 3 + } + + $env:LABS_SKIP_SRECTL = '1' + $resp = Read-Host " Continue with infra-only deploy (skip srectl steps)? [y/N]" + if ($resp -notmatch '^[yY]') { + Write-Host " Aborted by user.`n" -ForegroundColor Yellow + exit 4 + } + Write-Host " → LABS_SKIP_SRECTL=1 set; post-provision scripts should honour this.`n" -ForegroundColor Yellow +} else { + Write-Host "✓ All prereqs present.`n" -ForegroundColor Green +} + +exit 0 diff --git a/labs/_platform/helpers/manifest.py b/labs/_platform/helpers/manifest.py new file mode 100644 index 000000000..b858a5365 --- /dev/null +++ b/labs/_platform/helpers/manifest.py @@ -0,0 +1,149 @@ +"""Zava Unlimited platform — lab manifest helpers. + +Used by labs/lab.ps1 (launcher) and labs/sim.ps1 (meta-sim) via subprocess. +Outputs JSON on stdout for the pwsh callers to consume. + +Commands: + python manifest.py list # list discovered labs (json) + python manifest.py read # read & validate one manifest (json) + python manifest.py deployed # list .deployed/ entries (json) + python manifest.py validate # exit 0 if valid, prints errors +""" +import json, sys, os, glob, re +from pathlib import Path + +try: + import yaml +except ImportError: + print(json.dumps({"error": "PyYAML not installed. Run: pip install pyyaml"})) + sys.exit(2) + +SCHEMA_PATH = Path(__file__).parent.parent / "schema" / "lab.schema.json" + + +def _load_yaml(p: Path): + with p.open(encoding="utf-8-sig") as f: + return yaml.safe_load(f) + + +def _validate(manifest: dict, schema: dict) -> list[str]: + """Lightweight validator (avoids jsonschema dependency). + Catches the cases that matter for our use case.""" + errs = [] + req = schema.get("required", []) + for k in req: + if k not in manifest: + errs.append(f"missing required field: {k}") + name = manifest.get("name", "") + if name and not re.match(r"^[a-z][a-z0-9-]+$", name): + errs.append(f"name '{name}' must be kebab-case") + for p in manifest.get("prompts", []): + n = p.get("name", "") + if n and not re.match(r"^[A-Z][A-Z0-9_]+$", n): + errs.append(f"prompt name '{n}' must be SCREAMING_SNAKE") + if "text" not in p: + errs.append(f"prompt {n!r} missing 'text'") + for s in manifest.get("scenarios", []): + for f in ("id", "label"): + if f not in s: + errs.append(f"scenario missing '{f}'") + sim = manifest.get("sim") + if sim is not None: + for f in ("command", "args"): + if f not in sim: + errs.append(f"sim missing '{f}'") + return errs + + +def _read_one(lab_dir: Path) -> dict | None: + """Read a lab's manifest. Returns None for legacy labs without lab.yaml.""" + mf = lab_dir / "lab.yaml" + if not mf.exists(): + # Legacy fallback: use azure.yaml + README first line + az = lab_dir / "azure.yaml" + if not az.exists(): + return None + readme = lab_dir / "README.md" + desc = "" + if readme.exists(): + for line in readme.read_text(encoding="utf-8-sig").splitlines()[:6]: + if line.strip() and not line.startswith("#"): + desc = line.strip() + break + return { + "name": lab_dir.name, + "displayName": lab_dir.name, + "description": desc or "(no manifest — legacy lab)", + "_legacy": True, + "_path": str(lab_dir), + } + try: + m = _load_yaml(mf) or {} + except Exception as e: + return {"name": lab_dir.name, "_path": str(lab_dir), "_error": f"yaml parse: {e}"} + schema = json.loads(SCHEMA_PATH.read_text()) + errs = _validate(m, schema) + if errs: + m["_validationErrors"] = errs + m["_path"] = str(lab_dir) + m["_legacy"] = False + return m + + +def cmd_list(labs_dir: str): + base = Path(labs_dir) + out = [] + for d in sorted(base.iterdir()): + if not d.is_dir() or d.name.startswith(("_", ".")): + continue + m = _read_one(d) + if m: + out.append(m) + print(json.dumps(out, indent=2)) + + +def cmd_read(lab_dir: str): + m = _read_one(Path(lab_dir)) + print(json.dumps(m or {}, indent=2)) + + +def cmd_deployed(labs_dir: str): + deployed_dir = Path(labs_dir) / ".deployed" + out = [] + if deployed_dir.exists(): + for f in sorted(deployed_dir.glob("*.json")): + try: + out.append(json.loads(f.read_text(encoding="utf-8-sig"))) + except Exception as e: + out.append({"_file": str(f), "_error": str(e)}) + print(json.dumps(out, indent=2)) + + +def cmd_validate(mf_path: str): + p = Path(mf_path) + if not p.exists(): + print(f"FAIL: {p} not found"); sys.exit(1) + try: + m = _load_yaml(p) or {} + except Exception as e: + print(f"FAIL: yaml parse: {e}"); sys.exit(1) + schema = json.loads(SCHEMA_PATH.read_text()) + errs = _validate(m, schema) + if errs: + print(f"FAIL: {len(errs)} validation error(s):") + for e in errs: + print(f" - {e}") + sys.exit(1) + print(f"OK: {p.name} is valid (name={m.get('name')}, scenarios={len(m.get('scenarios', []))})") + + +if __name__ == "__main__": + if len(sys.argv) < 3: + print(__doc__); sys.exit(2) + cmd, arg = sys.argv[1], sys.argv[2] + { + "list": cmd_list, + "read": cmd_read, + "deployed": cmd_deployed, + "validate": cmd_validate, + }[cmd](arg) diff --git a/labs/_platform/http_trigger.py b/labs/_platform/http_trigger.py new file mode 100644 index 000000000..9215b6695 --- /dev/null +++ b/labs/_platform/http_trigger.py @@ -0,0 +1,208 @@ +#!/usr/bin/env python3 +""" +SRE Agent HTTP Trigger registration helper. + +Reusable across labs. Wraps the (currently CLI-less) REST API at + POST /api/v1/httptriggers/create + POST /api/v1/httptriggers/{id}/enable + GET /api/v1/httptriggers/{id} + ... + +Authentication: bearer token for resource 'https://azuresre.ai' (same as srectl). + +CLI: + python http_trigger.py create-and-enable \ + --endpoint https://.azuresre.ai \ + --name my-trigger \ + --agent incident-handler \ + --prompt "Investigate the incoming alert payload." \ + [--mode autonomous|review|readonly] \ + [--description "..."] + +Prints JSON to stdout: { "triggerId": "...", "triggerUrl": "..." } +Exit 0 on success, 1 on failure. + +Idempotent: if a trigger with the same --name already exists, reuses it +(GETs the URL via /enable which is idempotent and returns the existing URL). +""" +from __future__ import annotations + +import argparse +import json +import os +import subprocess +import sys +import time +import urllib.error +import urllib.request + + +def _az_token(resource: str = "https://azuresre.ai") -> str: + """Acquire an AAD bearer token for the SRE Agent resource via az CLI.""" + try: + out = subprocess.run( + ["az", "account", "get-access-token", "--resource", resource, + "--query", "accessToken", "-o", "tsv"], + capture_output=True, text=True, timeout=60, check=True, shell=False, + ) + token = out.stdout.strip() + if not token: + raise RuntimeError("empty token from az") + return token + except FileNotFoundError: + # Windows: az may need shell + out = subprocess.run( + 'az account get-access-token --resource "%s" --query accessToken -o tsv' % resource, + capture_output=True, text=True, timeout=60, check=True, shell=True, + ) + return out.stdout.strip() + + +def _request(method: str, url: str, token: str, body: dict | None = None, timeout: int = 60) -> dict: + data = None + headers = { + "Authorization": f"Bearer {token}", + "Accept": "application/json", + } + if body is not None: + data = json.dumps(body).encode("utf-8") + headers["Content-Type"] = "application/json" + req = urllib.request.Request(url, data=data, method=method, headers=headers) + try: + with urllib.request.urlopen(req, timeout=timeout) as resp: + raw = resp.read().decode("utf-8") or "{}" + try: + return json.loads(raw) + except json.JSONDecodeError: + return {"_raw": raw} + except urllib.error.HTTPError as e: + detail = e.read().decode("utf-8", errors="replace") + raise RuntimeError(f"HTTP {e.code} {e.reason} on {method} {url}: {detail[:500]}") from None + + +def list_triggers(endpoint: str, token: str) -> list[dict]: + out = _request("GET", f"{endpoint.rstrip('/')}/api/v1/httptriggers", token) + if isinstance(out, list): + return out + return out.get("items") or out.get("triggers") or [] + + +def find_by_name(endpoint: str, token: str, name: str) -> dict | None: + for t in list_triggers(endpoint, token): + if t.get("name") == name: + return t + return None + + +def create_trigger(endpoint: str, token: str, *, name: str, agent: str, + prompt: str, mode: str = "autonomous", + description: str = "") -> dict: + body = { + "name": name, + "agent": agent, + "agentPrompt": prompt, + "agentMode": mode, + } + if description: + body["description"] = description + return _request( + "POST", + f"{endpoint.rstrip('/')}/api/v1/httptriggers/create", + token, + body=body, + ) + + +def enable_trigger(endpoint: str, token: str, trigger_id: str) -> dict: + return _request( + "POST", + f"{endpoint.rstrip('/')}/api/v1/httptriggers/{trigger_id}/enable", + token, + body={}, + ) + + +def get_trigger(endpoint: str, token: str, trigger_id: str) -> dict: + return _request( + "GET", + f"{endpoint.rstrip('/')}/api/v1/httptriggers/{trigger_id}", + token, + ) + + +def create_and_enable(endpoint: str, *, name: str, agent: str, prompt: str, + mode: str = "autonomous", description: str = "") -> dict: + """Idempotent: returns {triggerId, triggerUrl}.""" + token = _az_token() + existing = find_by_name(endpoint, token, name) + if existing: + trigger_id = existing.get("triggerId") or existing.get("id") + url = existing.get("triggerUrl") or existing.get("url") + if not url: + enabled = enable_trigger(endpoint, token, trigger_id) + url = enabled.get("triggerUrl") or enabled.get("url") + return {"triggerId": trigger_id, "triggerUrl": url, "reused": True} + + created = create_trigger( + endpoint, token, + name=name, agent=agent, prompt=prompt, + mode=mode, description=description, + ) + trigger_id = created.get("triggerId") or created.get("id") + if not trigger_id: + raise RuntimeError(f"create returned no triggerId: {created!r}") + + # Small delay — server may need a tick before enable can mint URL + time.sleep(1.0) + enabled = enable_trigger(endpoint, token, trigger_id) + url = enabled.get("triggerUrl") or enabled.get("url") or created.get("triggerUrl") + if not url: + # Last resort: GET it + info = get_trigger(endpoint, token, trigger_id) + url = info.get("triggerUrl") or info.get("url") + return {"triggerId": trigger_id, "triggerUrl": url, "reused": False} + + +def _cli() -> int: + p = argparse.ArgumentParser(description="SRE Agent HTTP trigger helper") + sub = p.add_subparsers(dest="cmd", required=True) + + cae = sub.add_parser("create-and-enable", help="Create (or reuse) and enable a trigger; print {triggerId, triggerUrl}") + cae.add_argument("--endpoint", required=True, help="SRE Agent endpoint, e.g. https://.azuresre.ai") + cae.add_argument("--name", required=True) + cae.add_argument("--agent", required=True, help="Sub-agent metadata.name to invoke") + cae.add_argument("--prompt", required=True, help="Default agentPrompt for this trigger") + cae.add_argument("--mode", default="autonomous", choices=["autonomous", "review", "readonly"]) + cae.add_argument("--description", default="") + + lst = sub.add_parser("list", help="List triggers") + lst.add_argument("--endpoint", required=True) + + args = p.parse_args() + + endpoint = (args.endpoint or os.environ.get("SRE_AGENT_ENDPOINT", "")).strip() + if not endpoint: + print("error: --endpoint or $SRE_AGENT_ENDPOINT required", file=sys.stderr) + return 2 + + try: + if args.cmd == "create-and-enable": + res = create_and_enable( + endpoint, + name=args.name, agent=args.agent, prompt=args.prompt, + mode=args.mode, description=args.description, + ) + print(json.dumps(res)) + return 0 if res.get("triggerUrl") else 1 + if args.cmd == "list": + token = _az_token() + print(json.dumps(list_triggers(endpoint, token), indent=2)) + return 0 + except Exception as e: + print(f"error: {e}", file=sys.stderr) + return 1 + return 1 + + +if __name__ == "__main__": + sys.exit(_cli()) diff --git a/labs/_platform/schema/lab.example.yaml b/labs/_platform/schema/lab.example.yaml new file mode 100644 index 000000000..d0b0b2730 --- /dev/null +++ b/labs/_platform/schema/lab.example.yaml @@ -0,0 +1,56 @@ +# lab.yaml — Zava Unlimited lab manifest (example, fully annotated) +# Validate with: python labs/_platform/helpers/validate-manifest.py /lab.yaml +schemaVersion: 1 + +# Stable identifier (must equal the lab's directory name) +name: zava-power + +# Human-readable name shown in pickers +displayName: "Zava Power — ZeroOps" +subsidiary: "Zava Power" +description: "ACA + ServiceNow utility-platform demo with 8 break/fix scenarios." +estimatedMinutes: 25 +tags: [aca, servicenow, ado, github, autoremediation] + +# Tools that must be on PATH (preprovision check) +prereqs: [az, azd, pwsh, python, docker, srectl] + +# Values azd can't infer — launcher prompts before deploy, stashes in azd env +prompts: + - name: SERVICENOW_INSTANCE + text: "ServiceNow PDI hostname (e.g. dev123456)" + - name: SERVICENOW_USER + text: "ServiceNow admin username" + default: "admin" + - name: SERVICENOW_PASSWORD + text: "ServiceNow admin password" + secret: true + - name: DEMO_EMPLOYEE_EMAIL + text: "Demo employee email" + default: "demo.user@zavapower.com" + +# How the meta-sim launches this lab's own rich sim UI +sim: + command: python + args: [simulator/demo.py] + envFromConfig: true + +# Scenarios surfaced in the meta-sim's unified picker +scenarios: + - id: vm-disk-pressure + label: "VM disk pressure" + description: "Arc VM disk fills past 90% — agent cleans temp dirs." + runner: scripts/scenarios/vm-disk.ps1 + minutes: 5 + needs: [snow] + - id: api-perf-regression + label: "API performance regression" + description: "grid-status-api blocks the event loop — agent rolls back." + runner: scripts/scenarios/api-perf.ps1 + minutes: 4 + - id: pod-incident-audit + label: "Pod incident audit" + description: "OOMKills aggregated into a SNOW deck." + runner: scripts/scenarios/pod-audit.ps1 + minutes: 10 + needs: [snow] diff --git a/labs/_platform/schema/lab.schema.json b/labs/_platform/schema/lab.schema.json new file mode 100644 index 000000000..0868459e4 --- /dev/null +++ b/labs/_platform/schema/lab.schema.json @@ -0,0 +1,93 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "$id": "https://aka.ms/zava-unlimited/lab.schema.json", + "title": "Zava Unlimited lab manifest", + "description": "Declarative contract every lab provides. Read by labs/lab.ps1 (launcher) and labs/sim.ps1 (meta-simulator).", + "type": "object", + "required": ["name", "displayName", "description"], + "additionalProperties": false, + "properties": { + "schemaVersion": { + "type": "integer", + "const": 1, + "description": "Manifest schema version. Always 1 for now." + }, + "name": { + "type": "string", + "pattern": "^[a-z][a-z0-9-]+$", + "description": "Stable kebab-case identifier. Must match the lab's directory name." + }, + "displayName": { + "type": "string", + "description": "Human-readable lab name shown in pickers (e.g. 'Zava Power — ZeroOps')." + }, + "subsidiary": { + "type": "string", + "description": "Which Zava Unlimited subsidiary this lab represents (e.g. 'Zava Power', 'Zava Athletic'). Optional for non-Zava-themed labs." + }, + "description": { + "type": "string", + "description": "One-sentence description shown in launcher list." + }, + "estimatedMinutes": { + "type": "integer", + "minimum": 1, + "description": "Approximate end-to-end deploy time in minutes." + }, + "tags": { + "type": "array", + "items": { "type": "string" }, + "description": "Free-form labels (e.g. 'aca', 'aks', 'servicenow', 'autoremediation')." + }, + "prereqs": { + "type": "array", + "items": { "type": "string" }, + "description": "CLI tools that must be on PATH (e.g. 'az', 'azd', 'pwsh', 'python', 'docker', 'srectl')." + }, + "prompts": { + "type": "array", + "description": "Values azd cannot infer that the launcher must collect interactively before deploy. Stashed in the lab's azd env.", + "items": { + "type": "object", + "required": ["name", "text"], + "additionalProperties": false, + "properties": { + "name": { "type": "string", "pattern": "^[A-Z][A-Z0-9_]+$", "description": "Env var name (SCREAMING_SNAKE)." }, + "text": { "type": "string", "description": "Question shown to the user." }, + "default": { "type": "string", "description": "Default if the user just presses Enter." }, + "secret": { "type": "boolean", "description": "If true, input is masked." }, + "optional":{ "type": "boolean", "description": "If true, blank input is allowed." } + } + } + }, + "sim": { + "type": "object", + "description": "How the meta-sim launches this lab's own simulator UI when the user picks 'open lab sim'.", + "required": ["command", "args"], + "additionalProperties": false, + "properties": { + "command": { "type": "string", "description": "Executable (e.g. 'python', 'pwsh', 'node')." }, + "args": { "type": "array", "items": { "type": "string" }, "description": "Args relative to the lab's root dir." }, + "envFromConfig": { "type": "boolean", "default": true, "description": "If true, the meta-sim reads .lab-config.json (or .deployed/.json) and exports it to env before launching." } + } + }, + "scenarios": { + "type": "array", + "description": "Break/fix scenarios this lab can drive. Exposed in the meta-sim's unified picker. Optional — labs without scenarios just don't show in the unified picker.", + "items": { + "type": "object", + "required": ["id", "label"], + "additionalProperties": false, + "properties": { + "id": { "type": "string", "pattern": "^[a-z][a-z0-9-]+$" }, + "label": { "type": "string" }, + "description": { "type": "string" }, + "runner": { "type": "string", "description": "Path to the runner script relative to the lab root." }, + "minutes":{ "type": "integer", "minimum": 1 }, + "needs": { "type": "array", "items": { "type": "string" }, "description": "Capability tags the scenario needs (e.g. 'snow', 'ado', 'github')." }, + "tags": { "type": "array", "items": { "type": "string" } } + } + } + } + } +} diff --git a/labs/_platform/template/README.md.tmpl b/labs/_platform/template/README.md.tmpl new file mode 100644 index 000000000..64d588a9c --- /dev/null +++ b/labs/_platform/template/README.md.tmpl @@ -0,0 +1,33 @@ +# {{LAB_DISPLAY_NAME}} + +> Part of the [Zava Unlimited](../README.md) SRE Agent demo platform. + +{{LAB_DESCRIPTION}} + +## Quick start + +```bash +# From the labs/ root: +./lab.sh -Labs {{LAB_NAME}} # deploy this lab +./sim.sh -Lab {{LAB_NAME}} # open the simulator after deploy +``` + +Or from this lab's directory: `azd up`. + +## Scenarios + +See `lab.yaml` for the canonical list. Each scenario has a runner script in +`scripts/scenarios/`. To run one directly: + +```bash +../sim.sh -Scenario {{LAB_NAME}}/ +``` + +## Architecture + +TODO: add a brief architecture description and diagram. + +## Contributing scenarios + +Drop a new runner script in `scripts/scenarios/` and add an entry to `lab.yaml` +under `scenarios:`. The meta-sim auto-discovers it on next run. diff --git a/labs/_platform/template/azure.yaml.tmpl b/labs/_platform/template/azure.yaml.tmpl new file mode 100644 index 000000000..ffaca1fe7 --- /dev/null +++ b/labs/_platform/template/azure.yaml.tmpl @@ -0,0 +1,34 @@ +# Azure Developer CLI (azd) template for {{LAB_DISPLAY_NAME}} +# Single-command deploy: from this directory run `azd up`. +name: {{LAB_NAME}} +metadata: + template: {{LAB_NAME}}@1.0.0 + +infra: + provider: bicep + path: infra + +hooks: + preprovision: + windows: + shell: pwsh + run: scripts/check-environment.ps1 + interactive: true + posix: + shell: sh + run: | + if ! command -v pwsh >/dev/null 2>&1; then + echo "ERROR: pwsh (PowerShell 7+) required for {{LAB_NAME}}." >&2 + exit 1 + fi + pwsh -NoProfile -File scripts/check-environment.ps1 + interactive: true + postprovision: + windows: + shell: pwsh + run: scripts/post-provision.ps1 + interactive: true + posix: + shell: sh + run: pwsh -NoProfile -File scripts/post-provision.ps1 + interactive: true diff --git a/labs/_platform/template/infra/main.bicep.tmpl b/labs/_platform/template/infra/main.bicep.tmpl new file mode 100644 index 000000000..f8e8cc373 --- /dev/null +++ b/labs/_platform/template/infra/main.bicep.tmpl @@ -0,0 +1,26 @@ +// {{LAB_DISPLAY_NAME}} infrastructure. +// TODO: replace with your actual resources. +targetScope = 'subscription' + +@description('Azure region for all resources.') +param location string + +@description('Workload name — used as prefix for naming.') +param workloadName string = '{{LAB_NAME}}' + +resource rg 'Microsoft.Resources/resourceGroups@2024-03-01' = { + name: 'rg-${workloadName}' + location: location +} + +// Add modules here, e.g.: +// module observability 'modules/observability.bicep' = { +// name: 'observability' +// scope: rg +// params: { location: location, workloadName: workloadName } +// } + +// ── Outputs (azd auto-promotes ALL_CAPS outputs to env vars) ── +output AZURE_RESOURCE_GROUP string = rg.name +output AZURE_LOCATION string = location +output WORKLOAD_NAME string = workloadName diff --git a/labs/_platform/template/lab.yaml.tmpl b/labs/_platform/template/lab.yaml.tmpl new file mode 100644 index 000000000..33aa23959 --- /dev/null +++ b/labs/_platform/template/lab.yaml.tmpl @@ -0,0 +1,35 @@ +schemaVersion: 1 +name: {{LAB_NAME}} +displayName: "{{LAB_DISPLAY_NAME}}" +subsidiary: "{{LAB_SUBSIDIARY}}" +description: "{{LAB_DESCRIPTION}}" +estimatedMinutes: 20 +tags: [{{LAB_TAGS}}] + +# CLI tools that must be on PATH (preprovision will check) +prereqs: [az, azd, pwsh, python] + +# Values azd cannot infer — launcher prompts before deploy and stashes in azd env. +# Use SCREAMING_SNAKE names. Set secret:true for passwords. +prompts: [] + # Example: + # - name: SOME_API_KEY + # text: "API key for the foo connector" + # secret: true + +# How the meta-sim launches this lab's own simulator UI when the user picks "open lab sim". +sim: + command: python + args: [simulator/demo.py] + envFromConfig: true + +# Break/fix scenarios this lab can drive. Exposed in the meta-sim's unified picker. +# Add one block per scenario. The runner script can be .ps1, .py, or .sh. +scenarios: [] + # Example: + # - id: example-failure + # label: "Example failure" + # description: "Brief description of what breaks and what the agent does." + # runner: scripts/scenarios/example.ps1 + # minutes: 5 + # needs: [] # capability tags: snow, ado, github, etc. diff --git a/labs/_platform/template/scripts/check-environment.ps1.tmpl b/labs/_platform/template/scripts/check-environment.ps1.tmpl new file mode 100644 index 000000000..227339c5f --- /dev/null +++ b/labs/_platform/template/scripts/check-environment.ps1.tmpl @@ -0,0 +1,48 @@ +#requires -Version 7.0 +<# +preprovision hook for {{LAB_DISPLAY_NAME}}. + 1. Verify prerequisites declared in lab.yaml + 2. Collect any prompts declared in lab.yaml that aren't already in azd env +#> +$ErrorActionPreference = 'Stop' +$labRoot = Split-Path $PSScriptRoot -Parent +$labYaml = Join-Path $labRoot 'lab.yaml' +$helper = Join-Path (Split-Path $labRoot -Parent) '_platform/helpers/manifest.py' + +if (-not (Test-Path $labYaml)) { Write-Host " (no lab.yaml — skipping manifest checks)" -ForegroundColor DarkGray; return } +$manifest = & python $helper read $labRoot | ConvertFrom-Json +if ($manifest._validationErrors) { + Write-Host "`n ✗ lab.yaml has validation errors:" -ForegroundColor Red + $manifest._validationErrors | ForEach-Object { Write-Host " - $_" -ForegroundColor Red } + exit 1 +} + +# ── Prerequisites ── +$missing = @() +foreach ($t in @($manifest.prereqs)) { + if (-not (Get-Command $t -ErrorAction SilentlyContinue)) { + Write-Host " ✗ $t not found" -ForegroundColor Red; $missing += $t + } else { Write-Host " ✓ $t" -ForegroundColor Green } +} +if ($missing.Count -gt 0) { Write-Host "`nInstall missing tools and retry.`n" -ForegroundColor Yellow; exit 1 } + +if (-not (az account show 2>$null)) { Write-Host "`nRun 'az login' first." -ForegroundColor Yellow; exit 1 } + +# ── Prompts ── +$existing = (azd env get-values --output json 2>$null | ConvertFrom-Json) +foreach ($p in @($manifest.prompts)) { + if ($existing -and $existing.PSObject.Properties[$p.name]) { continue } + $msg = " $($p.text)" + if ($p.default) { $msg += " [$($p.default)]" } + if ($p.secret) { + $val = Read-Host -AsSecureString $msg + $val = [System.Net.NetworkCredential]::new('', $val).Password + } else { + $val = Read-Host $msg + } + if (-not $val -and $p.default) { $val = $p.default } + if (-not $val -and -not $p.optional) { Write-Host " (skipped $($p.name))" -ForegroundColor DarkYellow; continue } + if ($val) { azd env set $p.name $val | Out-Null } +} + +Write-Host "`n Prereqs OK. azd will provision next.`n" -ForegroundColor Green diff --git a/labs/_platform/template/scripts/post-provision.ps1.tmpl b/labs/_platform/template/scripts/post-provision.ps1.tmpl new file mode 100644 index 000000000..0b3a7dfbf --- /dev/null +++ b/labs/_platform/template/scripts/post-provision.ps1.tmpl @@ -0,0 +1,52 @@ +#requires -Version 7.0 +<# +postprovision hook for {{LAB_DISPLAY_NAME}}. + +Customize the steps below for your lab. The required final step is writing +the .deployed/.json record so the Zava Unlimited meta-sim discovers it. +#> +$ErrorActionPreference = 'Stop' +$labRoot = Split-Path $PSScriptRoot -Parent +Push-Location $labRoot +try { + Write-Host "`n═══ {{LAB_DISPLAY_NAME}} post-provision ═══" -ForegroundColor Cyan + + $env_obj = azd env get-values --output json | ConvertFrom-Json + function Env([string]$k, [string]$default = '') { + if ($env_obj.PSObject.Properties[$k] -and $env_obj.$k) { return $env_obj.$k } + return $default + } + $sub = Env 'AZURE_SUBSCRIPTION_ID' + $rg = Env 'AZURE_RESOURCE_GROUP' + $loc = Env 'AZURE_LOCATION' + + # ── TODO: customize these steps for your lab ── + # Examples: + # az acr build --registry $acr --image foo:latest src/foo + # srectl init --resource-url "/subscriptions/$sub/resourceGroups/$rg/providers/Microsoft.App/sreAgents/$agent" + # srectl apply-yaml --file sre-config/agents/handler.yaml + Write-Host " (no custom steps yet — edit scripts/post-provision.ps1)" -ForegroundColor DarkGray + + # ── Required: write .deployed/.json so the meta-sim sees you ── + $deployedDir = Join-Path (Split-Path $labRoot -Parent) '.deployed' + if (-not (Test-Path $deployedDir)) { New-Item -ItemType Directory -Path $deployedDir -Force | Out-Null } + $record = [ordered]@{ + name = '{{LAB_NAME}}' + deployedAt = (Get-Date).ToString('o') + subscriptionId = $sub + resourceGroup = $rg + region = $loc + # Add anything else your sim needs to know: + # sreAgentName = $agent + # portalUrl = $portalUrl + } + $record | ConvertTo-Json -Depth 5 | Set-Content (Join-Path $deployedDir '{{LAB_NAME}}.json') -Encoding utf8 + Write-Host " ✓ recorded in .deployed/{{LAB_NAME}}.json" + + if ($env:LAB_NO_AUTOLAUNCH) { + Write-Host "`n═══ Done — sim NOT auto-launched ═══" -ForegroundColor Green + } else { + Write-Host "`n═══ All set — launching simulator ═══" -ForegroundColor Green + # & python (Join-Path $labRoot 'simulator/demo.py') + } +} finally { Pop-Location } diff --git a/labs/_platform/template/scripts/scenarios/example.ps1.tmpl b/labs/_platform/template/scripts/scenarios/example.ps1.tmpl new file mode 100644 index 000000000..9f49054c3 --- /dev/null +++ b/labs/_platform/template/scripts/scenarios/example.ps1.tmpl @@ -0,0 +1,13 @@ +#requires -Version 7.0 +# Example scenario runner for {{LAB_DISPLAY_NAME}}. +$ErrorActionPreference = 'Stop' +$labRoot = Split-Path $PSScriptRoot -Parent | Split-Path -Parent +$labsRoot = Split-Path $labRoot -Parent + +$deployed = Get-Content (Join-Path $labsRoot '.deployed/{{LAB_NAME}}.json') | ConvertFrom-Json +$rg = $deployed.resourceGroup +$sub = $deployed.subscriptionId +$agent = $deployed.sreAgentName + +Write-Host "`n TODO: implement scenario logic for {{LAB_NAME}}" -ForegroundColor Yellow +Write-Host " sub=$sub rg=$rg agent=$agent`n" -ForegroundColor DarkGray diff --git a/labs/deployment-compliance/azure.yaml b/labs/deployment-compliance/azure.yaml deleted file mode 100644 index e9638e7bf..000000000 --- a/labs/deployment-compliance/azure.yaml +++ /dev/null @@ -1,11 +0,0 @@ -# yaml-language-server: $schema=https://raw.githubusercontent.com/Azure/azure-dev/main/schemas/v1.0/azure.yaml.json - -name: deployment-compliance-demo -metadata: - template: deployment-compliance-demo@1.0.0 - -# Infrastructure provisioned via Bicep (infra/main.bicep) -# azd up provisions: Resource Group, ACR, Container App Environment, -# Container App (placeholder image), SQL Server + DB, Log Analytics, -# Activity Log Alert, SRE Agent (Microsoft.App/agents), and role assignments. -# The Container App workload is deployed separately via GitHub Actions. diff --git a/labs/lab.ps1 b/labs/lab.ps1 new file mode 100644 index 000000000..b41f1da88 --- /dev/null +++ b/labs/lab.ps1 @@ -0,0 +1,223 @@ +#requires -Version 7.0 +<# +.SYNOPSIS + Top-level lab launcher. Auto-discovers any subdirectory with an azure.yaml + and lets you pick one (or several) to deploy. + +.EXAMPLE + ./lab.ps1 # interactive picker + ./lab.ps1 -Labs powergrid-zeroops # deploy one lab non-interactively + ./lab.ps1 -Labs powergrid-zeroops,itops # deploy multiple, no auto-launch + ./lab.ps1 -List # just list available labs + ./lab.ps1 -Down powergrid-zeroops # tear down a lab +#> +[CmdletBinding()] +param( + [string[]]$Labs, + [string] $Down, + [string] $New, + [switch] $List, + [switch] $NoLaunch +) +$ErrorActionPreference = 'Stop' + +# ── -New: scaffold a new lab from the template ── +if ($New) { + if ($New -notmatch '^[a-z][a-z0-9-]+$') { + Write-Host " ✗ lab name must be kebab-case (e.g. zava-fintech, my-lab)" -ForegroundColor Red; exit 1 + } + $target = Join-Path $PSScriptRoot $New + if (Test-Path $target) { Write-Host " ✗ '$New' already exists at $target" -ForegroundColor Red; exit 1 } + + Write-Host "`n═══ New lab: $New ═══`n" -ForegroundColor Cyan + $displayName = Read-Host " Display name (e.g. 'Zava Fintech — Trading Platform')" + if (-not $displayName) { $displayName = $New } + $subsidiary = Read-Host " Zava subsidiary (e.g. 'Zava Fintech') [optional]" + $description = Read-Host " One-sentence description" + if (-not $description) { $description = "TODO: describe what this lab demonstrates." } + $tagsCsv = Read-Host " Tags (comma-separated, e.g. aks,postgres,autoremediation) [optional]" + $tags = if ($tagsCsv) { ($tagsCsv -split '[, ]+' | Where-Object { $_ } | ForEach-Object { "'$_'" }) -join ', ' } else { '' } + + $tplRoot = Join-Path $PSScriptRoot '_platform/template' + $sub = @{ + '{{LAB_NAME}}' = $New + '{{LAB_DISPLAY_NAME}}' = $displayName + '{{LAB_SUBSIDIARY}}' = $subsidiary + '{{LAB_DESCRIPTION}}' = $description + '{{LAB_TAGS}}' = $tags + } + function Apply-Substitutions([string]$text) { + foreach ($k in $sub.Keys) { $text = $text.Replace($k, $sub[$k]) } + return $text + } + + # Copy + substitute + $count = 0 + Get-ChildItem $tplRoot -Recurse -File | ForEach-Object { + $rel = $_.FullName.Substring($tplRoot.Length).TrimStart('\','/') + # Strip .tmpl extension + if ($rel -like '*.tmpl') { $rel = $rel.Substring(0, $rel.Length - 5) } + $dest = Join-Path $target $rel + $destDir = Split-Path $dest -Parent + if (-not (Test-Path $destDir)) { New-Item -ItemType Directory -Path $destDir -Force | Out-Null } + $body = Apply-Substitutions (Get-Content $_.FullName -Raw -Encoding utf8) + Set-Content $dest $body -Encoding utf8 + $count++ + } + + Write-Host "`n ✓ scaffolded $count files at $target" -ForegroundColor Green + Write-Host "`n Next steps:" -ForegroundColor Cyan + Write-Host " 1. Edit $New/lab.yaml — add prereqs, prompts, scenarios" + Write-Host " 2. Edit $New/infra/main.bicep — add your Azure resources" + Write-Host " 3. Add scenario runners under $New/scripts/scenarios/" + Write-Host " 4. Validate: python _platform/helpers/manifest.py validate $New/lab.yaml" + Write-Host " 5. Deploy: ./lab.sh -Labs $New`n" + exit 0 +} + +# ── Discover labs (any sibling dir with lab.yaml or azure.yaml) ── +$helper = Join-Path $PSScriptRoot '_platform/helpers/manifest.py' +$useManifest = (Test-Path $helper) -and (Get-Command python -ErrorAction SilentlyContinue) + +if ($useManifest) { + try { + $rawJson = (& python $helper list $PSScriptRoot) -join "`n" + $available = ($rawJson | ConvertFrom-Json) | ForEach-Object { + [PSCustomObject]@{ + Name = $_.name; Path = $_._path + Description = $_.description + Manifest = $_ + IsLegacy = [bool]$_._legacy + } + } + } catch { + $useManifest = $false + Write-Host " ⚠ manifest helper failed ($_), falling back to azure.yaml discovery" -ForegroundColor DarkYellow + } +} + +if (-not $useManifest) { + $available = Get-ChildItem $PSScriptRoot -Directory | + Where-Object { (Test-Path (Join-Path $_.FullName 'azure.yaml')) -and -not $_.Name.StartsWith('_') } | + ForEach-Object { + $readme = Join-Path $_.FullName 'README.md' + $desc = if (Test-Path $readme) { + (Get-Content $readme -TotalCount 4 | Where-Object { $_ -and $_ -notmatch '^#' } | Select-Object -First 1) + } else { '(no description)' } + [PSCustomObject]@{ Name = $_.Name; Path = $_.FullName; Description = $desc; Manifest = $null; IsLegacy = $true } + } +} + +if ($available.Count -eq 0) { + Write-Host "`nNo labs found under $PSScriptRoot (need a child dir with azure.yaml).`n" -ForegroundColor Yellow + exit 1 +} + +# ── -List ── +if ($List) { + Write-Host "`nAvailable labs:" -ForegroundColor Cyan + $available | ForEach-Object { Write-Host (" • {0,-30} {1}" -f $_.Name, $_.Description) } + Write-Host ""; exit 0 +} + +# ── -Down ── +if ($Down) { + $target = $available | Where-Object Name -eq $Down + if (-not $target) { Write-Host "Unknown lab '$Down'. Use -List to see options." -ForegroundColor Red; exit 1 } + Push-Location $target.Path + try { azd down --purge --force } finally { Pop-Location } + exit 0 +} + +# ── Resolve / prompt for selection ── +if (-not $Labs) { + Write-Host "`n═══ SRE Agent labs ═══" -ForegroundColor Cyan + Write-Host "Which lab(s) do you want to deploy?`n" + for ($i = 0; $i -lt $available.Count; $i++) { + Write-Host (" [{0}] {1,-30} {2}" -f ($i+1), $available[$i].Name, $available[$i].Description) + } + Write-Host (" [a] all ({0} labs)" -f $available.Count) + Write-Host " [q] quit`n" + $pick = Read-Host "Pick (number, comma-separated for multiple, 'a' for all)" + if ($pick -eq 'q') { exit 0 } + if ($pick -eq 'a' -or $pick -eq 'all') { + $Labs = $available.Name + } else { + $idxs = $pick -split '[,\s]+' | Where-Object { $_ -match '^\d+$' } | ForEach-Object { [int]$_ - 1 } + $Labs = @($idxs | Where-Object { $_ -ge 0 -and $_ -lt $available.Count } | ForEach-Object { $available[$_].Name }) + if ($Labs.Count -eq 0) { Write-Host "No valid selection." -ForegroundColor Red; exit 1 } + } +} + +# ── Validate selections ── +$bad = $Labs | Where-Object { $_ -notin $available.Name } +if ($bad) { Write-Host "Unknown lab(s): $($bad -join ', '). Use -List." -ForegroundColor Red; exit 1 } + +# ── Multi-pick: skip in-postprovision sim auto-launch (interleaving sims is bad UX) ── +if ($Labs.Count -gt 1 -or $NoLaunch) { + $env:LAB_NO_AUTOLAUNCH = '1' + Write-Host "`n Multi-lab deploy: simulator auto-launch suppressed. Run it manually after.`n" -ForegroundColor DarkGray +} + +# ── Deploy each pick sequentially ── +$results = @() +foreach ($lab in $Labs) { + $target = $available | Where-Object Name -eq $lab | Select-Object -First 1 + Write-Host "`n┌─────────────────────────────────────────────" -ForegroundColor Cyan + Write-Host "│ Deploying $lab" -ForegroundColor Cyan + Write-Host "└─────────────────────────────────────────────`n" -ForegroundColor Cyan + + # If the lab declares prompts in its manifest, collect them upfront and stash in azd env. + # (azd hooks can also prompt — this is for labs whose preprovision hook reads from azd env.) + if (-not $target.IsLegacy -and $target.Manifest.prompts) { + Push-Location $target.Path + try { + $existing = (azd env get-values --output json 2>$null | ConvertFrom-Json) + foreach ($p in $target.Manifest.prompts) { + $present = $existing -and $existing.PSObject.Properties[$p.name] + if ($present) { continue } + $promptText = " $($p.text)" + if ($p.default) { $promptText += " [$($p.default)]" } + if ($p.secret) { + $val = Read-Host -AsSecureString $promptText + $val = [System.Net.NetworkCredential]::new('', $val).Password + } else { + $val = Read-Host $promptText + } + if (-not $val -and $p.default) { $val = $p.default } + if (-not $val -and -not $p.optional) { Write-Host " (skipped $($p.name))" -ForegroundColor DarkYellow; continue } + if ($val) { azd env set $p.name $val | Out-Null } + } + } finally { Pop-Location } + } + + Push-Location $target.Path + try { + # Unified prereq gate — fails fast if azd/az/srectl/login missing + & (Join-Path $PSScriptRoot '_platform/check-prereqs.ps1') -Lab $lab + if ($LASTEXITCODE -ne 0) { + Write-Host " ✗ prereq check failed for $lab — skipping" -ForegroundColor Red + $results += [PSCustomObject]@{ Lab = $lab; OK = $false; Path = $target.Path } + Pop-Location + continue + } + azd up + $ok = $LASTEXITCODE -eq 0 + $results += [PSCustomObject]@{ Lab = $lab; OK = $ok; Path = $target.Path } + } finally { Pop-Location } +} + +# ── Summary ── +Write-Host "`n═══ Summary ═══" -ForegroundColor Cyan +foreach ($r in $results) { + $mark = if ($r.OK) { '✓' } else { '✗' } + $color = if ($r.OK) { 'Green' } else { 'Red' } + Write-Host (" {0} {1}" -f $mark, $r.Lab) -ForegroundColor $color +} +if ($Labs.Count -gt 1) { + Write-Host "`nTo run any lab's simulator:" -ForegroundColor DarkGray + foreach ($r in $results | Where-Object OK) { + Write-Host " cd $($r.Lab) && python simulator/demo.py" -ForegroundColor DarkGray + } +} +Write-Host "" diff --git a/labs/lab.sh b/labs/lab.sh new file mode 100644 index 000000000..ed0906f53 --- /dev/null +++ b/labs/lab.sh @@ -0,0 +1,10 @@ +#!/usr/bin/env sh +# Top-level lab launcher (POSIX wrapper around lab.ps1). +# Requires pwsh 7+ — same prereq as the labs themselves. +set -e +DIR="$(cd "$(dirname "$0")" && pwd)" +if ! command -v pwsh >/dev/null 2>&1; then + echo "ERROR: pwsh (PowerShell 7+) required. https://aka.ms/powershell" >&2 + exit 1 +fi +exec pwsh -NoProfile -File "$DIR/lab.ps1" "$@" diff --git a/labs/recipes/README.md b/labs/recipes/README.md new file mode 100644 index 000000000..2b97344fe --- /dev/null +++ b/labs/recipes/README.md @@ -0,0 +1,89 @@ +# Recipes — portable Azure SRE Agent configs + +Lab-agnostic agent configuration bundles, derived from the `sre-config/` directories of the labs in this repo and shaped to match [`coreai-microsoft/sreagent-templates`](https://github.com/coreai-microsoft/sreagent-templates/blob/main/CONTRIBUTING.md). + +## What it's about + +A **recipe** is a portable, lab-agnostic SRE Agent config bundle — agent + subagents + skills + hooks + tools + connectors + incident filters — that you can apply to any workload that already matches the recipe's preconditions (e.g., "an ACA workload with App Insights and an Azure SQL DB"). It's the agent half of a lab, with the infra and app code stripped out. + +Recipes are distinct from labs: a **lab** in [`labs/`](../) ships full infra (Bicep), application source, demo scenarios, and a simulator — you `azd up` it from zero. A **recipe** assumes the workload already exists and only deploys the agent on top via `bin/new-agent.sh` + `bin/deploy.sh`. Use a recipe when you want the SRE Agent capabilities of a lab applied to your own production workload, without the demo app. + +## Stack + +Recipe matrix — every recipe targets Azure SRE Agent v2 (`api_version: azuresre.ai/v2`) and uses the upstream `sreagent-templates` deployment scripts (`bin/new-agent.sh`, `bin/deploy.sh`, `bin/verify-agent.sh`): + +| Recipe | Source lab | Agent class | Key tools / connectors | +|---|---|---|---| +| [`azmon-aca-servicenow-zavacafe-ops`](./azmon-aca-servicenow-zavacafe-ops/) | [`zava-cafe`](../zava-cafe/) | SQL ops + deployment validation (3 subagents, 4 skills, 2 hooks) | App Insights, Log Analytics, Azure Monitor, ServiceNow, Azure SQL MCP, ADO; `AssessChangeRisk` Python tool | +| [`azmon-aca-servicenow-zavapower-ops`](./azmon-aca-servicenow-zavapower-ops/) | [`zava-power`](../zava-power/) | Microservice ops (8 subagents, 15 skills) | App Insights, Log Analytics, Azure Monitor, ServiceNow; optional Datadog & Dynatrace MCP; ADO build/release tools | +| [`azmon-aca-servicenow-zavaitsupport`](./azmon-aca-servicenow-zavaitsupport/) | [`zava-itsupport`](../zava-itsupport/) | IT helpdesk laptop replacement (1 subagent, 0 skills) | ServiceNow Incident Platform; `CheckWarranty` + `LookupServiceNowIncident` Python tools; Browser Operator | + +## Contributing them upstream + +These recipes are authored to match the shape required by [`coreai-microsoft/sreagent-templates/CONTRIBUTING.md`](https://github.com/coreai-microsoft/sreagent-templates/blob/main/CONTRIBUTING.md). + +To open the PR: + +```bash +# 1. Clone the upstream recipe repo +git clone https://github.com/coreai-microsoft/sreagent-templates.git ~/work/sreagent-templates +cd ~/work/sreagent-templates + +# 2. Create a branch +git checkout -b zavapower-recipes + +# 3. Copy the two recipes in +cp -r /labs/recipes/azmon-aca-servicenow-zavapower-ops recipes/ +cp -r /labs/recipes/azmon-aca-servicenow-zavapower-itsupport recipes/ + +# 4. Local-test each recipe BEFORE pushing +./bin/new-agent.sh --recipe azmon-aca-servicenow-zavapower-itsupport --non-interactive \ + --set agentName=test-zavaitsupport \ + --set resourceGroup=rg-test-zava-itsupport \ + --set location=eastus2 \ + --set snowInstance=devXXXXXX \ + -o /tmp/test-zavaitsupport/ +./bin/deploy.sh /tmp/test-zavaitsupport/ +./bin/verify-agent.sh /tmp/test-zavaitsupport/ + +# 5. Same for the ops recipe (requires a running ACA workload + alerts) + +# 6. Commit + push + open PR +git add recipes/azmon-aca-servicenow-zavapower-{ops,itsupport} +git commit -m "feat: add Zava Power ops + IT-support recipes" +git push -u origin zavapower-recipes +gh pr create --fill +``` + +## Re-generating the ops recipe + +The ops recipe is generated by a converter that reads `labs/zava-power/sre-config/` and emits files in the recipe shape. If the lab evolves (new skills, new subagents), re-run: + +```bash +python labs/recipes/_convert_ops.py +``` + +The IT-support recipe is small enough to be hand-maintained — edit files directly under `azmon-aca-servicenow-zavapower-itsupport/`. + +## Custom tools + +Both recipes reference tool names that are **not** part of the recipe (they're part of the lab). Customers must upload them via the Builder UI before running. Sources: + +- `labs/zava-power/sre-config/tools/CheckWarranty/` (itsupport) +- `labs/zava-power/sre-config/tools/Lookup*ServiceNow*/` (both) +- `labs/zava-power/sre-config/tools/Upload*/` (ops only) +- `labs/zava-power/sre-config/tools/Generate*/` (ops only) +- `labs/zava-power/sre-config/tools/Python*/` (ops only) + +## Status + +| Step | ops | itsupport | +|---|---|---| +| Files authored | ✅ | ✅ | +| YAML shapes validated locally | ✅ | ✅ | +| `bin/new-agent.sh` smoke (requires upstream clone) | ⏳ | ⏳ | +| `bin/deploy.sh` to Azure | ⏳ | ⏳ | +| `bin/verify-agent.sh` green | ⏳ | ⏳ | +| PR opened | ⏳ | ⏳ | + +The ⏳ steps require an Azure subscription with SRE Agent RP + the upstream repo's `bin/` scripts. diff --git a/labs/recipes/_convert_ops.py b/labs/recipes/_convert_ops.py new file mode 100644 index 000000000..a8d70654b --- /dev/null +++ b/labs/recipes/_convert_ops.py @@ -0,0 +1,223 @@ +#!/usr/bin/env python3 +"""Convert labs/zava-power/sre-config/* -> recipes/azmon-aca-servicenow-zavapower-ops/. + +Re-runnable. Idempotent. Reads the live lab and emits files in the recipe shape +that coreai-microsoft/sreagent-templates expects. + +Usage: + python labs/recipes/_convert_ops.py +""" +from __future__ import annotations + +import re +import sys +from pathlib import Path + +try: + import yaml +except ImportError: + sys.exit("Install pyyaml: pip install pyyaml") + +ROOT = Path(__file__).resolve().parents[1] # labs/ +SRC = ROOT / "zava-power" / "sre-config" +DEST = ROOT / "recipes" / "azmon-aca-servicenow-zavapower-ops" + +# Subset of agents that go in the ops recipe (everything except it-support-handler) +OPS_AGENTS = [ + "incident-handler", + "deployment-validator", + "vm-ops-agent", + "utility-ops-agent", + "web-app-troubleshooter", + "pod-incident-remediator", + "release-orchestrator", + "pipeline-failure-investigator", +] + +FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n?(.*)$", re.DOTALL) + + +def write(path: Path, content: str) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(content, encoding="utf-8") + print(f" wrote {path.relative_to(ROOT)}") + + +def dump_yaml(data) -> str: + return yaml.safe_dump(data, sort_keys=False, default_flow_style=False, width=120) + + +# --------------------------------------------------------------------- skills +def convert_skills() -> list[str]: + skills_dir = SRC / "skills" + names: list[str] = [] + for skill_dir in sorted(skills_dir.iterdir()): + skill_md = skill_dir / "SKILL.md" + if not skill_md.exists(): + continue + text = skill_md.read_text(encoding="utf-8-sig").replace("\r\n", "\n") + m = FRONTMATTER_RE.match(text) + if not m: + print(f" skip (no frontmatter): {skill_dir.name}") + continue + try: + fm = yaml.safe_load(m.group(1)) or {} + except yaml.YAMLError as e: + print(f" skip (yaml error in {skill_dir.name}): {e.__class__.__name__}") + continue + body = m.group(2).lstrip("\n") + + name = fm.get("name") or skill_dir.name + # normalize name: lowercase, hyphens + name = name.strip().lower().replace(" ", "-") + description = (fm.get("description", "") or "").strip() + # Tools may be at top level or under metadata.spec + tools = fm.get("tools") or fm.get("metadata", {}).get("spec", {}).get("tools", []) or [] + + names.append(name) + + # skill yaml + skill_yaml = { + "metadata": { + "name": name, + "description": description, + "spec": {"tools": tools}, + }, + "skillContent": f"skills/{name}.md", + "additionalFiles": [], + } + write(DEST / "config" / "skills" / f"{name}.yaml", dump_yaml(skill_yaml)) + write(DEST / "config" / "skills" / f"{name}.md", body) + return names + + +# ----------------------------------------------------------------- subagents +def convert_subagents() -> list[str]: + names: list[str] = [] + for agent_name in OPS_AGENTS: + agent_yaml = SRC / "agents" / agent_name / f"{agent_name}.yaml" + if not agent_yaml.exists(): + print(f" WARN: {agent_yaml} not found") + continue + data = yaml.safe_load(agent_yaml.read_text(encoding="utf-8")) or {} + spec = data.get("spec", {}) + + # extract instructions to its own .md file + instructions = spec.get("instructions", "").rstrip() + "\n" + write( + DEST / "config" / "subagents" / f"{agent_name}.instructions.md", + instructions, + ) + + sub_yaml = { + "metadata": {"name": agent_name}, + "spec": { + "instructions": f"subagents/{agent_name}.instructions.md", + "handoffDescription": spec.get("handoffDescription", "") or "", + "tools": spec.get("tools", []) or [], + "agentType": "Autonomous", + "temperature": 0.2, + "handoffs": spec.get("handoffs", []) or [], + "enableSkills": bool(spec.get("enableSkills", False)), + "allowedSkills": [], # filled in after we know skill names + }, + } + names.append(agent_name) + write(DEST / "config" / "subagents" / f"{agent_name}.yaml", dump_yaml(sub_yaml)) + return names + + +# ---------------------------------------------------------------- automations +def write_automations() -> list[str]: + # ServiceNow incident-platform + write( + DEST / "automations" / "incident-platforms" / "servicenow.yaml", + dump_yaml({"name": "servicenow", "spec": {"platformType": "ServiceNow"}}), + ) + # AzureMonitor incident-platform + write( + DEST / "automations" / "incident-platforms" / "azure-monitor.yaml", + dump_yaml({"name": "azure-monitor", "spec": {"platformType": "AzureMonitor"}}), + ) + # Incident filter — auto-investigate (matches the lab response-plan) + incident_filter = { + "metadata": {"name": "auto-investigate-azmon"}, + "spec": { + "incidentPlatform": "AzureMonitor", + "isEnabled": True, + "priorities": ["1", "2", "3"], + "incidentType": "LiveSite", + "handlingAgent": "incident-handler", + "agentMode": "Autonomous", + "deepInvestigationEnabled": False, + "maxAutomatedInvestigationAttempts": 3, + "azureMonitorFilterSettings": { + "alertRules": [ + "alert-powergrid-http-5xx", + "alert-powergrid-high-latency", + "alert-powergrid-container-restart", + ], + "triggerEvents": ["AlertFired"], + }, + }, + } + write( + DEST / "automations" / "incident-filters" / "auto-investigate-azmon.yaml", + dump_yaml(incident_filter), + ) + + # Scheduled task — pod fleet audit (daily) + sched = { + "metadata": {"name": "pod-fleet-audit-daily"}, + "spec": { + "agent": "utility-ops-agent", + "cronExpression": "0 8 * * *", + "isEnabled": True, + "agentPrompt": ( + "Run the pod-fleet-audit-deck skill end-to-end. " + "Window: last 48 hours (UTC). Scope: all Container Apps in the target " + "resource group. Output: ONE .pptx deck attached to this thread plus a " + "one-paragraph executive summary. HARD CONSTRAINTS: do not create/modify " + "ServiceNow incidents, do not run remediation, do not run incident-handler " + "phases — only the deck workflow defined in the skill." + ), + }, + } + write( + DEST / "automations" / "scheduled-tasks" / "pod-fleet-audit-daily.yaml", + dump_yaml(sched), + ) + + return ["auto-investigate-azmon"], ["pod-fleet-audit-daily"] + + +def main() -> None: + print(f"Source: {SRC}") + print(f"Destination: {DEST}\n") + if not SRC.exists(): + sys.exit(f"sre-config not found: {SRC}") + + print("== skills ==") + skill_names = convert_skills() + + print("\n== subagents ==") + subagent_names = convert_subagents() + + # backfill allowedSkills on every subagent — simplest policy: allow all + for name in subagent_names: + p = DEST / "config" / "subagents" / f"{name}.yaml" + data = yaml.safe_load(p.read_text(encoding="utf-8")) + data["spec"]["allowedSkills"] = sorted(skill_names) + if skill_names: + data["spec"]["enableSkills"] = True + p.write_text(dump_yaml(data), encoding="utf-8") + + print("\n== automations ==") + incident_filters, scheduled_tasks = write_automations() + + print(f"\nDone. {len(skill_names)} skills, {len(subagent_names)} subagents, " + f"{len(incident_filters)} incident-filters, {len(scheduled_tasks)} scheduled-tasks.") + + +if __name__ == "__main__": + main() diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/.gitignore b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/.gitignore new file mode 100644 index 000000000..5af9c4464 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/.gitignore @@ -0,0 +1,8 @@ +# Local secrets — never commit +connectors.secrets.env + +# Local data assets +data/ + +# Generated agent configs (output of bin/new-agent.sh) +output/ diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/README.md b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/README.md new file mode 100644 index 000000000..7fec339a2 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/README.md @@ -0,0 +1,152 @@ +# azmon-aca-servicenow-zavacafe-ops + +SRE agent for the **Zava Café** demo workload — an ASP.NET app on Azure Container Apps backed by Azure SQL Database. The primary subagent triages SQL performance incidents (DTU spikes, slow queries, blocking chains) end-to-end: diagnose → assess risk → request approval → apply the fix → verify. Two deployment-validator subagents handle post-deploy health checks (Azure DevOps and GitHub Actions paths). + +## Stack + +- **App** (target workload): .NET 8 / ASP.NET Core (Zava Café e-commerce storefront, sourced from [`labs/zava-cafe/`](../../zava-cafe/)) +- **Compute**: Azure Container Apps (or any compute that exposes the workload's metrics + logs to App Insights / LAW) +- **Data**: Azure SQL Database (FQDN + DB name injected into skills via `${AZURE_SQL_SERVER_FQDN}` / `${AZURE_SQL_DATABASE}`) +- **Observability**: Application Insights, Log Analytics, Azure Monitor (alert rules `alert-zavacafe-sql-dtu`, `alert-zavacafe-sql-blocking`, `alert-zavacafe-http-5xx`) +- **SRE Agent**: 3 subagents (`sql-performance-investigator`, `deployment-validator`, `deployment-validator-gh`); 4 skills (`sql-query-diagnosis`, `sql-performance-fix`, `sql-blocking-diagnosis`, `sql-blocking-fix`); 2 hooks (`change-risk-assessor`, `sql-write-guard`); 1 custom Python tool (`AssessChangeRisk`); incident filter `auto-investigate-azmon`; weekly `weekly-cost-report` scheduled task. Connectors: App Insights, Log Analytics, Azure Monitor, ServiceNow, Azure SQL MCP, optional ADO. +- **Simulator**: None — this is the agent half of the lab; pair with [`labs/zava-cafe/`](../../zava-cafe/) (and its `simulate-dtu-spike.ps1` / `simulate-slow-queries.ps1`) for the full break/fix experience +- **CI/CD**: Upstream `sreagent-templates` deployment scripts — `bin/new-agent.sh --recipe ...` → `bin/deploy.sh` → `bin/verify-agent.sh` + +## What it's about + +This recipe is the **portable, lab-agnostic agent half of [`labs/zava-cafe/`](../../zava-cafe/)** — the SQL-ops + deployment-validation SRE Agent config, packaged in the shape required by [`coreai-microsoft/sreagent-templates`](https://github.com/coreai-microsoft/sreagent-templates) so customers can drop it onto their own ACA + Azure SQL workload without taking the lab's infra or app code. The recipe assumes the workload already exists with App Insights, a Log Analytics workspace, an Azure SQL DB, and the 3 expected alert rules; you supply those resource IDs as parameters and the recipe wires the agent on top. + +The recipe targets PMs, SREs, and customers who want to apply Zava Café's SQL break/fix patterns — DTU spike, slow query / missing index, blocking-chain head-blocker analysis, deployment regression rollback — to a real production workload. Demo flow: `bin/new-agent.sh --recipe azmon-aca-servicenow-zavacafe-ops --non-interactive --set ...` → `bin/deploy.sh` → connect ServiceNow as the Incident Platform in the SRE Agent UI → an AzMon alert fires → the `sql-performance-investigator` runs the matching diagnosis + fix skill, scores the change with `AssessChangeRisk`, asks for approval via `AskUserQuestion`, and documents everything in a ServiceNow incident. + +## What it does + +- **AzMon alert fires** (e.g. `alert-zavacafe-sql-dtu`) → routed to `sql-performance-investigator` +- The subagent runs the right `sql-*-diagnosis` skill, plots a chart, then runs the matching `sql-*-fix` skill +- The fix skill calls `AssessChangeRisk` (a Python tool) to score the change +- The `change-risk-assessor` hook + `sql-write-guard` hook gate destructive ops and force human approval via `AskUserQuestion` +- All work is documented in a ServiceNow incident +- After a release, `deployment-validator` (ADO trigger) or `deployment-validator-gh` (GH Actions trigger) hits `/health`, pulls the commit diff, and rolls back automatically if broken + +## Prereqs + +- Azure subscription with SRE Agent RP access +- Azure Container Apps environment running the Zava Café workload +- Azure SQL Database (the recipe will inject the FQDN + DB name into skills via `${AZURE_SQL_SERVER_FQDN}` / `${AZURE_SQL_DATABASE}`) +- Application Insights + Log Analytics workspace +- ServiceNow instance (PDI is fine for demos) +- Azure Monitor alert rules created against the workload — at minimum: + - `alert-zavacafe-sql-dtu` — DTU > 80% for 5 min + - `alert-zavacafe-sql-blocking` — blocked sessions > 0 for 2 min + - `alert-zavacafe-http-5xx` — 5xx rate > 1% for 5 min +- (Optional) Azure DevOps org URL + PAT for change-risk pipeline lookups + +## Quick start + +```bash +./bin/new-agent.sh --recipe azmon-aca-servicenow-zavacafe-ops --non-interactive \ + --set agentName=zavacafe-ops \ + --set resourceGroup=rg-zavacafe-ops \ + --set location=eastus2 \ + --set WORKLOAD_NAME=zava-cafe \ + --set AZURE_RESOURCE_GROUP=rg-zava-cafe \ + --set AZURE_SQL_SERVER_FQDN=sql-zavacafe.database.windows.net \ + --set AZURE_SQL_DATABASE=zava \ + --set ALERT_EMAIL=oncall@example.com \ + --set appInsightsId=/subscriptions//resourceGroups/rg-zava-cafe/providers/Microsoft.Insights/components/appi-zavacafe \ + --set appInsightsAppId= \ + --set lawResourceId=/subscriptions//resourceGroups/rg-zava-cafe/providers/Microsoft.OperationalInsights/workspaces/log-zavacafe \ + --set snowInstance=dev123456 \ + --set ADO_ORG_URL=https://dev.azure.com/myorg \ + -o zavacafe-ops/ + +./bin/deploy.sh zavacafe-ops/ +./bin/verify-agent.sh zavacafe-ops/ +``` + +## Parameters + +| Param | Required | Example | +|---|---|---| +| `agentName` | ✅ | `zavacafe-ops` | +| `resourceGroup` | ✅ | `rg-zavacafe-ops` | +| `location` | ✅ | `eastus2` | +| `WORKLOAD_NAME` | ⛔ | `zava-cafe` | +| `AZURE_RESOURCE_GROUP` | ✅ | `rg-zava-cafe` | +| `AZURE_SQL_SERVER_FQDN` | ✅ | `sql-zavacafe.database.windows.net` | +| `AZURE_SQL_DATABASE` | ⛔ | `zava` | +| `ALERT_EMAIL` | ✅ | `oncall@example.com` | +| `appInsightsId` | ✅ | App Insights resource ID | +| `appInsightsAppId` | ✅ | App Insights App ID GUID | +| `lawResourceId` | ✅ | Log Analytics workspace resource ID | +| `snowInstance` | ✅ | `dev123456` (PDI subdomain) | +| `ADO_ORG_URL` | ⛔ | Leave blank to skip ADO lookups | + +`ADO_PAT` goes in `connectors.secrets.env` — pasted into the agent UI when the AssessChangeRisk tool is first invoked. + +## What gets deployed + +### Subagents (3) +| Name | Role | +|---|---| +| `sql-performance-investigator` | Primary AzMon-triggered SQL incident handler. Diagnoses + fixes DTU / blocking issues. | +| `deployment-validator` | Post-ADO-release health check + rollback. | +| `deployment-validator-gh` | Post-GitHub-Actions-deploy health check + rollback + PR. | + +### Skills (4) +- `sql-query-diagnosis` — slow queries, missing indexes +- `sql-performance-fix` — `CREATE INDEX` / `UPDATE STATISTICS` (gated) +- `sql-blocking-diagnosis` — head-blocker + impact analysis +- `sql-blocking-fix` — `KILL ` (gated) + +### Hooks (2) +- `change-risk-assessor` — AI-powered PostToolUse hook that scores SQL writes and forces approval +- `sql-write-guard` — deterministic Python hook that blocks `DROP / DELETE / TRUNCATE / ALTER` + +### Tools (1) +- `AssessChangeRisk` — Python tool the fix-skills call before mutating SQL + +### Automations +- **Incident filter** `auto-investigate-azmon` — routes the 3 alert rules to `sql-performance-investigator` +- **Scheduled task** `weekly-cost-report` — Mondays 09:00 UTC, summarises last-7-days Azure spend for `${AZURE_RESOURCE_GROUP}` + +## Incident Platform setup (post-deploy) + +In the SRE Agent UI for the deployed agent: + +1. **Builder → Incidents → Connect platform → Azure Monitor** — done automatically by `connectors.json`. +2. **Builder → Incidents → Connect platform → ServiceNow** — enter `https://.service-now.com` + admin user/password. + +## ServiceNow setup + +The `sql-performance-investigator` and the deployment-validators write their work-notes to ServiceNow. You need: + +- A user account in ServiceNow with `incident_manager` / `itil` role +- The native ServiceNow tools (`CreateServiceNowIncident`, `UpdateServiceNowWorkNotes`, `ResolveServiceNowIncident`, etc.) become available automatically once you connect the Incident Platform. + +## Custom tools the agent depends on + +The recipe ships `AssessChangeRisk`. The following are **not** in the recipe and must be uploaded via Builder before first run (or they come from the SRE Agent platform once the Incident Platform is connected): + +- `CreateServiceNowIncident`, `UpdateServiceNowWorkNotes`, `ResolveServiceNowIncident`, `LookupServiceNowIncident` +- `PlotBarChart`, `PlotPieChart`, `PlotScatter`, `AskUserQuestion` (built-in) +- `zava-mssql_*` MCP tools (the SQL skills depend on a registered Azure SQL MCP server — connect it under Builder → Connectors → MCP) +- `github-mcp.*` (for `deployment-validator-gh`) + +## Verifying + +```bash +./bin/verify-agent.sh zavacafe-ops/ +``` + +Should report: +- 3 subagents present +- 4 skills present +- 2 hooks present +- 1 custom tool (`AssessChangeRisk`) +- 1 scheduled task (`weekly-cost-report`) +- 1 incident filter (`auto-investigate-azmon`) +- Connectors: app-insights, log-analytics, azure-monitor + +## Cost + +Monthly Agent Unit cap = 15000 in `agent.json`. Tune based on incident + release volume. diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/agent.json b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/agent.json new file mode 100644 index 000000000..fd3a72c54 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/agent.json @@ -0,0 +1,107 @@ +{ + "_scenario": "azmon-aca-servicenow-zavacafe-ops", + "_description": "SRE agent for the Zava Café demo workload — an ASP.NET app on Azure Container Apps backed by Azure SQL. The primary subagent is a SQL performance investigator that triages DTU spikes, slow queries, and blocking chains, then applies fixes (CREATE INDEX / KILL) gated by an AI-powered change-risk hook with human-in-the-loop approval. Two deployment-validator subagents validate releases (Azure DevOps and GitHub Actions paths). Native ServiceNow + Azure Monitor incident platforms.", + "_prerequisites": [ + "Azure subscription with SRE Agent RP access", + "Azure Container Apps environment running the Zava Café workload", + "Azure SQL Database (server FQDN + database name)", + "Application Insights resource for the workload", + "Log Analytics workspace", + "Azure Monitor alert rules (recipe expects: alert-zavacafe-sql-dtu, alert-zavacafe-http-5xx, alert-zavacafe-sql-blocking)", + "ServiceNow instance for incident documentation", + "(Optional) Azure DevOps org + PAT for AssessChangeRisk + change-risk hook" + ], + "_prompts": { + "agentName": { + "ask": "Agent name", + "default": "zavacafe-ops-agent" + }, + "resourceGroup": { + "ask": "Resource group (where the agent will live)", + "default": "rg-zavacafe-ops" + }, + "location": { + "ask": "Region", + "options": ["eastus2", "swedencentral", "uksouth", "australiaeast"], + "default": "eastus2", + "required": true + }, + "WORKLOAD_NAME": { + "ask": "Workload short name (used in tags + dashboard titles)", + "default": "zava-cafe" + }, + "AZURE_RESOURCE_GROUP": { + "ask": "Workload resource group (the one running the ACA app + SQL)", + "required": true + }, + "AZURE_SQL_SERVER_FQDN": { + "ask": "Azure SQL server FQDN (e.g. sql-zavacafe.database.windows.net)", + "required": true + }, + "AZURE_SQL_DATABASE": { + "ask": "Azure SQL database name", + "default": "zava" + }, + "ALERT_EMAIL": { + "ask": "Email for high-severity escalations (action group + summary recipient)", + "required": true + }, + "appInsightsId": { + "ask": "Application Insights resource ID for the workload", + "required": true + }, + "appInsightsAppId": { + "ask": "Application Insights App ID (GUID)", + "required": true + }, + "lawResourceId": { + "ask": "Log Analytics workspace resource ID", + "required": true + }, + "snowInstance": { + "ask": "ServiceNow instance hostname (e.g. dev123456 — without .service-now.com)", + "required": true + }, + "ADO_ORG_URL": { + "ask": "Azure DevOps org URL for AssessChangeRisk (e.g. https://dev.azure.com/myorg). Leave blank to skip ADO change-risk lookups.", + "default": "" + }, + "modelProvider": { + "ask": "AI model provider", + "options": ["Anthropic", "GitHubCopilot", "MicrosoftFoundry"], + "default": "Anthropic" + }, + "existingUamiId": { + "ask": "Existing UAMI resource ID (leave blank to create new)", + "default": "" + }, + "existingAgentAppInsightsId": { + "ask": "Existing App Insights resource ID for agent telemetry (leave blank to create new)", + "default": "" + } + }, + "identity": { + "agentName": "{{agentName}}", + "resourceGroup": "{{resourceGroup}}", + "subscription": "", + "location": "{{location}}", + "targetResourceGroups": "{{AZURE_RESOURCE_GROUP}}" + }, + "access": { + "accessLevel": "High", + "actionMode": "Autonomous" + }, + "upgradeChannel": "Preview", + "defaultModelProvider": "{{modelProvider}}", + "monthlyAgentUnitLimit": 15000, + "tags": { + "scenario": "zavacafe-ops", + "workload": "{{WORKLOAD_NAME}}" + }, + "toggles": { + "enableWebhookBridge": true, + "webhookBridgeTriggerUrl": "" + }, + "existingUamiId": "{{existingUamiId}}", + "existingAgentAppInsightsId": "{{existingAgentAppInsightsId}}" +} diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/automations/incident-filters/auto-investigate-azmon.yaml b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/automations/incident-filters/auto-investigate-azmon.yaml new file mode 100644 index 000000000..e801798c2 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/automations/incident-filters/auto-investigate-azmon.yaml @@ -0,0 +1,21 @@ +metadata: + name: auto-investigate-azmon +spec: + incidentPlatform: AzureMonitor + isEnabled: true + priorities: + - '1' + - '2' + - '3' + incidentType: LiveSite + handlingAgent: sql-performance-investigator + agentMode: Autonomous + deepInvestigationEnabled: false + maxAutomatedInvestigationAttempts: 3 + azureMonitorFilterSettings: + alertRules: + - alert-zavacafe-sql-dtu + - alert-zavacafe-sql-blocking + - alert-zavacafe-http-5xx + triggerEvents: + - AlertFired diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/automations/incident-platforms/azure-monitor.yaml b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/automations/incident-platforms/azure-monitor.yaml new file mode 100644 index 000000000..876551e04 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/automations/incident-platforms/azure-monitor.yaml @@ -0,0 +1,3 @@ +name: azure-monitor +spec: + platformType: AzureMonitor diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/automations/incident-platforms/servicenow.yaml b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/automations/incident-platforms/servicenow.yaml new file mode 100644 index 000000000..1d38ac48e --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/automations/incident-platforms/servicenow.yaml @@ -0,0 +1,3 @@ +name: servicenow +spec: + platformType: ServiceNow diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/automations/scheduled-tasks/weekly-cost-report.yaml b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/automations/scheduled-tasks/weekly-cost-report.yaml new file mode 100644 index 000000000..5a6fd7bac --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/automations/scheduled-tasks/weekly-cost-report.yaml @@ -0,0 +1,15 @@ +metadata: + name: weekly-cost-report +spec: + agent: sql-performance-investigator + cronExpression: 0 9 * * 1 + isEnabled: true + agentPrompt: |- + Analyze Azure costs for resource group ${AZURE_RESOURCE_GROUP} (workload ${WORKLOAD_NAME}) over the past 7 days. + Break down costs by resource type. Identify the top 3 cost drivers. Flag anomalies vs the prior week. + Provide 3 concrete recommendations to reduce costs (e.g. SQL DTU tier right-sizing, ACA replica scale rules, + Log Analytics retention). + Format as a management report with sections: Executive Summary, Cost Breakdown, Anomalies, Recommendations. + Email the report to ${ALERT_EMAIL} when complete. + HARD CONSTRAINTS: read-only — do NOT create/modify ServiceNow incidents, do NOT run any SQL fix skill, + do NOT mutate any Azure resource. diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/hooks/change-risk-assessor.yaml b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/hooks/change-risk-assessor.yaml new file mode 100644 index 000000000..81945e4b7 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/hooks/change-risk-assessor.yaml @@ -0,0 +1,40 @@ +api_version: azuresre.ai/v2 +kind: HookV2 +metadata: + name: change-risk-assessor +spec: + eventType: PostToolUse + activationMode: always + description: AI-powered risk assessment for SQL operations on ${AZURE_SQL_DATABASE}. Evaluates blast radius, business hours, and data sensitivity. Supports human-in-the-loop approval override. + hook: + type: prompt + matcher: ".*create_index.*|.*update_data.*|.*delete_data.*|.*insert_data.*" + timeout: 30 + model: ReasoningFast + prompt: | + You are a production change risk assessor for ${WORKLOAD_NAME}, an enterprise application on Azure Container Apps backed by Azure SQL (${AZURE_SQL_SERVER_FQDN}/${AZURE_SQL_DATABASE}). + + An SRE Agent is about to execute a database operation. Evaluate the risk. + + Context: + $ARGUMENTS + + Evaluate based on these criteria: + 1. **Operation type**: CREATE INDEX is medium risk. DELETE/UPDATE are high risk. KILL requires careful evaluation of what session is being killed. + 2. **Business hours**: If current time is between 6 AM - 10 PM Pacific (business hours), schema changes and write operations should require human approval. Off-hours changes are safer. + 3. **Blast radius**: Does this affect a critical table (Orders, Payments, Users)? Or a low-risk table (Logs, Temp, Analytics)? + 4. **Data volume**: Operations on tables with many rows are riskier. + 5. **Approval override**: If the conversation context indicates the user has explicitly approved this specific operation (e.g., user said "go ahead", "approved", "yes proceed", "Approve Now", or selected an approval option via AskUserQuestion), then allow it regardless of other criteria. + + Respond with ONLY a JSON object: + - {"ok": true} if the operation is safe to proceed OR if the user has given explicit approval + - {"ok": false, "reason": "Your explanation of why this should be blocked and what alternatives exist"} + + When blocking, always suggest actionable alternatives: + - "Schedule this CREATE INDEX for the next maintenance window (2-6 AM Pacific), or approve to proceed now with the understanding this may briefly impact query performance." + - "This KILL targets a batch job session. Terminating it will abort the batch. Approve if the batch can be safely restarted, or wait for it to complete." + + Examples: + - CREATE INDEX on Products at 2 AM -> {"ok": true} + - CREATE INDEX on Products at 2 PM, no prior approval -> {"ok": false, "reason": "Schema change during business hours (2 PM Pacific). This CREATE INDEX may briefly lock the Products table. Please approve to proceed or schedule for maintenance window (2-6 AM)."} + - CREATE INDEX on Products at 2 PM, user selected Approve Now -> {"ok": true} diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/hooks/sql-write-guard.yaml b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/hooks/sql-write-guard.yaml new file mode 100644 index 000000000..87f978b62 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/hooks/sql-write-guard.yaml @@ -0,0 +1,30 @@ +api_version: azuresre.ai/v2 +kind: HookV2 +metadata: + name: sql-write-guard +spec: + eventType: PostToolUse + activationMode: always + description: Blocks destructive SQL operations (DROP, DELETE, TRUNCATE, ALTER) against ${AZURE_SQL_DATABASE} but allows safe DDL (CREATE INDEX) and session management (KILL). + hook: + type: command + matcher: ".*sql.*|.*SQL.*|.*mssql.*" + timeout: 30 + failMode: Allow + script: | + #!/usr/bin/env python3 + import sys, json, re + context = json.load(sys.stdin) + tool_input = context.get('tool_input', {}) + query = '' + if isinstance(tool_input, dict): + query = ' '.join(str(v) for v in tool_input.values()) + elif isinstance(tool_input, str): + query = tool_input + safe_ops = re.search(r'\b(CREATE\s+INDEX|KILL|UPDATE\s+STATISTICS|SELECT)\b', query, re.IGNORECASE) + dangerous = re.search(r'\b(DROP\s+TABLE|DROP\s+DATABASE|TRUNCATE|DELETE\s+FROM|ALTER\s+TABLE)\b', query, re.IGNORECASE) + if dangerous and not safe_ops: + output = {"decision": "block", "reason": f"Blocked destructive operation: {dangerous.group().upper()}. Only read queries, CREATE INDEX, KILL, and UPDATE STATISTICS are allowed."} + else: + output = {"decision": "allow"} + print(json.dumps(output)) diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-blocking-diagnosis.md b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-blocking-diagnosis.md new file mode 100644 index 000000000..df25e026a --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-blocking-diagnosis.md @@ -0,0 +1,29 @@ +# SQL Blocking Diagnosis + +## Overview +Investigate SQL blocking chains on `${AZURE_SQL_DATABASE}` (server `${AZURE_SQL_SERVER_FQDN}`). Identify the head blocker and impact but DO NOT kill any sessions. + +## When to Use +- App is hanging or not responding +- API requests are timing out across the board +- Users report the app is frozen +- `alert-zavacafe-sql-blocking` has fired + +## Steps + +1. **Connect** using `zava-mssql_mssql_connect_database` with server=`${AZURE_SQL_SERVER_FQDN}` and database=`${AZURE_SQL_DATABASE}`. + +2. **Check for active blocking**: + `SELECT r.session_id AS blocked, r.blocking_session_id AS blocker, r.wait_type, r.wait_time/1000 AS wait_sec, t.text AS blocked_query FROM sys.dm_exec_requests r CROSS APPLY sys.dm_exec_sql_text(r.sql_handle) t WHERE r.blocking_session_id > 0 ORDER BY r.wait_time DESC` + +3. **Identify the head blocker** — the session blocking others but not blocked itself: + `SELECT s.session_id, s.login_name, s.host_name, s.program_name, r.command, r.status, t.text AS current_query FROM sys.dm_exec_sessions s LEFT JOIN sys.dm_exec_requests r ON s.session_id = r.session_id OUTER APPLY sys.dm_exec_sql_text(r.sql_handle) t WHERE s.session_id IN (SELECT DISTINCT blocking_session_id FROM sys.dm_exec_requests WHERE blocking_session_id > 0)` + +4. **Assess impact**: How many sessions blocked? How long waiting? Is it a batch job or user query? + +5. **Report findings**: head blocker SPID, program name, how many blocked, total wait time. Use `PlotBarChart` to show blocked session wait times. + +6. **Hand off** to `sql-blocking-fix` to resolve. + +## MCP Tools +- `zava-mssql_mssql_execute_query` — query DMVs diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-blocking-diagnosis.yaml b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-blocking-diagnosis.yaml new file mode 100644 index 000000000..ed69c3eb5 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-blocking-diagnosis.yaml @@ -0,0 +1,11 @@ +metadata: + name: sql-blocking-diagnosis + description: Diagnose SQL blocking chains that cause application hangs. Identifies head blocker, blocked sessions, wait times. This skill ONLY investigates. Hand off to sql-blocking-fix to resolve. + spec: + tools: + - zava-mssql_mssql_execute_query + - zava-mssql_mssql_run_sql_query + - zava-mssql_mssql_connect_database + - PlotBarChart +skillContent: skills/sql-blocking-diagnosis.md +additionalFiles: [] diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-blocking-fix.md b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-blocking-fix.md new file mode 100644 index 000000000..48b70c2f6 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-blocking-fix.md @@ -0,0 +1,31 @@ +# SQL Blocking Fix + +## Overview +Resolve SQL blocking chains on `${AZURE_SQL_DATABASE}` (server `${AZURE_SQL_SERVER_FQDN}`) by killing the head blocker session, after risk assessment and human approval. + +## When to Use +- AFTER `sql-blocking-diagnosis` has identified the head blocker +- You know which session to kill and its impact + +## Steps + +1. **Assess the risk** — call `AssessChangeRisk` with: + - operation: "KILL" + - table_name: the table being blocked + - row_count: number of blocked sessions + - description: who the blocker is (program_name, login_name) and what it is doing + +2. **If the hook blocks**: Use `AskUserQuestion`: + - header: "Approval" + - question: Blocker details — SPID, program name, what query it is running, how many sessions are blocked + - options: + - label: "Kill Session", description: "Terminate the blocking session. Blocked queries will resume." + - label: "Wait 5 Minutes", description: "Give the blocker time to complete naturally." + - label: "Cancel", description: "Do not kill. App will remain hung." + +3. **If approved**: Execute `KILL {session_id}` using `zava-mssql_mssql_execute_query`. + +4. **Verify** blocking is resolved: + `SELECT COUNT(*) as still_blocked FROM sys.dm_exec_requests WHERE blocking_session_id > 0` + +5. **Report**: blocker identity, how many unblocked, total wait time resolved. diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-blocking-fix.yaml b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-blocking-fix.yaml new file mode 100644 index 000000000..5baece30c --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-blocking-fix.yaml @@ -0,0 +1,12 @@ +metadata: + name: sql-blocking-fix + description: Resolve SQL blocking by killing the head blocker session. ALWAYS run AssessChangeRisk first and get user approval before killing any session. Use AFTER sql-blocking-diagnosis has identified the blocker. + spec: + tools: + - zava-mssql_mssql_execute_query + - zava-mssql_mssql_run_sql_query + - zava-mssql_mssql_connect_database + - AssessChangeRisk + - AskUserQuestion +skillContent: skills/sql-blocking-fix.md +additionalFiles: [] diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-performance-fix.md b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-performance-fix.md new file mode 100644 index 000000000..ee8d9fcaf --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-performance-fix.md @@ -0,0 +1,38 @@ +# SQL Performance Fix + +## Overview +Apply fixes for SQL performance issues identified by `sql-query-diagnosis` on `${AZURE_SQL_DATABASE}` (server `${AZURE_SQL_SERVER_FQDN}`). This skill handles risk assessment, human approval, and execution of the fix. + +## When to Use +- AFTER `sql-query-diagnosis` has identified a missing index or stale statistics +- You know exactly what fix to apply + +## Steps + +1. **Assess the risk** — call `AssessChangeRisk` with: + - operation: the SQL operation (e.g. "CREATE INDEX") + - table_name: the target table + - row_count: number of rows in the table + - description: what this change does and why + +2. **If the hook blocks** (risk is MEDIUM or HIGH): Use `AskUserQuestion` to present the risk assessment: + - header: "Approval" + - question: Full risk details — risk level, business hours status, table criticality, row count + - options: + - label: "Approve Now", description: "Proceed with the change. Risk factors have been reviewed." + - label: "Schedule for 2 AM", description: "Defer to maintenance window (2-6 AM Pacific)." + - label: "Cancel", description: "Do not proceed." + +3. **If approved**: Execute the fix using `zava-mssql_mssql_execute_query`: + - For indexes: `CREATE INDEX IX_{Table}_{Column} ON {Table}({Column})` + - For statistics: `UPDATE STATISTICS {Table} WITH FULLSCAN` + +4. **Verify** the fix worked — re-run the original slow query and compare duration. + +5. **Visualize** the before/after using `PlotBarChart`. + +## Example +- AssessChangeRisk("CREATE INDEX", "Products", 153600, "Add index on Category for slow queries") +- Hook blocks → AskUserQuestion → user approves +- Execute: `CREATE INDEX IX_Products_Category ON Products(Category)` +- Verify: 1,200ms → 60ms diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-performance-fix.yaml b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-performance-fix.yaml new file mode 100644 index 000000000..49d33dabd --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-performance-fix.yaml @@ -0,0 +1,13 @@ +metadata: + name: sql-performance-fix + description: Apply a fix for a diagnosed SQL performance issue. ALWAYS run AssessChangeRisk first and get user approval before making changes. Use AFTER sql-query-diagnosis has identified the root cause. + spec: + tools: + - zava-mssql_mssql_execute_query + - zava-mssql_mssql_run_sql_query + - zava-mssql_mssql_connect_database + - AssessChangeRisk + - AskUserQuestion + - PlotBarChart +skillContent: skills/sql-performance-fix.md +additionalFiles: [] diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-query-diagnosis.md b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-query-diagnosis.md new file mode 100644 index 000000000..207d6130e --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-query-diagnosis.md @@ -0,0 +1,31 @@ +# SQL Query Diagnosis + +## Overview +Investigate SQL query performance issues on `${AZURE_SQL_DATABASE}` (server `${AZURE_SQL_SERVER_FQDN}`). Identify root cause but DO NOT make any changes. + +## When to Use +- Users report slow page loads or API timeouts +- DTU alert fires (`alert-zavacafe-sql-dtu`) +- App Insights shows high query duration + +## Steps + +1. **Connect** to the database using `zava-mssql_mssql_connect_database` with server=`${AZURE_SQL_SERVER_FQDN}` and database=`${AZURE_SQL_DATABASE}` if not already connected. + +2. **Get table info** using `zava-mssql_mssql_get_schema` — check table sizes and existing indexes. + +3. **Find slow queries** by checking query stats: + `SELECT TOP 5 qs.total_elapsed_time/qs.execution_count as avg_ms, qs.execution_count, SUBSTRING(st.text, 1, 200) as query_text FROM sys.dm_exec_query_stats qs CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) st ORDER BY avg_ms DESC` + +4. **Analyze the execution plan** — run the slow query with `SET SHOWPLAN_TEXT ON` to see if there are Table Scans indicating missing indexes. + +5. **Check for missing index recommendations** from SQL Server: + `SELECT d.statement as table_name, d.equality_columns, d.inequality_columns, s.avg_user_impact FROM sys.dm_db_missing_index_details d JOIN sys.dm_db_missing_index_groups g ON d.index_handle = g.index_handle JOIN sys.dm_db_missing_index_group_stats s ON g.index_group_handle = s.group_handle ORDER BY s.avg_user_impact DESC` + +6. **Report findings**: table name, row count, missing index columns, estimated improvement. Use `PlotBarChart` to visualize query durations. + +7. **Hand off** to `sql-performance-fix` skill to apply the fix. + +## MCP Tools +- `zava-mssql_mssql_execute_query` — run diagnostic queries +- `zava-mssql_mssql_get_schema` — check table schema and indexes diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-query-diagnosis.yaml b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-query-diagnosis.yaml new file mode 100644 index 000000000..aa0a440b8 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/skills/sql-query-diagnosis.yaml @@ -0,0 +1,11 @@ +metadata: + name: sql-query-diagnosis + description: Diagnose slow SQL queries by analyzing execution plans and identifying missing indexes. This skill ONLY investigates. Hand off to sql-performance-fix to apply the fix. + spec: + tools: + - zava-mssql_mssql_execute_query + - zava-mssql_mssql_run_sql_query + - zava-mssql_mssql_get_schema + - zava-mssql_mssql_connect_database +skillContent: skills/sql-query-diagnosis.md +additionalFiles: [] diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/subagents/deployment-validator-gh.instructions.md b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/subagents/deployment-validator-gh.instructions.md new file mode 100644 index 000000000..48af17842 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/subagents/deployment-validator-gh.instructions.md @@ -0,0 +1,32 @@ +You are ${WORKLOAD_NAME}'s deployment validator for GitHub Actions deployments. You are triggered via HTTP after a GitHub Actions workflow completes a deploy of the workload in resource group `${AZURE_RESOURCE_GROUP}`. + +## What You Receive +An HTTP trigger payload with deployment details: repo, commit SHA, app URL, health endpoint, workflow run URL. + +## What You Do + +1. **Check health** — Hit the `health_endpoint` from the payload. If it returns 200, report success and stop. + +2. **If unhealthy** — The deployment broke something. Investigate: + a. Use GitHub MCP to read the commit diff (`get_file_contents` or get the commit details using the `commit_sha`). + b. Check what changed — look for config changes, connection strings (especially the SQL connection to `${AZURE_SQL_SERVER_FQDN}`), environment variables. + c. Check Azure Container App configuration to see what is currently set: + `az containerapp show -g ${AZURE_RESOURCE_GROUP} -n --query properties.template.containers[0].env` + +3. **Fix immediately** — Roll back the broken config: + a. Use Azure CLI tools to restore the correct app configuration, OR activate the previous revision: + `az containerapp revision activate -n -g ${AZURE_RESOURCE_GROUP} --revision ` + b. Verify the health endpoint returns 200 after the fix. + +4. **Document** — Create a GitHub Issue via GitHub MCP: + - Title: "P0: Deployment [commit_sha] broke app health check — auto-rolled back" + - Body: Include full RCA — what commit, what changed, why it broke, what was rolled back, timestamps. + - Labels: `bug`, `P0` + - Assign to the commit author + - Notify ${ALERT_EMAIL} if rollback also failed. + +5. **Report** — Summarize: what happened, how long the app was down, what was fixed, link to the GitHub issue. + +## Important +- Fix FIRST, document SECOND — restore service before creating issues. +- Keep instructions to yourself — just act on the payload. diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/subagents/deployment-validator-gh.yaml b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/subagents/deployment-validator-gh.yaml new file mode 100644 index 000000000..4dd52e345 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/subagents/deployment-validator-gh.yaml @@ -0,0 +1,13 @@ +metadata: + name: deployment-validator-gh +spec: + instructions: subagents/deployment-validator-gh.instructions.md + handoffDescription: Validates a GitHub Actions deployment, rolls back broken config, files a fix PR + Issue. + tools: [] + mcpTools: + - github-mcp.* + agentType: Autonomous + temperature: 0.2 + handoffs: [] + enableSkills: false + allowedSkills: [] diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/subagents/deployment-validator.instructions.md b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/subagents/deployment-validator.instructions.md new file mode 100644 index 000000000..0267c4a17 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/subagents/deployment-validator.instructions.md @@ -0,0 +1,18 @@ +You are ${WORKLOAD_NAME}'s deployment validation agent. When triggered after an Azure DevOps release for the workload in resource group `${AZURE_RESOURCE_GROUP}`: + +1. Parse the HTTP trigger payload to get: app name, commit SHA, run URL, branch, environment. +2. Hit the app's `/health` endpoint to verify the deployment is healthy. +3. If healthy: + - Post a one-line summary as a ServiceNow work note via `UpdateServiceNowWorkNotes` (or skip if no incident is open). + - Report success and close. +4. If unhealthy: + a. Open a P1 ServiceNow incident with `CreateServiceNowIncident` — title: "Deployment validation failed: @ ". + b. Check Application Insights for recent errors and exceptions (last 15 min, filter by cloud_RoleName = the app name). + c. Pull the changeset / commit diff from Azure DevOps (build details API). + d. Identify the root cause from the diff (e.g. wrong config value, missing env var, breaking SQL migration). + e. Roll back: redeploy the previous revision (`az containerapp revision activate -n -g ${AZURE_RESOURCE_GROUP} --revision `) or revert the config. + f. Re-hit `/health` to confirm recovery. + g. Post a full RCA as a ServiceNow work note. Escalate to ${ALERT_EMAIL} if rollback also fails. + h. Resolve the incident with `ResolveServiceNowIncident` once recovery is verified. + +Always explain your reasoning step by step. Fix FIRST, document SECOND — restore service before deep RCA. diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/subagents/deployment-validator.yaml b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/subagents/deployment-validator.yaml new file mode 100644 index 000000000..1ad0d28e5 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/subagents/deployment-validator.yaml @@ -0,0 +1,14 @@ +metadata: + name: deployment-validator +spec: + instructions: subagents/deployment-validator.instructions.md + handoffDescription: Validates an Azure DevOps release against the deployed Container App, rolls back automatically if the health check fails. + tools: + - CreateServiceNowIncident + - UpdateServiceNowWorkNotes + - ResolveServiceNowIncident + agentType: Autonomous + temperature: 0.2 + handoffs: [] + enableSkills: false + allowedSkills: [] diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/subagents/sql-performance-investigator.instructions.md b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/subagents/sql-performance-investigator.instructions.md new file mode 100644 index 000000000..5fddcbb24 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/subagents/sql-performance-investigator.instructions.md @@ -0,0 +1,52 @@ +You are ${WORKLOAD_NAME}'s SQL performance specialist. You investigate and resolve SQL performance issues on the Azure SQL database `${AZURE_SQL_DATABASE}` (server `${AZURE_SQL_SERVER_FQDN}`, resource group `${AZURE_RESOURCE_GROUP}`). + +You have four skills — two for diagnosis, two for fixing: + +**Diagnosis skills** (investigate only, no changes): +1. **sql-query-diagnosis** — Slow pages, timeouts, DTU spikes. Analyzes execution plans, finds missing indexes. +2. **sql-blocking-diagnosis** — App hanging, requests stuck. Finds head blocker session and impact. + +**Fix skills** (require risk assessment + approval): +3. **sql-performance-fix** — Creates missing indexes. MUST call AssessChangeRisk first, then AskUserQuestion for approval. +4. **sql-blocking-fix** — Kills blocking sessions. MUST call AssessChangeRisk first, then AskUserQuestion for approval. + +## Workflow when triggered by an Azure Monitor alert + +1. Open a ServiceNow incident with `CreateServiceNowIncident`: + - short_description: alert name + workload (e.g. "DTU > 80% on ${AZURE_SQL_DATABASE}") + - urgency: 2 (High), impact: 2 (High) + - escalate to ${ALERT_EMAIL} if severity is critical +2. Run the matching **diagnosis skill** to understand the problem +3. Present findings to the user with charts (`PlotBarChart`) +4. Switch to the matching **fix skill** to apply the solution +5. The fix skill assesses risk, requests approval, executes, and verifies +6. Document every step in ServiceNow via `UpdateServiceNowWorkNotes` +7. Resolve the incident with `ResolveServiceNowIncident` once the fix is verified + +NEVER skip the diagnosis step. NEVER apply a fix without running AssessChangeRisk first. + +## Visualization +Always use charts to help the user visualize the issue: +- Use **PlotBarChart** to show DTU consumption by query, blocking session counts, or query duration comparisons (before vs after fix) +- Use **PlotPieChart** to show distribution of query types, wait types, or resource consumption by category +- Use **PlotScatter** to show correlation between query duration and row counts, or DTU vs time +- When showing before/after results (e.g. query plan improvement), create a bar chart comparing old vs new metrics +- Always include a descriptive title and summary with each chart + +## Summary Report +After completing an investigation and fix, always provide a structured summary AND post it as a final ServiceNow work-note: + +### Issue Summary +- **Problem**: What was reported (e.g. "Products page loading slowly") +- **Impact**: Who/what was affected (e.g. "All users querying by category, avg response time 3.2s") + +### Analysis +- **Root Cause**: What you found (e.g. "Missing index on Products.Category causing full table scan on 4,800 rows") +- **Evidence**: Key data points from your investigation (execution plan, DMV results, metrics) + +### Resolution +- **Action Taken**: What you did to fix it (e.g. "Created index IX_Products_Category on Products(Category)") +- **Verification**: Before/after comparison (e.g. "Query time reduced from 3.2s to 0.05s, plan changed from Table Scan to Index Seek") + +### Recommendations +- Any follow-up actions (e.g. "Monitor DTU for the next hour to confirm stability", "Consider adding similar indexes for other filtered columns") diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/subagents/sql-performance-investigator.yaml b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/subagents/sql-performance-investigator.yaml new file mode 100644 index 000000000..d9aec3c24 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/subagents/sql-performance-investigator.yaml @@ -0,0 +1,23 @@ +metadata: + name: sql-performance-investigator +spec: + instructions: subagents/sql-performance-investigator.instructions.md + handoffDescription: '' + tools: + - PlotPieChart + - PlotBarChart + - PlotScatter + - AssessChangeRisk + - CreateServiceNowIncident + - UpdateServiceNowWorkNotes + - ResolveServiceNowIncident + - LookupServiceNowIncident + agentType: Autonomous + temperature: 0.2 + handoffs: [] + enableSkills: true + allowedSkills: + - sql-blocking-diagnosis + - sql-blocking-fix + - sql-performance-fix + - sql-query-diagnosis diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/tools/AssessChangeRisk.yaml b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/tools/AssessChangeRisk.yaml new file mode 100644 index 000000000..a530f2663 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/config/tools/AssessChangeRisk.yaml @@ -0,0 +1,110 @@ +api_version: azuresre.ai/v2 +kind: ExtendedAgentTool +metadata: + name: AssessChangeRisk +spec: + type: PythonTool + connector: '' + toolMode: Auto + description: |- + Assess the risk of a database change before executing it on ${AZURE_SQL_DATABASE} (${AZURE_SQL_SERVER_FQDN}). Evaluates business hours, table criticality, row count, and operation type. ALWAYS call this tool before making any SQL write operation (CREATE INDEX, UPDATE, DELETE, KILL). The result determines whether human approval is needed. If ${ADO_ORG_URL} is configured, recent pipeline runs may be cross-referenced via the ADO_PAT secret to detect coincident deploys. + functionCode: |- + from datetime import datetime + + CRITICAL_TABLES = ['Orders', 'OrderItems', 'Payments', 'Users', 'Customers'] + HIGH_RISK_OPS = ['DELETE', 'DROP', 'TRUNCATE', 'ALTER'] + MEDIUM_RISK_OPS = ['CREATE INDEX', 'UPDATE', 'KILL'] + LOW_RISK_OPS = ['SELECT', 'INSERT'] + + def assess_change_risk(operation: str, table_name: str, row_count: int = 0, description: str = '') -> dict: + risks = [] + risk_score = 0 + + # 1. Business hours check (rough US Pacific window) + hour = datetime.utcnow().hour + is_business_hours = 14 <= hour or hour <= 6 + if is_business_hours: + risks.append({'factor': 'Business Hours', 'level': 'WARNING', 'detail': f'Current time is {datetime.utcnow().strftime("%H:%M")} UTC — business hours in US Pacific. Schema changes may impact active users.'}) + risk_score += 30 + else: + risks.append({'factor': 'Business Hours', 'level': 'OK', 'detail': f'Current time is {datetime.utcnow().strftime("%H:%M")} UTC — outside peak business hours.'}) + + # 2. Table criticality + table_clean = table_name.replace('[', '').replace(']', '').replace('dbo.', '').strip() + is_critical = table_clean in CRITICAL_TABLES + if is_critical: + risks.append({'factor': 'Table Criticality', 'level': 'HIGH', 'detail': f'{table_clean} is a CRITICAL table — changes directly affect business operations.'}) + risk_score += 40 + else: + risks.append({'factor': 'Table Criticality', 'level': 'LOW', 'detail': f'{table_clean} is not in the critical tables list.'}) + risk_score += 5 + + # 3. Row count + if row_count > 100000: + risks.append({'factor': 'Data Volume', 'level': 'HIGH', 'detail': f'{row_count:,} rows — large table, operation may cause locks or high DTU.'}) + risk_score += 30 + elif row_count > 10000: + risks.append({'factor': 'Data Volume', 'level': 'MEDIUM', 'detail': f'{row_count:,} rows — moderate table size.'}) + risk_score += 15 + else: + risks.append({'factor': 'Data Volume', 'level': 'LOW', 'detail': f'{row_count:,} rows — small table.'}) + risk_score += 5 + + # 4. Operation type + op_upper = operation.upper().strip() + if any(op in op_upper for op in HIGH_RISK_OPS): + risks.append({'factor': 'Operation Type', 'level': 'HIGH', 'detail': f'{operation} is a destructive operation.'}) + risk_score += 40 + elif any(op in op_upper for op in MEDIUM_RISK_OPS): + risks.append({'factor': 'Operation Type', 'level': 'MEDIUM', 'detail': f'{operation} is a schema/session change.'}) + risk_score += 20 + else: + risks.append({'factor': 'Operation Type', 'level': 'LOW', 'detail': f'{operation} is a read/safe operation.'}) + + if risk_score >= 60: + overall = 'HIGH' + recommendation = 'Human approval required before proceeding. Use AskUserQuestion to present the risk assessment and get explicit approval.' + requires_approval = True + elif risk_score >= 30: + overall = 'MEDIUM' + recommendation = 'Human approval recommended. Use AskUserQuestion to present options: approve now or schedule for maintenance window.' + requires_approval = True + else: + overall = 'LOW' + recommendation = 'Safe to proceed without approval.' + requires_approval = False + + return { + 'overall_risk': overall, + 'risk_score': risk_score, + 'requires_approval': requires_approval, + 'recommendation': recommendation, + 'description': description, + 'assessed_at': datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S UTC'), + 'risk_factors': risks + } + timeoutSeconds: 240 + dependencies: + - '' + authScopes: + parameters: + - name: operation + type: string + description: |- + string:The SQL operation to assess (e.g. CREATE INDEX, DELETE, UPDATE, KILL) + required: true + - name: table_name + type: string + description: |- + string:The target table name (e.g. Products, Orders) + required: true + - name: row_count + type: string + description: |- + integer:Approximate number of rows affected + required: true + - name: description + type: string + description: |- + string:Brief description of what this change does + required: true diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/connectors.json b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/connectors.json new file mode 100644 index 000000000..566b9da77 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/connectors.json @@ -0,0 +1,12 @@ +{ + "toggles": { + "enableAppInsightsConnector": true, + "appInsightsResourceId": "{{appInsightsId}}", + "appInsightsAppId": "{{appInsightsAppId}}", + "enableLogAnalyticsConnector": true, + "lawResourceId": "{{lawResourceId}}", + "enableAzureMonitorConnector": true, + "azureMonitorLookbackDays": 7 + }, + "connectors": [] +} diff --git a/labs/recipes/azmon-aca-servicenow-zavacafe-ops/expected-config.json b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/expected-config.json new file mode 100644 index 000000000..68963e672 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavacafe-ops/expected-config.json @@ -0,0 +1,44 @@ +{ + "_scenario": "azmon-aca-servicenow-zavacafe-ops", + "agent": { + "accessLevel": "High", + "actionMode": "Autonomous", + "upgradeChannel": "Preview", + "defaultModelProvider": "Anthropic", + "incidentPlatform": "AzureMonitor" + }, + "connectors": [ + { "name": "app-insights", "type": "AppInsights" }, + { "name": "log-analytics", "type": "LogAnalytics" }, + { "name": "azure-monitor", "type": "AzureMonitor" } + ], + "skills": [ + "sql-blocking-diagnosis", + "sql-blocking-fix", + "sql-performance-fix", + "sql-query-diagnosis" + ], + "subagents": [ + "sql-performance-investigator", + "deployment-validator", + "deployment-validator-gh" + ], + "hooks": [ + "change-risk-assessor", + "sql-write-guard" + ], + "tools": [ + "AssessChangeRisk" + ], + "commonPrompts": [], + "scheduledTasks": [ + "weekly-cost-report" + ], + "responsePlans": [ + { + "name": "auto-investigate-azmon", + "handlingAgent": "sql-performance-investigator" + } + ], + "repos": [] +} diff --git a/labs/recipes/azmon-aca-servicenow-zavaitsupport/.gitignore b/labs/recipes/azmon-aca-servicenow-zavaitsupport/.gitignore new file mode 100644 index 000000000..5af9c4464 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavaitsupport/.gitignore @@ -0,0 +1,8 @@ +# Local secrets — never commit +connectors.secrets.env + +# Local data assets +data/ + +# Generated agent configs (output of bin/new-agent.sh) +output/ diff --git a/labs/recipes/azmon-aca-servicenow-zavaitsupport/README.md b/labs/recipes/azmon-aca-servicenow-zavaitsupport/README.md new file mode 100644 index 000000000..48603a28f --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavaitsupport/README.md @@ -0,0 +1,114 @@ +# azmon-aca-servicenow-zavaitsupport + +> **Unified IT-support recipe.** This is the single canonical recipe for the +> ServiceNow IT-support / laptop-replacement agent. It supersedes the prior +> `azmon-aca-servicenow-azurefriday-itsupport` and +> `azmon-aca-servicenow-zavapower-itsupport` recipes, which have been removed. +> The agent and tools here are sourced from +> [`labs/zava-itsupport/sre-config/`](../../zava-itsupport/sre-config/). + +ServiceNow IT-support helpdesk agent for the **Zava IT Support** Zava demo. Polls ServiceNow for laptop-replacement requests, validates warranty via a custom Python tool against the lab's warranty API, submits a replacement order through Browser Operator, and resolves the SNOW ticket — fully autonomous. + +## Stack + +- **App** (target workload): Node.js 20 IT-portal + Python 3.11 / FastAPI warranty API (sourced from [`labs/zava-itsupport/`](../../zava-itsupport/)) +- **Compute**: Azure Container Apps (recipe is workload-agnostic — only requires the warranty API to be reachable at `${WARRANTY_API_URL}`) +- **Data**: None (warranty data is mocked in the warranty-tool service; ServiceNow PDI is the system of record for tickets) +- **Observability**: ServiceNow incident telemetry only (no AzMon dependency in this recipe) +- **SRE Agent**: 1 subagent (`it-support-handler`, autonomous); 2 custom Python tools (`CheckWarranty`, `LookupServiceNowIncident`) shipped in `config/tools/`; incident filter `snow-laptop-replacement` (categoryFilter=hardware, shortDescriptionContains=laptop, priorities 3/4/5). Connectors: ServiceNow Incident Platform; native ServiceNow tools (`GetServiceNowIncident`, `PostServiceNowDiscussionEntry`, `AcknowledgeServiceNowIncident`, `ResolveServiceNowIncident`) become available after platform connection; Browser Operator for portal submission; `SendOutlookEmail` for employee notifications. No skills, no hooks, no scheduled tasks — single-purpose automation. +- **Simulator**: None — pair with [`labs/zava-itsupport/scripts/laptop-request-demo.sh`](../../zava-itsupport/scripts/) to file a sample request +- **CI/CD**: Upstream `sreagent-templates` deployment scripts — `bin/new-agent.sh --recipe ...` → `bin/deploy.sh` + +## What it's about + +This recipe is the **portable, lab-agnostic agent half of [`labs/zava-itsupport/`](../../zava-itsupport/)** — the ServiceNow IT-support / laptop-replacement SRE Agent config, packaged in the shape required by [`coreai-microsoft/sreagent-templates`](https://github.com/coreai-microsoft/sreagent-templates) so customers can drop it onto their own ServiceNow + warranty-API setup without taking the lab's infra or app code. It is the unified, canonical IT-support recipe — superseding the prior `azmon-aca-servicenow-azurefriday-itsupport` and `azmon-aca-servicenow-zavapower-itsupport` recipes. + +The recipe targets PMs, SREs, and customers who want to see the SRE Agent automate a **ServiceNow-driven helpdesk workflow with custom Python tools** — distinct from the AzMon-driven infrastructure-ops recipes. The break/fix pattern is single-purpose: a hardware/laptop SNOW ticket arrives → the filter routes to `it-support-handler` → the agent calls `LookupServiceNowIncident` to fetch the ticket → calls `CheckWarranty` against the warranty API → if eligible, submits a replacement via Browser Operator and resolves the ticket; if not, posts a discussion entry explaining next steps. Demo flow: `bin/new-agent.sh --recipe azmon-aca-servicenow-zavaitsupport ...` → `bin/deploy.sh` → connect ServiceNow as the Incident Platform → create a SNOW incident with category=Hardware + "Laptop replacement request" → watch the agent run the workflow autonomously. + +## What it does + +1. A ServiceNow incident with category=`hardware` and short_description containing `laptop` is created. +2. The `snow-laptop-replacement` filter routes it to `it-support-handler`. +3. The agent calls `LookupServiceNowIncident` (custom tool) to fetch ticket details by INC number. +4. The agent calls `CheckWarranty` (custom tool, hits `${WARRANTY_API_URL}`) with the device serial number. +5. If eligible, the agent uses Browser Operator to file a laptop request, then resolves the SNOW ticket and emails the employee. +6. If not eligible, the agent posts a discussion entry explaining next steps and resolves the ticket. + +## Prereqs + +- Azure subscription with SRE Agent RP access +- ServiceNow instance (PDI works) — reachable from the agent +- ServiceNow admin user with `incident_manager` / `itil` role +- The lab's warranty API reachable at `${WARRANTY_API_URL}` (returns JSON like `{ "found": true, "eligible_for_replacement": true, "warranty_expiry": "...", "recommended_replacement": "Dell XPS 15 9530" }`) + +## Quick start + +```bash +./bin/new-agent.sh --recipe azmon-aca-servicenow-zavaitsupport-itsupport --non-interactive \ + --set agentName=zavaitsupport-itsupport \ + --set resourceGroup=rg-zavaitsupport-itsupport \ + --set location=eastus2 \ + --set WORKLOAD_NAME=zava-itsupport \ + --set WARRANTY_API_URL=https://app-zava-warranty.azurewebsites.net \ + --set SERVICENOW_INSTANCE_URL=https://dev123456.service-now.com \ + --set SERVICENOW_USERNAME=admin \ + --set demoEmployeeEmail=demo.user@zavaitsupport.com \ + -o zavaitsupport-itsupport/ + +./bin/deploy.sh zavaitsupport-itsupport/ +``` + +`SERVICENOW_PASSWORD` is supplied via the SRE Agent UI when the Incident Platform is connected (and is also pasted into the `LookupServiceNowIncident` tool's secret slot on first invocation). + +## Parameters + +| Param | Required | Example | How to get it | +|---|---|---|---| +| `agentName` | ✅ | `zavaitsupport-itsupport` | You choose (lowercase, hyphens) | +| `resourceGroup` | ✅ | `rg-zavaitsupport-itsupport` | You choose or use existing RG | +| `location` | ✅ | `eastus2` | Where to host the agent | +| `WORKLOAD_NAME` | ⛔ | `zava-itsupport` | Workload tag | +| `WARRANTY_API_URL` | ⛔ | `https://app-zava-warranty.azurewebsites.net` | Lab's warranty service endpoint | +| `SERVICENOW_INSTANCE_URL` | ✅ | `https://dev123456.service-now.com` | ServiceNow instance URL | +| `SERVICENOW_USERNAME` | ⛔ | `admin` | ServiceNow user the LookupServiceNowIncident tool authenticates as | +| `demoEmployeeEmail` | ⛔ | `demo.user@zavaitsupport.com` | Fallback email when not in the ticket | + +## What gets deployed + +- **Subagent:** `it-support-handler` (Autonomous, native ServiceNow tools + `CheckWarranty` + `LookupServiceNowIncident` + `SendOutlookEmail`) +- **Tools:** `CheckWarranty`, `LookupServiceNowIncident` (custom Python tools, shipped in `config/tools/`) +- **Incident platform:** ServiceNow +- **Incident filter:** `snow-laptop-replacement` — routes hardware/laptop tickets to the handler +- No skills, no hooks, no scheduled tasks — single-purpose automation + +## ServiceNow setup (post-deploy) + +In the SRE Agent UI for the deployed agent: + +1. **Builder → Incidents → Connect platform → ServiceNow** +2. Instance URL: `${SERVICENOW_INSTANCE_URL}` +3. Username: `${SERVICENOW_USERNAME}` +4. Password: (your admin password or OAuth token) +5. Save. + +Once connected, the native SNOW tools (`GetServiceNowIncident`, `PostServiceNowDiscussionEntry`, `AcknowledgeServiceNowIncident`, `ResolveServiceNowIncident`) become available to the subagent automatically. + +### Filter behaviour + +The `snow-laptop-replacement` filter triggers on: +- `categoryFilter: hardware` +- `shortDescriptionContains: laptop` +- Priorities 3, 4, 5 +- Events: `IncidentCreated`, `IncidentUpdated` + +To test: create a SNOW incident with category=Hardware, priority=4, short description "Laptop replacement request", and a description containing employee details + serial number. + +## Azure Monitor alerts + +This recipe **does not** subscribe to Azure Monitor — it is purely SNOW-driven. If you want AzMon-driven incidents in this same agent, use the `azmon-aca-servicenow-zavaitsupport-ops` recipe instead. + +## Cost + +Single-subagent, low-volume helpdesk automation. Monthly Agent Unit budget capped at 5000 in `agent.json` — adjust if you process many tickets. + + diff --git a/labs/recipes/azmon-aca-servicenow-zavaitsupport/agent.json b/labs/recipes/azmon-aca-servicenow-zavaitsupport/agent.json new file mode 100644 index 000000000..3c4e9f04a --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavaitsupport/agent.json @@ -0,0 +1,86 @@ +{ + "_scenario": "azmon-aca-servicenow-zavaitsupport-itsupport", + "_description": "ServiceNow IT-support helpdesk agent for the Zava IT Support demo (Zava). Polls ServiceNow for laptop-replacement requests, validates warranty via a custom Python tool against the lab's warranty API, submits a replacement order via Browser Operator, and resolves the SNOW ticket. Native ServiceNow incident-platform integration — no Kusto, no AzMon.", + "_prerequisites": [ + "Azure subscription with SRE Agent RP access", + "ServiceNow instance reachable from the agent (PDI works for demos)", + "ServiceNow admin creds — entered when connecting the Incident Platform in the SRE Agent UI", + "Zava warranty API reachable at WARRANTY_API_URL (the lab's Azure Function or Web App)", + "(Optional) Outlook mailbox for the SendOutlookEmail tool" + ], + "_prompts": { + "agentName": { + "ask": "Agent name", + "default": "zavaitsupport-itsupport-agent" + }, + "resourceGroup": { + "ask": "Resource group", + "default": "rg-zavaitsupport-itsupport" + }, + "location": { + "ask": "Region", + "options": ["eastus2", "swedencentral", "uksouth", "australiaeast"], + "default": "eastus2", + "required": true + }, + "WORKLOAD_NAME": { + "ask": "Workload short name (used in tags + email body templates)", + "default": "zava-itsupport" + }, + "WARRANTY_API_URL": { + "ask": "Zava warranty API base URL (the CheckWarranty tool calls this for serial-number lookup)", + "default": "https://app-zava-warranty.azurewebsites.net" + }, + "SERVICENOW_INSTANCE_URL": { + "ask": "ServiceNow instance URL (e.g. https://dev123456.service-now.com)", + "required": true + }, + "SERVICENOW_USERNAME": { + "ask": "ServiceNow username for the LookupServiceNowIncident tool (e.g. admin)", + "default": "admin" + }, + "demoEmployeeEmail": { + "ask": "Demo employee email used as fallback when not in the ticket", + "default": "demo.user@zavaitsupport.com" + }, + "modelProvider": { + "ask": "AI model provider", + "options": ["Anthropic", "GitHubCopilot", "MicrosoftFoundry"], + "default": "Anthropic" + }, + "existingUamiId": { + "ask": "Existing UAMI resource ID (leave blank to create new)", + "default": "" + }, + "existingAgentAppInsightsId": { + "ask": "Existing App Insights resource ID for agent telemetry (leave blank to create new)", + "default": "" + } + }, + "identity": { + "agentName": "{{agentName}}", + "resourceGroup": "{{resourceGroup}}", + "subscription": "", + "location": "{{location}}", + "targetResourceGroups": "" + }, + "access": { + "accessLevel": "Low", + "actionMode": "Autonomous" + }, + "upgradeChannel": "Preview", + "defaultModelProvider": "{{modelProvider}}", + "monthlyAgentUnitLimit": 5000, + "tags": { + "scenario": "zavaitsupport-itsupport", + "workload": "{{WORKLOAD_NAME}}" + }, + "toggles": { + "enableWebhookBridge": false, + "webhookBridgeTriggerUrl": "" + }, + "existingUamiId": "{{existingUamiId}}", + "existingAgentAppInsightsId": "{{existingAgentAppInsightsId}}" +} + + diff --git a/labs/recipes/azmon-aca-servicenow-zavaitsupport/automations/incident-filters/snow-laptop-replacement.yaml b/labs/recipes/azmon-aca-servicenow-zavaitsupport/automations/incident-filters/snow-laptop-replacement.yaml new file mode 100644 index 000000000..6ddac51bd --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavaitsupport/automations/incident-filters/snow-laptop-replacement.yaml @@ -0,0 +1,20 @@ +metadata: + name: snow-laptop-replacement +spec: + incidentPlatform: ServiceNow + isEnabled: true + priorities: + - "3" + - "4" + - "5" + incidentType: Request + handlingAgent: it-support-handler + agentMode: Autonomous + deepInvestigationEnabled: false + maxAutomatedInvestigationAttempts: 2 + serviceNowFilterSettings: + triggerEvents: + - IncidentCreated + - IncidentUpdated + categoryFilter: hardware + shortDescriptionContains: laptop diff --git a/labs/recipes/azmon-aca-servicenow-zavaitsupport/automations/incident-platforms/servicenow.yaml b/labs/recipes/azmon-aca-servicenow-zavaitsupport/automations/incident-platforms/servicenow.yaml new file mode 100644 index 000000000..1d38ac48e --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavaitsupport/automations/incident-platforms/servicenow.yaml @@ -0,0 +1,3 @@ +name: servicenow +spec: + platformType: ServiceNow diff --git a/labs/recipes/azmon-aca-servicenow-zavaitsupport/config/subagents/it-support-handler.instructions.md b/labs/recipes/azmon-aca-servicenow-zavaitsupport/config/subagents/it-support-handler.instructions.md new file mode 100644 index 000000000..53e370be9 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavaitsupport/config/subagents/it-support-handler.instructions.md @@ -0,0 +1,48 @@ +You are ${WORKLOAD_NAME}'s IT Support automation agent. You handle employee laptop replacement requests that arrive as ServiceNow incidents. + +## Step 1: Fetch the ServiceNow Ticket + +The native ServiceNow tools require a `sys_id`, NOT the incident number. First, use `LookupServiceNowIncident` with the incident number (e.g. `INC0010005`) to get the `sys_id` and full ticket details. Then use the `sys_id` for all subsequent ServiceNow tool calls. + +Extract from the ticket: +- Employee name +- Employee email (default: `${demoEmployeeEmail}` if not specified) +- Employee ID +- Department +- Current laptop serial number +- Description of the issue + +## Step 2: Validate Warranty + +Use the `CheckWarranty` tool with the serial number extracted from the ticket. The tool calls `${WARRANTY_API_URL}/warranty/`. + +Evaluate the result: +- If `eligible_for_replacement` is `true` → proceed to Step 3. +- If warranty is still active → post a discussion entry to the ServiceNow ticket explaining the device is under warranty and should be repaired, not replaced. Use `PostServiceNowDiscussionEntry`. +- If device not found → post a discussion entry asking the requester to verify the serial number. + +## Step 3: Submit Laptop Request via Browser Operator + +Navigate to the internal IT request portal and fill the laptop request form with: +- Employee Name: from the ticket +- Employee Email: from the ticket (or `${demoEmployeeEmail}`) +- Employee ID: from the ticket +- Department: from the ticket +- Current Laptop Serial Number: from the ticket +- Reason for Request: `Warranty Expired` +- Preferred Laptop Model: use the `recommended_replacement` from the `CheckWarranty` result +- ServiceNow Ticket Reference: the incident number (e.g. `INC0010001`) +- Additional Notes: warranty expiry date and eligibility details + +Submit the form and capture the Request ID from the confirmation (e.g. `LR-2026-XXXXX`). + +## Step 4: Update ServiceNow and Notify + +- Use `PostServiceNowDiscussionEntry` to update the ticket with the laptop request details and Request ID. +- Use `ResolveServiceNowIncident` to resolve the ticket with a summary of actions taken. +- Send an email to the employee using `SendOutlookEmail` with the Request ID and next steps. + +## Important + +- Always verify warranty eligibility BEFORE submitting the form. +- If any step fails, report the issue clearly and suggest manual remediation steps. diff --git a/labs/recipes/azmon-aca-servicenow-zavaitsupport/config/subagents/it-support-handler.yaml b/labs/recipes/azmon-aca-servicenow-zavaitsupport/config/subagents/it-support-handler.yaml new file mode 100644 index 000000000..f78177430 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavaitsupport/config/subagents/it-support-handler.yaml @@ -0,0 +1,18 @@ +metadata: + name: it-support-handler +spec: + instructions: subagents/it-support-handler.instructions.md + handoffDescription: Handles employee laptop replacement requests from ServiceNow — fetches the ticket, validates warranty, submits the replacement order, and resolves the ticket. + tools: + - CheckWarranty + - LookupServiceNowIncident + - GetServiceNowIncident + - PostServiceNowDiscussionEntry + - AcknowledgeServiceNowIncident + - ResolveServiceNowIncident + - SendOutlookEmail + agentType: Autonomous + temperature: 0.2 + handoffs: [] + enableSkills: false + allowedSkills: [] diff --git a/labs/recipes/azmon-aca-servicenow-zavaitsupport/config/tools/CheckWarranty.yaml b/labs/recipes/azmon-aca-servicenow-zavaitsupport/config/tools/CheckWarranty.yaml new file mode 100644 index 000000000..4abda106a --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavaitsupport/config/tools/CheckWarranty.yaml @@ -0,0 +1,45 @@ +api_version: azuresre.ai/v2 +kind: ExtendedAgentTool +metadata: + name: CheckWarranty +spec: + type: PythonTool + connector: '' + toolMode: Auto + description: |- + Check device warranty status by serial number against the ${WORKLOAD_NAME} warranty API at ${WARRANTY_API_URL}. Returns warranty expiry, eligibility for replacement, and recommended replacement model. Use when handling laptop replacement requests or hardware support tickets. + functionCode: |- + import os + import requests + + WARRANTY_API_URL = os.environ.get("WARRANTY_API_URL", "${WARRANTY_API_URL}") + + def check_warranty(serial_number: str) -> dict: + """Check warranty status by calling the Zava Warranty API. + + Calls the warranty lookup service to check device warranty status, + eligibility for replacement, and recommended replacement. + """ + try: + response = requests.get( + f"{WARRANTY_API_URL}/warranty/{serial_number}", + timeout=10 + ) + if response.status_code == 404: + return {"found": False, "error": "Device not found in warranty database"} + response.raise_for_status() + return response.json() + except requests.exceptions.Timeout: + return {"found": False, "error": "Warranty API timed out"} + except Exception as e: + return {"found": False, "error": f"Failed to reach warranty API: {str(e)}"} + timeoutSeconds: 240 + dependencies: + - requests + authScopes: + parameters: + - name: serial_number + type: string + description: |- + string:Device serial number (e.g. SN-2023-XPS-4471) + required: true diff --git a/labs/recipes/azmon-aca-servicenow-zavaitsupport/config/tools/LookupServiceNowIncident.yaml b/labs/recipes/azmon-aca-servicenow-zavaitsupport/config/tools/LookupServiceNowIncident.yaml new file mode 100644 index 000000000..deb74ca19 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavaitsupport/config/tools/LookupServiceNowIncident.yaml @@ -0,0 +1,60 @@ +api_version: azuresre.ai/v2 +kind: ExtendedAgentTool +metadata: + name: LookupServiceNowIncident +spec: + type: PythonTool + connector: '' + toolMode: Auto + description: |- + Look up a ServiceNow incident by its number (e.g. INC0010005) on ${SERVICENOW_INSTANCE_URL} and return the sys_id and full incident details. Use this FIRST before calling GetServiceNowIncident, PostServiceNowDiscussionEntry, AcknowledgeServiceNowIncident, or ResolveServiceNowIncident — those tools require the sys_id, not the INC number. + functionCode: |- + import os + import requests + + SERVICENOW_URL = os.environ.get("SERVICENOW_INSTANCE_URL", "${SERVICENOW_INSTANCE_URL}") + SERVICENOW_USER = os.environ.get("SERVICENOW_USERNAME", "${SERVICENOW_USERNAME}") + SERVICENOW_PASS = os.environ.get("SERVICENOW_PASSWORD", "") + + def lookup_servicenow_incident(incident_number: str) -> dict: + try: + response = requests.get( + f"{SERVICENOW_URL}/api/now/table/incident", + params={ + "sysparm_query": f"number={incident_number}", + "sysparm_limit": "1", + "sysparm_fields": "sys_id,number,short_description,description,state,priority,category,subcategory,caller_id,assignment_group,assigned_to,opened_at,resolved_at" + }, + auth=(SERVICENOW_USER, SERVICENOW_PASS), + headers={"Accept": "application/json"}, + timeout=10 + ) + response.raise_for_status() + results = response.json().get("result", []) + if not results: + return {"found": False, "error": f"Incident {incident_number} not found"} + incident = results[0] + return { + "found": True, + "sys_id": incident["sys_id"], + "number": incident["number"], + "short_description": incident.get("short_description", ""), + "description": incident.get("description", ""), + "state": incident.get("state", ""), + "priority": incident.get("priority", ""), + "category": incident.get("category", ""), + "subcategory": incident.get("subcategory", ""), + "opened_at": incident.get("opened_at", "") + } + except Exception as e: + return {"found": False, "error": str(e)} + timeoutSeconds: 240 + dependencies: + - requests + authScopes: + parameters: + - name: incident_number + type: string + description: |- + string:ServiceNow incident number (e.g. INC0010005) + required: true diff --git a/labs/recipes/azmon-aca-servicenow-zavaitsupport/connectors.json b/labs/recipes/azmon-aca-servicenow-zavaitsupport/connectors.json new file mode 100644 index 000000000..120098b9f --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavaitsupport/connectors.json @@ -0,0 +1,12 @@ +{ + "toggles": { + "enableAppInsightsConnector": false, + "appInsightsResourceId": "", + "appInsightsAppId": "", + "enableLogAnalyticsConnector": false, + "lawResourceId": "", + "enableAzureMonitorConnector": false, + "azureMonitorLookbackDays": 7 + }, + "connectors": [] +} diff --git a/labs/recipes/azmon-aca-servicenow-zavaitsupport/expected-config.json b/labs/recipes/azmon-aca-servicenow-zavaitsupport/expected-config.json new file mode 100644 index 000000000..35cf9ae0e --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavaitsupport/expected-config.json @@ -0,0 +1,30 @@ +{ + "_scenario": "azmon-aca-servicenow-zavaitsupport-itsupport", + "agent": { + "accessLevel": "Low", + "actionMode": "Autonomous", + "upgradeChannel": "Preview", + "defaultModelProvider": "Anthropic", + "incidentPlatform": "ServiceNow" + }, + "connectors": [], + "skills": [], + "subagents": [ + "it-support-handler" + ], + "hooks": [], + "tools": [ + "CheckWarranty", + "LookupServiceNowIncident" + ], + "commonPrompts": [], + "scheduledTasks": [], + "responsePlans": [ + { + "name": "snow-laptop-replacement", + "handlingAgent": "it-support-handler" + } + ], + "repos": [] +} + diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/.gitignore b/labs/recipes/azmon-aca-servicenow-zavapower-ops/.gitignore new file mode 100644 index 000000000..9893e196e --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/.gitignore @@ -0,0 +1,3 @@ +connectors.secrets.env +data/ +output/ diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/README.md b/labs/recipes/azmon-aca-servicenow-zavapower-ops/README.md new file mode 100644 index 000000000..0ce862c37 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/README.md @@ -0,0 +1,121 @@ +# azmon-aca-servicenow-zavapower-ops + +Production-ops SRE agent for **Zava Power**'s microservices platform on Azure Container Apps. Handles 5xx alerts, latency, container restarts, deployment validation, VM disk pressure, pipeline failures, and pod fleet audits — across 5 services. Native Azure Monitor + ServiceNow incident platforms; optional Datadog and Dynatrace MCP connectors. + +## Stack + +- **App** (target workload): 5-service microservices fleet — Python/Flask, .NET 8, Node.js 20, Go 1.22, React (sourced from [`labs/zava-power/`](../../zava-power/)) +- **Compute**: Azure Container Apps (recipe is parameterized on `containerAppPrefix` + `targetRGs`) +- **Data**: None (microservices are stateless; ServiceNow PDI is the system of record for tickets) +- **Observability**: Application Insights, Log Analytics (`ContainerAppConsoleLogs`), Azure Monitor; optional Datadog and Dynatrace MCP connectors +- **SRE Agent**: 8 subagents (`incident-handler`, `deployment-validator`, `vm-ops-agent`, `utility-ops-agent`, `web-app-troubleshooter`, `pod-incident-remediator`, `release-orchestrator`, `pipeline-failure-investigator`); 15 skills (per-service diagnosis + crash/config/perf classes + ops procedures); incident filter `auto-investigate-azmon`; daily `pod-fleet-audit-daily` scheduled task. Connectors: App Insights, Log Analytics, Azure Monitor, ServiceNow, optional Datadog/Dynatrace. +- **Simulator**: None — pair with [`labs/zava-power/simulator/demo.py`](../../zava-power/simulator/) for the full 7-scenario break/fix experience +- **CI/CD**: Upstream `sreagent-templates` deployment scripts — `bin/new-agent.sh --recipe ...` → `bin/deploy.sh` → `bin/verify-agent.sh` + +## What it's about + +This recipe is the **portable, lab-agnostic agent half of [`labs/zava-power/`](../../zava-power/)** — the production-ops SRE Agent config for a multi-language microservices fleet on Azure Container Apps, packaged in the shape required by [`coreai-microsoft/sreagent-templates`](https://github.com/coreai-microsoft/sreagent-templates) so customers can drop it onto their own ACA workload without the lab's infra or simulator. The recipe assumes the workload already exists with App Insights, a Log Analytics workspace, AzMon alert rules, and a ServiceNow PDI; you supply those resource IDs as parameters and the recipe wires the agent on top. + +The recipe targets PMs, SREs, and customers who want to apply Zava Power's break/fix patterns — 5xx investigations, perf regressions, container restarts, VM disk pressure (Azure Arc), ADO pipeline failures, post-rollout validation, daily fleet audit decks — to their own ACA fleet. Demo flow: `bin/new-agent.sh --recipe azmon-aca-servicenow-zavapower-ops --non-interactive --set ...` → `bin/deploy.sh` → connect ServiceNow as the Incident Platform in the SRE Agent UI → an AzMon alert fires → the `incident-handler` (or the right specialist subagent) opens a ServiceNow incident, investigates, remediates, and documents the work. + +## Quick start + +```bash +./bin/new-agent.sh --recipe azmon-aca-servicenow-zavapower-ops --non-interactive \ + --set agentName=zavapower-ops \ + --set resourceGroup=rg-zavapower-ops \ + --set location=eastus2 \ + --set targetRGs=rg-zavapower-prod \ + --set appInsightsId=/subscriptions//resourceGroups/rg-zavapower-prod/providers/Microsoft.Insights/components/appi-zavapower \ + --set appInsightsAppId= \ + --set lawResourceId=/subscriptions//resourceGroups/rg-zavapower-prod/providers/Microsoft.OperationalInsights/workspaces/log-zavapower \ + --set snowInstance=dev123456 \ + --set containerAppPrefix=powergrid \ + --set workloadName=zavapower \ + -o zavapower-ops/ + +./bin/deploy.sh zavapower-ops/ +./bin/verify-agent.sh zavapower-ops/ +``` + +## Parameters + +| Param | Required | Example | +|---|---|---| +| `agentName` | ✅ | `zavapower-ops` | +| `resourceGroup` | ✅ | `rg-zavapower-ops` | +| `location` | ✅ | `eastus2` | +| `targetRGs` | ✅ | `rg-zavapower-prod` | +| `appInsightsId` | ✅ | `/subscriptions/.../components/appi-zavapower` | +| `appInsightsAppId` | ✅ | App Insights App ID GUID | +| `lawResourceId` | ✅ | Log Analytics workspace resource ID | +| `snowInstance` | ✅ | `dev123456` (PDI subdomain) | +| `containerAppPrefix` | ⛔ | `powergrid` | +| `workloadName` | ⛔ | `zavapower` | +| `datadogApiKey` | ⛔ | leave blank to skip Datadog | +| `dynatraceTenantUrl` | ⛔ | leave blank to skip Dynatrace | + +## What gets deployed + +### Connectors +- App Insights (workload telemetry) +- Log Analytics (`ContainerAppConsoleLogs`) +- Azure Monitor (alert routing) +- Datadog MCP (optional) +- Dynatrace MCP (optional) +- ServiceNow Incident Platform (configured via UI after deploy) + +### Subagents (8) +| Name | Role | +|---|---| +| `incident-handler` | Primary 5xx / latency / restart investigator. Documents to ServiceNow. | +| `deployment-validator` | Validates a rollout against alert noise & error baseline. | +| `vm-ops-agent` | Disk pressure & VM-level remediation (Azure Arc / VMs). | +| `utility-ops-agent` | Daily fleet audit deck — read-only, report-only. | +| `web-app-troubleshooter` | App Service / front-end specific path. | +| `pod-incident-remediator` | ACA-replica-level remediation (restarts, scale-out). | +| `release-orchestrator` | Pipeline → SRE → release flow coordinator. | +| `pipeline-failure-investigator` | ADO build/release failure diagnosis. | + +### Skills (15) +Service-specific diagnosis skills (`outage-api-diagnosis`, `meter-api-diagnosis`, …), classes of regression (`crash-`, `config-`, `perf-`), and ops procedures (`deployment-rollback`, `deployment-validation`, `repo-routing`, `release-on-sre-fix`, `pod-fleet-audit-deck`, `plot-incident-metrics`, `disk-pressure-diagnosis`, `sre-agent-customizer`). + +### Automations +- **Incident filter** `auto-investigate-azmon` — routes the 3 ACA alert rules to `incident-handler` +- **Scheduled task** `pod-fleet-audit-daily` — 8 AM UTC daily, runs `utility-ops-agent` to produce a .pptx deck + +## Incident Platform setup (post-deploy) + +In the SRE Agent UI for the deployed agent: + +1. **Builder → Incidents → Connect platform → Azure Monitor** → done automatically by `connectors.json`. +2. **Builder → Incidents → Connect platform → ServiceNow** → enter `https://.service-now.com` + creds. + +## Custom tools the agent depends on + +The recipe references these tool names but does not ship them. They are part of the lab and must be uploaded via Builder before the agent runs: + +- `CreateServiceNowIncident`, `UpdateServiceNowWorkNotes`, `LookupServiceNowIncident` +- `UploadChartToServiceNow`, `UploadDeckToServiceNow`, `UploadServiceNowAttachment` +- `GenerateAuditDeck`, `PythonChartGenerator`, `PythonScriptRunner` +- `RunAzCliReadCommands`, `RunAzCliWriteCommands` (provided by SRE Agent platform) +- `GetADOBuildDetails`, `GetADOReleaseDetails`, `RestartADOBuild` (if ADO integration enabled) + +See `labs/zava-power/sre-config/tools/` for the source YAMLs. + +## Verifying + +```bash +./bin/verify-agent.sh zavapower-ops/ +``` + +Should report: +- 8 subagents present +- 15 skills present +- 1 scheduled task +- 1 incident filter (`auto-investigate-azmon`) +- Connectors: app-insights, log-analytics, azure-monitor (datadog/dynatrace if their params were set) + +## Cost + +Higher than the IT-support recipe — incidents fan out to multiple subagents. Default monthly Agent Unit cap = 25000. Tune in `agent.json` based on incident volume. diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/agent.json b/labs/recipes/azmon-aca-servicenow-zavapower-ops/agent.json new file mode 100644 index 000000000..01fc63dd5 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/agent.json @@ -0,0 +1,103 @@ +{ + "_scenario": "azmon-aca-servicenow-zavapower-ops", + "_description": "Production-ops SRE agent for Zava Power's microservices platform on Azure Container Apps. Handles 5xx alerts, latency, container restarts, deployment validation, VM disk pressure, pipeline failures, and pod fleet audits — across 5 services. Native ServiceNow + Azure Monitor incident platforms; optional Datadog and Dynatrace MCP connectors.", + "_prerequisites": [ + "Azure subscription with SRE Agent RP access", + "Azure Container Apps environment running the 5 PowerGrid services (or another ACA workload)", + "Application Insights resource for the workload", + "Log Analytics workspace", + "Azure Monitor alert rules (the recipe expects: alert-powergrid-http-5xx, alert-powergrid-high-latency, alert-powergrid-container-restart)", + "ServiceNow instance for incident documentation", + "(Optional) Datadog API key", + "(Optional) Dynatrace tenant + token" + ], + "_prompts": { + "agentName": { + "ask": "Agent name", + "default": "zavapower-ops-agent" + }, + "resourceGroup": { + "ask": "Resource group (where the agent will live)", + "default": "rg-zavapower-ops" + }, + "location": { + "ask": "Region", + "options": ["eastus2", "swedencentral", "uksouth", "australiaeast"], + "default": "eastus2", + "required": true + }, + "targetRGs": { + "ask": "Workload resource groups to monitor (comma-separated)", + "required": true + }, + "appInsightsId": { + "ask": "Application Insights resource ID for the workload", + "required": true + }, + "appInsightsAppId": { + "ask": "Application Insights App ID (GUID)", + "required": true + }, + "lawResourceId": { + "ask": "Log Analytics workspace resource ID", + "required": true + }, + "snowInstance": { + "ask": "ServiceNow instance hostname (e.g. dev123456 — without .service-now.com)", + "required": true + }, + "containerAppPrefix": { + "ask": "Container App name prefix (used in az CLI calls — e.g. powergrid)", + "default": "powergrid" + }, + "workloadName": { + "ask": "Workload short name (used in tags + dashboard titles)", + "default": "zavapower" + }, + "datadogApiKey": { + "ask": "Datadog API key (leave blank to skip Datadog integration)", + "default": "" + }, + "dynatraceTenantUrl": { + "ask": "Dynatrace tenant URL (e.g. https://abc12345.live.dynatrace.com — leave blank to skip)", + "default": "" + }, + "modelProvider": { + "ask": "AI model provider", + "options": ["Anthropic", "GitHubCopilot", "MicrosoftFoundry"], + "default": "Anthropic" + }, + "existingUamiId": { + "ask": "Existing UAMI resource ID (leave blank to create new)", + "default": "" + }, + "existingAgentAppInsightsId": { + "ask": "Existing App Insights resource ID for agent telemetry (leave blank to create new)", + "default": "" + } + }, + "identity": { + "agentName": "{{agentName}}", + "resourceGroup": "{{resourceGroup}}", + "subscription": "", + "location": "{{location}}", + "targetResourceGroups": "{{targetRGs}}" + }, + "access": { + "accessLevel": "High", + "actionMode": "Autonomous" + }, + "upgradeChannel": "Preview", + "defaultModelProvider": "{{modelProvider}}", + "monthlyAgentUnitLimit": 25000, + "tags": { + "scenario": "zavapower-ops", + "workload": "{{workloadName}}" + }, + "toggles": { + "enableWebhookBridge": true, + "webhookBridgeTriggerUrl": "" + }, + "existingUamiId": "{{existingUamiId}}", + "existingAgentAppInsightsId": "{{existingAgentAppInsightsId}}" +} diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/automations/incident-filters/auto-investigate-azmon.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/automations/incident-filters/auto-investigate-azmon.yaml new file mode 100644 index 000000000..a34abf764 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/automations/incident-filters/auto-investigate-azmon.yaml @@ -0,0 +1,21 @@ +metadata: + name: auto-investigate-azmon +spec: + incidentPlatform: AzureMonitor + isEnabled: true + priorities: + - '1' + - '2' + - '3' + incidentType: LiveSite + handlingAgent: incident-handler + agentMode: Autonomous + deepInvestigationEnabled: false + maxAutomatedInvestigationAttempts: 3 + azureMonitorFilterSettings: + alertRules: + - alert-powergrid-http-5xx + - alert-powergrid-high-latency + - alert-powergrid-container-restart + triggerEvents: + - AlertFired diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/automations/incident-platforms/azure-monitor.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/automations/incident-platforms/azure-monitor.yaml new file mode 100644 index 000000000..876551e04 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/automations/incident-platforms/azure-monitor.yaml @@ -0,0 +1,3 @@ +name: azure-monitor +spec: + platformType: AzureMonitor diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/automations/incident-platforms/servicenow.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/automations/incident-platforms/servicenow.yaml new file mode 100644 index 000000000..1d38ac48e --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/automations/incident-platforms/servicenow.yaml @@ -0,0 +1,3 @@ +name: servicenow +spec: + platformType: ServiceNow diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/automations/scheduled-tasks/pod-fleet-audit-daily.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/automations/scheduled-tasks/pod-fleet-audit-daily.yaml new file mode 100644 index 000000000..e34e37ed3 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/automations/scheduled-tasks/pod-fleet-audit-daily.yaml @@ -0,0 +1,10 @@ +metadata: + name: pod-fleet-audit-daily +spec: + agent: utility-ops-agent + cronExpression: 0 8 * * * + isEnabled: true + agentPrompt: "Run the pod-fleet-audit-deck skill end-to-end. Window: last 48 hours (UTC). Scope: all Container Apps in the\ + \ target resource group. Output: ONE .pptx deck attached to this thread plus a one-paragraph executive summary. HARD CONSTRAINTS:\ + \ do not create/modify ServiceNow incidents, do not run remediation, do not run incident-handler phases \u2014 only the\ + \ deck workflow defined in the skill." diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/config-regression-diagnosis.md b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/config-regression-diagnosis.md new file mode 100644 index 000000000..6f8f5266a --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/config-regression-diagnosis.md @@ -0,0 +1,105 @@ +# Config Regression Diagnosis + +## When to use +Invoke after `deployment-validation` returns FAIL with category +`config` (5xx errors but no AppExceptions). Common signatures: +- A previously-required env var was removed from the new revision. +- A downstream URL was changed to point to a wrong / non-existent host. +- A feature flag was flipped on without the dependent code path ready. +- A secret was rotated but the app still references the old value. + +## Investigation steps + +### 1. Diff env vars: new revision vs previous +For the affected Container App: + +```bash +# Current revision env +az containerapp revision show -g {{AZ_RG}} \ + -n {APP_NAME} --revision {NEW_REVISION} \ + --query "properties.template.containers[0].env" -o json + +# Previous revision env +az containerapp revision show -g {{AZ_RG}} \ + -n {APP_NAME} --revision {PREV_REVISION} \ + --query "properties.template.containers[0].env" -o json +``` + +Identify env vars REMOVED, ADDED, or VALUE-CHANGED. Cross-reference +with the per-service diagnosis skill (e.g. `notification-svc-diagnosis`) +which lists each service's REQUIRED env vars. + +### 2. Look for "missing config" responses +```kusto +AppRequests +| where TimeGenerated >= datetime({DEPLOY_TIME}) +| where cloud_RoleInstance has "{REVISION_NAME}" +| where Success == false +| summarize count() by ResultCode, Name +| order by count_ desc +``` +Then sample a failing request body via `AppDependencies` / +`AppTraces` for that OperationId — the response body often says +"REQUIRED_CONFIG not set" or similar. + +### 3. Container console for explicit warnings +```kusto +ContainerAppConsoleLogs_CL +| where TimeGenerated >= datetime({DEPLOY_TIME}) +| where RevisionName_s == "{REVISION_NAME}" +| where Log_s contains "config" or Log_s contains "env" + or Log_s contains "missing" or Log_s contains "REQUIRED" +| take 20 +``` + +### 4. Validate downstream URLs respond +For each external URL referenced by the app's env, do a quick +`ProbeServiceLatency(name, url, '/healthz', count=2)` to confirm +reachability. A 5xx because the app can't reach `https://api.partner/` +is still a config-shaped failure (wrong URL or firewall change). + +### 5. Pinpoint the exact config delta + the code path that fails (REQUIRED) +A generic "config missing" is NOT acceptable. The lab's config +regressions are typically **hardcoded source-level constants** (not +runtime env vars), so the diff must inspect actual code: + a. Get the build commit SHA from the failing build via + `GetPipelineRunHistory` on **PowerGrid-Build** → `sourceVersion`, + plus the previous healthy build's SHA. + b. Browse the failing service's source dir for changed constants — + URLs, ports, hostnames, feature flags, timeout values. + c. The actual failure path is usually a downstream call that times + out or refuses connection because the constant points to the + wrong endpoint. Trace it back to its declaration site. + d. Quote the offending line(s) (≤5 lines) verbatim from the file, + with file path and line numbers, plus the dependent call site. + e. State the mechanism: WHICH constant changed, WHERE it's used, + WHY the new value is wrong (port closed, host renamed, TLS + mismatch, etc.), and HOW the failure surfaces (timeout, conn + refused, 503, etc.). + +## Output to caller + +Output schema (fill from your investigation — do NOT invent values): + +``` +CONFIG REGRESSION RCA + service: + revision: + deploy_time: + symptom: + count_5min: + prior revision: + code_cause: | + : + (commit , build #): + + + + + fix direction: +``` + +Hand off to `deployment-rollback` → `servicenow-incident-mgmt` → +`repo-routing`. The `code_cause` block goes verbatim into the +SNOW **Root Cause** section. diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/config-regression-diagnosis.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/config-regression-diagnosis.yaml new file mode 100644 index 000000000..ff2fc0a7c --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/config-regression-diagnosis.yaml @@ -0,0 +1,9 @@ +metadata: + name: config-regression-diagnosis + description: "Deep-dive diagnosis when deployment-validation has flagged a service\nwith elevated 5xx errors but NO exceptions\ + \ in App Insights \u2014 the\nclassic shape of a missing-env-var or bad-downstream-URL regression\nwhere the app returns\ + \ a clean 5xx without crashing." + spec: + tools: [] +skillContent: skills/config-regression-diagnosis.md +additionalFiles: [] diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/crash-regression-diagnosis.md b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/crash-regression-diagnosis.md new file mode 100644 index 000000000..49808c26d --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/crash-regression-diagnosis.md @@ -0,0 +1,93 @@ +# Crash Regression Diagnosis + +## When to use +Invoke after `deployment-validation` returns FAIL with category +`crash` (5xx errors AND exceptions present in revision-scoped AI). + +## Investigation steps + +### 1. Top exceptions on the new revision +```kusto +AppExceptions +| where TimeGenerated >= datetime({DEPLOY_TIME}) +| where cloud_RoleInstance has "{REVISION_NAME}" +| summarize count() by Type, OuterMessage +| order by count_ desc +``` + +### 2. Sample full stack trace +```kusto +AppExceptions +| where TimeGenerated >= datetime({DEPLOY_TIME}) +| where cloud_RoleInstance has "{REVISION_NAME}" +| project TimeGenerated, Type, OuterMessage, Method, Details +| take 3 +``` + +### 3. Check for container-level failures (OOM, ImagePullBackOff) +Use **Monitor Resource Log Query** on +`ContainerAppSystemLogs_CL` table filtered by +`RevisionName_s == "{REVISION_NAME}"`. Look for: +- `OOMKilled` — bump memory request OR fix leak +- `ImagePullBackOff` / `ErrImagePull` — image tag missing in ACR +- `CrashLoopBackOff` — process exits at startup; check console logs +- `Liveness probe failed` — endpoint never came up + +### 4. Check container console logs for startup errors +```kusto +ContainerAppConsoleLogs_CL +| where TimeGenerated >= datetime({DEPLOY_TIME}) +| where RevisionName_s == "{REVISION_NAME}" +| where Log_s contains "Error" or Log_s contains "Exception" + or Log_s contains "ImportError" or Log_s contains "ModuleNotFound" + or Log_s contains "Traceback" +| project TimeGenerated, Log_s +| take 20 +``` + +### 5. Diff against previous revision env +If exceptions reference missing config / undefined vars, also call the +service-specific diagnosis skill (e.g. `outage-api-diagnosis`) to +inspect env-var differences vs the prior revision. + +### 6. Pinpoint the code change (REQUIRED for the SNOW summary) +A generic "exception in the code" is NOT acceptable. Identify the +SPECIFIC change: + a. Get the build commit SHA from the failing build via + `GetPipelineRunHistory` on **PowerGrid-Build** → `sourceVersion`. + b. Get the previous healthy build's SHA the same way. + c. Browse the diff for the failing service — focus on the file and + line referenced in the exception stack trace. + d. Quote the exact function and the offending lines (≤5 lines) + verbatim from the file, with file path and line numbers. + e. State the mechanism: WHICH line throws, WHAT input causes it, + WHY it slipped past tests, what the safe call should be. + +## Output to caller + +Output schema (fill from your investigation — do NOT invent values): + +``` +CRASH REGRESSION RCA + service: + revision: + deploy_time: + symptom: + exception: + count_5min: (compare to request count for the endpoint) + prior revision: + code_cause: | + : + (commit , build #): + + + + + fix direction: +``` + +Pass to `deployment-rollback` (immediate mitigation), then +`servicenow-incident-mgmt` (open ticket with this RCA), then +`repo-routing` (file fix PR with this body). The `code_cause` +block goes verbatim into the SNOW **Root Cause** section. diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/crash-regression-diagnosis.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/crash-regression-diagnosis.yaml new file mode 100644 index 000000000..f42464fdd --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/crash-regression-diagnosis.yaml @@ -0,0 +1,15 @@ +metadata: + name: crash-regression-diagnosis + description: 'Deep-dive diagnosis when deployment-validation has flagged a service + + with elevated 5xx errors AND exceptions present in App Insights. + + Identifies whether the cause is an unhandled exception, OOMKilled, + + ImagePullBackOff, missing dependency, or import error. Produces a + + structured RCA suitable for SNOW work note + fix PR.' + spec: + tools: [] +skillContent: skills/crash-regression-diagnosis.md +additionalFiles: [] diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/deployment-rollback.md b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/deployment-rollback.md new file mode 100644 index 000000000..164fc683a --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/deployment-rollback.md @@ -0,0 +1,352 @@ +# deployment-rollback + +## Scope +Generic procedure for safely rolling back any Azure Container App to a previous healthy revision. This skill covers the full lifecycle: identifying which revision to roll back to, pre-rollback safety checks, executing the rollback, and validating recovery. + +--- + +## When to Use This Skill + +Rollback is appropriate when: +- An incident started immediately after a deployment +- The previous revision was known to be healthy +- The fix requires a code change that will take time to develop +- The bad deployment introduced a misconfiguration + +Rollback is **NOT** appropriate when: +- The issue is infrastructure-related (database, networking) — fix the infrastructure instead +- The previous revision has the same issue — rollback won't help +- A database migration was applied that is incompatible with old code — fix forward instead + +--- + +## Phase 1: IDENTIFY — List Revisions and Determine Which Is Healthy + +### 1.1 List All Revisions +```bash +az containerapp revision list \ + -g \ + -n \ + -o table +``` + +Note the output columns: **Name**, **Active**, **Created**, **Traffic Weight**, **Health State**, **Provisioning State**. + +Identify: +- **Current (potentially bad) revision**: the one receiving traffic now +- **Previous revision(s)**: candidates for rollback + +### 1.2 Inspect a Specific Revision +```bash +az containerapp revision show \ + -g \ + -n \ + --revision \ + -o json +``` + +### 1.3 Compare Configurations Between Revisions +```bash +# Current revision's env vars +az containerapp show \ + -g \ + -n \ + --query "properties.template.containers[0].env" \ + -o json + +# Current revision's image +az containerapp show \ + -g \ + -n \ + --query "properties.template.containers[0].image" \ + -o tsv + +# Previous revision's image (to see what changed) +az containerapp revision show \ + -g \ + -n \ + --revision \ + --query "properties.template.containers[0].image" \ + -o tsv +``` + +### 1.4 Confirm Deployment Correlates with Incident Onset +```kql +let errors = ContainerAppConsoleLogs_CL +| where TimeGenerated > ago(24h) +| where ContainerAppName_s == "" +| where Log_s contains "error" or Log_s contains "Error" or Log_s contains "500" or Log_s contains "503" +| summarize ErrorCount = count() by bin(TimeGenerated, 10m); +let deploys = ContainerAppSystemLogs_CL +| where TimeGenerated > ago(24h) +| where ContainerAppName_s == "" +| where Log_s contains "revision" or Log_s contains "Pulling" or Log_s contains "created" +| summarize DeployEvents = count() by bin(TimeGenerated, 10m); +errors +| join kind=fullouter deploys on TimeGenerated +| project TimeGenerated, + ErrorCount = coalesce(ErrorCount, 0), + DeployEvents = coalesce(DeployEvents, 0) +| order by TimeGenerated asc +``` + +If errors spiked at the same time as a deploy event, the deployment is confirmed as the cause. + +--- + +## Phase 2: PRE-ROLLBACK SAFETY CHECKS + +Before rolling back, verify each of these: + +### 2.1 Is the Previous Revision Still Available? +```bash +az containerapp revision list \ + -g \ + -n \ + -o table +``` + +The target revision must appear in the list. If it's been garbage-collected, you cannot roll back to it. + +### 2.2 Was the Previous Revision Actually Healthy? +```kql +ContainerAppConsoleLogs_CL +| where TimeGenerated > ago(48h) +| where ContainerAppName_s == "" +| where RevisionName_s == "" +| where Log_s contains "error" or Log_s contains "Error" or Log_s contains "500" +| summarize ErrorCount = count() +``` + +If ErrorCount is high for the previous revision too, rolling back won't help. Find an older healthy revision or fix forward. + +### 2.3 Were There Database Migrations? +Check whether the current deployment included schema changes. If it did, rolling back to old code that expects the old schema may break things. If unsure, check with the development team before proceeding. + +### 2.4 Will Rollback Break Other Services? +If the new revision introduced a new API contract that other services now depend on, rolling back will break those callers. Check whether other services were deployed at the same time. + +### Safety Check Summary + +| Check | How to Verify | Pass Criteria | +|-------|---------------|---------------| +| Previous revision exists | `az containerapp revision list` | Listed in output | +| Previous revision was healthy | Query historical logs above | Low/zero error count | +| No breaking database migrations | Check deployment notes/changelog | No schema changes | +| No breaking API contract changes | Check caller dependencies | Backwards compatible | + +--- + +## Phase 3: EXECUTE ROLLBACK + +### 3.0 CRITICAL — Detect Revision Mode FIRST +The rollback procedure differs based on the container app's revision mode. +**Always check this first** — calling `ingress traffic set` against a Single +mode app will fail with: *"Containerapp '' is configured for single +revision. Set revision mode to multiple in order to set ingress traffic."* + +```bash +az containerapp show \ + -g \ + -n \ + --query "properties.configuration.activeRevisionsMode" \ + -o tsv +``` + +- Output `Single` → use **3.1A** (image-swap rollback) +- Output `Multiple` → use **3.1B** (traffic-shift rollback) + +--- + +### 3.1A Single-Revision Mode Rollback (image swap) + +In single-revision mode, only one revision serves traffic and it is always +the *latest* one. To roll back you create a NEW revision pointing at the +previous container image — do NOT activate the old revision and do NOT +attempt to set traffic weights. + +Step 1 — discover the previous good image tag (the image of the revision +that was active immediately before the bad one): +```bash +az containerapp revision list \ + -g \ + -n \ + --all \ + --query "sort_by([], &properties.createdTime)[-3:].{name:name, image:properties.template.containers[0].image, created:properties.createdTime}" \ + -o table +``` + +Step 2 — execute the rollback by updating the image. This creates a new +revision whose code is the previous-good code, immediately taking traffic: +```bash +az containerapp update \ + -g \ + -n \ + --image \ + --revision-suffix "rollback$(date +%H%M%S)" +``` + +The `--revision-suffix` makes the rollback revision easy to identify in +later audits (e.g. `{{AZ_APP_PREFIX}}-grid--rollback143052`). + +Step 3 — confirm the new active revision: +```bash +az containerapp revision list -g -n \ + --query "[?properties.active] | [].{name:name, image:properties.template.containers[0].image}" \ + -o table +``` + +The bad revision is automatically deactivated by ACA when the new revision +becomes ready (single-revision mode behavior). No deactivate call needed. + +> Tip — for PowerGrid services the convention is that a stable image tag +> (`acrpowergrid.azurecr.io/:stable`) always points at the last +> known-good build. If unsure of the previous build's numeric tag, swap to +> `:stable` instead. + +--- + +### 3.1B Multiple-Revision Mode Rollback (traffic shift) + +Only use these commands when `activeRevisionsMode == "Multiple"`. + +```bash +az containerapp revision activate \ + -g \ + -n \ + --revision + +az containerapp ingress traffic set \ + -g \ + -n \ + --revision-weight =100 + +az containerapp revision deactivate \ + -g \ + -n \ + --revision +``` + +### Alternative: Gradual Traffic Shift (Canary Rollback) — Multiple mode only +If you want to be cautious, shift traffic gradually: +```bash +# Step 1: 80/20 split — send most traffic to the good revision +az containerapp ingress traffic set \ + -g \ + -n \ + --revision-weight =80 =20 + +# Step 2: Monitor for 5 minutes, then shift fully +az containerapp ingress traffic set \ + -g \ + -n \ + --revision-weight =100 + +# Step 3: Deactivate the bad revision +az containerapp revision deactivate \ + -g \ + -n \ + --revision +``` + +### Quick Reference + +**Single-revision mode (most common for PowerGrid services):** +```bash +# Detect mode + roll back via image swap to the :stable tag in 1 line +PREV_IMG=$(az containerapp revision list -g -n --all \ + --query "sort_by([?properties.active==\`false\`], &properties.createdTime)[-1].properties.template.containers[0].image" -o tsv) +az containerapp update -g -n --image "$PREV_IMG" \ + --revision-suffix "rollback$(date +%H%M%S)" +``` + +**Multiple-revision mode:** +```bash +# Full rollback in 3 commands: +az containerapp revision activate -g -n --revision +az containerapp ingress traffic set -g -n --revision-weight =100 +az containerapp revision deactivate -g -n --revision +``` + +--- + +## Phase 4: VALIDATE — Confirm Rollback Success + +### 4.1 Confirm Active Revision +```bash +az containerapp revision list \ + -g \ + -n \ + --query "[?properties.active==\`true\`].{Name:name, TrafficWeight:properties.trafficWeight, Created:properties.createdTime}" \ + -o table +``` + +The good revision should be the only active revision with 100% traffic. + +### 4.2 Health Check +```bash +curl -s -w "\nHTTP Status: %{http_code}\n" https:///health +``` + +### 4.3 Error Rate After Rollback +```kql +ContainerAppConsoleLogs_CL +| where TimeGenerated > ago(15m) +| where ContainerAppName_s == "" +| summarize + TotalLogs = count(), + ErrorLogs = countif(Log_s contains "error" or Log_s contains "Error" or Log_s contains "500" or Log_s contains "503") +by bin(TimeGenerated, 5m) +| extend ErrorRate = round(100.0 * ErrorLogs / TotalLogs, 2) +| order by TimeGenerated desc +``` + +Error rate should be dropping toward 0%. + +### 4.4 Latency Returned to Normal +```kql +requests +| where timestamp > ago(15m) +| where cloud_RoleName contains "" +| summarize + AvgDuration = avg(duration), + P95 = percentile(duration, 95) +by bin(timestamp, 5m) +| order by timestamp desc +``` + +### 4.5 No Container Restarts +```kql +AzureMetrics +| where TimeGenerated > ago(15m) +| where ResourceProvider == "MICROSOFT.APP" +| where MetricName == "RestartCount" +| where _ResourceId contains "" +| summarize MaxRestarts = max(Maximum) by bin(TimeGenerated, 5m) +| order by TimeGenerated desc +``` + +If any validation step fails, investigate whether the previous revision also has issues. You may need to find an even older revision or fix forward. + +--- + +## Rollback Decision Matrix + +| Situation | Recommended Action | +|-----------|-------------------| +| Bad env var introduced | Remove env var with `--remove-env-vars` (faster than rollback) | +| Bad code in new container image | Rollback to previous revision (this skill) | +| Bad config + bad code | Rollback to previous revision (this skill) | +| Database migration applied | **Do NOT rollback** — fix forward to avoid data integrity issues | +| Infrastructure issue (networking, DB down) | **Do NOT rollback** — fix the infrastructure | + +--- + +## Escalation + +Escalate if: +- The previous revision is also unhealthy +- No known good revision exists (all revisions have been garbage-collected) +- Database schema changes prevent safe rollback +- Multiple services need coordinated rollback +- Rollback does not resolve the incident within 10 minutes diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/deployment-rollback.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/deployment-rollback.yaml new file mode 100644 index 000000000..3a26bc800 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/deployment-rollback.yaml @@ -0,0 +1,8 @@ +metadata: + name: deployment-rollback + description: Execute safe rollback of Azure Container Apps to a previous healthy revision. Use after identifying a bad deployment + as root cause. Includes pre-rollback safety checks and validation steps. + spec: + tools: [] +skillContent: skills/deployment-rollback.md +additionalFiles: [] diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/deployment-validation.md b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/deployment-validation.md new file mode 100644 index 000000000..0abd9a137 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/deployment-validation.md @@ -0,0 +1,173 @@ +# Deployment Validation + +## When to use this skill +Invoke at the start of every post-deploy validation triggered by a +`ReleaseSucceeded` event. The output of this skill is the input to +the rollback / incident decision: only `PASS` for ALL services means +the deploy is healthy. + +## Why a structured skill (not ad-hoc Python) +Prior practice — agent writes urllib code on the fly — caused these +real, observed misses: + +1. Active probes hit only `portal-web`; internal-ingress services were + skipped despite the env having no VNet. Result: any regression in + grid/meter/outage/notify was invisible. +2. Application Insights queries were not scoped to the new revision; + aggregate metrics over 15 min mixed pre-deploy traffic and ambient + simulator probes with the new revision and reported "P95 60ms" while + the new revision was actually 9000ms. +3. No synthetic burst → concurrency-sensitive bugs (lock contention, + pool exhaustion) impossible to catch. + +This skill enforces every probe, scopes every query, and runs a burst. + +--- + +## Phase 1: IDENTIFY THE NEW REVISION (per service) +For every Container App in the release, call: + +``` +GetActiveRevision(app_name, resource_group) +``` + +Record `revision_name` and `created_time_utc` for each. These are the +**only** values that should be used to scope subsequent Log Analytics +queries. Resource group is `{{AZ_RG}}` for all PowerGrid apps. + +Apps deployed by PowerGrid-Release: + +| Logical service | Container App | Public URL (probe target) | +|---|---|---| +| outage-api | {{AZ_APP_PREFIX}}-outage | https://{{AZ_APP_PREFIX}}-outage.proudmoss-f0b5f310.eastus2.azurecontainerapps.io | +| meter-api | {{AZ_APP_PREFIX}}-meter | https://{{AZ_APP_PREFIX}}-meter.proudmoss-f0b5f310.eastus2.azurecontainerapps.io | +| grid-status-api | {{AZ_APP_PREFIX}}-grid | https://{{AZ_APP_PREFIX}}-grid.proudmoss-f0b5f310.eastus2.azurecontainerapps.io | +| notification-svc | {{AZ_APP_PREFIX}}-notify | https://{{AZ_APP_PREFIX}}-notify.proudmoss-f0b5f310.eastus2.azurecontainerapps.io | +| portal-web | app-powergrid-portal | https://app-powergrid-portal.azurewebsites.net | + +If `GetActiveRevision` returns `health_state != Healthy` OR +`provisioning_state != Provisioned` for ANY service → immediate FAIL, +skip to "FAIL handling" below. + +--- + +## Phase 2: ACTIVE PROBES (sequential, ground truth, no telemetry lag) +For EVERY service in the table above, call: + +``` +ProbeServiceLatency(service_name, url, path, count=5, timeout_s=10) +``` + +Suggested primary endpoints (the ones with realistic per-request CPU): + +| Service | Path | +|------------------|----------------| +| outage-api | /healthz | +| meter-api | /healthz | +| grid-status-api | /regions | +| notification-svc | /healthz | +| portal-web | / | + +A service PASSES Phase 2 if `verdict == "PASS"` (all 5/5 OK and +p95 < 1500 ms). Otherwise it is a regression candidate — record but +continue Phase 3 to gather more data before deciding. + +**Do NOT skip any service.** No-VNet means every endpoint is reachable. + +--- + +## Phase 3: SYNTHETIC BURST (concurrency, warm telemetry) +For EVERY service that holds real traffic (skip portal-web /healthz — +portal-web is fronted by App Service, no need to burst), call: + +``` +BurstLoadTest(url, path, concurrency=10, duration_s=15) +``` + +Two purposes: +1. Detect concurrency-sensitive regressions invisible to sequential + probes (lock contention, pool exhaustion). +2. Drive ≥ 50 requests/service into App Insights so Phase 4 KQL + returns useful sample counts before ingestion lag is a problem. + +A service PASSES Phase 3 if `verdict == "PASS"` (zero errors AND +p95 < 1500 ms). + +--- + +## Phase 4: REVISION-SCOPED TELEMETRY (confirmation, post-warm) +Wait 60 seconds after Phase 3 to allow App Insights ingestion. Then +for each service, invoke the existing MCP tool **Monitor Workspace +Log Query** with this KQL template (substituting `{REVISION_NAME}`, +`{DEPLOY_TIME}`, `{SERVICE_NAME}`): + +```kusto +AppRequests +| where TimeGenerated >= datetime({DEPLOY_TIME}) +| where Properties.RevisionName == "{REVISION_NAME}" + or cloud_RoleInstance has "{REVISION_NAME}" +| summarize + sample_count = count(), + p50_ms = percentile(DurationMs, 50), + p95_ms = percentile(DurationMs, 95), + p99_ms = percentile(DurationMs, 99), + error_rate = todouble(countif(Success == false)) / count() + by AppRoleName +``` + +Decision rules: + +- `sample_count < 20` → ingestion still cold; rely on Phases 2 + 3. +- `p95_ms > 1500` → confirmed perf regression. +- `error_rate > 0.05` → confirmed crash/error regression. + +If Phases 2/3 said PASS but Phase 4 says FAIL, trust Phase 4 (it has +real production-like sample size). + +--- + +## Phase 5: VERDICT MATRIX +Combine results from Phases 2 + 3 + 4 per service: + +| Phase 2 | Phase 3 | Phase 4 | Verdict | Category | Next skill | +|---------|---------|---------|---------|----------|---| +| PASS | PASS | PASS or cold | PASS | — | (none — post Teams success) | +| FAIL (latency) | * | * | FAIL | perf | `perf-regression-diagnosis` | +| PASS | FAIL_LATENCY | * | FAIL | perf | `perf-regression-diagnosis` | +| * | FAIL_ERRORS | * | FAIL | crash/config (decide via Phase 4) | `crash-regression-diagnosis` if exceptions present, else `config-regression-diagnosis` | +| PASS | PASS | FAIL p95 | FAIL | perf | `perf-regression-diagnosis` | +| PASS | PASS | FAIL errors | FAIL | crash/config | as above | + +--- + +## Output (return to caller) +Emit a structured summary like: + +``` +DEPLOYMENT VALIDATION RESULT + release_id: + build_id: + per_service: + grid-status-api FAIL (perf) — probe p95=9837ms, burst err=80% + outage-api PASS — probe p95=210ms, burst p95=440ms + meter-api PASS — probe p95=180ms, burst p95=520ms + notification-svc PASS — health_state=Healthy + portal-web PASS — probe p95=425ms + overall: FAIL — proceed to perf-regression-diagnosis on grid-status-api +``` + +## On PASS (all services) +Post to Teams channel: "✅ Deployment validated — no +regression found across 5 services." Include per-service p95 numbers +and link to ADO release. Done. + +## On FAIL (any service) +Hand off to the per-category diagnosis skill identified in Phase 5. +Then (in order): +1. `deployment-rollback` — restore the previous healthy revision. +2. `servicenow-incident-mgmt` — open SNOW with RCA + consolidated + chart from `plot-incident-metrics`. +3. `repo-routing` — file a fix PR against + `{{ADO_ORG}}/{{ADO_REPO}}` with the `sre-agent-fix` ADO build tag + so the `release-orchestrator` agent will trigger the next release + when the fix build succeeds. diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/deployment-validation.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/deployment-validation.yaml new file mode 100644 index 000000000..580bc76fa --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/deployment-validation.yaml @@ -0,0 +1,17 @@ +metadata: + name: deployment-validation + description: 'Authoritative runbook for validating that a just-deployed release did + + NOT introduce a regression. Combines active probing, synthetic burst + + load, and revision-scoped Application Insights queries to produce a + + per-service verdict (PASS / FAIL with category). Replaces the prior + + practice of writing ad-hoc Python in the agent. Always invoke this + + skill at the start of any post-deploy validation flow.' + spec: + tools: [] +skillContent: skills/deployment-validation.md +additionalFiles: [] diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/disk-pressure-diagnosis.md b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/disk-pressure-diagnosis.md new file mode 100644 index 000000000..e8938e35a --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/disk-pressure-diagnosis.md @@ -0,0 +1,135 @@ +# Disk Pressure Diagnosis (Windows) + +## Overview +Investigate and remediate disk space issues on Windows Azure VMs (including Arc-enabled servers). + +## Run-Command Guidelines + +**Variable escaping**: When passing PowerShell via `az vm run-command invoke --scripts`, variables like `$_` and `$PSItem` are mangled by intermediate shell layers. Use these safe patterns: +- `foreach ($var in $collection)` with named variables — never `ForEach-Object { $_ }` +- `Select-Object PropertyName` with direct property names — never calculated properties `@{E={$_...}}` +- `Where-Object PropertyName -eq Value` (simplified syntax) — never `Where-Object { $_.Prop }` +- `Get-CimInstance` for WMI queries that return clean tabular output without pipeline variables + +**Output limits**: Run Command returns only the last ~4096 bytes. Always use `Select-Object -First N` or targeted queries. Never scan all of C:\ recursively — target specific directories. + +**Serial execution (v1)**: Only one `invoke` runs at a time per VM. If a previous command is stuck or timed out, new invocations will block. Use the v2 managed run-command API as a fallback: +```bash +# Create a named run-command (runs independently of the v1 queue) +az vm run-command create --resource-group --vm-name \ + --name --location \ + --script "Get-CimInstance Win32_LogicalDisk -Filter 'DriveType=3' | Format-List DeviceID, Size, FreeSpace" + +# Check result +az vm run-command show --resource-group --vm-name \ + --name --instance-view + +# Clean up afterward (mandatory — this creates a persistent ARM resource) +az vm run-command delete --resource-group --vm-name \ + --name --yes +``` + +## Phase 1: Detect — Confirm Disk Pressure + +### Check disk utilization +```bash +az vm run-command invoke --resource-group --name \ + --command-id RunPowerShellScript \ + --scripts "Get-CimInstance Win32_LogicalDisk -Filter 'DriveType=3' | Format-List DeviceID, Size, FreeSpace" +``` +Compute used percentage from raw Size and FreeSpace values. Warning threshold: above 85% used. Critical: above 95% used. + +### Check disk metrics in Azure Monitor +```kql +InsightsMetrics +| where Namespace == "LogicalDisk" and Name == "FreeSpacePercentage" +| where Computer contains "" +| where TimeGenerated > ago(24h) +| summarize avg(Val) by bin(TimeGenerated, 1h), Tags +| order by TimeGenerated desc +``` + +## Phase 2: Investigate — Find What Is Using Space + +### List top-level directories +```bash +az vm run-command invoke --resource-group --name \ + --command-id RunPowerShellScript \ + --scripts "Get-ChildItem C:\ -Directory -ErrorAction SilentlyContinue | Select-Object Name" +``` + +### Check size of a specific directory +Run this per suspect directory. Do NOT scan all of C:\ recursively — it will timeout. +```bash +az vm run-command invoke --resource-group --name \ + --command-id RunPowerShellScript \ + --scripts "Get-ChildItem C:\data -Recurse -File -ErrorAction SilentlyContinue | Measure-Object -Property Length -Sum" +``` + +### Find large files in a directory +```bash +az vm run-command invoke --resource-group --name \ + --command-id RunPowerShellScript \ + --scripts "Get-ChildItem C:\data -Recurse -File -ErrorAction SilentlyContinue | Sort-Object Length -Descending | Select-Object -First 20 FullName, Length" +``` + +### Common Windows culprits +- `C:\data` — application data, backups, database dumps +- `C:\Windows\Temp` — temporary installer files +- `C:\Windows\Logs\CBS` — component servicing logs +- `C:\Windows\SoftwareDistribution` — Windows Update cache +- `C:\inetpub\logs` — IIS access logs +- `C:\Users\*\AppData\Local\Temp` — user temp files +- `C:\ProgramData` — application state and logs + +## Phase 3: Root Cause — Classify the Problem + +| Pattern | Likely Cause | Check | +|---|---|---| +| Large .bak/.dump files in C:\data | Backup retention not configured | Is there a scheduled task? Is retention set? | +| Single giant .log file | App logging at DEBUG/TRACE level | Check app log config | +| C:\Windows\SoftwareDistribution large | Windows Update cache buildup | Run Dism cleanup | +| C:\Windows\Temp growing | Failed installers or stale temp | Check file ages | +| Sudden spike in usage | One-time event (dump, export, failed job) | Check timestamps of large files | +| Steady growth over days | Data accumulation without cleanup | Check scheduled task outputs | + +## Phase 4: Remediate + +### Option A: Clean up (after confirming files are safe to delete) +```bash +az vm run-command invoke --resource-group --name \ + --command-id RunPowerShellScript \ + --scripts " + Remove-Item C:\data\scada-backups\*.bak -Force -ErrorAction SilentlyContinue + Remove-Item C:\data\grid-logs\*.log -Force -ErrorAction SilentlyContinue + Remove-Item C:\data\grid-logs\*.tmp -Force -ErrorAction SilentlyContinue + Remove-Item C:\Windows\Temp\* -Recurse -Force -ErrorAction SilentlyContinue + Write-Output 'Cleanup complete.' + " +``` + +### Option B: Expand disk +```bash +# Check current disk size +az disk show --resource-group --name --query diskSizeGb + +# Expand (can only increase, not decrease) +az disk update --resource-group --name --size-gb + +# Then extend the partition inside the VM +az vm run-command invoke --resource-group --name \ + --command-id RunPowerShellScript \ + --scripts " + $maxSize = (Get-PartitionSupportedSize -DriveLetter C).SizeMax + Resize-Partition -DriveLetter C -Size $maxSize + Get-CimInstance Win32_LogicalDisk -Filter 'DriveType=3' | Format-List DeviceID, Size, FreeSpace + " +``` + +## Phase 5: Validate +```bash +az vm run-command invoke --resource-group --name \ + --command-id RunPowerShellScript \ + --scripts "Get-CimInstance Win32_LogicalDisk -Filter 'DriveType=3' | Format-List DeviceID, Size, FreeSpace" +``` +Confirm target drive is below 80% used. diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/disk-pressure-diagnosis.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/disk-pressure-diagnosis.yaml new file mode 100644 index 000000000..faaa7fea6 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/disk-pressure-diagnosis.yaml @@ -0,0 +1,9 @@ +metadata: + name: disk-pressure-diagnosis + description: Diagnose and remediate disk pressure on Windows Azure VMs. Investigates disk usage, identifies large files, + old backups, runaway logs, and recommends cleanup or disk expansion. Use when disk utilization alerts fire or a VM reports + low free space. + spec: + tools: [] +skillContent: skills/disk-pressure-diagnosis.md +additionalFiles: [] diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/grid-status-diagnosis.md b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/grid-status-diagnosis.md new file mode 100644 index 000000000..1c381a0e4 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/grid-status-diagnosis.md @@ -0,0 +1,268 @@ +# grid-status-diagnosis + +## Scope +The **grid-status-api** is a Node.js/Express service (`{{AZ_APP_PREFIX}}-grid`) that provides real-time grid status and regional power data. This skill guides you through diagnosing any performance regression — from detecting latency spikes, through identifying whether the cause is code, configuration, or infrastructure, to applying the right fix. + +--- + +## Phase 1: DETECT — Measure the Latency + +### 1.1 Measure Current Response Times +```bash +# Time a request to grid-status-api +curl -s -o /dev/null -w "HTTP Status: %{http_code}\nTime: %{time_total}s\n" \ + https:///api/grid/status + +curl -s -o /dev/null -w "HTTP Status: %{http_code}\nTime: %{time_total}s\n" \ + https:///health +``` + +Compare response times. Are ALL endpoints slow, or only specific ones? This distinction matters in Phase 2. + +### 1.2 App Insights Latency Percentiles +```kql +requests +| where timestamp > ago(2h) +| where cloud_RoleName contains "grid" +| summarize + AvgDuration = avg(duration), + P50 = percentile(duration, 50), + P95 = percentile(duration, 95), + P99 = percentile(duration, 99), + RequestCount = count() +by bin(timestamp, 5m), name +| order by timestamp desc +``` + +Establish: what is the current latency, and when did it change from baseline? Normal baseline is P50 < 200ms, P95 < 500ms. + +### 1.3 Latency Trend — Find the Inflection Point +```kql +requests +| where timestamp > ago(12h) +| where cloud_RoleName contains "grid" +| summarize + P50 = percentile(duration, 50), + P95 = percentile(duration, 95) +by bin(timestamp, 10m) +| order by timestamp asc +``` + +Note the exact time latency spiked. You'll compare this to deployments and other events in Phase 2. + +### 1.4 Upstream Impact — Timeouts from Callers +```kql +ContainerAppConsoleLogs_CL +| where TimeGenerated > ago(2h) +| where Log_s contains "timeout" or Log_s contains "ETIMEDOUT" or Log_s contains "ECONNRESET" +| where Log_s contains "grid" or ContainerAppName_s contains "portal" or ContainerAppName_s contains "outage" +| project TimeGenerated, ContainerAppName_s, Log_s +| order by TimeGenerated desc +``` + +### 1.5 Console Log Errors +```kql +ContainerAppConsoleLogs_CL +| where TimeGenerated > ago(1h) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-grid" +| where Log_s contains "Error" + or Log_s contains "error" + or Log_s contains "WARN" + or Log_s contains "timeout" + or Log_s contains "FATAL" +| project TimeGenerated, Log_s, RevisionName_s +| order by TimeGenerated desc +| take 50 +``` + +--- + +## Phase 2: INVESTIGATE — Find What's Causing the Latency + +### 2.1 Check CPU and Memory — Is It a Resource Issue? +```kql +AzureMetrics +| where TimeGenerated > ago(2h) +| where ResourceProvider == "MICROSOFT.APP" +| where MetricName in ("UsageNanoCores", "WorkingSetBytes") +| where _ResourceId contains "{{AZ_APP_PREFIX}}-grid" +| summarize AvgValue = avg(Average), MaxValue = max(Maximum) by bin(TimeGenerated, 5m), MetricName +| order by TimeGenerated desc +``` + +- **High CPU + high latency** → CPU-bound operation blocking the Node.js event loop (e.g., synchronous computation, tight loop) +- **Normal CPU + high latency** → Not a CPU problem; likely an artificial delay, slow dependency, or connection pool exhaustion +- **High memory** → Possible memory pressure causing GC pauses + +### 2.2 Check Environment Variables +```bash +az containerapp show \ + -g \ + -n {{AZ_APP_PREFIX}}-grid \ + --query "properties.template.containers[0].env" \ + -o table +``` + +Look for any env var that could inject delays, alter timeouts, or change service behavior. + +### 2.3 Correlate Latency Onset with Deployments +```kql +ContainerAppSystemLogs_CL +| where TimeGenerated > ago(24h) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-grid" +| where Log_s contains "revision" or Log_s contains "Pulling" or Log_s contains "Started" or Log_s contains "created" +| project TimeGenerated, Log_s, RevisionName_s +| order by TimeGenerated desc +``` + +Did the latency spike start at the same time as a new revision? If so, the deployment is the likely cause. + +### 2.4 List Revisions — Compare Current with Previous +```bash +az containerapp revision list \ + -g \ + -n {{AZ_APP_PREFIX}}-grid \ + -o table +``` + +### 2.5 Check Dependency Performance +```kql +dependencies +| where timestamp > ago(1h) +| where cloud_RoleName contains "grid" +| summarize + AvgDuration = avg(duration), + P95 = percentile(duration, 95), + FailureCount = countif(success == false) +by target, type +| order by AvgDuration desc +``` + +Slow dependencies (database, downstream APIs) can cause the service to appear slow even if its own code is fast. + +### 2.6 Look for Event Loop Blocking Indicators +```kql +ContainerAppConsoleLogs_CL +| where TimeGenerated > ago(1h) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-grid" +| where Log_s contains "event loop" + or Log_s contains "blocked" + or Log_s contains "CPU" + or Log_s contains "sync" + or Log_s contains "intensive" +| project TimeGenerated, Log_s +| order by TimeGenerated desc +``` + +### 2.7 Check for Connection Pool Issues +```kql +ContainerAppConsoleLogs_CL +| where TimeGenerated > ago(1h) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-grid" +| where Log_s contains "pool" + or Log_s contains "connection" + or Log_s contains "ECONNREFUSED" + or Log_s contains "ENOTFOUND" + or Log_s contains "socket hang up" +| project TimeGenerated, Log_s +| order by TimeGenerated desc +``` + +--- + +## Phase 3: ROOT CAUSE — Interpret Findings + +| Finding | Likely Root Cause | Next Step | +|---------|-------------------|-----------| +| All endpoints uniformly slow, normal CPU/memory | Artificial delay injected via env var or middleware | Check env vars, remove the offending setting | +| Latency spike aligns exactly with deployment | Bad deployment introduced slow code or config | Rollback to previous revision | +| High CPU correlating with latency | CPU-bound synchronous operation blocking event loop | Rollback or fix blocking code | +| Slow on specific endpoints only, others fast | Endpoint-specific issue (slow query, slow dependency) | Check dependency latency for those endpoints | +| Normal latency in App Insights but callers report timeouts | Network-level issue or ingress timeout misconfiguration | Check ingress settings and Container App networking | +| Dependency calls show high latency | Downstream dependency is slow, not this service | Investigate the slow dependency | +| Memory growing + increasing GC pauses | Node.js memory leak causing GC-induced latency | Check for memory leaks, restart or increase memory | + +### Latency Thresholds + +| Metric | Normal | Warning | Critical | +|--------|--------|---------|----------| +| Avg Response Time | < 300ms | 300ms - 2s | > 2s | +| P95 Response Time | < 500ms | 500ms - 5s | > 5s | +| P99 Response Time | < 1s | 1s - 10s | > 10s | +| Timeout Rate | 0% | < 1% | > 1% | + +--- + +## Phase 4: FIX — Apply the Appropriate Remediation + +Choose based on Phase 3 findings. Do NOT guess — match the fix to the diagnosed cause. + +### Option A: Remove a Problematic Environment Variable +```bash +az containerapp update \ + -g \ + -n {{AZ_APP_PREFIX}}-grid \ + --remove-env-vars +``` + +### Option B: Rollback to Previous Revision +Use the `deployment-rollback` skill for a safe rollback procedure. + +### Option C: Scale Out +```bash +az containerapp update \ + -g \ + -n {{AZ_APP_PREFIX}}-grid \ + --min-replicas 2 \ + --max-replicas 5 +``` + +### Verify the Fix +```bash +# Wait 30-60 seconds, then time a request +curl -s -o /dev/null -w "HTTP Status: %{http_code}\nTime: %{time_total}s\n" \ + https:///api/grid/status +``` + +```kql +// Confirm latency has returned to baseline +requests +| where timestamp > ago(15m) +| where cloud_RoleName contains "grid" +| summarize + AvgDuration = avg(duration), + P95 = percentile(duration, 95), + RequestCount = count() +by bin(timestamp, 5m) +| order by timestamp desc +``` + +```kql +// Confirm no more upstream timeouts +ContainerAppConsoleLogs_CL +| where TimeGenerated > ago(15m) +| where Log_s contains "timeout" or Log_s contains "ETIMEDOUT" +| where Log_s contains "grid" +| summarize Count = count() +``` + +```bash +# Confirm active revision +az containerapp revision list \ + -g \ + -n {{AZ_APP_PREFIX}}-grid \ + --query "[?properties.active==\`true\`].{Name:name, Created:properties.createdTime, TrafficWeight:properties.trafficWeight}" \ + -o table +``` + +If latency persists after fix, re-enter Phase 2 with fresh data. + +--- + +## Escalation + +Escalate if: +- Root cause cannot be determined from available metrics and logs +- The fix does not return latency to normal within 5 minutes +- Latency is caused by infrastructure or networking outside your control +- Multiple downstream services are affected (systemic issue) diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/grid-status-diagnosis.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/grid-status-diagnosis.yaml new file mode 100644 index 000000000..0b092e4cd --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/grid-status-diagnosis.yaml @@ -0,0 +1,8 @@ +metadata: + name: grid-status-diagnosis + description: Diagnose and fix grid-status-api performance regressions including high latency, slow /regions responses, and + Node.js event loop blocking. Use when response times exceed 1 second. + spec: + tools: [] +skillContent: skills/grid-status-diagnosis.md +additionalFiles: [] diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/meter-api-diagnosis.md b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/meter-api-diagnosis.md new file mode 100644 index 000000000..836afe7fa --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/meter-api-diagnosis.md @@ -0,0 +1,291 @@ +# meter-api-diagnosis + +## Scope +The **meter-api** is a .NET 8 Web API service (`{{AZ_APP_PREFIX}}-meter`) that manages meter readings and data. This skill guides you through a systematic investigation of container restarts, memory pressure, OOM kills, and other .NET-specific failures. + +--- + +## Phase 1: DETECT — Check Health and Container Stability + +### 1.1 Check Service Health +```bash +curl -s -w "\nHTTP Status: %{http_code}\n" https:///health +curl -s -w "\nHTTP Status: %{http_code}\n" https:///api/meters +``` + +### 1.2 Check Container Restart Count +```kql +AzureMetrics +| where TimeGenerated > ago(4h) +| where ResourceProvider == "MICROSOFT.APP" +| where MetricName == "RestartCount" +| where _ResourceId contains "{{AZ_APP_PREFIX}}-meter" +| summarize MaxRestarts = max(Maximum) by bin(TimeGenerated, 5m) +| order by TimeGenerated desc +``` + +A climbing restart count indicates crash-looping. Note the pattern — constant restarts vs. periodic restarts. + +### 1.3 Memory Usage Trend +```kql +AzureMetrics +| where TimeGenerated > ago(4h) +| where ResourceProvider == "MICROSOFT.APP" +| where MetricName == "WorkingSetBytes" or MetricName == "MemoryPercentage" +| where _ResourceId contains "{{AZ_APP_PREFIX}}-meter" +| summarize AvgValue = avg(Average), MaxValue = max(Maximum) by bin(TimeGenerated, 5m), MetricName +| order by TimeGenerated asc +``` + +Look for the shape: **flat** (healthy), **sawtooth** (OOM crash + restart cycle), or **steady climb** (leak without crash yet). + +### 1.4 Check for System-Level Events (OOM kills, restarts, failures) +```kql +ContainerAppSystemLogs_CL +| where TimeGenerated > ago(2h) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-meter" +| where Log_s contains "restart" + or Log_s contains "OOMKilled" + or Log_s contains "BackOff" + or Log_s contains "Unhealthy" + or Log_s contains "killed" + or Log_s contains "exit" + or Log_s contains "Failed" +| project TimeGenerated, Log_s, RevisionName_s, Reason_s +| order by TimeGenerated desc +``` + +### 1.5 Console Log Errors — Get the Full Picture +```kql +ContainerAppConsoleLogs_CL +| where TimeGenerated > ago(2h) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-meter" +| where Log_s contains "Error" + or Log_s contains "Exception" + or Log_s contains "OutOfMemory" + or Log_s contains "Killed" + or Log_s contains "FATAL" + or Log_s contains "Unhandled" +| project TimeGenerated, Log_s, RevisionName_s +| order by TimeGenerated desc +| take 50 +``` + +--- + +## Phase 2: INVESTIGATE — Diagnose the .NET Failure + +### 2.1 Look for OOM Indicators +```kql +ContainerAppConsoleLogs_CL +| where TimeGenerated > ago(2h) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-meter" +| where Log_s contains "OutOfMemory" + or Log_s contains "System.OutOfMemoryException" + or Log_s contains "OOM" + or Log_s has_any ("GC", "heap", "Heap", "gen0", "gen1", "gen2", "LOH", "finaliz") +| project TimeGenerated, Log_s +| order by TimeGenerated desc +| take 30 +``` + +If OOM-related entries appear, proceed to 2.2. If not, check 2.3 for other exception types. + +### 2.2 Correlate Memory Growth with Restart Events +```kql +let memoryEvents = ContainerAppConsoleLogs_CL +| where TimeGenerated > ago(4h) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-meter" +| where Log_s contains "OutOfMemory" or Log_s contains "OOM" or Log_s contains "memory" +| summarize OOMCount = count() by bin(TimeGenerated, 5m); +let errorEvents = ContainerAppConsoleLogs_CL +| where TimeGenerated > ago(4h) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-meter" +| where Log_s contains "500" or Log_s contains "Error" or Log_s contains "Exception" +| summarize ErrorCount = count() by bin(TimeGenerated, 5m); +memoryEvents +| join kind=fullouter errorEvents on TimeGenerated +| project TimeGenerated, OOMCount = coalesce(OOMCount, 0), ErrorCount = coalesce(ErrorCount, 0) +| order by TimeGenerated asc +``` + +### 2.3 .NET Exception Stack Traces +```kql +ContainerAppConsoleLogs_CL +| where TimeGenerated > ago(2h) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-meter" +| where Log_s contains "Exception" or Log_s contains "StackTrace" or Log_s contains "at " or Log_s contains "Unhandled" +| project TimeGenerated, Log_s +| order by TimeGenerated desc +| take 50 +``` + +Read the stack trace: what exception type, what class/method, what line? This tells you whether it's a memory issue, a dependency failure, or a code bug. + +### 2.4 Check Environment Variables for Suspicious Settings +```bash +az containerapp show \ + -g \ + -n {{AZ_APP_PREFIX}}-meter \ + --query "properties.template.containers[0].env" \ + -o table +``` + +Look for any env vars that could alter memory behavior, enable debug/simulation modes, or misconfigure the runtime. + +### 2.5 Check Container Resource Limits +```bash +az containerapp show \ + -g \ + -n {{AZ_APP_PREFIX}}-meter \ + --query "properties.template.containers[0].resources" \ + -o json +``` + +Is the memory limit sufficient for this workload? A limit that's too low will cause OOM kills even under normal load. + +### 2.6 Check for Recent Deployments +```kql +ContainerAppSystemLogs_CL +| where TimeGenerated > ago(24h) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-meter" +| where Log_s contains "revision" or Log_s contains "Pulling" or Log_s contains "Started" or Log_s contains "created" +| project TimeGenerated, Log_s, RevisionName_s +| order by TimeGenerated desc +``` + +Did the restarts start after a deployment? Compare error onset time with deployment time. + +### 2.7 App Insights Exception Breakdown +```kql +exceptions +| where timestamp > ago(2h) +| where cloud_RoleName contains "meter" +| summarize Count = count(), FirstSeen = min(timestamp), LastSeen = max(timestamp) +by type, problemId, outerMessage +| order by Count desc +``` + +### 2.8 Dependency Health (database, downstream services) +```kql +dependencies +| where timestamp > ago(1h) +| where cloud_RoleName contains "meter" +| where success == false +| summarize FailureCount = count() by target, type, resultCode +| order by FailureCount desc +``` + +--- + +## Phase 3: ROOT CAUSE — Interpret Findings + +| Finding | Likely Root Cause | Next Step | +|---------|-------------------|-----------| +| Memory sawtooth pattern + OOM logs | Memory leak — code or config is causing unbounded allocation | Remove cause of leak (env var, code fix) or increase limits | +| Memory stable but restarts still occur | Non-memory crash — check exit codes and stack traces | Read .NET exception logs | +| Suspicious env var altering memory behavior | Environment-driven simulation/misconfiguration | Remove or correct the env var | +| Memory usage is flat, near limit, no growth | Memory limit too low for normal workload | Increase container memory | +| Stack trace shows dependency connection failure | Database or downstream service is down | Fix dependency, not this service | +| Errors started at exact deployment time | Bad deployment introduced the issue | Rollback to previous revision | +| No recent deployment, errors appear gradually | Resource exhaustion or external change | Check connection pools, dependency health | + +### .NET Memory Indicators + +| Indicator | Normal | Concerning | Critical | +|-----------|--------|------------|----------| +| Working Set | < 200 MB stable | 200-800 MB growing | > 800 MB / near limit | +| GC Gen2 Collections | Infrequent | Increasing | Continuous | +| Restart Count | 0 | 1-2 in 1h | 3+ in 1h | + +--- + +## Phase 4: FIX — Apply the Appropriate Remediation + +Choose based on Phase 3 findings. Do NOT guess — apply the fix that matches the diagnosed root cause. + +### Option A: Remove a Problematic Environment Variable +```bash +az containerapp update \ + -g \ + -n {{AZ_APP_PREFIX}}-meter \ + --remove-env-vars +``` + +### Option B: Increase Memory Limit +```bash +az containerapp update \ + -g \ + -n {{AZ_APP_PREFIX}}-meter \ + --cpu 0.5 \ + --memory 2Gi +``` + +### Option C: Rollback to Previous Revision +Use the `deployment-rollback` skill for a safe rollback procedure. + +### Option D: Restart the Container +```bash +az containerapp revision list \ + -g \ + -n {{AZ_APP_PREFIX}}-meter \ + -o table + +az containerapp revision restart \ + -g \ + -n {{AZ_APP_PREFIX}}-meter \ + --revision +``` + +### Verify the Fix +```bash +# Wait 60 seconds, then: +curl -s -w "\nHTTP Status: %{http_code}\n" https:///health +curl -s -w "\nHTTP Status: %{http_code}\n" https:///api/meters +``` + +```kql +// Confirm no restarts after fix +ContainerAppSystemLogs_CL +| where TimeGenerated > ago(15m) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-meter" +| where Log_s contains "restart" or Log_s contains "OOMKilled" or Log_s contains "killed" +| summarize Count = count() +``` + +```kql +// Confirm memory is stable +AzureMetrics +| where TimeGenerated > ago(30m) +| where ResourceProvider == "MICROSOFT.APP" +| where MetricName == "WorkingSetBytes" +| where _ResourceId contains "{{AZ_APP_PREFIX}}-meter" +| summarize AvgMemory = avg(Average), MaxMemory = max(Maximum) by bin(TimeGenerated, 5m) +| order by TimeGenerated desc +``` + +```kql +// Confirm requests are succeeding +requests +| where timestamp > ago(15m) +| where cloud_RoleName contains "meter" +| summarize + Total = count(), + Success = countif(resultCode startswith "2"), + Failed = countif(resultCode startswith "5") +by bin(timestamp, 5m) +| extend SuccessRate = round(100.0 * Success / Total, 2) +| order by timestamp desc +``` + +If errors persist after fix, re-enter Phase 2 with fresh data. + +--- + +## Escalation + +Escalate if: +- Root cause cannot be determined from available logs and metrics +- Memory continues to grow after removing all suspicious env vars (real application memory leak) +- The issue is in an external dependency (database, networking) outside your control +- Multiple services are experiencing restarts simultaneously diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/meter-api-diagnosis.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/meter-api-diagnosis.yaml new file mode 100644 index 000000000..38ec1f019 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/meter-api-diagnosis.yaml @@ -0,0 +1,8 @@ +metadata: + name: meter-api-diagnosis + description: Diagnose and fix meter-api issues including OOM kills, memory leaks, and .NET container restarts. Use when + meter-api health degrades or containers restart repeatedly. + spec: + tools: [] +skillContent: skills/meter-api-diagnosis.md +additionalFiles: [] diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/notification-svc-diagnosis.md b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/notification-svc-diagnosis.md new file mode 100644 index 000000000..c2014f178 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/notification-svc-diagnosis.md @@ -0,0 +1,282 @@ +# notification-svc-diagnosis + +## Scope +The **notification-svc** is a Go service (`{{AZ_APP_PREFIX}}-notify`) that handles sending notifications and alerts to customers. This skill guides you through diagnosing container crashes, CrashLoopBackOff patterns, and request-level failures by systematically reading logs, checking configuration, and applying the right fix. + +--- + +## Phase 1: DETECT — Is the Container Running? + +### 1.1 Check Container Status — Running, Restarting, or Crashed? +```kql +ContainerAppSystemLogs_CL +| where TimeGenerated > ago(1h) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-notify" +| where Log_s contains "restart" + or Log_s contains "BackOff" + or Log_s contains "CrashLoopBackOff" + or Log_s contains "Unhealthy" + or Log_s contains "exit" + or Log_s contains "terminated" + or Log_s contains "Failed" + or Log_s contains "killed" +| project TimeGenerated, Log_s, RevisionName_s, Reason_s +| order by TimeGenerated desc +``` + +### 1.2 Check Exit Codes +```kql +ContainerAppSystemLogs_CL +| where TimeGenerated > ago(1h) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-notify" +| where Log_s contains "exit code" or Log_s contains "ExitCode" or Log_s contains "exitCode" +| project TimeGenerated, Log_s, RevisionName_s +| order by TimeGenerated desc +``` + +Exit code interpretation: +| Code | Meaning | +|------|---------| +| 0 | Normal exit (shouldn't happen for a long-running service) | +| 1 | Application error (startup validation failure, config issue) | +| 2 | Go runtime panic | +| 137 | SIGKILL (OOM killed by platform) | +| 143 | SIGTERM (graceful shutdown request) | + +### 1.3 Restart Frequency +```kql +AzureMetrics +| where TimeGenerated > ago(2h) +| where ResourceProvider == "MICROSOFT.APP" +| where MetricName == "RestartCount" +| where _ResourceId contains "{{AZ_APP_PREFIX}}-notify" +| summarize MaxRestarts = max(Maximum) by bin(TimeGenerated, 5m) +| order by TimeGenerated desc +``` + +### 1.4 Try to Reach the Service +```bash +curl -s -w "\nHTTP Status: %{http_code}\n" https:///health +curl -s -w "\nHTTP Status: %{http_code}\n" https:///send +``` + +If you get connection refused, 502, or no response — the container is not healthy. If you get a specific error response (400, 500), the container IS running but failing on requests — skip to Phase 2, section 2.5. + +--- + +## Phase 2: INVESTIGATE — Read the Logs to Find Why + +### 2.1 Read Startup Logs — What's the Last Log Before Crash? +```kql +ContainerAppConsoleLogs_CL +| where TimeGenerated > ago(1h) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-notify" +| project TimeGenerated, Log_s, RevisionName_s +| order by TimeGenerated desc +| take 100 +``` + +Look for: +- **Missing config messages**: `not set`, `required`, `missing`, `configuration`, `FATAL` +- **Connection failures**: `connection refused`, `dial tcp`, `no such host`, `DNS` +- **Permission errors**: `permission denied`, `access denied`, `unauthorized` +- **Panic/fatal**: `panic:`, `fatal error:`, `goroutine` + +The last log line before silence is often the smoking gun. + +### 2.2 Go Panic / Fatal Error Stack Traces +```kql +ContainerAppConsoleLogs_CL +| where TimeGenerated > ago(1h) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-notify" +| where Log_s contains "panic" + or Log_s contains "fatal" + or Log_s contains "FATAL" + or Log_s contains "goroutine" + or Log_s contains "runtime error" + or Log_s contains ".go:" + or Log_s contains "signal" +| project TimeGenerated, Log_s +| order by TimeGenerated asc +| take 50 +``` + +Read Go stack traces bottom-up: the goroutine dump shows which function panicked and at which line. + +### 2.3 Check Environment Variables +```bash +az containerapp show \ + -g \ + -n {{AZ_APP_PREFIX}}-notify \ + --query "properties.template.containers[0].env" \ + -o table +``` + +Look for: missing required variables, wrong values, wrong endpoint URLs, wrong port numbers. + +### 2.4 Container Lifecycle Timeline +```kql +ContainerAppSystemLogs_CL +| where TimeGenerated > ago(1h) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-notify" +| project TimeGenerated, Log_s, RevisionName_s, Reason_s +| order by TimeGenerated desc +| take 100 +``` + +This shows the full cycle. A healthy container shows: Pull → Start → Running. A crashing container shows: Pull → Start → Exit → BackOff → Start → Exit → BackOff... + +### 2.5 If Running But Failing — Check Request-Level Errors +```kql +ContainerAppConsoleLogs_CL +| where TimeGenerated > ago(1h) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-notify" +| where Log_s contains "Error" + or Log_s contains "error" + or Log_s contains "timeout" + or Log_s contains "refused" + or Log_s contains "500" + or Log_s contains "failed" +| project TimeGenerated, Log_s +| order by TimeGenerated desc +| take 50 +``` + +### 2.6 Check for DNS / Network Failures +```kql +ContainerAppConsoleLogs_CL +| where TimeGenerated > ago(1h) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-notify" +| where Log_s contains "dial" + or Log_s contains "DNS" + or Log_s contains "no such host" + or Log_s contains "connection refused" + or Log_s contains "ECONNREFUSED" + or Log_s contains "i/o timeout" +| project TimeGenerated, Log_s +| order by TimeGenerated desc +``` + +### 2.7 Check for Recent Deployments +```kql +ContainerAppSystemLogs_CL +| where TimeGenerated > ago(24h) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-notify" +| where Log_s contains "revision" or Log_s contains "Pulling" or Log_s contains "Started" or Log_s contains "created" +| project TimeGenerated, Log_s, RevisionName_s +| order by TimeGenerated desc +``` + +--- + +## Phase 3: ROOT CAUSE — Interpret Findings + +| Finding | Likely Root Cause | Next Step | +|---------|-------------------|-----------| +| Crash on startup, exit code 1, log says "missing" or "not set" | Required environment variable is missing | Add the env var with the correct value | +| Crash on startup, exit code 1, log says "connection refused" or "dial tcp" | Service can't reach a required dependency at startup | Fix the endpoint URL or ensure the dependency is running | +| Crash on startup, exit code 2, panic with stack trace | Go runtime panic — nil pointer, index out of range, etc. | Fix the code bug or rollback | +| Crash on startup, exit code 137 | OOM kill — container uses too much memory at startup | Increase memory limit or fix startup memory usage | +| Container running but `/send` returns 502 | Ingress can't reach the container — wrong port config | Check container port matches ingress target port | +| Container running but `/send` returns 500 | Application error on the request path | Read the error logs for that endpoint | +| Container running but requests timeout | Downstream dependency is slow or unreachable | Check DNS, endpoint URLs, dependency health | +| Errors started exactly at deployment time | Bad deployment | Rollback to previous revision | + +### Go Crash Pattern Reference + +| Pattern | Log Signature | +|---------|---------------| +| Missing config | `FATAL: ... not set`, `missing required`, exit code 1 | +| Nil pointer dereference | `panic: runtime error: invalid memory address` | +| Go panic with stack dump | `goroutine 1 [running]:` followed by `.go:` lines | +| OOM kill | `signal: killed`, exit code 137 | +| Segfault | `signal: segmentation fault` | + +--- + +## Phase 4: FIX — Apply the Appropriate Remediation + +Choose based on Phase 3 findings. The fix depends entirely on what the logs revealed. + +### Option A: Add a Missing Environment Variable +```bash +az containerapp update \ + -g \ + -n {{AZ_APP_PREFIX}}-notify \ + --set-env-vars = +``` + +### Option B: Fix an Incorrect Environment Variable +```bash +az containerapp update \ + -g \ + -n {{AZ_APP_PREFIX}}-notify \ + --set-env-vars = +``` + +### Option C: Remove a Problematic Environment Variable +```bash +az containerapp update \ + -g \ + -n {{AZ_APP_PREFIX}}-notify \ + --remove-env-vars +``` + +### Option D: Rollback to Previous Revision +Use the `deployment-rollback` skill for a safe rollback procedure. + +### Verify the Fix +```bash +# Wait 30-60 seconds for new revision, then: +az containerapp revision list \ + -g \ + -n {{AZ_APP_PREFIX}}-notify \ + -o table + +curl -s -w "\nHTTP Status: %{http_code}\n" https:///health +``` + +```kql +// Confirm no more crash events +ContainerAppSystemLogs_CL +| where TimeGenerated > ago(15m) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-notify" +| where Log_s contains "restart" or Log_s contains "BackOff" or Log_s contains "exit" +| summarize Count = count() +``` + +```kql +// Confirm healthy startup logs +ContainerAppConsoleLogs_CL +| where TimeGenerated > ago(15m) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-notify" +| where Log_s contains "started" or Log_s contains "listening" or Log_s contains "ready" or Log_s contains "healthy" +| project TimeGenerated, Log_s +| order by TimeGenerated desc +``` + +```kql +// Confirm requests are succeeding +requests +| where timestamp > ago(15m) +| where cloud_RoleName contains "notify" +| summarize + Total = count(), + Success = countif(resultCode startswith "2"), + Failed = countif(resultCode startswith "5") +by bin(timestamp, 5m) +| order by timestamp desc +``` + +If the container still crashes after your fix, re-enter Phase 2 — the original root cause diagnosis may have been incomplete. Check for a secondary failure that was masked by the first. + +--- + +## Escalation + +Escalate if: +- Root cause cannot be determined from the available logs +- The container continues to crash after applying the indicated fix +- Go panic stack traces point to a code-level bug requiring a developer +- The issue is in an external dependency or network configuration outside your control +- Multiple services are in crash loops simultaneously diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/notification-svc-diagnosis.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/notification-svc-diagnosis.yaml new file mode 100644 index 000000000..7c138a503 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/notification-svc-diagnosis.yaml @@ -0,0 +1,8 @@ +metadata: + name: notification-svc-diagnosis + description: Diagnose and fix notification-svc failures including CrashLoopBackOff from missing REQUIRED_CONFIG and gateway + timeout from wrong port configuration. Use when notification-svc containers crash or /send returns 502. + spec: + tools: [] +skillContent: skills/notification-svc-diagnosis.md +additionalFiles: [] diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/outage-api-diagnosis.md b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/outage-api-diagnosis.md new file mode 100644 index 000000000..afc22fee0 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/outage-api-diagnosis.md @@ -0,0 +1,254 @@ +# outage-api-diagnosis + +## Scope +The **outage-api** is a Python/Flask service (`{{AZ_APP_PREFIX}}-outage`) that manages power outage reports. This skill guides you through a systematic investigation of any failure in this service — from detecting symptoms, through root-cause analysis of Python tracebacks, to applying the appropriate fix. + +--- + +## Phase 1: DETECT — Identify Symptoms + +### 1.1 Check Service Health +```bash +# Test health endpoint +curl -s -w "\nHTTP Status: %{http_code}\n" https:///health + +# Test functional endpoint +curl -s -w "\nHTTP Status: %{http_code}\n" https:///api/outages +``` + +Compare the results: does `/health` pass while other endpoints fail, or do ALL endpoints fail? This distinction narrows the investigation. + +### 1.2 Query Console Logs for Errors and Tracebacks +```kql +ContainerAppConsoleLogs_CL +| where TimeGenerated > ago(1h) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-outage" +| where Log_s contains "Error" + or Log_s contains "Traceback" + or Log_s contains "Exception" + or Log_s contains "500" + or Log_s contains "503" + or Log_s contains "FATAL" +| project TimeGenerated, Log_s, RevisionName_s +| order by TimeGenerated desc +| take 50 +``` + +### 1.3 Error Rate Over Time — When Did It Start? +```kql +ContainerAppConsoleLogs_CL +| where TimeGenerated > ago(6h) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-outage" +| summarize + TotalLogs = count(), + ErrorLogs = countif(Log_s contains "Error" or Log_s contains "Exception" or Log_s contains "500" or Log_s contains "503") +by bin(TimeGenerated, 5m) +| extend ErrorRate = round(100.0 * ErrorLogs / TotalLogs, 2) +| order by TimeGenerated asc +``` + +Look for the inflection point — when did errors start? Note this timestamp for Phase 2. + +### 1.4 App Insights Request Failures +```kql +requests +| where timestamp > ago(2h) +| where cloud_RoleName contains "outage" +| summarize + Total = count(), + Success = countif(resultCode startswith "2"), + ServerErrors = countif(resultCode startswith "5"), + ClientErrors = countif(resultCode startswith "4") +by bin(timestamp, 5m) +| extend ErrorRate = round(100.0 * ServerErrors / Total, 2) +| order by timestamp desc +``` + +### 1.5 Check Container Status — Is It Running? +```kql +ContainerAppSystemLogs_CL +| where TimeGenerated > ago(1h) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-outage" +| where Log_s contains "restart" + or Log_s contains "Unhealthy" + or Log_s contains "BackOff" + or Log_s contains "killed" + or Log_s contains "exit" +| project TimeGenerated, Log_s, RevisionName_s, Reason_s +| order by TimeGenerated desc +``` + +--- + +## Phase 2: INVESTIGATE — Find the Root Cause + +### 2.1 Read the Full Python Traceback +```kql +ContainerAppConsoleLogs_CL +| where TimeGenerated > ago(1h) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-outage" +| where Log_s contains "Traceback" + or Log_s contains "File \"" + or Log_s contains "raise " + or Log_s has_any ("TypeError", "ValueError", "KeyError", "AttributeError", "ImportError", "ConnectionError", "NoneType") +| project TimeGenerated, Log_s +| order by TimeGenerated desc +| take 50 +``` + +Read the traceback bottom-up. Identify: **which file**, **which line**, **which function**, and **what exception type** was raised. + +### 2.2 Check Environment Variables +```bash +az containerapp show \ + -g \ + -n {{AZ_APP_PREFIX}}-outage \ + --query "properties.template.containers[0].env" \ + -o table +``` + +Look for anything suspicious: unexpected values, missing expected vars, or vars that could change service behavior (e.g., feature flags, error simulation flags, wrong database URLs). + +### 2.3 Check for Recent Deployments — Correlate with Error Onset +```kql +ContainerAppSystemLogs_CL +| where TimeGenerated > ago(24h) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-outage" +| where Log_s contains "revision" or Log_s contains "Pulling" or Log_s contains "Started" or Log_s contains "created" +| project TimeGenerated, Log_s, RevisionName_s +| order by TimeGenerated desc +``` + +Did the errors start immediately after a new revision was deployed? If so, the deployment is likely the cause. + +### 2.4 Compare Current vs Previous Revision +```bash +# List all revisions +az containerapp revision list \ + -g \ + -n {{AZ_APP_PREFIX}}-outage \ + -o table + +# Check current revision's container image +az containerapp show \ + -g \ + -n {{AZ_APP_PREFIX}}-outage \ + --query "properties.template.containers[0].image" \ + -o tsv + +# Check current revision's env vars +az containerapp show \ + -g \ + -n {{AZ_APP_PREFIX}}-outage \ + --query "properties.template.containers[0].env" \ + -o json +``` + +### 2.5 App Insights Exception Details +```kql +exceptions +| where timestamp > ago(2h) +| where cloud_RoleName contains "outage" +| summarize Count = count(), FirstSeen = min(timestamp), LastSeen = max(timestamp) +by type, problemId, outerMessage +| order by Count desc +``` + +### 2.6 Check Dependency Failures +```kql +dependencies +| where timestamp > ago(1h) +| where cloud_RoleName contains "outage" +| where success == false +| summarize FailureCount = count() by target, type, resultCode +| order by FailureCount desc +``` + +--- + +## Phase 3: ROOT CAUSE — Interpret Findings + +Based on your investigation in Phase 2, match your findings to one of these common patterns: + +| Finding | Likely Root Cause | Next Step | +|---------|-------------------|-----------| +| Traceback shows `NoneType` / `AttributeError` | Code bug — variable is None when it shouldn't be | Fix code or rollback revision | +| Traceback shows `ImportError` / `ModuleNotFoundError` | Missing dependency in container image | Rebuild image or rollback | +| Traceback shows `ConnectionError` / `ConnectionRefusedError` | Downstream dependency is unreachable | Check database/dependency health | +| Traceback shows `KeyError` on config | Missing or wrong environment variable | Add/fix the env var | +| All endpoints return same HTTP error, no traceback | Middleware or env-var-driven error mode | Check env vars for flags that alter behavior | +| Errors started exactly when a new revision deployed | Bad deployment | Rollback to previous revision | +| Errors started without any deployment | External dependency failure or config change | Check dependencies and config sources | +| `500` on specific endpoints only | Endpoint-specific code bug | Read traceback for that endpoint | + +--- + +## Phase 4: FIX — Apply the Appropriate Remediation + +Choose the fix based on what Phase 3 revealed. Do NOT apply a fix without first confirming the root cause. + +### Option A: Remove or Fix a Problematic Environment Variable +```bash +# Remove a problematic env var +az containerapp update \ + -g \ + -n {{AZ_APP_PREFIX}}-outage \ + --remove-env-vars + +# Or set an env var to the correct value +az containerapp update \ + -g \ + -n {{AZ_APP_PREFIX}}-outage \ + --set-env-vars = +``` + +### Option B: Rollback to Previous Revision +Use the `deployment-rollback` skill for a safe rollback procedure. + +### Option C: Restart the Container +```bash +az containerapp revision restart \ + -g \ + -n {{AZ_APP_PREFIX}}-outage \ + --revision +``` + +### Verify the Fix +```bash +# Wait 30-60 seconds, then: +curl -s -w "\nHTTP Status: %{http_code}\n" https:///health +curl -s -w "\nHTTP Status: %{http_code}\n" https:///api/outages +``` + +```kql +ContainerAppConsoleLogs_CL +| where TimeGenerated > ago(15m) +| where ContainerAppName_s == "{{AZ_APP_PREFIX}}-outage" +| where Log_s contains "Error" or Log_s contains "Traceback" or Log_s contains "Exception" +| summarize ErrorCount = count() +``` + +Confirm: ErrorCount should be 0 or near 0 after the fix. If errors persist, re-enter Phase 2 with the new data. + +```kql +requests +| where timestamp > ago(15m) +| where cloud_RoleName contains "outage" +| summarize + Total = count(), + Success = countif(resultCode startswith "2"), + Failed = countif(resultCode startswith "5") +by bin(timestamp, 5m) +| extend SuccessRate = round(100.0 * Success / Total, 2) +| order by timestamp desc +``` + +--- + +## Escalation + +Escalate if: +- The root cause cannot be determined from logs and traces +- The fix does not resolve errors within 5 minutes +- The issue is in an external dependency outside your control +- Multiple services are simultaneously affected diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/outage-api-diagnosis.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/outage-api-diagnosis.yaml new file mode 100644 index 000000000..925cc5eeb --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/outage-api-diagnosis.yaml @@ -0,0 +1,8 @@ +metadata: + name: outage-api-diagnosis + description: Diagnose and fix outage-api failures including HTTP 500/503 errors, SCADA enrichment crashes, and FORCE_ERROR + issues. Use when outage-api health checks fail or /outages endpoint returns errors. + spec: + tools: [] +skillContent: skills/outage-api-diagnosis.md +additionalFiles: [] diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/perf-regression-diagnosis.md b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/perf-regression-diagnosis.md new file mode 100644 index 000000000..2691ce99e --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/perf-regression-diagnosis.md @@ -0,0 +1,130 @@ +# Perf Regression Diagnosis + +## When to use +Invoke after `deployment-validation` returns FAIL with category +`perf` for one or more services. Input from caller: `service_name`, +`revision_name`, `deploy_time`, observed p95 from probes. + +## Investigation steps + +### 1. Confirm scope of slowness — which endpoints? +Use **Monitor Workspace Log Query** with: + +```kusto +AppRequests +| where TimeGenerated >= datetime({DEPLOY_TIME}) +| where cloud_RoleInstance has "{REVISION_NAME}" +| summarize p95 = percentile(DurationMs, 95), count() by Name +| order by p95 desc +``` + +If only ONE endpoint is slow → isolated code path (most likely a new +feature). If ALL endpoints are slow → infrastructure / framework / GC. + +### 2. Inspect dependencies +```kusto +AppDependencies +| where TimeGenerated >= datetime({DEPLOY_TIME}) +| where cloud_RoleInstance has "{REVISION_NAME}" +| summarize p95 = percentile(DurationMs, 95), count() by Type, Target +| order by p95 desc +``` + +If dependency p95 ≈ request p95 → downstream is the bottleneck (DB, +external API). If dependency p95 << request p95 → bottleneck is in- +process (CPU, GC, sync code). + +### 3. Sample slow traces +```kusto +AppRequests +| where TimeGenerated >= datetime({DEPLOY_TIME}) +| where cloud_RoleInstance has "{REVISION_NAME}" +| where DurationMs > 1500 +| project TimeGenerated, Name, DurationMs, Url, OperationId +| take 5 +``` +For each OperationId, follow with `union AppRequests, AppDependencies, +AppExceptions, AppTraces | where OperationId == "..."` to see the full +call tree. + +### 4. Compare to previous revision +Repeat (1) for the PREVIOUS revision (use the previous revision_name +from the ACA revision history) over the same kind of window. If the +prior revision had p95 < 200 ms on the same endpoint, that confirms +the new code is the cause. + +### 5. Check container console logs for hot loops / GC +Use **Monitor Resource Log Query** on the Container App's console log +stream filtered by `RevisionName == "{REVISION_NAME}"`. Look for: +- Repeated identical log lines (hot loop) +- GC pause warnings +- "EVENTLOOP_BLOCKED", "long task" warnings (Node) +- Thread pool saturation (Python) +- Per-request log lines emitted from a new code path that signal a + freshly added expensive computation in the handler. + +### 6. Check chaos / latency-injection endpoints +Some services expose admin endpoints that inject server-side latency +(scenario 5 uses these to simulate organic load). For each slow +service, GET `https:///chaos/status`. If `active: true` +or `latency_ms > 0`, **that is your root cause** — not a code +regression. Disable via `DELETE /chaos/latency`. Note: this is an +ORTHOGONAL failure mode to a deploy regression — if you got here from +post-deploy validation, chaos status will normally be inactive and +the cause is in the new image. + +### 7. Pinpoint the code change (REQUIRED for the SNOW summary) +A generic "latency in the code" is NOT acceptable. You must identify +the SPECIFIC change. Steps: + a. Get the build commit SHA from the failing build: + `GetPipelineRunHistory` on **PowerGrid-Build** for buildId + → `sourceVersion` field. + b. Get the previous healthy build's commit SHA the same way. + c. Use `GetFileContents` / repo browse to inspect the diff for the + failing service's source dir. Pay attention to: + - new synchronous CPU-heavy loops over request payloads + (e.g. nested loops, repeated hashing, large JSON walks) + - new external HTTP/DB calls without timeouts + - new locks / mutex contention + - blocking I/O introduced into an async handler (e.g. + `fs.readFileSync` instead of `fs.promises.readFile`) + - new middleware registered on every request + d. Quote the exact function name and the offending lines (≤5 lines) + of source — verbatim from the file, with file path and line + numbers — in the RCA. + e. State the mechanism in plain English: WHICH function, WHAT it + does, WHY it slows requests, by HOW MUCH (latency added per + call, which endpoints are on the affected path, etc.). + +## Output to caller +Return a structured RCA. The `code_cause` field is REQUIRED and must +quote actual source lines (verbatim from the file), not paraphrase. + +Output schema (fill from your investigation — do NOT invent values): + +``` +PERF REGRESSION RCA + service: + revision: + deploy_time: + scope: + p95 before: + p95 after: + dependencies: + chaos_endpoint: + console_log: + code_cause: | + : + (commit , build #): + + + + + fix direction: +``` + +This RCA is the body for the `servicenow-incident-mgmt` work note and +the `repo-routing` PR description. The `code_cause` block goes +verbatim into the SNOW **Root Cause** section so on-callers see the +exact lines without re-investigating. diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/perf-regression-diagnosis.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/perf-regression-diagnosis.yaml new file mode 100644 index 000000000..d0123a67e --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/perf-regression-diagnosis.yaml @@ -0,0 +1,10 @@ +metadata: + name: perf-regression-diagnosis + description: "Deep-dive diagnosis when deployment-validation has flagged a service\nas a perf regression (sequential or\ + \ burst p95 > 1500 ms, no errors).\nIdentifies whether the cause is CPU-bound code (e.g. O(n\xB2) loop),\nslow synchronous\ + \ I/O, blocking dependency, or cold-start. Produces a\none-paragraph root-cause hypothesis suitable for the SNOW work\ + \ note\nand the fix PR description." + spec: + tools: [] +skillContent: skills/perf-regression-diagnosis.md +additionalFiles: [] diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/plot-incident-metrics.md b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/plot-incident-metrics.md new file mode 100644 index 000000000..cbdd852e1 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/plot-incident-metrics.md @@ -0,0 +1,97 @@ +# Plot Incident Metrics + +## Overview +This skill is the **single, canonical way** for any SRE Agent to produce +incident charts. It enforces a **one-chart-per-incident** rule: all +related metrics are overlaid on the same time axis so the responder sees +correlation at a glance, instead of context-switching across multiple +images. + +## Hard rule — ONE chart per incident +- ❌ Do **NOT** call the plotting primitives multiple times per incident +- ❌ Do **NOT** upload more than one chart attachment per SNOW INC +- ✅ Always pass the FULL metric set (below) to a single chart call +- ✅ If a metric is unavailable, omit just that series — still emit one chart + +## Required series (all overlaid, shared time axis) +| # | Series | Source | Notes | +|---|-----------------------|--------------------------|--------------------------------------| +| 1 | Request rate | App Insights `requests` | requests/sec, bin 1m | +| 2 | Error rate (5xx %) | App Insights `requests` | `countif(success==false)/count()*100`| +| 3 | P95 latency (ms) | App Insights `requests` | `percentile(duration,95)` | +| 4 | CPU utilization (%) | ACA / Azure Monitor | `UsageNanoCores / cpuLimit * 100` | +| 5 | Memory utilization (%)| ACA / Azure Monitor | `WorkingSetBytes / memoryLimit * 100`| +| 6 | Request queue depth | ACA ingress metrics | `Requests` queue / pending | +| 7 | Replica count | ACA scale metrics | `Replicas` count | + +## Required annotations on the chart +- **Vertical line** at the deploy timestamp (label: `Deploy `) +- **Vertical line** at the incident detection timestamp (label: `Incident detected`) +- Time window: **30 min before deploy → now** (or +60 min, whichever is shorter) + +## Tools used +- `PlotAreaChartWithCorrelation` (preferred — handles multi-series overlay + and correlation visualization natively) **OR** + `PlotTimeSeriesData` (fallback — one call, multiple series) +- `UploadChartToServiceNow` — exactly **one** invocation, immediately after + the plotting call + +## Standard KQL (App Insights side, for series 1–3) +Use a single union/join query that produces one timeseries per metric: + +```kusto +let svc = ""; +let deployTime = datetime(); +let window = totimespan(90m); +requests +| where cloud_RoleName == svc +| where timestamp between (deployTime - 30m .. deployTime + window) +| summarize + request_rate = count() / 60.0, + error_rate_pct = 100.0 * countif(success == false) / count(), + p95_latency_ms = percentile(duration, 95) + by bin(timestamp, 1m) +| order by timestamp asc +``` + +For series 4–7, query Azure Monitor on the Container App resource for +`UsageNanoCores`, `WorkingSetBytes`, `Requests`, `Replicas` over the +same time range. Pass all results into the single chart call. + +## Workflow +1. **Determine context:** the calling agent provides: + - `service` — e.g. `outage-api` + - `inc_number` — e.g. `INC0010042` + - `deploy_time` — ISO 8601, optional (omit annotation if unknown) + - `incident_time` — ISO 8601, when the issue was detected +2. **Build the multi-series dataset** (KQL above + Azure Monitor queries) +3. **Call the plotting tool ONCE** with all 7 series + annotations +4. **Call `UploadChartToServiceNow` ONCE** with the resulting base64 PNG, + filename `incident-overview-.png` +5. **Add a SNOW work note** linking the attachment and listing the + metrics included (so reviewers know what's in the chart) + +## Constraints +- Do not chart unrelated services in the same image — one chart per + affected service if multiple services are involved +- Do not lower the time resolution below 1-min bins (loses signal) +- Do not crop the time window to exclude the deploy — the deploy + timestamp is the most important reference point +- If chart generation fails twice in a row, fall back to a SNOW work + note that links the raw KQL query instead of looping further + +## Example invocation context +> "outage-api started returning 500s 4 minutes after the v2027.04.17.1 +> deploy. Use plot-incident-metrics with service=outage-api, +> inc_number=INC0010042, deploy_time=2026-04-17T22:30:00Z, +> incident_time=2026-04-17T22:34:12Z." + +The skill produces **one** PNG attached to INC0010042 showing all 7 +series with both vertical reference lines, and the agent moves on to +remediation — no further charting needed. + +## Related +- `deployment-validator` — primary consumer (Phase A visualization) +- `incident-handler` — consumer (PHASE 5 visualization, replaces + ad-hoc charting) +- `servicenow-incident-mgmt` — for the SNOW INC the chart attaches to diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/plot-incident-metrics.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/plot-incident-metrics.yaml new file mode 100644 index 000000000..64810cf5f --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/plot-incident-metrics.yaml @@ -0,0 +1,9 @@ +metadata: + name: plot-incident-metrics + description: "Produce ONE consolidated multi-series chart capturing all incident-relevant\nmetrics for a service, then upload\ + \ it to the ServiceNow incident.\nUse this skill any time an agent needs to visualize an incident \u2014 never\ngenerate\ + \ multiple separate charts per incident." + spec: + tools: [] +skillContent: skills/plot-incident-metrics.md +additionalFiles: [] diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/pod-fleet-audit-deck.md b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/pod-fleet-audit-deck.md new file mode 100644 index 000000000..ff4d5e55c --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/pod-fleet-audit-deck.md @@ -0,0 +1,366 @@ +# Pod Fleet Audit Deck + +## 0. Output contract (must satisfy ALL) + +- Exactly ONE `.pptx` file attached to the agent thread. +- Filename: `powergrid-fleet-audit-.pptx`. +- Slide size: 16:9 widescreen, 13.333 in x 7.5 in (NOT 4:3). +- Final assistant message: 1-paragraph executive summary + link to the + attachment + 4 KPI tiles (Findings / Categories / Auto-fixable / + Fleet status emoji). Total <= 80 words. +- NEVER emit more than one deck per run. +- NEVER emit Markdown instead of a deck. + +--- + +## 1. Inputs + +| Param | Default | Notes | +|---|---|---| +| `window_hours` | `48` | lookback in hours | +| `services` | `["{{AZ_APP_PREFIX}}-outage","{{AZ_APP_PREFIX}}-meter","{{AZ_APP_PREFIX}}-grid","{{AZ_APP_PREFIX}}-notify","{{AZ_APP_PREFIX}}-portal"]` | MUST use the {{AZ_APP_PREFIX}}-* short names because that's what `ContainerAppName_s` contains in this workspace. | +| `friendly_names` | mapping below | shown on slides | +| `resource_group` | `{{AZ_RG}}` | | +| `subscription` | `{{AZ_SUBSCRIPTION_ID}}` | | +| `workspace_id` | `1b8e5f73-805d-4efe-9a29-2489e255f607` | law-powergrid customerId | + +Friendly-name map (use `friendly` anywhere a service appears in user-visible text): + +| ContainerAppName_s | friendly | +|---|---| +| {{AZ_APP_PREFIX}}-outage | outage-api | +| {{AZ_APP_PREFIX}}-meter | meter-api | +| {{AZ_APP_PREFIX}}-grid | grid-status-api | +| {{AZ_APP_PREFIX}}-notify | notification-svc | +| {{AZ_APP_PREFIX}}-portal | portal-web | + +--- + +## 2. Step 1 - fetch data (5 queries, run via Log Analytics REST or the agent's KQL tool) + +> Do NOT skip Step 1. No drafting allowed until all 5 results are in memory. +> Token audience for direct REST: `https://api.loganalytics.io/.default`. +> Endpoint: `https://api.loganalytics.io/v1/workspaces/{workspace_id}/query`. + +`{{services_kql}}` below = the 5 names above as a Kusto string list, e.g. +`'{{AZ_APP_PREFIX}}-outage','{{AZ_APP_PREFIX}}-meter','{{AZ_APP_PREFIX}}-grid','{{AZ_APP_PREFIX}}-notify','{{AZ_APP_PREFIX}}-portal'`. + +### Q1 - Failure events per service (the headline signal) +```kusto +let _start = ago({{window_hours}}h); +let _failure_reasons = dynamic([ + "ReplicaUnhealthy","ContainerBackOff","AssigningReplicaFailed", + "ScaledObjectCheckFailed","Error","OOMKilled","BackOff", + "CrashLoopBackOff","Killing","Unhealthy" +]); +ContainerAppSystemLogs_CL +| where TimeGenerated >= _start +| where ContainerAppName_s in ({{services_kql}}) +| where Reason_s in (_failure_reasons) +| summarize n = count() by ContainerAppName_s, Reason_s +| order by n desc +``` + +### Q2 - Failure events binned for the heat map (1-hour bins) +```kusto +let _start = ago({{window_hours}}h); +let _failure_reasons = dynamic([ + "ReplicaUnhealthy","ContainerBackOff","AssigningReplicaFailed", + "ScaledObjectCheckFailed","Error","OOMKilled","Killing","Unhealthy" +]); +ContainerAppSystemLogs_CL +| where TimeGenerated >= _start +| where ContainerAppName_s in ({{services_kql}}) +| where Reason_s in (_failure_reasons) +| summarize n = count() by ContainerAppName_s, bin(TimeGenerated, 1h) +| order by TimeGenerated asc +``` + +### Q3 - Probe failures (console log scan) +```kusto +let _start = ago({{window_hours}}h); +ContainerAppConsoleLogs_CL +| where TimeGenerated >= _start +| where ContainerAppName_s in ({{services_kql}}) +| where Log_s matches regex @"(?i)(probe|liveness|readiness|unhealthy)" +| summarize probe_failures = count() by ContainerAppName_s +``` + +### Q4 - Sample evidence (first + most-recent failure log line per service) +```kusto +let _start = ago({{window_hours}}h); +ContainerAppSystemLogs_CL +| where TimeGenerated >= _start +| where ContainerAppName_s in ({{services_kql}}) +| where Reason_s in ("ReplicaUnhealthy","ContainerBackOff","Error","AssigningReplicaFailed") +| summarize + first_seen = min(TimeGenerated), + last_seen = max(TimeGenerated), + sample_msg = take_any(Log_s) + by ContainerAppName_s, Reason_s +| order by ContainerAppName_s asc, Reason_s asc +``` + +### Q5 - Replica state (per service, current; NOT KQL - `az` calls) +For each `ContainerAppName_s`: +``` +az containerapp show -n -g {{AZ_RG}} \ + --query "{min:properties.template.scale.minReplicas, max:properties.template.scale.maxReplicas, status:properties.runningStatus}" \ + -o json +``` + +> Do NOT filter App Insights `AppRequests` by `AppRoleName` - it is +> empty/`unknown_service` on this workspace and will silently return zero +> results. Skip traffic metrics; the system logs above already tell the +> story. + +--- + +## 3. Step 2 - classification + +For each service, pick exactly ONE category using its Q1 totals (highest match wins): + +| Category | Rule | +|---|---| +| `crash-loop` | `ContainerBackOff >= 50` OR `BackOff >= 20` OR `CrashLoopBackOff >= 5` | +| `oom` | `OOMKilled >= 1` | +| `probe-misconfig` | `Q3 probe_failures >= 100` AND `ReplicaUnhealthy >= 50` | +| `scaling-flap` | `ScaledObjectCheckFailed >= 50` OR `AssigningReplicaFailed >= 10` | +| `unhealthy-replicas` | `ReplicaUnhealthy >= 50` (catch-all if no above category) | +| `errors-only` | `Error >= 10` (and no above) | +| `healthy` | total failure events == 0 | + +Track per-service: `{service, friendly, category, total_events, top_reason, top_reason_count, evidence_first, evidence_last, sample_msg, recommendation}`. + +--- + +## 4. Step 3 - build the deck (HARD layout rules) + +> Most reported pain in v1 was overflow + cluttered look. Fix it by +> following these rules verbatim. Do NOT improvise sizes. + +### 4.1 Global setup + +```python +from pptx import Presentation +from pptx.util import Inches, Pt, Emu +from pptx.dml.color import RGBColor +from pptx.enum.shapes import MSO_SHAPE +from pptx.enum.text import PP_ALIGN, MSO_ANCHOR, MSO_AUTO_SIZE + +prs = Presentation() +prs.slide_width = Inches(13.333) +prs.slide_height = Inches(7.5) +BLANK = prs.slide_layouts[6] # blank layout - we draw everything ourselves +``` + +### 4.2 Brand tokens (MUST be used everywhere - no other colors) + +| Token | RGB | When | +|---|---|---| +| `BRAND` | `0x0078D4` | titles, accent bars | +| `INK` | `0x1F2937` | body text | +| `MUTED` | `0x6B7280` | sub-labels, footers | +| `BG_LIGHT` | `0xF3F4F6` | KPI cards, table header band | +| `OK` | `0x10B981` | healthy / green pills | +| `WARN` | `0xF59E0B` | scaling-flap, errors-only | +| `BAD` | `0xEF4444` | crash-loop, oom, probe-misconfig, unhealthy-replicas | + +### 4.3 Type scale (no other sizes) + +| Role | Font | Size | Weight | +|---|---|---|---| +| Slide title | Calibri | 28pt | bold | +| Section header | Calibri | 18pt | bold | +| Body bullet | Calibri | 14pt | regular | +| Table cell | Calibri | 12pt | regular | +| KPI big number | Calibri | 36pt | bold | +| KPI label | Calibri | 11pt | regular | +| Footer | Calibri | 9pt | regular | + +### 4.4 Anti-overflow rules (MANDATORY) + +For EVERY text frame you create: +```python +tf = shape.text_frame +tf.word_wrap = True +tf.auto_size = MSO_AUTO_SIZE.NONE # NEVER let pptx auto-grow boxes +tf.margin_left = tf.margin_right = Inches(0.1) +tf.margin_top = tf.margin_bottom = Inches(0.05) +``` + +Truncate before writing. NEVER paste a string longer than the limits below - if it's longer, cut and append `...`: + +| Field | Char limit | +|---|---| +| Slide title | 60 | +| Section header | 50 | +| Bullet line | 90 | +| Table cell | 32 | +| KPI big number | 5 (e.g. "1,473") | +| KPI label | 18 | +| Code/`az` line | 80 (then break to a 2nd line) | +| Sample log evidence | 110 (one line, monospace, then `...`) | + +Bullet lists: max 5 bullets per text box. If you have more, drop the lowest-priority ones (don't shrink font). + +### 4.5 Page grid (in inches; everything snaps to this) + +| Region | Left | Top | Width | Height | +|---|---|---|---|---| +| Title bar | 0.5 | 0.3 | 12.333 | 0.7 | +| Accent rule under title | 0.5 | 1.05 | 12.333 | 0.04 | +| Content area | 0.5 | 1.25 | 12.333 | 5.7 | +| Footer | 0.5 | 7.05 | 12.333 | 0.3 | + +Always draw the title bar + accent rule + footer using a helper function so every slide looks identical. + +Footer text: `PowerGrid Fleet Audit | h window | Generated | Slide N of M` + +--- + +## 5. Slide-by-slide spec (FIXED - do not invent extra slides) + +### Slide 1 - Title (cover) +- BRAND background bar across the top 1.5 inch +- Title (white, 44pt bold): `PowerGrid Fleet Health Audit` +- Subtitle (white, 20pt): `Last h | ` +- Bottom-right small text (MUTED, 10pt): `Zava Power Limited | Confidential` + +### Slide 2 - Executive Summary +- Title: `Executive Summary` +- One sentence (16pt, INK): e.g. `4 of 5 services experienced pod failures in the last 48h; grid-status-api is the largest contributor with 687 events.` +- Below: a row of 4 KPI cards, equally spaced, each card 2.7"w x 1.6"h, BG_LIGHT fill, 0.02" border in BRAND, anchored top:3.0", lefts at 0.5 / 3.5 / 6.5 / 9.5 + - Card 1: total failure events (number + label `Failure events`) + - Card 2: services affected (e.g. `4 / 5`, label `Services affected`) + - Card 3: distinct categories triggered (label `Categories`) + - Card 4: fleet status (one big colored emoji + label `Fleet status`). Use OK/WARN/BAD color band per rule: + - green if 0 services in non-healthy + - amber if 1-2 non-healthy + - red if >=3 non-healthy + +### Slide 3 - Cluster Snapshot (table) +- Title: `Cluster Snapshot` +- Table 6 cols x 6 rows (header + 5 services), pinned at left=0.5", top=1.4", width=12.333", row_height=0.55" + - Cols: `Service` (3.0") | `Status` (1.4") | `Replicas (active/min/max)` (2.5") | `Top failure reason` (2.5") | `Events (48h)` (1.5") | `Category` (1.4") + - Header row: BRAND fill, white bold text + - Body rows: alternate white / BG_LIGHT + - `Status` cell: colored pill (rounded rect inside cell) using OK/WARN/BAD per category + - `Events (48h)` right-aligned + +### Slide 4 - Failure Heat Map +- Title: `Failure Frequency - last h (1-hour bins)` +- Render Q2 result with matplotlib: + ```python + fig, ax = plt.subplots(figsize=(12.0, 4.0), dpi=150) + # rows = services in fixed order, cols = hour bins + im = ax.imshow(matrix, aspect='auto', cmap='Reds') + ax.set_yticks(range(len(services))); ax.set_yticklabels(friendly_names) + ax.set_xticks(every Nth bin); ax.set_xticklabels(HH:MM rotation=0) + ax.set_title('Pod failure events per hour'); fig.colorbar(im, ax=ax, label='events') + fig.tight_layout(); fig.savefig(buf, format='png', bbox_inches='tight') + ``` +- Insert PNG anchored left=0.5", top=1.4", width=12.333", height=5.5" +- Below the chart, one-line caption (MUTED, 12pt) explaining the color scale + +### Slides 5..N - One per non-healthy finding (max 5) +> If more than 5 non-healthy findings, group the lowest-event ones into a single "Other" slide. + +Layout - 4 quadrants, each in a fixed BG_LIGHT rounded-rect tile: + +``` ++----------------------------------------------------------+ +| Title: | [colored pill]| ++--------------------------+-------------------------------+ +| ISSUE | MITIGATION | +| (1.4" tall) | (1.4" tall) | +| - 1-line root cause | `` | +| - Top reason: x | OR "engineering required" | +| - First seen: | | +| - Last seen: | | ++--------------------------+-------------------------------+ +| IMPACT | RECOMMENDATION | +| (1.4" tall) | (1.4" tall) | +| - Repeat count: | - Bullet 1 (<=90 ch) | +| - Operator-min saved: | - Bullet 2 | +| (assume 12 min/event) | - Bullet 3 | ++--------------------------+-------------------------------+ +``` + +Tile dimensions: each 6.0"w x 2.7"h, gap 0.16". Lefts: 0.5 / 6.66. Tops: 1.4 / 4.26. + +Section header (`ISSUE` / `MITIGATION` / `IMPACT` / `RECOMMENDATION`): +- 11pt bold, MUTED color, ALL CAPS +- Anchored top of its tile, left-padded 0.15" + +Body inside each tile: +- 14pt INK bullets, max 4 lines + +Color pill in title-bar (right side): rounded rect, BAD/WARN/OK per category. Width = enough for category label + 0.4" padding. + +### Slide N+1 - Recommendations Roll-up +- Title: `Prevention Recommendations (prioritized)` +- Table 4 cols x (1 + N findings rows), max 6 rows total + - Cols: `Priority` (1.0") | `Service` (2.5") | `Action` (7.5") | `Owner hint` (1.3") + - Priority cell: colored pill `P1` (BAD), `P2` (WARN), `P3` (OK) + - Action text wraps to max 2 lines (<=180 chars total) + +Priority rule: +- `P1` if category in `{crash-loop, oom, probe-misconfig}` +- `P2` if category in `{unhealthy-replicas, scaling-flap}` +- `P3` everything else + +### Slide N+2 - Audit ROI +- Title: `Audit ROI` +- Left half (5.5"w): bullet list (16pt INK, max 5 bullets) + - `Findings detected: ` + - `Auto-remediable: ` + - `Estimated operator-minutes saved: ` (assume 12 min/finding) + - `Time to insight: ~10s` (vs human triage) + - `Coverage: 5/5 services audited` +- Right half (5.5"w): a small horizontal bar chart of "events by category" (matplotlib, 6.0x4.0 in, dpi=150, BRAND-colored bars) + +### Slide N+3 - Appendix (KQL) +- Title: `Appendix - KQL Queries` +- 3 columns of code text (10pt monospace `Consolas`, INK), one per Q1/Q2/Q3, each in its own BG_LIGHT rounded rect +- Code lines wrap at 60 chars (use a hard wrapper in Python before inserting) + +--- + +## 6. Step 4 - attach + summary message + +1. Save as `/tmp/powergrid-fleet-audit-.pptx`. +2. Attach via the runtime's standard thread-attachment mechanism. If + attachment fails, base64-encode and emit as + `data:application/vnd.openxmlformats-officedocument.presentationml.presentation;base64,<...>`. +3. Final assistant message (<= 80 words): + +``` +**PowerGrid Fleet Audit - last h** + +Fleet status: | findings | auto-fixable | ~ operator-min saved + +[powergrid-fleet-audit-.pptx]() + +Top finding: () - x. +``` + +--- + +## 7. Failure handling + +- Single KQL fails -> render its slide with `data unavailable` footer band; continue. Never abort the whole deck. +- python-pptx pip-install fails -> emit a Markdown deck-equivalent labeled `Fallback: pptx unavailable`. +- Empty fleet (no events) -> 2-slide deck (Title + "No activity in window - fleet is healthy"). + +--- + +## 8. Anti-patterns (do NOT do these) + +- Use `requests` / `cloud_RoleName` (App Insights tables) - `AppRoleName` is empty on this workspace. +- Use reason names like `Started`, `BackOff`, `CrashLoopBackOff` as the primary failure filter - they're not what ACA emits. Use the list in section 2 Q1. +- Auto-grow text frames (`MSO_AUTO_SIZE.TEXT_TO_FIT_SHAPE`) - produces uneven slides. +- Multiple charts per finding slide - use the 4-quadrant layout only. +- Add new colors beyond the 7 brand tokens. +- Add slides not listed in section 5. +- Create SNOW tickets, call remediation tools, or run any Phase 1-4 of `utility-ops-agent`. diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/pod-fleet-audit-deck.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/pod-fleet-audit-deck.yaml new file mode 100644 index 000000000..96eeccb08 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/pod-fleet-audit-deck.yaml @@ -0,0 +1,15 @@ +metadata: + name: pod-fleet-audit-deck + description: 'Generate ONE polished executive PowerPoint deck summarizing pod-level + + health across all 5 PowerGrid Container Apps over a configurable + + lookback window (default 48h). Read-only. NEVER creates SNOW tickets. + + NEVER calls remediation tools. Used by the daily "PowerGrid Fleet + + Audit Deck" scheduled task.' + spec: + tools: [] +skillContent: skills/pod-fleet-audit-deck.md +additionalFiles: [] diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/release-on-sre-fix.md b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/release-on-sre-fix.md new file mode 100644 index 000000000..ccd303d98 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/release-on-sre-fix.md @@ -0,0 +1,92 @@ +# Release on SRE Fix + +## When to use +Triggered by `BuildSucceeded` events on the **PowerGrid-Build** +pipeline. The release-orchestrator agent loads this skill on every +such event. + +## Why a filter is necessary +ADO release triggers do not natively filter by author or commit tag. +Without this skill, every successful build (including human dev +commits) would auto-release. We want auto-release ONLY for SRE-Agent +fixes that have already been validated by the agent's own diagnosis. + +## Configuration (lab-specific — edit for your environment) +- **SRE Agent service principal UPN** (used to identify SRE-authored + builds when the `sre-agent-fix` tag is missing): set this in your + agent prose, e.g. `sre-agent@yourtenant.onmicrosoft.com`. If your + PR-creation flow always tags the resulting build with + `sre-agent-fix`, the SP UPN check is optional. +- **Build pipeline name**: `PowerGrid-Build`. +- **Release pipeline name**: `PowerGrid-Release`. + +The built-in ADO MCP tools accept either pipeline names or numeric +IDs; prefer names so this skill is portable across environments +where the IDs differ. + +## Decision flow + +### 1. Read the build +Use the built-in ADO MCP tool **`GetPipelineRunHistory`** on the +**PowerGrid-Build** pipeline, filtered to the run that triggered the +event. Capture: +- `tags` (array of strings) +- `requestedFor.uniqueName` +- `result` (must be `succeeded`) + +### 2. Idempotency check +If `tags` already contains a value matching `sre-agent-release-*`, a +release has already been triggered for this build. Post Teams note +"release already in flight" and EXIT. + +### 3. Determine is_sre_agent_fix +``` +is_sre_agent_fix = + ('sre-agent-fix' in tags) + OR + (requestedFor.uniqueName.lower() == SRE_AGENT_SP_UPN.lower()) +``` + +### 4. Decision matrix + +| `result == succeeded` | `is_sre_agent_fix` | Action | +|---|---|---| +| no | any | Post Teams note "build N did not succeed; nothing to release", exit. | +| yes | false | Post Teams note "Build #N succeeded — human-author build, leaving release to normal CI/CD", exit. | +| yes | true | Proceed to step 5. | + +### 5. Trigger PowerGrid-Release +Use the built-in ADO MCP run-pipeline tool (the same one the +pipeline-failure-investigator agent uses for `TriggerBuildPipelineRun` +— there is an equivalent for the release pipeline) to start the +**PowerGrid-Release** pipeline with variables: +- `SOURCE_BUILD_ID = ` +- `TRIGGERED_BY = sre-agent` +- `REASON = auto-release of SRE-Agent fix for buildId=` + +Then add an audit tag to the SOURCE build (best-effort): tag value +`sre-agent-release-`. Use the built-in ADO MCP add-tag +tool. + +### 6. Post to Teams +"🤖 Auto-release triggered: PowerGrid-Release run #<release_id> +materializing SRE-Agent fix from build #<build_id>. The +deployment-validator agent will validate post-deploy." + +EXIT. The deployment-validator agent picks up from ReleaseSucceeded. + +## Why no custom PythonTool +The SRE Agent runtime exposes ADO operations through built-in MCP +tools that are pre-authenticated via delegated OAuth. Custom +PythonTools that call ADO directly require either a PAT (a secret to +manage) or the agent's managed identity to be added as a user in the +ADO org (extra setup). Built-in MCP tools require neither. + +## Loop-safety notes +- Never trigger a release for a build that wasn't tagged — even if + the build originated from an SRE-Agent-authored commit, untagged + builds suggest something is off; let humans investigate. +- Never trigger a release if `result != succeeded`. +- Do not chain triggers: this agent does not trigger another build + from a release (the deployment-validator handles rollback + + fix-PR + new build chain on regression). diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/release-on-sre-fix.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/release-on-sre-fix.yaml new file mode 100644 index 000000000..30a907835 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/release-on-sre-fix.yaml @@ -0,0 +1,10 @@ +metadata: + name: release-on-sre-fix + description: "Authoritative skill for the release-orchestrator agent. When a\nPowerGrid-Build succeeds, this skill decides\ + \ whether to trigger\nPowerGrid-Release. It triggers ONLY for SRE-Agent-authored fixes\n(identified via ADO build tag\ + \ 'sre-agent-fix' or service-principal\nrequestedFor). Human developer commits flow through normal CI/CD\ngates and are\ + \ NOT auto-released by this agent. Uses the runtime's\nbuilt-in ADO MCP tools (delegated OAuth) \u2014 no PAT required." + spec: + tools: [] +skillContent: skills/release-on-sre-fix.md +additionalFiles: [] diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/repo-routing.md b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/repo-routing.md new file mode 100644 index 000000000..98a167890 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/repo-routing.md @@ -0,0 +1,101 @@ +# Repo Routing — single source of truth + +> **Audience.** Both the agent runtime (this file is loaded as a skill) +> AND humans setting up or modifying the lab. Same content; do not +> fork into a separate ARCHITECTURE doc. + +## The three repos and what each is for + +``` +┌──────────────────────────────────────────┐ ┌──────────────────────────────────────────┐ +│ TEMPLATE (public) │ │ PER-USER SRE CONFIG (GitHub) │ +│ {{GH_TEMPLATE_ORG}}/{{GH_TEMPLATE_REPO}} │ │ {{GH_USER}}/{{GH_REPO}} │ +│ │ │ │ +│ - simulator/ setup/ templates/ │ │ - skills/ tools/ agents/ │ +│ - docs/ README.md │ │ - knowledge-base/ │ +│ - NO secrets, NO per-user values │ │ - scheduled-tasks/ hooks/ │ +│ │ │ │ +│ AGENT WRITES: NEVER │ │ AGENT WRITES: KB updates, post-mortems │ +│ HUMAN: clone to bootstrap a new lab │ │ HUMAN: source of truth for agent config │ +└──────────────────────────────────────────┘ └──────────────────────────────────────────┘ + ┌──────────────────────────────────────────┐ + │ PER-USER APP SOURCE (Azure DevOps) │ + │ {{ADO_ORG}}/{{ADO_PROJECT}}/_git/ │ + │ {{ADO_REPO}} │ + │ │ + │ - src/ infra/ pipelines/ │ + │ - bicep modules │ + │ │ + │ AGENT WRITES: fix PRs │ + │ HUMAN: PR review, prod code │ + └──────────────────────────────────────────┘ +``` + +## Allowed tools by operation (HARD CONTRACT) + +| Operation | Target | Use this tool ONLY | +|---|---|---| +| Read GitHub file | `{{GH_USER}}/{{GH_REPO}}` | `get_file_contents` | +| Create GitHub branch | `{{GH_USER}}/{{GH_REPO}}` | `create_branch` | +| Commit single file to GitHub | `{{GH_USER}}/{{GH_REPO}}` | `create_or_update_file` | +| Commit multiple files to GitHub | `{{GH_USER}}/{{GH_REPO}}` | `push_files` | +| Open GitHub PR | `{{GH_USER}}/{{GH_REPO}}` | `create_pull_request` | +| Open GitHub issue | `{{GH_USER}}/{{GH_REPO}}` | `create_issue` | +| Update KB doc / runbook | `{{GH_USER}}/{{GH_REPO}}` `knowledge-base/` | branch + `create_or_update_file` + `create_pull_request` | +| Read ADO file | `{{ADO_ORG}}/{{ADO_PROJECT}}/{{ADO_REPO}}` | ADO MCP `get_file` | +| Open ADO PR (fix) | `{{ADO_ORG}}/{{ADO_PROJECT}}/{{ADO_REPO}}` | `CreateFixPullRequest` | +| Trigger ADO build | `{{ADO_ORG}}/{{ADO_PROJECT}}` | ADO MCP run-pipeline | + +## Forbidden — DO NOT do any of these + +- ❌ `git clone`, `git push`, `git commit` (no git binary in runtime) +- ❌ `gh pr create`, `gh issue create`, any `gh` invocation +- ❌ writing to `{{GH_TEMPLATE_ORG}}/{{GH_TEMPLATE_REPO}}` (template, read-only) +- ❌ writing app source code to `{{GH_USER}}/{{GH_REPO}}` (that's ADO's job) +- ❌ writing agent config to ADO (that's GitHub's job) + +If the right tool is missing, **stop and report** — do NOT shell out to `git`/`gh`. + +## How to file a fix PR (the common case) + +You diagnosed a bug in `src/grid-status-api/server.js` and want to open a fix PR. + +``` +1. branch_name = "fix/grid-status-perf-INC0010069" +2. ADO MCP: GetFileContents owner={{ADO_ORG}} project={{ADO_PROJECT}} + repo={{ADO_REPO}} path=src/grid-status-api/server.js + → modify content in memory +3. CreateFixPullRequest with the modified content + branch_name + title + "Fix grid-status-api perf regression (INC0010069)" +4. Post the PR URL back to ServiceNow via UpdateServiceNowWorkNotes. +``` + +## How to update a knowledge-base doc + +You learned a new diagnostic pattern during incident response and want +to capture it in `knowledge-base/grid-status-tsg.md`. + +``` +1. branch_name = "kb/INC0010069-checksum-pattern" +2. get_file_contents owner={{GH_USER}} repo={{GH_REPO}} + path=knowledge-base/grid-status-tsg.md ref=main +3. Modify content in memory. +4. create_branch owner={{GH_USER}} repo={{GH_REPO}} + branch= from_branch=main +5. create_or_update_file owner={{GH_USER}} repo={{GH_REPO}} + branch= + path=knowledge-base/grid-status-tsg.md + content= message= +6. create_pull_request owner={{GH_USER}} repo={{GH_REPO}} + head= base=main + title="KB: " body="See INC" +7. Post PR URL to SNOW work notes. +``` + +## Why this exists + +Earlier the agent failed a doc update because the prior `create-pr-or-issue` +skill mandated `git push` — but the runtime has no git binary and no +PAT for it. This skill mandates **MCP tools only**, which work autonomously +without portal interaction once the GitHub MCP server is wired with a PAT +that has `Contents:write` + `Pull requests:write` + `Issues:write`. diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/repo-routing.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/repo-routing.yaml new file mode 100644 index 000000000..7e7deacb3 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/repo-routing.yaml @@ -0,0 +1,13 @@ +metadata: + name: repo-routing + description: 'AUTHORITATIVE contract for which repository the agent reads from and + + writes to, and which tools are allowed for each operation. Replaces + + the legacy create-pr-or-issue skill (which incorrectly mandated the + + `gh` and `git` CLIs that are not available in the agent runtime).' + spec: + tools: [] +skillContent: skills/repo-routing.md +additionalFiles: [] diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/sre-agent-customizer.md b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/sre-agent-customizer.md new file mode 100644 index 000000000..bbef1f088 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/sre-agent-customizer.md @@ -0,0 +1,27 @@ +--- +name: SRE Agent customizer +description: +tools: + - sreagent-runtime-mcp_agent_tools + - sreagent-runtime-mcp_agents + - sreagent-runtime-mcp_connectors + - sreagent-runtime-mcp_get_documentation + - sreagent-runtime-mcp_hooks + - sreagent-runtime-mcp_incidents + - sreagent-runtime-mcp_investigate_with_agent + - sreagent-runtime-mcp_investigate_with_agent_yolo + - sreagent-runtime-mcp_memory + - sreagent-runtime-mcp_plan_agent_architecture + - sreagent-runtime-mcp_scheduled_tasks + - sreagent-runtime-mcp_skills + - sreagent-runtime-mcp_threads + - sreagent-runtime-mcp_yaml +--- + + +Use the SRE Agent MCP to help create custom agents and skills. +Don't create additional tools if there are system tools already available. +You will first understand the ask, look at any existing skills, tools and leverage that +You will ask if you need it connected to a trigger or not +You will ask for any design choices +Will finalize the plan and only after user approval create the necessary YAML and md files and then apply it to the agent diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/sre-agent-customizer.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/sre-agent-customizer.yaml new file mode 100644 index 000000000..7d60ea1f5 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/skills/sre-agent-customizer.yaml @@ -0,0 +1,21 @@ +metadata: + name: sre-agent-customizer + description: '' + spec: + tools: + - sreagent-runtime-mcp_agent_tools + - sreagent-runtime-mcp_agents + - sreagent-runtime-mcp_connectors + - sreagent-runtime-mcp_get_documentation + - sreagent-runtime-mcp_hooks + - sreagent-runtime-mcp_incidents + - sreagent-runtime-mcp_investigate_with_agent + - sreagent-runtime-mcp_investigate_with_agent_yolo + - sreagent-runtime-mcp_memory + - sreagent-runtime-mcp_plan_agent_architecture + - sreagent-runtime-mcp_scheduled_tasks + - sreagent-runtime-mcp_skills + - sreagent-runtime-mcp_threads + - sreagent-runtime-mcp_yaml +skillContent: skills/sre-agent-customizer.md +additionalFiles: [] diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/deployment-validator.instructions.md b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/deployment-validator.instructions.md new file mode 100644 index 000000000..0c69007b0 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/deployment-validator.instructions.md @@ -0,0 +1,212 @@ +You are the PowerGrid deployment validator for Zava Power Limited. +You are triggered AUTOMATICALLY after every PowerGrid-Release pipeline +deployment (ReleaseSucceeded on the **PowerGrid-Release** pipeline). + +Your job: validate that the latest release did NOT introduce a +regression. On PASS, post to Teams and exit. On FAIL, mitigate +immediately (rollback), open SNOW with RCA + chart, and file a fix +PR tagged `sre-agent-fix`. The release-orchestrator agent will pick +up the resulting build success and trigger the next release; you +will then be re-invoked to re-validate. The loop is event-driven — +you never poll. + +───────────────────────────────────────────────────────────────────────── +LOOP-SAFETY CHECK (ALWAYS RUN FIRST) +───────────────────────────────────────────────────────────────────────── +1. GetPipelineRunHistory on the **PowerGrid-Release** pipeline → + capture buildId + and runId of the run that triggered you. +2. LookupServiceNowIncident with tag `buildId=` and state + `in-progress`. + - If a matching INC exists AND was updated < 30 min ago, EXIT with + work note "duplicate trigger; INC already handling". +3. Tag every SNOW artifact in this run with `buildId=`. + +───────────────────────────────────────────────────────────────────────── +PHASE 1 — VALIDATE +───────────────────────────────────────────────────────────────────────── +Invoke the **deployment-validation** skill. +It will, for EACH of the 5 services (outage-api, meter-api, +grid-status-api, notification-svc, portal-web): + • call GetActiveRevision → identify the new revision + deploy time + • call ProbeServiceLatency (5 sequential probes, ground truth) + • call BurstLoadTest (concurrent load to surface concurrency bugs) + • wait for ≥20 requests on the new revision then run a + revision-scoped App Insights query (Monitor Workspace Log Query) + +The skill returns a per-service verdict: + verdict ∈ { PASS, FAIL } + category ∈ { perf, crash, config, unknown } (when FAIL) + +Do NOT improvise your own probes. Do NOT skip services. Do NOT query +AI without scoping to the new revision. The skill enforces all of +these. + +───────────────────────────────────────────────────────────────────────── +PHASE 2 — DECIDE +───────────────────────────────────────────────────────────────────────── +IF every service verdict == PASS: + • Post a Teams notification to the configured channel: + "✅ Deployment validated — PowerGrid-Release run # + (buildId=): all 5 services healthy across active probes, + burst load, and revision-scoped telemetry. No regression." + • Add SNOW work note (no incident): "deployment validated, + buildId=" + • EXIT. + +IF any service verdict == FAIL: + • Continue to PHASE 3 (mitigate) and PHASE 4 (long-term fix). + +───────────────────────────────────────────────────────────────────────── +PHASE 3 — IMMEDIATE MITIGATION +───────────────────────────────────────────────────────────────────────── +3a. CreateServiceNowIncident + short_description: "post-deploy regression: " + urgency: 2 (High), impact: 2 (High) + tags: buildId=, runId=, category= + +3b. Invoke skill: **plot-incident-metrics** + ONE consolidated chart (req rate, 5xx %, P95, CPU%, Mem%, + replicas) with deploy timestamp annotated. Upload to SNOW + AND include the returned `markdown` field VERBATIM in your + assistant reply so the chart renders inline in the SRE Agent + thread. Do NOT generate any additional charts. + CAPTURE the returned chart URL — you will reuse it in 4c. + +3c. ROLLBACK — call **RollbackContainerAppRevision** directly with: + app_name= + resource_group={{AZ_RG}} + target_image_tag= + This is the autonomous path — uses the agent's MI, no approval + prompt. Do NOT use system Azure tools (they will gate on human + confirmation). If the previous tag is unknown, query ACR + repository tags or use GetActiveRevision to inspect the prior + active revision's image reference. + +3d. Re-invoke **deployment-validation** to confirm rollback restored + health. + +3e. UpdateServiceNowWorkNotes: + "mitigated via rollback to ; long-term fix in + progress." + +───────────────────────────────────────────────────────────────────────── +PHASE 4 — LONG-TERM FIX (run only after Phase 3 rollback succeeded) +───────────────────────────────────────────────────────────────────────── +4a. DIAGNOSE ROOT CAUSE — pick the skill matching the verdict + category from Phase 1: + category=perf → invoke skill **perf-regression-diagnosis** + category=crash → invoke skill **crash-regression-diagnosis** + category=config → invoke skill **config-regression-diagnosis** + category=unknown → start with crash-regression-diagnosis; + if no exceptions, fall back to + perf-regression-diagnosis. + ALSO consult the per-service diagnosis skill for context: + outage-api → outage-api-diagnosis + meter-api → meter-api-diagnosis + grid-status-api → grid-status-diagnosis + notification-svc → notification-svc-diagnosis + +4b. OPEN FIX PR — invoke skill **repo-routing** with: + • repo: {{ADO_ORG}}/{{ADO_REPO}} + • source branch: `sre-agent/fix--` + • target branch: `main` + • title: "fix(): " + • body MUST include: + - Resolves SNOW + - Root cause (from 4a, 1-3 sentences) + - Summary of the change + - Rolled-back revision id + - Original failed buildId + • The PR MUST contain a REAL CODE FIX authored from the diagnosis + in 4a — surgical edits that preserve engineering intent. NEVER + file a PR that only reverts the bad commit; engineering owns + the feature, the SRE Agent provides the fix. + • Set the PR to **auto-complete with squash + delete source + branch**. Branch policies on main are intentionally permissive + so the PR will merge server-side within seconds. + • IMPORTANT: when the PR is merged and the resulting build + starts, that build MUST be tagged `sre-agent-fix`. Either the + PR-creation skill or your service principal identity provides + this — verify by including the tag instruction in the PR body + and ensuring you commit as the SRE Agent service principal. + The `release-orchestrator` agent gates release on this tag. + +4c. UpdateServiceNowWorkNotes on the original incident with a + complete artifact summary. Use Markdown so links are clickable + in both SNOW and the agent thread reply: + + ``` + ## Root Cause + + + ## Config Delta (if applicable) + + + ## Failed Deployment Artifacts + - Failed Build: [#]() + - Failed Release: [#]() + - Rolled-back revision: `` + - Incident chart: [view chart]() + + ## Long-term Fix + - Fix PR: [#]() + - Branch: `sre-agent/fix--` + - Tag on resulting build: `sre-agent-fix` + + ## Next Step + Build will auto-trigger on PR merge. release-orchestrator agent + will trigger PowerGrid-Release on build success; this validator + will be re-invoked on ReleaseSucceeded. + ``` + + ALSO emit the same Markdown block VERBATIM in your assistant + reply so the SRE Agent thread shows clickable links and the + chart inline. The thread reply is the operator's primary view — + do not summarize, do not strip URLs. + +4d. EXIT. Do NOT trigger build or release yourself. Do NOT poll. + +───────────────────────────────────────────────────────────────────────── +RE-INVOCATION (same agent, new buildId) +───────────────────────────────────────────────────────────────────────── +When release-orchestrator triggers a release for the fix build and +that release succeeds, you are re-invoked. The new buildId differs +from the original, so loop-safety does not skip you. +- Re-run Phase 1 against the new deployment. +- Re-run plot-incident-metrics ONCE for the post-fix window + (deploy timestamp = new release time). Capture chart URL. +- If PASS → ResolveServiceNowIncident on the original INC with a + Markdown close-out (also emit verbatim in your assistant reply): + + ``` + ## Fix Validated ✅ + - Fix Build: [#]() (tag `sre-agent-fix`) + - Fix Release: [#]() + - New active revision: `` + - Post-fix chart: [view chart]() + - Original fix PR: [#]() + + All 5 services healthy across active probes, burst load, and + revision-scoped telemetry. Closing INC. + ``` + + Also post Teams success. +- If FAIL → repeat Phase 3 + Phase 4. + +───────────────────────────────────────────────────────────────────────── +GUARDRAILS +───────────────────────────────────────────────────────────────────────── +• Never improvise probe logic — always go through deployment-validation. +• Never roll back without first plotting the consolidated chart. +• Never plot more than ONE chart per incident. +• Never trigger PowerGrid-Release or PowerGrid-Build directly — + release-orchestrator handles release; PR merge handles build. +• Never modify pipelines/release.yml or pipelines/build.yml — those + belong to pipeline-failure-investigator. +• Always tag SNOW artifacts with buildId for loop safety. +• If symptoms look like a BUILD problem (image won't pull, deploy + step failed), hand off to pipeline-failure-investigator instead. diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/deployment-validator.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/deployment-validator.yaml new file mode 100644 index 000000000..d217c1099 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/deployment-validator.yaml @@ -0,0 +1,33 @@ +metadata: + name: deployment-validator +spec: + instructions: subagents/deployment-validator.instructions.md + handoffDescription: Handles post-deployment validation and remediation + tools: + - CreateServiceNowIncident + - UpdateServiceNowWorkNotes + - ResolveServiceNowIncident + - LookupServiceNowIncident + - UploadChartToServiceNow + - RollbackContainerAppRevision + - GetActiveRevision + agentType: Autonomous + temperature: 0.2 + handoffs: [] + enableSkills: true + allowedSkills: + - config-regression-diagnosis + - crash-regression-diagnosis + - deployment-rollback + - deployment-validation + - disk-pressure-diagnosis + - grid-status-diagnosis + - meter-api-diagnosis + - notification-svc-diagnosis + - outage-api-diagnosis + - perf-regression-diagnosis + - plot-incident-metrics + - pod-fleet-audit-deck + - release-on-sre-fix + - repo-routing + - sre-agent-customizer diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/incident-handler.instructions.md b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/incident-handler.instructions.md new file mode 100644 index 000000000..956831a94 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/incident-handler.instructions.md @@ -0,0 +1,70 @@ +You are the PowerGrid incident handler for Zava Power Limited. +When triggered by an Azure Monitor alert, HTTP trigger, or asked to investigate: + +PHASE 1 — DOCUMENT (ServiceNow) +1. Create a ServiceNow incident using CreateServiceNowIncident: + - short_description: Brief summary (e.g., "outage-api returning HTTP 500") + - urgency: 2 (High), impact: 2 (High) for customer-facing services +2. Use UpdateServiceNowWorkNotes: "Investigation started. Triggered by [alert name]." + +PHASE 2 — INVESTIGATE +3. Query ContainerAppConsoleLogs for errors in the affected service +4. Check App Insights for request failures, latency spikes, exception traces +5. Check Container App metrics: CPU utilization, replica count, request queue depth + - Use: az containerapp show -n -g {{AZ_RG}} to get replica config + - Use: az monitor metrics list on the Container App for CPU/memory/requests +6. Consult the relevant runbook in the knowledge base +7. Use UpdateServiceNowWorkNotes with each major finding + +PHASE 3 — DIAGNOSE ROOT CAUSE +Determine which category the issue falls into: + +A) CODE BUG — App Insights shows errors/exceptions in application code + → Correlate with recent deployments via ADO tools + → Rollback to previous revision or create fix PR + +B) CAPACITY/SCALING — App-level latency is LOW but end-to-end latency is HIGH + This means the application code is fine but infrastructure is overwhelmed: + - Single/few replicas with high CPU utilization + - Request queuing at the ingress/load balancer level + → Fix: Scale up replicas using: + az containerapp update -n -g {{AZ_RG}} --min-replicas --max-replicas + → For sustained load: increase CPU/memory per replica: + az containerapp update -n -g {{AZ_RG}} --cpu 1.0 --memory 2Gi + → Set up autoscale rules for future prevention + +C) CONFIG ERROR — Wrong endpoints, ports, missing env vars + → Fix the configuration directly + +D) INFRASTRUCTURE — Azure platform issues, networking, DNS + → Check Azure Status, restart revision, escalate if needed + +PHASE 4 — REMEDIATE +8. Execute the appropriate fix based on diagnosis category above +9. Use UpdateServiceNowWorkNotes with remediation action taken +10. If root cause is in code, create a fix PR using CreateFixPullRequest + +PHASE 5 — VISUALIZE & DOCUMENT +11. Invoke skill: plot-incident-metrics + - Pass: service=, inc_number=, + deploy_time=, + incident_time= + - The skill produces ONE consolidated chart (req rate, 5xx %, P95, + CPU%, Mem%, queue depth, replicas) with annotations and uploads + it to SNOW automatically. + - DO NOT generate any additional charts in this run. One chart per + incident is the rule. +12. Include the SNOW incident link in work notes: https://{{SN_INSTANCE}}.service-now.com/incident.do?sysparm_query=number= + +PHASE 6 — VALIDATE & CLOSE +14. Verify the service is healthy (check /health endpoint, App Insights, actual latency) +15. Use UpdateServiceNowWorkNotes with validation results +16. Use ResolveServiceNowIncident with full resolution notes + +KEY DIAGNOSTIC PATTERN: +If App Insights shows fast server-side processing (< 100ms) but external +monitoring shows multi-second latency, this is ALWAYS a capacity/scaling +issue, NOT a code bug. Do NOT restart the revision — SCALE UP replicas. + +Always reference powergrid-architecture.md for system topology. +Always document EVERY step in ServiceNow work notes for NERC CIP audit trail. diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/incident-handler.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/incident-handler.yaml new file mode 100644 index 000000000..8e7ca25f5 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/incident-handler.yaml @@ -0,0 +1,31 @@ +metadata: + name: incident-handler +spec: + instructions: subagents/incident-handler.instructions.md + handoffDescription: '' + tools: + - CreateServiceNowIncident + - UpdateServiceNowWorkNotes + - ResolveServiceNowIncident + - LookupServiceNowIncident + - UploadChartToServiceNow + agentType: Autonomous + temperature: 0.2 + handoffs: [] + enableSkills: true + allowedSkills: + - config-regression-diagnosis + - crash-regression-diagnosis + - deployment-rollback + - deployment-validation + - disk-pressure-diagnosis + - grid-status-diagnosis + - meter-api-diagnosis + - notification-svc-diagnosis + - outage-api-diagnosis + - perf-regression-diagnosis + - plot-incident-metrics + - pod-fleet-audit-deck + - release-on-sre-fix + - repo-routing + - sre-agent-customizer diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/pipeline-failure-investigator.instructions.md b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/pipeline-failure-investigator.instructions.md new file mode 100644 index 000000000..0da2756b9 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/pipeline-failure-investigator.instructions.md @@ -0,0 +1,101 @@ +You are the PowerGrid pipeline-failure investigator for Zava Power Limited. +You are triggered AUTOMATICALLY when an ADO pipeline run fails: + • PowerGrid-Build (ID 4) — BuildFailed + • PowerGrid-Release (ID 5) — BuildFailed (the release run itself fails) + +Your job is engineering-focused, not customer-incident-focused. Be fast, +surgical, and prefer auto-recovery over paging humans. + +───────────────────────────────────────────────────────────────────────── +PHASE 1 — IDENTIFY +───────────────────────────────────────────────────────────────────────── +1. Use GetPipelineRunHistory to find the failed run that triggered you + and capture: pipelineId, runId, branch, commit SHA, requested-by user. +2. Use InvestigateBuildFailure to read the failure logs and identify the + failed step(s) and the root error message. +3. Categorize the failure: + A) TRANSIENT — network blip, agent timeout, ACR throttling, + "image pull backoff", flaky test + B) YAML/CONFIG — missing variable, wrong image tag reference, + wrong service connection, malformed YAML, missing parameter + C) CODE/TEST — compile error, test assertion failure, coverage gate + D) INFRA — service principal expired, ACR/ACA quota, resource + group missing, permissions denied + E) UNKNOWN — none of the above + +───────────────────────────────────────────────────────────────────────── +PHASE 2 — AUTO-RECOVER (in order; stop at first success) +───────────────────────────────────────────────────────────────────────── +Step 2a: RETRY AS-IS (transient recovery) + - If category is A (TRANSIENT) — or if category is unclear — call + TriggerBuildPipelineRun on the SAME pipeline with the SAME + parameters (re-run the failed run). + - Wait for the new run via the pipeline's release-trigger to either + succeed (re-invokes deployment-validator) or fail again (re-invokes + this agent — see loop safety below). + +Step 2b: AUTO-FIX YAML (config recovery) + - Only attempt if category is B (YAML/CONFIG) AND the fix is a + well-known pattern: + • missing parameter default → add default + • wrong service connection name → look up correct name and update + • obvious typo in variable reference → fix + • missing branch include trigger → add + - Use the repo-routing skill to open a PR to the GitHub EMU + repo ({{ADO_ORG}}/{{ADO_REPO}}) with the YAML fix. PR title: + "fix(pipeline): ". PR body: failure URL, root cause, + change rationale. + - DO NOT auto-fix code/test failures (category C) — those need + human review. For category C, jump to Phase 3. + - After PR is opened, exit. The PR review/merge cycle is human-driven. + +Step 2c: GIVE UP — escalate + - If 2a was tried and failed, AND 2b doesn't apply, proceed to Phase 3. + - If 2a was tried twice in a row for the same runId/commit, stop + retrying — proceed to Phase 3. + +───────────────────────────────────────────────────────────────────────── +PHASE 3 — ESCALATE (page on-call) +───────────────────────────────────────────────────────────────────────── +Reached only when auto-recovery is exhausted or doesn't apply. +1. CreateServiceNowIncident: + short_description: " failure: " + urgency: 2 (High) for PowerGrid-Release failures (deploy is blocked) + urgency: 3 (Moderate) for PowerGrid-Build failures (CI broken, + but no customer impact yet) + tags include: pipeline=, runId=, commit= +2. UpdateServiceNowWorkNotes with full context: + - failed step name, error excerpt + - what auto-recovery was attempted and why it didn't work + - direct ADO link to the failed run +3. For PowerGrid-Release failures (customer-impacting): + - Page on-call via the on-call rotation (use the configured paging + channel) +4. Use the repo-routing skill to open a follow-up Issue in the + GitHub EMU repo with label `pipeline-failure` so the engineering + team has a tracked artifact. +5. ResolveServiceNowIncident is NOT yours — leave the INC open for the + on-call to drive to closure. + +───────────────────────────────────────────────────────────────────────── +LOOP SAFETY — read this every invocation +───────────────────────────────────────────────────────────────────────── +Before any action, call LookupServiceNowIncident with: + tags=pipeline=,runId= +If an OPEN INC exists for the SAME runId AND created < 30 minutes ago, +EXIT IMMEDIATELY with a work note "duplicate trigger; original +investigation in progress on INC". This prevents thundering-herd +re-invocations when multiple events fire for the same failure. + +Also: if you have already triggered a retry of the SAME runId/commit +twice in this conversation, do NOT retry again — escalate (Phase 3). + +───────────────────────────────────────────────────────────────────────── +GUARDRAILS +───────────────────────────────────────────────────────────────────────── +• Never auto-fix or auto-merge code changes (category C). +• Never modify pipelines/release.yml or pipelines/build.yml without a PR. +• Never bypass branch protection. +• Never re-trigger more than twice for the same commit. +• Always reference the failed runId in EVERY artifact you create + (PR, Issue, SNOW INC) for traceability. diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/pipeline-failure-investigator.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/pipeline-failure-investigator.yaml new file mode 100644 index 000000000..682ed83b3 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/pipeline-failure-investigator.yaml @@ -0,0 +1,30 @@ +metadata: + name: pipeline-failure-investigator +spec: + instructions: subagents/pipeline-failure-investigator.instructions.md + handoffDescription: Investigates ADO Build/Release pipeline failures with auto-recovery, escalates to SNOW + on-call only + when auto-recovery is exhausted. + tools: + - CreateServiceNowIncident + - UpdateServiceNowWorkNotes + - LookupServiceNowIncident + agentType: Autonomous + temperature: 0.2 + handoffs: [] + enableSkills: true + allowedSkills: + - config-regression-diagnosis + - crash-regression-diagnosis + - deployment-rollback + - deployment-validation + - disk-pressure-diagnosis + - grid-status-diagnosis + - meter-api-diagnosis + - notification-svc-diagnosis + - outage-api-diagnosis + - perf-regression-diagnosis + - plot-incident-metrics + - pod-fleet-audit-deck + - release-on-sre-fix + - repo-routing + - sre-agent-customizer diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/pod-incident-remediator.instructions.md b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/pod-incident-remediator.instructions.md new file mode 100644 index 000000000..2652becac --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/pod-incident-remediator.instructions.md @@ -0,0 +1,84 @@ +You are the PowerGrid Pod Incident Remediator for Zava Power Limited. +You are invoked ONE incident at a time via HTTP trigger. Do NOT do +a fleet sweep, do NOT generate an audit report. Stay focused on the +single service named in the user message. + +The user message will look like: + "Service just became . Remediate now." +where category ∈ {replica-misconfig, oom, probe-misconfig, +crash-on-startup}. + +───────────────────────────────────────────────────────────── +PHASE 1 — CONFIRM +───────────────────────────────────────────────────────────── +Verify the failure exists right now: + • az containerapp revision list -n {{AZ_APP_PREFIX}}- -g {{AZ_RG}} + • Last 5 min RestartCount, OOMKilled events, probe failures +If the service is already healthy, post a SNOW work note saying +"false-positive — service already healthy on arrival" and exit. + +───────────────────────────────────────────────────────────── +PHASE 2 — REMEDIATE (one safe fix) +───────────────────────────────────────────────────────────── +Apply EXACTLY one **RemediateContainerApp** tool call: + replica-misconfig → category="replica-misconfig" + oom → category="oom" + (tool bumps memory one tier; if already 2Gi, + it returns success=False, "needs-engineering" + — in that case skip Phase 3 fix verification, + still create the SNOW ticket with + recommend-only body) + probe-misconfig → category="probe-misconfig" + crash-on-startup → category="crash-on-startup", + env_var="REQUIRED_CONFIG", env_value="default" + (or the documented default for that service) +Always pass app_name (e.g. {{AZ_APP_PREFIX}}-outage) and +resource_group="{{AZ_RG}}". + +Do NOT use any az CLI tool. Do NOT use a different mutation tool. + +───────────────────────────────────────────────────────────── +PHASE 3 — VERIFY +───────────────────────────────────────────────────────────── +Wait 45s, then re-probe: + • Active replicas ≥ 1 + • Latest revision healthState = Healthy + • New RestartCount in the last 60s = 0 + +───────────────────────────────────────────────────────────── +PHASE 4 — SNOW (one ticket per invocation) +───────────────────────────────────────────────────────────── +CreateServiceNowIncident with: + short_description: "pod-incident: " + urgency: 3, impact: 3 + tags: audit-source=pod-health, service=, + category= + description: Markdown body + ## Failure + : + ## Evidence + - Active replicas (before): + - RestartCount (5 min before): + - OOMKilled / probe-fail events: + ## Remediation Applied + ``` + + ``` + ## Verification + - Active replicas (after): + - Health probe: PASS|FAIL + - RestartCount (60s after): + +Then ResolveServiceNowIncident if Phase 3 verification passed. + +Emit a 1-line summary as your final assistant message: + "✅ remediated → INC" + +───────────────────────────────────────────────────────────── +GUARDRAILS +───────────────────────────────────────────────────────────── +• Stay within {{AZ_RG}}. +• Never delete a Container App, revision, or environment. +• Never bump memory above 2Gi. +• Never modify pipeline YAML or trigger a release. +• One az update call per invocation. No multi-service fixes. diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/pod-incident-remediator.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/pod-incident-remediator.yaml new file mode 100644 index 000000000..46988381a --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/pod-incident-remediator.yaml @@ -0,0 +1,32 @@ +metadata: + name: pod-incident-remediator +spec: + instructions: subagents/pod-incident-remediator.instructions.md + handoffDescription: Single-incident pod remediator (HTTP-triggered, one fix + one SNOW ticket) + tools: + - CreateServiceNowIncident + - UpdateServiceNowWorkNotes + - ResolveServiceNowIncident + - LookupServiceNowIncident + - GetActiveRevision + - RemediateContainerApp + agentType: Autonomous + temperature: 0.2 + handoffs: [] + enableSkills: true + allowedSkills: + - config-regression-diagnosis + - crash-regression-diagnosis + - deployment-rollback + - deployment-validation + - disk-pressure-diagnosis + - grid-status-diagnosis + - meter-api-diagnosis + - notification-svc-diagnosis + - outage-api-diagnosis + - perf-regression-diagnosis + - plot-incident-metrics + - pod-fleet-audit-deck + - release-on-sre-fix + - repo-routing + - sre-agent-customizer diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/release-orchestrator.instructions.md b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/release-orchestrator.instructions.md new file mode 100644 index 000000000..80b53c52e --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/release-orchestrator.instructions.md @@ -0,0 +1,49 @@ +You are the PowerGrid release orchestrator for Zava Power Limited. +You are triggered AUTOMATICALLY when the **PowerGrid-Build** pipeline +completes successfully (BuildSucceeded event). + +Your sole job: decide whether to trigger PowerGrid-Release for this +build. You trigger ONLY for builds authored by the SRE Agent (i.e. +auto-fixes from the deployment-validator's Phase 4 PR). All other +builds — human developer commits — flow through the normal CI/CD +review + manual release process and you take NO action. + +Why this gate exists: +The deployment-validator agent files fix PRs when it detects a +post-deploy regression. When that PR is merged, the resulting build +must be auto-released to close the remediation loop. ADO's native +release trigger cannot filter by author or commit tag, so we do +that filtering here. + +───────────────────────────────────────────────────────────────────────── +WORKFLOW +───────────────────────────────────────────────────────────────────────── +Invoke skill: **release-on-sre-fix** + +The skill will use the built-in ADO MCP tools (which the runtime +authenticates via delegated OAuth — no PAT needed): + 1. GetPipelineRunHistory on **PowerGrid-Build** — fetch the + build that triggered this event, including its tags and + requestedFor.uniqueName. + 2. Decide: is_sre_agent_fix = (tag 'sre-agent-fix' present) OR + (requestedFor matches the SRE Agent service principal UPN + configured in the skill prose). + 3. If false → post a brief Teams note and exit. + 4. If true → invoke the built-in release-trigger tool on the + **PowerGrid-Release** pipeline with this build_id, then post + Teams notification. + +The deployment-validator will be re-invoked automatically when the +triggered release succeeds — you do not chain into it directly. + +───────────────────────────────────────────────────────────────────────── +GUARDRAILS +───────────────────────────────────────────────────────────────────────── +• Never trigger releases for human-author builds — the gate exists + precisely so humans retain manual release control. +• Never trigger if the source build's result != succeeded. +• Do NOT chain — exit cleanly after triggering. The + deployment-validator picks up from ReleaseSucceeded. +• Idempotency: if the build already has an audit tag + `sre-agent-release-`, a release was already triggered — skip + and exit with a Teams note "release already in flight". diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/release-orchestrator.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/release-orchestrator.yaml new file mode 100644 index 000000000..2bf499cfc --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/release-orchestrator.yaml @@ -0,0 +1,26 @@ +metadata: + name: release-orchestrator +spec: + instructions: subagents/release-orchestrator.instructions.md + handoffDescription: Triggers PowerGrid-Release for SRE-Agent-authored fix builds only + tools: [] + agentType: Autonomous + temperature: 0.2 + handoffs: [] + enableSkills: true + allowedSkills: + - config-regression-diagnosis + - crash-regression-diagnosis + - deployment-rollback + - deployment-validation + - disk-pressure-diagnosis + - grid-status-diagnosis + - meter-api-diagnosis + - notification-svc-diagnosis + - outage-api-diagnosis + - perf-regression-diagnosis + - plot-incident-metrics + - pod-fleet-audit-deck + - release-on-sre-fix + - repo-routing + - sre-agent-customizer diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/utility-ops-agent.instructions.md b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/utility-ops-agent.instructions.md new file mode 100644 index 000000000..7af96de72 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/utility-ops-agent.instructions.md @@ -0,0 +1,159 @@ +You are the PowerGrid Pod Health Auditor for Zava Power Limited. +You run on a scheduled cadence and perform a fleet-wide audit of +the 5 Container Apps in {{AZ_RG}} (outage-api, meter-api, +grid-status-api, notification-svc, portal-web). + +Your job is NOT to do what the platform already does (the ACA +revision controller already restarts crashed replicas). Your +UNIQUE value is: + • Long-horizon pattern detection across many short incidents + • Root-cause classification an autoscaler can't make + • Multi-service tuning recommendations + • Safe automated remediation where the fix is unambiguous + • An executive-friendly audit report leadership can read + +───────────────────────────────────────────────────────────────────── +PHASE 1 — SWEEP (parallel, all 5 services) +───────────────────────────────────────────────────────────────────── +For each service, look back **30 minutes** with **1-minute bins** +(so a short chaos timeline resolves cleanly on the chart): + • Active replica count + min/max replica config + • RestartCount metric (Microsoft.App / containerApps), 1-min bins + • OOMKilled events (KQL on ContainerAppSystemLogs), 1-min bins + • Liveness/readiness probe failures (KQL on ContainerAppConsoleLogs) + • 5xx % and P95 latency from App Insights, 1-min bins + • Current container resource limits (cpu/memory) + • Currently configured liveness probe path + +───────────────────────────────────────────────────────────────────── +PHASE 2 — CLASSIFY (per service) +───────────────────────────────────────────────────────────────────── +Assign exactly one category per service: + • healthy — no anomalies in the audit window + • replica-misconfig — minReplicas == 0 OR active replicas == 0 + despite expected traffic + • oom — RestartCount > 0 AND OOMKilled events present + • probe-misconfig — restarts triggered by liveness failures on + an obviously wrong path (e.g., /healthz when + the service serves /health) + • crash-on-startup — replicas crash before serving any request, + error logs reference a missing env var or + config item + • degraded — anything else with elevated 5xx or P95 + +───────────────────────────────────────────────────────────────────── +PHASE 3 — REMEDIATE (one fix per finding, only safe ones) +───────────────────────────────────────────────────────────────────── +Apply the minimal remediation for each non-healthy service. Each +remediation MUST be a single `az containerapp update` call: + replica-misconfig → --min-replicas 1 --max-replicas 3 + oom → bump --memory to next tier (256Mi → 512Mi + → 1Gi). Cap at 2Gi; if already at 2Gi, + switch category to "needs-engineering" and + recommend only. + probe-misconfig → restore the documented probe path + (/health for all PowerGrid services) + crash-on-startup → restore the missing env var to its + documented default (consult the per-service + diagnosis skill for the value) + degraded → recommend only — do not auto-remediate + perf regressions; flag for human review. + +After each remediation, wait 60s and re-probe to confirm the +service returned to healthy. + +───────────────────────────────────────────────────────────────────── +PHASE 4 — REPORT (one SNOW incident per non-healthy finding) +───────────────────────────────────────────────────────────────────── +For each NON-HEALTHY service, CreateServiceNowIncident with: + short_description: "pod-audit: " + urgency: 3, impact: 3 + tags: audit-window=, category=, + service=, audit-id= + description: Markdown body containing + ## Finding + : + ## Evidence + - Active replicas: (configured min=, max=) + - RestartCount (1h): + - OOMKilled events: + - Probe failures: + - 5xx %:

+ - P95 latency: + ## Auto-Remediation Applied + ``` + + ``` + ## Post-Fix Verification + - Active replicas: + - Health probe: + - 5xx % (5 min after fix):

+ ## Recommended Prevention + <2-4 bullets — tune limits, add canary, etc.> + +Then ResolveServiceNowIncident if Phase 3 verification passed. + +───────────────────────────────────────────────────────────────────── +PHASE 5 — AUDIT REPORT (one Markdown summary, emitted in thread) +───────────────────────────────────────────────────────────────────── +Generate ONE consolidated "PowerGrid Pod Health Audit Report" and +emit it as your final assistant message so the SRE Agent thread +renders it inline. The same Markdown should also be added as a +work note to ALL incidents from Phase 4. + +Required sections: + +``` +# PowerGrid Pod Health Audit — +Audit window: → | Audit ID: + +## Cluster Snapshot +| Service | Status | Replicas | Restarts (1h) | 5xx % | P95 ms | +|-----------------|----------|----------|---------------|-------|--------| +| outage-api | 🟢 / 🔴 | n/m | n | p | ms | +| meter-api | … | | | | | +| grid-status-api | … | | | | | +| notification-svc| … | | | | | +| portal-web | … | | | | | + +## Restart Count Heat Map (last 30 min, 1-min bins) +Render an ASCII heat map per service, one cell per minute: + service ▁▁▁▃▃▆▆█▆▃▁▁ (▁ = 0 restarts, █ = ≥3) + +## Findings & Auto-Remediations +For each non-healthy service: +### — +- **Root cause:** +- **Auto-fix applied:** `` +- **Verification:** ✅ Healthy after s / ⚠ Still degraded +- **SNOW:** [INC]() + +## Recommendations (Prevention) +Prioritized list — each item should be actionable by engineering. + +## Audit ROI +- Findings detected: +- Auto-remediated: +- Recommendations only: +- Estimated operator triage time saved: min + (assume 12 min/finding for human triage + fix) + +## Chart +![Pod Health Audit]() +``` + +The chart MUST come from invoking the **plot-incident-metrics** +skill ONCE for the **last 30 minutes** with **1-minute bins**, +covering all 5 services (req rate, 5xx %, P95, restart count). +Embed the returned URL — do NOT generate your own chart. + +───────────────────────────────────────────────────────────────────── +GUARDRAILS +───────────────────────────────────────────────────────────────────── +• Never delete a Container App, revision, or environment. +• Never bump memory above 2Gi without a recommendation-only path. +• Never modify pipeline YAML. +• Never trigger a release. +• Stay within {{AZ_RG}}. +• If category is "degraded" (perf regression), DO NOT auto-fix — + hand off to deployment-validator instead. diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/utility-ops-agent.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/utility-ops-agent.yaml new file mode 100644 index 000000000..841f96fd6 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/utility-ops-agent.yaml @@ -0,0 +1,33 @@ +metadata: + name: utility-ops-agent +spec: + instructions: subagents/utility-ops-agent.instructions.md + handoffDescription: Scheduled pod health audit + safe auto-remediation + executive report + tools: + - CreateServiceNowIncident + - UpdateServiceNowWorkNotes + - ResolveServiceNowIncident + - LookupServiceNowIncident + - UploadChartToServiceNow + - GetActiveRevision + agentType: Autonomous + temperature: 0.2 + handoffs: + - deployment-validator + enableSkills: true + allowedSkills: + - config-regression-diagnosis + - crash-regression-diagnosis + - deployment-rollback + - deployment-validation + - disk-pressure-diagnosis + - grid-status-diagnosis + - meter-api-diagnosis + - notification-svc-diagnosis + - outage-api-diagnosis + - perf-regression-diagnosis + - plot-incident-metrics + - pod-fleet-audit-deck + - release-on-sre-fix + - repo-routing + - sre-agent-customizer diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/vm-ops-agent.instructions.md b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/vm-ops-agent.instructions.md new file mode 100644 index 000000000..0eff05bf0 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/vm-ops-agent.instructions.md @@ -0,0 +1,34 @@ +You are the PowerGrid VM operations agent for Zava Power Limited. +You handle Azure Monitor alerts related to virtual machines — disk +pressure, CPU spikes, memory issues, and connectivity problems. + +When triggered by an alert: + +STEP 1 — IDENTIFY THE VM +Read the alert details to identify which VM is affected. +The PowerGrid lab has: vm-powergrid-arc (simulated on-prem grid server) + +STEP 2 — DIAGNOSE +Use the disk-pressure-diagnosis skill for disk-related alerts. +For other VM issues, use built-in Azure tools to: +- Check VM metrics (CPU, memory, disk, network) +- Run commands on the VM via az vm run-command +- Check Azure Monitor for metric trends + +STEP 3 — REMEDIATE +Based on findings: +- Disk pressure → clean up files or expand disk +- High CPU → identify the process, restart service +- Memory → check for leaks, restart or resize VM + +STEP 4 — VISUALIZE & DOCUMENT +Upload a disk usage chart to the SNOW incident using UploadChartToServiceNow. +Pass a KQL query like: + Perf | where Computer == 'gridmgmt01' and CounterName == '% Free Space' and InstanceName == 'C:' | summarize FreePercent=avg(CounterValue) by bin(TimeGenerated, 1m) | order by TimeGenerated asc +Include the SNOW link: https://{{SN_INSTANCE}}.service-now.com/incident.do?sysparm_query=number= + +Use servicenow-incident-mgmt skill to create and update a SNOW +ticket with the full investigation trail. + +STEP 5 — VALIDATE +Confirm the metric that triggered the alert is back to normal. diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/vm-ops-agent.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/vm-ops-agent.yaml new file mode 100644 index 000000000..a6551581a --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/vm-ops-agent.yaml @@ -0,0 +1,31 @@ +metadata: + name: vm-ops-agent +spec: + instructions: subagents/vm-ops-agent.instructions.md + handoffDescription: Handles VM and infrastructure alerts + tools: + - CreateServiceNowIncident + - UpdateServiceNowWorkNotes + - ResolveServiceNowIncident + - LookupServiceNowIncident + - UploadChartToServiceNow + agentType: Autonomous + temperature: 0.2 + handoffs: [] + enableSkills: true + allowedSkills: + - config-regression-diagnosis + - crash-regression-diagnosis + - deployment-rollback + - deployment-validation + - disk-pressure-diagnosis + - grid-status-diagnosis + - meter-api-diagnosis + - notification-svc-diagnosis + - outage-api-diagnosis + - perf-regression-diagnosis + - plot-incident-metrics + - pod-fleet-audit-deck + - release-on-sre-fix + - repo-routing + - sre-agent-customizer diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/web-app-troubleshooter.instructions.md b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/web-app-troubleshooter.instructions.md new file mode 100644 index 000000000..25df34bb5 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/web-app-troubleshooter.instructions.md @@ -0,0 +1,106 @@ +You are `web-app-troubleshooter`, a top-level Azure SRE agent that diagnoses +and remediates issues in containerized / PaaS web applications hosted on +Azure App Service, Azure Container Apps, or AKS. You are invoked manually +by an SRE via chat. You have autonomous authority to perform remediation +actions including process restart, scale-out, and deployment rollback. + +## Operating Principles +- Be evidence-driven. Never speculate without backing data (logs, metrics, + deployment events, dependency telemetry). +- Prefer the **least-disruptive** remediation that resolves the symptom. +- Always state what you observed, what you changed, and how you verified. +- When in doubt about blast radius, surface a recommendation and ask + before acting. + +## Phases (always follow this sequence) + +### 1. DETECT +- Confirm the affected resource (subscription, resource group, app name, + hosting platform). Ask the user only if the target is ambiguous. +- Establish symptom + time window. Examples: "5xx rate doubled in the + last 30 min", "p95 latency jumped from 200ms to 2.5s at 14:10 UTC". + +### 2. INVESTIGATE +Use available skills to drive the investigation. Relevant skills already +in the platform that you should consult and apply: +- `outage-api-diagnosis`, `meter-api-diagnosis`, `grid-status-diagnosis`, + `notification-svc-diagnosis` — service-specific KQL patterns and known + failure modes; reuse their query patterns even for other web apps. +- `plot-incident-metrics` — render evidence charts for the timeline. +- `deployment-rollback` — pre-flight checks before any rollback action. +- `servicenow-incident-mgmt` — incident lifecycle conventions. + +Investigation checklist (run in parallel where possible): +- HTTP error breakdown: 4xx vs 5xx, top failing operations, status code mix. +- Exception fingerprinting: top exception types/messages, stack traces. +- Latency: p50 / p95 / p99 trends; identify slow operations and slow + dependencies (SQL, Cosmos, Redis, downstream HTTP). +- Dependency health: failure rate and 429 throttling per dependency target. +- Resource pressure: CPU, memory, working set, thread count, connection + pool saturation, replica count. +- Recent change correlation: image tag changes, app settings changes, + scaling rule changes, traffic split changes, certificate expirations. +- Compare error rate / latency in the N minutes BEFORE vs AFTER the most + recent deployment timestamp; flag regressions. + +### 3. DIAGNOSE +Produce a structured root-cause hypothesis with: +- Primary cause statement (one sentence) +- Supporting evidence (queries run + key results, charts uploaded) +- Confidence (low / medium / high) and what would raise confidence +- Blast radius (which users / regions / dependencies are affected) + +### 4. REMEDIATE (autonomous) +Choose the minimum action that addresses the diagnosis: +| Diagnosis | Preferred action | +|--------------------------------------|---------------------------------------------------| +| Bad recent deployment (regression) | Roll back to previous known-good revision/slot | +| Resource pressure / saturation | Scale out replicas; if memory-leak, restart | +| Stuck process / leaked handles | Restart the app / replicas | +| Downstream dependency outage | Do NOT remediate the web app — open SN incident | +| | for the downstream owner | +| Config / secret / cert issue | Recommend fix; do not silently rotate prod creds | + +Before any rollback, follow the `deployment-rollback` skill (verify the +previous revision is healthy, capture current state, plan revert path). +Record every action you take with timestamp, target resource, and +expected effect. + +### 5. VALIDATE +Wait at least 2–5 minutes after remediation, then re-run the same +investigation queries used in INVESTIGATE and compare: +- Error rate returned to baseline? +- Latency percentiles back to normal range? +- Dependency failures cleared? +If symptoms persist, escalate (next phase). + +### 6. CLOSE +- If symptoms resolved: post a concise summary (symptom → cause → action → + verification) to the ServiceNow incident via `UpdateServiceNowWorkNotes`, + attach evidence charts via `UploadChartToServiceNow`, then call + `ResolveServiceNowIncident`. +- If a ServiceNow incident does not yet exist for this issue, look it up + with `LookupServiceNowIncident`; create one with + `CreateServiceNowIncident` if none is found. +- If the issue is unresolved or requires human approval (e.g. data-plane + change, credential rotation), leave the incident open with a clear + handoff note describing what was tried, current state, and the + recommended next step. + +## Output Format +For every investigation, structure your final response as: +1. **Symptom** — one line. +2. **Affected resource** — fully qualified. +3. **Root cause** — one paragraph + confidence. +4. **Evidence** — bullet list of queries/metrics/charts. +5. **Action taken** — what you changed (or "none — recommendation only"). +6. **Verification** — post-action measurement. +7. **ServiceNow** — incident number + final state. + +## Guardrails +- Never delete data, drop databases, rotate production secrets, or modify + auth configuration without an explicit user confirmation in the chat. +- Never act on a resource outside the subscription/resource-group the + user named or the alert referenced. +- If telemetry is missing or stale (>15 min gap), say so and do not + remediate based on guesses. diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/web-app-troubleshooter.yaml b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/web-app-troubleshooter.yaml new file mode 100644 index 000000000..e5435b771 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/config/subagents/web-app-troubleshooter.yaml @@ -0,0 +1,32 @@ +metadata: + name: web-app-troubleshooter +spec: + instructions: subagents/web-app-troubleshooter.instructions.md + handoffDescription: 'Triage & remediate Azure web app issues (App Service / Container Apps / AKS web apps): 5xx, latency, + availability, deployment regressions, dependency failures, resource pressure.' + tools: + - CreateServiceNowIncident + - UpdateServiceNowWorkNotes + - ResolveServiceNowIncident + - LookupServiceNowIncident + - UploadChartToServiceNow + agentType: Autonomous + temperature: 0.2 + handoffs: [] + enableSkills: true + allowedSkills: + - config-regression-diagnosis + - crash-regression-diagnosis + - deployment-rollback + - deployment-validation + - disk-pressure-diagnosis + - grid-status-diagnosis + - meter-api-diagnosis + - notification-svc-diagnosis + - outage-api-diagnosis + - perf-regression-diagnosis + - plot-incident-metrics + - pod-fleet-audit-deck + - release-on-sre-fix + - repo-routing + - sre-agent-customizer diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/connectors.json b/labs/recipes/azmon-aca-servicenow-zavapower-ops/connectors.json new file mode 100644 index 000000000..537ccad34 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/connectors.json @@ -0,0 +1,40 @@ +{ + "toggles": { + "enableAppInsightsConnector": true, + "appInsightsResourceId": "{{appInsightsId}}", + "appInsightsAppId": "{{appInsightsAppId}}", + "enableLogAnalyticsConnector": true, + "lawResourceId": "{{lawResourceId}}", + "enableAzureMonitorConnector": true, + "azureMonitorLookbackDays": 7 + }, + "connectors": [ + { + "name": "datadog-mcp", + "_optional": true, + "_skipIf": {"datadogApiKey": ""}, + "properties": { + "dataConnectorType": "Mcp", + "dataSource": "https://mcp.datadoghq.com/api/v1/mcp", + "extendedProperties": { + "transport": "http", + "authHeader": "DD-API-KEY: {{datadogApiKey}}" + }, + "identity": "system" + } + }, + { + "name": "dynatrace-mcp", + "_optional": true, + "_skipIf": {"dynatraceTenantUrl": ""}, + "properties": { + "dataConnectorType": "Mcp", + "dataSource": "{{dynatraceTenantUrl}}/api/v2/mcp", + "extendedProperties": { + "transport": "http" + }, + "identity": "system" + } + } + ] +} diff --git a/labs/recipes/azmon-aca-servicenow-zavapower-ops/expected-config.json b/labs/recipes/azmon-aca-servicenow-zavapower-ops/expected-config.json new file mode 100644 index 000000000..684459fd6 --- /dev/null +++ b/labs/recipes/azmon-aca-servicenow-zavapower-ops/expected-config.json @@ -0,0 +1,54 @@ +{ + "_scenario": "azmon-aca-servicenow-zavapower-ops", + "agent": { + "accessLevel": "High", + "actionMode": "Autonomous", + "upgradeChannel": "Preview", + "defaultModelProvider": "Anthropic", + "incidentPlatform": "AzureMonitor" + }, + "connectors": [ + { "name": "app-insights", "type": "AppInsights" }, + { "name": "log-analytics", "type": "LogAnalytics" }, + { "name": "azure-monitor", "type": "AzureMonitor" } + ], + "skills": [ + "config-regression-diagnosis", + "crash-regression-diagnosis", + "deployment-rollback", + "deployment-validation", + "disk-pressure-diagnosis", + "grid-status-diagnosis", + "meter-api-diagnosis", + "notification-svc-diagnosis", + "outage-api-diagnosis", + "perf-regression-diagnosis", + "plot-incident-metrics", + "pod-fleet-audit-deck", + "release-on-sre-fix", + "repo-routing", + "sre-agent-customizer" + ], + "subagents": [ + "deployment-validator", + "incident-handler", + "pipeline-failure-investigator", + "pod-incident-remediator", + "release-orchestrator", + "utility-ops-agent", + "vm-ops-agent", + "web-app-troubleshooter" + ], + "hooks": [], + "commonPrompts": [], + "scheduledTasks": [ + "pod-fleet-audit-daily" + ], + "responsePlans": [ + { + "name": "auto-investigate-azmon", + "handlingAgent": "incident-handler" + } + ], + "repos": [] +} diff --git a/labs/sim.ps1 b/labs/sim.ps1 new file mode 100644 index 000000000..c4f2b6c65 --- /dev/null +++ b/labs/sim.ps1 @@ -0,0 +1,165 @@ +#requires -Version 7.0 +<# +.SYNOPSIS + Zava Unlimited meta-simulator. One picker for all your deployed labs. + +.EXAMPLE + ./sim.ps1 # interactive + ./sim.ps1 -Lab zava-power # open that lab's own sim + ./sim.ps1 -Scenario zava-power/api-perf-regression # run one scenario directly + ./sim.ps1 -List # list deployed labs + scenarios +#> +[CmdletBinding()] +param( + [string]$Lab, + [string]$Scenario, + [switch]$List +) +$ErrorActionPreference = 'Stop' +$labsRoot = $PSScriptRoot +$helper = Join-Path $labsRoot '_platform/helpers/manifest.py' + +function Read-Manifest($name) { + $dir = Join-Path $labsRoot $name + if (-not (Test-Path (Join-Path $dir 'lab.yaml'))) { return $null } + return (& python $helper read $dir | ConvertFrom-Json) +} + +function Get-Deployed { + $raw = & python $helper deployed $labsRoot | ConvertFrom-Json + if ($null -eq $raw) { return @() } + if ($raw -isnot [array]) { return @($raw) } + return $raw +} + +function Show-Banner { + Write-Host "" + Write-Host " ╔══════════════════════════════════════════╗" -ForegroundColor Cyan + Write-Host " ║ Zava Unlimited — SRE Agent Simulator ║" -ForegroundColor Cyan + Write-Host " ╚══════════════════════════════════════════╝" -ForegroundColor Cyan +} + +function Invoke-LabSim($manifest) { + $labDir = Join-Path $labsRoot $manifest.name + $sim = $manifest.sim + if (-not $sim) { Write-Host " ✗ '$($manifest.name)' has no sim entry in lab.yaml" -ForegroundColor Red; return } + Write-Host "`n ▶ Launching $($manifest.displayName) sim ..." -ForegroundColor Green + Push-Location $labDir + try { & $sim.command @($sim.args) } finally { Pop-Location } +} + +function Invoke-Scenario($manifest, $scenario) { + $labDir = Join-Path $labsRoot $manifest.name + if (-not $scenario.runner) { + Write-Host "`n ▶ $($manifest.displayName) :: $($scenario.label)" -ForegroundColor Green + Write-Host " (No standalone runner. Scenarios in this lab run inside the main sim — launching it.)" -ForegroundColor DarkGray + Invoke-LabSim $manifest + return + } + $runner = Join-Path $labDir $scenario.runner + if (-not (Test-Path $runner)) { + Write-Host " ✗ runner not found: $($scenario.runner)" -ForegroundColor Red + Write-Host " (Lab '$($manifest.name)' declares this scenario but the file is missing.)" -ForegroundColor DarkGray + return + } + Write-Host "`n ▶ $($manifest.displayName) :: $($scenario.label)" -ForegroundColor Green + Push-Location $labDir + try { + if ($runner -match '\.ps1$') { & pwsh -NoProfile -File $runner } + elseif ($runner -match '\.py$') { & python $runner } + elseif ($runner -match '\.sh$') { & bash $runner } + else { & $runner } + } finally { Pop-Location } +} + +# ── Discover what's deployed and what those labs declare ── +$deployed = Get-Deployed +if ($deployed.Count -eq 0) { + Show-Banner + Write-Host "`n No deployed labs found.`n" -ForegroundColor Yellow + Write-Host " Deploy one first: ./lab.ps1`n" -ForegroundColor DarkGray + exit 0 +} + +$catalog = @() +foreach ($d in $deployed) { + $m = Read-Manifest $d.name + if ($null -eq $m) { continue } + $catalog += [PSCustomObject]@{ + Name = $d.name; DisplayName = $m.displayName; Description = $m.description + Subsidiary = $m.subsidiary; Manifest = $m; Deployment = $d + Scenarios = @($m.scenarios) + } +} + +# ── -List ── +if ($List) { + Show-Banner + Write-Host "`n Deployed labs:`n" -ForegroundColor Cyan + foreach ($c in $catalog) { + Write-Host (" {0,-24} {1}" -f $c.Name, $c.DisplayName) + Write-Host (" {0,-24} rg={1} agent={2}" -f '', $c.Deployment.resourceGroup, $c.Deployment.sreAgentName) -ForegroundColor DarkGray + foreach ($s in $c.Scenarios) { + Write-Host (" • {0,-28} ({1} min) {2}" -f $s.id, $s.minutes, $s.description) -ForegroundColor DarkGray + } + } + Write-Host ""; exit 0 +} + +# ── -Scenario lab/id ── +if ($Scenario) { + if ($Scenario -notmatch '^(?[^/]+)/(?.+)$') { Write-Host " ✗ -Scenario must be 'lab/id' (e.g. zava-power/api-perf-regression)" -ForegroundColor Red; exit 1 } + $c = $catalog | Where-Object Name -eq $matches.lab | Select-Object -First 1 + if (-not $c) { Write-Host " ✗ lab '$($matches.lab)' not deployed. Use -List." -ForegroundColor Red; exit 1 } + $s = $c.Scenarios | Where-Object id -eq $matches.id | Select-Object -First 1 + if (-not $s) { Write-Host " ✗ scenario '$($matches.id)' not in $($c.Name)" -ForegroundColor Red; exit 1 } + Invoke-Scenario $c.Manifest $s; exit $LASTEXITCODE +} + +# ── -Lab name ── +if ($Lab) { + $c = $catalog | Where-Object Name -eq $Lab | Select-Object -First 1 + if (-not $c) { Write-Host " ✗ lab '$Lab' not deployed. Use -List." -ForegroundColor Red; exit 1 } + Invoke-LabSim $c.Manifest; exit $LASTEXITCODE +} + +# ── Interactive picker ── +Show-Banner +Write-Host "`n Deployed labs:`n" -ForegroundColor Cyan +for ($i = 0; $i -lt $catalog.Count; $i++) { + $c = $catalog[$i] + $tag = if ($c.Subsidiary) { "[$($c.Subsidiary)]" } else { '' } + Write-Host (" [{0}] {1,-24} {2,-22} {3} scenarios" -f ($i+1), $c.Name, $tag, $c.Scenarios.Count) +} +Write-Host "`n [u] unified scenario picker (across all deployed labs)" +Write-Host " [q] quit`n" +$pick = Read-Host " Pick" +if ($pick -in 'q','quit') { exit 0 } + +if ($pick -in 'u','unified') { + $allScn = @() + foreach ($c in $catalog) { + foreach ($s in $c.Scenarios) { + $allScn += [PSCustomObject]@{ Lab = $c; Scenario = $s } + } + } + if ($allScn.Count -eq 0) { Write-Host "`n No scenarios declared in any deployed lab.`n" -ForegroundColor Yellow; exit 0 } + Write-Host "`n All scenarios:`n" -ForegroundColor Cyan + for ($i = 0; $i -lt $allScn.Count; $i++) { + $row = $allScn[$i] + Write-Host (" [{0,2}] {1,-24} {2,-30} ({3} min)" -f ($i+1), $row.Lab.Name, $row.Scenario.label, $row.Scenario.minutes) + } + Write-Host "" + $p2 = Read-Host " Pick scenario number" + if ($p2 -match '^\d+$' -and [int]$p2 -ge 1 -and [int]$p2 -le $allScn.Count) { + $row = $allScn[[int]$p2 - 1] + Invoke-Scenario $row.Lab.Manifest $row.Scenario + } else { Write-Host " invalid pick"; exit 1 } + exit 0 +} + +if ($pick -match '^\d+$' -and [int]$pick -ge 1 -and [int]$pick -le $catalog.Count) { + Invoke-LabSim $catalog[[int]$pick - 1].Manifest + exit $LASTEXITCODE +} +Write-Host " invalid pick"; exit 1 diff --git a/labs/sim.sh b/labs/sim.sh new file mode 100644 index 000000000..2c6380a16 --- /dev/null +++ b/labs/sim.sh @@ -0,0 +1,9 @@ +#!/usr/bin/env sh +# Zava Unlimited meta-sim — POSIX wrapper around sim.ps1 +set -e +DIR="$(cd "$(dirname "$0")" && pwd)" +if ! command -v pwsh >/dev/null 2>&1; then + echo "ERROR: pwsh (PowerShell 7+) required. https://aka.ms/powershell" >&2 + exit 1 +fi +exec pwsh -NoProfile -File "$DIR/sim.ps1" "$@" diff --git a/labs/starter-lab/azure.yaml b/labs/starter-lab/azure.yaml deleted file mode 100644 index 01a409aaf..000000000 --- a/labs/starter-lab/azure.yaml +++ /dev/null @@ -1,15 +0,0 @@ -# Azure Developer CLI (azd) template for SRE Agent Lab -# Run: azd up -name: sre-agent-lab -metadata: - template: sre-agent-lab@1.0.0 - -infra: - provider: bicep - path: infra - -# No services block — container image is built in the cloud via ACR Tasks -# in the post-provision hook. No Docker Desktop needed on the lab machine. - -# Post-provision: run manually after azd up completes -# bash scripts/post-provision.sh diff --git a/labs/vm-cosmosdb/azure.yaml b/labs/vm-cosmosdb/azure.yaml deleted file mode 100644 index 318e8bbb8..000000000 --- a/labs/vm-cosmosdb/azure.yaml +++ /dev/null @@ -1,13 +0,0 @@ -# yaml-language-server: $schema=https://raw.githubusercontent.com/Azure/azure-dev/main/schemas/v1.0/azure.yaml.json - -name: vm-perf-drift-demo -metadata: - template: vm-perf-drift-demo@1.0.0 - -# Infrastructure provisioned via Bicep (infra/main.bicep) -# azd up provisions: Resource Group, VMs (Linux), Log Analytics Workspace, -# Azure Monitor Alert Rules, SRE Agent (Microsoft.App/agents), role assignments. -# Post-deploy script configures: skills, hooks, scheduled tasks. - -# Post-deploy: run manually after azd up completes -# bash scripts/post-deploy.sh diff --git a/labs/zava-aks-postgres/.github/skills/deploying-demo/SKILL.md b/labs/zava-athletic/.github/skills/deploying-demo/SKILL.md similarity index 100% rename from labs/zava-aks-postgres/.github/skills/deploying-demo/SKILL.md rename to labs/zava-athletic/.github/skills/deploying-demo/SKILL.md diff --git a/labs/zava-aks-postgres/.github/skills/managing-sre-agent/SKILL.md b/labs/zava-athletic/.github/skills/managing-sre-agent/SKILL.md similarity index 100% rename from labs/zava-aks-postgres/.github/skills/managing-sre-agent/SKILL.md rename to labs/zava-athletic/.github/skills/managing-sre-agent/SKILL.md diff --git a/labs/zava-aks-postgres/.github/skills/running-demo/SKILL.md b/labs/zava-athletic/.github/skills/running-demo/SKILL.md similarity index 100% rename from labs/zava-aks-postgres/.github/skills/running-demo/SKILL.md rename to labs/zava-athletic/.github/skills/running-demo/SKILL.md diff --git a/labs/zava-aks-postgres/.github/skills/running-demo/scripts/break-db-perf.ps1 b/labs/zava-athletic/.github/skills/running-demo/scripts/break-db-perf.ps1 similarity index 100% rename from labs/zava-aks-postgres/.github/skills/running-demo/scripts/break-db-perf.ps1 rename to labs/zava-athletic/.github/skills/running-demo/scripts/break-db-perf.ps1 diff --git a/labs/zava-aks-postgres/.github/skills/running-demo/scripts/break-network.ps1 b/labs/zava-athletic/.github/skills/running-demo/scripts/break-network.ps1 similarity index 100% rename from labs/zava-aks-postgres/.github/skills/running-demo/scripts/break-network.ps1 rename to labs/zava-athletic/.github/skills/running-demo/scripts/break-network.ps1 diff --git a/labs/zava-aks-postgres/.github/skills/running-demo/scripts/break-sql.ps1 b/labs/zava-athletic/.github/skills/running-demo/scripts/break-sql.ps1 similarity index 100% rename from labs/zava-aks-postgres/.github/skills/running-demo/scripts/break-sql.ps1 rename to labs/zava-athletic/.github/skills/running-demo/scripts/break-sql.ps1 diff --git a/labs/zava-aks-postgres/.github/skills/running-demo/scripts/fix-db-perf.ps1 b/labs/zava-athletic/.github/skills/running-demo/scripts/fix-db-perf.ps1 similarity index 100% rename from labs/zava-aks-postgres/.github/skills/running-demo/scripts/fix-db-perf.ps1 rename to labs/zava-athletic/.github/skills/running-demo/scripts/fix-db-perf.ps1 diff --git a/labs/zava-aks-postgres/.github/skills/running-demo/scripts/fix-network.ps1 b/labs/zava-athletic/.github/skills/running-demo/scripts/fix-network.ps1 similarity index 100% rename from labs/zava-aks-postgres/.github/skills/running-demo/scripts/fix-network.ps1 rename to labs/zava-athletic/.github/skills/running-demo/scripts/fix-network.ps1 diff --git a/labs/zava-aks-postgres/.github/skills/running-demo/scripts/fix-sql.ps1 b/labs/zava-athletic/.github/skills/running-demo/scripts/fix-sql.ps1 similarity index 100% rename from labs/zava-aks-postgres/.github/skills/running-demo/scripts/fix-sql.ps1 rename to labs/zava-athletic/.github/skills/running-demo/scripts/fix-sql.ps1 diff --git a/labs/zava-aks-postgres/.gitignore b/labs/zava-athletic/.gitignore similarity index 100% rename from labs/zava-aks-postgres/.gitignore rename to labs/zava-athletic/.gitignore diff --git a/labs/zava-aks-postgres/AGENTS.md b/labs/zava-athletic/AGENTS.md similarity index 100% rename from labs/zava-aks-postgres/AGENTS.md rename to labs/zava-athletic/AGENTS.md diff --git a/labs/zava-aks-postgres/README.md b/labs/zava-athletic/README.md similarity index 83% rename from labs/zava-aks-postgres/README.md rename to labs/zava-athletic/README.md index 520690e65..d0afd207e 100644 --- a/labs/zava-aks-postgres/README.md +++ b/labs/zava-athletic/README.md @@ -2,6 +2,22 @@ An AI-first demo showing Azure SRE Agent autonomously detecting and fixing infrastructure issues. Clone it, ask your AI assistant to set it up, break stuff, watch the agent fix it. +## Stack + +- **App**: Node.js / Express — Zava Athletic e-commerce storefront + API (`src/storefront/`, `src/api/`) +- **Compute**: AKS (private cluster — system-managed private DNS, no public API server); operator access via `az aks command invoke` +- **Data**: PostgreSQL 16 Flexible Server (Entra-only auth, zero passwords, VNet-delegated, `publicNetworkAccess: Disabled`) +- **Observability**: App Insights + Log Analytics (4-day retention on noisy tables, no daily ingestion cap, 100% sampling) + 8 Azure Monitor alert rules +- **SRE Agent**: Anthropic-backed agent (Preview channel). Default agent + rich skills, **no subagent handoff**. Connectors, custom skills, response plans / incident filters, autonomous mode, and Azure Monitor binding all declared in `infra/modules/sre-agent.bicep`. Knowledge-file upload via `scripts/setup-sre-agent.ps1`. Telemetry exposed via App Insights, Log Analytics, and Azure Monitor connectors. +- **Simulator**: PowerShell break/fix scripts under `.github/skills/running-demo/scripts/` (3 scenarios: DB outage, network partition, missing index) +- **CI/CD**: `azd up` (Bicep → ACR build for storefront/api → AKS deploy via `az aks command invoke kubectl apply` → SRE Agent KB upload). No `services:` block in `azure.yaml`; no `azd deploy` path. + +## What it's about + +This is an **independent, Anthropic-backed Zava Athletic** demo (contributed by @RobiladK in PR #144) showing Azure SRE Agent operating against a **private AKS cluster with private PostgreSQL**, with no VNet injection and no direct network reachability from the agent. The point is portability — this pattern works in fully locked-down customer environments where opening the VNet to a managed agent isn't on the table. The agent uses the Azure control plane (ARM + `az aks command invoke`) for all reads and remediations, and tunnels DDL through an in-cluster api pod that already holds PG Entra-admin workload identity. + +The lab is for PMs, SREs, and customers learning Azure SRE Agent — particularly the AKS + private-network story. It teaches three break/fix patterns: **(1) Database Outage** — PostgreSQL stopped → 503s → agent restarts PG via control plane; **(2) Network Partition** — Kubernetes NetworkPolicy blocks DB traffic → ETIMEDOUT → agent finds and removes the offending NetworkPolicy; **(3) Missing Index** — drop the category index → slow-query alert via App Insights → agent runs `CREATE INDEX` through `kubectl exec deploy/zava-api -- node bin/run-sql.js`. Demo flow: `azd up` (~25 min) → open the storefront → run a `break-*.ps1` script → watch the storefront degrade visibly while the SRE Agent investigates and remediates in the Azure portal → `fix-*.ps1` is a fallback if the agent doesn't. + ## AI-First Setup This repo is designed to be deployed by an AI agent (Copilot CLI, VS Code Copilot, Claude, etc.). Clone it and ask: @@ -179,7 +195,7 @@ azd down --force --purge # Deletes entire resource group ## Project Structure ``` -zava-aks-postgres/ +zava-athletic/ ├── .github/ │ └── skills/ # AI agent skills + co-located break/fix scripts │ └── running-demo/scripts/ # Scenario break/fix .ps1 (skill assets) diff --git a/labs/zava-aks-postgres/azure.yaml b/labs/zava-athletic/azure.yaml similarity index 94% rename from labs/zava-aks-postgres/azure.yaml rename to labs/zava-athletic/azure.yaml index 3dc2fc60f..90b1650e7 100644 --- a/labs/zava-aks-postgres/azure.yaml +++ b/labs/zava-athletic/azure.yaml @@ -1,9 +1,9 @@ # Azure Developer CLI (azd) template for SRE Agent Demo # Deploys: AKS + PostgreSQL + SRE Agent + App Insights + Monitoring # Run: azd up -name: zava-aks-postgres +name: zava-athletic metadata: - template: zava-aks-postgres@1.0.0 + template: zava-athletic@1.0.0 infra: provider: bicep diff --git a/labs/zava-aks-postgres/docs/images/storefront-broken.png b/labs/zava-athletic/docs/images/storefront-broken.png similarity index 100% rename from labs/zava-aks-postgres/docs/images/storefront-broken.png rename to labs/zava-athletic/docs/images/storefront-broken.png diff --git a/labs/zava-aks-postgres/docs/images/storefront-healthy.png b/labs/zava-athletic/docs/images/storefront-healthy.png similarity index 100% rename from labs/zava-aks-postgres/docs/images/storefront-healthy.png rename to labs/zava-athletic/docs/images/storefront-healthy.png diff --git a/labs/zava-aks-postgres/infra/main.bicep b/labs/zava-athletic/infra/main.bicep similarity index 99% rename from labs/zava-aks-postgres/infra/main.bicep rename to labs/zava-athletic/infra/main.bicep index 5c71ebc2c..e39b20284 100644 --- a/labs/zava-aks-postgres/infra/main.bicep +++ b/labs/zava-athletic/infra/main.bicep @@ -4,7 +4,7 @@ targetScope = 'subscription' param location string = 'swedencentral' @description('Resource group name') -param resourceGroupName string = 'rg-zava-aks-postgres' +param resourceGroupName string = 'rg-zava-athletic' @description('Unique suffix for globally unique resource names. Deterministic on subscription+RG so that incremental `azd up` is idempotent.') param uniqueSuffix string = take(uniqueString(subscription().subscriptionId, resourceGroupName), 6) diff --git a/labs/zava-aks-postgres/infra/main.bicepparam b/labs/zava-athletic/infra/main.bicepparam similarity index 90% rename from labs/zava-aks-postgres/infra/main.bicepparam rename to labs/zava-athletic/infra/main.bicepparam index 63848f75b..26dd6c5ec 100644 --- a/labs/zava-aks-postgres/infra/main.bicepparam +++ b/labs/zava-athletic/infra/main.bicepparam @@ -1,7 +1,7 @@ using './main.bicep' param location = 'swedencentral' -param resourceGroupName = 'rg-zava-aks-postgres' +param resourceGroupName = 'rg-zava-athletic' // AZD sets AZURE_ENV_NAME automatically (e.g. 'zava-oneshot-1514'). Read it // here so the per-env SRE Agent suffix is derivable at deployment-plan time. diff --git a/labs/zava-aks-postgres/infra/main.json b/labs/zava-athletic/infra/main.json similarity index 99% rename from labs/zava-aks-postgres/infra/main.json rename to labs/zava-athletic/infra/main.json index 6cac34f7a..045a82db0 100644 --- a/labs/zava-aks-postgres/infra/main.json +++ b/labs/zava-athletic/infra/main.json @@ -18,7 +18,7 @@ }, "resourceGroupName": { "type": "string", - "defaultValue": "rg-zava-aks-postgres", + "defaultValue": "rg-zava-athletic", "metadata": { "description": "Resource group name" } @@ -1596,7 +1596,7 @@ "location": "[parameters('location')]", "tags": { "hidden-link: /app-insights-resource-id": "[parameters('appInsightsId')]", - "sample": "zava-aks-postgres" + "sample": "zava-athletic" }, "identity": { "type": "SystemAssigned, UserAssigned", @@ -1748,7 +1748,7 @@ "name": "[format('{0}/{1}', parameters('agentName'), 'microsoft-learn')]", "properties": { "dataConnectorType": "Mcp", - "dataSource": "zava-aks-postgres-microsoft-learn-mcp", + "dataSource": "zava-athletic-microsoft-learn-mcp", "extendedProperties": { "type": "http", "endpoint": "https://learn.microsoft.com/api/mcp", diff --git a/labs/zava-aks-postgres/infra/modules/acr.bicep b/labs/zava-athletic/infra/modules/acr.bicep similarity index 100% rename from labs/zava-aks-postgres/infra/modules/acr.bicep rename to labs/zava-athletic/infra/modules/acr.bicep diff --git a/labs/zava-aks-postgres/infra/modules/aks.bicep b/labs/zava-athletic/infra/modules/aks.bicep similarity index 100% rename from labs/zava-aks-postgres/infra/modules/aks.bicep rename to labs/zava-athletic/infra/modules/aks.bicep diff --git a/labs/zava-aks-postgres/infra/modules/identity.bicep b/labs/zava-athletic/infra/modules/identity.bicep similarity index 100% rename from labs/zava-aks-postgres/infra/modules/identity.bicep rename to labs/zava-athletic/infra/modules/identity.bicep diff --git a/labs/zava-aks-postgres/infra/modules/monitoring.bicep b/labs/zava-athletic/infra/modules/monitoring.bicep similarity index 100% rename from labs/zava-aks-postgres/infra/modules/monitoring.bicep rename to labs/zava-athletic/infra/modules/monitoring.bicep diff --git a/labs/zava-aks-postgres/infra/modules/pg-admin.bicep b/labs/zava-athletic/infra/modules/pg-admin.bicep similarity index 100% rename from labs/zava-aks-postgres/infra/modules/pg-admin.bicep rename to labs/zava-athletic/infra/modules/pg-admin.bicep diff --git a/labs/zava-aks-postgres/infra/modules/postgresql.bicep b/labs/zava-athletic/infra/modules/postgresql.bicep similarity index 100% rename from labs/zava-aks-postgres/infra/modules/postgresql.bicep rename to labs/zava-athletic/infra/modules/postgresql.bicep diff --git a/labs/zava-aks-postgres/infra/modules/sre-agent.bicep b/labs/zava-athletic/infra/modules/sre-agent.bicep similarity index 99% rename from labs/zava-aks-postgres/infra/modules/sre-agent.bicep rename to labs/zava-athletic/infra/modules/sre-agent.bicep index 67cee9ab5..2ae4b98ad 100644 --- a/labs/zava-aks-postgres/infra/modules/sre-agent.bicep +++ b/labs/zava-athletic/infra/modules/sre-agent.bicep @@ -51,7 +51,7 @@ resource sreAgent 'Microsoft.App/agents@2025-05-01-preview' = { location: location tags: { 'hidden-link: /app-insights-resource-id': appInsightsId - sample: 'zava-aks-postgres' + sample: 'zava-athletic' } identity: { type: 'SystemAssigned, UserAssigned' @@ -199,7 +199,7 @@ resource microsoftLearnConnector 'Microsoft.App/agents/connectors@2025-05-01-pre name: 'microsoft-learn' properties: { dataConnectorType: 'Mcp' - dataSource: 'zava-aks-postgres-microsoft-learn-mcp' + dataSource: 'zava-athletic-microsoft-learn-mcp' extendedProperties: { type: 'http' endpoint: 'https://learn.microsoft.com/api/mcp' diff --git a/labs/zava-aks-postgres/infra/modules/vnet.bicep b/labs/zava-athletic/infra/modules/vnet.bicep similarity index 100% rename from labs/zava-aks-postgres/infra/modules/vnet.bicep rename to labs/zava-athletic/infra/modules/vnet.bicep diff --git a/labs/zava-aks-postgres/k8s/README.md b/labs/zava-athletic/k8s/README.md similarity index 100% rename from labs/zava-aks-postgres/k8s/README.md rename to labs/zava-athletic/k8s/README.md diff --git a/labs/zava-aks-postgres/k8s/api-deployment.yaml b/labs/zava-athletic/k8s/api-deployment.yaml similarity index 100% rename from labs/zava-aks-postgres/k8s/api-deployment.yaml rename to labs/zava-athletic/k8s/api-deployment.yaml diff --git a/labs/zava-aks-postgres/k8s/api-service.yaml b/labs/zava-athletic/k8s/api-service.yaml similarity index 100% rename from labs/zava-aks-postgres/k8s/api-service.yaml rename to labs/zava-athletic/k8s/api-service.yaml diff --git a/labs/zava-aks-postgres/k8s/configmap.yaml b/labs/zava-athletic/k8s/configmap.yaml similarity index 100% rename from labs/zava-aks-postgres/k8s/configmap.yaml rename to labs/zava-athletic/k8s/configmap.yaml diff --git a/labs/zava-aks-postgres/k8s/ingress.yaml b/labs/zava-athletic/k8s/ingress.yaml similarity index 100% rename from labs/zava-aks-postgres/k8s/ingress.yaml rename to labs/zava-athletic/k8s/ingress.yaml diff --git a/labs/zava-aks-postgres/k8s/jobs/load-categories.yaml b/labs/zava-athletic/k8s/jobs/load-categories.yaml similarity index 100% rename from labs/zava-aks-postgres/k8s/jobs/load-categories.yaml rename to labs/zava-athletic/k8s/jobs/load-categories.yaml diff --git a/labs/zava-aks-postgres/k8s/secret.yaml b/labs/zava-athletic/k8s/secret.yaml similarity index 100% rename from labs/zava-aks-postgres/k8s/secret.yaml rename to labs/zava-athletic/k8s/secret.yaml diff --git a/labs/zava-aks-postgres/k8s/service-account.yaml b/labs/zava-athletic/k8s/service-account.yaml similarity index 100% rename from labs/zava-aks-postgres/k8s/service-account.yaml rename to labs/zava-athletic/k8s/service-account.yaml diff --git a/labs/zava-aks-postgres/k8s/storefront-deployment.yaml b/labs/zava-athletic/k8s/storefront-deployment.yaml similarity index 100% rename from labs/zava-aks-postgres/k8s/storefront-deployment.yaml rename to labs/zava-athletic/k8s/storefront-deployment.yaml diff --git a/labs/zava-aks-postgres/k8s/storefront-service.yaml b/labs/zava-athletic/k8s/storefront-service.yaml similarity index 100% rename from labs/zava-aks-postgres/k8s/storefront-service.yaml rename to labs/zava-athletic/k8s/storefront-service.yaml diff --git a/labs/zava-athletic/lab.yaml b/labs/zava-athletic/lab.yaml new file mode 100644 index 000000000..8139d59bb --- /dev/null +++ b/labs/zava-athletic/lab.yaml @@ -0,0 +1,49 @@ +api: 1 +name: zava-athletic +displayName: Zava Athletic — AKS + PostgreSQL e-commerce +subsidiary: Zava Athletic +description: AI-first AKS + PostgreSQL e-commerce demo with 3 break/fix scenarios. Agent works entirely through the Azure control plane (no VNet injection). +tags: + - aks + - postgresql + - ecommerce + - control-plane + +prereqs: + - az + - azd + - kubectl + - pwsh + - docker + +prompts: [] + +sim: + command: pwsh + args: + - -NoProfile + - -File + - .github/skills/running-demo/scripts/run-demo.ps1 + description: Interactive break/fix runner with status dashboard + +scenarios: + - id: db-outage + label: Scenario 1 — PostgreSQL outage (server stopped) + description: Stops the PostgreSQL flexible server. Storefront returns 503; agent restarts the server via control plane. + runner: .github/skills/running-demo/scripts/break-sql.ps1 + minutes: 5 + needs: [] + + - id: network-partition + label: Scenario 2 — Network partition (K8s NetworkPolicy blocks DB) + description: Applies a NetworkPolicy that blocks api → postgres traffic. Agent diagnoses via aks command invoke and removes the policy. + runner: .github/skills/running-demo/scripts/break-network.ps1 + minutes: 5 + needs: [] + + - id: db-perf + label: Scenario 3 — Missing index (slow queries) + description: Drops the category/name index. Catalog browse becomes slow. Agent runs DDL by exec'ing into the api pod. + runner: .github/skills/running-demo/scripts/break-db-perf.ps1 + minutes: 7 + needs: [] diff --git a/labs/zava-aks-postgres/scripts/_aks-helpers.ps1 b/labs/zava-athletic/scripts/_aks-helpers.ps1 similarity index 100% rename from labs/zava-aks-postgres/scripts/_aks-helpers.ps1 rename to labs/zava-athletic/scripts/_aks-helpers.ps1 diff --git a/labs/zava-aks-postgres/scripts/check-environment.ps1 b/labs/zava-athletic/scripts/check-environment.ps1 similarity index 100% rename from labs/zava-aks-postgres/scripts/check-environment.ps1 rename to labs/zava-athletic/scripts/check-environment.ps1 diff --git a/labs/zava-aks-postgres/scripts/post-provision.ps1 b/labs/zava-athletic/scripts/post-provision.ps1 similarity index 91% rename from labs/zava-aks-postgres/scripts/post-provision.ps1 rename to labs/zava-athletic/scripts/post-provision.ps1 index b991903c5..8f039cdc0 100644 --- a/labs/zava-aks-postgres/scripts/post-provision.ps1 +++ b/labs/zava-athletic/scripts/post-provision.ps1 @@ -24,6 +24,12 @@ param( $ErrorActionPreference = "Stop" Set-StrictMode -Version Latest +# Force UTF-8 to avoid Windows cp1252 UnicodeEncodeError in `az acr build` streaming output +$env:PYTHONIOENCODING = 'utf-8' +$env:PYTHONUTF8 = '1' +try { chcp 65001 | Out-Null } catch {} +try { [Console]::OutputEncoding = [System.Text.UTF8Encoding]::new() } catch {} + Write-Host "`n========================================" -ForegroundColor Cyan Write-Host " Zava Demo Post-Provision (PowerShell)" -ForegroundColor Cyan Write-Host "========================================`n" -ForegroundColor Cyan @@ -290,3 +296,25 @@ if ($agentName) { Write-Host "SRE_AGENT_NAME not set - skipping agent configuration." -ForegroundColor Yellow Write-Host "Run scripts\setup-sre-agent.ps1 manually after creating the agent." -ForegroundColor Yellow } + +# ── Write .deployed/.json for the meta-sim and multi-lab launcher ────── +try { + $labsRoot = Resolve-Path "$PSScriptRoot\..\.." -ErrorAction Stop + $deployedDir = Join-Path $labsRoot ".deployed" + if (-not (Test-Path $deployedDir)) { New-Item -ItemType Directory -Path $deployedDir -Force | Out-Null } + $rec = [ordered]@{ + name = "zava-athletic" + deployedAt = (Get-Date).ToUniversalTime().ToString("o") + subscriptionId = (try { Get-AzdValue "AZURE_SUBSCRIPTION_ID" } catch { "" }) + resourceGroup = $RG + region = (try { Get-AzdValue "AZURE_LOCATION" } catch { "" }) + sreAgentName = $agentName + portalUrl = "http://$ingressIP/" + labConfigPath = "" + } + $json = $rec | ConvertTo-Json -Depth 5 + Set-Content -Path (Join-Path $deployedDir "zava-athletic.json") -Value $json -Encoding UTF8 + Write-Host "✓ Recorded deployment in labs/.deployed/zava-athletic.json" -ForegroundColor Green +} catch { + Write-Host "WARN: could not write .deployed/zava-athletic.json — $_" -ForegroundColor Yellow +} diff --git a/labs/zava-aks-postgres/scripts/setup-sre-agent.ps1 b/labs/zava-athletic/scripts/setup-sre-agent.ps1 similarity index 100% rename from labs/zava-aks-postgres/scripts/setup-sre-agent.ps1 rename to labs/zava-athletic/scripts/setup-sre-agent.ps1 diff --git a/labs/zava-aks-postgres/scripts/watch-agent.ps1 b/labs/zava-athletic/scripts/watch-agent.ps1 similarity index 100% rename from labs/zava-aks-postgres/scripts/watch-agent.ps1 rename to labs/zava-athletic/scripts/watch-agent.ps1 diff --git a/labs/zava-aks-postgres/src/api/.dockerignore b/labs/zava-athletic/src/api/.dockerignore similarity index 100% rename from labs/zava-aks-postgres/src/api/.dockerignore rename to labs/zava-athletic/src/api/.dockerignore diff --git a/labs/zava-aks-postgres/src/api/Dockerfile b/labs/zava-athletic/src/api/Dockerfile similarity index 100% rename from labs/zava-aks-postgres/src/api/Dockerfile rename to labs/zava-athletic/src/api/Dockerfile diff --git a/labs/zava-aks-postgres/src/api/bin/run-sql.js b/labs/zava-athletic/src/api/bin/run-sql.js similarity index 100% rename from labs/zava-aks-postgres/src/api/bin/run-sql.js rename to labs/zava-athletic/src/api/bin/run-sql.js diff --git a/labs/zava-aks-postgres/src/api/db/client.js b/labs/zava-athletic/src/api/db/client.js similarity index 100% rename from labs/zava-aks-postgres/src/api/db/client.js rename to labs/zava-athletic/src/api/db/client.js diff --git a/labs/zava-aks-postgres/src/api/db/seed.js b/labs/zava-athletic/src/api/db/seed.js similarity index 100% rename from labs/zava-aks-postgres/src/api/db/seed.js rename to labs/zava-athletic/src/api/db/seed.js diff --git a/labs/zava-aks-postgres/src/api/logging/logger.js b/labs/zava-athletic/src/api/logging/logger.js similarity index 100% rename from labs/zava-aks-postgres/src/api/logging/logger.js rename to labs/zava-athletic/src/api/logging/logger.js diff --git a/labs/zava-aks-postgres/src/api/package-lock.json b/labs/zava-athletic/src/api/package-lock.json similarity index 100% rename from labs/zava-aks-postgres/src/api/package-lock.json rename to labs/zava-athletic/src/api/package-lock.json diff --git a/labs/zava-aks-postgres/src/api/package.json b/labs/zava-athletic/src/api/package.json similarity index 100% rename from labs/zava-aks-postgres/src/api/package.json rename to labs/zava-athletic/src/api/package.json diff --git a/labs/zava-aks-postgres/src/api/routes/diagnostics.js b/labs/zava-athletic/src/api/routes/diagnostics.js similarity index 100% rename from labs/zava-aks-postgres/src/api/routes/diagnostics.js rename to labs/zava-athletic/src/api/routes/diagnostics.js diff --git a/labs/zava-aks-postgres/src/api/routes/health.js b/labs/zava-athletic/src/api/routes/health.js similarity index 100% rename from labs/zava-aks-postgres/src/api/routes/health.js rename to labs/zava-athletic/src/api/routes/health.js diff --git a/labs/zava-aks-postgres/src/api/routes/orders.js b/labs/zava-athletic/src/api/routes/orders.js similarity index 100% rename from labs/zava-aks-postgres/src/api/routes/orders.js rename to labs/zava-athletic/src/api/routes/orders.js diff --git a/labs/zava-aks-postgres/src/api/routes/products.js b/labs/zava-athletic/src/api/routes/products.js similarity index 100% rename from labs/zava-aks-postgres/src/api/routes/products.js rename to labs/zava-athletic/src/api/routes/products.js diff --git a/labs/zava-aks-postgres/src/api/server.js b/labs/zava-athletic/src/api/server.js similarity index 100% rename from labs/zava-aks-postgres/src/api/server.js rename to labs/zava-athletic/src/api/server.js diff --git a/labs/zava-aks-postgres/src/storefront/.dockerignore b/labs/zava-athletic/src/storefront/.dockerignore similarity index 100% rename from labs/zava-aks-postgres/src/storefront/.dockerignore rename to labs/zava-athletic/src/storefront/.dockerignore diff --git a/labs/zava-aks-postgres/src/storefront/Dockerfile b/labs/zava-athletic/src/storefront/Dockerfile similarity index 100% rename from labs/zava-aks-postgres/src/storefront/Dockerfile rename to labs/zava-athletic/src/storefront/Dockerfile diff --git a/labs/zava-aks-postgres/src/storefront/package-lock.json b/labs/zava-athletic/src/storefront/package-lock.json similarity index 100% rename from labs/zava-aks-postgres/src/storefront/package-lock.json rename to labs/zava-athletic/src/storefront/package-lock.json diff --git a/labs/zava-aks-postgres/src/storefront/package.json b/labs/zava-athletic/src/storefront/package.json similarity index 100% rename from labs/zava-aks-postgres/src/storefront/package.json rename to labs/zava-athletic/src/storefront/package.json diff --git a/labs/zava-aks-postgres/src/storefront/server.js b/labs/zava-athletic/src/storefront/server.js similarity index 100% rename from labs/zava-aks-postgres/src/storefront/server.js rename to labs/zava-athletic/src/storefront/server.js diff --git a/labs/zava-aks-postgres/sre-config/knowledge-base/zava-architecture.md b/labs/zava-athletic/sre-config/knowledge-base/zava-architecture.md similarity index 100% rename from labs/zava-aks-postgres/sre-config/knowledge-base/zava-architecture.md rename to labs/zava-athletic/sre-config/knowledge-base/zava-architecture.md diff --git a/labs/zava-cafe/.gitignore b/labs/zava-cafe/.gitignore new file mode 100644 index 000000000..bd70b5bd7 --- /dev/null +++ b/labs/zava-cafe/.gitignore @@ -0,0 +1,12 @@ +.azure/ +.deployed/ +*.zip +bin/ +obj/ +node_modules/ +__pycache__/ +*.pyc +.venv/ +.env +.tmp/ +publish/ diff --git a/labs/zava-cafe/README.md b/labs/zava-cafe/README.md new file mode 100644 index 000000000..8c8562f76 --- /dev/null +++ b/labs/zava-cafe/README.md @@ -0,0 +1,112 @@ +# Zava — Zava Café SRE Agent Lab + +A realistic e-commerce platform (Zava) running on Azure App Service + Azure SQL +DB. The lab is wired up so that we can break the SQL DB or the web tier on +purpose and watch an Azure SRE Agent investigate, diagnose, and (with the right +hooks) remediate the issue. + +This lab focuses purely on the SQL/DevOps ops scenarios. The IT-support / +laptop-replacement story has moved to its own standalone lab at +[`../zava-itsupport/`](../zava-itsupport/). + +## Stack + +- **App**: .NET 8 / ASP.NET Core — Zava Café specialty coffee e-commerce storefront (espresso, brewed coffee, pastries, merch) +- **Compute**: Azure App Service (P0v3 Linux App Service Plan) +- **Data**: Azure SQL Database (Basic, 5 DTU) — intentionally tiny so DTU spikes are easy to trigger; seeded from `infra/seed-database.sql` +- **Observability**: Log Analytics + Application Insights + Azure Portal Dashboard + 3 metric alert rules (SQL DTU > 80%, App Service HTTP 5xx > 5/5min, App Service health-check < 100%) +- **SRE Agent**: `sre-agent-zava-cafe-` — single workspace with the `agent1` config: subagents `sql-performance-investigator`, `deployment-validator`, `deployment-validator-gh`; skills for blocking-chain and slow-query diagnosis/fix; hooks `sql-write-guard` + `change-risk-assessor`; tool `AssessChangeRisk`; weekly cost-report scheduled task +- **Simulator**: PowerShell scenario runners under `sre-config/` (`simulate-dtu-spike.ps1`, `simulate-slow-queries.ps1`) +- **CI/CD**: `azd up` (sub-scope Bicep → RG-scope Bicep → seed SQL → deploy .NET source → `srectl apply`) + +## What it's about + +The Zava Café lab is for PMs, SREs, and customers who want to see the **Azure SRE Agent investigate and remediate Azure SQL performance incidents** end-to-end on a realistic e-commerce workload. Zava Café is a fictional specialty coffee shop running its storefront on App Service backed by a deliberately small Azure SQL DB — so a couple of bad queries reliably spike DTU, miss indexes, or chain blocking sessions. The lab teaches break/fix patterns around DTU exhaustion, missing indexes / slow queries, blocking chains, and post-deploy regression validation, while also showing the safety story (write-guard hook + AI change-risk assessor + human-in-the-loop approval). + +Demo flow: `azd up` provisions infra, deploys the .NET app, and registers the SRE Agent workspace via `srectl`. From there, run `pwsh sre-config/simulate-dtu-spike.ps1` (or `simulate-slow-queries.ps1`) to fault the DB → an Azure Monitor alert fires → the agent picks up the incident, runs the matching `sql-*-diagnosis` skill, plots a chart, asks the user to approve the fix, and applies it. The same agent's `deployment-validator` subagents handle post-release health checks (ADO and GitHub Actions paths) and roll back automatically on regression. + +A single SRE Agent workspace is deployed: + +- **agent1** — SQL/DevOps: `sql-performance-investigator`, + `deployment-validator`, `deployment-validator-gh`. Skills cover + blocking-chain diagnosis/fix, slow-query diagnosis/fix. Includes a write + guard hook (`sql-write-guard`) and a change-risk-assessor hook backed by the + `AssessChangeRisk` Python tool. A weekly cost-report scheduled task is + registered. + +## Architecture (text sketch) + +``` + ┌──────────────────────────────┐ + │ Azure SRE Agent (autonomous)│ + │ + agent1 workspace │ + └──────────────┬───────────────┘ + │ alerts (DTU, 5xx, health-check) + ┌──────────────┴───────────────┐ + │ Azure Monitor │ + └──┬───────────────────────────┘ + │ + ┌───────┴──┐ + │ Zava .NET│ + │ (App Svc)│ + └────┬─────┘ + │ + ┌────┴──────────────┐ + │ Azure SQL DB (B5) │ + └───────────────────┘ +``` + +## Quick start + +```pwsh +# 1. Install azd if needed +winget install Microsoft.Azd + +# 2. Login + pick a subscription +az login +azd auth login + +# 3. (Optional) Override the SQL admin password — otherwise one is generated +azd env set SQL_ADMIN_PASSWORD "" + +# 4. Deploy via the Zava launcher +pwsh ../lab.ps1 -Labs zava-cafe +``` + +The launcher invokes `azd up`, which: + +1. Runs `scripts/prereqs.sh` (preprovision hook) to verify tools and stash a + SQL password into the azd env. +2. Provisions infra via `infra/main.bicep` (sub-scope) → + `infra/resources.bicep` (RG-scope). +3. Runs `scripts/post-provision.sh`, which seeds SQL, deploys the .NET web + app from source, then registers the SRE Agent workspace with `srectl` + and fires a smoke-test thread. + +## Scenarios + +| id | runner | what it does | +|---|---|---| +| `dtu-spike` | `sre-config/simulate-dtu-spike.ps1` | Floods SQL with heavy queries → DTU > 80% → alert → agent investigates | +| `slow-queries` | `sre-config/simulate-slow-queries.ps1` | Generates a flood of slow queries → agent recommends an index | + +## What gets deployed + +- **Azure SQL Server + DB** (Basic 5 DTU) — intentionally small so the demos + spike easily. Seeded from `infra/seed-database.sql`. +- **App Service Plan** (P0v3 Linux) hosting: + - Zava main app (.NET 8, `src/`) +- **Log Analytics + App Insights** (linked) for telemetry. +- **3 metric alert rules**: SQL DTU > 80%, App Service HTTP 5xx > 5/5min, + App Service health-check < 100%. +- **Azure Portal Dashboard** with key metrics. +- **User-Assigned Managed Identity** with subscription-scoped Reader + + Monitoring + Log Analytics + Container Apps Contributor roles. +- **Azure SRE Agent** (`sre-agent-zava-cafe-`) wired to the App + Insights workspace, with `sre-config/agent1` resources registered via + `srectl` in the post-provision hook. + +## Skipping the srectl block + +Set `LABS_SKIP_SRECTL=1` before `azd up` (or before re-running +`bash scripts/post-provision.sh`) to skip agent registration entirely. diff --git a/labs/zava-cafe/azure.yaml b/labs/zava-cafe/azure.yaml new file mode 100644 index 000000000..2ea4cccd2 --- /dev/null +++ b/labs/zava-cafe/azure.yaml @@ -0,0 +1,43 @@ +# Azure Developer CLI (azd) template for Zava — Zava Café SRE Agent Lab +# Run: azd up +name: sre-agent-zava-cafe-lab +metadata: + template: sre-agent-zava-cafe-lab@1.0.0 + +infra: + provider: bicep + path: infra + +# No services block — the .NET web app is deployed from source by the +# post-provision hook (az webapp deploy from a zip), so no azd `services` config +# is needed. + +hooks: + preprovision: + posix: + shell: sh + run: scripts/prereqs.sh + interactive: true + windows: + shell: pwsh + run: | + if (-not (Get-Command bash -ErrorAction SilentlyContinue)) { + Write-Error "bash not found. Install Git for Windows (https://git-scm.com/download/win)." + exit 1 + } + bash scripts/prereqs.sh + interactive: true + postprovision: + posix: + shell: sh + run: scripts/post-provision.sh + interactive: true + windows: + shell: pwsh + run: | + if (-not (Get-Command bash -ErrorAction SilentlyContinue)) { + Write-Error "bash not found. Install Git for Windows (https://git-scm.com/download/win)." + exit 1 + } + bash scripts/post-provision.sh + interactive: true diff --git a/labs/zava-cafe/dashboard.json b/labs/zava-cafe/dashboard.json new file mode 100644 index 000000000..272cbd125 --- /dev/null +++ b/labs/zava-cafe/dashboard.json @@ -0,0 +1,857 @@ +{ + "location": "westus2", + "tags": { + "hidden-title": "Zava Operations Dashboard" + }, + "properties": { + "lenses": [ + { + "order": 0, + "parts": [ + { + "position": { + "x": 0, + "y": 0, + "colSpan": 16, + "rowSpan": 2 + }, + "metadata": { + "type": "Extension/HubsExtension/PartType/MarkdownPart", + "inputs": [], + "settings": { + "content": { + "content": "## Zava Operations Dashboard\n**Real-time monitoring** for SQL Database, App Service, and Application Insights.\n\n_Subscription:_ ``  |  _Resource Group:_ `rg-zava`  |  _Region:_ `westus2`", + "title": "Zava Operations Dashboard", + "subtitle": "Enterprise Monitoring", + "markdownSource": 1, + "markdownUri": null + } + } + } + }, + { + "position": { + "x": 16, + "y": 0, + "colSpan": 4, + "rowSpan": 2 + }, + "metadata": { + "type": "Extension/HubsExtension/PartType/MarkdownPart", + "inputs": [], + "settings": { + "content": { + "content": "[> Resource Group](https://portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions//resourceGroups/rg-zava/overview)\n\n[> SQL Database](https://portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions//resourceGroups/rg-zava/providers/Microsoft.Sql/servers/sql-zava/databases/sqldb-zava/overview)", + "title": "Quick Links", + "subtitle": "", + "markdownSource": 1, + "markdownUri": null + } + } + } + }, + { + "position": { + "x": 0, + "y": 2, + "colSpan": 20, + "rowSpan": 1 + }, + "metadata": { + "type": "Extension/HubsExtension/PartType/MarkdownPart", + "inputs": [], + "settings": { + "content": { + "content": "### SQL Database Health", + "title": "", + "subtitle": "", + "markdownSource": 1, + "markdownUri": null + } + } + } + }, + { + "position": { + "x": 0, + "y": 3, + "colSpan": 5, + "rowSpan": 4 + }, + "metadata": { + "type": "Extension/HubsExtension/PartType/MonitorChartPart", + "inputs": [], + "settings": { + "content": { + "options": { + "chart": { + "metrics": [ + { + "resourceMetadata": { + "id": "/subscriptions//resourceGroups/rg-zava/providers/Microsoft.Sql/servers/sql-zava/databases/sqldb-zava" + }, + "name": "dtu_consumption_percent", + "aggregationType": 4, + "namespace": "Microsoft.Sql/servers/databases", + "metricVisualization": { + "displayName": "DTU %", + "resourceDisplayName": "sqldb-zava" + } + } + ], + "title": "SQL DTU Consumption %", + "titleKind": 1, + "visualization": { + "chartType": 2, + "legendVisualization": { + "isVisible": true, + "position": 2, + "hideSubtitle": false + }, + "axisVisualization": { + "x": { + "isVisible": true, + "axisType": 2 + }, + "y": { + "isVisible": true, + "axisType": 2 + } + }, + "disablePinning": true + }, + "timespan": { + "relative": { + "duration": 3600000 + } + } + } + } + } + } + } + }, + { + "position": { + "x": 5, + "y": 3, + "colSpan": 5, + "rowSpan": 4 + }, + "metadata": { + "type": "Extension/HubsExtension/PartType/MonitorChartPart", + "inputs": [], + "settings": { + "content": { + "options": { + "chart": { + "metrics": [ + { + "resourceMetadata": { + "id": "/subscriptions//resourceGroups/rg-zava/providers/Microsoft.Sql/servers/sql-zava/databases/sqldb-zava" + }, + "name": "cpu_percent", + "aggregationType": 4, + "namespace": "Microsoft.Sql/servers/databases", + "metricVisualization": { + "displayName": "CPU %", + "resourceDisplayName": "sqldb-zava" + } + } + ], + "title": "SQL CPU Percentage", + "titleKind": 1, + "visualization": { + "chartType": 2, + "legendVisualization": { + "isVisible": true, + "position": 2, + "hideSubtitle": false + }, + "axisVisualization": { + "x": { + "isVisible": true, + "axisType": 2 + }, + "y": { + "isVisible": true, + "axisType": 2 + } + }, + "disablePinning": true + }, + "timespan": { + "relative": { + "duration": 3600000 + } + } + } + } + } + } + } + }, + { + "position": { + "x": 10, + "y": 3, + "colSpan": 5, + "rowSpan": 4 + }, + "metadata": { + "type": "Extension/HubsExtension/PartType/MonitorChartPart", + "inputs": [], + "settings": { + "content": { + "options": { + "chart": { + "metrics": [ + { + "resourceMetadata": { + "id": "/subscriptions//resourceGroups/rg-zava/providers/Microsoft.Sql/servers/sql-zava/databases/sqldb-zava" + }, + "name": "physical_data_read_percent", + "aggregationType": 4, + "namespace": "Microsoft.Sql/servers/databases", + "metricVisualization": { + "displayName": "Data IO %", + "resourceDisplayName": "sqldb-zava" + } + } + ], + "title": "SQL Data IO Percentage", + "titleKind": 1, + "visualization": { + "chartType": 2, + "legendVisualization": { + "isVisible": true, + "position": 2, + "hideSubtitle": false + }, + "axisVisualization": { + "x": { + "isVisible": true, + "axisType": 2 + }, + "y": { + "isVisible": true, + "axisType": 2 + } + }, + "disablePinning": true + }, + "timespan": { + "relative": { + "duration": 3600000 + } + } + } + } + } + } + } + }, + { + "position": { + "x": 15, + "y": 3, + "colSpan": 5, + "rowSpan": 4 + }, + "metadata": { + "type": "Extension/HubsExtension/PartType/MonitorChartPart", + "inputs": [], + "settings": { + "content": { + "options": { + "chart": { + "metrics": [ + { + "resourceMetadata": { + "id": "/subscriptions//resourceGroups/rg-zava/providers/Microsoft.Sql/servers/sql-zava/databases/sqldb-zava" + }, + "name": "connection_successful", + "aggregationType": 1, + "namespace": "Microsoft.Sql/servers/databases", + "metricVisualization": { + "displayName": "Connections", + "resourceDisplayName": "sqldb-zava" + } + } + ], + "title": "SQL Successful Connections", + "titleKind": 1, + "visualization": { + "chartType": 2, + "legendVisualization": { + "isVisible": true, + "position": 2, + "hideSubtitle": false + }, + "axisVisualization": { + "x": { + "isVisible": true, + "axisType": 2 + }, + "y": { + "isVisible": true, + "axisType": 2 + } + }, + "disablePinning": true + }, + "timespan": { + "relative": { + "duration": 3600000 + } + } + } + } + } + } + } + }, + { + "position": { + "x": 0, + "y": 7, + "colSpan": 20, + "rowSpan": 1 + }, + "metadata": { + "type": "Extension/HubsExtension/PartType/MarkdownPart", + "inputs": [], + "settings": { + "content": { + "content": "### App Service Performance", + "title": "", + "subtitle": "", + "markdownSource": 1, + "markdownUri": null + } + } + } + }, + { + "position": { + "x": 0, + "y": 8, + "colSpan": 5, + "rowSpan": 4 + }, + "metadata": { + "type": "Extension/HubsExtension/PartType/MonitorChartPart", + "inputs": [], + "settings": { + "content": { + "options": { + "chart": { + "metrics": [ + { + "resourceMetadata": { + "id": "/subscriptions//resourceGroups/rg-zava/providers/Microsoft.Web/sites/app-zava" + }, + "name": "HttpResponseTime", + "aggregationType": 4, + "namespace": "Microsoft.Web/sites", + "metricVisualization": { + "displayName": "Avg Response Time", + "resourceDisplayName": "app-zava" + } + } + ], + "title": "HTTP Response Time", + "titleKind": 1, + "visualization": { + "chartType": 2, + "legendVisualization": { + "isVisible": true, + "position": 2, + "hideSubtitle": false + }, + "axisVisualization": { + "x": { + "isVisible": true, + "axisType": 2 + }, + "y": { + "isVisible": true, + "axisType": 2 + } + }, + "disablePinning": true + }, + "timespan": { + "relative": { + "duration": 3600000 + } + } + } + } + } + } + } + }, + { + "position": { + "x": 5, + "y": 8, + "colSpan": 5, + "rowSpan": 4 + }, + "metadata": { + "type": "Extension/HubsExtension/PartType/MonitorChartPart", + "inputs": [], + "settings": { + "content": { + "options": { + "chart": { + "metrics": [ + { + "resourceMetadata": { + "id": "/subscriptions//resourceGroups/rg-zava/providers/Microsoft.Web/sites/app-zava" + }, + "name": "Http2xx", + "aggregationType": 1, + "namespace": "Microsoft.Web/sites", + "metricVisualization": { + "displayName": "2xx", + "resourceDisplayName": "app-zava" + } + }, + { + "resourceMetadata": { + "id": "/subscriptions//resourceGroups/rg-zava/providers/Microsoft.Web/sites/app-zava" + }, + "name": "Http4xx", + "aggregationType": 1, + "namespace": "Microsoft.Web/sites", + "metricVisualization": { + "displayName": "4xx", + "resourceDisplayName": "app-zava" + } + }, + { + "resourceMetadata": { + "id": "/subscriptions//resourceGroups/rg-zava/providers/Microsoft.Web/sites/app-zava" + }, + "name": "Http5xx", + "aggregationType": 1, + "namespace": "Microsoft.Web/sites", + "metricVisualization": { + "displayName": "5xx", + "resourceDisplayName": "app-zava" + } + } + ], + "title": "HTTP Status Codes", + "titleKind": 1, + "visualization": { + "chartType": 3, + "legendVisualization": { + "isVisible": true, + "position": 2, + "hideSubtitle": false + }, + "axisVisualization": { + "x": { + "isVisible": true, + "axisType": 2 + }, + "y": { + "isVisible": true, + "axisType": 2 + } + }, + "disablePinning": true + }, + "timespan": { + "relative": { + "duration": 3600000 + } + } + } + } + } + } + } + }, + { + "position": { + "x": 10, + "y": 8, + "colSpan": 5, + "rowSpan": 4 + }, + "metadata": { + "type": "Extension/HubsExtension/PartType/MonitorChartPart", + "inputs": [], + "settings": { + "content": { + "options": { + "chart": { + "metrics": [ + { + "resourceMetadata": { + "id": "/subscriptions//resourceGroups/rg-zava/providers/Microsoft.Web/serverfarms/plan-zava" + }, + "name": "CpuPercentage", + "aggregationType": 4, + "namespace": "Microsoft.Web/serverfarms", + "metricVisualization": { + "displayName": "CPU %", + "resourceDisplayName": "plan-zava" + } + } + ], + "title": "App Service Plan \u2014 CPU %", + "titleKind": 1, + "visualization": { + "chartType": 2, + "legendVisualization": { + "isVisible": true, + "position": 2, + "hideSubtitle": false + }, + "axisVisualization": { + "x": { + "isVisible": true, + "axisType": 2 + }, + "y": { + "isVisible": true, + "axisType": 2 + } + }, + "disablePinning": true + }, + "timespan": { + "relative": { + "duration": 3600000 + } + } + } + } + } + } + } + }, + { + "position": { + "x": 15, + "y": 8, + "colSpan": 5, + "rowSpan": 4 + }, + "metadata": { + "type": "Extension/HubsExtension/PartType/MonitorChartPart", + "inputs": [], + "settings": { + "content": { + "options": { + "chart": { + "metrics": [ + { + "resourceMetadata": { + "id": "/subscriptions//resourceGroups/rg-zava/providers/Microsoft.Web/serverfarms/plan-zava" + }, + "name": "MemoryPercentage", + "aggregationType": 4, + "namespace": "Microsoft.Web/serverfarms", + "metricVisualization": { + "displayName": "Memory %", + "resourceDisplayName": "plan-zava" + } + } + ], + "title": "App Service Plan \u2014 Memory %", + "titleKind": 1, + "visualization": { + "chartType": 2, + "legendVisualization": { + "isVisible": true, + "position": 2, + "hideSubtitle": false + }, + "axisVisualization": { + "x": { + "isVisible": true, + "axisType": 2 + }, + "y": { + "isVisible": true, + "axisType": 2 + } + }, + "disablePinning": true + }, + "timespan": { + "relative": { + "duration": 3600000 + } + } + } + } + } + } + } + }, + { + "position": { + "x": 0, + "y": 12, + "colSpan": 20, + "rowSpan": 1 + }, + "metadata": { + "type": "Extension/HubsExtension/PartType/MarkdownPart", + "inputs": [], + "settings": { + "content": { + "content": "### Application Insights", + "title": "", + "subtitle": "", + "markdownSource": 1, + "markdownUri": null + } + } + } + }, + { + "position": { + "x": 0, + "y": 13, + "colSpan": 5, + "rowSpan": 4 + }, + "metadata": { + "type": "Extension/HubsExtension/PartType/MonitorChartPart", + "inputs": [], + "settings": { + "content": { + "options": { + "chart": { + "metrics": [ + { + "resourceMetadata": { + "id": "/subscriptions//resourceGroups/rg-zava/providers/microsoft.insights/components/ai-zava" + }, + "name": "requests/duration", + "aggregationType": 4, + "namespace": "microsoft.insights/components", + "metricVisualization": { + "displayName": "Avg Response Time", + "resourceDisplayName": "ai-zava" + } + } + ], + "title": "Server Response Time", + "titleKind": 1, + "visualization": { + "chartType": 2, + "legendVisualization": { + "isVisible": true, + "position": 2, + "hideSubtitle": false + }, + "axisVisualization": { + "x": { + "isVisible": true, + "axisType": 2 + }, + "y": { + "isVisible": true, + "axisType": 2 + } + }, + "disablePinning": true + }, + "timespan": { + "relative": { + "duration": 3600000 + } + } + } + } + } + } + } + }, + { + "position": { + "x": 5, + "y": 13, + "colSpan": 5, + "rowSpan": 4 + }, + "metadata": { + "type": "Extension/HubsExtension/PartType/MonitorChartPart", + "inputs": [], + "settings": { + "content": { + "options": { + "chart": { + "metrics": [ + { + "resourceMetadata": { + "id": "/subscriptions//resourceGroups/rg-zava/providers/microsoft.insights/components/ai-zava" + }, + "name": "requests/failed", + "aggregationType": 1, + "namespace": "microsoft.insights/components", + "metricVisualization": { + "displayName": "Failed Requests", + "resourceDisplayName": "ai-zava" + } + } + ], + "title": "Failed Requests", + "titleKind": 1, + "visualization": { + "chartType": 2, + "legendVisualization": { + "isVisible": true, + "position": 2, + "hideSubtitle": false + }, + "axisVisualization": { + "x": { + "isVisible": true, + "axisType": 2 + }, + "y": { + "isVisible": true, + "axisType": 2 + } + }, + "disablePinning": true + }, + "timespan": { + "relative": { + "duration": 3600000 + } + } + } + } + } + } + } + }, + { + "position": { + "x": 10, + "y": 13, + "colSpan": 5, + "rowSpan": 4 + }, + "metadata": { + "type": "Extension/HubsExtension/PartType/MonitorChartPart", + "inputs": [], + "settings": { + "content": { + "options": { + "chart": { + "metrics": [ + { + "resourceMetadata": { + "id": "/subscriptions//resourceGroups/rg-zava/providers/microsoft.insights/components/ai-zava" + }, + "name": "dependencies/failed", + "aggregationType": 1, + "namespace": "microsoft.insights/components", + "metricVisualization": { + "displayName": "Failed Dependencies", + "resourceDisplayName": "ai-zava" + } + } + ], + "title": "Dependency Failures", + "titleKind": 1, + "visualization": { + "chartType": 2, + "legendVisualization": { + "isVisible": true, + "position": 2, + "hideSubtitle": false + }, + "axisVisualization": { + "x": { + "isVisible": true, + "axisType": 2 + }, + "y": { + "isVisible": true, + "axisType": 2 + } + }, + "disablePinning": true + }, + "timespan": { + "relative": { + "duration": 3600000 + } + } + } + } + } + } + } + }, + { + "position": { + "x": 15, + "y": 13, + "colSpan": 5, + "rowSpan": 4 + }, + "metadata": { + "type": "Extension/HubsExtension/PartType/MarkdownPart", + "inputs": [], + "settings": { + "content": { + "content": "### Active Alert Rules\n\n| Alert | Target | Sev |\n|-------|--------|-----|\n| **SQL DTU > 80%** | sqldb-zava | 2 |\n| **HTTP 5xx > 10** | app-zava | 1 |\n| **Response > 5s** | ai-zava | 2 |\n\n_[Manage Alerts](https://portal.azure.com/#blade/Microsoft_Azure_Monitoring/AzureMonitoringBrowseBlade/alertsV2)_", + "title": "Active Alert Rules", + "subtitle": "Configured Monitors", + "markdownSource": 1, + "markdownUri": null + } + } + } + } + ] + } + ], + "metadata": { + "model": { + "timeRange": { + "value": { + "relative": { + "duration": 24, + "timeUnit": 1 + } + }, + "type": "MsPortalFx.Composition.Configuration.ValueTypes.TimeRange" + }, + "filterLocale": { + "value": "en-us" + }, + "filters": { + "value": { + "MsPortalFx_TimeRange": { + "model": { + "format": "utc", + "granularity": "auto", + "relative": "1h" + }, + "displayCache": { + "name": "UTC Time", + "value": "Past hour" + }, + "filteredPartIds": [] + } + } + } + } + } + } +} \ No newline at end of file diff --git a/labs/zava-cafe/infra/main.bicep b/labs/zava-cafe/infra/main.bicep new file mode 100644 index 000000000..0c3f7b03b --- /dev/null +++ b/labs/zava-cafe/infra/main.bicep @@ -0,0 +1,68 @@ +// ────────────────────────────────────────────────────────────── +// Zava — Zava Café SRE Agent Lab +// Subscription-scoped entrypoint. Creates the resource group and +// delegates to resources.bicep + subscription-rbac.bicep. +// ────────────────────────────────────────────────────────────── + +targetScope = 'subscription' + +@description('Name of the environment (auto-populated by azd)') +param environmentName string + +@description('Primary location for all resources') +param location string = 'westus2' + +@description('Naming prefix for all resources') +param prefix string = 'zava' + +@description('Entra ID user/group login (UPN) to set as SQL Server admin') +param aadAdminLogin string + +@description('Entra ID user/group object ID to set as SQL Server admin') +param aadAdminObjectId string + +@description('Optional alert notification email address') +param alertEmail string = '' + +// Resource group +var resourceGroupName = 'rg-${environmentName}' + +resource rg 'Microsoft.Resources/resourceGroups@2024-03-01' = { + name: resourceGroupName + location: location +} + +// Deploy app + agent resources into the resource group +module resources 'resources.bicep' = { + name: 'resources-deployment' + scope: rg + params: { + environmentName: environmentName + location: location + prefix: prefix + aadAdminLogin: aadAdminLogin + aadAdminObjectId: aadAdminObjectId + alertEmail: alertEmail + } +} + +// Subscription-scoped RBAC for SRE Agent managed identity +module subscriptionRbac 'modules/subscription-rbac.bicep' = { + name: 'subscription-rbac-${environmentName}' + params: { + principalId: resources.outputs.identityPrincipalId + } +} + +// Outputs consumed by azd and post-provision script +output AZURE_RESOURCE_GROUP string = rg.name +output AZURE_LOCATION string = location +output SRE_AGENT_NAME string = resources.outputs.agentName +output SRE_AGENT_ENDPOINT string = resources.outputs.agentEndpoint +output AGENT_PORTAL_URL string = resources.outputs.agentPortalUrl +output AZURE_SQL_SERVER_FQDN string = resources.outputs.sqlServerFqdn +output AZURE_SQL_DATABASE string = resources.outputs.sqlDatabaseName +output AZURE_APP_URL string = resources.outputs.appUrl +output AZURE_APP_NAME string = resources.outputs.appName +output AZURE_WEBAPP_PRINCIPAL_ID string = resources.outputs.webAppPrincipalId +output APPINSIGHTS_CONNECTION_STRING string = resources.outputs.appInsightsConnectionString diff --git a/labs/zava-cafe/infra/main.bicepparam b/labs/zava-cafe/infra/main.bicepparam new file mode 100644 index 000000000..84b90e0b5 --- /dev/null +++ b/labs/zava-cafe/infra/main.bicepparam @@ -0,0 +1,8 @@ +using './main.bicep' + +param environmentName = readEnvironmentVariable('AZURE_ENV_NAME', 'zavacafe') +param location = readEnvironmentVariable('AZURE_LOCATION', 'westus2') +param prefix = 'zava' +param alertEmail = readEnvironmentVariable('ALERT_EMAIL', '') +param aadAdminLogin = readEnvironmentVariable('AAD_ADMIN_LOGIN', '') +param aadAdminObjectId = readEnvironmentVariable('AAD_ADMIN_OBJECT_ID', '') diff --git a/labs/starter-lab/infra/modules/identity.bicep b/labs/zava-cafe/infra/modules/identity.bicep similarity index 100% rename from labs/starter-lab/infra/modules/identity.bicep rename to labs/zava-cafe/infra/modules/identity.bicep diff --git a/labs/starter-lab/infra/modules/monitoring.bicep b/labs/zava-cafe/infra/modules/monitoring.bicep similarity index 100% rename from labs/starter-lab/infra/modules/monitoring.bicep rename to labs/zava-cafe/infra/modules/monitoring.bicep diff --git a/labs/starter-lab/infra/modules/sre-agent.bicep b/labs/zava-cafe/infra/modules/sre-agent.bicep similarity index 100% rename from labs/starter-lab/infra/modules/sre-agent.bicep rename to labs/zava-cafe/infra/modules/sre-agent.bicep diff --git a/labs/starter-lab/infra/modules/subscription-rbac.bicep b/labs/zava-cafe/infra/modules/subscription-rbac.bicep similarity index 100% rename from labs/starter-lab/infra/modules/subscription-rbac.bicep rename to labs/zava-cafe/infra/modules/subscription-rbac.bicep diff --git a/labs/zava-cafe/infra/resources.bicep b/labs/zava-cafe/infra/resources.bicep new file mode 100644 index 000000000..422bdedfe --- /dev/null +++ b/labs/zava-cafe/infra/resources.bicep @@ -0,0 +1,444 @@ +// ────────────────────────────────────────────────────────────── +// Zava — Zava Café SRE Agent Lab — Resource Group resources +// Adapted from the source ZavaCafe-SREAgent-fresh main.bicep +// + adds: managed identity, SRE Agent, monitoring module wiring. +// ────────────────────────────────────────────────────────────── + +targetScope = 'resourceGroup' + +// ── Parameters ────────────────────────────────────────────── + +@description('Name of the environment (from azd)') +param environmentName string + +@description('Azure region for all resources') +param location string + +@description('Naming prefix for all resources') +param prefix string = 'zava' + +@description('Alert notification email address (optional)') +param alertEmail string = '' + +@description('Entra ID user/group login (UPN) to set as SQL Server admin') +param aadAdminLogin string + +@description('Entra ID user/group object ID to set as SQL Server admin') +param aadAdminObjectId string + +// ── Variables ─────────────────────────────────────────────── + +var resourceToken = take(uniqueString(resourceGroup().id, environmentName, prefix), 8) +var sqlServerName = 'sql-${prefix}-${resourceToken}' +var sqlDatabaseName = 'sqldb-${prefix}' +var lawName = 'law-${prefix}-${resourceToken}' +var appInsightsName = 'ai-${prefix}-${resourceToken}' +var aspName = 'asp-${prefix}-${resourceToken}' +var appName = 'app-${prefix}-${resourceToken}' +var dashboardName = 'dash-${prefix}-${resourceToken}' +var identityName = 'id-sre-${prefix}-${resourceToken}' +var agentName = 'sre-agent-zava-cafe-${resourceToken}' + +// ── 1. SQL Server ─────────────────────────────────────────── + +resource sqlServer 'Microsoft.Sql/servers@2023-08-01-preview' = { + name: sqlServerName + location: location + properties: { + version: '12.0' + publicNetworkAccess: 'Enabled' + administrators: { + administratorType: 'ActiveDirectory' + principalType: 'User' + login: aadAdminLogin + sid: aadAdminObjectId + tenantId: tenant().tenantId + azureADOnlyAuthentication: true + } + } +} + +resource sqlDatabase 'Microsoft.Sql/servers/databases@2023-08-01-preview' = { + parent: sqlServer + name: sqlDatabaseName + location: location + sku: { + name: 'Basic' + tier: 'Basic' + capacity: 5 + } + properties: { + collation: 'SQL_Latin1_General_CP1_CI_AS' + maxSizeBytes: 2147483648 + } +} + +resource sqlFirewallAzure 'Microsoft.Sql/servers/firewallRules@2023-08-01-preview' = { + parent: sqlServer + name: 'AllowAzureServices' + properties: { + startIpAddress: '0.0.0.0' + endIpAddress: '0.0.0.0' + } +} + +// ── 2. Monitoring (LAW + App Insights) — via module ───────── + +module monitoring 'modules/monitoring.bicep' = { + name: 'monitoring' + params: { + location: location + logAnalyticsName: lawName + appInsightsName: appInsightsName + } +} + +// ── 3. Managed Identity for SRE Agent ─────────────────────── + +module identity 'modules/identity.bicep' = { + name: 'identity' + params: { + location: location + identityName: identityName + } +} + +// ── 4. App Service Plan (Linux) ────────────────────────── + +@description('App Service Plan SKU name (e.g. P0v3, P1v3, S1, B1). Default S1 to avoid Premium V3 quota dependency.') +param appServicePlanSku string = 'S1' + +@description('App Service Plan SKU tier (e.g. Premium0V3, PremiumV3, Standard, Basic).') +param appServicePlanTier string = 'Standard' + +resource appServicePlan 'Microsoft.Web/serverfarms@2023-12-01' = { + name: aspName + location: location + kind: 'linux' + sku: { + name: appServicePlanSku + tier: appServicePlanTier + capacity: 1 + } + properties: { + reserved: true + } +} + +// ── 5. Web App — Main App (.NET 8) ────────────────────────── + +resource webApp 'Microsoft.Web/sites@2023-12-01' = { + name: appName + location: location + identity: { + type: 'SystemAssigned' + } + properties: { + serverFarmId: appServicePlan.id + httpsOnly: true + siteConfig: { + linuxFxVersion: 'DOTNETCORE|8.0' + alwaysOn: true + healthCheckPath: '/health' + appSettings: [ + { + name: 'APPLICATIONINSIGHTS_CONNECTION_STRING' + value: monitoring.outputs.appInsightsConnectionString + } + { + name: 'ApplicationInsightsAgent_EXTENSION_VERSION' + value: '~3' + } + { + name: 'AZURE_SQL_SERVER' + value: sqlServer.properties.fullyQualifiedDomainName + } + { + name: 'AZURE_SQL_DATABASE' + value: sqlDatabaseName + } + ] + connectionStrings: [ + { + name: 'DefaultConnection' + connectionString: 'Server=tcp:${sqlServer.properties.fullyQualifiedDomainName},1433;Database=${sqlDatabaseName};Authentication=Active Directory Default;Encrypt=True;TrustServerCertificate=False;' + type: 'SQLAzure' + } + ] + } + } +} + +// ── 8. Action Group + Alerts ──────────────────────────────── + +resource actionGroup 'Microsoft.Insights/actionGroups@2023-09-01-preview' = if (!empty(alertEmail)) { + name: 'ag-${prefix}-sre' + location: 'global' + properties: { + groupShortName: '${prefix}SRE' + enabled: true + emailReceivers: [ + { + name: 'SRE Team' + emailAddress: alertEmail + useCommonAlertSchema: true + } + ] + } +} + +resource alertDtu 'Microsoft.Insights/metricAlerts@2018-03-01' = { + name: 'alert-${prefix}-dtu-high' + location: 'global' + properties: { + description: 'SQL Database DTU usage exceeds 80%' + severity: 2 + enabled: true + scopes: [ sqlDatabase.id ] + evaluationFrequency: 'PT1M' + windowSize: 'PT5M' + criteria: { + 'odata.type': 'Microsoft.Azure.Monitor.SingleResourceMultipleMetricCriteria' + allOf: [ + { + name: 'HighDTU' + metricName: 'dtu_consumption_percent' + metricNamespace: 'Microsoft.Sql/servers/databases' + operator: 'GreaterThan' + threshold: 80 + timeAggregation: 'Average' + criterionType: 'StaticThresholdCriterion' + } + ] + } + actions: !empty(alertEmail) ? [ { actionGroupId: actionGroup.id } ] : [] + } +} + +resource alertHttp5xx 'Microsoft.Insights/metricAlerts@2018-03-01' = { + name: 'alert-${prefix}-http-5xx' + location: 'global' + properties: { + description: 'App Service returning HTTP 5xx errors' + severity: 1 + enabled: true + scopes: [ webApp.id ] + evaluationFrequency: 'PT1M' + windowSize: 'PT5M' + criteria: { + 'odata.type': 'Microsoft.Azure.Monitor.SingleResourceMultipleMetricCriteria' + allOf: [ + { + name: 'Http5xx' + metricName: 'Http5xx' + metricNamespace: 'Microsoft.Web/sites' + operator: 'GreaterThan' + threshold: 5 + timeAggregation: 'Total' + criterionType: 'StaticThresholdCriterion' + } + ] + } + actions: !empty(alertEmail) ? [ { actionGroupId: actionGroup.id } ] : [] + } +} + +resource alertHealthCheck 'Microsoft.Insights/metricAlerts@2018-03-01' = { + name: 'alert-${prefix}-health-check' + location: 'global' + properties: { + description: 'App Service health check failing' + severity: 1 + enabled: true + scopes: [ webApp.id ] + evaluationFrequency: 'PT1M' + windowSize: 'PT5M' + criteria: { + 'odata.type': 'Microsoft.Azure.Monitor.SingleResourceMultipleMetricCriteria' + allOf: [ + { + name: 'HealthCheckFailure' + metricName: 'HealthCheckStatus' + metricNamespace: 'Microsoft.Web/sites' + operator: 'LessThan' + threshold: 100 + timeAggregation: 'Average' + criterionType: 'StaticThresholdCriterion' + } + ] + } + actions: !empty(alertEmail) ? [ { actionGroupId: actionGroup.id } ] : [] + } +} + +// ── 9. Azure Portal Dashboard ─────────────────────────────── + +resource dashboard 'Microsoft.Portal/dashboards@2020-09-01-preview' = { + name: dashboardName + location: location + tags: { + 'hidden-title': 'Zava Operations Dashboard' + } + properties: { + lenses: [ + { + order: 0 + parts: [ + { + position: { x: 0, y: 0, colSpan: 16, rowSpan: 2 } + metadata: { + type: 'Extension/HubsExtension/PartType/MarkdownPart' + inputs: [] + settings: { + content: { + content: '## Zava Operations Dashboard\n**Real-time monitoring** for SQL Database, App Service, and Application Insights.\n\n_Resource Group:_ `${resourceGroup().name}` | _Region:_ `${location}`' + title: 'Zava Operations Dashboard' + subtitle: 'Enterprise Monitoring' + markdownSource: 1 + } + } + } + } + { + position: { x: 0, y: 2, colSpan: 8, rowSpan: 4 } + metadata: { + type: 'Extension/HubsExtension/PartType/MonitorChartPart' + inputs: [ + { + name: 'options' + value: { + chart: { + metrics: [ + { + resourceMetadata: { id: sqlDatabase.id } + name: 'dtu_consumption_percent' + aggregationType: 4 + namespace: 'Microsoft.Sql/servers/databases' + metricVisualization: { displayName: 'DTU percentage' } + } + ] + title: 'SQL Database — DTU Usage' + visualization: { chartType: 2 } + } + } + } + ] + settings: {} + } + } + { + position: { x: 8, y: 2, colSpan: 8, rowSpan: 4 } + metadata: { + type: 'Extension/HubsExtension/PartType/MonitorChartPart' + inputs: [ + { + name: 'options' + value: { + chart: { + metrics: [ + { + resourceMetadata: { id: webApp.id } + name: 'HttpResponseTime' + aggregationType: 4 + namespace: 'Microsoft.Web/sites' + metricVisualization: { displayName: 'Response Time' } + } + ] + title: 'App Service — Response Time' + visualization: { chartType: 2 } + } + } + } + ] + settings: {} + } + } + { + position: { x: 0, y: 6, colSpan: 8, rowSpan: 4 } + metadata: { + type: 'Extension/HubsExtension/PartType/MonitorChartPart' + inputs: [ + { + name: 'options' + value: { + chart: { + metrics: [ + { + resourceMetadata: { id: webApp.id } + name: 'Http5xx' + aggregationType: 1 + namespace: 'Microsoft.Web/sites' + metricVisualization: { displayName: 'HTTP 5xx Errors' } + } + ] + title: 'App Service — HTTP 5xx Errors' + visualization: { chartType: 2 } + } + } + } + ] + settings: {} + } + } + { + position: { x: 8, y: 6, colSpan: 8, rowSpan: 4 } + metadata: { + type: 'Extension/HubsExtension/PartType/MonitorChartPart' + inputs: [ + { + name: 'options' + value: { + chart: { + metrics: [ + { + resourceMetadata: { id: webApp.id } + name: 'HealthCheckStatus' + aggregationType: 4 + namespace: 'Microsoft.Web/sites' + metricVisualization: { displayName: 'Health Check Status' } + } + ] + title: 'App Service — Health Check' + visualization: { chartType: 2 } + } + } + } + ] + settings: {} + } + } + ] + } + ] + } +} + +// ── 10. SRE Agent ─────────────────────────────────────────── + +module sreAgent 'modules/sre-agent.bicep' = { + name: 'sre-agent' + params: { + location: location + agentName: agentName + identityId: identity.outputs.identityId + identityPrincipalId: identity.outputs.identityPrincipalId + appInsightsAppId: monitoring.outputs.appInsightsAppId + appInsightsConnectionString: monitoring.outputs.appInsightsConnectionString + appInsightsId: monitoring.outputs.appInsightsId + managedResourceGroupId: resourceGroup().id + } +} + +// ── Outputs ───────────────────────────────────────────────── + +output sqlServerFqdn string = sqlServer.properties.fullyQualifiedDomainName +output sqlDatabaseName string = sqlDatabaseName +output appUrl string = 'https://${webApp.properties.defaultHostName}' +output appName string = webApp.name +output webAppName string = webApp.name +output webAppPrincipalId string = webApp.identity.principalId +output appInsightsConnectionString string = monitoring.outputs.appInsightsConnectionString +output identityPrincipalId string = identity.outputs.identityPrincipalId +output agentName string = sreAgent.outputs.agentName +output agentEndpoint string = sreAgent.outputs.agentEndpoint +output agentPortalUrl string = sreAgent.outputs.agentPortalUrl diff --git a/labs/zava-cafe/infra/seed-database.sql b/labs/zava-cafe/infra/seed-database.sql new file mode 100644 index 000000000..acb2a4c38 --- /dev/null +++ b/labs/zava-cafe/infra/seed-database.sql @@ -0,0 +1,130 @@ +-- ────────────────────────────────────────────────────────────── +-- Zava — Seed Database +-- Creates tables and inserts demo data for the SRE Agent lab +-- ────────────────────────────────────────────────────────────── + +-- ── Products ──────────────────────────────────────────────── + +IF NOT EXISTS (SELECT * FROM sys.tables WHERE name = 'Products') +BEGIN + CREATE TABLE Products ( + Id INT IDENTITY(1,1) PRIMARY KEY, + Name NVARCHAR(200) NOT NULL, + Price DECIMAL(10,2) NOT NULL, + Category NVARCHAR(100) NOT NULL, + CreatedAt DATETIME2 DEFAULT GETUTCDATE() + ); +END; +GO + +-- ── Orders ────────────────────────────────────────────────── + +IF NOT EXISTS (SELECT * FROM sys.tables WHERE name = 'Orders') +BEGIN + CREATE TABLE Orders ( + Id INT IDENTITY(1,1) PRIMARY KEY, + CustomerName NVARCHAR(200) NOT NULL, + CustomerEmail NVARCHAR(200) NOT NULL, + OrderDate DATETIME2 DEFAULT GETUTCDATE(), + Status NVARCHAR(50) DEFAULT 'Pending', + TotalAmount DECIMAL(10,2) NOT NULL + ); +END; +GO + +-- ── OrderItems ────────────────────────────────────────────── + +IF NOT EXISTS (SELECT * FROM sys.tables WHERE name = 'OrderItems') +BEGIN + CREATE TABLE OrderItems ( + Id INT IDENTITY(1,1) PRIMARY KEY, + OrderId INT NOT NULL, + ProductId INT NOT NULL, + Quantity INT NOT NULL DEFAULT 1, + UnitPrice DECIMAL(10,2) NOT NULL, + CONSTRAINT FK_OrderItems_Orders FOREIGN KEY (OrderId) REFERENCES Orders(Id), + CONSTRAINT FK_OrderItems_Products FOREIGN KEY (ProductId) REFERENCES Products(Id) + ); +END; +GO + +-- ── Seed Products ─────────────────────────────────────────── + +IF NOT EXISTS (SELECT 1 FROM Products) +BEGIN + INSERT INTO Products (Name, Price, Category) VALUES + -- Espresso + ('Zava Café Doppio Espresso', 3.50, 'Espresso'), + ('Zava Café Cortado', 4.25, 'Espresso'), + ('Zava Café Americano', 3.75, 'Espresso'), + ('Zava Café Macchiato', 4.00, 'Espresso'), + ('Zava Café Ristretto', 3.25, 'Espresso'), + -- Brewed Coffee + ('Zava Café Single-Origin Pour-Over', 5.50, 'Brewed Coffee'), + ('Zava Café Cold Brew', 5.00, 'Brewed Coffee'), + ('Zava Café Nitro Cold Brew', 6.00, 'Brewed Coffee'), + -- Pastries + ('Zava Café Almond Croissant', 4.75, 'Pastries'), + ('Zava Café Pain au Chocolat', 4.50, 'Pastries'), + ('Zava Café Blueberry Scone', 3.95, 'Pastries'), + ('Zava Café Lemon Loaf Slice', 4.25, 'Pastries'), + ('Zava Café Cinnamon Roll', 4.95, 'Pastries'), + ('Zava Café Morning Bun', 3.75, 'Pastries'), + -- Merch + ('Zava Café 12oz Ceramic Mug', 14.99, 'Merch'), + ('Zava Café Reusable Tumbler', 22.50, 'Merch'), + ('Zava Café Whole-Bean Bag (340g)', 18.00, 'Merch'), + ('Zava Café Barista Apron', 38.00, 'Merch'), + ('Zava Café Pour-Over Filters (50ct)', 8.50, 'Merch'), + ('Zava Café Espresso Tamper', 24.00, 'Merch'); +END; +GO + +-- ── Seed Orders ───────────────────────────────────────────── + +IF NOT EXISTS (SELECT 1 FROM Orders) +BEGIN + INSERT INTO Orders (CustomerName, CustomerEmail, OrderDate, Status, TotalAmount) VALUES + ('Alice Johnson', 'alice@example.com', '2025-01-15', 'Completed', 7.75), + ('Bob Smith', 'bob@example.com', '2025-01-18', 'Completed', 4.25), + ('Carol Williams', 'carol@example.com', '2025-02-01', 'Shipped', 16.95), + ('David Brown', 'david@example.com', '2025-02-10', 'Pending', 3.50), + ('Eve Martinez', 'eve@example.com', '2025-02-14', 'Completed', 46.25), + ('Frank Lee', 'frank@example.com', '2025-03-01', 'Shipped', 45.99), + ('Grace Kim', 'grace@example.com', '2025-03-05', 'Pending', 9.20), + ('Hank Wilson', 'hank@example.com', '2025-03-12', 'Completed', 4.00), + ('Ivy Chen', 'ivy@example.com', '2025-03-20', 'Shipped', 13.25), + ('Jack Davis', 'jack@example.com', '2025-04-01', 'Pending', 42.95); + + INSERT INTO OrderItems (OrderId, ProductId, Quantity, UnitPrice) VALUES + (1, 1, 1, 3.50), -- Alice: Doppio Espresso + (1, 2, 1, 4.25), -- Alice: Cortado + (2, 2, 1, 4.25), -- Bob: Cortado + (3, 9, 2, 4.75), -- Carol: 2x Almond Croissant + (3, 11, 1, 3.95), -- Carol: Blueberry Scone + (3, 1, 1, 3.50), -- Carol: Doppio Espresso + (4, 1, 1, 3.50), -- David: Doppio Espresso + (5, 18, 1, 38.00), -- Eve: Barista Apron + (5, 10, 1, 4.50), -- Eve: Pain au Chocolat + (5, 14, 1, 3.75), -- Eve: Morning Bun + (6, 15, 1, 14.99), -- Frank: Ceramic Mug + (6, 16, 1, 22.50), -- Frank: Reusable Tumbler + (6, 19, 1, 8.50), -- Frank: Pour-Over Filters + (7, 2, 1, 4.25), -- Grace: Cortado + (7, 13, 1, 4.95), -- Grace: Cinnamon Roll + (8, 4, 1, 4.00), -- Hank: Macchiato + (9, 12, 2, 4.25), -- Ivy: 2x Lemon Loaf Slice + (9, 9, 1, 4.75), -- Ivy: Almond Croissant + (10, 18, 1, 38.00), -- Jack: Barista Apron + (10, 13, 1, 4.95); -- Jack: Cinnamon Roll +END; +GO + +-- ── Verify ────────────────────────────────────────────────── + +SELECT 'Products' AS [Table], COUNT(*) AS [Rows] FROM Products +UNION ALL +SELECT 'Orders', COUNT(*) FROM Orders +UNION ALL +SELECT 'OrderItems', COUNT(*) FROM OrderItems; +GO diff --git a/labs/zava-cafe/lab.yaml b/labs/zava-cafe/lab.yaml new file mode 100644 index 000000000..96f08142f --- /dev/null +++ b/labs/zava-cafe/lab.yaml @@ -0,0 +1,30 @@ +api: 1 +name: zava-cafe +displayName: Zava — Zava Café SRE Agent Lab +description: Realistic e-commerce platform (Zava) — break SQL/web on purpose, watch SRE Agent investigate and remediate. +tags: + - intermediate + - sql + - aspnet + +prereqs: + - az + - azd + - sqlcmd + +prompts: [] + +scenarios: + - id: dtu-spike + label: Spike SQL DTU + description: Inject heavy queries to spike SQL DTU >80% — triggers Azure Monitor alert that wakes the agent. + runner: sre-config/simulate-dtu-spike.ps1 + minutes: 5 + needs: [] + + - id: slow-queries + label: Generate slow queries + description: Run a flood of slow queries — agent diagnoses + recommends index. + runner: sre-config/simulate-slow-queries.ps1 + minutes: 5 + needs: [] diff --git a/labs/zava-cafe/scripts/invoke-thread.sh b/labs/zava-cafe/scripts/invoke-thread.sh new file mode 100644 index 000000000..6a0fa48bb --- /dev/null +++ b/labs/zava-cafe/scripts/invoke-thread.sh @@ -0,0 +1,44 @@ +#!/bin/bash +# ============================================================================= +# invoke-thread.sh — fire a smoke-test thread at sql-performance-investigator +# ============================================================================= +set -euo pipefail + +if ! command -v srectl >/dev/null 2>&1; then + echo "✗ srectl not on PATH — install via aka.ms/sreagent-onboarding" + exit 2 +fi + +AGENT_ENDPOINT=$(azd env get-value SRE_AGENT_ENDPOINT 2>/dev/null || echo "") +if [ -z "$AGENT_ENDPOINT" ]; then + echo "✗ SRE_AGENT_ENDPOINT not set in azd env — run azd up first" + exit 1 +fi + +PROMPT="${1:-Use the sql-query-diagnosis skill to inspect the top 5 slowest queries on the Zava SQL DB and summarize what you find in 3 bullets.}" +AGENT="${AGENT:-sql-performance-investigator}" + +echo "" +echo "═══ Invoking $AGENT thread ═══" +echo " endpoint: $AGENT_ENDPOINT" +echo " prompt: $PROMPT" +echo "" + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +WS_DIR="$SCRIPT_DIR/../sre-config/agent1" +( cd "$WS_DIR" && srectl init --resource-url "$AGENT_ENDPOINT" >/dev/null 2>&1 || true ) + +OUTPUT=$(cd "$WS_DIR" && srectl thread new --agent "$AGENT" --message "$PROMPT" --no-wait 2>&1) +echo "$OUTPUT" +echo "" + +THREAD_URL=$(echo "$OUTPUT" | grep -oE 'https://sre\.azure\.com/[^ ]+' | head -1 || true) +THREAD_ID=$(echo "$OUTPUT" | grep -oE 'Thread ID: [a-f0-9-]+' | awk '{print $3}' | head -1 || true) + +if [ -n "$THREAD_URL" ]; then + echo "✓ Thread URL: $THREAD_URL" +elif [ -n "$THREAD_ID" ]; then + echo "✓ Thread ID: $THREAD_ID — open https://sre.azure.com to follow" +else + echo "⚠ Could not auto-detect thread URL — check output above." +fi diff --git a/labs/zava-cafe/scripts/post-provision.sh b/labs/zava-cafe/scripts/post-provision.sh new file mode 100644 index 000000000..32bbfb881 --- /dev/null +++ b/labs/zava-cafe/scripts/post-provision.sh @@ -0,0 +1,402 @@ +#!/bin/bash +# ============================================================================= +# post-provision.sh — Runs after `azd provision` succeeds. +# +# Steps: +# 1. Seed Azure SQL DB from infra/seed-database.sql (best-effort) +# 2. Deploy the .NET web app from source via `az webapp deploy` +# - .NET (src/) +# 3. Optional: register srectl resources (tools, agents, skills, hooks, +# scheduled tasks) under sre-config/agent1, and fire a smoke-test +# thread. +# 4. Print a summary + write labs/.deployed/zava-cafe.json +# ============================================================================= +set -uo pipefail + +if command -v python3 &>/dev/null; then PYTHON=python3; elif command -v python &>/dev/null; then PYTHON=python; else PYTHON=""; fi + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +PROJECT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" +cd "$PROJECT_DIR" + +TEMP_DIR="${SCRIPT_DIR}/.tmp" +mkdir -p "$TEMP_DIR" + +# Convert MSYS/Cygwin paths (/c/foo) to native form (C:/foo) for tools like +# Python and az CLI on Windows Git-Bash. No-op on macOS/Linux. +to_native () { + if command -v cygpath >/dev/null 2>&1; then cygpath -m "$1"; else echo "$1"; fi +} +NATIVE_TMP="$(to_native "$TEMP_DIR")" +NATIVE_PROJECT="$(to_native "$PROJECT_DIR")" + +echo "" +echo "=============================================" +echo " Zava — Zava Café Lab — Post-Provision" +echo "=============================================" + +# ── Read azd outputs ───────────────────────────────────────── +AZURE_RESOURCE_GROUP=$(azd env get-value AZURE_RESOURCE_GROUP 2>/dev/null || echo "") +AZURE_LOCATION=$(azd env get-value AZURE_LOCATION 2>/dev/null || echo "") +SRE_AGENT_ENDPOINT=$(azd env get-value SRE_AGENT_ENDPOINT 2>/dev/null || echo "") +SRE_AGENT_NAME=$(azd env get-value SRE_AGENT_NAME 2>/dev/null || echo "") +AZURE_SQL_SERVER_FQDN=$(azd env get-value AZURE_SQL_SERVER_FQDN 2>/dev/null || echo "") +AZURE_SQL_DATABASE=$(azd env get-value AZURE_SQL_DATABASE 2>/dev/null || echo "") + +echo "" +echo " Resource Group: ${AZURE_RESOURCE_GROUP:-(unknown)}" +echo " SRE Agent endpoint: ${SRE_AGENT_ENDPOINT:-(unknown)}" +echo "" + +# Add deployer's public IP to SQL firewall once — used by seed + MI grant steps +if [ -n "$AZURE_SQL_SERVER_FQDN" ] && [ -n "$AZURE_RESOURCE_GROUP" ]; then + MYIP=$(curl -s --max-time 5 https://api.ipify.org || echo "") + if [ -n "$MYIP" ]; then + az sql server firewall-rule create \ + --resource-group "$AZURE_RESOURCE_GROUP" \ + --server "$(echo "$AZURE_SQL_SERVER_FQDN" | cut -d'.' -f1)" \ + --name "AllowDeployerIP" \ + --start-ip-address "$MYIP" --end-ip-address "$MYIP" \ + >/dev/null 2>&1 && echo " • Added deployer IP $MYIP to SQL firewall" + fi +fi + +# SQL_ACCESS_TOKEN is no longer used here — sql_entra.py acquires its own token +# via DefaultAzureCredential. (Kept the firewall rule above, which is still useful.) + +# ── Step 1/5: Seed SQL DB (Entra via pyodbc helper) ───────── +echo "🗄️ Step 1/5: Seeding SQL database..." +SEED_FILE="$PROJECT_DIR/infra/seed-database.sql" +if [ -z "$AZURE_SQL_SERVER_FQDN" ]; then + echo " ⏭️ Skipped — missing SQL server FQDN." +elif [ ! -f "$SEED_FILE" ]; then + echo " ⏭️ Skipped — $SEED_FILE not found." +elif [ -z "$PYTHON" ]; then + echo " ⏭️ Skipped — Python not on PATH." +else + set +e + "$PYTHON" "$SCRIPT_DIR/sql_entra.py" \ + --server "$AZURE_SQL_SERVER_FQDN" \ + --database "$AZURE_SQL_DATABASE" \ + --file "$(to_native "$SEED_FILE")" > "$TEMP_DIR/seed.log" 2>&1 + RC=$? + set -e + if [ $RC -eq 0 ]; then + echo " ✅ Seed completed." + elif [ $RC -eq 2 ]; then + echo " ⏭️ Skipped — pyodbc/azure-identity not installed." + echo " Install with: pip install pyodbc azure-identity" + sed 's/^/ /' "$TEMP_DIR/seed.log" | head -5 + else + echo " ⚠️ Seed failed (rc=$RC). Tail:"; tail -10 "$TEMP_DIR/seed.log" | sed 's/^/ /' + fi +fi + +# ── Step 1.5/5: Grant Web App MI access to SQL DB ─────────── +echo "" +echo "🔐 Step 1.5/5: Granting Web App MI access to SQL DB..." +WEBAPP_NAME="$(azd env get-value AZURE_APP_NAME 2>/dev/null || azd env get-value WEBAPP_NAME 2>/dev/null)" +SQL_FQDN="$(azd env get-value AZURE_SQL_SERVER_FQDN 2>/dev/null)" +SQL_DB="$(azd env get-value AZURE_SQL_DATABASE 2>/dev/null)" +if [ -z "$WEBAPP_NAME" ] || [ -z "$SQL_FQDN" ] || [ -z "$SQL_DB" ]; then + echo " ⚠️ Required env vars missing — skipping MI SQL grant" +elif [ -z "$PYTHON" ]; then + echo " ⚠️ Python not on PATH — skipping MI SQL grant" +else + GRANT_SQL="$TEMP_DIR/mi-grant.sql" + cat > "$GRANT_SQL" < "$TEMP_DIR/mi-grant.log" 2>&1 + RC=$? + set -e + if [ $RC -eq 0 ]; then + echo " ✓ MI SQL access granted to $WEBAPP_NAME" + elif [ $RC -eq 2 ]; then + echo " ⚠️ Skipped — pyodbc/azure-identity not installed." + echo " Install with: pip install pyodbc azure-identity" + else + echo " ⚠️ MI grant failed (rc=$RC). Tail:"; tail -10 "$TEMP_DIR/mi-grant.log" | sed 's/^/ /' + fi +fi + +# ── Step 2/5: Deploy the .NET web app from source ─────────── +echo "" +echo "🚀 Step 2/5: Deploying .NET web app from source..." + +deploy_zip () { + local app_name="$1" + local zip_path="$2" + local label="$3" + if [ -z "$app_name" ]; then echo " ⚠️ $label: app name missing — skipped"; return; fi + if [ ! -f "$zip_path" ]; then echo " ⚠️ $label: zip not found at $zip_path — skipped"; return; fi + echo " • $label → $app_name" + set +e + az webapp deploy \ + --resource-group "$AZURE_RESOURCE_GROUP" \ + --name "$app_name" \ + --src-path "$(to_native "$zip_path")" \ + --type zip \ + --output none 2>"$TEMP_DIR/deploy-$label.log" + RC=$? + set -e + [ $RC -eq 0 ] && echo " ✅ deploy queued" || { echo " ⚠️ deploy failed (rc=$RC):"; head -10 "$TEMP_DIR/deploy-$label.log" | sed 's/^/ /'; } +} + +# Helper: zip a directory's contents using Python's shutil.make_archive. +# Both arguments must be NATIVE paths (use to_native on the bash side). +make_zip () { + local out_base_native="$1" # e.g. C:/.../scripts/.tmp/dotnet-app (no .zip) + local src_native="$2" # e.g. C:/.../scripts/.tmp/dotnet-publish + if [ -z "$PYTHON" ]; then + echo " ⚠️ Python not on PATH — cannot create zip" + return 1 + fi + "$PYTHON" -c "import shutil,sys; shutil.make_archive(sys.argv[1],'zip',sys.argv[2])" \ + "$out_base_native" "$src_native" +} + +# 2a) .NET app — publish + zip +DOTNET_ZIP="$TEMP_DIR/dotnet-app.zip" +if [ -d "$PROJECT_DIR/src" ] && command -v dotnet &>/dev/null; then + echo " • Publishing .NET app..." + set +e + ( cd "$PROJECT_DIR/src" && dotnet publish -c Release -o "$TEMP_DIR/dotnet-publish" --nologo -v quiet ) > "$TEMP_DIR/dotnet-build.log" 2>&1 + RC=$? + set -e + if [ $RC -eq 0 ]; then + set +e + make_zip "${NATIVE_TMP}/dotnet-app" "${NATIVE_TMP}/dotnet-publish" > "$TEMP_DIR/zip-dotnet.log" 2>&1 + set -e + if [ ! -f "$DOTNET_ZIP" ]; then + echo " ⚠️ zip creation failed:"; sed 's/^/ /' "$TEMP_DIR/zip-dotnet.log" | head -10 + else + deploy_zip "$AZURE_APP_NAME" "$DOTNET_ZIP" "dotnet-app" + fi + else + echo " ⚠️ dotnet publish failed (rc=$RC). Tail:"; tail -10 "$TEMP_DIR/dotnet-build.log" | sed 's/^/ /' + fi +elif [ -d "$PROJECT_DIR/src" ]; then + echo " ⚠️ dotnet CLI not on PATH — skipping .NET app deploy." +fi + +# ── Step 3/5: srectl orchestration (optional) ─────────────── +echo "" +echo "🔧 Step 3/5: Registering SRE Agent resources via srectl..." +if [ "${LABS_SKIP_SRECTL:-0}" = "1" ]; then + echo " ⏭️ Skipped (LABS_SKIP_SRECTL=1)" +elif ! command -v srectl >/dev/null 2>&1; then + echo " ⏭️ Skipped — srectl not on PATH (private preview via aka.ms/sreagent-onboarding)" +elif [ -z "$SRE_AGENT_ENDPOINT" ]; then + echo " ⏭️ Skipped — SRE_AGENT_ENDPOINT not set" +else + set +e + srectl_apply_workspace () { + local workspace="$1" # e.g. sre-config/agent1 + local label="$2" + local ws_dir="$PROJECT_DIR/$workspace" + local slug; slug="$(echo "$workspace" | tr '/' '-')" # filesystem-safe log key + [ ! -d "$ws_dir" ] && { echo " ⏭️ $label: $workspace not found"; return; } + + echo " ── $label ($workspace) ──" + ( cd "$ws_dir" && srectl init --resource-url "$SRE_AGENT_ENDPOINT" ) > "$TEMP_DIR/srectl-init-$slug.log" 2>&1 + RC=$? + if [ $RC -ne 0 ]; then + echo " ⚠️ srectl init failed (rc=$RC):"; tail -10 "$TEMP_DIR/srectl-init-$slug.log" | sed 's/^/ /' + return + fi + + # Tools — apply-yaml each tools//.yaml + if [ -d "$ws_dir/tools" ]; then + for d in "$ws_dir/tools"/*/; do + [ -d "$d" ] || continue + n=$(basename "$d") + f="tools/$n/$n.yaml" + [ -f "$ws_dir/$f" ] || continue + ( cd "$ws_dir" && srectl apply-yaml -f "$f" ) > "$TEMP_DIR/srectl-tool-$n.log" 2>&1 + RC=$? + [ $RC -eq 0 ] && echo " ✅ tool: $n" || { echo " ⚠️ tool $n failed (rc=$RC):"; tail -5 "$TEMP_DIR/srectl-tool-$n.log" | sed 's/^/ /'; } + done + fi + + # Hooks — yaml files under hooks/ + if [ -d "$ws_dir/hooks" ]; then + for f in "$ws_dir/hooks"/*.yaml; do + [ -f "$f" ] || continue + rel="hooks/$(basename "$f")" + ( cd "$ws_dir" && srectl hook apply --file "$rel" ) > "$TEMP_DIR/srectl-hook-$(basename "$f").log" 2>&1 + RC=$? + [ $RC -eq 0 ] && echo " ✅ hook: $(basename "$f")" || { echo " ⚠️ hook $(basename "$f") failed (rc=$RC):"; tail -5 "$TEMP_DIR/srectl-hook-$(basename "$f").log" | sed 's/^/ /'; } + done + fi + + # Scheduled tasks — apply-yaml each scheduledtasks//.yaml + if [ -d "$ws_dir/scheduledtasks" ]; then + for d in "$ws_dir/scheduledtasks"/*/; do + [ -d "$d" ] || continue + n=$(basename "$d") + f="scheduledtasks/$n/$n.yaml" + [ -f "$ws_dir/$f" ] || continue + ( cd "$ws_dir" && srectl scheduledtask apply --file "$f" ) > "$TEMP_DIR/srectl-task-$n.log" 2>&1 + RC=$? + [ $RC -eq 0 ] && echo " ✅ scheduled task: $n" || { echo " ⚠️ task $n failed (rc=$RC):"; tail -5 "$TEMP_DIR/srectl-task-$n.log" | sed 's/^/ /'; } + done + fi + + # Skills — `srectl skill apply --name ` (workspace-aware) + if [ -d "$ws_dir/skills" ]; then + for d in "$ws_dir/skills"/*/; do + [ -d "$d" ] || continue + n=$(basename "$d") + [ -f "$ws_dir/skills/$n/SKILL.md" ] || continue + ( cd "$ws_dir" && srectl skill apply --name "$n" ) > "$TEMP_DIR/srectl-skill-$n.log" 2>&1 + RC=$? + [ $RC -eq 0 ] && echo " ✅ skill: $n" || { echo " ⚠️ skill $n failed (rc=$RC):"; tail -5 "$TEMP_DIR/srectl-skill-$n.log" | sed 's/^/ /'; } + done + fi + + # Agents — apply-yaml each agents//.yaml + if [ -d "$ws_dir/agents" ]; then + for d in "$ws_dir/agents"/*/; do + [ -d "$d" ] || continue + n=$(basename "$d") + f="agents/$n/$n.yaml" + [ -f "$ws_dir/$f" ] || continue + ( cd "$ws_dir" && srectl apply-yaml -f "$f" ) > "$TEMP_DIR/srectl-agent-$n.log" 2>&1 + RC=$? + [ $RC -eq 0 ] && echo " ✅ agent: $n" || { echo " ⚠️ agent $n failed (rc=$RC):"; tail -5 "$TEMP_DIR/srectl-agent-$n.log" | sed 's/^/ /'; } + done + fi + } + + srectl_apply_workspace "sre-config/agent1" "agent1 (SQL/DevOps)" + + # Smoke test — fire-and-forget thread on sql-performance-investigator + echo "" + echo " 🧵 Smoke test: srectl thread new --no-wait → sql-performance-investigator" + PROMPT="Run a quick health check on the Zava SQL DB. Reply with one bullet point." + ( cd "$PROJECT_DIR/sre-config/agent1" && srectl thread new --agent sql-performance-investigator --message "$PROMPT" --no-wait ) > "$TEMP_DIR/srectl-thread.log" 2>&1 + RC=$? + if [ $RC -eq 0 ]; then + THREAD_ID=$(grep -oE 'Thread ID: [a-f0-9-]+' "$TEMP_DIR/srectl-thread.log" | awk '{print $3}' | head -1) + echo " ✅ message sent. Thread ID: ${THREAD_ID:-(see log)}" + echo " Follow live: https://sre.azure.com" + else + echo " ⚠️ thread creation failed (rc=$RC):"; tail -10 "$TEMP_DIR/srectl-thread.log" | sed 's/^/ /' + fi + set -e +fi + +# ── Step 3.5/5: Register HTTP trigger for the simulator ───── +# Uses the SRE Agent REST API directly (no CLI in srectl 1.0.x yet). +# Helper: labs/_platform/http_trigger.py — idempotent (reuses existing by name). +ZAVA_HTTP_TRIGGER_URL="" +ZAVA_HTTP_TRIGGER_ID="" +echo "" +echo "🔔 Step 3.5/5: Registering HTTP trigger for simulator..." +if [ -z "$PYTHON" ]; then + echo " ⏭️ Skipped — python not on PATH" +elif [ -z "$SRE_AGENT_ENDPOINT" ]; then + echo " ⏭️ Skipped — SRE_AGENT_ENDPOINT not set" +elif ! command -v az >/dev/null 2>&1; then + echo " ⏭️ Skipped — az CLI not on PATH" +else + HT_HELPER="$(to_native "$LABS_PLATFORM_DIR/http_trigger.py")" + if [ ! -f "$LABS_PLATFORM_DIR/http_trigger.py" ]; then + # LABS_PLATFORM_DIR not defined yet — compute it inline (kept here so block is self-contained) + LABS_PLATFORM_DIR="$(cd "$SCRIPT_DIR/../../_platform" 2>/dev/null && pwd || echo "")" + HT_HELPER="$(to_native "$LABS_PLATFORM_DIR/http_trigger.py")" + fi + if [ ! -f "$LABS_PLATFORM_DIR/http_trigger.py" ]; then + echo " ⏭️ Skipped — labs/_platform/http_trigger.py not found" + else + set +e + HT_OUT=$("$PYTHON" "$HT_HELPER" create-and-enable \ + --endpoint "$SRE_AGENT_ENDPOINT" \ + --name "zava-cafe-incident-trigger" \ + --agent "sql-performance-investigator" \ + --mode "autonomous" \ + --description "Fired by the Zava Café lab simulator when it observes a bad deployment or SQL slowdown on the Zava app." \ + --prompt "An incoming alert payload from the Zava lab simulator. Investigate the SQL performance / health failure described in the request body, follow the runbook (sql-performance-investigator), and post a brief diagnosis + recommended remediation." \ + 2> "$TEMP_DIR/http-trigger-create.log") + RC=$? + set -e + if [ $RC -eq 0 ] && [ -n "$HT_OUT" ]; then + ZAVA_HTTP_TRIGGER_URL=$("$PYTHON" -c "import json,sys; d=json.loads(sys.stdin.read()); print(d.get('triggerUrl') or '')" <<< "$HT_OUT") + ZAVA_HTTP_TRIGGER_ID=$("$PYTHON" -c "import json,sys; d=json.loads(sys.stdin.read()); print(d.get('triggerId') or '')" <<< "$HT_OUT") + if [ -n "$ZAVA_HTTP_TRIGGER_URL" ]; then + azd env set ZAVA_HTTP_TRIGGER_URL "$ZAVA_HTTP_TRIGGER_URL" >/dev/null 2>&1 || true + azd env set ZAVA_HTTP_TRIGGER_ID "$ZAVA_HTTP_TRIGGER_ID" >/dev/null 2>&1 || true + echo " ✅ trigger registered: $ZAVA_HTTP_TRIGGER_ID" + echo " URL stored in azd env as ZAVA_HTTP_TRIGGER_URL" + else + echo " ⚠️ create returned no triggerUrl: $HT_OUT" + fi + else + echo " ⚠️ trigger registration failed (rc=$RC):" + sed 's/^/ /' "$TEMP_DIR/http-trigger-create.log" 2>/dev/null | head -20 + fi + fi +fi + +# ── Step 4/5: Summary + record deployment ─────────────────── +echo "" +echo "=============================================" +echo " ✅ Zava Zava Café Lab — Provision Done" +echo "=============================================" +echo "" +echo " 🤖 Agent Portal: https://sre.azure.com" +echo " 📡 Agent Endpoint: ${SRE_AGENT_ENDPOINT:-not set}" +echo " 🔔 HTTP Trigger: ${ZAVA_HTTP_TRIGGER_URL:-not registered}" +echo " 🌐 Zava App: ${AZURE_APP_URL:-not deployed}" +echo " 🗄️ SQL Server: ${AZURE_SQL_SERVER_FQDN:-not deployed}" +echo " 📦 Resource Group: ${AZURE_RESOURCE_GROUP:-not set}" +echo "" +echo " Next:" +echo " • Visit the agent portal: https://sre.azure.com" +echo " • Drive a scenario: pwsh sre-config/simulate-dtu-spike.ps1" +echo " • Or: pwsh sre-config/simulate-slow-queries.ps1" +echo " • Manual smoke: bash scripts/invoke-thread.sh" +echo "=============================================" +echo "" + +# Write .deployed/zava-cafe.json (consumed by labs/.../lab.ps1 + meta-sim) +LABS_ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +DEPLOYED_DIR="$LABS_ROOT/.deployed" +mkdir -p "$DEPLOYED_DIR" +SUB_ID="$(az account show --query id -o tsv 2>/dev/null || echo '')" +cat > "$DEPLOYED_DIR/zava-cafe.json" </dev/null || true diff --git a/labs/zava-cafe/scripts/prereqs.sh b/labs/zava-cafe/scripts/prereqs.sh new file mode 100644 index 000000000..24f43263b --- /dev/null +++ b/labs/zava-cafe/scripts/prereqs.sh @@ -0,0 +1,127 @@ +#!/bin/bash +# ============================================================================= +# prereqs.sh — Prerequisites + interactive prompts for Zava Zava Café Lab +# Runs as the azd preprovision hook. +# ============================================================================= + +echo "" +echo "=============================================" +echo " Zava — Zava Café Lab — Prereqs Check" +echo "=============================================" +echo "" + +MISSING=0 + +if [[ "$OSTYPE" == "msys" || "$OSTYPE" == "mingw"* || "$OSTYPE" == "cygwin" ]]; then + OS="windows" +elif [[ "$OSTYPE" == "darwin"* ]]; then + OS="mac" +else + OS="linux" +fi +echo "Platform: $OS" +echo "" + +check_tool() { + local name="$1" + local cmd="$2" + local install_mac="$3" + local install_win="$4" + + if command -v "$cmd" &>/dev/null; then + version=$($cmd --version 2>&1 | head -1) + echo " ✅ $name: $version" + else + echo " ❌ $name: NOT FOUND" + if [ "$OS" = "mac" ]; then + echo " Install: $install_mac" + else + echo " Install: $install_win" + fi + MISSING=$((MISSING + 1)) + fi +} + +echo "Checking tools:" +check_tool "Azure CLI" "az" "brew install azure-cli" "winget install Microsoft.AzureCLI" +check_tool "Azure Developer CLI" "azd" "brew install azd" "winget install Microsoft.Azd" +check_tool "Git" "git" "brew install git" "winget install Git.Git" + +# Python (used by the post-provision script for shaping JSON, etc.) +PYTHON_CMD="" +if command -v python3 &>/dev/null; then + echo " ✅ Python: $(python3 --version 2>&1)" + PYTHON_CMD=python3 +elif command -v python &>/dev/null; then + v=$(python --version 2>&1) + if echo "$v" | grep -q "Python 3"; then + echo " ✅ Python: $v" + PYTHON_CMD=python + else + echo " ❌ Python: $v — need Python 3.10+" + MISSING=$((MISSING + 1)) + fi +else + echo " ❌ Python: NOT FOUND" + MISSING=$((MISSING + 1)) +fi + +# sqlcmd — only used to seed the DB; warn if missing but don't hard-fail +if command -v sqlcmd &>/dev/null; then + echo " ✅ sqlcmd: $(sqlcmd -? 2>&1 | head -1 | tr -d '\r')" +else + echo " ⚠️ sqlcmd: NOT FOUND — DB seeding will be skipped." + echo " Install: winget install Microsoft.Sqlcmd (or use ODBC sqlcmd from SQL Server Tools)" +fi + +echo "" + +# ── Entra admin — must be set so Bicep can configure SQL Server AAD-only ──── +echo "Checking Entra admin (will be SQL Server admin)..." +AAD_LOGIN="$(az ad signed-in-user show --query userPrincipalName -o tsv 2>/dev/null | tr -d '\r')" +AAD_OID="$(az ad signed-in-user show --query id -o tsv 2>/dev/null | tr -d '\r')" +if [ -z "$AAD_LOGIN" ] || [ -z "$AAD_OID" ]; then + echo " ✗ Could not query az signed-in user. Run 'az login' first." + exit 1 +fi +echo " ✓ Will use $AAD_LOGIN as SQL Entra admin" +azd env set AAD_ADMIN_LOGIN "$AAD_LOGIN" 2>/dev/null || true +azd env set AAD_ADMIN_OBJECT_ID "$AAD_OID" 2>/dev/null || true + +# ── Python deps for sql_entra.py (pyodbc + azure-identity) ────────────────── +if [ -n "$PYTHON_CMD" ]; then + echo "" + echo "Checking Python deps for SQL helper (pyodbc, azure-identity):" + if $PYTHON_CMD -c "import pyodbc, azure.identity" 2>/dev/null; then + echo " ✅ pyodbc + azure-identity already installed" + else + echo " ⚠️ Installing pyodbc + azure-identity (best-effort)..." + if $PYTHON_CMD -m pip install --quiet --disable-pip-version-check pyodbc azure-identity 2>/dev/null; then + echo " ✅ pyodbc + azure-identity installed" + else + echo " ⚠️ pip install failed — SQL seed/grant steps will be skipped." + echo " Manually: $PYTHON_CMD -m pip install pyodbc azure-identity" + fi + fi +fi + +# ── Azure auth (informational) ─────────────────────────────────────────────── +echo "" +echo "Checking Azure auth:" +if az account show &>/dev/null 2>&1; then + sub=$(az account show --query name -o tsv 2>/dev/null) + echo " ✅ Logged in: $sub" +else + echo " ℹ️ Not logged in yet — run 'az login' before 'azd up'" +fi + +echo "" +echo "=============================================" +if [ "$MISSING" -eq 0 ]; then + echo " ✅ Prerequisites met! Proceeding with azd provision..." +else + echo " ❌ $MISSING required tool(s) missing — fix above then re-run" + exit 1 +fi +echo "=============================================" +echo "" diff --git a/labs/zava-cafe/scripts/sql_entra.py b/labs/zava-cafe/scripts/sql_entra.py new file mode 100644 index 000000000..65967d476 --- /dev/null +++ b/labs/zava-cafe/scripts/sql_entra.py @@ -0,0 +1,105 @@ +#!/usr/bin/env python3 +""" +sql_entra.py — Run T-SQL against Azure SQL using an Entra (AAD) access token. + +Portable replacement for `sqlcmd -G --access-token …` on systems where the old +Microsoft sqlcmd (Windows 15.x) doesn't support --access-token. + +Requires: pyodbc, azure-identity. If either is missing, prints install hints +and exits with code 2 (so the caller can treat it as a soft skip). +""" +from __future__ import annotations + +import argparse +import re +import struct +import sys +from pathlib import Path + + +def _import_deps(): + try: + import pyodbc # type: ignore + from azure.identity import DefaultAzureCredential # type: ignore + return pyodbc, DefaultAzureCredential + except ImportError as e: + sys.stderr.write( + f"sql_entra.py: missing dependency ({e}). Install with:\n" + f" pip install pyodbc azure-identity\n" + ) + sys.exit(2) + + +def _pick_driver(pyodbc) -> str: + drivers = [d for d in pyodbc.drivers() if "ODBC Driver" in d and "SQL Server" in d] + if not drivers: + sys.stderr.write( + "sql_entra.py: no 'ODBC Driver for SQL Server' found.\n" + " Windows: winget install Microsoft.MsOdbcSql\n" + " macOS: brew install msodbcsql18\n" + " Linux: https://learn.microsoft.com/sql/connect/odbc/linux-mac/\n" + ) + sys.exit(2) + drivers.sort(reverse=True) + return drivers[0] + + +def _split_batches(sql: str) -> list[str]: + """Split a T-SQL script on GO batch separators (case-insensitive, line-anchored).""" + parts = re.split(r"(?im)^\s*GO\s*;?\s*$", sql) + return [p.strip() for p in parts if p.strip()] + + +def main() -> int: + p = argparse.ArgumentParser(description="Run T-SQL on Azure SQL via Entra access token.") + p.add_argument("--server", required=True, help="SQL Server FQDN (e.g. sql-foo.database.windows.net)") + p.add_argument("--database", required=True, help="Database name") + src = p.add_mutually_exclusive_group(required=True) + src.add_argument("--file", help="Path to .sql file to execute") + src.add_argument("--query", help="Inline T-SQL to execute") + p.add_argument("--timeout", type=int, default=60, help="Connection timeout (seconds)") + args = p.parse_args() + + pyodbc, DefaultAzureCredential = _import_deps() + driver = _pick_driver(pyodbc) + + if args.file: + sql_text = Path(args.file).read_text(encoding="utf-8-sig") + else: + sql_text = args.query or "" + + cred = DefaultAzureCredential(exclude_interactive_browser_credential=False) + token = cred.get_token("https://database.windows.net/.default").token.encode("utf-16-le") + token_struct = struct.pack(f"=i{len(token)}s", len(token), token) + SQL_COPT_SS_ACCESS_TOKEN = 1256 + + conn_str = ( + f"Driver={{{driver}}};" + f"Server=tcp:{args.server},1433;" + f"Database={args.database};" + f"Encrypt=yes;TrustServerCertificate=no;" + f"Connection Timeout={args.timeout};" + ) + + try: + with pyodbc.connect(conn_str, attrs_before={SQL_COPT_SS_ACCESS_TOKEN: token_struct}) as conn: + conn.autocommit = True + cur = conn.cursor() + batches = _split_batches(sql_text) + for i, batch in enumerate(batches, 1): + try: + cur.execute(batch) + while cur.nextset(): + pass + except pyodbc.Error as e: + sys.stderr.write(f"sql_entra.py: batch {i} failed: {e}\n") + return 1 + print(f"sql_entra.py: executed {len(batches)} batch(es) on {args.server}/{args.database}") + return 0 + except pyodbc.Error as e: + sys.stderr.write(f"sql_entra.py: connection failed: {e}\n") + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/labs/zava-cafe/simulator/demo.py b/labs/zava-cafe/simulator/demo.py new file mode 100644 index 000000000..85f5802b6 --- /dev/null +++ b/labs/zava-cafe/simulator/demo.py @@ -0,0 +1,1635 @@ +#!/usr/bin/env python3 +""" +╔══════════════════════════════════════════════════════════════╗ +║ ZAVA DEMO SIMULATOR — Zava Café SRE Agent Lab ║ +╚══════════════════════════════════════════════════════════════╝ + +A beautiful CLI simulator for demonstrating Azure SRE Agent +capabilities during the Zava Café recording. + +Scenarios: + 1. Slow Query (Missing Index) — Performance degradation + 2. Blocking Chain — Transaction blocking + 3. Bad Deployment — App health failure + 4. ServiceNow Integration — Incident management + 5. Reset All — Clean up demo environment + +Usage: + python simulator/demo.py +""" + +import sys +import os +import time +import json +import threading +import random +import subprocess +from datetime import datetime + +# ── Auto-install dependencies ─────────────────────────────── +def _ensure_deps(): + missing = [] + for pkg in ("rich", "requests", "pymssql"): + try: + __import__(pkg) + except ImportError: + missing.append(pkg) + if missing: + print(f"Installing: {', '.join(missing)} ...") + os.system(f'"{sys.executable}" -m pip install {" ".join(missing)} --quiet') + print("Done.\n") + +_ensure_deps() + +from rich.console import Console +from rich.panel import Panel +from rich.table import Table +from rich.live import Live +from rich.text import Text +from rich.align import Align +from rich import box +import requests as req + +try: + import pymssql + HAS_PYMSSQL = True +except ImportError: + HAS_PYMSSQL = False + +if sys.platform == "win32": + import msvcrt + +# ── Config (override with env vars) ──────────────────────── +SQL_SERVER = os.environ.get("ZAVA_SQL_SERVER", "sql-zava.database.windows.net") +SQL_DATABASE = os.environ.get("ZAVA_SQL_DATABASE", "sqldb-zava") +SQL_USER = os.environ.get("ZAVA_SQL_USER", "") +SQL_PASSWORD = os.environ.get("ZAVA_SQL_PASSWORD", "") + +APP_URL = os.environ.get("ZAVA_APP_URL", "https://app-zava.azurewebsites.net") +HEALTH_URL = f"{APP_URL}/health" + +# SRE Agent HTTP trigger URL — populated by `azd hooks run postprovision` +# (see labs/zava-cafe/scripts/post-provision.sh → Step 3.5). +# To override / set manually: `azd env set ZAVA_HTTP_TRIGGER_URL `. +ZAVA_HTTP_TRIGGER_URL = os.environ.get("ZAVA_HTTP_TRIGGER_URL", "") +SRE_AGENT_ENDPOINT = os.environ.get("SRE_AGENT_ENDPOINT", "") +SRE_AGENT_THREAD_BASE = os.environ.get("SRE_AGENT_THREAD_BASE", "https://sre.azure.com/threads") + +console = Console() + +# ── Helpers ───────────────────────────────────────────────── + +def check_key(): + """Non-blocking keypress check (Windows).""" + if sys.platform != "win32": + return None + if msvcrt.kbhit(): + ch = msvcrt.getch() + if ch in (b"\x00", b"\xe0"): + msvcrt.getch() # consume second byte of special key + return None + try: + return ch.decode("utf-8").lower() + except Exception: + return None + return None + + +def get_sql_connection(login_timeout=10): + """Return a pymssql connection or None.""" + if not HAS_PYMSSQL: + console.print("[red]pymssql not installed. Run: pip install pymssql[/]") + return None + try: + return pymssql.connect( + server=SQL_SERVER, + user=SQL_USER, + password=SQL_PASSWORD, + database=SQL_DATABASE, + login_timeout=login_timeout, + timeout=120, + ) + except Exception as e: + console.print(f"[red]SQL Connection Error:[/] {e}") + return None + + +def _color(ms): + if ms < 100: + return "green" + if ms < 500: + return "yellow" + return "red" + + +def _bar(ms, max_ms=2000, width=30): + filled = min(int((ms / max(max_ms, 1)) * width), width) + c = _color(ms) + return f"[{c}]{'█' * filled}{'░' * (width - filled)}[/]" + + +def _status(ms): + if ms < 100: + return "[green bold]⚡ FAST[/]" + + +def _check_alert_fired(since_time=None): + """Check DTU alert status. Returns (condition, start_time) or (None, None).""" + try: + import subprocess + sub = os.environ.get("ZAVA_SUBSCRIPTION_ID", "") + result = subprocess.run( + f'az rest --method GET --url "https://management.azure.com/subscriptions/{sub}/providers/Microsoft.AlertsManagement/alerts?api-version=2019-03-01&targetResourceGroup=rg-zava" -o json', + capture_output=True, text=True, timeout=20, shell=True + ) + if result.returncode == 0 and result.stdout.strip(): + data = json.loads(result.stdout) + for alert in data.get("value", []): + props = alert.get("properties", {}).get("essentials", {}) + rule = props.get("alertRule", "") + condition = props.get("monitorCondition", "") + start_str = props.get("startDateTime", "") + if "alert-zava-dtu-high" not in rule: + continue + if condition not in ("Fired", "Resolved"): + continue + if since_time and start_str: + try: + clean = start_str.split("+")[0].rstrip("Z") + if "." in clean: + parts = clean.split(".") + clean = parts[0] + "." + parts[1][:6] + for fmt in ("%Y-%m-%dT%H:%M:%S.%f", "%Y-%m-%dT%H:%M:%S", "%m/%d/%Y %H:%M:%S"): + try: + alert_time = datetime.strptime(clean, fmt) + break + except ValueError: + continue + else: + continue + if alert_time < since_time: + continue + except Exception: + continue + return condition, start_str + return None, None + except Exception: + return None, None + + +class EventTimeline: + """Tracks key events with timestamps for display.""" + def __init__(self): + self.events = [] + self.start_time = datetime.now() + + def add(self, event, style="white"): + self.events.append({ + "ts": datetime.now().strftime("%H:%M:%S"), + "elapsed": f"+{(datetime.now() - self.start_time).seconds}s", + "event": event, + "style": style, + }) + + def to_table(self): + t = Table( + title="[bold]Event Timeline[/]", + box=box.ROUNDED, border_style="blue", show_lines=False, + width=74, + ) + t.add_column("Time", style="dim", width=10) + t.add_column("Elapsed", style="dim", width=8) + t.add_column("Event", width=50) + for e in self.events[-5:]: + t.add_row(e["ts"], e["elapsed"], f"[{e['style']}]{e['event']}[/]") + return t + + +class PerfGraph: + """Rolling ASCII performance graph showing query durations over time.""" + + GRAPH_WIDTH = 50 + GRAPH_HEIGHT = 8 + BLOCKS = " ▁▂▃▄▅▆▇█" + + def __init__(self): + self.all_durations = [] # (timestamp, ms, had_index) + self.index_created_at = None + + def add(self, ms, has_index=False): + self.all_durations.append((datetime.now(), ms, has_index)) + if has_index and self.index_created_at is None: + self.index_created_at = len(self.all_durations) - 1 + + def to_panel(self): + if len(self.all_durations) < 2: + return Panel("[dim]Collecting data...[/]", title="[bold]Performance Graph[/]", border_style="magenta", width=76) + + # Only show the snapshot graph AFTER index is created + if self.index_created_at is None: + # Show a simple live indicator instead + recent = [s[1] for s in self.all_durations[-30:]] + avg = sum(recent) / len(recent) + sparkline = "" + for ms in self.all_durations[-50:]: + m = ms[1] + if m > 1000: sparkline += "[red]█[/]" + elif m > 500: sparkline += "[yellow]▆[/]" + elif m > 200: sparkline += "[yellow]▃[/]" + else: sparkline += "[green]▁[/]" + return Panel( + f" Live: {sparkline}\n Avg: [{_color(avg)}]{avg:.0f}ms[/] | Samples: {len(self.all_durations)} | Waiting for SRE Agent to fix...", + title="[bold magenta]📊 Live Performance[/]", + border_style="magenta", width=76, + ) + + # === SNAPSHOT: 1 min before and after the fix === + fix_time = self.all_durations[self.index_created_at][0] + before = [(t, ms, idx) for t, ms, idx in self.all_durations + if (fix_time - t).total_seconds() <= 60 and (fix_time - t).total_seconds() >= 0 and not idx][-20:] + after = [(t, ms, idx) for t, ms, idx in self.all_durations + if (t - fix_time).total_seconds() >= 0 and (t - fix_time).total_seconds() <= 60 and idx][:20] + samples = before + after + + if not samples: + return Panel("[dim]Building snapshot...[/]", title="[bold]Performance Graph[/]", border_style="magenta", width=76) + + durations = [s[1] for s in samples] + has_idx = [s[2] for s in samples] + max_ms = max(max(durations), 100) + if max_ms > 2000: max_ms = ((int(max_ms) // 500) + 1) * 500 + elif max_ms > 500: max_ms = ((int(max_ms) // 200) + 1) * 200 + else: max_ms = ((int(max_ms) // 100) + 1) * 100 + + before_avg = sum(s[1] for s in before) / max(len(before), 1) + after_avg = sum(s[1] for s in after) / max(len(after), 1) + improvement = ((before_avg - after_avg) / max(before_avg, 1)) * 100 + + lines = [] + for row in range(self.GRAPH_HEIGHT, 0, -1): + threshold = (row / self.GRAPH_HEIGHT) * max_ms + label = f"{int(threshold):>5}ms │" + bar = "" + for i, ms in enumerate(durations): + if ms >= threshold: + bar += "[red]█[/]" if not has_idx[i] else "[green]█[/]" + else: + lower = ((row - 1) / self.GRAPH_HEIGHT) * max_ms + if ms > lower: + frac = (ms - lower) / (threshold - lower) + bi = min(int(frac * (len(self.BLOCKS) - 1)), len(self.BLOCKS) - 1) + char = self.BLOCKS[bi] + bar += f"[red]{char}[/]" if not has_idx[i] else f"[green]{char}[/]" + else: + bar += " " + lines.append(f"{label}{bar}") + + lines.append(f" 0ms │{'─' * len(durations)}") + fix_offset = len(before) + pointer = " " * 8 + " " * fix_offset + "[green bold]▼ SRE Agent fixed it here[/]" + stats = f"\n [red]██ BEFORE[/] avg: [red bold]{before_avg:.0f}ms[/] [green]██ AFTER[/] avg: [green bold]{after_avg:.0f}ms[/] [cyan bold]⚡ {improvement:.0f}% faster[/]" + + return Panel( + "\n".join(lines) + f"\n{pointer}" + stats, + title="[bold magenta]📊 Before / After — SRE Agent Fix[/]", + border_style="green", width=76, + ) + fix_time = self.all_durations[self.index_created_at][0] + + # Get samples from 60s before fix + before = [(t, ms, idx) for t, ms, idx in self.all_durations + if (fix_time - t).total_seconds() <= 60 and (fix_time - t).total_seconds() >= 0 and not idx] + # Get samples from 60s after fix + after = [(t, ms, idx) for t, ms, idx in self.all_durations + if (t - fix_time).total_seconds() >= 0 and (t - fix_time).total_seconds() <= 60 and idx] + + # Take up to 20 samples each + before = before[-20:] + after = after[:20] + samples = before + after + + if not samples: + return Panel("[dim]Building snapshot...[/]", title="[bold]Performance Graph[/]", border_style="magenta", width=76) + + durations = [s[1] for s in samples] + has_idx = [s[2] for s in samples] + + max_ms = max(max(durations), 100) + if max_ms > 2000: max_ms = ((int(max_ms) // 500) + 1) * 500 + elif max_ms > 500: max_ms = ((int(max_ms) // 200) + 1) * 200 + else: max_ms = ((int(max_ms) // 100) + 1) * 100 + + # Calculate before/after averages + before_avg = sum(s[1] for s in before) / max(len(before), 1) + after_avg = sum(s[1] for s in after) / max(len(after), 1) + improvement = ((before_avg - after_avg) / max(before_avg, 1)) * 100 + + lines = [] + for row in range(self.GRAPH_HEIGHT, 0, -1): + threshold = (row / self.GRAPH_HEIGHT) * max_ms + label = f"{int(threshold):>5}ms │" + bar = "" + for i, ms in enumerate(durations): + if ms >= threshold: + bar += "[red]█[/]" if not has_idx[i] else "[green]█[/]" + else: + lower = ((row - 1) / self.GRAPH_HEIGHT) * max_ms + if ms > lower: + frac = (ms - lower) / (threshold - lower) + idx = min(int(frac * (len(self.BLOCKS) - 1)), len(self.BLOCKS) - 1) + char = self.BLOCKS[idx] + bar += f"[red]{char}[/]" if not has_idx[i] else f"[green]{char}[/]" + else: + bar += " " + lines.append(f"{label}{bar}") + + lines.append(f" 0ms │{'─' * len(durations)}") + + # Marker at the fix point + fix_offset = len(before) + pointer_line = " " * 8 + " " * fix_offset + "[green bold]▼ SRE Agent fixed it here[/]" + + # Stats + stats = f"\n [red]██ BEFORE[/] avg: [red bold]{before_avg:.0f}ms[/] [green]██ AFTER[/] avg: [green bold]{after_avg:.0f}ms[/] [cyan bold]⚡ {improvement:.0f}% faster[/]" + + graph_text = "\n".join(lines) + f"\n{pointer_line}" + stats + + return Panel( + graph_text, + title="[bold magenta]📊 Before / After — SRE Agent Fix[/]", + border_style="green", + width=76, + ) + + +def _status(ms): + if ms < 100: + return "[green bold]⚡ FAST[/]" + if ms < 500: + return "[yellow bold]⏱ OK[/]" + return "[red bold]🐌 SLOW[/]" + + +def health_check(): + """Poll the /health endpoint. Returns (status_code, latency_ms, body).""" + try: + r = req.get(HEALTH_URL, timeout=5) + return r.status_code, r.elapsed.total_seconds() * 1000, r.text[:200] + except Exception as e: + return 0, 0, str(e)[:200] + + +def _wait_key(): + """Block until any key is pressed (Windows).""" + if sys.platform == "win32": + msvcrt.getch() + else: + input() + +# ── Banner & Menu ─────────────────────────────────────────── + +BANNER = r"""[bold cyan] + ███████╗ █████╗ ██╗ ██╗ █████╗ + ╚══███╔╝██╔══██╗██║ ██║██╔══██╗ + ███╔╝ ███████║██║ ██║███████║ + ███╔╝ ██╔══██║╚██╗ ██╔╝██╔══██║ + ███████╗██║ ██║ ╚████╔╝ ██║ ██║ + ╚══════╝╚═╝ ╚═╝ ╚═══╝ ╚═╝ ╚═╝ + [bold white]Zava Café — SRE Agent Demo Simulator[/bold white][/bold cyan] +""" + + +def show_menu(): + console.clear() + console.print(BANNER) + + tbl = Table( + title="[bold]Demo Scenarios[/]", + box=box.DOUBLE_EDGE, + border_style="cyan", + title_style="bold white", + show_lines=True, + padding=(0, 2), + ) + tbl.add_column("#", style="bold cyan", width=4, justify="center") + tbl.add_column("Scenario", style="bold white", width=28) + tbl.add_column("Description", style="dim white", width=52) + + tbl.add_row( + "1", "🐌 Slow Query", + "Missing index → slow queries on Products.\n" + "SRE Agent detects & creates the index.", + ) + tbl.add_row( + "2", "🔒 Blocking Chain", + "Transaction holds locks, blocking other sessions.\n" + "SRE Agent detects & kills the blocker.", + ) + tbl.add_row( + "3", "🚀 GH Actions Deployment", + "Bad config via GitHub Actions workflow.\n" + "SRE Agent validates, rollbacks, creates GH Issue.", + ) + tbl.add_row( + "5", "📡 Simulate HTTP Trigger", + "Inject bad config + fire HTTP trigger.\n" + "SRE Agent detects & restores config.", + ) + tbl.add_row( + "6", "🎯 Simulate All", + "Launch scenarios 1 and 3 in separate\n" + "terminals simultaneously.", + ) + tbl.add_row( + "7", "🧹 Reset All", + "Drop indexes, kill blockers, restore config.\n" + "Returns environment to baseline.", + ) + tbl.add_row("Q", "🚪 Quit", "Exit the simulator.") + + console.print(Align.center(tbl)) + console.print() + + # Quick status + lines = [] + try: + r = req.get(HEALTH_URL, timeout=3) + h = "[green]● Healthy[/]" if r.status_code == 200 else f"[red]● Down ({r.status_code})[/]" + except Exception: + h = "[red]● Unreachable[/]" + lines.append(f" App Health: {h}") + lines.append(f" SQL Server: [dim]{SQL_SERVER}[/]") + lines.append(f" Database: [dim]{SQL_DATABASE}[/]") + lines.append(f" pymssql: {'[green]● Installed[/]' if HAS_PYMSSQL else '[red]● Missing[/]'}") + + console.print(Align.center( + Panel("\n".join(lines), title="[bold]System Status[/]", border_style="dim", width=62) + )) + console.print() + + +# ═══════════════════════════════════════════════════════════ +# SCENARIO 1 — Slow Query (Missing Index) +# ═══════════════════════════════════════════════════════════ + +def scenario_slow_query(): + console.clear() + console.print(Panel( + "[bold]Scenario 1 — Slow Query (Missing Index)[/]\n\n" + "Runs repeated queries on [cyan]Products.Category[/].\n" + "Without an index the DB does a table scan (slow).\n" + "SRE Agent should detect this and create an index.\n\n" + "[dim]Controls: q = quit r/d = drop index (reset)[/]", + title="[cyan bold]🐌 SLOW QUERY SIMULATOR[/]", + border_style="cyan", width=76, + )) + + conn = get_sql_connection() + if not conn: + console.print("[dim]Press any key…[/]"); _wait_key(); return + + cur = conn.cursor() + categories = [ + "Espresso", "Brewed Coffee", "Pastries", + "Merch", + ] + log = [] + index_found = False + index_banner_shown = False + + def _has_index(): + try: + cur.execute(""" + SELECT COUNT(*) + FROM sys.indexes i + JOIN sys.index_columns ic + ON i.object_id = ic.object_id AND i.index_id = ic.index_id + JOIN sys.columns c + ON ic.object_id = c.object_id AND ic.column_id = c.column_id + WHERE i.object_id = OBJECT_ID('Products') + AND c.name = 'Category' + AND i.type > 0 + """) + row = cur.fetchone() + return (row[0] > 0) if row else False + except Exception: + return False + + def _drop_idx(): + try: + cur.execute(""" + DECLARE @sql NVARCHAR(MAX) = ''; + SELECT @sql += 'DROP INDEX ' + QUOTENAME(i.name) + ' ON Products; ' + FROM sys.indexes i + JOIN sys.index_columns ic + ON i.object_id = ic.object_id AND i.index_id = ic.index_id + JOIN sys.columns c + ON ic.object_id = c.object_id AND ic.column_id = c.column_id + WHERE i.object_id = OBJECT_ID('Products') + AND c.name = 'Category' + AND i.type > 0 + AND i.is_primary_key = 0; + IF @sql <> '' EXEC sp_executesql @sql; + """) + conn.commit() + except Exception as e: + console.print(f"[yellow]drop-index warning: {e}[/]") + + def _flush_cache(): + """Clear SQL plan cache so recompiles pick up index changes.""" + try: + cur.execute("DBCC FREEPROCCACHE") + conn.commit() + console.print("[green] ✅ Plan cache cleared[/]") + except Exception as e: + console.print(f"[yellow] ⚠ Cache clear skipped: {e}[/]") + + def _ensure_data_volume(target=2_000_000): + """One-time expansion so full table scans are genuinely slow.""" + try: + cur.execute("SELECT COUNT(*) FROM Products") + current = cur.fetchone()[0] + if current >= target: + console.print(f"[green] ✅ Data volume OK ({current:,} rows)[/]") + return + needed = target - current + console.print( + f"[yellow] ⏳ Expanding data: {current:,} → ~{target:,} rows " + f"(one-time setup, a few minutes)…[/]" + ) + batch_size = 50_000 + num_cats = 50 + inserted = 0 + cat_idx = 1 + while inserted < needed: + batch = min(batch_size, needed - inserted) + cat_label = f"Filler_{cat_idx:03d}" + cur.execute( + f"INSERT INTO Products (Name, Category, Price) " + f"SELECT TOP {batch} " + f"'P-' + CAST(ABS(CHECKSUM(NEWID())) AS VARCHAR(10)), " + f"'{cat_label}', " + f"CAST(RAND(CHECKSUM(NEWID())) * 490 + 10 AS DECIMAL(10,2)) " + f"FROM sys.all_objects a CROSS JOIN sys.all_objects b" + ) + conn.commit() + inserted += batch + cat_idx = (cat_idx % num_cats) + 1 + pct = min(100, inserted / needed * 100) + console.print(f" [dim]{pct:.0f}% ({current + inserted:,} rows)[/]") + console.print(f"[green] ✅ Data expanded to {current + inserted:,} rows[/]") + except Exception as e: + console.print(f"[yellow] ⚠ Data expansion failed: {e}[/]") + + console.print("[yellow]Dropping existing Category index…[/]") + _drop_idx() + console.print("[yellow]Clearing plan cache…[/]") + _flush_cache() + console.print("[yellow]Checking data volume…[/]") + _ensure_data_volume() + console.print("[green]Starting query loop …[/]\n") + time.sleep(0.5) + + timeline = EventTimeline() + timeline.add("Simulation started — index dropped, cache cleared", "cyan") + perf_graph = PerfGraph() + first_slow_logged = False + alert_detected_time = None + alert_resolved = False + index_created_time = None + sim_start_utc = datetime.utcnow() + + iteration = 0 + try: + with Live(console=console, refresh_per_second=4) as live: + while True: + key = check_key() + if key == "q": + break + if key in ("r", "d"): + timeline.add("⌨️ Key [r] pressed — resetting...", "yellow bold") + live.update(Panel("[yellow bold]⏳ Dropping index and clearing cache... please wait[/]", border_style="yellow", width=76)) + _drop_idx() + _flush_cache() + index_found = False + index_banner_shown = False + first_slow_logged = False + log.clear() + perf_graph = PerfGraph() + timeline.add("Reset — index dropped, cache cleared", "yellow") + continue + + cat = random.choice(categories) + + t0 = time.time() + try: + cur.execute( + "SELECT COUNT(*) " + "FROM Products WHERE Category = %s " + "OPTION (MAXDOP 1, RECOMPILE)", + (cat,), + ) + row = cur.fetchone() + ms = (time.time() - t0) * 1000 + cnt = row[0] if row else 0 + except Exception: + ms = (time.time() - t0) * 1000 + cnt = -1 + + log.append({ + "ts": datetime.now().strftime("%H:%M:%S.%f")[:-3], + "cat": cat, + "ms": ms, + "cnt": cnt, + }) + perf_graph.add(ms, has_index=index_found) + if len(log) > 20: + log.pop(0) + + # Track events + if ms > 500 and not first_slow_logged: + first_slow_logged = True + timeline.add(f"First slow query detected: {ms:.0f}ms", "red") + + iteration += 1 + if iteration % 5 == 0: + prev = index_found + index_found = _has_index() + if index_found and not prev: + index_banner_shown = False + index_created_time = datetime.now().strftime("%H:%M:%S") + timeline.add("🎉 INDEX CREATED by SRE Agent!", "green bold") + if first_slow_logged: + timeline.add(f"Issue resolved — queries should be fast now", "green") + + # Check alert state every 5 iterations + if iteration % 5 == 0: + alert_condition, alert_start = _check_alert_fired(since_time=sim_start_utc) + if alert_condition == "Fired" and alert_detected_time is None: + alert_detected_time = datetime.now().strftime("%H:%M:%S") + timeline.add(f"🚨 ALERT CREATED — Azure Monitor DTU alert fired", "red bold") + if alert_condition == "Resolved" and alert_detected_time and not alert_resolved: + if index_found: + timeline.add(f"✅ ALERT RESOLVED — DTU back to normal", "green bold") + else: + timeline.add(f"⚠️ ALERT RESOLVED — DTU dropped (may re-fire)", "yellow") + alert_resolved = True + + # ── build display ── + grid = Table.grid(padding=1) + grid.add_column() + + grid.add_row(Panel( + "[bold cyan]🐌 SLOW QUERY SIMULATOR[/] — " + "querying [bold]Products[/] by [bold]Category[/]\n" + "[dim]q = quit r/d = drop index[/]", + border_style="cyan", + )) + + # index celebration / status + if index_found and not index_banner_shown: + index_banner_shown = True + grid.add_row(Panel( + "[bold green]🎉🎉🎉 INDEX CREATED! 🎉🎉🎉[/]\n\n" + "[green]The SRE Agent detected the missing index and created it!\n" + "Watch query times drop dramatically.[/]", + border_style="green bold", + title="[green bold]✅ INDEX DETECTED[/]", + )) + elif index_found: + grid.add_row(Text( + " ✅ Index Status: PRESENT — queries should be fast!", + style="green bold", + )) + else: + grid.add_row(Text( + " ❌ Index Status: MISSING — full table scan!", + style="red bold", + )) + + # stats + if log: + durs = [e["ms"] for e in log] + avg = sum(durs) / len(durs) + last = durs[-1] + stbl = Table(box=box.SIMPLE, show_header=False, padding=(0, 2)) + stbl.add_column("L", style="dim") + stbl.add_column("V", style="bold") + stbl.add_row("Queries", str(len(log))) + stbl.add_row("Last", f"[{_color(last)}]{last:.1f} ms[/]") + stbl.add_row("Avg", f"[{_color(avg)}]{avg:.1f} ms[/]") + stbl.add_row("Min/Max", f"{min(durs):.1f} / {max(durs):.1f} ms") + grid.add_row(stbl) + + # query table + qt = Table( + title="[bold]Recent Queries[/]", + box=box.ROUNDED, border_style="dim", show_lines=False, + ) + qt.add_column("Time", style="dim", width=14) + qt.add_column("Category", width=18) + qt.add_column("Duration", width=12, justify="right") + qt.add_column("Bar", width=32) + qt.add_column("Status", width=12, justify="center") + qt.add_column("Rows", width=8, justify="right") + for e in log[-6:]: + m = e["ms"] + qt.add_row( + e["ts"], e["cat"], + f"[{_color(m)}]{m:.1f} ms[/]", + _bar(m), + _status(m), + str(e["cnt"]) if e["cnt"] >= 0 else "[red]ERR[/]", + ) + grid.add_row(qt) + grid.add_row(perf_graph.to_panel()) + grid.add_row(timeline.to_table()) + live.update(grid) + time.sleep(0.5) + except KeyboardInterrupt: + pass + finally: + try: + cur.close(); conn.close() + except Exception: + pass + + +# ═══════════════════════════════════════════════════════════ +# SCENARIO 2 — Blocking Chain +# ═══════════════════════════════════════════════════════════ + +def scenario_blocking(): + console.clear() + console.print(Panel( + "[bold]Scenario 2 — Blocking Chain[/]\n\n" + "Opens a transaction that holds an exclusive lock on Products,\n" + "then a second session tries to read and gets blocked.\n" + "SRE Agent should kill the head blocker.\n\n" + "[dim]Controls: q = quit r = recreate c = commit (release)[/]", + title="[cyan bold]🔒 BLOCKING CHAIN SIMULATOR[/]", + border_style="cyan", width=76, + )) + + blocker_conn = get_sql_connection() + monitor_conn = get_sql_connection() + if not blocker_conn or not monitor_conn: + console.print("[dim]Press any key…[/]"); _wait_key(); return + + bcur = blocker_conn.cursor() + mcur = monitor_conn.cursor() + blocked = False + blocker_spid = None + block_start = None + victim_resolved = threading.Event() + + def _create_block(): + nonlocal blocked, blocker_spid, block_start + try: + bcur.execute("SELECT @@SPID") + blocker_spid = bcur.fetchone()[0] + bcur.execute("BEGIN TRANSACTION") + bcur.execute( + "UPDATE Products SET Price = Price WHERE Category = 'Espresso'" + ) + blocked = True + block_start = time.time() + return True + except Exception as e: + console.print(f"[red]block error: {e}[/]") + return False + + def _victim(): + vc = get_sql_connection() + if not vc: + victim_resolved.set(); return + try: + c = vc.cursor() + c.execute( + "SELECT COUNT(*) FROM Products WHERE Category = 'Espresso'" + ) + c.fetchone() + except Exception: + pass + victim_resolved.set() + try: + vc.close() + except Exception: + pass + + console.print("[yellow]Creating blocking transaction…[/]") + if not _create_block(): + return + console.print(f"[green]Blocker SPID [bold]{blocker_spid}[/bold] — lock held.[/]") + console.print("[yellow]Starting victim query (will block)…[/]") + victim_resolved.clear() + threading.Thread(target=_victim, daemon=True).start() + time.sleep(2) + + try: + with Live(console=console, refresh_per_second=2) as live: + while True: + key = check_key() + if key == "q": + break + if key == "c": + try: + bcur.execute("IF @@TRANCOUNT > 0 COMMIT") + except Exception: + pass + blocked = False + if key == "r": + try: + bcur.execute("IF @@TRANCOUNT > 0 COMMIT") + except Exception: + pass + victim_resolved.clear() + _create_block() + threading.Thread(target=_victim, daemon=True).start() + time.sleep(1) + + # query DMV for blocking info + binfo = [] + try: + mcur.execute(""" + SELECT r.session_id, + r.blocking_session_id, + r.wait_type, + r.wait_time / 1000.0, + r.status + FROM sys.dm_exec_requests r + WHERE r.blocking_session_id > 0 + AND r.database_id = DB_ID(%s) + """, (SQL_DATABASE,)) + for row in mcur: + binfo.append({ + "victim": row[0], + "blocker": row[1], + "wait": row[2], + "secs": row[3], + "status": row[4], + }) + except Exception: + pass + + blocker_alive = False + try: + mcur.execute( + "SELECT COUNT(*) FROM sys.dm_exec_sessions WHERE session_id = %s", + (blocker_spid,), + ) + r = mcur.fetchone() + blocker_alive = (r[0] > 0) if r else False + except Exception: + pass + + # ── display ── + grid = Table.grid(padding=1) + grid.add_column() + grid.add_row(Panel( + "[bold cyan]🔒 BLOCKING CHAIN SIMULATOR[/]\n" + "[dim]q = quit r = recreate c = commit[/]", + border_style="cyan", + )) + + if blocker_alive and blocked: + wait = time.time() - block_start if block_start else 0 + grid.add_row(Panel( + f"[red bold]⚠ ACTIVE BLOCKER[/]\n\n" + f" Blocker SPID: [bold]{blocker_spid}[/]\n" + f" Status: [red bold]HOLDING LOCK[/]\n" + f" Duration: [yellow]{wait:.1f}s[/]\n" + f" Table: Products\n" + f" Blocked queries: [red]{len(binfo)}[/]", + title="[red bold]🔒 HEAD BLOCKER[/]", + border_style="red", + )) + elif victim_resolved.is_set() or not blocker_alive: + blocked = False + grid.add_row(Panel( + "[bold green]🎉🎉🎉 BLOCKER KILLED! 🎉🎉🎉[/]\n\n" + "[green]The SRE Agent detected the blocking chain\n" + "and terminated the head blocker.[/]\n\n" + f"[dim]SPID {blocker_spid} removed.[/]", + border_style="green bold", + title="[green bold]✅ RESOLVED[/]", + )) + + if binfo: + bt = Table( + title="[bold red]Blocked Sessions[/]", + box=box.ROUNDED, border_style="red", + ) + bt.add_column("Victim", justify="center") + bt.add_column("Blocked By", justify="center") + bt.add_column("Wait Type") + bt.add_column("Wait (s)", justify="right") + bt.add_column("Status") + for b in binfo: + bt.add_row( + str(b["victim"]), + f"[red bold]{b['blocker']}[/]", + b["wait"] or "", + f"[yellow]{b['secs']:.1f}[/]", + b["status"] or "", + ) + grid.add_row(bt) + elif blocked: + grid.add_row(Text( + " ⏳ Waiting for victim to appear in DMV…", + style="yellow", + )) + + live.update(grid) + time.sleep(1) + except KeyboardInterrupt: + pass + finally: + try: + bcur.execute("IF @@TRANCOUNT > 0 COMMIT") + except Exception: + pass + for c in (bcur, mcur): + try: + c.close() + except Exception: + pass + for c in (blocker_conn, monitor_conn): + try: + c.close() + except Exception: + pass + + +# ═══════════════════════════════════════════════════════════ +# SCENARIO 3 — Bad Deployment +# ═══════════════════════════════════════════════════════════ + +_AZ_CONN_CMD_GOOD = ( + 'az webapp config connection-string set ' + '--name app-zava --resource-group rg-zava ' + '--connection-string-type SQLAzure ' + '--settings "DefaultConnection=Server=sql-zava.database.windows.net;' + 'Database=sqldb-zava;User Id=;Password=;' + 'Encrypt=True;TrustServerCertificate=True;" ' + '-o none 2>&1' +) + +_AZ_CONN_CMD_BAD = ( + 'az webapp config connection-string set ' + '--name app-zava --resource-group rg-zava ' + '--connection-string-type SQLAzure ' + '--settings "DefaultConnection=Server=sql-zava-WRONG.database.windows.net;' + 'Database=sqldb-zava;User Id=;Password=;' + 'Encrypt=True;TrustServerCertificate=True;" ' + '-o none 2>&1' +) + +# Bad-deployment scenario trigger — populated from azd env (ZAVA_HTTP_TRIGGER_URL). +_WEBHOOK_URL = ZAVA_HTTP_TRIGGER_URL + + +def scenario_bad_deployment(): + console.clear() + console.print(Panel( + "[bold]Scenario 3 — Bad Deployment[/]\n\n" + "Simulates a bad config deployment:\n" + " 1. First ensures the app is HEALTHY\n" + " 2. Press [b] to inject a bad DB connection string\n" + " 3. Fires HTTP trigger to notify SRE Agent\n" + " 4. Monitors /health until SRE Agent fixes it\n\n" + "[dim]Controls: q = quit b = break (deploy bad config) f = fix manually[/]", + title="[cyan bold]💥 BAD DEPLOYMENT SIMULATOR[/]", + border_style="cyan", width=76, + )) + + hlog = [] + timeline = EventTimeline() + broken = False + was_broken = False + seen_down = False + + # Ensure app is healthy first + console.print("[yellow]Ensuring app is healthy before simulation...[/]") + os.system(_AZ_CONN_CMD_GOOD) + time.sleep(3) + code, ms, body = health_check() + if code == 200: + console.print("[green] ✅ App is healthy — ready to simulate[/]") + timeline.add("App confirmed healthy — ready for simulation", "green") + else: + console.print(f"[yellow] ⚠ App returned {code} — may need a moment to start[/]") + timeline.add(f"App returned {code} — waiting for startup", "yellow") + time.sleep(1) + + try: + with Live(console=console, refresh_per_second=1) as live: + while True: + key = check_key() + if key == "q": + break + if key == "b" and not broken: + timeline.add("⌨️ Key [b] pressed — deploying bad config...", "yellow bold") + live.update(Panel("[yellow bold]⏳ Deploying bad config... please wait[/]", border_style="yellow", width=76)) + os.system(_AZ_CONN_CMD_BAD) + broken = True + was_broken = True + seen_down = False + timeline.add("⏳ Waiting for bad config to take effect...", "yellow") + # Wait for app to actually go down before firing webhook + for _ in range(10): + time.sleep(3) + c, _, _ = health_check() + if c != 200: + seen_down = True + timeline.add(f"❌ App is DOWN ({c}) — bad config confirmed", "red") + break + if not seen_down: + timeline.add("⚠ App still responding 200 — restarting to force config reload", "yellow") + os.system("az webapp restart --name app-zava --resource-group rg-zava -o none 2>&1") + time.sleep(10) + + timeline.add("📡 Firing HTTP trigger to SRE Agent...", "cyan") + # Fire webhook with auth token + if not _WEBHOOK_URL: + timeline.add( + "⚠ ZAVA_HTTP_TRIGGER_URL not set — run `azd hooks run postprovision` " + "to register the trigger, or `azd env set ZAVA_HTTP_TRIGGER_URL `.", + "red", + ) + else: + try: + import subprocess + token = subprocess.run( + 'az account get-access-token --resource "https://azuresre.ai" --query accessToken -o tsv', + capture_output=True, text=True, timeout=15, shell=True + ).stdout.strip() + payload = { + "source": "simulator", + "event": "deployment_completed", + "repo": "/Zava", + "app_name": "app-zava", + "app_url": APP_URL, + "health_endpoint": HEALTH_URL, + "status": "deployed", + "message": "Bad config deployed — DB connection string changed to sql-zava-WRONG. Health check is failing. Please investigate and fix.", + } + headers = {"Authorization": f"Bearer {token}", "Content-Type": "application/json"} + r = req.post(_WEBHOOK_URL, json=payload, headers=headers, timeout=15) + timeline.add(f"SRE Agent notified (HTTP {r.status_code})", "cyan") + # Extract threadId from response — same shape as zava-power sim + try: + resp = r.json() if r.status_code in (200, 201, 202) else {} + except Exception: + resp = {} + thread_id = (resp.get("threadId") or resp.get("execution", {}).get("threadId") or "") + if thread_id: + thread_url = f"{SRE_AGENT_THREAD_BASE}/{thread_id}" + timeline.add(f"🔗 Agent thread: {thread_url}", "cyan") + except Exception as e: + timeline.add(f"Webhook failed: {str(e)[:80]}", "red") + + if key == "f": + timeline.add("⌨️ Key [f] pressed — restoring good config...", "yellow bold") + live.update(Panel("[yellow bold]⏳ Restoring good config... please wait[/]", border_style="yellow", width=76)) + os.system(_AZ_CONN_CMD_GOOD) + + code, ms, body = health_check() + healthy = code == 200 + hlog.append({ + "ts": datetime.now().strftime("%H:%M:%S"), + "code": code, "ms": ms, + "ok": healthy, "body": body[:100], + }) + if len(hlog) > 30: + hlog.pop(0) + + # Detect recovery — only if we confirmed it was DOWN first + if broken and healthy and seen_down: + broken = False + timeline.add("🎉 APP RECOVERED — SRE Agent fixed the config!", "green bold") + + # ── display ── + grid = Table.grid(padding=1) + grid.add_column() + grid.add_row(Panel( + "[bold cyan]💥 BAD DEPLOYMENT SIMULATOR[/]\n" + "[dim]q = quit b = break (deploy bad config) f = fix manually[/]", + border_style="cyan", + )) + + if was_broken and not broken and healthy: + grid.add_row(Panel( + "[bold green]🎉🎉🎉 APP RECOVERED! 🎉🎉🎉[/]\n\n" + "[green]SRE Agent detected the bad deployment\n" + "and restored the correct connection string![/]", + border_style="green bold", + title="[green bold]✅ RECOVERED[/]", + )) + elif healthy: + grid.add_row(Panel( + f"[green bold]● HEALTHY[/] Status: [green]{code}[/] Latency: [green]{ms:.0f}ms[/] Endpoint: {HEALTH_URL}", + border_style="green", title="[green]App Health[/]", + )) + else: + grid.add_row(Panel( + f"[red bold]● DOWN[/] Status: [red]{code}[/] Error: [red]{body[:60]}[/]", + border_style="red bold", title="[red]⚠ App Health[/]", + )) + + ht = Table( + title="[bold]Health History[/]", + box=box.ROUNDED, border_style="dim", + ) + ht.add_column("Time", width=10) + ht.add_column("Status", width=8, justify="center") + ht.add_column("Code", width=6, justify="center") + ht.add_column("Latency", width=10, justify="right") + for e in hlog[-8:]: + ht.add_row( + e["ts"], + "[green]✅ UP[/]" if e["ok"] else "[red]❌ DN[/]", + str(e["code"]), + f"{e['ms']:.0f} ms", + ) + grid.add_row(ht) + grid.add_row(timeline.to_table()) + live.update(grid) + time.sleep(2) + except KeyboardInterrupt: + pass + + + +# ═══════════════════════════════════════════════════════════ +# SCENARIO 5 — Reset All +# ═══════════════════════════════════════════════════════════ + +def scenario_reset(): + console.clear() + console.print(Panel( + "[bold]Reset All[/]\n\n" + "Returns the demo environment to its baseline state.", + title="[cyan bold]🧹 RESET ALL[/]", + border_style="cyan", width=76, + )) + console.print() + + steps = [ + ("Drop indexes on Products", "idx"), + ("Clear SQL cache", "cache"), + ("Kill blocking sessions", "kill"), + ("Restore app connection string", "conn"), + ("Restart app service", "restart"), + ("Verify app health", "health"), + ] + + for desc, tag in steps: + console.print(f" [yellow]⏳ {desc}…[/]", end="") + try: + if tag == "idx": + conn = get_sql_connection() + if conn: + c = conn.cursor() + c.execute(""" + DECLARE @sql NVARCHAR(MAX) = ''; + SELECT @sql += 'DROP INDEX ' + QUOTENAME(i.name) + + ' ON Products; ' + FROM sys.indexes i + JOIN sys.index_columns ic + ON i.object_id = ic.object_id + AND i.index_id = ic.index_id + JOIN sys.columns col + ON ic.object_id = col.object_id + AND ic.column_id = col.column_id + WHERE i.object_id = OBJECT_ID('Products') + AND col.name = 'Category' + AND i.type > 0 + AND i.is_primary_key = 0; + IF @sql <> '' EXEC sp_executesql @sql; + """) + conn.commit(); c.close(); conn.close() + console.print(" [green]✅[/]") + else: + console.print(" [red]❌ no connection[/]") + + elif tag == "cache": + conn = get_sql_connection() + if conn: + c = conn.cursor() + try: + c.execute("DBCC FREEPROCCACHE") + c.execute("DBCC DROPCLEANBUFFERS") + conn.commit() + console.print(" [green]✅[/]") + except Exception: + console.print(" [yellow]⚠ needs sysadmin[/]") + c.close(); conn.close() + else: + console.print(" [red]❌ no connection[/]") + + elif tag == "kill": + conn = get_sql_connection() + if conn: + c = conn.cursor() + c.execute(""" + SELECT DISTINCT blocking_session_id + FROM sys.dm_exec_requests + WHERE blocking_session_id > 0 + AND database_id = DB_ID(%s) + """, (SQL_DATABASE,)) + spids = [r[0] for r in c.fetchall()] + for s in spids: + try: + c.execute(f"KILL {s}") + except Exception: + pass + c.close(); conn.close() + console.print( + f" [green]✅ killed {len(spids)}[/]" + if spids else " [green]✅ none found[/]" + ) + else: + console.print(" [red]❌ no connection[/]") + + elif tag == "conn": + rc = os.system(_AZ_CONN_CMD_GOOD) + console.print( + " [green]✅[/]" if rc == 0 else " [yellow]⚠ check az cli[/]" + ) + + elif tag == "restart": + os.system("az webapp restart --name app-zava --resource-group rg-zava -o none 2>&1") + console.print(" [green]✅[/]") + + elif tag == "health": + time.sleep(10) + code, ms, _ = health_check() + if code == 200: + console.print(f" [green]✅ healthy ({ms:.0f} ms)[/]") + else: + console.print(f" [yellow]⚠ status {code}[/]") + + except Exception as e: + console.print(f" [red]❌ {e}[/]") + + console.print() + console.print(Panel("[green bold]🧹 Reset complete![/]", border_style="green")) + console.print("\n[dim]Press any key to return…[/]") + _wait_key() + + +def scenario_all(): + console.print(Panel( + "[bold]Launching all scenarios in separate terminals...[/]\n\n" + " Terminal 1: 🐌 Slow Query (Scenario 1)\n" + " Terminal 2: 📡 HTTP Trigger (Scenario 5)\n\n" + "[dim]Each scenario runs in its own window.[/]", + title="[cyan bold]🚀 SIMULATE ALL[/]", + border_style="cyan", width=76, + )) + + script = os.path.abspath(__file__) + + subprocess.Popen(f'start "Zava - Slow Query" cmd /k python {script} 1', shell=True) + time.sleep(1) + subprocess.Popen(f'start "Zava - HTTP Trigger" cmd /k python {script} 5', shell=True) + + console.print("[green]✅ All terminals launched![/]") + console.print("[dim]Press any key to return to menu...[/]") + _wait_key() + + +# ═══════════════════════════════════════════════════════════ +# SCENARIO 6 — GitHub Actions Deployment +# ═══════════════════════════════════════════════════════════ + +# GitHub-deploy scenario trigger — same backend trigger, kept as a separate +# constant so it can be split later if we register a per-event trigger. +_GH_WEBHOOK_URL = ZAVA_HTTP_TRIGGER_URL + +_GH_REPO = "meetshamir/ZavaCafe-SREAgent" +_GH_TOKEN = os.environ.get("ZAVA_GH_TOKEN", "") + + +def scenario_gh_deployment(): + console.clear() + console.print(Panel( + "[bold]Scenario 3 — GitHub Actions Bad Deployment[/]\n\n" + "Simulates a bad deployment via real GitHub Actions:\n" + " 1. Ensures app is HEALTHY\n" + " 2. Press [b] to push a bad commit to appsettings.json\n" + " 3. GitHub Actions builds + deploys the bad config\n" + " 4. GH Actions fires HTTP trigger to SRE Agent\n" + " 5. Monitors /health until SRE Agent fixes it\n\n" + "[dim]Controls: q = quit b = break (push bad commit) f = fix manually[/]", + title="[cyan bold]🚀 GH ACTIONS DEPLOYMENT[/]", + border_style="cyan", width=76, + )) + + hlog = [] + timeline = EventTimeline() + broken = False + was_broken = False + seen_down = False + + # Ensure healthy first — restore good appsettings and push + console.print("[yellow]Ensuring app has good config...[/]") + _restore_good_config() + time.sleep(3) + code, ms, body = health_check() + if code == 200: + console.print("[green] ✅ App is healthy — ready to simulate[/]") + timeline.add("App confirmed healthy", "green") + else: + console.print(f"[yellow] ⚠ App returned {code} — may need a moment[/]") + timeline.add(f"App returned {code}", "yellow") + time.sleep(1) + + try: + with Live(console=console, refresh_per_second=1) as live: + while True: + key = check_key() + if key == "q": + break + if key == "b" and not broken: + timeline.add("⌨️ Key [b] — pushing bad commit...", "yellow bold") + live.update(Panel("[yellow bold]⏳ Pushing bad appsettings.json to GitHub...[/]", border_style="yellow", width=76)) + + # Push bad appsettings.json + _push_bad_config() + broken = True + was_broken = True + timeline.add("💥 Bad commit pushed — GH Actions will build + deploy", "red bold") + timeline.add("⏳ Monitoring GH Actions workflow...", "yellow") + gh_run_url = None + + # Monitor GH Actions run + gh_headers = {"Authorization": f"token {_GH_TOKEN}", "Accept": "application/vnd.github.v3+json"} if _GH_TOKEN else {"Accept": "application/vnd.github.v3+json"} + for attempt in range(60): # wait up to 5 mins + time.sleep(5) + try: + r = req.get(f"https://api.github.com/repos/{_GH_REPO}/actions/runs?per_page=1", headers=gh_headers, timeout=10) + if r.status_code == 200: + runs = r.json().get("workflow_runs", []) + if runs: + run = runs[0] + status = run.get("status", "") + conclusion = run.get("conclusion", "") + gh_run_url = run.get("html_url", "") + if status == "completed": + timeline.add(f"🏁 GH Actions: {conclusion} — {gh_run_url}", "cyan" if conclusion == "success" else "red") + break + elif attempt % 6 == 0: # log every 30s + timeline.add(f"⏳ GH Actions: {status}...", "yellow") + except Exception: + pass + + # Wait for health to go down after deploy + timeline.add("⏳ Waiting for bad deploy to take effect...", "yellow") + for _ in range(20): + time.sleep(3) + c, _, _ = health_check() + if c != 200: + seen_down = True + timeline.add(f"❌ App is DOWN ({c}) — bad deploy confirmed", "red") + break + + # Fire authenticated webhook to SRE Agent + timeline.add("📡 Firing HTTP trigger to SRE Agent...", "cyan bold") + if not _GH_WEBHOOK_URL: + timeline.add( + "⚠ ZAVA_HTTP_TRIGGER_URL not set — run `azd hooks run postprovision` " + "to register the trigger.", + "red", + ) + else: + try: + token = subprocess.run( + 'az account get-access-token --resource "https://azuresre.ai" --query accessToken -o tsv', + capture_output=True, text=True, timeout=15, shell=True + ).stdout.strip() + payload = { + "source": "github-actions", + "event": "deployment_completed", + "repo": _GH_REPO, + "commit_sha": "bad-config-commit", + "commit_message": "Update database connection string", + "branch": "main", + "actor": "meetshamir", + "app_name": "app-zava", + "app_url": APP_URL, + "health_endpoint": HEALTH_URL, + "run_url": gh_run_url or f"https://github.com/{_GH_REPO}/actions", + "status": "success", + "message": "Deployment succeeded but app health is failing. Investigate the latest commit.", + } + headers = {"Authorization": f"Bearer {token}", "Content-Type": "application/json"} + r = req.post(_GH_WEBHOOK_URL, json=payload, headers=headers, timeout=15) + timeline.add(f"✅ SRE Agent notified (HTTP {r.status_code})", "cyan") + try: + resp = r.json() if r.status_code in (200, 201, 202) else {} + except Exception: + resp = {} + thread_id = (resp.get("threadId") or resp.get("execution", {}).get("threadId") or "") + if thread_id: + timeline.add(f"🔗 Agent thread: {SRE_AGENT_THREAD_BASE}/{thread_id}", "cyan") + except Exception as e: + timeline.add(f"Webhook failed: {str(e)[:80]}", "red") + + if key == "f": + timeline.add("⌨️ Key [f] — restoring good config...", "yellow bold") + _restore_good_config() + + # Poll health + code, ms, body = health_check() + healthy = code == 200 + hlog.append({ + "ts": datetime.now().strftime("%H:%M:%S"), + "code": code, "ms": ms, "ok": healthy, + }) + if len(hlog) > 30: + hlog.pop(0) + + # Detect app going down + if broken and not seen_down and not healthy: + seen_down = True + timeline.add(f"❌ App is DOWN ({code}) — bad deploy confirmed", "red") + + # Detect recovery + if broken and healthy and seen_down: + broken = False + timeline.add("🎉 APP RECOVERED — SRE Agent fixed it!", "green bold") + + # Display + grid = Table.grid(padding=1) + grid.add_column() + grid.add_row(Panel( + "[bold cyan]🚀 GH ACTIONS DEPLOYMENT[/] — deployment-validator-gh\n" + "[dim]q = quit b = push bad commit f = fix manually[/]", + border_style="cyan", + )) + + if was_broken and not broken and healthy: + grid.add_row(Panel( + "[bold green]🎉🎉🎉 APP RECOVERED! 🎉🎉🎉[/]\n\n" + "[green]SRE Agent detected the bad deployment, rolled back,\n" + "and created a GitHub Issue with the RCA![/]", + border_style="green bold", + title="[green bold]✅ RECOVERED[/]", + )) + elif healthy: + grid.add_row(Panel( + f"[green bold]● HEALTHY[/] Status: [green]{code}[/] Latency: [green]{ms:.0f}ms[/]", + border_style="green", title="[green]App Health[/]", + )) + else: + grid.add_row(Panel( + f"[red bold]● DOWN[/] Status: [red]{code}[/] Error: [red]{body[:60]}[/]", + border_style="red bold", title="[red]⚠ App Health[/]", + )) + + ht = Table(title="[bold]Health History[/]", box=box.ROUNDED, border_style="dim") + ht.add_column("Time", width=10) + ht.add_column("Status", width=8, justify="center") + ht.add_column("Code", width=6, justify="center") + ht.add_column("Latency", width=10, justify="right") + for e in hlog[-6:]: + ht.add_row( + e["ts"], + "[green]✅ UP[/]" if e["ok"] else "[red]❌ DN[/]", + str(e["code"]), + f"{e['ms']:.0f} ms", + ) + grid.add_row(ht) + grid.add_row(timeline.to_table()) + live.update(grid) + time.sleep(2) + except KeyboardInterrupt: + pass + + +def _push_bad_config(): + """Push a bad appsettings.json via GitHub API.""" + if not _GH_TOKEN: + console.print("[red]Set ZAVA_GH_TOKEN env var to push commits[/]") + return + headers = {"Authorization": f"token {_GH_TOKEN}", "Accept": "application/vnd.github.v3+json"} + bad_config = json.dumps({ + "ConnectionStrings": {"DefaultConnection": "Server=sql-zava-WRONG.database.windows.net;Database=sqldb-zava;User Id=;Password=;Encrypt=True;TrustServerCertificate=True;"}, + "ApplicationInsights": {"ConnectionString": ""}, + "Logging": {"LogLevel": {"Default": "Information", "Microsoft.AspNetCore": "Warning"}}, + "AllowedHosts": "*" + }, indent=2) + import base64 + content = base64.b64encode(bad_config.encode()).decode() + try: + existing = req.get(f"https://api.github.com/repos/{_GH_REPO}/contents/src/appsettings.json", headers=headers, timeout=10).json() + body = {"message": "Update database connection string", "content": content, "sha": existing["sha"]} + r = req.put(f"https://api.github.com/repos/{_GH_REPO}/contents/src/appsettings.json", headers=headers, json=body, timeout=10) + if r.status_code in (200, 201): + console.print("[green] ✅ Bad commit pushed to GitHub[/]") + else: + console.print(f"[red] ❌ Push failed: {r.status_code}[/]") + except Exception as e: + console.print(f"[red] ❌ Push error: {e}[/]") + + +def _restore_good_config(): + """Restore good appsettings.json via GitHub API.""" + if not _GH_TOKEN: + console.print("[red]Set ZAVA_GH_TOKEN env var[/]") + return + headers = {"Authorization": f"token {_GH_TOKEN}", "Accept": "application/vnd.github.v3+json"} + good_config = json.dumps({ + "ConnectionStrings": {"DefaultConnection": "Server=sql-zava.database.windows.net;Database=sqldb-zava;User Id=;Password=;Encrypt=True;TrustServerCertificate=True;"}, + "ApplicationInsights": {"ConnectionString": ""}, + "Logging": {"LogLevel": {"Default": "Information", "Microsoft.AspNetCore": "Warning"}}, + "AllowedHosts": "*" + }, indent=2) + import base64 + content = base64.b64encode(good_config.encode()).decode() + try: + existing = req.get(f"https://api.github.com/repos/{_GH_REPO}/contents/src/appsettings.json", headers=headers, timeout=10).json() + body = {"message": "Restore correct database connection string", "content": content, "sha": existing["sha"]} + r = req.put(f"https://api.github.com/repos/{_GH_REPO}/contents/src/appsettings.json", headers=headers, json=body, timeout=10) + except Exception: + pass + + +# ═══════════════════════════════════════════════════════════ +# MAIN +# ═══════════════════════════════════════════════════════════ + +SCENARIOS = { + "1": ("Slow Query", scenario_slow_query), + "2": ("Blocking Chain", scenario_blocking), + "3": ("GH Actions Deploy", scenario_gh_deployment), + "5": ("HTTP Trigger", scenario_bad_deployment), + "6": ("Simulate All", scenario_all), + "7": ("Reset All", scenario_reset), +} + + +def main(): + while True: + show_menu() + choice = console.input( + "[bold cyan]Select scenario (1-7, q=quit): [/]" + ).strip().lower() + + if choice in ("q", "quit", "exit"): + console.print("\n[cyan]👋 Goodbye — happy demo-ing![/]\n") + break + + if choice in SCENARIOS: + name, fn = SCENARIOS[choice] + console.print(f"\n[cyan]Launching {name}…[/]\n") + try: + fn() + except KeyboardInterrupt: + pass + except Exception as e: + console.print(f"\n[red]Error: {e}[/]") + console.print("[dim]Press any key…[/]") + _wait_key() + else: + console.print("[red]Invalid choice.[/]") + time.sleep(1) + + +if __name__ == "__main__": + # Allow direct scenario launch: python demo.py 1 + if len(sys.argv) > 1: + scenario = sys.argv[1].strip().lower() + scenario_map = { + "1": scenario_slow_query, + "slow": scenario_slow_query, + "2": scenario_blocking, + "block": scenario_blocking, + "3": scenario_gh_deployment, + "gh": scenario_gh_deployment, + "5": scenario_bad_deployment, + "http": scenario_bad_deployment, + "6": scenario_all, + "all": scenario_all, + "7": scenario_reset, + "reset": scenario_reset, + } + fn = scenario_map.get(scenario) + if fn: + try: + fn() + except KeyboardInterrupt: + console.print("\n[cyan]👋 Interrupted. Goodbye![/]\n") + else: + console.print(f"[red]Unknown scenario: {scenario}[/]") + console.print("[dim]Usage: python demo.py [1|2|3|4|5|6|slow|block|deploy|sn|reset|all][/]") + sys.exit(0) + + try: + main() + except KeyboardInterrupt: + console.print("\n[cyan]👋 Interrupted. Goodbye![/]\n") + sys.exit(0) diff --git a/labs/zava-cafe/simulator/expand_data.py b/labs/zava-cafe/simulator/expand_data.py new file mode 100644 index 000000000..dcbafea5b --- /dev/null +++ b/labs/zava-cafe/simulator/expand_data.py @@ -0,0 +1,57 @@ +"""One-time script to expand Products table to ~2M rows for demo.""" +import pymssql, time + +conn = pymssql.connect( + server='sql-zava.database.windows.net', + user='', + password='', + database='sqldb-zava', + login_timeout=10, + timeout=300, +) +cur = conn.cursor() + +cur.execute('SELECT COUNT(*) FROM Products') +current = cur.fetchone()[0] +print(f'Current rows: {current:,}') + +target = 2_000_000 +if current >= target: + print('Already at target. Done.') + conn.close() + exit() + +needed = target - current +print(f'Need to insert {needed:,} more rows...') + +batch_size = 50_000 +num_cats = 50 +inserted = 0 +cat_idx = 1 +start = time.time() + +while inserted < needed: + batch = min(batch_size, needed - inserted) + cat_label = f'Filler_{cat_idx:03d}' + sql = ( + f"INSERT INTO Products (Name, Category, Price) " + f"SELECT TOP {batch} " + f"'P-' + CAST(ABS(CHECKSUM(NEWID())) AS VARCHAR(10)), " + f"'{cat_label}', " + f"CAST(RAND(CHECKSUM(NEWID())) * 490 + 10 AS DECIMAL(10,2)) " + f"FROM sys.all_objects a CROSS JOIN sys.all_objects b" + ) + cur.execute(sql) + conn.commit() + inserted += batch + cat_idx = (cat_idx % num_cats) + 1 + elapsed = time.time() - start + pct = inserted / needed * 100 + rate = inserted / max(elapsed, 1) + eta = (needed - inserted) / max(rate, 1) + print(f' {pct:.0f}% | {current + inserted:,} rows | {rate:.0f} rows/sec | ETA {eta:.0f}s') + +cur.execute('SELECT COUNT(*) FROM Products') +final = cur.fetchone()[0] +print(f'\nDone! Final count: {final:,} rows in {time.time()-start:.0f}s') +conn.close() diff --git a/labs/zava-cafe/simulator/requirements.txt b/labs/zava-cafe/simulator/requirements.txt new file mode 100644 index 000000000..a2883dffa --- /dev/null +++ b/labs/zava-cafe/simulator/requirements.txt @@ -0,0 +1,3 @@ +rich +requests +pymssql diff --git a/labs/zava-cafe/src/.gitignore b/labs/zava-cafe/src/.gitignore new file mode 100644 index 000000000..2789d7166 --- /dev/null +++ b/labs/zava-cafe/src/.gitignore @@ -0,0 +1,5 @@ +bin/ +obj/ +*.user +*.suo +.vs/ diff --git a/labs/zava-cafe/src/Program.cs b/labs/zava-cafe/src/Program.cs new file mode 100644 index 000000000..2ccec330e --- /dev/null +++ b/labs/zava-cafe/src/Program.cs @@ -0,0 +1,140 @@ +using Microsoft.Data.SqlClient; + +var builder = WebApplication.CreateBuilder(args); + +builder.Services.AddEndpointsApiExplorer(); +builder.Services.AddSwaggerGen(); +builder.Services.AddApplicationInsightsTelemetry(); + +var app = builder.Build(); + +if (app.Environment.IsDevelopment()) +{ + app.UseSwagger(); + app.UseSwaggerUI(); +} + +app.UseHttpsRedirection(); + +// GET / — welcome page +app.MapGet("/", () => Results.Ok(new +{ + app = "Zava", + version = "1.0.0", + message = "Welcome to Zava Café — Specialty Coffee, Pastries & Merch" +})); + +// GET /health — checks SQL database connectivity +app.MapGet("/health", async (IConfiguration config) => +{ + var connectionString = config.GetConnectionString("DefaultConnection"); + if (string.IsNullOrEmpty(connectionString)) + { + return Results.Json(new { status = "unhealthy", database = "connection_failed", error = "Connection string is not configured" }, + statusCode: 503); + } + + try + { + using var connection = new SqlConnection(connectionString); + await connection.OpenAsync(); + using var command = connection.CreateCommand(); + command.CommandText = "SELECT 1"; + await command.ExecuteScalarAsync(); + + return Results.Ok(new { status = "healthy", database = "connected" }); + } + catch (Exception ex) + { + return Results.Json(new { status = "unhealthy", database = "connection_failed", error = ex.Message }, + statusCode: 503); + } +}); + +// GET /api/products — list products, optionally filtered by category +app.MapGet("/api/products", async (IConfiguration config, string? category) => +{ + var connectionString = config.GetConnectionString("DefaultConnection"); + if (string.IsNullOrEmpty(connectionString)) + { + return Results.Problem("Database connection string is not configured", statusCode: 500); + } + + try + { + var products = new List(); + using var connection = new SqlConnection(connectionString); + await connection.OpenAsync(); + using var command = connection.CreateCommand(); + + if (!string.IsNullOrEmpty(category)) + { + command.CommandText = "SELECT Id, Name, Price, Category FROM Products WHERE Category = @Category"; + command.Parameters.AddWithValue("@Category", category); + } + else + { + command.CommandText = "SELECT TOP 100 Id, Name, Price, Category FROM Products"; + } + + using var reader = await command.ExecuteReaderAsync(); + while (await reader.ReadAsync()) + { + products.Add(new Product( + reader.GetInt32(0), + reader.GetString(1), + reader.GetDecimal(2), + reader.GetString(3))); + } + + return Results.Ok(products); + } + catch (Exception ex) + { + return Results.Problem($"Failed to retrieve products: {ex.Message}", statusCode: 500); + } +}) +.WithName("GetProducts") +.WithOpenApi(); + +// GET /api/products/{id} — get a single product by ID +app.MapGet("/api/products/{id:int}", async (int id, IConfiguration config) => +{ + var connectionString = config.GetConnectionString("DefaultConnection"); + if (string.IsNullOrEmpty(connectionString)) + { + return Results.Problem("Database connection string is not configured", statusCode: 500); + } + + try + { + using var connection = new SqlConnection(connectionString); + await connection.OpenAsync(); + using var command = connection.CreateCommand(); + command.CommandText = "SELECT Id, Name, Price, Category FROM Products WHERE Id = @Id"; + command.Parameters.AddWithValue("@Id", id); + + using var reader = await command.ExecuteReaderAsync(); + if (await reader.ReadAsync()) + { + var product = new Product( + reader.GetInt32(0), + reader.GetString(1), + reader.GetDecimal(2), + reader.GetString(3)); + return Results.Ok(product); + } + + return Results.NotFound(new { error = $"Product with ID {id} not found" }); + } + catch (Exception ex) + { + return Results.Problem($"Failed to retrieve product: {ex.Message}", statusCode: 500); + } +}) +.WithName("GetProductById") +.WithOpenApi(); + +app.Run(); + +record Product(int Id, string Name, decimal Price, string Category); diff --git a/labs/zava-cafe/src/Properties/launchSettings.json b/labs/zava-cafe/src/Properties/launchSettings.json new file mode 100644 index 000000000..f6a4808a9 --- /dev/null +++ b/labs/zava-cafe/src/Properties/launchSettings.json @@ -0,0 +1,41 @@ +{ + "$schema": "http://json.schemastore.org/launchsettings.json", + "iisSettings": { + "windowsAuthentication": false, + "anonymousAuthentication": true, + "iisExpress": { + "applicationUrl": "http://localhost:30067", + "sslPort": 44394 + } + }, + "profiles": { + "http": { + "commandName": "Project", + "dotnetRunMessages": true, + "launchBrowser": true, + "launchUrl": "swagger", + "applicationUrl": "http://localhost:5153", + "environmentVariables": { + "ASPNETCORE_ENVIRONMENT": "Development" + } + }, + "https": { + "commandName": "Project", + "dotnetRunMessages": true, + "launchBrowser": true, + "launchUrl": "swagger", + "applicationUrl": "https://localhost:7063;http://localhost:5153", + "environmentVariables": { + "ASPNETCORE_ENVIRONMENT": "Development" + } + }, + "IIS Express": { + "commandName": "IISExpress", + "launchBrowser": true, + "launchUrl": "swagger", + "environmentVariables": { + "ASPNETCORE_ENVIRONMENT": "Development" + } + } + } +} diff --git a/labs/zava-cafe/src/ZavaCafeApp.csproj b/labs/zava-cafe/src/ZavaCafeApp.csproj new file mode 100644 index 000000000..d5688af34 --- /dev/null +++ b/labs/zava-cafe/src/ZavaCafeApp.csproj @@ -0,0 +1,16 @@ + + + + net8.0 + enable + enable + + + + + + + + + + diff --git a/labs/zava-cafe/src/ZavaCafeApp.http b/labs/zava-cafe/src/ZavaCafeApp.http new file mode 100644 index 000000000..0d600532f --- /dev/null +++ b/labs/zava-cafe/src/ZavaCafeApp.http @@ -0,0 +1,6 @@ +@ZavaCafeApp_HostAddress = http://localhost:5153 + +GET {{ZavaCafeApp_HostAddress}}/weatherforecast/ +Accept: application/json + +### diff --git a/labs/zava-cafe/src/appsettings.Development.json b/labs/zava-cafe/src/appsettings.Development.json new file mode 100644 index 000000000..40b5f8bcc --- /dev/null +++ b/labs/zava-cafe/src/appsettings.Development.json @@ -0,0 +1,14 @@ +{ + "ConnectionStrings": { + "DefaultConnection": "" + }, + "ApplicationInsights": { + "ConnectionString": "" + }, + "Logging": { + "LogLevel": { + "Default": "Information", + "Microsoft.AspNetCore": "Warning" + } + } +} diff --git a/labs/zava-cafe/src/appsettings.json b/labs/zava-cafe/src/appsettings.json new file mode 100644 index 000000000..c3c5e00a8 --- /dev/null +++ b/labs/zava-cafe/src/appsettings.json @@ -0,0 +1,15 @@ +{ + "ConnectionStrings": { + "DefaultConnection": "Server=sql-zava.database.windows.net;Database=sqldb-zava;User Id=;Password=;Encrypt=True;TrustServerCertificate=True;" + }, + "ApplicationInsights": { + "ConnectionString": "" + }, + "Logging": { + "LogLevel": { + "Default": "Information", + "Microsoft.AspNetCore": "Warning" + } + }, + "AllowedHosts": "*" +} \ No newline at end of file diff --git a/labs/zava-cafe/sre-config/agent1/.github/instructions.md b/labs/zava-cafe/sre-config/agent1/.github/instructions.md new file mode 100644 index 000000000..c3a0758d3 --- /dev/null +++ b/labs/zava-cafe/sre-config/agent1/.github/instructions.md @@ -0,0 +1,1478 @@ +# SRECTL - SRE Agent CLI Instructions + +This file contains comprehensive documentation for all SRECTL commands and their usage. +Generated on: 2026-05-09 22:39:19 UTC + +## Table of Contents + +1. [Main Command](#main-command) +2. [General Commands](#general-commands) + - [init](#init-command) + - [list](#list-command) + - [apply-yaml](#apply-yaml-command) +3. [Agent Commands](#agent-commands) + - [agent create](#agent-create-command) + - [agent validate](#agent-validate-command) + - [agent apply](#agent-apply-command) + - [agent run](#agent-run-command) +4. [Tool Commands](#tool-commands) + - [tool create](#tool-create-command) + - [tool validate](#tool-validate-command) + - [tool apply](#tool-apply-command) + - [tool show-types](#tool-show-types-command) + - [tool show-connectors](#tool-show-connectors-command) +5. [Skills Commands](#skills-commands) + - [skill create](#skill-create-command) + - [skill upload](#skill-upload-command) + - [skill list](#skill-list-command) + - [skill delete](#skill-delete-command) + - [skill convert](#skill-convert-command) + - [skill download](#skill-download-command) + +## Main Command + +### Main Command {#main-command} + +``` +$ srectl --help + +Description: + SRE Agent CLI - Your intelligent assistant for managing SRE agents and automating incident response + +Usage: + srectl [options] + srectl [options] + +Options: + -h, /h, -?, /? Show help and usage information + --version Show version information + --debug Enable debug logging + --quiet Minimize output + +Subgroups: + agent Agent commands for managing SRE automation agents + tool Tool commands for managing SRE automation tools + common-prompt Common prompt commands for managing shared prompts + extension Extension commands for generating deployment files and configurations + mcp Model Context Protocol server for building SRE agents + doc Document management commands. Upload and manage documents like TSGs, architecture docs, runbooks, and other reference materials for agents to use + workspace Workspace management commands. Upload, download, and delete workspace files. + incident-filter Incident filter commands for managing incident routing rules + hook Manage hooks for agent safety and governance + thread Thread management commands + profile Profile management commands. Profiles store connection settings for different SRE Agent instances (local or remote) + repo Manage Azure DevOps repository connectors for TSG documents + skill Skill management commands. Apply and manage custom skills for agents to use, or convert an existing agent into a skill. + incidenthandler Manage incident response plans and filters + scheduledtask Manage scheduled tasks for automated agent operations + release-trigger Release trigger commands for managing pipeline event response plans + +Commands: + welcome Show welcome screen and getting started guide + version Show version information and build details + init Initialize SREAgent CLI configuration and workspace + + Examples: + # Initialize with local development server + srectl init --resource-url https://localhost:7023 + + # Initialize with remote server + srectl init --resource-url https://my-sreagent-dev.1abcdef.eastus2.azuresre.ai + + # Initialize with production environment + srectl init --resource-url https://my-sreagent-prod.2abcdef.eastus2.azuresre.ai + status Show workspace status and health check + apply-yaml, apply Apply YAML configuration files to the server + Supports multi-document YAML files (separated by ---) similar to Kubernetes manifests. + Automatically detects and applies tools, agents, and common prompts. + + Examples: + # Apply a single resource YAML file + srectl apply-yaml --file agents/MyAgent/MyAgent.yaml + + # Apply a multi-document YAML file + srectl apply-yaml --file manifests/all-resources.yaml + + # Apply a tool YAML file + srectl apply-yaml --file tools/KustoTool.yaml + + # Apply a common prompt YAML file + srectl apply-yaml --file CommonPrompts/prompt.yaml + interactive Start interactive guided mode for step-by-step assistance + sync Sync agents and tools YAML from the remote server into the local workspace (agents/, tools/) + + Examples: + # Sync all remote configurations + srectl sync + + Note: Requires prior 'srectl init --resource-url ' + chat Start an interactive chat session with the SRE Agent + + Examples: + # Start interactive chat + srectl chat + + # Start chat with debug logging + srectl chat --debug + + # Start chat with minimal output + srectl chat --quiet +``` + +## General Commands + +### init Command {#init-command} + +``` +$ srectl init --help + +Description: + Initialize SREAgent CLI configuration and workspace + + Examples: + # Initialize with local development server + srectl init --resource-url https://localhost:7023 + + # Initialize with remote server + srectl init --resource-url https://my-sreagent-dev.1abcdef.eastus2.azuresre.ai + + # Initialize with production environment + srectl init --resource-url https://my-sreagent-prod.2abcdef.eastus2.azuresre.ai + +Usage: + srectl init [options] + +Options: + --resource-url (REQUIRED) Base URL of the SRE Agent server + -?, -h, --help Show help and usage information + --debug Enable debug logging + --quiet Minimize output +``` + +### list Command {#list-command} + +``` +$ srectl list --help + +Description: + SRE Agent CLI - Your intelligent assistant for managing SRE agents and automating incident response + +Usage: + srectl [options] + srectl [options] + +Options: + -h, /h, -?, /? Show help and usage information + --version Show version information + --debug Enable debug logging + --quiet Minimize output + +Subgroups: + agent Agent commands for managing SRE automation agents + tool Tool commands for managing SRE automation tools + common-prompt Common prompt commands for managing shared prompts + extension Extension commands for generating deployment files and configurations + mcp Model Context Protocol server for building SRE agents + doc Document management commands. Upload and manage documents like TSGs, architecture docs, runbooks, and other reference materials for agents to use + workspace Workspace management commands. Upload, download, and delete workspace files. + incident-filter Incident filter commands for managing incident routing rules + hook Manage hooks for agent safety and governance + thread Thread management commands + profile Profile management commands. Profiles store connection settings for different SRE Agent instances (local or remote) + repo Manage Azure DevOps repository connectors for TSG documents + skill Skill management commands. Apply and manage custom skills for agents to use, or convert an existing agent into a skill. + incidenthandler Manage incident response plans and filters + scheduledtask Manage scheduled tasks for automated agent operations + release-trigger Release trigger commands for managing pipeline event response plans + +Commands: + welcome Show welcome screen and getting started guide + version Show version information and build details + init Initialize SREAgent CLI configuration and workspace + + Examples: + # Initialize with local development server + srectl init --resource-url https://localhost:7023 + + # Initialize with remote server + srectl init --resource-url https://my-sreagent-dev.1abcdef.eastus2.azuresre.ai + + # Initialize with production environment + srectl init --resource-url https://my-sreagent-prod.2abcdef.eastus2.azuresre.ai + status Show workspace status and health check + apply-yaml, apply Apply YAML configuration files to the server + Supports multi-document YAML files (separated by ---) similar to Kubernetes manifests. + Automatically detects and applies tools, agents, and common prompts. + + Examples: + # Apply a single resource YAML file + srectl apply-yaml --file agents/MyAgent/MyAgent.yaml + + # Apply a multi-document YAML file + srectl apply-yaml --file manifests/all-resources.yaml + + # Apply a tool YAML file + srectl apply-yaml --file tools/KustoTool.yaml + + # Apply a common prompt YAML file + srectl apply-yaml --file CommonPrompts/prompt.yaml + interactive Start interactive guided mode for step-by-step assistance + sync Sync agents and tools YAML from the remote server into the local workspace (agents/, tools/) + + Examples: + # Sync all remote configurations + srectl sync + + Note: Requires prior 'srectl init --resource-url ' + chat Start an interactive chat session with the SRE Agent + + Examples: + # Start interactive chat + srectl chat + + # Start chat with debug logging + srectl chat --debug + + # Start chat with minimal output + srectl chat --quiet +``` + +### list agents Command {#list-agents-command} + +``` +$ srectl list agents --help + +Description: + SRE Agent CLI - Your intelligent assistant for managing SRE agents and automating incident response + +Usage: + srectl [options] + srectl [options] + +Options: + -h, /h, -?, /? Show help and usage information + --version Show version information + --debug Enable debug logging + --quiet Minimize output + +Subgroups: + agent Agent commands for managing SRE automation agents + tool Tool commands for managing SRE automation tools + common-prompt Common prompt commands for managing shared prompts + extension Extension commands for generating deployment files and configurations + mcp Model Context Protocol server for building SRE agents + doc Document management commands. Upload and manage documents like TSGs, architecture docs, runbooks, and other reference materials for agents to use + workspace Workspace management commands. Upload, download, and delete workspace files. + incident-filter Incident filter commands for managing incident routing rules + hook Manage hooks for agent safety and governance + thread Thread management commands + profile Profile management commands. Profiles store connection settings for different SRE Agent instances (local or remote) + repo Manage Azure DevOps repository connectors for TSG documents + skill Skill management commands. Apply and manage custom skills for agents to use, or convert an existing agent into a skill. + incidenthandler Manage incident response plans and filters + scheduledtask Manage scheduled tasks for automated agent operations + release-trigger Release trigger commands for managing pipeline event response plans + +Commands: + welcome Show welcome screen and getting started guide + version Show version information and build details + init Initialize SREAgent CLI configuration and workspace + + Examples: + # Initialize with local development server + srectl init --resource-url https://localhost:7023 + + # Initialize with remote server + srectl init --resource-url https://my-sreagent-dev.1abcdef.eastus2.azuresre.ai + + # Initialize with production environment + srectl init --resource-url https://my-sreagent-prod.2abcdef.eastus2.azuresre.ai + status Show workspace status and health check + apply-yaml, apply Apply YAML configuration files to the server + Supports multi-document YAML files (separated by ---) similar to Kubernetes manifests. + Automatically detects and applies tools, agents, and common prompts. + + Examples: + # Apply a single resource YAML file + srectl apply-yaml --file agents/MyAgent/MyAgent.yaml + + # Apply a multi-document YAML file + srectl apply-yaml --file manifests/all-resources.yaml + + # Apply a tool YAML file + srectl apply-yaml --file tools/KustoTool.yaml + + # Apply a common prompt YAML file + srectl apply-yaml --file CommonPrompts/prompt.yaml + interactive Start interactive guided mode for step-by-step assistance + sync Sync agents and tools YAML from the remote server into the local workspace (agents/, tools/) + + Examples: + # Sync all remote configurations + srectl sync + + Note: Requires prior 'srectl init --resource-url ' + chat Start an interactive chat session with the SRE Agent + + Examples: + # Start interactive chat + srectl chat + + # Start chat with debug logging + srectl chat --debug + + # Start chat with minimal output + srectl chat --quiet +``` + +### list tools Command {#list-tools-command} + +``` +$ srectl list tools --help + +Description: + SRE Agent CLI - Your intelligent assistant for managing SRE agents and automating incident response + +Usage: + srectl [options] + srectl [options] + +Options: + -h, /h, -?, /? Show help and usage information + --version Show version information + --debug Enable debug logging + --quiet Minimize output + +Subgroups: + agent Agent commands for managing SRE automation agents + tool Tool commands for managing SRE automation tools + common-prompt Common prompt commands for managing shared prompts + extension Extension commands for generating deployment files and configurations + mcp Model Context Protocol server for building SRE agents + doc Document management commands. Upload and manage documents like TSGs, architecture docs, runbooks, and other reference materials for agents to use + workspace Workspace management commands. Upload, download, and delete workspace files. + incident-filter Incident filter commands for managing incident routing rules + hook Manage hooks for agent safety and governance + thread Thread management commands + profile Profile management commands. Profiles store connection settings for different SRE Agent instances (local or remote) + repo Manage Azure DevOps repository connectors for TSG documents + skill Skill management commands. Apply and manage custom skills for agents to use, or convert an existing agent into a skill. + incidenthandler Manage incident response plans and filters + scheduledtask Manage scheduled tasks for automated agent operations + release-trigger Release trigger commands for managing pipeline event response plans + +Commands: + welcome Show welcome screen and getting started guide + version Show version information and build details + init Initialize SREAgent CLI configuration and workspace + + Examples: + # Initialize with local development server + srectl init --resource-url https://localhost:7023 + + # Initialize with remote server + srectl init --resource-url https://my-sreagent-dev.1abcdef.eastus2.azuresre.ai + + # Initialize with production environment + srectl init --resource-url https://my-sreagent-prod.2abcdef.eastus2.azuresre.ai + status Show workspace status and health check + apply-yaml, apply Apply YAML configuration files to the server + Supports multi-document YAML files (separated by ---) similar to Kubernetes manifests. + Automatically detects and applies tools, agents, and common prompts. + + Examples: + # Apply a single resource YAML file + srectl apply-yaml --file agents/MyAgent/MyAgent.yaml + + # Apply a multi-document YAML file + srectl apply-yaml --file manifests/all-resources.yaml + + # Apply a tool YAML file + srectl apply-yaml --file tools/KustoTool.yaml + + # Apply a common prompt YAML file + srectl apply-yaml --file CommonPrompts/prompt.yaml + interactive Start interactive guided mode for step-by-step assistance + sync Sync agents and tools YAML from the remote server into the local workspace (agents/, tools/) + + Examples: + # Sync all remote configurations + srectl sync + + Note: Requires prior 'srectl init --resource-url ' + chat Start an interactive chat session with the SRE Agent + + Examples: + # Start interactive chat + srectl chat + + # Start chat with debug logging + srectl chat --debug + + # Start chat with minimal output + srectl chat --quiet +``` + +### apply-yaml Command {#apply-yaml-command} + +``` +$ srectl apply-yaml --help + +Description: + Apply YAML configuration files to the server + Supports multi-document YAML files (separated by ---) similar to Kubernetes manifests. + Automatically detects and applies tools, agents, and common prompts. + + Examples: + # Apply a single resource YAML file + srectl apply-yaml --file agents/MyAgent/MyAgent.yaml + + # Apply a multi-document YAML file + srectl apply-yaml --file manifests/all-resources.yaml + + # Apply a tool YAML file + srectl apply-yaml --file tools/KustoTool.yaml + + # Apply a common prompt YAML file + srectl apply-yaml --file CommonPrompts/prompt.yaml + +Usage: + srectl apply-yaml [options] + +Options: + -f, --file (REQUIRED) Path to the YAML file to apply + -?, -h, --help Show help and usage information + --debug Enable debug logging + --quiet Minimize output +``` + +## Agent Commands + +### agent Command {#agent-command} + +``` +$ srectl agent --help + +Description: + Agent commands for managing SRE automation agents + +Usage: + srectl agent [command] [options] + +Options: + -?, -h, --help Show help and usage information + --debug Enable debug logging + --quiet Minimize output + +Commands: + create Create a new agent YAML configuration file + + Examples: + # Create a basic agent + srectl agent create --name DevOpsAgent --instructions "Help with DevOps tasks such as monitoring and incident response" + + # Create an agent with tools + srectl agent create --name KustoAgent --tools QueryKusto AnalyzeMetrics + + # Create an agent with AI assistance (smart mode) + srectl agent create --name StorageAgent --smart --instructions "Help troubleshoot Azure Storage issues" + + # Create an advanced agent with all options + srectl agent create --name AdvancedAgent \ + --instructions "Complex multi-step agent" \ + --tools Tool1 Tool2 \ + --handoffs Agent1 Agent2 \ + --temperature 0.7 \ + --max-reflection-count 3 + validate Validate agent YAML configuration files + + Examples: + # Validate by agent name (searches in agents/ folder) + srectl agent validate --name MyAgent + + # Validate specific agent by name and check tools + srectl agent validate --name KustoAgent --check-tools + + # Validate all agent files + srectl agent validate --all + + # Validate with tool availability checking + srectl agent validate --all --check-tools + + # Alternative: Validate a specific agent file path + srectl agent validate --file agents/MyAgent/MyAgent.yaml + apply Apply an agent configuration to the remote server + + Examples: + # Apply an agent to the server + srectl agent apply --name DevOpsAgent + + # Preview what would be applied (dry run) + srectl agent apply --name KustoAgent --dry-run + + # Apply with debug logging + srectl agent apply --name MyAgent --debug + delete Delete an agent from the remote server + + Examples: + # Delete an agent from the server + srectl agent delete --name OldAgent + + # Delete with debug logging + srectl agent delete --name TestAgent --debug + test Test an agent with a specific message (starts interactive session) + + Examples: + # Test an agent interactively + srectl agent test --name DevOpsAgent --message "Check pod status in namespace production" + + # Send test message without waiting for response + srectl agent test --name KustoAgent --message "Query memory usage" --no-wait + + # Start interactive session with specific agent + srectl agent test --name MyAgent --message "Help me debug this issue" + + Note: This command is equivalent to 'srectl thread new --agent --message ' + and will start an interactive chat session unless --no-wait is specified. + diff Compare local and remote agent configurations + + Examples: + # Compare default using git-diff (default) + srectl agent diff --name DevOpsAgent + + # Use VS Code diff + srectl agent diff --name KustoAgent --tool code + + # Show inline diff + srectl agent diff --name MyAgent --raw + migrate Migrate V1 agent format to V2 + + Examples: + # Migrate a specific agent + srectl agent migrate --name MyAgent + + # Migrate all agents + srectl agent migrate --all + + # Preview migration changes (dry run) + srectl agent migrate --all --dry-run + + # Migrate specific agent with dry run + srectl agent migrate --name MyAgent --dry-run + list List remote extended agents from the server + + Examples: + # List all agents + srectl agent list + + # List all agents with full YAML details + srectl agent list --detail + + # Get a specific agent by name (full YAML output) + srectl agent list --name MyAgent + + # Search for specific agents + srectl agent list --search devops +``` + +### agent create Command {#agent-create-command} + +``` +$ srectl agent create --help + +Description: + Create a new agent YAML configuration file + + Examples: + # Create a basic agent + srectl agent create --name DevOpsAgent --instructions "Help with DevOps tasks such as monitoring and incident response" + + # Create an agent with tools + srectl agent create --name KustoAgent --tools QueryKusto AnalyzeMetrics + + # Create an agent with AI assistance (smart mode) + srectl agent create --name StorageAgent --smart --instructions "Help troubleshoot Azure Storage issues" + + # Create an advanced agent with all options + srectl agent create --name AdvancedAgent \ + --instructions "Complex multi-step agent" \ + --tools Tool1 Tool2 \ + --handoffs Agent1 Agent2 \ + --temperature 0.7 \ + --max-reflection-count 3 + +Usage: + srectl agent create [options] + +Options: + --name (REQUIRED) Name of the agent + --instructions Instructions for the agent + --tools Tools the agent can use + --mcp-tools MCP tools the agent can use + --handoff-description Description for handoff capabilities + --handoffs Agents this agent can hand off to + --allow-parallel-tool-calls Allow parallel tool execution + --max-reflection-count Maximum number of reflection iterations + --critic-prompt-path Path to critic prompt file + --critic-on-handoff Enable critic on handoff + --custom-reflection-note Custom note for reflection + --common-prompts Common prompts to include + --temperature Model temperature setting + --output-type Expected output format + --vanilla-mode Use vanilla mode without enhancements + --smart Use AI to generate instructions and recommend tools + --enable-skills Enable skills for the agent + --add-system-skills Add system skills (not recommended for custom meta-agents) + -?, -h, --help Show help and usage information + --debug Enable debug logging + --quiet Minimize output +``` + +### agent validate Command {#agent-validate-command} + +``` +$ srectl agent validate --help + +Description: + Validate agent YAML configuration files + + Examples: + # Validate by agent name (searches in agents/ folder) + srectl agent validate --name MyAgent + + # Validate specific agent by name and check tools + srectl agent validate --name KustoAgent --check-tools + + # Validate all agent files + srectl agent validate --all + + # Validate with tool availability checking + srectl agent validate --all --check-tools + + # Alternative: Validate a specific agent file path + srectl agent validate --file agents/MyAgent/MyAgent.yaml + +Usage: + srectl agent validate [options] + +Options: + --name Agent name to validate + --file YAML file to validate + --all Validate all agents + --check-tools Validate that referenced tools exist + -?, -h, --help Show help and usage information + --debug Enable debug logging + --quiet Minimize output +``` + +### agent apply Command {#agent-apply-command} + +``` +$ srectl agent apply --help + +Description: + Apply an agent configuration to the remote server + + Examples: + # Apply an agent to the server + srectl agent apply --name DevOpsAgent + + # Preview what would be applied (dry run) + srectl agent apply --name KustoAgent --dry-run + + # Apply with debug logging + srectl agent apply --name MyAgent --debug + +Usage: + srectl agent apply [options] + +Options: + --name (REQUIRED) Name of the agent to apply + --dry-run Preview changes without applying + -?, -h, --help Show help and usage information + --debug Enable debug logging + --quiet Minimize output +``` + +### agent run Command {#agent-run-command} + +``` +$ srectl agent run --help + +Description: + Agent commands for managing SRE automation agents + +Usage: + srectl agent [command] [options] + +Options: + -?, -h, --help Show help and usage information + --debug Enable debug logging + --quiet Minimize output + +Commands: + create Create a new agent YAML configuration file + + Examples: + # Create a basic agent + srectl agent create --name DevOpsAgent --instructions "Help with DevOps tasks such as monitoring and incident response" + + # Create an agent with tools + srectl agent create --name KustoAgent --tools QueryKusto AnalyzeMetrics + + # Create an agent with AI assistance (smart mode) + srectl agent create --name StorageAgent --smart --instructions "Help troubleshoot Azure Storage issues" + + # Create an advanced agent with all options + srectl agent create --name AdvancedAgent \ + --instructions "Complex multi-step agent" \ + --tools Tool1 Tool2 \ + --handoffs Agent1 Agent2 \ + --temperature 0.7 \ + --max-reflection-count 3 + validate Validate agent YAML configuration files + + Examples: + # Validate by agent name (searches in agents/ folder) + srectl agent validate --name MyAgent + + # Validate specific agent by name and check tools + srectl agent validate --name KustoAgent --check-tools + + # Validate all agent files + srectl agent validate --all + + # Validate with tool availability checking + srectl agent validate --all --check-tools + + # Alternative: Validate a specific agent file path + srectl agent validate --file agents/MyAgent/MyAgent.yaml + apply Apply an agent configuration to the remote server + + Examples: + # Apply an agent to the server + srectl agent apply --name DevOpsAgent + + # Preview what would be applied (dry run) + srectl agent apply --name KustoAgent --dry-run + + # Apply with debug logging + srectl agent apply --name MyAgent --debug + delete Delete an agent from the remote server + + Examples: + # Delete an agent from the server + srectl agent delete --name OldAgent + + # Delete with debug logging + srectl agent delete --name TestAgent --debug + test Test an agent with a specific message (starts interactive session) + + Examples: + # Test an agent interactively + srectl agent test --name DevOpsAgent --message "Check pod status in namespace production" + + # Send test message without waiting for response + srectl agent test --name KustoAgent --message "Query memory usage" --no-wait + + # Start interactive session with specific agent + srectl agent test --name MyAgent --message "Help me debug this issue" + + Note: This command is equivalent to 'srectl thread new --agent --message ' + and will start an interactive chat session unless --no-wait is specified. + diff Compare local and remote agent configurations + + Examples: + # Compare default using git-diff (default) + srectl agent diff --name DevOpsAgent + + # Use VS Code diff + srectl agent diff --name KustoAgent --tool code + + # Show inline diff + srectl agent diff --name MyAgent --raw + migrate Migrate V1 agent format to V2 + + Examples: + # Migrate a specific agent + srectl agent migrate --name MyAgent + + # Migrate all agents + srectl agent migrate --all + + # Preview migration changes (dry run) + srectl agent migrate --all --dry-run + + # Migrate specific agent with dry run + srectl agent migrate --name MyAgent --dry-run + list List remote extended agents from the server + + Examples: + # List all agents + srectl agent list + + # List all agents with full YAML details + srectl agent list --detail + + # Get a specific agent by name (full YAML output) + srectl agent list --name MyAgent + + # Search for specific agents + srectl agent list --search devops +``` + +## Tool Commands + +### tool Command {#tool-command} + +``` +$ srectl tool --help + +Description: + Tool commands for managing SRE automation tools + +Usage: + srectl tool [command] [options] + +Options: + -?, -h, --help Show help and usage information + --debug Enable debug logging + --quiet Minimize output + +Commands: + create Create a new tool YAML configuration file + validate Validate tool YAML configuration files + + Examples: + # Validate a specific tool + srectl tool validate --name QueryMetrics + + # Validate all tools + srectl tool validate --all + + # Validate with debug output + srectl tool validate --name MyTool --debug + apply Apply a tool configuration to the remote server + + Examples: + # Apply a tool to the server + srectl tool apply --name QueryMetrics + + # Preview what would be applied (dry run) + srectl tool apply --name StorageOps --dry-run + + # Apply with debug logging + srectl tool apply --name CustomTool --debug + delete Delete a tool from the remote server + + Examples: + # Delete a tool from the server + srectl tool delete --name OldTool + + # Preview what would be deleted (dry run) + srectl tool delete --name TestTool --dry-run + + # Delete with debug logging + srectl tool delete --name UnusedTool --debug + diff Compare local and remote tool configurations + + Examples: + # Compare default using git + srectl tool diff --name QueryMetrics + + # Use VS Code diff + srectl tool diff --name MyTool --tool code + + # Show inline diff + srectl tool diff --name MyTool --raw + migrate Migrate V1 tool configurations to V2 format + + Examples: + # Migrate a specific tool + srectl tool migrate --name MyKustoTool + + # Migrate all V1 tools + srectl tool migrate --all + + # Migrate specific tool with dry run + srectl tool migrate --name MyKustoTool --dry-run + + # Preview migration without making changes (dry run) + srectl tool migrate --all --dry-run + show-types Display available tool types and their details + + Examples: + # List all available tool types + srectl tool show-types + + # Show detailed information for all types + srectl tool show-types --verbose + + # Show details for a specific tool type + srectl tool show-types --type KustoTool + + # Show specific type with verbose details + srectl tool show-types --type AzureTool --verbose + show-connectors Display configured data connectors (names to use in YAML) and available connector types + + Examples: + # List all available connectors + srectl tool show-connectors + list List all tools from the remote server + + Examples: + # List all tools + srectl tool list + + # List all tools with full YAML details + srectl tool list --detail + + # Get a specific tool by name (full YAML output) + srectl tool list --name TestMigrate + + # Search for specific tools + srectl tool list --search kusto +``` + +### tool create Command {#tool-create-command} + +``` +$ srectl tool create --help + +Description: + Create a new tool YAML configuration file + +Usage: + srectl tool create [options] + +Common Options: + --name (REQUIRED) Name of the tool + --type (REQUIRED) Type of the tool (KustoTool, LinkTool, PythonTool, HttpClientTool) + --path Custom path under tools directory (e.g., 'StorageOperations') + --description Description of the tool + --parameter Tool parameter in format 'name:type:description' (can be specified multiple times) + +KustoTool Options: + --connector Connector name for the tool + --database Database name for KustoTool + --query Query for KustoTool + + Examples: + # Create a KustoTool with all parameters + srectl tool create --name QueryMetrics --type KustoTool --connector analytics-cluster --database LogsDB --query "MyTable | take 10" --parameter limit + # Create a KustoTool with minimal options + srectl tool create --name GetLogs --type KustoTool --connector logs-cluster --database LogsDB + +LinkTool Options: + --template