diff --git a/CHANGELOG.md b/CHANGELOG.md index 344a967a..bb090e1a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **`qwen3-8-27b-l40s` serves NVIDIA's own Qwen3.8-27B-NVFP4 checkpoint from its signed model image** (giantswarm/agent-platform#591): `nvidia/Qwen3.8-27B-NVFP4` (20.4 GiB — NVFP4 on the MLPs and the language-model head, FP8 on the attention layers) replaces the Inferact repack's `hf://` source; the preset gains the model-image environment (HOME, HF_HOME and VLLM_CACHE_ROOT under /tmp) and declares 21 + 20 GiB. Parsers, the thinking-off default and the 64k context are unchanged; `qwen3-8-27b`, the Blackwell recipe, still serves the Inferact repack. - **`modelServing.imageVerification` is on by default: a model pod's images are verified at admission against the Giant Swarm signing identity** (giantswarm/agent-platform#575; epic giantswarm/giantswarm#37853). 4.41.0 gave the block its defaults — `images: [gsoci.azurecr.io/giantswarm/*]`, one keyless attestor for the identity the architect orb's cosign keyless signing gives every image a Giant Swarm CircleCI project builds or re-signs (issuer `https://oidc.circleci.com`, subject the pipeline definition that ran), `type: SigstoreBundle` — and left the switch off until every image a model pod runs was signed. Every one is now, verified with `cosign verify` against that identity: the curated model image the shipped `oci://` preset names, `gsoci.azurecr.io/giantswarm/models/qwen3-8-flash-next-nvfp4:7c4f1bc1a2d6`, which giantswarm/models builds, signs and publishes (giantswarm/agent-platform#554); the llm-d runtime and sidecars of the well-known `LLMInferenceServiceConfig`s at the `llm-d-fast/` prefix (`llm-d-cuda:v0.8.0`, `llm-d-router-disagg-sidecar:v0.9.0`, the endpoint picker, tokenizer and latency-predictor images; giantswarm/llm-d signs every mirror and repack); the KServe `storage-initializer:v0.20.0` and `agent:v0.20.0` retagger copies (giantswarm/retagger#1230); the pre-pull DaemonSet's `pause:3.10.1`; and the `vllm-b12x:20260913` mirror an installation serves the flash-next preset with. Both charts' defaults carry `enabled: true`, so a serving shape under Kyverno renders `-model-serving-image-verification` without being asked: one `verifyImages` rule per pod shape over the model pods of the serving namespace at CREATE and UPDATE, every verified image pinned to its digest, a signature required, Enforce. A container image matching no pattern is left alone, so an installation serving from a registry of its own is untouched until it names its pattern and its signer; an installation whose model pods run an image under the platform's namespace that none of these signed adds its signer or turns the switch off before it upgrades (UPGRADE.md). `make verify-model-serving-policies` asserts the default render carries the policy with its defaults and that the switch off or no Kyverno renders none; `make verify-target` holds the switch on both sides of the golden comparison until `origin/main` carries it. Proof: `kyverno apply` of the rendered default policy over a pod of the llm-d workload shape naming every image above admits it with every image rewritten to its digest, and denies a pod naming an unsigned image under the pattern with `sigstore bundle verification failed`; the same two cases on a lab's Kyverno. +- **An Overview board, the entry point of the platform's Grafana folder** (giantswarm/giantswarm#36711). `dashboards` in both charts renders one ConfigMap per JSON under the connectivity chart's `dashboards/` directory, discovered by observability-operator through `app.giantswarm.io/kind: dashboard` and placed by the organization and folder annotations (both values carry a space, so neither can be a label). It ships `overview.json`, uid `agent-platform-overview`, in `Shared Org / Agent Platform`: what the platform is and which board covers what; component health; what it is doing; and the Postgres and Valkey rows, which have no board of their own. Every health panel reads the **same series as the alert that pages for it** — `gotk_resource_info`, `kube_deployment_status_replicas_available`, `muster_mcpserver_state`, `kube_cnpg_cluster_info`, `kube_pod_container_status_restarts_total`, `muster_session_store_backend`, `redis_memory_used_bytes` — so a panel and a page cannot disagree. Today the six alerts of `agent-platform.rules.yml` hand on-call a runbook and no graph. The ConfigMaps render under the resolved `global.observability.metrics.serviceMonitor.enabled`: the operator that picks them up is part of the observability platform the monitors write to. A component's own board still comes from its own chart. The Postgres and the Valkey row, and the six agentgateway panels of the "what it is doing" row, are gated on the series they read: each one repeats over a hidden `label_values(, __name__)` variable, which yields one value while the metric exists and none while it does not, so an installation that runs neither CloudNativePG nor Valkey nor the agentgateway data plane opens the board without a row of "No data". A component that is on but not yet scraped is hidden the same way, which a Helm conditional on `components..enabled` could not do. + ### Changed - **Every agent platform dashboard lands in one folder customers can reach: `Shared Org / Agent Platform`** (giantswarm/giantswarm#36711). The muster board (`Muster / MCP Gateway`) was loaded into the staff-only `Giant Swarm` organization, in a folder named after the component, so the people who run agents on the platform could not see it: `muster.observability.grafanaDashboard.folder` is `Agent Platform` and `.giantswarm.organization` is `Shared Org`. `Shared Org` is the organization every logged-in customer reaches as a Viewer and it carries the observability data of Giant Swarm managed components, which is where these boards belong; the `Giant Swarm` organization is staff-only. The boards that follow — the agentgateway gateway board, the platform's own overview — land in the same folder. diff --git a/Makefile.custom.mk b/Makefile.custom.mk index 9ac6b2ca..0203801b 100644 --- a/Makefile.custom.mk +++ b/Makefile.custom.mk @@ -89,6 +89,17 @@ GOLDEN_REF ?= origin/main # is on by default with no backend (giantswarm/agent-platform#329), an intended difference # held equal on both sides — a chart that predates the default accepts the key. Empty this # once GOLDEN_REF carries the line. +# The platform's own board ConfigMaps are dropped from a golden render by name, +# never by a value: GOLDEN_REF's schema has no `dashboards` key, so --set on it +# fails the render outright and every document then reads as added +# (giantswarm/giantswarm#36711). One name per board; the list goes with the line. +DASHBOARDS_GOLDEN_DROP := agent-platform-connectivity-dashboard-overview +# Drop those documents from a rendered manifest in place, by metadata.name, and +# keep the leading document separator whatever was dropped — a stripped first +# document would otherwise read as a one-line diff of its own. +define drop_dashboards + @python3 -c 'import re,sys; ex=set(sys.argv[2].split()); docs=open(sys.argv[1]).read().split("\n---\n"); keep=[d for d in docs if not (re.search(r"^ name: (\S+)", d, re.M) and re.search(r"^ name: (\S+)", d, re.M).group(1) in ex)]; out="\n---\n".join(keep).lstrip("-\n"); open(sys.argv[1],"w").write("---\n"+out.rstrip("\n")+"\n")' $(1) "$(DASHBOARDS_GOLDEN_DROP)" +endef WIRING_PG_GOLDEN_HOLD := --set components.model-manager.enabled=false # Objects the 4.0 line changes on purpose, dropped from BOTH renders before the # golden diff (by metadata.name): the v1alpha2 agent Deployments' seccomp @@ -99,7 +110,7 @@ WIRING_PG_GOLDEN_HOLD := --set components.model-manager.enabled=false # on (templates/kagent/harness.yaml — a new object with no 3.x counterpart; its # shape is asserted by verify-kagent-harness / verify-kagent-crds). Empty this # list once GOLDEN_REF carries the line. -GOLDEN_EXCLUDE := kagent-declarative-seccomp agent-platform-connectivity-kagent-controller-ingress kagent +GOLDEN_EXCLUDE := kagent-declarative-seccomp agent-platform-connectivity-kagent-controller-ingress kagent $(DASHBOARDS_GOLDEN_DROP) # Any reference is enough: the assertions read the rendered exception, not the image. PGVECTOR_IMG := gsoci.azurecr.io/giantswarm/pgvector:0.8.2-18-bookworm @@ -228,7 +239,7 @@ verify-modes: ## Assert ingress.mode fail-guards fire (connectivity chart owns t $(GOLDEN_RETIRED) $$out/golden; \ helm template t $(CONNECTIVITY_DIR) $(KYVERNO_GOLDEN) >$$out/head 2>&1 \ || { echo "FAIL: the working-tree render failed"; cat $$out/head; exit 1; }; \ - for f in golden head; do python3 -c 'import re,sys; ex=set(sys.argv[2].split()); docs=open(sys.argv[1]).read().split("\n---\n"); keep=[d for d in docs if not (re.search(r"^ name: (\S+)", d, re.M) and re.search(r"^ name: (\S+)", d, re.M).group(1) in ex)]; out="\n---\n".join(keep).rstrip("\n"); open(sys.argv[1],"w").write(out+"\n")' $$out/$$f "$(GOLDEN_EXCLUDE)"; done; \ + for f in golden head; do python3 -c 'import re,sys; ex=set(sys.argv[2].split()); docs=open(sys.argv[1]).read().split("\n---\n"); keep=[d for d in docs if not (re.search(r"^ name: (\S+)", d, re.M) and re.search(r"^ name: (\S+)", d, re.M).group(1) in ex)]; out="\n---\n".join(keep).lstrip("-\n"); open(sys.argv[1],"w").write("---\n"+out.rstrip("\n")+"\n")' $$out/$$f "$(GOLDEN_EXCLUDE)"; done; \ if diff -u $$out/golden $$out/head; then echo "ok: default render unchanged (excluding $(GOLDEN_EXCLUDE))"; \ else echo "FAIL: the default render drifted from $(GOLDEN_REF)"; exit 1; fi; \ fi @@ -2439,6 +2450,8 @@ verify-wiring: ## Assert the standalone's ported wiring: toggles off = no object for flavor in cilium kubernetes; do \ helm template t $(CONNECTIVITY_DIR) $(WIRING_PG) $(WIRING_PG_GOLDEN_HOLD) --set networkPolicy.flavor=$$flavor 2>/dev/null >/tmp/vw-pg-new-$$flavor.out; \ helm template t /tmp/vw-pg-ref/$(CONNECTIVITY_DIR) $(WIRING_PG) $(WIRING_PG_GOLDEN_HOLD) --set networkPolicy.flavor=$$flavor 2>/dev/null >/tmp/vw-pg-old-$$flavor.out; \ + python3 -c 'import re,sys; ex=set(sys.argv[2].split()); docs=open(sys.argv[1]).read().split("\n---\n"); keep=[d for d in docs if not (re.search(r"^ name: (\S+)", d, re.M) and re.search(r"^ name: (\S+)", d, re.M).group(1) in ex)]; out="\n---\n".join(keep).lstrip("-\n"); open(sys.argv[1],"w").write("---\n"+out.rstrip("\n")+"\n")' /tmp/vw-pg-new-$$flavor.out "$(DASHBOARDS_GOLDEN_DROP)"; \ + python3 -c 'import re,sys; ex=set(sys.argv[2].split()); docs=open(sys.argv[1]).read().split("\n---\n"); keep=[d for d in docs if not (re.search(r"^ name: (\S+)", d, re.M) and re.search(r"^ name: (\S+)", d, re.M).group(1) in ex)]; out="\n---\n".join(keep).lstrip("-\n"); open(sys.argv[1],"w").write("---\n"+out.rstrip("\n")+"\n")' /tmp/vw-pg-old-$$flavor.out "$(DASHBOARDS_GOLDEN_DROP)"; \ diff -u /tmp/vw-pg-old-$$flavor.out /tmp/vw-pg-new-$$flavor.out || { echo "FAIL: the $$flavor render changed with postgres.imagePullSecrets and .affinity unset"; git worktree remove --force /tmp/vw-pg-ref; exit 1; }; \ done; \ git worktree remove --force /tmp/vw-pg-ref; \ diff --git a/helm/agent-platform-connectivity/README.md b/helm/agent-platform-connectivity/README.md index 291fa504..96e3d0b4 100644 --- a/helm/agent-platform-connectivity/README.md +++ b/helm/agent-platform-connectivity/README.md @@ -874,6 +874,10 @@ The kagent block is open in the schema, so the template refuses a key under `kag | hooks.opensslImage.repository | string | `"giantswarm/alpine-openssl"` | | | hooks.opensslImage.tag | string | `"3.5.8"` | | | extraObjects | list | `[]` | | +| dashboards.enabled | bool | `true` | | +| dashboards.namespace | string | `""` | | +| dashboards.organization | string | `"Shared Org"` | | +| dashboards.folder | string | `"Agent Platform"` | | | dicebear | object | `{}` | | | muster.enabled | bool | `true` | | | muster.image.registry | string | `"gsoci.azurecr.io"` | | diff --git a/helm/agent-platform-connectivity/dashboards/overview.json b/helm/agent-platform-connectivity/dashboards/overview.json new file mode 100644 index 00000000..f2559b0f --- /dev/null +++ b/helm/agent-platform-connectivity/dashboards/overview.json @@ -0,0 +1,2245 @@ +{ + "annotations": { + "list": [] + }, + "editable": false, + "fiscalYearStartMonth": 0, + "graphTooltip": 1, + "links": [ + { + "asDropdown": false, + "icon": "external link", + "includeVars": false, + "keepTime": true, + "tags": [ + "topic:agents" + ], + "targetBlank": false, + "title": "Agent Platform boards", + "tooltip": "", + "type": "dashboards", + "url": "" + } + ], + "panels": [ + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 1, + "panels": [], + "title": "What this is", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "gridPos": { + "h": 6, + "w": 24, + "x": 0, + "y": 1 + }, + "id": 2, + "options": { + "code": { + "language": "plaintext", + "showLineNumbers": false, + "showMiniMap": false + }, + "content": "The **Giant Swarm Agent Platform** runs AI agents on your own infrastructure and gives them governed access to curated tools.\n\nThis board is the entry point: is the platform healthy, and what is it doing. Every health panel reads the same series as the alert that pages for it, so a panel and a page cannot disagree.\n\n| Board | What it covers |\n|---|---|\n| **LLM usage** | tokens, cost and latency per agent and per model |\n| **Usage by person** | the same, per signed-in person |\n| **Agentgateway** | the gateway itself: requests, MCP tool calls, control-plane config distribution, runtime |\n| **Muster / MCP Gateway** | the MCP server fleet, tool calls and workflows |\n\nPostgres and Valkey have no board of their own: their rows are below. The full CloudNativePG board lives in the Giant Swarm organization.", + "mode": "markdown" + }, + "pluginVersion": "12.1.0", + "title": "", + "transparent": true, + "type": "text" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 7 + }, + "id": 3, + "panels": [], + "title": "Component health", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Platform HelmReleases that have not been Ready. AgentPlatformHelmReleaseNotReady pages after an hour on the same series.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 0, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 1 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 3, + "x": 0, + "y": 8 + }, + "id": 4, + "options": { + "colorMode": "value", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "count(gotk_resource_info{job=\"flux-ksm\", ready=\"False\", customresource_kind=\"HelmRelease\", cluster_type=\"management_cluster\", exported_namespace=\"flux-giantswarm\", team=\"bumblebee\", cluster_id=~\"$cluster\"}) or vector(0)", + "instant": false, + "legendFormat": "not ready", + "range": true, + "refId": "A" + } + ], + "title": "HelmReleases not ready", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Available replicas of the kagent control plane. KagentControllerDown pages after 30 minutes at zero; agents on the installation are down.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 0, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": 0 + }, + { + "color": "green", + "value": 1 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 3, + "x": 3, + "y": 8 + }, + "id": 5, + "options": { + "colorMode": "value", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum(kube_deployment_status_replicas_available{cluster_type=\"management_cluster\", namespace=\"kagent\", deployment=\"kagent-controller\", cluster_id=~\"$cluster\"}) or vector(0)", + "instant": false, + "legendFormat": "available", + "range": true, + "refId": "A" + } + ], + "title": "kagent-controller replicas", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "MCP servers muster cannot reach. MusterMCPServerFailed notifies after 20 minutes.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 0, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 1 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 3, + "x": 6, + "y": 8 + }, + "id": 6, + "options": { + "colorMode": "value", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "count(max by (mcpserver_name, mcpserver_namespace) (muster_mcpserver_state{state=\"Failed\", cluster_type=\"management_cluster\", cluster_id=~\"$cluster\"})) or vector(0)", + "instant": false, + "legendFormat": "failed", + "range": true, + "refId": "A" + } + ], + "title": "MCP servers failed", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "MCP servers muster has a live session with.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 0, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": 0 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 3, + "x": 9, + "y": 8 + }, + "id": 7, + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "count(max by (mcpserver_name, mcpserver_namespace) (muster_mcpserver_state{state=\"Connected\", cluster_type=\"management_cluster\", cluster_id=~\"$cluster\"})) or vector(0)", + "instant": false, + "legendFormat": "connected", + "range": true, + "refId": "A" + } + ], + "title": "MCP servers connected", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "CloudNativePG clusters of the platform outside a healthy phase. AgentPlatformPostgresClusterNotHealthy pages after 30 minutes.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 0, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 1 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 3, + "x": 12, + "y": 8 + }, + "id": 8, + "options": { + "colorMode": "value", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "count(kube_cnpg_cluster_info{cluster_type=\"management_cluster\", namespace=~\"agent-platform|kagent\", phase!=\"Cluster in healthy state\", cluster_id=~\"$cluster\"}) or vector(0)", + "instant": false, + "legendFormat": "unhealthy", + "range": true, + "refId": "A" + } + ], + "title": "Postgres clusters unhealthy", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Platform containers with more than five restarts in the last hour. AgentPlatformContainerRestartingTooOften pages after 15 minutes.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 0, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 1 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 3, + "x": 15, + "y": 8 + }, + "id": 9, + "options": { + "colorMode": "value", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "count(sum by (namespace, container) (increase(kube_pod_container_status_restarts_total{cluster_type=\"management_cluster\", namespace=~\"agent-platform|kagent|mcp-kubernetes|mcp-capi|mcp-prometheus\", cluster_id=~\"$cluster\"}[1h])) > 5) or vector(0)", + "instant": false, + "legendFormat": "restarting", + "range": true, + "refId": "A" + } + ], + "title": "Containers restarting", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "1 means muster keeps OAuth sessions in memory, so every restart signs every person out. AgentPlatformMusterSessionStoresInMemory pages in working hours.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 0, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "red", + "value": 1 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 3, + "x": 18, + "y": 8 + }, + "id": 10, + "options": { + "colorMode": "value", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "max(muster_session_store_backend{cluster_type=\"management_cluster\", backend=\"memory\", cluster_id=~\"$cluster\"}) or vector(0)", + "instant": false, + "legendFormat": "in memory", + "range": true, + "refId": "A" + } + ], + "title": "muster session store", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Share of the Valkey memory limit in use. AgentPlatformValkeyMemoryHigh notifies above 90%.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "orange", + "value": 0.8 + }, + { + "color": "red", + "value": 0.9 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 3, + "x": 21, + "y": 8 + }, + "id": 11, + "options": { + "colorMode": "value", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "max(redis_memory_used_bytes{cluster_type=\"management_cluster\", namespace=~\"agent-platform|kagent|mcp-kubernetes|mcp-capi|mcp-prometheus\", cluster_id=~\"$cluster\"}) / max(redis_memory_max_bytes{cluster_type=\"management_cluster\", namespace=~\"agent-platform|kagent|mcp-kubernetes|mcp-capi|mcp-prometheus\", cluster_id=~\"$cluster\"} > 0)", + "instant": false, + "legendFormat": "used", + "range": true, + "refId": "A" + } + ], + "title": "Valkey memory used", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "One line per platform HelmRelease: 1 while it is Ready.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": 0 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 0, + "y": 12 + }, + "id": 12, + "options": { + "legend": { + "calcs": [ + "lastNotNull", + "max" + ], + "displayMode": "table", + "placement": "right", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "12.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "max by (name) (gotk_resource_info{job=\"flux-ksm\", ready=\"True\", customresource_kind=\"HelmRelease\", cluster_type=\"management_cluster\", exported_namespace=\"flux-giantswarm\", team=\"bumblebee\", cluster_id=~\"$cluster\"})", + "instant": false, + "legendFormat": "{{name}}", + "range": true, + "refId": "A" + } + ], + "title": "HelmRelease readiness by release", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Restarts in the last hour per container, over the platform's namespaces.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": 0 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 12, + "y": 12 + }, + "id": 13, + "options": { + "legend": { + "calcs": [ + "lastNotNull", + "max" + ], + "displayMode": "table", + "placement": "right", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "12.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (namespace, container) (increase(kube_pod_container_status_restarts_total{cluster_type=\"management_cluster\", namespace=~\"agent-platform|kagent|mcp-kubernetes|mcp-capi|mcp-prometheus\", cluster_id=~\"$cluster\"}[1h]))", + "instant": false, + "legendFormat": "{{namespace}}/{{container}}", + "range": true, + "refId": "A" + } + ], + "title": "Container restarts per hour", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 19 + }, + "id": 14, + "panels": [], + "title": "What it is doing", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Every token the gateway proxied for an agent, all models and token types. LLM usage breaks it down.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 1, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": 0 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 4, + "x": 0, + "y": 20 + }, + "id": 15, + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum(rate(agentgateway_gen_ai_client_token_usage_sum{cluster_id=~\"$cluster\"}[$__rate_interval]))", + "instant": false, + "legendFormat": "tokens/s", + "range": true, + "refId": "A" + } + ], + "title": "Tokens / s", + "type": "stat", + "repeat": "agentgateway_present", + "repeatDirection": "h" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Estimated spend at the current rate. A model with no price in the catalog reads as zero, not as an error; the Price catalog row of LLM usage names them.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 2, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": 0 + } + ] + }, + "unit": "currencyUSD" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 4, + "x": 4, + "y": 20 + }, + "id": 16, + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum(rate(agentgateway_gen_ai_client_cost_usd_total{cluster_id=~\"$cluster\"}[$__rate_interval])) * 3600", + "instant": false, + "legendFormat": "USD/h", + "range": true, + "refId": "A" + } + ], + "title": "Cost / hour", + "type": "stat", + "repeat": "agentgateway_present", + "repeatDirection": "h" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "MCP tool calls through the gateway. The Agentgateway board breaks them down by tool.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 2, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": 0 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 4, + "x": 8, + "y": 20 + }, + "id": 17, + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum(rate(agentgateway_mcp_requests_total{cluster_id=~\"$cluster\"}[$__rate_interval]))", + "instant": false, + "legendFormat": "calls/s", + "range": true, + "refId": "A" + } + ], + "title": "Tool calls / s", + "type": "stat", + "repeat": "agentgateway_present", + "repeatDirection": "h" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Distinct agents that called a model in the window.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 0, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": 0 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 4, + "x": 12, + "y": 20 + }, + "id": 18, + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "count(count by (agent) (rate(agentgateway_gen_ai_client_token_usage_count{cluster_id=~\"$cluster\"}[$__rate_interval]) > 0)) or vector(0)", + "instant": false, + "legendFormat": "agents", + "range": true, + "refId": "A" + } + ], + "title": "Agents calling", + "type": "stat", + "repeat": "agentgateway_present", + "repeatDirection": "h" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Share of gateway responses that were not 2xx, every listener.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 2, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": 0 + }, + { + "color": "orange", + "value": 0.01 + }, + { + "color": "red", + "value": 0.05 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 4, + "x": 16, + "y": 20 + }, + "id": 19, + "options": { + "colorMode": "value", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum(rate(agentgateway_requests_total{status!~\"2..\", cluster_id=~\"$cluster\"}[$__rate_interval])) / sum(rate(agentgateway_requests_total{cluster_id=~\"$cluster\"}[$__rate_interval]))", + "instant": false, + "legendFormat": "errors", + "range": true, + "refId": "A" + } + ], + "title": "Gateway error ratio", + "type": "stat", + "repeat": "agentgateway_present", + "repeatDirection": "h" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "MCP servers muster knows, whatever their state.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "decimals": 0, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": 0 + } + ] + }, + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 4, + "x": 20, + "y": 20 + }, + "id": 20, + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "12.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "count(max by (mcpserver_name, mcpserver_namespace) (muster_mcpserver_state{cluster_type=\"management_cluster\", cluster_id=~\"$cluster\"})) or vector(0)", + "instant": false, + "legendFormat": "servers", + "range": true, + "refId": "A" + } + ], + "title": "MCP servers", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Which agents are spending. LLM usage has the cost and the models.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": 0 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 0, + "y": 24 + }, + "id": 21, + "options": { + "legend": { + "calcs": [ + "lastNotNull", + "max" + ], + "displayMode": "table", + "placement": "right", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "12.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (agent) (rate(agentgateway_gen_ai_client_token_usage_sum{cluster_id=~\"$cluster\"}[$__rate_interval]))", + "instant": false, + "legendFormat": "{{agent}}", + "range": true, + "refId": "A" + } + ], + "title": "Tokens per second by agent", + "type": "timeseries", + "repeat": "agentgateway_present", + "repeatDirection": "h" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Which tool backends the agents are using, from muster's dispatch metrics.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": 0 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 12, + "y": 24 + }, + "id": 22, + "options": { + "legend": { + "calcs": [ + "lastNotNull", + "max" + ], + "displayMode": "table", + "placement": "right", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "12.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (mcpserver_name) (rate(muster_downstream_tool_calls_total{cluster_id=~\"$cluster\"}[$__rate_interval]))", + "instant": false, + "legendFormat": "{{mcpserver_name}}", + "range": true, + "refId": "A" + } + ], + "title": "Tool calls per second by MCP server", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 31 + }, + "id": 23, + "panels": [], + "title": "Postgres", + "type": "row", + "repeat": "postgres_present", + "repeatDirection": "v" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "1 while the CloudNativePG cluster is in a healthy phase. The same series AgentPlatformPostgresClusterNotHealthy reads.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": 0 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 0, + "y": 32 + }, + "id": 24, + "options": { + "legend": { + "calcs": [ + "lastNotNull", + "max" + ], + "displayMode": "table", + "placement": "right", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "12.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "max by (namespace, name) (kube_cnpg_cluster_info{cluster_type=\"management_cluster\", namespace=~\"agent-platform|kagent\", phase=\"Cluster in healthy state\", cluster_id=~\"$cluster\"})", + "instant": false, + "legendFormat": "{{namespace}}/{{name}}", + "range": true, + "refId": "A" + } + ], + "title": "Cluster phase", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Connections waiting on a lock. Sustained growth is a query or a migration holding one.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": 0 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 8, + "y": 32 + }, + "id": 25, + "options": { + "legend": { + "calcs": [ + "lastNotNull", + "max" + ], + "displayMode": "table", + "placement": "right", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "12.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "max by (namespace, pod) (cnpg_backends_waiting_total{cluster_id=~\"$cluster\"})", + "instant": false, + "legendFormat": "{{pod}}", + "range": true, + "refId": "A" + } + ], + "title": "Backends waiting", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Seconds between the last archived and the last failed WAL archive. Above zero means archiving is failing, so point-in-time recovery is not available.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": 0 + } + ] + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 16, + "y": 32 + }, + "id": 26, + "options": { + "legend": { + "calcs": [ + "lastNotNull", + "max" + ], + "displayMode": "table", + "placement": "right", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "12.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "max by (namespace, pod) (cnpg_pg_stat_archiver_last_failed_time{cluster_id=~\"$cluster\"} - cnpg_pg_stat_archiver_last_archived_time{cluster_id=~\"$cluster\"})", + "instant": false, + "legendFormat": "{{pod}}", + "range": true, + "refId": "A" + } + ], + "title": "WAL archive lag", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 39 + }, + "id": 27, + "panels": [], + "title": "Valkey", + "type": "row", + "repeat": "valkey_present", + "repeatDirection": "v" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "muster keeps its OAuth sessions here. At the limit Valkey evicts, and an evicted session signs a person out.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": 0 + } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 0, + "y": 40 + }, + "id": 28, + "options": { + "legend": { + "calcs": [ + "lastNotNull", + "max" + ], + "displayMode": "table", + "placement": "right", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "12.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "max by (namespace) (redis_memory_used_bytes{cluster_type=\"management_cluster\", namespace=~\"agent-platform|kagent|mcp-kubernetes|mcp-capi|mcp-prometheus\", cluster_id=~\"$cluster\"}) / max by (namespace) (redis_memory_max_bytes{cluster_type=\"management_cluster\", namespace=~\"agent-platform|kagent|mcp-kubernetes|mcp-capi|mcp-prometheus\", cluster_id=~\"$cluster\"} > 0)", + "instant": false, + "legendFormat": "{{namespace}}", + "range": true, + "refId": "A" + } + ], + "title": "Memory used against the limit", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Clients holding a connection to Valkey.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": 0 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 8, + "y": 40 + }, + "id": 29, + "options": { + "legend": { + "calcs": [ + "lastNotNull", + "max" + ], + "displayMode": "table", + "placement": "right", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "12.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "max by (namespace) (redis_connected_clients{cluster_type=\"management_cluster\", namespace=~\"agent-platform|kagent|mcp-kubernetes|mcp-capi|mcp-prometheus\", cluster_id=~\"$cluster\"})", + "instant": false, + "legendFormat": "{{namespace}}", + "range": true, + "refId": "A" + } + ], + "title": "Connected clients", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "description": "Above zero means Valkey is dropping data to stay inside its limit, and a dropped session signs a person out.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "text", + "value": 0 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 8, + "x": 16, + "y": 40 + }, + "id": 30, + "options": { + "legend": { + "calcs": [ + "lastNotNull", + "max" + ], + "displayMode": "table", + "placement": "right", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "12.1.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "editorMode": "code", + "expr": "sum by (namespace) (rate(redis_evicted_keys_total{cluster_type=\"management_cluster\", namespace=~\"agent-platform|kagent|mcp-kubernetes|mcp-capi|mcp-prometheus\", cluster_id=~\"$cluster\"}[$__rate_interval]))", + "instant": false, + "legendFormat": "{{namespace}}", + "range": true, + "refId": "A" + } + ], + "title": "Keys evicted per second", + "type": "timeseries" + } + ], + "preload": false, + "refresh": "1m", + "schemaVersion": 41, + "tags": [ + "owner:team-bumblebee", + "component:agent-platform", + "topic:agents" + ], + "templating": { + "list": [ + { + "current": {}, + "hide": 0, + "includeAll": false, + "label": "Data source", + "multi": false, + "name": "datasource", + "options": [], + "query": "prometheus", + "refresh": 1, + "regex": "", + "skipUrlSync": false, + "type": "datasource" + }, + { + "current": {}, + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "definition": "label_values(gotk_resource_info{job=\"flux-ksm\", team=\"bumblebee\"}, cluster_id)", + "includeAll": true, + "label": "Cluster", + "multi": true, + "name": "cluster", + "options": [], + "query": { + "qryType": 1, + "query": "label_values(gotk_resource_info{job=\"flux-ksm\", team=\"bumblebee\"}, cluster_id)", + "refId": "StandardVariableQuery" + }, + "refresh": 2, + "regex": "", + "skipUrlSync": false, + "sort": 1, + "type": "query" + }, + { + "current": {}, + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "definition": "label_values(kube_cnpg_cluster_info{cluster_type=\"management_cluster\", namespace=~\"agent-platform|kagent\", cluster_id=~\"$cluster\"}, __name__)", + "hide": 2, + "includeAll": false, + "label": "Postgres present", + "multi": false, + "name": "postgres_present", + "options": [], + "query": { + "qryType": 1, + "query": "label_values(kube_cnpg_cluster_info{cluster_type=\"management_cluster\", namespace=~\"agent-platform|kagent\", cluster_id=~\"$cluster\"}, __name__)", + "refId": "StandardVariableQuery" + }, + "refresh": 2, + "regex": "", + "skipUrlSync": true, + "sort": 0, + "type": "query" + }, + { + "current": {}, + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "definition": "label_values(redis_memory_used_bytes{cluster_type=\"management_cluster\", namespace=~\"agent-platform|kagent|mcp-kubernetes|mcp-capi|mcp-prometheus\", cluster_id=~\"$cluster\"}, __name__)", + "hide": 2, + "includeAll": false, + "label": "Valkey present", + "multi": false, + "name": "valkey_present", + "options": [], + "query": { + "qryType": 1, + "query": "label_values(redis_memory_used_bytes{cluster_type=\"management_cluster\", namespace=~\"agent-platform|kagent|mcp-kubernetes|mcp-capi|mcp-prometheus\", cluster_id=~\"$cluster\"}, __name__)", + "refId": "StandardVariableQuery" + }, + "refresh": 2, + "regex": "", + "skipUrlSync": true, + "sort": 0, + "type": "query" + }, + { + "current": {}, + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "definition": "label_values(agentgateway_requests_total{cluster_id=~\"$cluster\"}, __name__)", + "hide": 2, + "includeAll": false, + "label": "agentgateway present", + "multi": false, + "name": "agentgateway_present", + "options": [], + "query": { + "qryType": 1, + "query": "label_values(agentgateway_requests_total{cluster_id=~\"$cluster\"}, __name__)", + "refId": "StandardVariableQuery" + }, + "refresh": 2, + "regex": "", + "skipUrlSync": true, + "sort": 0, + "type": "query" + } + ] + }, + "time": { + "from": "now-6h", + "to": "now" + }, + "timepicker": {}, + "timezone": "utc", + "title": "Overview", + "uid": "agent-platform-overview", + "weekStart": "" +} diff --git a/helm/agent-platform-connectivity/templates/dashboards/configmap.yaml b/helm/agent-platform-connectivity/templates/dashboards/configmap.yaml new file mode 100644 index 00000000..94114660 --- /dev/null +++ b/helm/agent-platform-connectivity/templates/dashboards/configmap.yaml @@ -0,0 +1,35 @@ +{{- /* The platform's own Grafana boards, one ConfigMap per board, discovered by +observability-operator through the app.giantswarm.io/kind label and placed by +the organization and folder annotations. Both annotation values are display +names that carry a space, so neither can be a label. + +Gated on the same resolved global.observability.metrics.serviceMonitor.enabled +as every monitor of this chart: the operator that picks a ConfigMap up is part +of the observability platform the monitors write to, and a board over series +nobody collects is worse than no board. + +The boards of a component come from that component's own chart (muster ships +its own, and agentgateway's is upstream's, vendored in the packaging chart). +These are the platform's own, across components, which no component chart can +render. */ -}} +{{- if and (include "agent-platform.serviceMonitor" .) .Values.dashboards.enabled }} +{{- $dashboards := .Values.dashboards }} +{{- range $path, $_ := .Files.Glob "dashboards/*.json" }} +{{- $name := base $path | trimSuffix ".json" }} +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ include "name" $ }}-dashboard-{{ $name }} + namespace: {{ $dashboards.namespace | default $.Release.Namespace }} + labels: + {{- include "labels.common" $ | nindent 4 }} + app.giantswarm.io/kind: dashboard + annotations: + observability.giantswarm.io/organization: {{ $dashboards.organization | quote }} + observability.giantswarm.io/folder: {{ $dashboards.folder | quote }} +data: + {{ base $path }}: |- +{{ $.Files.Get $path | indent 4 }} +{{- end }} +{{- end }} diff --git a/helm/agent-platform-connectivity/values.schema.json b/helm/agent-platform-connectivity/values.schema.json index 42b1fbe8..1a16217a 100644 --- a/helm/agent-platform-connectivity/values.schema.json +++ b/helm/agent-platform-connectivity/values.schema.json @@ -229,6 +229,24 @@ } } }, + "dashboards": { + "type": "object", + "additionalProperties": false, + "properties": { + "enabled": { + "type": "boolean" + }, + "folder": { + "type": "string" + }, + "namespace": { + "type": "string" + }, + "organization": { + "type": "string" + } + } + }, "dicebear": { "type": "object", "additionalProperties": true diff --git a/helm/agent-platform-connectivity/values.yaml b/helm/agent-platform-connectivity/values.yaml index 490846f3..f36fe6b2 100644 --- a/helm/agent-platform-connectivity/values.yaml +++ b/helm/agent-platform-connectivity/values.yaml @@ -652,6 +652,38 @@ hooks: # valkey-password: REPLACE_ME extraObjects: [] +# === The platform's own Grafana boards === +# One ConfigMap per JSON under the chart's dashboards/ directory +# (templates/dashboards/configmap.yaml), discovered by observability-operator +# through the app.giantswarm.io/kind: dashboard label and placed by the +# organization and folder annotations. +# +# Shipped: `overview.json` — Overview (uid agent-platform-overview), the folder's +# entry point. Its component-health panels read the same series as the alerts +# that page for them (agent-platform.rules.yml in giantswarm/prometheus-rules), +# so a panel and a page cannot disagree, and it carries the Postgres and Valkey +# rows, which have no board of their own. +# +# A component's own board comes from that component's chart: muster ships its +# own and agentgateway's is upstream's, vendored in the packaging chart. These +# are the platform's own, across components, which no component chart can render. +dashboards: + # The ConfigMaps render under the resolved + # global.observability.metrics.serviceMonitor.enabled as well: the operator + # that picks them up is part of the observability platform the monitors write + # to, and a board over series nobody collects is worse than no board. + enabled: true + # Namespace for the ConfigMaps. Empty renders into the release namespace. + namespace: "" + # The Grafana organization display name, which must match an existing + # organization. `Shared Org` is the one every logged-in customer reaches as a + # Viewer and it carries the observability data of Giant Swarm managed + # components; `Giant Swarm` is staff-only. + organization: "Shared Org" + # The Grafana folder, `/`-separated to nest. One folder for every board of + # the platform, so an operator opens one place. + folder: "Agent Platform" + # Dicebear avatar service values. Consumed by the dicebear component chart and # forwarded here via forwardAllValues; declared so the schema accepts the block. dicebear: {} # @schema skipProperties: true; additionalProperties: true diff --git a/helm/agent-platform/README.md b/helm/agent-platform/README.md index 4f993558..222e5664 100644 --- a/helm/agent-platform/README.md +++ b/helm/agent-platform/README.md @@ -913,6 +913,10 @@ The map is merged into each component's own `nodeSelector` (`muster.nodeSelector | kagent.uiRoute.backendTrafficPolicy.labels | object | `{}` | | | kagent.modelConfigs | list | `[]` | | | kagent.remoteMcpServers | list | `[]` | | +| dashboards.enabled | bool | `true` | | +| dashboards.namespace | string | `""` | | +| dashboards.organization | string | `"Shared Org"` | | +| dashboards.folder | string | `"Agent Platform"` | | | postgres.enabled | bool | `false` | | | postgres.namespace | string | `"kagent"` | | | postgres.clusterName | string | `"kagent-pg"` | | diff --git a/helm/agent-platform/values.schema.json b/helm/agent-platform/values.schema.json index c0f98953..6816bc9e 100644 --- a/helm/agent-platform/values.schema.json +++ b/helm/agent-platform/values.schema.json @@ -815,6 +815,24 @@ } } }, + "dashboards": { + "type": "object", + "additionalProperties": false, + "properties": { + "enabled": { + "type": "boolean" + }, + "folder": { + "type": "string" + }, + "namespace": { + "type": "string" + }, + "organization": { + "type": "string" + } + } + }, "dicebear": { "type": "object", "additionalProperties": true diff --git a/helm/agent-platform/values.yaml b/helm/agent-platform/values.yaml index 1d458b47..0d26470e 100644 --- a/helm/agent-platform/values.yaml +++ b/helm/agent-platform/values.yaml @@ -2968,6 +2968,32 @@ kagent: # @schema skipProperties: true; additionalProperties: true # tokenSecret: "" # optional: pre-existing Secret with key "token" # description: "external MCP server" +# === The platform's own Grafana boards === +# Forwarded to the connectivity chart, which renders one ConfigMap per board +# under its dashboards/ directory. Shipped: Overview (uid +# agent-platform-overview), the folder's entry point — its component-health +# panels read the same series as the alerts that page for them, and it carries +# the Postgres and Valkey rows, which have no board of their own. +# +# A component's own board comes from that component's chart: muster ships its +# own (muster.observability.grafanaDashboard above) and agentgateway's is +# upstream's, vendored in the packaging chart +# (agentgateway.monitoring.grafanaDashboard). Every one of them lands in the +# same organization and folder as these. +dashboards: + # The ConfigMaps render under the resolved + # global.observability.metrics.serviceMonitor.enabled as well: the operator + # that picks them up is part of the observability platform the monitors write + # to, and a board over series nobody collects is worse than no board. + enabled: true + namespace: "" + # Must match an existing Grafana organization's display name. `Shared Org` is + # the one every logged-in customer reaches as a Viewer; `Giant Swarm` is + # staff-only. + organization: "Shared Org" + # `/`-separated to nest. One folder for every board of the platform. + folder: "Agent Platform" + # Platform Postgres — CloudNativePG Cluster for kagent (and future Klaus sessions). # The CNPG operator and its CRDs are a cluster-level prerequisite (like Gateway # API CRDs): they must be installed before enabling this. Flip `enabled: false` diff --git a/tests/verify-target.py b/tests/verify-target.py index 6237d91e..ecb1b765 100755 --- a/tests/verify-target.py +++ b/tests/verify-target.py @@ -303,6 +303,43 @@ def strip(render: str) -> str: return h, strip(there) +# giantswarm/giantswarm#36711: the platform's own board ConfigMaps and the +# dashboards block that configures them. GOLDEN_REF has neither — its schema has +# no `dashboards` key either, so this cannot be held with --set — so the block is +# dropped from the meta renders' forwarded values and the ConfigMap documents +# from the connectivity renders. Both go with the line. +DASHBOARDS_KEY = re.compile(r"^(\s+)dashboards:\s*$") +DASHBOARDS_CONFIGMAP = "# Source: agent-platform-connectivity/templates/dashboards/configmap.yaml" + + +def drop_mapping(render: str, key: re.Pattern) -> str: + """The render without the mapping `key` names: its line and every line + indented deeper than it.""" + out, lines, i = [], render.split("\n"), 0 + while i < len(lines): + if m := key.match(lines[i]): + indent = len(m.group(1)) + i += 1 + while i < len(lines) and (not lines[i].strip() + or len(lines[i]) - len(lines[i].lstrip(" ")) > indent): + i += 1 + continue + out.append(lines[i]) + i += 1 + return "\n".join(out) + + +def hold_dashboards(here: str, there: str, is_meta: bool) -> tuple: + """The two renders with the platform's own boards left out of the comparison.""" + def strip(render: str) -> str: + if is_meta: + return drop_mapping(render, DASHBOARDS_KEY) + return "\n---\n".join(d for d in render.split("\n---\n") if DASHBOARDS_CONFIGMAP not in d) + if (h := strip(here)) != here or strip(there) != there: + print("note: #36711 hold — the platform's own dashboards block and board ConfigMaps are left out of the golden comparison") + return h, strip(there) + + def check_golden(meta: str, connectivity: str) -> None: ref = os.environ.get("GOLDEN_REF", "origin/main") if not ref: @@ -354,6 +391,7 @@ def check_golden(meta: str, connectivity: str) -> None: for label, chart, flags in shapes: here = helm(chart, flags) there = helm(os.path.join(tree, chart), [f.replace(f"{meta}/", f"{tree}/{meta}/") for f in flags]) + here, there = hold_dashboards(here, there, chart == meta) if chart == meta: here, there = drop_new_roster_entries(here, there) here, there = hold_llmd_only(here, there)