Grafana Agent: scrape eks-agent-platform operator metrics over authenticated HTTPS

[stxkxs]

Context

The eks-agent-platform operator's metrics endpoint is now secure by default (metrics.secure: true): it serves over HTTPS and requires every scrape to authenticate + authorize via controller-runtime's WithAuthenticationAndAuthorization filter (TokenReview + a SubjectAccessReview on the /metrics nonResourceURL). This closes a STRIDE information-disclosure finding where the metrics endpoint was plaintext HTTP gated only by NetworkPolicy.

Consequence for eks-gitops

The prod scrape path here is the Grafana Agent (Alloy) scraping by pod annotation (prometheus.io/scrape), which is plaintext and unauthenticated. That path can no longer read the operator's metrics, and the operator chart now suppresses those prometheus.io/* annotations while metrics.secure is on. On kube-prometheus-stack clusters (kx) the operator's ServiceMonitor already carries the scheme: https + insecureSkipVerify + bearer-token config, so the ServiceMonitor path works; the Grafana-Agent-scraped clusters need their scrape config updated.

Ask

Teach the Grafana Agent to scrape the operator's secure metrics endpoint over https with a bearer token. Either:

• Bind the agent's scrape ServiceAccount to the operator chart's <release>-metrics-reader ClusterRole (grants get on the /metrics nonResourceURL) and present that SA token with tls_config { insecure_skip_verify = true } (the endpoint uses an in-memory self-signed cert); or
• Scrape the operator via the authenticated ServiceMonitor path instead of pod annotations.

Until this lands, deploying the operator with metrics.secure: true to a Grafana-Agent-scraped cluster leaves its controller-runtime metrics unscraped — fail-safe (metrics not exposed rather than leaked), but a monitoring gap. Interim escape hatch: set metrics.secure: false on that cluster to restore the plaintext annotation path, at the cost of reopening the info-disclosure gap.

Refs

• eks-agent-platform operator chart: charts/operator/values.yaml (metrics.secure), charts/operator/templates/rbac.yaml (*-metrics-reader ClusterRole + system:auth-delegator binding), charts/operator/templates/servicemonitor.yaml
• Filed from eks-agent-platform quality-remediation Target 29 (metrics authn/authz)

[stxkxs]

Closing — the scrape exists and the agent this issue names does not.

addons/observability/otel-agent/values.yaml:194-198 runs a eks-agent-platform-operator job over scheme: https with bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token, which is what this asked for.

The Grafana Agent half is moot: 9e0e491 (2026-07-24) deleted addons/observability/alloy/* and applicationsets/addons-alloy.yaml, and a grep for alloy|grafana.agent across the tracked tree returns nothing.

Worth recording that the OTLP migration did not carry this across — the operator's metrics reached nothing from 2026-07-24 until f887241 closed it on 2026-08-07, while two in-repo comments asserted the opposite.

One residual, tracked separately rather than here: eks-agent-platform/docs/runbooks/deploy-end-to-end.md:67 still references the deleted grafana-agent addon.