Expose native Spanner metrics with an explicit exporter lifecycle

[apstndb]

Scope

Keep this issue for metrics, replacing the obsolete pool-statistics proposal #289/#403. The pinned Go client has no usable legacy session pool, and mycli explicitly disables native export in session.go.

Updated implementation plan

• Decide between the SDK's native Cloud Monitoring export and a caller-owned ClientMetricsProvider; expose an opt-in startup policy for the chosen path. The SDK documents that DisableNativeMetrics disables native Cloud Monitoring export but does not disable a caller-owned provider.

• OTEL_* environment variables alone are not an initialized exporter/provider. Configure the supported exporter/provider deliberately, document its destination and startup behavior, and retain ownership for shutdown.

• Reuse one lifecycle across session replacement/USE/DETACH. Preserve other embedded client config fields. Keep metrics and the new end-to-end tracing plan independently enabled.

• Do not add a SHOW METRICS snapshot UI or recreate removed session pool statistics in this PR. Avoid query/parameter/row values in metric labels.

Acceptance

• Disabled mode preserves current behavior; enabled configuration reaches the actual client factory.

• An in-memory metric reader verifies collection/export wiring without Cloud credentials; provider replacement/shutdown has no leak or duplicate exporter.

• Document native Cloud Monitoring versus caller export and any optional managed-service acceptance check.

References: Go SQL native metrics property, Java observability configuration, mycli session.go:60.

Verification and delivery

Plan refreshed on 2026-09-14 against a8fa2418b65af9b889cf92e02190366b3e6e02ea. This body supersedes obsolete implementation suggestions in earlier comments; history is retained. Tracked with #47.

For each implementation PR, capture focused test output to a file and check the exit code, then run make check before pushing; run make check-race for transaction concurrency/lifecycle changes. Keep the coverage gate at 80%. Update user docs and the compatibility matrix with the resulting behavior. No database/sql migration or unrelated dependency upgrade.

Accepted implementation contract (2026-09-14)

Reviewed against current main e0dfd9cfb82922fd4d253609a2bcaf9381d14dea, Go Spanner v1.95.0 and OTel v1.44.0. This section resolves the design choices above and supersedes conflicting suggestions in earlier comments.

• Choose a caller-owned provider using ClientMetricsProvider and spanner.ClientMetricsMeterProviderOptions(). Keep native Cloud Monitoring disabled when this opt-in is enabled. Do not install a global MeterProvider; the SDK views must stay dedicated to Spanner client metrics.
• Add startup-only flags/TOML --spanner-metrics-exporter=off|otlp (default off) and --spanner-metrics-endpoint, with read-only CLI_SPANNER_METRICS_EXPORTER and CLI_SPANNER_METRICS_ENDPOINT. SQL SET/SET LOCAL/RESET cannot change them. Off mode initializes no exporter/provider and preserves existing client configuration, including caller/embedded overrides; do not erase preexisting injected configuration merely to satisfy a nil-provider assertion.
• Use one OTLP HTTP/protobuf metric exporter matching the existing OTel v1.44.0 family. The endpoint must be an explicit absolute http/https URL with host; no userinfo, query or fragment. Interpret a missing/root path as /v1/metrics, otherwise preserve the explicit metrics path. http is explicit plaintext collector transport; https verifies the collector using its normal TLS configuration. Do not reuse Spanner endpoint, Google credentials or custom Spanner certificate options for the collector. Reject off plus a nonempty endpoint and otlp without an endpoint before constructing an exporter or clients. Do not accept an OTEL environment endpoint as an implicit destination or silently fall back on malformed URLs. Explicit endpoint/path/scheme must override conflicting exporter environment defaults. Other standard exporter settings may apply only after explicit opt-in; document that distinction without adding more CLI flags.
• Initialize one dedicated MeterProvider and PeriodicReader before the first Spanner client. Inject the provider into a copied client config while preserving embedded config fields and pointer ownership. Reuse it for USE/DETACH/reconnect/RecreateClient. Client/session replacement must not shut it down. Export only the SDK spanner/client/* client metrics; do not add SQL, parameter or row values as labels. Include database/endpoint and ordinary SDK operational labels in the documented data boundary.
• Own cleanup in runWithOutput, with no per-session exporter. All exits after initialization, including failed startup, batch success/error, MCP return and interactive exit, must release it. Stop/close the owned CLI clients before final metric export where needed; preserve already-closed paths safely. Use one fresh, total five-second cleanup context for flush/shutdown, independent of the cancelled command context. Cleanup diagnostics go to stderr and preserve the original exit result. Prove the final batch can be exported and shutdown stops the reader; avoid a separate five-second budget per cleanup step.
• Do not implement tracing in this PR or add an unused tracer slot/framework. Design opt-in OpenTelemetry tracing including Spanner end-to-end spans #967 will reuse the concrete ownership/cleanup boundary after this lands. Its design must separately suppress db.statement, which the pinned SDK adds to recording spans; metrics enablement does not authorize trace export.
• Full and slim share this core implementation. Preserve the SDK's suppression of caller metrics when SPANNER_EMULATOR_HOST is set and document it. No SHOW METRICS, legacy session-pool metrics or second export sink.

Acceptance requires an in-memory reader plus actual fake Spanner RPC proof, and a local fake HTTP OTLP receiver proving encoded metric export and the explicit destination/path. Assert disabled mode performs no initialization even with OTEL environment variables, malformed/missing/conflicting endpoint rejection, no sensitive SQL/params/rows in emitted attributes, source-config immutability, provider identity across replacement, and bounded cleanup on success/error/cancellation. Exercise conflicting OTEL endpoint/scheme defaults to prove the CLI-selected destination wins. Use no Cloud credentials or remote collectors. Distinguish source/local proof from optional managed-service evidence.

Promote only the already pinned OTel packages needed directly and add the matching OTLP HTTP metric exporter/transitive requirements; no unrelated upgrades. Capture outputs and exit codes, run full make check before each push and make check-race for lifecycle/concurrency, keep the CI coverage floor at 80%, and deliver one focused PR with user docs and compatibility updates.