feat(otel): db-span enrichment, IMeterFactory metrics, sub-second buckets

docs/mediator/adr/0005-opentelemetry-instrumentation.md

@@ -17,7 +17,7 @@ description: "Design of the OpenTelemetry companion package: automatic distribut
 
 ## Status
 
-**Released in v1.0.0**
+**Released in v1.0.0** — amended in v1.1.0-rc.2 (see [Amendment](#amendment--v110-rc2-database-enrichment--factory-based-metrics) below).
 
 ## Context
 
@@ -537,9 +537,36 @@ added if there is demand, but the default is full-enumeration spans.
 
 ---
 
+## Amendment — v1.1.0-rc.2 (database enrichment + factory-based metrics)
+
+Three capabilities were added after the original release. All are additive: `AddMediatorInstrumentation()` is unchanged at the call site, so existing consumers get them with no code change.
+
+### A1. Automatic database-span enrichment
+
+**Decision.** `AddMediatorInstrumentation()` on the `TracerProviderBuilder` now also registers a `BaseProcessor<Activity>` (`DatabaseSpanEnrichmentProcessor`). For any database client span (one carrying `db.system`) flowing through the same provider, it derives a redaction-safe `db.operation.name`, `db.sql.table` and `db.stored_procedure.name` from `db.statement` / `db.query.text` — but only when the underlying driver emitted just the raw statement (e.g. an older Npgsql).
+
+**Why.** Without those structured attributes, every query on a connection collapses into one `"{system} → {host}"` dependency in the Pipeline Explorer flame. Parsing happens **in-process** — where the application owns its own SQL — via `SqlStatementParser`, and only the verb plus a single bare identifier are surfaced; the raw statement, parameters and row values are never copied onto the span. This lets the trace consumer (the Pipeline Explorer importer) keep its "never read `db.statement`" boundary while still attributing time per operation (`SELECT` vs `INSERT` vs a stored procedure by name).
+
+**Properties.** Strictly additive — an attribute already supplied by native instrumentation is never overwritten. Gated on a cheap `db.system` check, so non-DB spans are untouched. AOT/trim-safe (no reflection). New files: `DatabaseSpanEnrichmentProcessor.cs`, `SqlStatementParser.cs`.
+
+### A2. Metrics created from `IMeterFactory` (replaces §3.2's static `Meter`)
+
+**Decision.** The three instruments (§3.2) are now created from the DI `IMeterFactory` via a singleton `MediatorMetrics`, registered by `AddMediatorInstrumentation()` — which also calls `services.AddMetrics()` so the factory is present even outside a Generic Host. The meter name is unchanged (`"DSoftStudio.Mediator"`), so `AddMeter("DSoftStudio.Mediator")` still subscribes. The `ActivitySource` stays static + named (the .NET convention — there is no per-DI ActivitySource factory).
+
+**Why.** Microsoft prescribes `IMeterFactory` for DI-aware libraries; a static `Meter` is an anti-pattern because it cannot be isolated per service collection, which cross-contaminates parallel tests and multiple hosts in one process. Adds a `Microsoft.Extensions.Diagnostics` package reference for `AddMetrics()`. New file: `MediatorMetrics.cs`.
+
+### A3. Histogram bucket advice
+
+**Decision.** `mediator.request.duration` carries explicit sub-second `InstrumentAdvice<double>` bucket boundaries (`0.0005 … 5 s`).
+
+**Why.** The OpenTelemetry default histogram buckets (`[0, 5, 10, 25, …] s`) collapse every sub-5-second request into the first bucket, making p95/p99 meaningless for millisecond-scale mediator requests. The advice is honoured by the OpenTelemetry .NET SDK (≥ 1.10) as the default boundaries.
+
+---
+
 ## Document History
 
 | Date       | Version | Changes |
 |------------|---------|---------|
 | —          | Draft   | Initial ADR with instrumentation design |
 | 2026-03-15 | v1.0.0  | Released as DSoftStudio.Mediator.OpenTelemetry companion package |
+| 2026-06-22 | v1.1.0-rc.2 | Amendment: automatic database-span enrichment, `IMeterFactory`-based metrics, sub-second histogram bucket advice |

docs/mediator/integrations/opentelemetry.md

@@ -47,7 +47,8 @@ builder.Services.AddOpenTelemetry()
 | Signal | Details |
 |---|---|
 | **Tracing** | One span per `Send()`, `Publish()`, `CreateStream()` with CQRS-aware naming (`CreateUser command`, `GetUsers query`). Per-handler child spans for notifications. |
-| **Metrics** | `mediator.request.duration` (histogram), `mediator.request.active` (up-down counter), `mediator.request.errors` (counter with `error.type` tag). |
+| **Metrics** | `mediator.request.duration` (histogram, **seconds**, with sub-second bucket boundaries), `mediator.request.active` (up-down counter), `mediator.request.errors` (counter with `error.type` tag). Instruments are created from the DI `IMeterFactory`. |
+| **Database enrichment** | Calling `AddMediatorInstrumentation()` on the `TracerProviderBuilder` also tags database spans with a redaction-safe `db.operation.name` / `db.sql.table` / `db.stored_procedure.name`, so each query (`SELECT`, `INSERT`, `CALL`…) is a distinct dependency instead of one aggregated row. No configuration. |
 | **Zero-cost when unused** | `HasListeners()` short-circuit adds ~1 ns when no exporter is configured. |
 
 ## Configuration

src/DSoftStudio.Mediator.OpenTelemetry/DSoftStudio.Mediator.OpenTelemetry.csproj

@@ -9,7 +9,7 @@
 
 		<!-- NuGet -->
 		<PackageId>DSoftStudio.Mediator.OpenTelemetry</PackageId>
-		<Version>1.1.0-rc.1</Version>
+		<Version>1.1.0-rc.2</Version>
 
 		<Authors>DSoftStudio</Authors>
 		<Company>DSoftStudio</Company>
@@ -66,6 +66,9 @@
 
 	<ItemGroup>
 		<PackageReference Include="OpenTelemetry" Version="1.16.0" />
+		<!-- IMeterFactory + AddMetrics(): create metric instruments from DI rather than a static Meter
+		     (Microsoft's prescribed pattern for a DI-aware instrumentation library). -->
+		<PackageReference Include="Microsoft.Extensions.Diagnostics" Version="9.0.0" />
 	</ItemGroup>
 
 	<ItemGroup>

src/DSoftStudio.Mediator.OpenTelemetry/DatabaseSpanEnrichmentProcessor.cs

@@ -0,0 +1,62 @@
+// Copyright (c) DSoftStudio. All rights reserved.
+// Licensed under the MIT License. See LICENSE in the project root for license information.
+
+using System.Diagnostics;
+
+namespace DSoftStudio.Mediator.OpenTelemetry;
+
+/// <summary>
+/// Enriches database client spans (those carrying <c>db.system</c>) with a redaction-safe
+/// <c>db.operation.name</c> and <c>db.sql.table</c> derived from the SQL statement, so that
+/// downstream tooling (e.g. the Pipeline Explorer) can attribute time to the <em>specific</em>
+/// operation — distinguishing a <c>SELECT</c> from an <c>INSERT</c> on the same connection instead
+/// of collapsing every query into a single <c>"{system} → {host}"</c> dependency row.
+/// </summary>
+/// <remarks>
+/// <para>
+/// Registered automatically by <c>AddMediatorInstrumentation()</c> on the
+/// <see cref="global::OpenTelemetry.Trace.TracerProviderBuilder"/>; no configuration is required.
+/// It runs in-process, where the application owns its own SQL, so reading <c>db.statement</c> here
+/// never crosses a trust boundary — and only the verb and a single bare table identifier are
+/// copied onto the span. The raw statement, parameters and row values are never propagated, which
+/// keeps the import-side contract intact (the trace consumer never has to read <c>db.statement</c>).
+/// </para>
+/// <para>
+/// The enrichment is strictly additive: an attribute already supplied by native instrumentation
+/// (a newer Npgsql / EF Core that emits <c>db.operation.name</c>) is never overwritten.
+/// </para>
+/// </remarks>
+internal sealed class DatabaseSpanEnrichmentProcessor : global::OpenTelemetry.BaseProcessor<Activity>
+{
+    public override void OnEnd(Activity activity)
+    {
+        // Cheap gate: only database client spans carry db.system / db.system.name.
+        if (activity.GetTagItem("db.system") is null && activity.GetTagItem("db.system.name") is null)
+            return;
+
+        bool hasOperation = activity.GetTagItem("db.operation.name") is not null
+                         || activity.GetTagItem("db.operation") is not null;
+        bool hasTable = activity.GetTagItem("db.sql.table") is not null
+                     || activity.GetTagItem("db.collection.name") is not null;
+        bool hasProcedure = activity.GetTagItem("db.stored_procedure.name") is not null;
+
+        if (hasOperation && hasTable && hasProcedure)
+            return; // Native instrumentation already described the operation — nothing to add.
+
+        var statement = (activity.GetTagItem("db.query.text") as string)
+                     ?? (activity.GetTagItem("db.statement") as string);
+        if (string.IsNullOrWhiteSpace(statement))
+            return;
+
+        if (!hasOperation && SqlStatementParser.Operation(statement) is { } operation)
+            activity.SetTag("db.operation.name", operation);
+
+        // DML target table (SELECT/INSERT/UPDATE/DELETE) — distinguishes queries on different tables.
+        if (!hasTable && SqlStatementParser.Table(statement) is { } table)
+            activity.SetTag("db.sql.table", table);
+
+        // Stored-procedure / function name (CALL/EXEC) — distinguishes one procedure from another.
+        if (!hasProcedure && SqlStatementParser.Procedure(statement) is { } procedure)
+            activity.SetTag("db.stored_procedure.name", procedure);
+    }
+}

src/DSoftStudio.Mediator.OpenTelemetry/InstrumentedNotificationPublisher.cs

@@ -11,7 +11,7 @@ namespace DSoftStudio.Mediator.OpenTelemetry;
 /// Decorator that wraps an <see cref="INotificationPublisher"/> with distributed tracing
 /// and metrics. Creates a parent span for the publish operation and per-handler child spans.
 /// </summary>
-internal sealed class InstrumentedNotificationPublisher(INotificationPublisher inner, MediatorInstrumentationOptions options) : INotificationPublisher
+internal sealed class InstrumentedNotificationPublisher(INotificationPublisher inner, MediatorInstrumentationOptions options, MediatorMetrics? metrics) : INotificationPublisher
 {
     private static readonly ActivitySource Source = MediatorInstrumentation.ActivitySource;
     private static readonly ConcurrentDictionary<Type, string> HandlerSpanNames = new();
@@ -23,7 +23,7 @@ public async Task Publish<TNotification>(
         where TNotification : INotification
     {
         bool tracingActive = options.EnableTracing && Source.HasListeners();
-        bool metricsActive = options.EnableMetrics && MediatorInstrumentation.RequestDuration.Enabled;
+        bool metricsActive = options.EnableMetrics && metrics is not null && metrics.RequestDuration.Enabled;
 
         if (!tracingActive && !metricsActive)
         {
@@ -65,7 +65,7 @@ public async Task Publish<TNotification>(
                 { "mediator.request.kind", MediatorNotificationMetadata<TNotification>.RequestKind }
             };
 
-            MediatorInstrumentation.RequestActive.Add(1, metricTags);
+            metrics!.RequestActive.Add(1, metricTags);
             startTimestamp = Stopwatch.GetTimestamp();
         }
 
@@ -96,7 +96,7 @@ public async Task Publish<TNotification>(
                     { "error.type", ex.GetType().FullName! }
                 };
 
-                MediatorInstrumentation.RequestErrors.Add(1, errorTags);
+                metrics!.RequestErrors.Add(1, errorTags);
             }
 
             throw;
@@ -108,8 +108,8 @@ public async Task Publish<TNotification>(
             if (metricsActive)
             {
                 var elapsed = Stopwatch.GetElapsedTime(startTimestamp);
-                MediatorInstrumentation.RequestDuration.Record(elapsed.TotalSeconds, metricTags);
-                MediatorInstrumentation.RequestActive.Add(-1, metricTags);
+                metrics!.RequestDuration.Record(elapsed.TotalSeconds, metricTags);
+                metrics!.RequestActive.Add(-1, metricTags);
             }
         }
     }

src/DSoftStudio.Mediator.OpenTelemetry/MediatorInstrumentation.cs

@@ -2,38 +2,25 @@
 // Licensed under the MIT License. See LICENSE in the project root for license information.
 
 using System.Diagnostics;
-using System.Diagnostics.Metrics;
 
 namespace DSoftStudio.Mediator.OpenTelemetry;
 
 /// <summary>
-/// Provides the <see cref="ActivitySource"/> and <see cref="Meter"/> used by the mediator instrumentation.
+/// Provides the <see cref="ActivitySource"/> used by the mediator instrumentation, plus the shared name/version.
+/// The metric instruments live on <see cref="MediatorMetrics"/> (created from the DI <c>IMeterFactory</c>).
 /// </summary>
 public static class MediatorInstrumentation
 {
     /// <summary>
-    /// The name used for both the <see cref="ActivitySource"/> and <see cref="Meter"/>.
+    /// The name used for both the <see cref="ActivitySource"/> and the metrics <c>Meter</c>.
     /// Use this constant when manually calling <c>AddSource()</c> or <c>AddMeter()</c>.
     /// </summary>
     public const string SourceName = "DSoftStudio.Mediator";
 
-    private static readonly string Version = typeof(MediatorInstrumentation)
+    /// <summary>The instrumentation version, stamped onto the <see cref="ActivitySource"/> and the metrics meter.</summary>
+    internal static readonly string Version = typeof(MediatorInstrumentation)
         .Assembly.GetName().Version?.ToString() ?? "0.0.0";
 
+    // The ActivitySource stays static + named (the .NET convention — there is no per-DI ActivitySource factory).
     internal static readonly ActivitySource ActivitySource = new(SourceName, Version);
-    internal static readonly Meter Meter = new(SourceName, Version);
-
-    // ── Metric instruments ─────────────────────────────────────────────
-
-    internal static readonly Histogram<double> RequestDuration =
-        Meter.CreateHistogram<double>("mediator.request.duration", "s",
-            "Time from behavior entry to handler completion");
-
-    internal static readonly UpDownCounter<long> RequestActive =
-        Meter.CreateUpDownCounter<long>("mediator.request.active", "{request}",
-            "Number of in-flight requests");
-
-    internal static readonly Counter<long> RequestErrors =
-        Meter.CreateCounter<long>("mediator.request.errors", "{error}",
-            "Count of failed requests");
 }

src/DSoftStudio.Mediator.OpenTelemetry/MediatorMetrics.cs

@@ -0,0 +1,56 @@
+// Copyright (c) DSoftStudio. All rights reserved.
+// Licensed under the MIT License. See LICENSE in the project root for license information.
+
+using System.Diagnostics.Metrics;
+
+namespace DSoftStudio.Mediator.OpenTelemetry;
+
+/// <summary>
+/// Owns the mediator metric instruments. The <see cref="Meter"/> is created from the DI
+/// <see cref="IMeterFactory"/> — the pattern Microsoft prescribes for a DI-aware library, because a
+/// <c>static</c> <see cref="Meter"/> cannot be isolated per service collection (it leaks measurements across
+/// parallel tests and across hosts in the same process). Registered as a singleton by
+/// <c>AddMediatorInstrumentation()</c>; the meter name is <see cref="MediatorInstrumentation.SourceName"/>, so an
+/// app still subscribes with the same <c>AddMeter("DSoftStudio.Mediator")</c> call.
+/// </summary>
+public sealed class MediatorMetrics
+{
+    // Explicit sub-second bucket boundaries (in SECONDS) so the duration histogram yields meaningful p50/p95/p99
+    // for millisecond-scale mediator requests. Without this, the OpenTelemetry default buckets ([0, 5, 10, 25, …]
+    // seconds) collapse every sub-5-second request into the first bucket → useless percentiles. Supplied via
+    // InstrumentAdvice, which the OpenTelemetry .NET SDK (>= 1.10) honours as the default boundaries.
+    private static readonly double[] DurationSecondsBuckets =
+        [0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5];
+
+    public MediatorMetrics(IMeterFactory meterFactory)
+    {
+        ArgumentNullException.ThrowIfNull(meterFactory);
+
+        var meter = meterFactory.Create(MediatorInstrumentation.SourceName, MediatorInstrumentation.Version);
+
+        RequestDuration = meter.CreateHistogram<double>(
+            name: "mediator.request.duration",
+            unit: "s",
+            description: "Time from behavior entry to handler completion",
+            advice: new InstrumentAdvice<double> { HistogramBucketBoundaries = DurationSecondsBuckets });
+
+        RequestActive = meter.CreateUpDownCounter<long>(
+            name: "mediator.request.active",
+            unit: "{request}",
+            description: "Number of in-flight requests");
+
+        RequestErrors = meter.CreateCounter<long>(
+            name: "mediator.request.errors",
+            unit: "{error}",
+            description: "Count of failed requests");
+    }
+
+    /// <summary>Histogram of request durations in SECONDS (record with <c>elapsed.TotalSeconds</c>).</summary>
+    public Histogram<double> RequestDuration { get; }
+
+    /// <summary>In-flight request count (+1 on entry, −1 on completion).</summary>
+    public UpDownCounter<long> RequestActive { get; }
+
+    /// <summary>Count of failed requests, tagged with <c>error.type</c>.</summary>
+    public Counter<long> RequestErrors { get; }
+}

src/DSoftStudio.Mediator.OpenTelemetry/MediatorMetricsBehavior.cs

@@ -9,7 +9,7 @@ namespace DSoftStudio.Mediator.OpenTelemetry;
 /// <summary>
 /// Pipeline behavior that records metrics (duration, active count, errors) for mediator requests.
 /// </summary>
-public sealed class MediatorMetricsBehavior<TRequest, TResponse>(MediatorInstrumentationOptions options) : IPipelineBehavior<TRequest, TResponse>
+public sealed class MediatorMetricsBehavior<TRequest, TResponse>(MediatorInstrumentationOptions options, MediatorMetrics metrics) : IPipelineBehavior<TRequest, TResponse>
     where TRequest : IRequest<TResponse>
 {
 
@@ -18,7 +18,7 @@ public async ValueTask<TResponse> Handle(
         IRequestHandler<TRequest, TResponse> next,
         CancellationToken cancellationToken)
     {
-        if (!options.EnableMetrics || !MediatorInstrumentation.RequestDuration.Enabled)
+        if (!options.EnableMetrics || !metrics.RequestDuration.Enabled)
             return await next.Handle(request, cancellationToken);
 
         if (options.Filter is not null && !options.Filter(typeof(TRequest)))
@@ -30,7 +30,7 @@ public async ValueTask<TResponse> Handle(
             { "mediator.request.kind", MediatorTelemetryMetadata<TRequest, TResponse>.RequestKind }
         };
 
-        MediatorInstrumentation.RequestActive.Add(1, tags);
+        metrics.RequestActive.Add(1, tags);
         var startTimestamp = Stopwatch.GetTimestamp();
 
         try
@@ -46,14 +46,14 @@ public async ValueTask<TResponse> Handle(
                 { "error.type", ex.GetType().FullName! }
             };
 
-            MediatorInstrumentation.RequestErrors.Add(1, errorTags);
+            metrics.RequestErrors.Add(1, errorTags);
             throw;
         }
         finally
         {
             var elapsed = Stopwatch.GetElapsedTime(startTimestamp);
-            MediatorInstrumentation.RequestDuration.Record(elapsed.TotalSeconds, tags);
-            MediatorInstrumentation.RequestActive.Add(-1, tags);
+            metrics.RequestDuration.Record(elapsed.TotalSeconds, tags);
+            metrics.RequestActive.Add(-1, tags);
         }
     }
 }