Configurable wasm-shim span buffer size

[javo8a]

Upstream feature request: configurable wasm-shim span buffer size

Use this document to file issues or PRs against Kuadrant when load on the MaaS gateway triggers:

wasm log ... Span buffer full (100), dropping oldest span

Problem

The Kuadrant wasm-shim uses a global BufferingSpanProcessor with a hardcoded capacity of 100 spans (src/tracing/processor.rs):

pub fn new() -> Self {
    Self::with_capacity(100)
}

Under concurrent gateway load, each HTTP request creates multiple trace spans (auth, rate-limit check/report, gRPC decorators). All in-flight requests share one buffer; spans are drained on request teardown via ExportTracesTask OTLP export. When span production outpaces export, the buffer fills and oldest spans are dropped.

There is no configuration surface today in:

• Kuadrant CR (spec.observability)
• WasmPlugin pluginConfig.observability
• Envoy / Istio Wasm filter settings

Proposed solution

1. wasm-shim

Add optional config field (name bikeshedding welcome):

observability:
  defaultLevel: WARN
  tracing:
    service: tracing-service
    spanBufferSize: 1000   # new, default 100

Wire in init_observability() / get_span_processor():

• On first init, call BufferingSpanProcessor::with_capacity(span_buffer_size) instead of new().
• Validate bounds (e.g. min 100, max 100000) to avoid unbounded WASM memory use.

2. kuadrant-operator

Extend API types in internal/wasm/types.go:

type Tracing struct {
    Service        string `json:"service"`
    SpanBufferSize *int   `json:"spanBufferSize,omitempty"`
}

Pass through BuildObservabilityConfig() in internal/wasm/utils.go.

Extend Kuadrant CR spec.observability.tracing (or dataPlane) with the same field.

3. Documentation

• Document recommended sizing: expected_concurrent_requests × spans_per_request × headroom (e.g. 50 concurrent × 5 spans × 2 = 500).
• Note memory trade-off in WASM VM.

4. Optional follow-up

• Per-request span isolation instead of a single global buffer.
• Partial flush under backpressure before request teardown.

GitHub issue template (wasm-shim)

Title: Make tracing span buffer size configurable

Body:

Summary

Under gateway load, wasm-shim logs Span buffer full (100), dropping oldest span. The BufferingSpanProcessor default capacity is hardcoded to 100 with a TODO in code. High-concurrency environments (e.g. MaaS inference gateways) need a supported way to increase this without rebuilding a custom .wasm.

Current behavior

• Global span queue, max 100 spans (processor.rs)
• Export on request teardown via ExportTracesTask
• with_capacity() exists but is not exposed via plugin config

Desired behavior

• observability.tracing.spanBufferSize (or similar) in plugin JSON
• Sensible default (100), operator passes value from Kuadrant CR

Alternatives considered

• Only documenting lower gateway concurrency — insufficient for production tracing under load
• OTEL_BSP_MAX_QUEUE_SIZE on Envoy — does not affect wasm-shim internal buffer

GitHub issue template (kuadrant-operator)

Title: Expose wasm-shim span buffer size in Kuadrant observability spec

Body:

Companion to Kuadrant/wasm-shim#XXX. Plumb spanBufferSize from spec.observability.tracing (or dataPlane) into WasmPlugin pluginConfig.observability.tracing via BuildObservabilityConfig.

How to file

# wasm-shim
open https://github.com/Kuadrant/wasm-shim/issues/new

# kuadrant-operator (after wasm-shim issue exists)
open https://github.com/Kuadrant/kuadrant-operator/issues/new

Paste the relevant sections from this document.