Skip to content

feat(connectors): OTLP/gRPC source connector for logs, metrics and traces #3815

Description

@mfyuce

Description

Getting OpenTelemetry data into Iggy currently means running a separate bridge
process in front of it. Collectors and SDKs already speak OTLP/gRPC, and the
connectors runtime already hosts long-lived plugins, so the bridge is avoidable:
the OTLP service can terminate inside the runtime and produce straight onto Iggy
topics.

OTLP is listed as a P0 source and sink on the connector roadmap (#2753).

Affected area / component

Connectors

Proposed solution

An otlp_source connector plugin that binds the OTLP standard port and serves
LogsService, MetricsService and TraceService from a single listener, since
standard OTLP deployments expose one endpoint and splitting them would force
operators to configure three addresses that match no existing tooling.

Decode with opentelemetry-proto, which already ships the generated tonic
stubs, so no build.rs or protoc enters the build.

Two storage formats:

  • json, one queryable document per record, for sinks that need field-level
    access.
  • proto, the raw request bytes as a single message, roughly 4-5x smaller and
    zero-copy when paired with an OTLP sink.

Backpressure must not fail the export. When the in-process channel fills, enqueue
what fits and report the remainder in the response's partial_success, counted
in OTLP records rather than internal messages, so a client retries only what was
actually rejected instead of resending a batch that was half delivered.

Alternatives considered

Running the OpenTelemetry Collector with a custom exporter. That keeps a second
process in the path and duplicates the connector runtime's lifecycle, config and
metrics handling.

Generating the protobuf stubs with tonic-build. It adds protoc to the build
for types opentelemetry-proto already publishes.

Contribution

Yes, I am willing to submit a PR.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions