Description
Getting OpenTelemetry data into Iggy currently means running a separate bridge
process in front of it. Collectors and SDKs already speak OTLP/gRPC, and the
connectors runtime already hosts long-lived plugins, so the bridge is avoidable:
the OTLP service can terminate inside the runtime and produce straight onto Iggy
topics.
OTLP is listed as a P0 source and sink on the connector roadmap (#2753).
Affected area / component
Connectors
Proposed solution
An otlp_source connector plugin that binds the OTLP standard port and serves
LogsService, MetricsService and TraceService from a single listener, since
standard OTLP deployments expose one endpoint and splitting them would force
operators to configure three addresses that match no existing tooling.
Decode with opentelemetry-proto, which already ships the generated tonic
stubs, so no build.rs or protoc enters the build.
Two storage formats:
json, one queryable document per record, for sinks that need field-level
access.
proto, the raw request bytes as a single message, roughly 4-5x smaller and
zero-copy when paired with an OTLP sink.
Backpressure must not fail the export. When the in-process channel fills, enqueue
what fits and report the remainder in the response's partial_success, counted
in OTLP records rather than internal messages, so a client retries only what was
actually rejected instead of resending a batch that was half delivered.
Alternatives considered
Running the OpenTelemetry Collector with a custom exporter. That keeps a second
process in the path and duplicates the connector runtime's lifecycle, config and
metrics handling.
Generating the protobuf stubs with tonic-build. It adds protoc to the build
for types opentelemetry-proto already publishes.
Contribution
Yes, I am willing to submit a PR.
Description
Getting OpenTelemetry data into Iggy currently means running a separate bridge
process in front of it. Collectors and SDKs already speak OTLP/gRPC, and the
connectors runtime already hosts long-lived plugins, so the bridge is avoidable:
the OTLP service can terminate inside the runtime and produce straight onto Iggy
topics.
OTLP is listed as a P0 source and sink on the connector roadmap (#2753).
Affected area / component
Connectors
Proposed solution
An
otlp_sourceconnector plugin that binds the OTLP standard port and servesLogsService,MetricsServiceandTraceServicefrom a single listener, sincestandard OTLP deployments expose one endpoint and splitting them would force
operators to configure three addresses that match no existing tooling.
Decode with
opentelemetry-proto, which already ships the generated tonicstubs, so no
build.rsorprotocenters the build.Two storage formats:
json, one queryable document per record, for sinks that need field-levelaccess.
proto, the raw request bytes as a single message, roughly 4-5x smaller andzero-copy when paired with an OTLP sink.
Backpressure must not fail the export. When the in-process channel fills, enqueue
what fits and report the remainder in the response's
partial_success, countedin OTLP records rather than internal messages, so a client retries only what was
actually rejected instead of resending a batch that was half delivered.
Alternatives considered
Running the OpenTelemetry Collector with a custom exporter. That keeps a second
process in the path and duplicates the connector runtime's lifecycle, config and
metrics handling.
Generating the protobuf stubs with
tonic-build. It addsprotocto the buildfor types
opentelemetry-protoalready publishes.Contribution
Yes, I am willing to submit a PR.