Feature comparison - Kafka vs Iggy for Enterprise consumption

[ryerraguntla]

Creating a place holder to compare and analyze feature comparison with Kafka. The output of this document is to list all the features of Kafka which are needed in Iggy to have full enterprise product suite. Once analysis is complete, plan for the next action items with respective PRs.

I have my initial analysis which covers the following main areas.

Production Grade
Cloud Native support
Production Resilience/DR
Observability
Schema Registry
API compatibility

[ryerraguntla]

Given the effort involved in implementing , I started with API Compatibility with Kafka. At a high level, I can think of two ways of implementing - a stand alone application which receives and translates Kafka requests to Iggy format and forwards to Iggy server. Another approach is to have a TCP Listener port and associated thread inside iggy server which receives and translates Kafka requests to Iggy format and forwards to Iggy server layer for processing.

Pros and Cons :
Approach 1 (Stand alone TCP Listener and Processing) :
Pro : Clean code , do not interfere with Iggy code base.
Con : Adds additional 1-2 ms to the round trip of message processing.

Approach 2 ( Iggy opens another TCP Port for Kafka API protocol) :
Pro : Easy to deploy and help for a quicker adaptability
Con : Need to make sure code and design patterns do not affect the Iggy code base.

Given my preference for Approach 2, below is the high level design diagram looks.

Kafka TCP Listener and Kafka Protocol Decoder (ApiKeys, VInts, etc.) will be implemented such that it will not (or minimal )interfere with Iggy current code base. This approach helps smooth or even zero code changes for transition and make use of Iggy's Ultra Performance capabilities even as a single node instance.

[hubcio]

Please see #3044 and #3038

[spetz]

We can't directly open the Kafka port and support this API natively as part of our server - this can only be done via an additional translation layer or some sort of proxy. Iggy is the new project within the message streaming infrastructure, and we can't embed another protocol into it.

[ryerraguntla]

Thank you @hubcio and @spetz . So general direction is to have a separate runtime process outside of server.It may add one or two milli seconds to the end to end roundtrip. We can debate about it later after functional completeness is available.

One of the major challenge I am facing is the complexity in terms number of API keys and versions in each of them.
At highlevel I am grouping them into two buckets. API Key 0-18 and all others..
Group of API Keys 0-20 provides the basic functionality of Kafka and other part is more of transactions and ZK free Kafka.

TODO :-
I will update with detailed info. Once done I will solicit reviews.

[ryerraguntla]

I am splitting the implementation discussion of kafka compatible messages to iggy (kafka to iggy bridge)into two threads.
Technical implementation discussion
Order of features implementation.

#3252 - Technical implementation discussion is about the design and architecture of the solution.
#3253 - The order of implementing the features is about the priority of the rollout of the kafka compatible features. I am creating separate GH discussions for each of the threads. -

I am keeping this discussion open for the other features of enterprise readiness and discussions.

[ryerraguntla]

#3252 is closed.