[Discuss] First-class Queue Semantics on top of the log

[Shekharrajak]

Today Iggy gives us queue-like behavior through consumer groups: multiple clients join one group and partitions get load-balanced across them. That works, but it's distribution at
partition granularity, so parallelism is capped by partition count and there's no per-message ack.

Let's learn from Kafka and make sure we are building iggy for both Log and Queues.

[spetz]

Hey,
Queue semantics are much different than an append-only log, and in general, implementing the message broker is a total infrastructure rewrite (there's no more immutability, replayability, etc.). TBH, I don't think that's something we would like to incorporate in the near future. Do you have any proposals for how the overall design and implementation could look?

[Shekharrajak]

We should probably work on current roadmap - keeping queue semantics in mind, at least for record commits. The offset approach needs to be flexible enough to track record states accurately ,so that it is easy to extend later. Kafka struggled for Queue Semantics and still community is working on improving the throughput.

I don't have a solid design yet, but I'll flesh it out and share a Google Doc soon.

[hubcio]

could you please be more concrete?