Kafka Transactional writer #174
Description
Is your feature request related to a problem? Please describe.
If change events read from the database are being output to Kafka, with state written to a file on disk, there is the possibility to get duplicate events in Kafka, e.g. if OLR writes to Kafka, then crashes before writing to the file on disk, when it restarts it will resend change events to Kafka, causing duplicates in Kafka.
Describe the solution you'd like
Exactly once delivery semantics are possible using Kafka's transactional support. One solution would be to enhance OLR to support storing state in a Kafka topic, with writes to store change events and state contained in the same Kafka transaction. Then Kafka either contains both the change events and the updated state, or neither.
Describe alternatives you've considered
If storing change events and state in two different technologies, e.g. Kafka and file on disk, it's not possible to support exactly once delivery semantics. So the only solution I can think of is to enhance OLR to support storing state in Kafka (either in its own state topic, or potentially in the headers of the change events it writes to Kafka), and leverage Kafka's transaction support to ensure no duplicates are published in the presence of partial failures.
Additional context
Currently running a spike to investigate OLR. At this point, I'm looking to touch base and see if such a feature seems possible for OLR...? It's been a long time since I've worked in C++, but there may be scope for me to help with the feature. TBC.