Application becomes unresponsive after KafkaThrottledLatestProcessedCommit warns about unacknowledged messages

[wesleysalimansdvb]

Description

Our application stops processing Kafka messages and enters a "zombie" state after KafkaThrottledLatestProcessedCommit logs a TooManyMessagesWithoutAckException warning. Once this occurs, the application remains running but no longer consumes or processes any messages from Kafka. We want to be able to detect this warning occurring and restart our application.

Warning Message

WARN  [io.sma.rea.mes.kafka] (vert.x-eventloop-thread-1) SRMSG18228: A failure has been reported for Kafka topics ‘[topic]’: io.smallrye.reactive.messaging.kafka.commit.KafkaThrottledLatestProcessedCommit$TooManyMessagesWithoutAckException: The record 48609276 from topic/partition ’topic-12' has waited for 52 seconds to be acknowledged. At the moment 88 messages from this partition are awaiting acknowledgement. The last committed offset for this partition was 48609275.

Environment

smallrye-reactive-messaging-kafka: 4.31.0
quarkus: 3.30.6
java21

Our config

mp.messaging.incoming.transaction-consumer.enabled=true
mp.messaging.incoming.transaction-consumer.connector=smallrye-kafka
mp.messaging.incoming.transaction-consumer.security.protocol=SSL

mp.messaging.incoming.transaction-consumer.ssl.keystore.location=keystore.jks
mp.messaging.incoming.transaction-consumer.ssl.keystore.password=password
mp.messaging.incoming.transaction-consumer.ssl.key.password=password

mp.messaging.incoming.transaction-consumer.topics=topic
mp.messaging.incoming.transaction-consumer.group.id=group
mp.messaging.incoming.transaction-consumer.auto.offset.reset=earliest

mp.messaging.incoming.transaction-consumer.commit-strategy=throttled
mp.messaging.incoming.transaction-consumer.throttled.unprocessed-record-max-age.ms=50000

mp.messaging.incoming.transaction-consumer.key.deserializer=io.confluent.kafka.serializers.KafkaAvroDeserializer
mp.messaging.incoming.transaction-consumer.value.deserializer=io.confluent.kafka.serializers.KafkaAvroDeserializer
mp.messaging.incoming.transaction-consumer.failure-strategy=dead-letter-queue
mp.messaging.incoming.transaction-consumer.dead-letter-queue.topic=dlq

Expected Behavior

The application should either:

• Recover automatically after acknowledgement timeout/issues
• Throw an exception that we can catch so we can restart our application

Actual Behavior

After the warning is logged, the application completely stops consuming messages from Kafka. The process continues running (no crash), but it becomes a "zombie" - not processing any messages and not recovering without a restart.

Questions

• How can we detect that the TooManyMessagesWithoutAckException has been triggered? It is not thrown or anything, so how can we see that it has become unhealthy?
• What is the recommended approach to handle this scenario where the TooManyMessagesWithoutAckException is triggered and how can we prevent the application from becoming "unresponsive" (stopping processing any more messages)?
• Are there specific configuration parameters we should adjust to prevent this issue (e.g., throttle limits, acknowledgement timeouts)?
• Is this behavior (stopping all processing) intentional, or should the consumer continue with other partitions/messages?

Any recommendations would be greatly appreciated!