Skip to content

fix(rabbitmq): count failed quorum deliveries on RabbitMQ 4.3+ (#533) - #541

Merged
brenpike merged 10 commits into
masterfrom
bugfix/533-rabbitmq-quorum-delivery-count
Sep 29, 2026
Merged

brenpike merged 10 commits into
masterfrom
bugfix/533-rabbitmq-quorum-delivery-count

Conversation

@brenpike

Copy link
Copy Markdown
Owner

Closes #533.

Problem

From RabbitMQ 4.3, an AMQP 0.9.1 basic.nack with requeue on a quorum queue no longer advances x-delivery-count. Only basic.reject (a failed delivery) does. The Receiver settled failed quorum deliveries with basic.nack, so on 4.3+ a message whose handler kept failing never reached maxReceiveAttempts. It never reached the broker delivery-limit either, so it was redelivered indefinitely. Measured on rabbitmq:4.3-management: about 4,200 handler invocations in 20 s, with ReceiveAttempts stuck at 1.

Fix

  • Settle verb. A failed quorum delivery is returned with basic.reject(requeue: true). Before 4.3 both verbs take the same server path (rabbit_channel.erl on v3.13.x, v4.0.x and v4.2.x), so no version detection is needed. On 4.3+ reject is the outcome that advances x-delivery-count.
  • Native counters are consumed at the receive boundary. BufferDeliveryAsync reads the prior-delivery count once from the key the queue type selects, carries it as an int, and removes x-delivery-count and x-acquired-count from the carried headers. They no longer appear in ReceivedMessage.Headers, on messages sent while handling, or on copies republished to the Dead-letter Queue, the Error Queue or a classic redelivery. This also closes a pre-existing leak of x-delivery-count onto republished copies. x-acquired-count is never an attempt source, because it counts assignments, not failures.
  • Docs. README quorum-queue section, CONTEXT.md, the design doc, ADR-0042 (new) and an ADR-0027 amendment.
  • Version. Chatter.MessageBrokers.RabbitMQ 0.6.1 -> 0.6.2 (PATCH), with a CHANGELOG entry.

Impact

  • Public API signatures: none.
  • Consumer configuration: none.
  • Behaviour on RabbitMQ before 4.3 and on classic queues: unchanged (3.13 integration collection green).
  • On 4.3+ quorum queues: failing messages are dead-lettered at maxReceiveAttempts, provided maxReceiveAttempts is below the queue's delivery-limit.
  • ReceivedMessage.Headers (public property) no longer contains the two native counters.

Tests

  • New focused rabbitmq:4.3-management integration collection (RabbitMq43Fixture). Its quorum scenarios were observed red on the original code before the fix. The 3.13 fixture remains the broad-surface pin.
  • Unit facts for the reject verb, the counter strip on every republish hop, and attempt resolution when both quorum counters are present. Mutations were measured and are recorded in the INVARIANT: comments.
  • dotnet test (Chatter.sln) green: all 9 projects.

Decisions

  • User decisions: accepted the reviewer's rejections of two Codex findings about publisher- or handler-controlled counters, which are inherited and recorded in ADR-0015. Approved the receive-boundary remediation. Stopped the local review after the final wording fix.
  • Agent decisions: basic.reject over reading x-acquired-count, over a quorum republish counter, and over a version probe (see ADR-0042). A focused 4.3 CI collection instead of promoting the main fixture. delivery-limit versus maxReceiveAttempts is documented, not guarded at runtime. PATCH bump.

Open items

…533)

The INVARIANT claiming the Receiver never reads x-acquired-count cited only the
acquired-count-ONLY fact, which cannot pin it: with x-delivery-count absent, a
resolver consulting x-acquired-count only WHEN x-delivery-count is present --
the normal RabbitMQ 4.3 redelivery shape, where the broker stamps BOTH -- stays
green.

Adds MustResolveReceiveAttemptsFromDeliveryCountWhenBothQuorumCountersArePresent,
which delivers both counters at divergent values (2 and 9) and asserts
ReceiveAttempts 3. The named mutation was MEASURED to redden that one fact and no
other of the module's 363 unit facts.

Also splits the quorum requeue-verb INVARIANT: its oracles and the BasicNackAsync
mutation stay on the claims they pin, the 3.13 fact is cited for the half it
proves, and the unbacked "both verbs run the same server path before 4.3" claim
is demoted to NOTE: with an explicit no-test-pins-this.

No production behavior change.
@brenpike
brenpike merged commit 6760601 into master Sep 29, 2026
16 checks passed
@brenpike
brenpike deleted the bugfix/533-rabbitmq-quorum-delivery-count branch September 29, 2026 03:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RabbitMQ: quorum-queue attempt counting never advances on RabbitMQ 4.3+ (basic.nack does not increment x-delivery-count)

1 participant