fix(rabbitmq): count failed quorum deliveries on RabbitMQ 4.3+ (#533) - #541
Merged
Merged
Conversation
…533) The INVARIANT claiming the Receiver never reads x-acquired-count cited only the acquired-count-ONLY fact, which cannot pin it: with x-delivery-count absent, a resolver consulting x-acquired-count only WHEN x-delivery-count is present -- the normal RabbitMQ 4.3 redelivery shape, where the broker stamps BOTH -- stays green. Adds MustResolveReceiveAttemptsFromDeliveryCountWhenBothQuorumCountersArePresent, which delivers both counters at divergent values (2 and 9) and asserts ReceiveAttempts 3. The named mutation was MEASURED to redden that one fact and no other of the module's 363 unit facts. Also splits the quorum requeue-verb INVARIANT: its oracles and the BasicNackAsync mutation stay on the claims they pin, the 3.13 fact is cited for the half it proves, and the unbacked "both verbs run the same server path before 4.3" claim is demoted to NOTE: with an explicit no-test-pins-this. No production behavior change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #533.
Problem
From RabbitMQ 4.3, an AMQP 0.9.1
basic.nackwith requeue on a quorum queue no longer advancesx-delivery-count. Onlybasic.reject(a failed delivery) does. The Receiver settled failed quorum deliveries withbasic.nack, so on 4.3+ a message whose handler kept failing never reachedmaxReceiveAttempts. It never reached the brokerdelivery-limiteither, so it was redelivered indefinitely. Measured onrabbitmq:4.3-management: about 4,200 handler invocations in 20 s, withReceiveAttemptsstuck at 1.Fix
basic.reject(requeue: true). Before 4.3 both verbs take the same server path (rabbit_channel.erlon v3.13.x, v4.0.x and v4.2.x), so no version detection is needed. On 4.3+ reject is the outcome that advancesx-delivery-count.BufferDeliveryAsyncreads the prior-delivery count once from the key the queue type selects, carries it as anint, and removesx-delivery-countandx-acquired-countfrom the carried headers. They no longer appear inReceivedMessage.Headers, on messages sent while handling, or on copies republished to the Dead-letter Queue, the Error Queue or a classic redelivery. This also closes a pre-existing leak ofx-delivery-countonto republished copies.x-acquired-countis never an attempt source, because it counts assignments, not failures.Chatter.MessageBrokers.RabbitMQ0.6.1 -> 0.6.2 (PATCH), with a CHANGELOG entry.Impact
maxReceiveAttempts, providedmaxReceiveAttemptsis below the queue'sdelivery-limit.ReceivedMessage.Headers(public property) no longer contains the two native counters.Tests
rabbitmq:4.3-managementintegration collection (RabbitMq43Fixture). Its quorum scenarios were observed red on the original code before the fix. The 3.13 fixture remains the broad-surface pin.INVARIANT:comments.dotnet test(Chatter.sln) green: all 9 projects.Decisions
basic.rejectover readingx-acquired-count, over a quorum republish counter, and over a version probe (see ADR-0042). A focused 4.3 CI collection instead of promoting the main fixture.delivery-limitversusmaxReceiveAttemptsis documented, not guarded at runtime. PATCH bump.Open items
INVARIANT:comment prose claiming more than its oracles pin, and six pre-existing blocks inRabbitMqReceiver.cs, are deferred to the existing audit in Audit INVARIANT comments so each claims only what its oracle pins (ADR-0027 trigger) #524 (recorded there).