docs: add the rationale doc for AG2-015 - #70
Open
bradAGI wants to merge 1 commit into
Open
Conversation
Pairs with AG2-015 in trustabl-rules. Follows the template guide and carries the machine-checked front-matter, so check_rulebook.py covers the rule. The threat model turns on AutoGen's conversation model supplying two retry paths a single-agent loop lacks. A tool response is a message re-sent in full on every later turn, so an inconclusive call stays re-invitable for the whole run rather than the next turn. And in a group chat the manager may select a different speaker to re-issue it — an agent that reads the same transcript but holds no private memory of having committed the side effect, and so is structurally unable to know it is issuing a duplicate. Confidence is 0.55, the lowest in the pack, and the doc defends that rather than glossing it: the rule reasons from a name and a parameter list, so update_cache is a false positive and book_flight is never examined at all. The exclusions lead with the failure that matters most in practice — a key derived from a fresh uuid4() per call is regenerated on the retry, so it deduplicates nothing while looking exactly like a fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pairs with trustabl/trustabl-rules#101 (AG2-015) and trustabl/trustabl#138 (fixture + tests).
Why AG2-015 exists
Six packs ship an idempotency rule — CSDK-006/016, OAI-009/019, ADK-006, MCP-007, CREW-006, PYD-007. AutoGen ships none.
The argument the doc makes
A retry is not an unusual event in an agent — it is the normal response to an ambiguous result. That is at-least-once delivery with a language model as the retry policy, and the retry policy cannot inspect the remote's state.
AutoGen adds two retry paths a single-agent loop does not have, and both come from its conversation model rather than the tool:
GroupChatthe manager selects who speaks next, and any selected agent can re-issue the tool. It reads the same transcript but holds no private memory of having committed the side effect, so the duplicate is issued by a participant structurally unable to know it is a duplicate.Confidence 0.55 is defended, not glossed
It is the lowest in this pack, and the doc says why in both directions: the rule reasons from a name and a parameter list.
update_cacheprefix-matches and is harmless;book_flightandtransfer_fundscarry no listed prefix and are never examined at all. The prefix list is a heuristic, not a taxonomy of side effects.The exclusions lead with the non-fix
The most important limitation is the one that looks like a fix:
Also named: a declared key the body never forwards, a downstream service that ignores keys entirely, and deduplication done in a wrapper the signature check cannot see.
One recommendation specific to this framework
And: put "reuse the same
idempotency_keywhen retrying" in the tool description, because the model chooses the argument — if it is not told, it invents a new one on the retry.Gate
Two pre-existing errors remain on
main(OAI-112, PYD-106), fixed by #53. AG2-015 is covered and consistent.