test(blackbox): stop stale commands from resurrecting TTL-destroyed slots - #670
Merged
Conversation
…lots ForceCheckTTLAndWait folds a TTL destroy into the model directly instead of leaving the destroy command in AcceptedCommands, so the drain's per-pass corrected map never learns about it and the recency ordering (newest command wins) breaks across passes. The authoritative-slot mark stood in for that ordering, but the correction path lets a create clear authoritative status, so a command accepted before the destroy that finished Failed while carrying a create-Success RU for a destroyed slot would clear the mark and resurrect the slot in the model, while the agent's cascade had already removed the inventory row. Cross-stack dependents of an expired provider stack were the visible victims: the existing purge only drops commands whose slots all live on the destroyed stack, so commands on other stacks escaped it. When a TTL destroy is observed, every command still in AcceptedCommands predates it. Mark the destroyed slots (the stack's own and the cascade-destroyed cross-stack dependents) as superseded on those commands; corrections skip superseded slots in both the RU pass and the unmentioned-slot revert, restoring the ordering the corrected map provides within a single pass.
This was referenced Aug 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a seed-dependent FullChaos failure surfaced by the first combined property run after #658 (observed on #659's CI):
inventory=NotExist but model expects Existsfor a cross-stack slot.The mechanism, traced from the failing run's log:
ForceCheckTTLAndWaitfolds an observed TTL destroy into the model directly rather than leaving the destroy command inAcceptedCommands, so the drain's per-passcorrectedmap never represents it and the "newest command wins" ordering breaks across correction passes. The authoritative-slot mark stood in for that ordering, but the correction path deliberately lets creates clear authoritative status (so post-TTL re-creates work). A command accepted before the destroy that finished Failed while carrying a create-Success RU for a destroyed slot therefore cleared the mark and resurrected the slot in the model, while the agent's cascade had already removed the inventory row. Cross-stack dependents of an expired provider stack were the visible victims: the existing supersession purge only drops commands whose slots all live on the destroyed stack, so commands on other stacks escaped it. Pre-#658 this was invisible because cross-stack slots were excluded from model-vs-inventory assertions.The fix restores the ordering information instead of adding another special case: when a TTL destroy is observed, every command still in
AcceptedCommandspredates it, so its destroyed slots (the stack's own and the cascade-destroyed cross-stack dependents) are marked superseded on those commands. Corrections skip superseded slots in both the RU pass and the unmentioned-slot revert.The regression test reproduces the exact shape (stale Failed command with a create-Success RU for a cascade-destroyed cross-stack slot) and fails without the fix. Full non-property suite green; FullChaos green at 25 checks locally.
Blocks #659's CI (which hit this on a rebase re-run) and #668; neither PR's content is related to the failure.