Skip to content

fix: bump cometbft and TxReplacementDecorator into ante handler to unblock stuck mempool sequences#661

Open
hoank101 wants to merge 15 commits into
classic-terra:mainfrom
orbitorg:hoa/fix-unjain
Open

fix: bump cometbft and TxReplacementDecorator into ante handler to unblock stuck mempool sequences#661
hoank101 wants to merge 15 commits into
classic-terra:mainfrom
orbitorg:hoa/fix-unjain

Conversation

@hoank101
Copy link
Copy Markdown
Collaborator

@hoank101 hoank101 commented May 9, 2026

Fix #648

Summary of changes

Problem

A validator operator wallet became permanently stuck and the validator was jailed with no way to unjail, because a CheckTx-accepted transaction at the committed sequence never made it into a block:

  1. On-chain (committed) sequence for the wallet = 432.
  2. A sequence-432 tx is accepted into the mempool (CheckTx passes, code 0).
  3. Rebroadcast of that tx returns ErrTxInMempoolCache / code 19 — CometBFT's cache believes the tx is still pending, so it refuses to re-admit it.
  4. Meanwhile subsequent txs queue up at 433…442, advancing the mempool's locally-expected (pending) sequence to 443.
  5. The sequence-432 tx is never selected/included, but the operator cannot replace it: a fresh submission at 432 is rejected against the mempool's pending sequence (expected 443, got 432), and a resubmission of the
    identical tx is rejected by the cache (code 19).

The result is a deadlock: every later tx from the wallet is stranded behind a sequence-432 tx that can neither be included nor replaced. For a validator this is fatal — the MsgUnjail / MsgEditValidator txs needed
to recover are themselves queued behind the stuck sequence.

Report of required housekeeping

  • Github issue OR spec proposal link
  • Wrote tests
  • Updated API documentation (client/lcd/swagger-ui/swagger.yaml)
  • Added a relevant changelog entry: clog add [section] [stanza] [message]

(FOR ADMIN) Before merging

  • Added appropriate labels to PR
  • Squashed all commits, uses message "Merge pull request #XYZ: [title]" (coding standards)
  • Confirm added tests are consistent with the intended behavior of changes
  • Ensure all tests pass

hoank101 and others added 14 commits April 25, 2026 22:44
* - fix ibc handling in wasm (#77)

- remove fork code to use upstream
- remove unused code

* Problem: cometbft not up to date

* add v14_2

* bump version

* bump version

* bump version

* fix legacy query staking (#79)

* fix legacy query staking

* bet

* fix legacy params for validator delegation fallback

* Bet

* bet

---------

Co-authored-by: Khanh Hoa <hoa@notional.ventures>
Co-authored-by: Tuan Tran <tropicaldog17@gmail.com>

* consistent alias

---------

Co-authored-by: StrathCole <7449529+StrathCole@users.noreply.github.com>
Co-authored-by: Kien Trinh <51135161+kien6034@users.noreply.github.com>
Co-authored-by: Tuan Tran <tropicaldog17@gmail.com>
Comment on lines +58 to +60
for k := range info.newTxBytesSet {
newSet[k] = struct{}{}
}
Comment on lines +89 to +93
for k := range info.newTxBytesSet {
if k != key {
newSet[k] = struct{}{}
}
}
Comment on lines +58 to +60
for k := range info.newTxBytesSet {
newSet[k] = struct{}{}
}
Comment on lines +89 to +93
for k := range info.newTxBytesSet {
if k != key {
newSet[k] = struct{}{}
}
}
@TropicalDog17
Copy link
Copy Markdown
Contributor

Hi @StrathCole @fragwuerdig can you guys review this PR?

@StrathCole
Copy link
Copy Markdown
Collaborator

Wouldn't it be rather necessary to fix the root cause (a tx that doesn't make it into a block should not pass checkTx)?

@hoank101
Copy link
Copy Markdown
Collaborator Author

Wouldn't it be rather necessary to fix the root cause (a tx that doesn't make it into a block should not pass checkTx)?

The fee.go changes address the most likely root cause (burn tax calculation inconsistency between CheckTx and RecheckTx). However, fixing it completely is not possible within Terra alone - CometBFT's CheckTx is by design a soft filter, and any chain state change (governance tax/gas-price update) between submission and inclusion creates a new discrepancy. The TxReplacementDecorator is defense-in-depth: it provides a recovery path when any future discrepancy causes the same deadlock pattern, which is especially critical for validator operations where getting stuck means being jailed indefinitely.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Unable to change validator commission -- account sequence mismatch

5 participants