Panic on dev (again)

[ananthb]

I'm on latest dev 5b04420 and I've hit another panic during catchup.

ananth@enterprise ~/s/machines (main)> journalctl -b -f -u mithril
Mar 15 01:20:30 enterprise bash[2355299]: │  [1] Continue from AccountsDB (replay 871,084 slots)                         │
Mar 15 01:20:30 enterprise bash[2355299]: │  [2] Start fresh from latest snapshot (faster to catch up)                   │
Mar 15 01:20:30 enterprise bash[2355299]: │                                                                              │
Mar 15 01:20:30 enterprise bash[2355299]: └──────────────────────────────────────────────────────────────────────────────┘
Mar 15 01:20:32 enterprise bash[2355299]: Enter choice (1 or 2): (+    4s) AccountsDB is 871084 slots behind chain tip
Mar 15 01:20:32 enterprise bash[2355299]: (+    4s) mode=auto: Resuming from existing AccountsDB at slot 405551839
Mar 15 01:20:32 enterprise bash[2355299]: (+    4s) StoreAsync=false
Mar 15 01:20:38 enterprise bash[2355299]: (+   10s) Started RPC server on port 8899
Mar 15 01:20:38 enterprise bash[2355299]: (+   10s) Block fetching configured with 2 RPC endpoints (primary + 1 backups)
Mar 15 01:20:40 enterprise bash[2355299]: (+   12s) Loading vote and stake caches (aggregate-only mode, 1486631 stake accounts)
Mar 15 01:21:05 enterprise bash[2355299]: === Replay Start ===
Mar 15 01:21:05 enterprise bash[2355299]: (+   38s) startup stake check: rawScanTotal=634361130941316969 epochEffectiveTotal=421071332193356228 delta=213289798747960741 (expected difference from warmup/cooldown)
Mar 15 01:21:05 enterprise bash[2355299]: (+   38s) loaded RecentBlockhashes sysvar: 150 entries, newest=b0b0778f5f795341, oldest=1ed4cc25e0f6a432
Mar 15 01:21:05 enterprise bash[2355299]: (+   38s) ERROR: [run:a531cd59] DIVERGENCE in slot 405551840: tx 3qcpe2xySzdn48KBq7Zff4e1wZHuj9kcqqJABVQ8w9VFuP1H9r7h3d5xHEPnXQbwwF4qNmfjNzmMDSH66GBGxyAT pre-balance mismatch for 76rcGHdPvgs8G1XrzCXUTWtwgT59AFDvpB4VbTS2TBBJ: mithril=51492474108, onchain=51492669108
Mar 15 01:21:05 enterprise bash[2355299]: (+   38s) ERROR: [run:a531cd59] DIVERGENCE in slot 405551840: tx 2i6F1Aj5UNJnJ1o5uJBLNvfxrNJvXYjLBq9nuu7L16rY9SzTkQ435CXTChZkhCRt8mSnanPgALodbnGHeruxr149 pre-balance mismatch for BUokhb8pPF9MZuzW3rHLr6jzakgcz3NDq2PZkpiVv3jb: mithril=12306539595, onchain=12306739595
Mar 15 01:21:05 enterprise bash[2355299]: panic: tx 3qcpe2xySzdn48KBq7Zff4e1wZHuj9kcqqJABVQ8w9VFuP1H9r7h3d5xHEPnXQbwwF4qNmfjNzmMDSH66GBGxyAT pre-balance divergence: lamport balance for 76rcGHdPvgs8G1XrzCXUTWtwgT59AFDvpB4VbTS2TBBJ was 51492474108 but onchain lamport balance was 51492669108
Mar 15 01:21:05 enterprise bash[2355299]:         acct - slot: 405551839, pubkey: 76rcGHdPvgs8G1XrzCXUTWtwgT59AFDvpB4VbTS2TBBJ, owner: 11111111111111111111111111111111, lamports: 51492474108, executable: false, rent epoch: 18446744073709551615, data len: 0, data hash: CeeM54NJ6EoxLi4VGXLDC1jvNL9SGmppXXUMbAQmoaZw
Mar 15 01:21:05 enterprise bash[2355299]:
Mar 15 01:21:05 enterprise bash[2355299]: goroutine 5637 [running]:
Mar 15 01:21:05 enterprise bash[2355299]: github.com/Overclock-Validator/mithril/pkg/replay.ProcessTransaction(0xc0abb7a000, 0xc051871560, 0xc083274fa0, 0xc07c1454a0, 0xc00041a6a0, 0xc00f1f5f20)
Mar 15 01:21:05 enterprise bash[2355299]:         github.com/Overclock-Validator/mithril/pkg/replay/transaction.go:410 +0x3174
Mar 15 01:21:05 enterprise bash[2355299]: github.com/Overclock-Validator/mithril/pkg/replay.parallelTxLoop.func2()
Mar 15 01:21:05 enterprise bash[2355299]:         github.com/Overclock-Validator/mithril/pkg/replay/block.go:2087 +0x21c
Mar 15 01:21:05 enterprise bash[2355299]: created by github.com/Overclock-Validator/mithril/pkg/replay.parallelTxLoop in goroutine 1
Mar 15 01:21:05 enterprise bash[2355299]:         github.com/Overclock-Validator/mithril/pkg/replay/block.go:2082 +0x31e
Mar 15 01:21:05 enterprise bash[2355299]: panic: tx 2i6F1Aj5UNJnJ1o5uJBLNvfxrNJvXYjLBq9nuu7L16rY9SzTkQ435CXTChZkhCRt8mSnanPgALodbnGHeruxr149 pre-balance divergence: lamport balance for BUokhb8pPF9MZuzW3rHLr6jzakgcz3NDq2PZkpiVv3jb was 12306539595 but onchain lamport balance was 12306739595
Mar 15 01:21:05 enterprise bash[2355299]:         acct - slot: 405551839, pubkey: BUokhb8pPF9MZuzW3rHLr6jzakgcz3NDq2PZkpiVv3jb, owner: 11111111111111111111111111111111, lamports: 12306539595, executable: false, rent epoch: 18446744073709551615, data len: 0, data hash: CezXZPyP6T9zYzqqJvwymFER6ndbMWpFzku3kZK9WPPX
Mar 15 01:21:05 enterprise bash[2355299]:
Mar 15 01:21:05 enterprise bash[2355299]: goroutine 5631 [running]:
Mar 15 01:21:05 enterprise bash[2355299]: github.com/Overclock-Validator/mithril/pkg/replay.ProcessTransaction(0xc0abb7a000, 0xc051871560, 0xc083274be0, 0xc07c144c60, 0xc00041a6a0, 0xc00f1f5e00)
Mar 15 01:21:05 enterprise bash[2355299]:         github.com/Overclock-Validator/mithril/pkg/replay/transaction.go:410 +0x3174
Mar 15 01:21:05 enterprise bash[2355299]: github.com/Overclock-Validator/mithril/pkg/replay.parallelTxLoop.func2()
Mar 15 01:21:05 enterprise bash[2355299]:         github.com/Overclock-Validator/mithril/pkg/replay/block.go:2087 +0x21c
Mar 15 01:21:05 enterprise bash[2355299]: created by github.com/Overclock-Validator/mithril/pkg/replay.parallelTxLoop in goroutine 1
Mar 15 01:21:05 enterprise bash[2355299]:         github.com/Overclock-Validator/mithril/pkg/replay/block.go:2082 +0x31e
Mar 15 01:21:05 enterprise systemd[1]: mithril.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Mar 15 01:21:05 enterprise systemd[1]: mithril.service: Failed with result 'exit-code'.
Mar 15 01:21:05 enterprise systemd[1]: mithril.service: Consumed 2min 7.748s CPU time over 39.429s wall clock time, 6.5G memory peak, 13.3G read from disk, 52K written to disk, 235.8M incoming IP traffic, 4.6M outgoing IP traffic.