Zisk v0.16.1 seems to get stuck

[jsign]

I've been using Zisk v0.16.1 to prove many EEST fixtures (at this point more than 4.7k ran okay), but suddenly I got into this:

2026-03-28T15:37:19.868133Z  INFO proofman::utils: --- PROOF INSTANCES SUMMARY ---
2026-03-28T15:37:19.868165Z  INFO proofman::utils: Zisk | Arith: 3 | ArithEq: 1 | Binary: 24 | BinaryAdd: 10 | BinaryExtension: 5 | Dma: 1 | Dma64AlignedInputCpy: 1 | Dma64AlignedMem: 1 | Dma64AlignedMemCpy: 2 | Dma64AlignedMemSet: 1 | DmaMemCpy: 1 | DmaPrePost: 1 | DmaUnaligned: 1 | InputData: 1 | Keccakf: 1 | Main: 157 | Mem: 51 | MemAlign: 6 | MemAlignByte: 1 | MemAlignReadByte: 17 | MemAlignWriteByte: 3 | Rom: 1 | RomData: 2 | Sha256f: 1 | SpecifiedRanges: 1 | VirtualTable0: 1 | VirtualTable1: 1 | Total global instances: 296
2026-03-28T15:37:19.868171Z  INFO proofman::utils: --------------------------------
2026-03-28T15:37:19.868177Z  INFO proofman::proofman: <<< EXECUTE (2295ms)
2026-03-28T15:37:19.868186Z  INFO proofman::proofman: >>> CALCULATING_CONTRIBUTIONS
2026-03-28T15:37:27.143574Z  INFO proofman::proofman: <<< CALCULATING_CONTRIBUTIONS (7275ms)
2026-03-28T15:37:27.143778Z  INFO proofman::proofman: ··· Global challenge: [1144121138736804980, 17616331269217287452, 16092880414434848899]
2026-03-28T15:37:27.143790Z  INFO proofman::proofman: >>> GENERATING_PROOFS
2026-03-28T15:37:27.143794Z  INFO proofman::proofman: >>> GENERATING_INNER_PROOFS
2026-03-28T15:37:46.728173Z  INFO proofman::proofman: <<< GENERATING_INNER_PROOFS (19584ms)
2026-03-28T15:37:46.728374Z  INFO proofman::proofman: Received [] aggregated proofs
2026-03-28T15:37:46.728410Z  INFO proofman::proofman: Last proof received. [0] proofs were received and waiting for 0 aggregated proofs to be generated...
2026-03-28T15:37:46.728716Z  INFO proofman::recursion: >>> GENERATE_VADCOP_FINAL_PROOF
2026-03-28T15:37:46.968294Z  INFO proofman::recursion: <<< GENERATE_VADCOP_FINAL_PROOF (239ms)
2026-03-28T15:37:47.010070Z  INFO benchmark_runner::runner: Saving report test_account_query.py::test_ext_account_query_warm[fork_Osaka-blockchain_test-initial_storage_False-initial_balance_True-empty_code_True-opcode_CALLCODE-benchmark-gas-value_25M]
2026-03-28T15:37:47.010514Z  INFO benchmark_runner::runner: Running test_transaction_types.py::test_ether_transfers_to_precompile[fork_Osaka-precompile_0x0000000000000000000000000000000000000007-blockchain_test-transfer_amount_0-benchmark-gas-value_30M]
2026-03-28T15:37:47.011984Z  INFO proofman_common::proof_ctx: Initializing publics custom_commits
2026-03-28T15:37:47.020076Z  INFO proofman::proofman: >>> EXECUTE
2026-03-28T15:37:47.241752Z  INFO executor::executor: >>> COMPUTE_MINIMAL_TRACE
2026-03-28T15:37:50.230852Z  INFO asm_runner::asm_mt_runner: ··· Assembly execution speed: 31MHz (2.988014238s)
2026-03-28T15:37:50.240550Z  INFO executor::executor: <<< COMPUTE_MINIMAL_TRACE (2998ms)
2026-03-28T15:37:50.240567Z  INFO executor::executor: >>> PLAN
2026-03-28T15:37:50.241779Z  INFO executor::executor: <<< PLAN (1ms)
2026-03-28T15:37:50.241814Z  INFO executor::executor: >>> WAIT_PLAN_MEM_CPP
2026-03-28T15:37:50.413838Z  INFO mem_planner_cpp::mem_planner: >>> COLLECT_MEM_PLANS
2026-03-28T15:37:50.414017Z  INFO mem_planner_cpp::mem_planner: <<< COLLECT_MEM_PLANS (0ms)
2026-03-28T15:37:50.414268Z  INFO executor::executor: <<< WAIT_PLAN_MEM_CPP (172ms)
2026-03-28T15:37:50.414301Z  INFO executor::executor: >>> WAIT_ASM_RH
2026-03-28T15:37:50.414307Z  INFO executor::executor: <<< WAIT_ASM_RH (0ms)
2026-03-28T15:37:50.415606Z  INFO proofman::utils: --- PROOF INSTANCES SUMMARY ---
2026-03-28T15:37:50.415626Z  INFO proofman::utils: Zisk | Arith: 1 | ArithEq: 11 | Binary: 5 | BinaryAdd: 3 | BinaryExtension: 2 | Dma: 1 | Dma64AlignedInputCpy: 1 | Dma64AlignedMem: 1 | Dma64AlignedMemCpy: 1 | Dma64AlignedMemSet: 1 | DmaPrePost: 1 | DmaUnaligned: 1 | InputData: 1 | Keccakf: 4 | Main: 23 | Mem: 9 | MemAlign: 3 | MemAlignReadByte: 2 | MemAlignWriteByte: 1 | Rom: 1 | RomData: 1 | Sha256f: 23 | SpecifiedRanges: 1 | VirtualTable0: 1 | VirtualTable1: 1 | Total global instances: 100
2026-03-28T15:37:50.415631Z  INFO proofman::utils: --------------------------------
2026-03-28T15:37:50.415636Z  INFO proofman::proofman: <<< EXECUTE (3395ms)
2026-03-28T15:37:50.415650Z  INFO proofman::proofman: >>> CALCULATING_CONTRIBUTIONS
2026-03-28T15:37:51.729760Z  INFO proofman::proofman: <<< CALCULATING_CONTRIBUTIONS (1314ms)
2026-03-28T15:37:51.729960Z  INFO proofman::proofman: ··· Global challenge: [17289586130210059402, 14002945680848330544, 5002707199477250415]
2026-03-28T15:37:51.729978Z  INFO proofman::proofman: >>> GENERATING_PROOFS
2026-03-28T15:37:51.729982Z  INFO proofman::proofman: >>> GENERATING_INNER_PROOFS
2026-03-28T15:38:59.154464Z  WARN proofman::proof_manager: Counter still waiting to reach zero after 66s - current: 2, expected: 0
2026-03-28T15:39:59.154675Z  WARN proofman::proof_manager: Counter still waiting to reach zero after 126s - current: 2, expected: 0
2026-03-28T15:40:59.154816Z  WARN proofman::proof_manager: Counter still waiting to reach zero after 186s - current: 2, expected: 0
2026-03-28T15:41:59.155018Z  WARN proofman::proof_manager: Counter still waiting to reach zero after 246s - current: 2, expected: 0
2026-03-28T15:42:59.155136Z  WARN proofman::proof_manager: Counter still waiting to reach zero after 306s - current: 2, expected: 0
2026-03-28T15:43:59.155335Z  WARN proofman::proof_manager: Counter still waiting to reach zero after 366s - current: 2, expected: 0
2026-03-28T15:44:59.155496Z  WARN proofman::proof_manager: Counter still waiting to reach zero after 426s - current: 2, expected: 0
2026-03-28T15:45:59.155657Z  WARN proofman::proof_manager: Counter still waiting to reach zero after 486s - current: 2, expected: 0
2026-03-28T15:46:59.155775Z  WARN proofman::proof_manager: Counter still waiting to reach zero after 546s - current: 2, expected: 0
2026-03-28T15:47:52.466099Z ERROR proofman::proof_manager: Counter timeout after 10 minutes to reach zero - current: 2, expected: 0. Cancelling.

And seems to be stuck here for around ~3hs. The prover is run with ZISK_PROVE_TIMEOUT_SEC=900.

I'll restart and can report back if I see this again -- but considering it went through 4.7k fixtures, maybe is some kind of race condition in Zisk.

[han0110]

Now we are running the prover using ZisK SDK, and it seems not as easy as before to set timeout (the timeout env for ZisK is not supported in Ere currently), will try to figure that out.

Is this fixture available in ethereum/execution-spec-tests@benchmark/v0.0.7? Want try to reproduce and debug this

[jsign]

Is not there since I filled a more recent branch after the release -- but in any case, I retried the fixture after, and it worked fine.

So looks like this isn't easily reproducible, nor does it feel like it's particular to any specific fixture (which makes some sense to me given that I've observed this after 4.7k other fixtures proven).

[jsign]

Now it's at ~6k of fixtures proven and haven't seen this again -- so might be something quite weird.

[jsign]

Here's the fixture if it is worth maybe trying multiple times.

[han0110]

Got it, will try to see if I can reproduce, thanks!