Leave a cell alone once it has had its four tries

[tamnd]

failures.json has counted attempts per cell since it was written, the doc on the field says a cell that has failed four times is a cell where something is actually wrong, and nothing read the count. So a cell this box cannot measure was attempted again on every sweep, at the cost of a full run each time, because a cell fails after memtier has finished rather than before it starts.

The wasted run is the smaller half. The five runs of one cell sit next to each other in the order the matrix goes in, so a shape a box cannot measure is three failures in a row, and three in a row is what puts the whole engine down for the rest of the session. A sweep of one engine stops there and every remaining cell is left with it. A wrapper that restarts the sweep then stops at the same place every time, so the matrix never gets past the first bad shape.

That is what is happening on the 32 core box right now. Sixteen load generator threads on a WSL2 VM finish wildly apart on the deeper pipelines, the parser refuses the rate because memtier divides the whole operation count by the first thread to finish, and one cell there has been attempted fourteen times. It is not particular to one engine: the published wsl32coarse failure file has 40 cells of rugo, 33 of yo and 10 of garnet refused for exactly that reason.

What changes

Four attempts and the cell is left alone. It is counted apart from the cells already on disk, in the line the sweep prints when it starts and in the summary it prints when it finishes, because a cell nobody is going to measure and a cell that is already measured are different kinds of absent.

The failure the sweep exits with says how many of the missing cells have had their tries, so a script that restarts it can tell a sweep worth running again from one that would do nothing. --retry-failed throws the counts away, which is what to pass on a machine that has changed.

Nothing about which cells are measured changes on a box where cells do not fail, and nothing about the numbers changes anywhere.

What checks it

a_cell_is_worth_another_run_until_it_has_had_its_tries walks a cell up to the limit and past it, and checks that another cell in the same file is its own question. the_matrix_comes_apart_into_measured_spent_and_left_to_do puts one cell on the disk and one over the limit and checks the three counts add up to the matrix, and that the cell with a file stops being a failure. the_way_out_says_whether_another_sweep_would_help checks the message a restarting script reads. a_cell_that_has_never_failed_has_been_attempted_no_times and throwing_the_counts_away_makes_every_cell_worth_trying_again cover the two new methods on Failures.

Part of M8 (#10), because a sweep that cannot reach the end of the matrix on the 32 core box is the thing standing between that milestone and two published results directories.