Extract ponder move and check it against PV

[robertnurnberg]

This PR allows the extraction of a ponder move if it is present on the bestmove output line. On successful extraction, the ponder move is checked against the second move of the last PV output (if that was for an exact score).

If the new fhe flag -warn-ponder-pv-short is passed, then fastchess also warns if the engine emits a ponder move alongside a PV of length 1.

No legality check is performed explicitly, but an illegal ponder move would always raise a warning, as either the PV move itself was illegal (master flags this already), or the ponder move does not match the PV.

We also add some test cases and extend the dummy engine for this purpose.

[robertnurnberg]

Ok, so latest sf-dev no longer emits this warning. But I think it is good to leave the ponder move check optional.

IMO the standard for all uci engines should be that bestmove and ponder move match the final PV. Here is what https://backscattering.de/chess/uci/#engine-bestmove has to say about this:

bestmove move1 [ ponder move2 ]
* the engine has stopped searching and found the move move best in this position.
* the engine can send the move it likes to ponder on. The engine must not start pondering automatically.
* this command must always be sent if the engine stops searching, also in pondering mode if there is a stop command, so for every go command a bestmove command is needed!
* Directly before that the engine should send a final info command with the final search information, the the GUI has the complete statistics about the last search.

So arguably the last point means that the GUI should be informed about the ponder move in the final info command. For example, thinking about chess tournaments, I believe it is nice to give the viewer the full information: i.e. what opponent move the engine would ponder on.

If engines would like to give the "true" search pv (say of length 1 after a fail high/low) to the user as well, they can do so in the penultimate info command, for example.

[vondele]

SF always tries to provide a ponder move, even if the PV lengths is just 1. This makes sense, i.e. ensure you have something to ponder about even if you have no PV yet (otherwise you can't use the time given to you in the 'pondering window').
The ponder move comes with less guarantees than a PV move, that's why it is fetched from TT in case there is no PV of sufficient length.

[robertnurnberg]

Right. And if SF does fetch a ponder move from TT, it lengthens the PV with it and outputs the longer PV to the gui.

Which is as it should be imo.

The fact that length 1 or length 2 PVs show some uncertainty of the engine is usually well understood by users. Similarly to the very end of PVs usually being somewhat unreliable.

[vondele]

actually I think that would be a change of behavior, at least when we last updated the cluster branch, we made sure that the PV was shown before that extension, so that the extension was immaterial? At least, IIRC.

[robertnurnberg]

Yes, it's part of the changes in official-stockfish/Stockfish#6679.

I don't think there's a reason to withhold information from the gui. If the engine thinks the move is good enough to ponder on, we might as well send it to the user.

The UCI protocol should not have to concern itself with a/b engine semantics, imo.

[vondele]

I don't think that's quite correct, or at least it might lead to confusion. What if for example the PV + pondermove happens to be a mate position or a 3-fold, but the eval is not yet consistent.

[robertnurnberg]

A game ending draw can no longer happen, that's one of the things the pr fixed. (Master used to emit ponder moves leading to a theoretical draw.) Actually, a game ending draw after the ponder move can of course still happen, sorry. But I don't think there is a requirement for PVs ending in theoretical draws to have a draw score at root?

If the ponder move leads to a checkmate then bestmove should ideally have mate -1, yes. And I guess only in extreme time trouble it wouldn't. But I think it's not necessary that any PV ending in checkmate should have a mate score. Indeed our PV extensions from TB end in checkmate without a mate score at root.

[vondele]

still not 100% sure that the SF change is good. However, probably best to split this question in two parts.

The first one for fastchess, and I would say if there is a PV that contains at least 2 moves, it is reasonable to verify that the pondermove and the 2nd PV move match. If there is not a second PV move that check is not done. I'm not 100% sure this check is correct (or I could imagine an engine coming up with a different scheme for whatever reason).

The other question is for SF, i.e. if adding the pondermove to the PV if the PV is only 1 move long. Here I have rather strong reservations. But that we can discuss elsewhere.

[robertnurnberg]

Agreed. I have amended the PR to make the contentious warning for ponder move vs. PV of length 1 optional.

Latest push causes CI to fail, but this looks unrelated to the PR? Maybe something @Disservin can look into?

[robertnurnberg]

Example run of ./fastchess -engine name=sf-17.1a cmd=./sf17.1 tc=1+0.01 -engine name=sf-17.1b cmd=./sf17.1 tc=0.2+0.01 -rounds 500 -concurrency -1 -openings file=UHO_Lichess_4852_v1.epd format=epd -warn-ponder-pv-short | grep Warning -A2 | head:

Warning; PV has length 1 despite ponder move output - move e5d6 from sf-17.1b
Info; info depth 3 seldepth 5 multipv 1 score cp -311 nodes 191 nps 191000 hashfull 0 tbhits 0 time 1 pv h7h5
Position; fen rnbqk1nr/pppp1ppp/3b4/4p3/3PP3/5N2/PPP2PPP/RNBQKB1R b KQkq - 0 3
--
Warning; PV has length 1 despite ponder move output - move c3g3 from sf-17.1b
Info; info depth 3 seldepth 5 multipv 1 score cp -245 nodes 472 nps 236000 hashfull 0 tbhits 0 time 2 pv c7d8
Position; fen rnbqkbnr/pp2pppp/2p5/8/2pP4/4P3/PP3PPP/RNBQKBNR w KQkq - 0 4
--
Warning; PV has length 1 despite ponder move output - move e4e1 from sf-17.1b
Info; info depth 5 seldepth 7 multipv 1 score cp -545 nodes 268 nps 268000 hashfull 0 tbhits 0 time 1 pv f7f5

Example run of ./fastchess -engine name=sf-17.1a cmd=./sf17.1 tc=1+0.01 -engine name=sf-17.1b cmd=./sf17.1 tc=0.2+0.01 -rounds 500 -concurrency -1 -openings file=UHO_Lichess_4852_v1.epd format=epd | grep Warning -A2 | head:

Warning; Pondermove does not match beginning of last PV - move c6d7 from sf-17.1a
Info; info depth 17 seldepth 18 multipv 1 score cp -780 nodes 11294 nps 941166 hashfull 6 tbhits 0 time 12 pv f5g6 e1e7 g6h6 c6a4 h6g6 e7f8 g6h5 f8h8 h5g6
Position; fen rn1qkbnr/pp2pppp/6b1/3p4/3P2P1/5N1P/PPP2P2/RNBQKB1R w KQkq - 1 7
--
Warning; Pondermove does not match beginning of last PV - move g8f7 from sf-17.1b
Info; info depth 58 seldepth 5 multipv 1 score mate -2 nodes 4460 nps 1486666 hashfull 0 tbhits 0 time 3 pv c4f7 g8h8 b1d2 g3h3
Position; fen rnbq1rk1/ppp2pp1/3p1n1p/2b1p3/2B1P3/3P1Q1N/PPP2PPP/RNB2RK1 w - - 4 9

[robertnurnberg]

Thanks for fixing the CI issue by doing a manual re-run. I have moved the separate discussion to official-stockfish/Stockfish#6725.