help me test multipv

[Ferdy]

Should the pv scores be sorted? Especially at high depths you don't want to wait 3x just to sort the pv lines by score.

The PV lines are printed as soon as they are computed. Ideally the search returns the scores in a decreasing order, but the search is full of instabilities (hash, lmr, nullmove) and sometimes the second move is searched more or less deeply.

I agree with that, but I took a different approach, after one pv is completed at a given iteration, I stored it first waiting for other pvs to be completed because there might be a pv later that is better than the previous pv. Once the pv are completed, I sort them by score, then send it to gui pv by pv, with multipv 1 having the highest score and is sent first. This would assure me that the gui will sort it properly for the user convenience to see which is best and second best and so on per iteration.

There might be gui that does not sort your pv properly in this case Arena. There could be others, but I did not try so far.

Also sorting this pv's is for my engine limit strength feature, multipv first is best and multi pv last is worst, per given iteration depth. So I can easily operate which move is best, etc per iteration by just looking at the multipv number.

[brtzsnr]

Below is what Stockfish does. Before the engine starts to display lowerbound/upperbound/currentmove the output is very similar to my engine's, except PVs are sorted by score. zurichess doesn't show upperbound/lowerbound/currentmove so the grouping is implicit. Sorting is not required (per UCI specifications), therefore zurichess conforms to the UCI protocol. I don't care about sorting, but I understand that the human player won't trust the pvs if the scores are not sorted.

As usual I'm really amazed by the amount of smart and friendly people on this forum. Thank you for the help.

Edit: Just a reminder. The shared binaries are just to test the new multipv feature. I think there is 30 Elo regression which I need to track down. I will release a new stable version sometime earliest at the end of August.

Code: Select all

% /usr/games/stockfish
Stockfish 09-06-13 64bit by Tord Romstad, Marco Costalba and Joona Kiiski
position startpos moves e2e4 e7e5 
setoption name MultiPV value 3
go infinite
info depth 1 seldepth 1 score cp 76 nodes 132 nps 132000 time 1 multipv 1 pv d2d4 e5d4 d1d4
info depth 1 seldepth 1 score cp 68 nodes 132 nps 132000 time 1 multipv 2 pv b1c3
info depth 1 seldepth 1 score cp 56 nodes 132 nps 132000 time 1 multipv 3 pv g1f3
info depth 2 seldepth 2 score cp 24 nodes 681 nps 340500 time 2 multipv 1 pv d2d4 b8c6 d4e5 c6e5
info depth 2 seldepth 2 score cp 12 nodes 681 nps 340500 time 2 multipv 2 pv b1c3 b8c6
info depth 2 seldepth 2 score cp -4 nodes 681 nps 340500 time 2 multipv 3 pv g1f3 d7d5 e4d5 d8d5
info depth 3 seldepth 3 score cp 60 nodes 1525 nps 381250 time 4 multipv 1 pv d2d4 b8c6 g1f3 e5d4 f3d4 c6d4 d1d4
info depth 3 seldepth 3 score cp 60 nodes 1525 nps 381250 time 4 multipv 2 pv g1f3 b8c6 d2d4 e5d4 f3d4 c6d4 d1d4
info depth 3 seldepth 3 score cp 56 nodes 1525 nps 381250 time 4 multipv 3 pv b1c3 b8c6 g1f3
info depth 4 seldepth 4 score cp 28 nodes 2939 nps 587800 time 5 multipv 1 pv d2d4 b8c6 g1f3 g8f6
info depth 4 seldepth 4 score cp 28 nodes 2939 nps 587800 time 5 multipv 2 pv g1f3 b8c6 d2d4 g8f6
info depth 4 seldepth 4 score cp 12 nodes 2939 nps 587800 time 5 multipv 3 pv b1c3 b8c6 g1f3 g8f6
info depth 5 seldepth 5 score cp 60 nodes 6226 nps 778250 time 8 multipv 1 pv g1f3 b8c6 b1c3 f8c5 d2d3
info depth 5 seldepth 5 score cp 60 nodes 6226 nps 778250 time 8 multipv 2 pv b1c3 b8c6 g1f3 f8c5 d2d3
info depth 5 seldepth 5 score cp 52 nodes 6226 nps 778250 time 8 multipv 3 pv d2d4 e5d4 g1f3 c7c5 c1g5
info depth 6 seldepth 7 score cp 64 nodes 13270 nps 1020769 time 13 multipv 1 pv g1f3 b8c6 b1c3 f7f5 d2d4 f5e4 f3e5
info depth 6 seldepth 7 score cp 64 nodes 13270 nps 1020769 time 13 multipv 2 pv b1c3 b8c6 g1f3 f7f5 d2d4 f5e4 f3e5
info depth 6 seldepth 7 score cp 4 nodes 13270 nps 1020769 time 13 multipv 3 pv d1h5 d8e7 g1f3 b8c6 b1c3 g8f6
info depth 7 seldepth 10 score cp 52 nodes 27931 nps 1396550 time 20 multipv 1 pv g1f3 b8c6 b1c3 g8f6 d2d4 e5d4 f3d4 d7d5 e4d5 f6d5 d4c6 b7c6
info depth 7 seldepth 10 score cp 52 nodes 27931 nps 1396550 time 20 multipv 2 pv b1c3 b8c6 g1f3 g8f6 d2d4 e5d4 f3d4 d7d5 e4d5 f6d5 d4c6 b7c6
info depth 7 seldepth 10 score cp 48 nodes 27931 nps 1396550 time 20 multipv 3 pv f1c4 g8f6 b1c3 b8c6 g1f3 f8c5 d2d3
info depth 8 seldepth 10 score cp 52 nodes 36973 nps 1422038 time 26 multipv 1 pv g1f3 b8c6 b1c3 g8f6 d2d4 e5d4 f3d4 d7d5 e4d5 f6d5 d4c6 b7c6
info depth 8 seldepth 10 score cp 52 nodes 36973 nps 1422038 time 26 multipv 2 pv b1c3 b8c6 g1f3 g8f6 d2d4 e5d4 f3d4 d7d5 e4d5 f6d5 d4c6 b7c6
info depth 8 seldepth 10 score cp 12 nodes 36973 nps 1422038 time 26 multipv 3 pv f1c4 g8f6 b1c3 b8c6 g1f3 f8c5 d2d3 d7d6
info depth 9 seldepth 14 score cp 52 nodes 92069 nps 1560491 time 59 multipv 1 pv g1f3 b8c6 b1c3 g8f6 d2d4 e5d4 f3d4 d7d5 e4d5 f6d5 d4c6 b7c6
info depth 9 seldepth 14 score cp 16 nodes 92069 nps 1560491 time 59 multipv 2 pv b1c3 b8c6 g1f3 f8c5 c3a4 c5d6 d2d4 e5d4 f3d4 g8f6
info depth 9 seldepth 14 score cp 12 nodes 92069 nps 1560491 time 59 multipv 3 pv f1c4 g8f6 b1c3 b8c6 g1f3 f8c5 d2d3 d7d6 e1g1 c8e6 c1e3 c5e3 f2e3 e6c4 d3c4
info depth 10 seldepth 16 score cp 52 nodes 295969 nps 2084288 time 142 multipv 1 pv g1f3 b8c6 d2d4 e5d4 f3d4 g8f6 b1c3 d7d5 e4d5 f6d5 d4c6 b7c6
info depth 10 seldepth 16 score cp 32 nodes 295969 nps 2084288 time 142 multipv 2 pv g1e2 d7d5 e4d5 d8d5 b1c3 d5e6 f2f4 e5f4 d2d4 f8d6 d4d5
info depth 10 seldepth 16 score cp 12 nodes 295969 nps 2084288 time 142 multipv 3 pv b1c3 b8c6 g1f3 f8c5 c3a4 c5e7 d2d4 e5d4 f3d4 d7d5 d4c6 b7c6
info depth 11 seldepth 18 score cp 40 nodes 568261 nps 2328938 time 244 multipv 1 pv b1c3 b8c6 g1f3 f8c5 f1b5 g8e7 e1g1 e8g8 d2d3 a7a6 b5c6 e7c6 c1g5 f7f6 g5e3 c5b4 c3d5
info depth 11 seldepth 18 score cp 36 nodes 568261 nps 2328938 time 244 multipv 2 pv g1f3 b8c6 f1b5 f7f5 d2d3 f5e4 d3e4 g8f6 b1c3 f8b4 b5c6 b4c3 b2c3 d7c6 d1d8 e8d8
info depth 11 seldepth 18 score cp 16 nodes 568261 nps 2328938 time 244 multipv 3 pv g1e2 b8c6 d2d4 e5d4 e2d4 g8f6 b1c3 f8b4 d4c6 b7c6 e4e5 f6d5
info depth 12 seldepth 24 score cp 36 nodes 975324 nps 2955527 time 330 multipv 1 pv g1f3 b8c6 f1b5 f7f5 d2d3 g8f6 b1c3 f5e4 d3e4 f8b4 b5c6 b4c3 b2c3 d7c6 d1d8 e8d8 f3e5 f6e4
info depth 12 seldepth 24 score cp 32 nodes 975324 nps 2955527 time 330 multipv 2 pv b1c3 b8c6 g1f3 g8f6 f1b5 f8b4 b5c6 d7c6 f3e5 e8g8 f2f3 c8e6 d2d4 f8e8
info depth 12 seldepth 24 score cp 20 nodes 975324 nps 2955527 time 330 multipv 3 pv f1c4 g8f6 b1c3 f6e4 d1h5 e4d6 h5e5 d8e7 e5e7 f8e7 c3d5 e7d8 d2d3 b8c6
info depth 13 seldepth 25 score cp 44 nodes 1724193 nps 3569757 time 483 multipv 1 pv g1f3 g8f6 f3e5 d7d6 e5f3 f6e4 d1e2 d8e7 b1c3 e4c3 d2c3 e7e2 f1e2 b8c6
info depth 13 seldepth 25 score cp 32 nodes 1724193 nps 3569757 time 483 multipv 2 pv b1c3 b8c6 g1f3 g8f6 f1b5 f8b4 b5c6 d7c6 f3e5 e8g8 f2f3 c8e6 d2d4 f8e8
info depth 13 seldepth 25 score cp 20 nodes 1724193 nps 3569757 time 483 multipv 3 pv f1c4 g8f6 b1c3 f6e4 d1h5 e4d6 h5e5 d8e7 e5e7 f8e7 c4b3 b7b5 a2a4 b5a4 a1a4 b8c6 g1f3 c8b7
info depth 14 seldepth 25 score cp 40 nodes 3321320 nps 3930556 time 845 multipv 1 pv g1f3 g8f6 f3e5 d7d6 e5f3 f6e4 f1b5 c7c6 b5e2 f8e7 d2d3 e4f6 b1c3 e8g8 e1g1 f8e8 c1e3 c6c5 d3d4 b8c6
info depth 14 seldepth 25 score cp 20 nodes 3321320 nps 3930556 time 845 multipv 2 pv b1c3 b8c6 g1f3 g8f6 f1b5 f8b4 d2d3 b4c3 b2c3 d7d6 b5c6 b7c6 e1g1 e8g8 a1b1 c8e6 c3c4 a8b8 b1b8 d8b8 c1d2
info depth 14 seldepth 25 score cp 20 nodes 3321320 nps 3930556 time 845 multipv 3 pv d2d3 b8c6 f2f4 d7d5 e4d5 d8d5 b1c3 f8b4 g1e2 c8g4 f4e5 d5e5 h2h3 b4c3 b2c3 g4h5
info depth 15 seldepth 25 score cp 28 nodes 5487280 nps 4410996 time 1244 multipv 1 pv g1f3 g8f6 f3e5 d7d6 e5f3 f6e4 f1b5 c7c6 b5e2 f8e7 d2d3 e4f6 b1c3 e8g8 e1g1 f8e8 c1d2 b8a6 a2a3 c8d7 d3d4 d6d5
info depth 15 seldepth 25 score cp 24 nodes 5487280 nps 4410996 time 1244 multipv 2 pv b1c3 g8f6 g1f3 b8c6 f1c4 f8b4 e1g1 d7d6 c3d5 f6e4 d2d4 b4a5 c2c3 c8e6 d1b3 e6d5 c4d5
info depth 15 seldepth 25 score cp 0 nodes 5487280 nps 4410996 time 1244 multipv 3 pv d2d3 b8c6 f2f4 d7d6 b1c3 f7f5 g1f3 g8f6 f1e2 f8e7 e1g1 e8g8 c1d2 c8d7 e4f5 d7f5 f4e5 d6e5 f3g5 e7c5
info depth 16 seldepth 27 score cp 24 nodes 6971682 nps 4771856 time 1461 multipv 1 pv g1f3 g8f6 b1c3 b8c6 f1c4 f8b4 e1g1 d7d6 c3d5 f6e4 d2d4 b4a5 c2c3 c8e6 d1b3 e6d5 c4d5
info depth 16 seldepth 27 score cp 24 nodes 6971682 nps 4771856 time 1461 multipv 2 pv b1c3 g8f6 g1f3 b8c6 f1c4 f8b4 e1g1 d7d6 c3d5 f6e4 d2d4 b4a5 c2c3 c8e6 d1b3 e6d5 c4d5
info depth 16 seldepth 27 score cp 12 nodes 6971682 nps 4771856 time 1461 multipv 3 pv d2d3 b8c6 f2f4 d7d6 b1c3 f7f5 g1f3 g8f6 f1e2 f8e7 e1g1 e8g8 c1d2 c8d7 a1b1 f5e4 d3e4 e5f4 d2f4
info depth 17 seldepth 27 score cp 48 nodes 11490781 nps 5443288 time 2111 multipv 1 pv g1f3 b8c6 f1b5 a7a6 b5c6 d7c6 e1g1 c8g4 d2d3 g8f6 b1d2 f8d6 d2c4 e8g8 c1d2 b7b5 c4e3 g4f3 d1f3 d6c5
info depth 17 seldepth 27 score cp 24 nodes 11490781 nps 5443288 time 2111 multipv 2 pv b1c3 g8f6 g1f3 b8c6 f1b5 f8b4 e1g1 e8g8 d2d3 b4c3 b2c3 d7d6 c1d2 c8d7 a1b1 h7h6 c3c4 a7a5 c4c5 c6b4 b5d7 d8d7 c5d6
info depth 17 seldepth 27 score cp 12 nodes 11490781 nps 5443288 time 2111 multipv 3 pv d2d3 b8c6 f2f4 d7d6 b1c3 f7f5 g1f3 g8f6 f1e2 f8e7 e1g1 e8g8 c1d2 c8d7 a1b1 f5e4 d3e4 e5f4 d2f4
info depth 18 seldepth 31 score cp 44 nodes 18842562 nps 6121690 time 3078 multipv 1 pv g1f3 b8c6 f1b5 g8f6 e1g1 f8d6 d2d3 a7a6 b5a4 e8g8 b1c3 b7b5 a4b3 c8b7 c1e3 c6a5 f1e1 c7c6 h2h3
info depth 18 seldepth 31 score cp 24 nodes 18842562 nps 6121690 time 3078 multipv 2 pv b1c3 g8f6 g1f3 b8c6 f1b5 f8b4 e1g1 e8g8 d2d3
info depth 18 seldepth 31 score cp 20 lowerbound nodes 18842562 nps 6121690 time 3078 multipv 3 pv d2d3 b8c6 f2f4 d7d6 b1c3 f7f5 g1f3 g8f6 f1e2 f8e7 e1g1 e8g8 g1h1 f5e4 d3e4 c8e6 f4f5
info depth 18 currmove d2d3 currmovenumber 3
info depth 18 currmove d1h5 currmovenumber 4
info depth 18 currmove d1e2 currmovenumber 9
info depth 18 currmove g2g4 currmovenumber 11
info depth 18 currmove d1f3 currmovenumber 16
info depth 18 currmove f1a6 currmovenumber 18
info depth 18 currmove f2f3 currmovenumber 19
info depth 18 currmove b2b4 currmovenumber 22
info depth 18 currmove a2a3 currmovenumber 24
info depth 18 currmove g2g3 currmovenumber 28
info depth 18 seldepth 31 score cp 44 nodes 19816033 nps 6200260 time 3196 multipv 1 pv g1f3 b8c6 f1b5 g8f6 e1g1 f8d6 d2d3 a7a6 b5a4 e8g8 b1c3 b7b5 a4b3 c8b7 c1e3 c6a5 f1e1 c7c6 h2h3
info depth 18 seldepth 31 score cp 24 nodes 19816033 nps 6200260 time 3196 multipv 2 pv b1c3 g8f6 g1f3 b8c6 f1b5 f8b4 e1g1 e8g8 d2d3
info depth 18 seldepth 31 score cp 4 upperbound nodes 19816033 nps 6200260 time 3196 multipv 3 pv d2d3 b8c6 f2f4 d7d6 b1c3 f7f5 g1f3 g8f6 f1e2 f8e7 e1g1 e8g8 g1h1 e5f4 c1f4 c8d7 e4f5 d7f5
info depth 18 currmove d2d3 currmovenumber 3
info depth 18 currmove d1f3 currmovenumber 7
info depth 18 seldepth 31 score cp 44 nodes 20778251 nps 6230360 time 3335 multipv 1 pv g1f3 b8c6 f1b5 g8f6 e1g1 f8d6 d2d3 a7a6 b5a4 e8g8 b1c3 b7b5 a4b3 c8b7 c1e3 c6a5 f1e1 c7c6 h2h3
info depth 18 seldepth 31 score cp 24 nodes 20778251 nps 6230360 time 3335 multipv 2 pv b1c3 g8f6 g1f3 b8c6 f1b5 f8b4 e1g1 e8g8 d2d3
info depth 18 seldepth 31 score cp 12 nodes 20778251 nps 6230360 time 3335 multipv 3 pv d2d3 b8c6 f2f4 d7d6 b1c3 f7f5 g1f3 g8f6 f1e2 f8e7 e1g1 e8g8 g1h1 e5f4 c1f4 c8d7 a1c1 a7a6 e4f5 d7f5 d3d4 d6d5
info depth 19 currmove g1f3 currmovenumber 1
info depth 19 currmove c2c3 currmovenumber 4
info depth 19 currmove a2a4 currmovenumber 28
info depth 19 seldepth 31 score cp 36 upperbound nodes 22031109 nps 6253508 time 3523 multipv 1 pv g1f3 b8c6 f1b5 g8f6 e1g1 f8d6 d2d3 a7a6 b5a4 e8g8 b1c3 b7b5 a4b3 c8b7 c1e3 c6a5 f1e1 h7h6 h2h3 g8h8
info depth 18 seldepth 31 score cp 24 nodes 22031109 nps 6253508 time 3523 multipv 2 pv b1c3 g8f6 g1f3 b8c6 f1b5 f8b4 e1g1 e8g8 d2d3
info depth 18 seldepth 31 score cp 12 nodes 22031109 nps 6253508 time 3523 multipv 3 pv d2d3 b8c6 f2f4 d7d6 b1c3 f7f5 g1f3 g8f6 f1e2 f8e7 e1g1 e8g8 g1h1 e5f4 c1f4 c8d7 a1c1 a7a6 e4f5 d7f5 d3d4 d6d5

[hgm]

Sorting is not required (per UCI specifications), therefore zurichess conforms to the UCI protocol.

Indeed, but an engine that responds with "resign" to any "go" command conforms to WB protocol, and is completely useless. So conforming to a protocol is only a small victory.

[brtzsnr]

After adding multipv now I added Skill Levels. I uploaded compilations for different os/platforms to [1]. I would be grateful if anyone wants to help me test this feature, too.

To use it

Code: Select all

setoption name Skill Level value 5

For example

Code: Select all

$ ./zurichess 
zurichess graubuenden https://bitbucket.org/zurichess/zurichess
build with devel +f3d5478 Fri Jun 17 19:15:29 2016 +0000 at 2016-07-13 00:26:37, running on amd64
setoption name Skill Level value 5
go depth 5
info depth 0 seldepth 0 multipv 1 score cp 0 nodes 2 time 0 nps 49422 pv
info depth 1 seldepth 1 multipv 1 score cp 104 nodes 95 time 0 nps 652208 pv e2e3
info depth 1 seldepth 1 multipv 2 score cp 98 nodes 95 time 0 nps 626194 pv e2e4
info depth 1 seldepth 1 multipv 3 score cp 80 nodes 95 time 0 nps 619655 pv b1c3
info depth 2 seldepth 2 multipv 1 score cp 0 nodes 581 time 0 nps 1025570 pv e2e3 e7e6
info depth 2 seldepth 2 multipv 2 score cp -6 nodes 581 time 0 nps 1019749 pv e2e4 e7e6
info depth 2 seldepth 2 multipv 3 score cp -15 nodes 581 time 0 nps 1016898 pv d2d4 e7e6
info depth 3 seldepth 3 multipv 1 score cp 80 nodes 1711 time 1 nps 1129982 pv e2e3 e7e6 b1c3
info depth 3 seldepth 3 multipv 2 score cp 80 nodes 1711 time 1 nps 1124945 pv b1c3 e7e6 e2e3
info depth 3 seldepth 3 multipv 3 score cp 78 nodes 1711 time 1 nps 1123559 pv d2d4 d7d5 g1f3
info depth 4 seldepth 4 multipv 1 score cp 0 nodes 5803 time 4 nps 1197991 pv e2e3 e7e6 d2d4 d7d5
info depth 4 seldepth 4 multipv 2 score cp 0 nodes 5803 time 4 nps 1196765 pv d2d4 d7d5 g1f3 g8f6
info depth 4 seldepth 4 multipv 3 score cp -9 nodes 5803 time 4 nps 1196236 pv b1c3 d7d5 e2e4 e7e6 e4d5 e6d5
info depth 5 seldepth 5 multipv 1 score cp 72 nodes 10999 time 9 nps 1197111 pv b1c3 d7d5 e2e4 d5e4 c3e4
info depth 5 seldepth 5 multipv 2 score cp 55 nodes 10999 time 9 nps 1196458 pv d2d4 g8f6 g1f3 d7d6 c2c4
info depth 5 seldepth 5 multipv 3 score cp 55 nodes 10999 time 9 nps 1195152 pv e2e3 e7e6 d2d4 d7d5 g1f3
bestmove e2e3 ponder e7e6

I ran a quick tournament:

Code: Select all

Rank Name                          ELO     +/-   Games   Score   Draws
   1 sl0                           396      13    6707     91%      6%
   2 fish12                        237      10    6707     80%      9%
   3 sl2                           152       9    6707     71%      8%
   4 sl4                            16       8    6707     52%      9%
   5 fish8                         -75       8    6707     39%      8%
   6 sl6                          -113       8    6707     34%      9%
   7 sl8                          -228      10    6706     21%      8%
   8 sl10                         -346      12    6706     12%      5%

Some questions I'm interested in
1) Does it work with your favorite GUI?
2) Is it ok to print more than 1 PV when skilllevel is > 0 and multipv == 1.
3) Is skill level 10 's Elo low enough for a casual player (I don't count).
4) Do I need more randomization in choosing the pv? Right now it's based on position's zobrist key.


Disclaimer: This is pre-alpha release. I'll do a full release in August or so, after I have more confidence MultiPV and Skill Level options work fine.


[1]

[petero2]

The UCI specs do not say that the 'min' and 'max' parameters are required, right? So if there is no maximum to a spin option you could just leave it out.

I think it does. From the specification:

Code: Select all

* option
...
	For all allowed combinations see the examples below,
	as some combinations of this tokens don't make sense.
...
	Examples:
	Here are 5 strings for each of the 5 possible types of options
		"option name Nullmove type check default true\n"
		"option name Selectivity type spin default 2 min 0 max 4\n"
		"option name Style type combo default Normal var Solid var Normal var Risky\n"
		"option name NalimovPath type string default c:\\n"
		"option name Clear Hash type button\n"

Since there is no spin option without min/max in the examples, it is not an allowed combination to leave them out.

[hgm]

These are presented as examples, not as an exhaustive list. I don't think you can conclude from it, for instance, that combo options must always have 3 possible settings, or that check options must always have default value 'true'. As far as I am concerned it says that min/max are only allowed on spin options, not that they are mandatory on such options. It makes little sense to require specification of a maximum on a setting that has no maximum. If you provide examples, it makes sense to show the most elaborate cases.