Leela Chess Zero against Stockfish, Houdini and Komodo

[Javier Ros]

In this test 100 games were played using Silver Opening Suite with 50 positions from

https://en.chessbase.com/post/test-your ... ings-suite

"This openings suite is comprised of 50 positions, leading to a round 100-game match, covering not only the major openings, but also standard pawn structures and opening systems. Therefore not only will you find an Alekhine, some French Defenses, and several flavors of the Sicilian, but there are also a Hedgehog, a Stonewall, a Colle, and a King’s Indian Attack for example.
To determine their frequency, the number of positions for each opening found in the suite is proportionate to the number of times played as per statistics culled from Mega Database 2010. In other words, the number of Sicilians found in the suite (9), is roughly proportionate to the number found in the 4.5 million game database. This isn’t true in an absolute sense, as minor concessions were made to favor variety as well.

That said, if your goal isn’t only to test engine strength, but also to watch the engines duke it, the most entertaining moments are often when all hell breaks loose. So, a second suite of 25 positions was put together, based entirely on gambits. In fact, three of the positions even start with a piece sacrifice, and while they may not hold up in long time-controls, if you watch them at five-minute games, chaos and cutthroat chess are virtually guaranteed.

Nevertheless, two criteria were applied in the selection:
The engines could not start out believing they were dead lost, which would have been dreadfully unfair. Thus a maximum -0.70 self-evaluation limit was set.
The theory could not be reduced to finding ‘only-moves’, meaning only one move is playable, since this would be tantamount to a tactical test suite rather than an openings playing suite."


Lc0 version 0.18 rc2 with net 11261 on Asus Geforce GTX 1060 3 Gb (300$) against Stockfish 9, Houdini 6 Pro and Komodo 12 with 4 threads on i7 4790S (300$) with 4 cores and 8 threads, ponder Off, Hash tables 1 Gb. for all, under Arena gui. Time 5 minutes plus 3 seconds.

I want to experiment with tablebases support for lc0, 5 men syzygy on SSD disk, to see how it is working.
Stockfish 9, Houdini 6 and Komodo 12 played without syzygy tablebases.
LeR=3300*875/4500000=0.64

Stockfish_9_x64_bmi2 - Lc018rc211261syzygy : 57,5/100 24-9-67 (===0=0=====1=1===1=1=101=1=1===01====1=====1======0======111=======1=0======1=0===111==1=11====110=0) 57.5%
Lc018rc211261syzygy - Stockfish_9_x64_bmi2 : 42,5/100 9-24-67 (===1=1=====0=0===0=0=010=0=0===10====0=====0======1======000=======0=1======0=1===000==0=00====001=1) 42,5%

Complete pgn from Arena at
https://sites.google.com/site/unoallavo ... ects=0&d=1

H6_Pro_x64-pext - Lc018rc211261syzygy : 53,0/100 18-12-70 (======0====001=1=1===10=00==0===1=1===0====1===1101==1===11==1=========1=========0==1====10====1===0) 53%
Lc018rc211261syzygy - H6_Pro_x64-pext : 47,0/100 12-18-70 (======1====110=0=0===01=11==1===0=0===1====0===0010==0===00==0=========0=========1==0====01====0===1) 47%

https://sites.google.com/site/unoallavo ... ects=0&d=1

Komodo-12-bmi2-64bit - Lc018rc211261syzygy : 46,5/100 18-25-57 (=1===1=00=0=00=====10=0101==01===011======01===10===01==0=1===0=====01====010=000=0===01==1====1=001) 46,5%
Lc018rc211261syzygy - Komodo-12-bmi2-64bit : 53,5/100 25-18-57 (=0===0=11=1=11=====01=1010==10===100======10===01===10==1=0===1=====10====101=111=1===10==0====0=110) 53,5%

https://sites.google.com/site/unoallavo ... ects=0&d=1

After the test started it was known that the implementation of syzygy was buggy, so I rerun the test with fixed version lc0 0.18.1 against Stockfish 9 and the score of lc0 improved slightly

Stockfish_9_x64_bmi2 - Lc0181TB11261 : 56,0/100 21-9-70 (1==1=00===========11=001====1=1====1=1==1==10==1011====1=====1==11===10=0========1=============1===0) 56%
Lc0181TB11261 - Stockfish_9_x64_bmi2 : 44,0/100 9-21-70 (0==0=11===========00=110====0=0====0=0==0==01==0100====0=====0==00===01=1========0=============0===1) 44%

https://sites.google.com/site/unoallavo ... ects=0&d=1

[Javier Ros]

The tactical errors appeared again on the side of lc0, like in the game 82 of the last match.
After lc0 played 63...Kc5 +0.35

[pgn][Event "Computer chess game"] [Date "2018.10.04"] [Round "82"] [White "Stockfish_9_x64_popcnt"] [Black "lc0 0.18.1 11261"] [Result "*"] [TimeControl "300+3"] [SetUp "1"] [FEN "8/8/8/ppk4p/5N2/P4qPK/1P1R1P1P/8 w - - 27 64"] [Termination "unterminated"] [PlyCount "0"] 64. Ne6+ Kc4 65.Rd4+ Kb3 66.Rd3+ Qd3 67.Nc5+ K * [/pgn]




Stockfish9 played 64.Ne6+ +0.39 expecting Kb6, but after lc0 blunder
64... Kc4?? +0.37
65.Rd4+ +7.87 Stockfish 9 won easily.