Comparing Leela Chess Zero with AlphaZero

[Javier Ros]

AlphaZero won Stockfish 8 +28 =72 -0 with 64%

I have played 100 games without opening book between Lc0 version 0.19 rc2 with net 11261 on Asus Geforce GTX 1060 3 Gb tc=1500+10 (secs) and Stockfish 8 on i7 4790S 6 logical cores tc=710+4.73 (secs)

that is equivalent to LeR=1 (same speed proportion of AlphaZero-Stockfish 8), ponder Off, no endgame tablebases, Hash tables 1 Gb. under Cutechess cli. Turbo Boost disabled to avoid speed variations due to other processes.

At 25 min plus 10 sec Lc0 has won Stockfish 8 with a score +14 =77 -9 and 52.5%

At 5 min plus 3 sec Lc0 has won Stockfish 8 with a score +14 =75 -11 and 51.5%

At 1 min plus 1 sec Stockfish 8 has won Lc0 with a score +16 =77 -7 and 54.5%

the evolution of the score suggests that with the hardware of the match Alphazero-Stockfish the victory of Lc0 would be wider. Unfortunately to get that amount of computation we need aprox 24 minutes per move on the GTX 1060 and to play 100 games about one year!!

I consider that one side has advantage if Stockfish 8 eval is greater than or equal to 1 or less than or equal to -1 (I consider eval of lc0 is not as precise as Stockfish`s)
In the analysis of the 100 games for time control 25 min+10 sec I write down the number of the move with the abs(Stockfish eval)>=1 and the result of the game.

lc0 got advantage in 22 times with an average of 32,55 of move and an average score 0,82.
Stockfish 8 got advantage in 10 times with an average of 52,7 of move and an average score 0,95.

The effectiveness of Stockfish is superior to the lc0 one.
The number of advantages of lc0 is higher than Stockfish one.
The average movement where the advantage is reached is much lower for lc0.

All this indicates that lc0 is superior in the opening than Stockfish but the advantage decreases as the game progresses and many times the game ends as a draw.
For sf8 it is more difficult to get an advantage but when he does he rarely loses it.

Advantage Lc0
Color num move result
W 1 24 0,5
W 3 28 0,5
B 6 83 0,5
W 13 26 1
W 17 30 1
W 21 26 0,5
W 23 25 1
B 28 31 1
W 31 23 0,5
W 33 24 1
W 35 37 1
W 43 28 1
W 53 24 0,5
B 70 40 1
B 72 27 1
W 73 45 0,5
B 78 45 0,5
W 79 22 1
W 81 44 1
W 83 35 1
W 85 24 1
W 97 25 1
22 32,55 0,82

Advantage Stockfish
Color num move result
W 2 38 1
W 10 56 0,5
W 12 45 1
W 20 50 1
W 38 52 1
W 46 32 1
W 64 50 1
W 67 45 1
W 82 114 1
W 84 45 1
10 52,7 0,95

After finished this test I read a test done by Greg Mattson, see

https://groups.google.com/forum/#!msg/l ... BI5EiMAwAJ

He played 24 games using 2 billion nodes per move (sf9, no opening book or TB) and 2.5 million nodes per move (lc0 11248) for a leela ratio of 1,
with results

lc0: 15.5/24 1=1===11=======1====1=1=
sf9: 8.5/24 0=0===00=======0====0=0=

The amount of computation is about the half of the AlphaZero-Stockfish 8 match.

The hardware was: i7-6700K 4 core, 4.00GHz cpu processor, GTX 1080 x 2 with 1 game per day

My point of view:

1) All these tests suggest that Lc0 is not so far from A0 as everyone thinks

2) Comparing this test with other tests I have done, where lc0 got much worse results, see for example

http://talkchess.com/forum3/viewtopic.p ... 40#p774276

the sensitivity in the level of play of lc0 in the election of the opening is evident.

I think the best possible scenario for lc0 and possibly for A0 is playing from the initial position, we must not forget that these programs have learned to play this way.

3) The opening book is an addition that most alpha-beta programs need but lc0 does not, this explains the good result of lc0

4) I have no doubt that the Google experiment was a real revolution for computer chess, however it is a pity that we do not have access to all the games played by A0 and that Sf8 would not have played with the opening book to be able to check the superiority of A0.

Pgn files:

https://sites.google.com/site/unoallavo ... ects=0&d=1

https://sites.google.com/site/unoallavo ... ects=0&d=1

https://sites.google.com/site/unoallavo ... ects=0&d=1

[kasinp]

Interesting test, thanks for sharing.
My impressions are similar. I find Leela strong in complex early middlegame positions, but perhaps the most striking feature is the ability to seize on a permanent pawn structure weakness, often around the enemy king. In many such games Leela plays as if on autopilot.
I found your comparison of the relative strength of Lc0 and A0 informative.

Cheers,
PK

[Javier Ros]

Interesting test, thanks for sharing.
My impressions are similar. I find Leela strong in complex early middlegame positions, but perhaps the most striking feature is the ability to seize on a permanent pawn structure weakness, often around the enemy king. In many such games Leela plays as if on autopilot.
I found your comparison of the relative strength of Lc0 and A0 informative.

Cheers,
PK

This was my purpose.

I have forgotten to reference the interesting experiment of Laskos, see

http://talkchess.com/forum3/viewtopic.p ... 60#p778260

where he did a similar test with Leela Ratio approx. 2. I thought that his experiments were important and they had to be revisited with LeR=1.

On the other hand, the statistics on advantages point in the same direction as the Laskos experiments, see

Official: Lc0 is the strongest engine In the Openings and Middlegames

http://talkchess.com/forum3/viewtopic.php?f=2&t=68621

[Nay Lin Tun]

Probably Leela is stronger than SF without opening book, but " What about SF having access to brainfish database"?

[marsell]

Yes i agree! And Lc0 is number one in the CCC 2 Blitz Battle.That says everything

[jp]

My point of view:

1) All these tests suggest that Lc0 is not so far from A0 as everyone thinks

Depends which "everyone" you mean. There've been growing claims for many months now that Lc0 has already passed A0.
e.g. one of the estimate lists has
"Alpha Zero 1080Ti [No Handicaps] 3545"
and a group of Lc0s 1124x, 1125x around the same.

[Laskos]

Probably Leela is stronger than SF without opening book, but " What about SF having access to brainfish database"?

Opening book for Stockfish is a big plus for it against Leela. A small book BookX.bin, enabled SF_dev to outperform its performance compared to always starting from initial positions by 120 Elo points. The results are here:

http://talkchess.com/forum3/viewtopic.p ... 0&start=11

So, it was a wrong methodology from DeepMind using just 1 starting position for 100 games. Might also be a premeditated one, as A0 gets much better results in the way they performed (wrongly) the test.

Thanks Javier for these interesting experiments.

[Uri Blass]

Yes i agree! And Lc0 is number one in the CCC 2 Blitz Battle.That says everything

I see stockfish is leading in the CCC 2 blitz battle

stockfish 37.5/59
komodo 28/59
lc0 27.5/59
houdini 25/59

[marsell]

Yes Ironic! Every day someone claims Lc0 is better than ...