AlphaZero won Stockfish 8 +28 =72 -0 with 64%
I have played 100 games without opening book between Lc0 version 0.19 rc2 with net 11261 on Asus Geforce GTX 1060 3 Gb tc=1500+10 (secs) and Stockfish 8 on i7 4790S 6 logical cores tc=710+4.73 (secs)
that is equivalent to LeR=1 (same speed proportion of AlphaZero-Stockfish 8), ponder Off, no endgame tablebases, Hash tables 1 Gb. under Cutechess cli. Turbo Boost disabled to avoid speed variations due to other processes.
At 25 min plus 10 sec Lc0 has won Stockfish 8 with a score +14 =77 -9 and 52.5%
At 5 min plus 3 sec Lc0 has won Stockfish 8 with a score +14 =75 -11 and 51.5%
At 1 min plus 1 sec Stockfish 8 has won Lc0 with a score +16 =77 -7 and 54.5%
the evolution of the score suggests that with the hardware of the match Alphazero-Stockfish the victory of Lc0 would be wider. Unfortunately to get that amount of computation we need aprox 24 minutes per move on the GTX 1060 and to play 100 games about one year!!
I consider that one side has advantage if Stockfish 8 eval is greater than or equal to 1 or less than or equal to -1 (I consider eval of lc0 is not as precise as Stockfish`s)
In the analysis of the 100 games for time control 25 min+10 sec I write down the number of the move with the abs(Stockfish eval)>=1 and the result of the game.
lc0 got advantage in 22 times with an average of 32,55 of move and an average score 0,82.
Stockfish 8 got advantage in 10 times with an average of 52,7 of move and an average score 0,95.
The effectiveness of Stockfish is superior to the lc0 one.
The number of advantages of lc0 is higher than Stockfish one.
The average movement where the advantage is reached is much lower for lc0.
All this indicates that lc0 is superior in the opening than Stockfish but the advantage decreases as the game progresses and many times the game ends as a draw.
For sf8 it is more difficult to get an advantage but when he does he rarely loses it.
Advantage Lc0
Color num move result
W 1 24 0,5
W 3 28 0,5
B 6 83 0,5
W 13 26 1
W 17 30 1
W 21 26 0,5
W 23 25 1
B 28 31 1
W 31 23 0,5
W 33 24 1
W 35 37 1
W 43 28 1
W 53 24 0,5
B 70 40 1
B 72 27 1
W 73 45 0,5
B 78 45 0,5
W 79 22 1
W 81 44 1
W 83 35 1
W 85 24 1
W 97 25 1
22 32,55 0,82
Advantage Stockfish
Color num move result
W 2 38 1
W 10 56 0,5
W 12 45 1
W 20 50 1
W 38 52 1
W 46 32 1
W 64 50 1
W 67 45 1
W 82 114 1
W 84 45 1
10 52,7 0,95
After finished this test I read a test done by Greg Mattson, see
https://groups.google.com/forum/#!msg/l ... BI5EiMAwAJ
He played 24 games using 2 billion nodes per move (sf9, no opening book or TB) and 2.5 million nodes per move (lc0 11248) for a leela ratio of 1,
with results
lc0: 15.5/24 1=1===11=======1====1=1=
sf9: 8.5/24 0=0===00=======0====0=0=
The amount of computation is about the half of the AlphaZero-Stockfish 8 match.
The hardware was: i7-6700K 4 core, 4.00GHz cpu processor, GTX 1080 x 2 with 1 game per day
My point of view:
1) All these tests suggest that Lc0 is not so far from A0 as everyone thinks
2) Comparing this test with other tests I have done, where lc0 got much worse results, see for example
http://talkchess.com/forum3/viewtopic.p ... 40#p774276
the sensitivity in the level of play of lc0 in the election of the opening is evident.
I think the best possible scenario for lc0 and possibly for A0 is playing from the initial position, we must not forget that these programs have learned to play this way.
3) The opening book is an addition that most alpha-beta programs need but lc0 does not, this explains the good result of lc0
4) I have no doubt that the Google experiment was a real revolution for computer chess, however it is a pity that we do not have access to all the games played by A0 and that Sf8 would not have played with the opening book to be able to check the superiority of A0.
Pgn files:
https://sites.google.com/site/unoallavo ... ects=0&d=1
https://sites.google.com/site/unoallavo ... ects=0&d=1
https://sites.google.com/site/unoallavo ... ects=0&d=1
Comparing Leela Chess Zero with AlphaZero
Moderators: hgm, Rebel, chrisw
-
- Posts: 200
- Joined: Fri Oct 12, 2012 12:48 pm
- Location: Seville (SPAIN)
- Full name: Javier Ros
-
- Posts: 251
- Joined: Sat Dec 02, 2006 10:47 pm
- Location: Toronto
- Full name: Peter Kasinski
Re: Comparing Leela Chess Zero with AlphaZero
Interesting test, thanks for sharing.
My impressions are similar. I find Leela strong in complex early middlegame positions, but perhaps the most striking feature is the ability to seize on a permanent pawn structure weakness, often around the enemy king. In many such games Leela plays as if on autopilot.
I found your comparison of the relative strength of Lc0 and A0 informative.
Cheers,
PK
My impressions are similar. I find Leela strong in complex early middlegame positions, but perhaps the most striking feature is the ability to seize on a permanent pawn structure weakness, often around the enemy king. In many such games Leela plays as if on autopilot.
I found your comparison of the relative strength of Lc0 and A0 informative.
Cheers,
PK
-
- Posts: 200
- Joined: Fri Oct 12, 2012 12:48 pm
- Location: Seville (SPAIN)
- Full name: Javier Ros
Re: Comparing Leela Chess Zero with AlphaZero
This was my purpose.kasinp wrote: ↑Wed Nov 14, 2018 5:06 am Interesting test, thanks for sharing.
My impressions are similar. I find Leela strong in complex early middlegame positions, but perhaps the most striking feature is the ability to seize on a permanent pawn structure weakness, often around the enemy king. In many such games Leela plays as if on autopilot.
I found your comparison of the relative strength of Lc0 and A0 informative.
Cheers,
PK
I have forgotten to reference the interesting experiment of Laskos, see
http://talkchess.com/forum3/viewtopic.p ... 60#p778260
where he did a similar test with Leela Ratio approx. 2. I thought that his experiments were important and they had to be revisited with LeR=1.
On the other hand, the statistics on advantages point in the same direction as the Laskos experiments, see
Official: Lc0 is the strongest engine In the Openings and Middlegames
http://talkchess.com/forum3/viewtopic.php?f=2&t=68621
-
- Posts: 708
- Joined: Mon Jan 16, 2012 6:34 am
Re: Comparing Leela Chess Zero with AlphaZero
Probably Leela is stronger than SF without opening book, but " What about SF having access to brainfish database"?
-
- Posts: 106
- Joined: Tue Feb 07, 2012 11:14 am
Re: Comparing Leela Chess Zero with AlphaZero
Yes i agree! And Lc0 is number one in the CCC 2 Blitz Battle.That says everything
-
- Posts: 1470
- Joined: Mon Apr 23, 2018 7:54 am
Re: Comparing Leela Chess Zero with AlphaZero
Depends which "everyone" you mean. There've been growing claims for many months now that Lc0 has already passed A0.Javier Ros wrote: ↑Wed Nov 14, 2018 12:53 am My point of view:
1) All these tests suggest that Lc0 is not so far from A0 as everyone thinks
e.g. one of the estimate lists has
"Alpha Zero 1080Ti [No Handicaps] 3545"
and a group of Lc0s 1124x, 1125x around the same.
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: Comparing Leela Chess Zero with AlphaZero
Opening book for Stockfish is a big plus for it against Leela. A small book BookX.bin, enabled SF_dev to outperform its performance compared to always starting from initial positions by 120 Elo points. The results are here:Nay Lin Tun wrote: ↑Wed Nov 14, 2018 8:34 am Probably Leela is stronger than SF without opening book, but " What about SF having access to brainfish database"?
http://talkchess.com/forum3/viewtopic.p ... 0&start=11
So, it was a wrong methodology from DeepMind using just 1 starting position for 100 games. Might also be a premeditated one, as A0 gets much better results in the way they performed (wrongly) the test.
Thanks Javier for these interesting experiments.
-
- Posts: 10282
- Joined: Thu Mar 09, 2006 12:37 am
- Location: Tel-Aviv Israel
-
- Posts: 106
- Joined: Tue Feb 07, 2012 11:14 am
Re: Comparing Leela Chess Zero with AlphaZero
Yes Ironic! Every day someone claims Lc0 is better than ...