So, it doesn't matter as methodology goes, 1200 games from only 12 starting positions usually give skewed results. And, hardly accidentally, 100 Elo points better results for Lc0 and A0 than from varied openings. My English is bad, maybe you had to explain me simpler: yes, there are only 12 openings, but they are the most popular human openings. I would have replied that the methodology for determining the strength compared to SF8 was still wrong, and it seems by now hardly by accident.Tobber wrote: ↑Mon Sep 24, 2018 12:48 pmSo the 12 most popular openings in human play favor Lc0 and A0? The idea was to show that A0 by self-play could figure out how to play the most popular openings. What is it you don't understand?Laskos wrote: ↑Mon Sep 24, 2018 11:22 amI don't understand what you are saying, and I won't read again the paper right now. How many positions they did use for 1200 games match? The fact is the 12 provided positions are very favorable for Lc0 and probably A0. Similarly, no openings at all heavily favors Lc0 and A0.Tobber wrote: ↑Mon Sep 24, 2018 11:14 amPlease read the paper and then check for yourself. The 1200 games match were selected as the 12 most popular openings from an online database.Laskos wrote: ↑Mon Sep 24, 2018 9:45 am
Our differences probably come from using different set of openings, as Lc0 and A0 seem sensitive to the choice. 12 openings for 1200 games match and 1 opening for 100 games match is bad testing methodology by A0 team. And probably on purpose bad, as it bolster A0 (or Lc0) by about 100 Elo points.
/John
/John
leela is official(?) better than sf9
Moderators: hgm, Rebel, chrisw
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: leela is official(?) better than sf9
-
- Posts: 3657
- Joined: Wed Nov 18, 2015 11:41 am
- Location: hungary
Re: leela is official(?) better than sf9
...and bigger NN.chrisw wrote: ↑Mon Sep 24, 2018 10:24 amThe fewer pieces there are, especially pieces that can combine in mating nets, the stronger is the AB search (the search width is naturally smaller, AB is less likely to prune away important lines. and, the opportunities to create imbalances are fewer, if there is an imbalance, it is probably already there). NN-MCTS is better at finding imbalances and potential mating net structures, the opportunity for that is early on.corres wrote: ↑Mon Sep 24, 2018 10:09 am AB engines are tested mainly for middle game.
NN engines play learning games from start position so their knowledge decreases going further from start
position. These facts cause that starting the games from a sort opening book or without opening book is favorable for NN engines and disadvantageous for AB engines.
Your point of greater learnt knowledge at start (eg most pieces still on board) is also true. What's also true is that there is no reason why more training and larger nets can't overcome the weaknesses in later parts of the game. Just give it training time.
If NN is too small during a longer training time the good content of NN will be overwritten.
-
- Posts: 143
- Joined: Wed Jan 17, 2018 1:26 pm
Re: leela is official(?) better than sf9
Bigger NN is a part of it, but this can be substituted or augmented with "better NN". What I'm talking about here is to change the neural net architecture from the standard residual neural net (ResNet), to a higher performing one. https://arxiv.org/pdf/1709.01507.pdf shows that significant improvements to ResNet are feasible for image recognition, by using SEResNet (Squeeze-and-Excitation), or even combining this technique with ResNeXt, a generalisation of the ResNet idea. The SEResNet architecture was tried on the standard dataset for Lc0 and achieved a significantly lower MSE and policy loss compared to our regular networks.corres wrote: ↑Mon Sep 24, 2018 1:08 pm...and bigger NN.chrisw wrote: ↑Mon Sep 24, 2018 10:24 amThe fewer pieces there are, especially pieces that can combine in mating nets, the stronger is the AB search (the search width is naturally smaller, AB is less likely to prune away important lines. and, the opportunities to create imbalances are fewer, if there is an imbalance, it is probably already there). NN-MCTS is better at finding imbalances and potential mating net structures, the opportunity for that is early on.corres wrote: ↑Mon Sep 24, 2018 10:09 am AB engines are tested mainly for middle game.
NN engines play learning games from start position so their knowledge decreases going further from start
position. These facts cause that starting the games from a sort opening book or without opening book is favorable for NN engines and disadvantageous for AB engines.
Your point of greater learnt knowledge at start (eg most pieces still on board) is also true. What's also true is that there is no reason why more training and larger nets can't overcome the weaknesses in later parts of the game. Just give it training time.
If NN is too small during a longer training time the good content of NN will be overwritten.
Long story short, our current neural nets are nowhere near the top performing network architectures, and when we get a working implementation of an improved architecture, we can expect a significant jump in performance.
-
- Posts: 4319
- Joined: Tue Apr 03, 2012 4:28 pm
Re: leela is official(?) better than sf9
Sounds more than brilliant. I've been hunting around, not very successfully, for some kind of source in C or + variants where the heavy work of defining architecture and GPU support is already done. Fann and gann are ok, but is a bit daunting to then assemble them into architectures more complex than fully connected, and they don't do GPU anyway.jkiliani wrote: ↑Mon Sep 24, 2018 2:41 pmBigger NN is a part of it, but this can be substituted or augmented with "better NN". What I'm talking about here is to change the neural net architecture from the standard residual neural net (ResNet), to a higher performing one. https://arxiv.org/pdf/1709.01507.pdf shows that significant improvements to ResNet are feasible for image recognition, by using SEResNet (Squeeze-and-Excitation), or even combining this technique with ResNeXt, a generalisation of the ResNet idea. The SEResNet architecture was tried on the standard dataset for Lc0 and achieved a significantly lower MSE and policy loss compared to our regular networks.corres wrote: ↑Mon Sep 24, 2018 1:08 pm...and bigger NN.chrisw wrote: ↑Mon Sep 24, 2018 10:24 amThe fewer pieces there are, especially pieces that can combine in mating nets, the stronger is the AB search (the search width is naturally smaller, AB is less likely to prune away important lines. and, the opportunities to create imbalances are fewer, if there is an imbalance, it is probably already there). NN-MCTS is better at finding imbalances and potential mating net structures, the opportunity for that is early on.corres wrote: ↑Mon Sep 24, 2018 10:09 am AB engines are tested mainly for middle game.
NN engines play learning games from start position so their knowledge decreases going further from start
position. These facts cause that starting the games from a sort opening book or without opening book is favorable for NN engines and disadvantageous for AB engines.
Your point of greater learnt knowledge at start (eg most pieces still on board) is also true. What's also true is that there is no reason why more training and larger nets can't overcome the weaknesses in later parts of the game. Just give it training time.
If NN is too small during a longer training time the good content of NN will be overwritten.
Long story short, our current neural nets are nowhere near the top performing network architectures, and when we get a working implementation of an improved architecture, we can expect a significant jump in performance.
-
- Posts: 200
- Joined: Fri Oct 12, 2012 12:48 pm
- Location: Seville (SPAIN)
- Full name: Javier Ros
Re: leela is official(?) better than sf9
Page 6 of the paper:Tobber wrote: ↑Mon Sep 24, 2018 11:14 amPlease read the paper and then check for yourself. The 1200 games match were selected as the 12 most popular openings from an online database.Laskos wrote: ↑Mon Sep 24, 2018 9:45 am
Our differences probably come from using different set of openings, as Lc0 and A0 seem sensitive to the choice. 12 openings for 1200 games match and 1 opening for 100 games match is bad testing methodology by A0 team. And probably on purpose bad, as it bolster A0 (or Lc0) by about 100 Elo points.
/John
Table 2: Analysis of the 12 most popular human openings (played more than 100,000 times
in an online database (1)). Each opening is labelled by its ECO code and common name. The
plot shows the proportion of self-play training games in which AlphaZero played each opening,
against training time. We also report the win/draw/loss results of 100 game AlphaZero vs.
Stockfish matches starting from each opening, as either white (w) or black (b), from AlphaZero’s
perspective. Finally, the principal variation (PV) of AlphaZero is provided from each opening
When you force a program to play an opening variant that it would not have played on its own you are forcing it to play outside its style and the performance goes down a lot. This is particularly important for lc0 that has only trained starting from the initial position.
Last edited by Javier Ros on Mon Sep 24, 2018 3:52 pm, edited 5 times in total.
-
- Posts: 200
- Joined: Fri Oct 12, 2012 12:48 pm
- Location: Seville (SPAIN)
- Full name: Javier Ros
Re: leela is official(?) better than sf9
I agree.Laskos wrote: ↑Mon Sep 24, 2018 12:58 pmSo, it doesn't matter as methodology goes, 1200 games from only 12 starting positions usually give skewed results. And, hardly accidentally, 100 Elo points better results for Lc0 and A0 than from varied openings. My English is bad, maybe you had to explain me simpler: yes, there are only 12 openings, but they are the most popular human openings. I would have replied that the methodology for determining the strength compared to SF8 was still wrong, and it seems by now hardly by accident.Tobber wrote: ↑Mon Sep 24, 2018 12:48 pmSo the 12 most popular openings in human play favor Lc0 and A0? The idea was to show that A0 by self-play could figure out how to play the most popular openings. What is it you don't understand?Laskos wrote: ↑Mon Sep 24, 2018 11:22 amI don't understand what you are saying, and I won't read again the paper right now. How many positions they did use for 1200 games match? The fact is the 12 provided positions are very favorable for Lc0 and probably A0. Similarly, no openings at all heavily favors Lc0 and A0.Tobber wrote: ↑Mon Sep 24, 2018 11:14 amPlease read the paper and then check for yourself. The 1200 games match were selected as the 12 most popular openings from an online database.Laskos wrote: ↑Mon Sep 24, 2018 9:45 am
Our differences probably come from using different set of openings, as Lc0 and A0 seem sensitive to the choice. 12 openings for 1200 games match and 1 opening for 100 games match is bad testing methodology by A0 team. And probably on purpose bad, as it bolster A0 (or Lc0) by about 100 Elo points.
/John
/John
-
- Posts: 379
- Joined: Fri Sep 28, 2012 5:53 pm
- Location: Sweden
Re: leela is official(?) better than sf9
I see, what exactly do you agree with? That the 12 openings, selected to favor A0, happens to be the 12 most popular openings played by humans? A remarkable coincidence no doubt.Javier Ros wrote: ↑Mon Sep 24, 2018 3:42 pmI agree.Laskos wrote: ↑Mon Sep 24, 2018 12:58 pmSo, it doesn't matter as methodology goes, 1200 games from only 12 starting positions usually give skewed results. And, hardly accidentally, 100 Elo points better results for Lc0 and A0 than from varied openings. My English is bad, maybe you had to explain me simpler: yes, there are only 12 openings, but they are the most popular human openings. I would have replied that the methodology for determining the strength compared to SF8 was still wrong, and it seems by now hardly by accident.Tobber wrote: ↑Mon Sep 24, 2018 12:48 pmSo the 12 most popular openings in human play favor Lc0 and A0? The idea was to show that A0 by self-play could figure out how to play the most popular openings. What is it you don't understand?Laskos wrote: ↑Mon Sep 24, 2018 11:22 amI don't understand what you are saying, and I won't read again the paper right now. How many positions they did use for 1200 games match? The fact is the 12 provided positions are very favorable for Lc0 and probably A0. Similarly, no openings at all heavily favors Lc0 and A0.Tobber wrote: ↑Mon Sep 24, 2018 11:14 amPlease read the paper and then check for yourself. The 1200 games match were selected as the 12 most popular openings from an online database.Laskos wrote: ↑Mon Sep 24, 2018 9:45 am
Our differences probably come from using different set of openings, as Lc0 and A0 seem sensitive to the choice. 12 openings for 1200 games match and 1 opening for 100 games match is bad testing methodology by A0 team. And probably on purpose bad, as it bolster A0 (or Lc0) by about 100 Elo points.
/John
/John
/John
-
- Posts: 4190
- Joined: Wed Nov 25, 2009 1:47 am
Re: leela is official(?) better than sf9
Did you even check the openings or you are just repeating like a parrot BS from that PR manifest?
Simple example take Reti from the paper. The actual opening used is 5 moves deep. After 2.c4 most further moves by both black and white are like 2nd and 3rd possibility. So from 70k possible games after 2.c4 we end up at only 6k after 5. .. O-O.
The most popular human openings my ass. These were deliberately chosen 12 positions from 12 most popular openings that have nothing to do with popularity between humans or representably of chess. They were only selected to give A0 advantage it needed for that PR stunt to succeed.
-
- Posts: 417
- Joined: Thu Aug 02, 2012 7:48 pm
- Location: Germany
Re: leela is official(?) better than sf9
thats pretty much an imputation from my personal point of view and my experience tells me not to do so if there is no evidence for such claims..to tell people that google manipulated the entries to get a result that was intended to get because otherwise it would not have been possible or likely is not the right way to talkchess imho...if u present indicators then its normal to discuss things like that but if u have NOTHING in hand then its a pure assumption that brings google in miscredit without evidence...if ud ask me id like to prefere to get google in miscredit by having sth in hand
Wahrheiten sind Illusionen von denen wir aber vergessen haben dass sie welche sind.
-
- Posts: 200
- Joined: Fri Oct 12, 2012 12:48 pm
- Location: Seville (SPAIN)
- Full name: Javier Ros
Re: leela is official(?) better than sf9
Tobber wrote: ↑Mon Sep 24, 2018 4:16 pmI see, what exactly do you agree with? That the 12 openings, selected to favor A0, happens to be the 12 most popular openings played by humans? A remarkable coincidence no doubt.Javier Ros wrote: ↑Mon Sep 24, 2018 3:42 pmI agree.Laskos wrote: ↑Mon Sep 24, 2018 12:58 pmSo, it doesn't matter as methodology goes, 1200 games from only 12 starting positions usually give skewed results. And, hardly accidentally, 100 Elo points better results for Lc0 and A0 than from varied openings. My English is bad, maybe you had to explain me simpler: yes, there are only 12 openings, but they are the most popular human openings. I would have replied that the methodology for determining the strength compared to SF8 was still wrong, and it seems by now hardly by accident.Tobber wrote: ↑Mon Sep 24, 2018 12:48 pmSo the 12 most popular openings in human play favor Lc0 and A0? The idea was to show that A0 by self-play could figure out how to play the most popular openings. What is it you don't understand?Laskos wrote: ↑Mon Sep 24, 2018 11:22 amI don't understand what you are saying, and I won't read again the paper right now. How many positions they did use for 1200 games match? The fact is the 12 provided positions are very favorable for Lc0 and probably A0. Similarly, no openings at all heavily favors Lc0 and A0.Tobber wrote: ↑Mon Sep 24, 2018 11:14 amPlease read the paper and then check for yourself. The 1200 games match were selected as the 12 most popular openings from an online database.Laskos wrote: ↑Mon Sep 24, 2018 9:45 am
Our differences probably come from using different set of openings, as Lc0 and A0 seem sensitive to the choice. 12 openings for 1200 games match and 1 opening for 100 games match is bad testing methodology by A0 team. And probably on purpose bad, as it bolster A0 (or Lc0) by about 100 Elo points.
/John
/John
/John
"So, it doesn't matter as methodology goes, 1200 games from only 12 starting positions usually give skewed results." I agree, I think that they should have chosen a wider sampler of openings.
I have played a lot of games with lc0 using these 12 positions and using other initial positions like Nunn or Noomen Sharp Gambits getting a completely different result, see
http://talkchess.com/forum3/viewtopic.p ... 40#p774276
Playing all these games starting from A0 12 positions I also noticed that the positions reached were of positional style, that lc0 understands better than classic alpha-beta engines and that similar games often occurred but without an exact repetition.
When you force a program to play an opening variant that it would not have played on its own you are forcing it to play outside its style and the performance goes down a lot.
So what is the real level of lc0?
35% obtained from Noomen Sharp Openings that I have forced it to play or the 40%-42% obtained from the other positions?
The rules of chess say that you have to start from the initial position and not from other artificially chosen ones.
From the 35% obtained from Noomen Sharp Openings against 65% of Stockfish you can conclude that Stockfish is better in that type of position, but this positions would never be played by lc0 starting from the initial position.
So I think that the real level of lc0 is reached starting from the initial position or opening variants that lc0 plays naturally by itself.