Hey, Carlos, would you mind uploading the setup files in mediafire for Leelazero v3 to be able to run in my arena.
(i can setup client version 3 in the network with your previous files)
LCZero update
Moderators: hgm, Rebel, chrisw
-
- Posts: 568
- Joined: Tue Dec 12, 2006 10:10 am
- Full name: Gary Linscott
Re: LCZero update
I agree the gap between random and SF is immense.lucasart wrote:Completely agree.hgm wrote:No one has a clue how much random play is. There just aren't enough intermediate opponents to bridge the gap between random movers and searching engines. The weak engines that seem to fit in there are all very buggy, and do not behave according to the Elo model: they have a fixed finite probability to lose against any opponent, no matter how weak, because they crash.
So random play could be 3000 Elo below Stockfish. Or 30,000 Elo. We just don't know.
The CCRL list in this region is a bit suspect. E.g. RAM is supposed to be a random mover. So how can it be at the level of NEG, ~300 Elo above Brutus RND? I am also pretty sure that NEG scores 99% against a random mover, much more than the Elo difference suggests. The rating of the random mover must be highly inflated because it gets free points from buggy engine higher in the list. You cannot make a sensible rating list with engines that hand out free points, irrespective of opponent performance.
I think the gap between random play and SF is immense. Much higher than what rating lists indicate.
I am looking at lczero.org now, and I can see that:
* game play is indistinguishable from random (to my untrained eye)
* yet, it is showing +3,100 elo vs. random play
By this metric, SF could easily be 30,000 elo.
However, the gameplay is definitely not random, if you look at match games. If you look at training games, there is actually randomness deliberately introduced to ensure variety, so that will look more random .
Check out some games from the recent promotion match: http://lczero.org/match/13
Or, for even more fun, a battle against the lichess SF at 3 ply, level 20: https://lichess.org/p7W6goyS (check out the beautiful queen windmill!).
-
- Posts: 291
- Joined: Wed May 08, 2013 6:49 am
Re: LCZero update
I want to know if LCZero in this mode still relies on GPU like in training mode.Nay Lin Tun wrote:Hey, Carlos, would you mind uploading the setup files in mediafire for Leelazero v3 to be able to run in my arena.
(i can setup client version 3 in the network with your previous files)
-
- Posts: 126
- Joined: Thu Jun 05, 2014 5:29 am
- Location: Malaysia
Re: LCZero update
Wow, that some human-alike moves right there!gladius wrote:Or, for even more fun, a battle against the lichess SF at 3 ply, level 20: https://lichess.org/p7W6goyS (check out the beautiful queen windmill!).
-
- Posts: 3232
- Joined: Mon May 31, 2010 1:29 pm
- Full name: lucasart
Re: LCZero update
Ok. Those games do look better. I'd say positionally reasonable, but tactics are the kryptonite of lczero.gladius wrote:I agree the gap between random and SF is immense.lucasart wrote:Completely agree.hgm wrote:No one has a clue how much random play is. There just aren't enough intermediate opponents to bridge the gap between random movers and searching engines. The weak engines that seem to fit in there are all very buggy, and do not behave according to the Elo model: they have a fixed finite probability to lose against any opponent, no matter how weak, because they crash.
So random play could be 3000 Elo below Stockfish. Or 30,000 Elo. We just don't know.
The CCRL list in this region is a bit suspect. E.g. RAM is supposed to be a random mover. So how can it be at the level of NEG, ~300 Elo above Brutus RND? I am also pretty sure that NEG scores 99% against a random mover, much more than the Elo difference suggests. The rating of the random mover must be highly inflated because it gets free points from buggy engine higher in the list. You cannot make a sensible rating list with engines that hand out free points, irrespective of opponent performance.
I think the gap between random play and SF is immense. Much higher than what rating lists indicate.
I am looking at lczero.org now, and I can see that:
* game play is indistinguishable from random (to my untrained eye)
* yet, it is showing +3,100 elo vs. random play
By this metric, SF could easily be 30,000 elo.
However, the gameplay is definitely not random, if you look at match games. If you look at training games, there is actually randomness deliberately introduced to ensure variety, so that will look more random .
Check out some games from the recent promotion match: http://lczero.org/match/13
Or, for even more fun, a battle against the lichess SF at 3 ply, level 20: https://lichess.org/p7W6goyS (check out the beautiful queen windmill!).
Theory and practice sometimes clash. And when that happens, theory loses. Every single time.
-
- Posts: 3232
- Joined: Mon May 31, 2010 1:29 pm
- Full name: lucasart
Re: LCZero update
At least one advantage of this approach is that you can create human-like weak levels, which is not really possible with alpha beta. Snapshots of the NN weights along the way would give you that.
Although you may end up with unhuman play in the opposite direction: reasonably good positional play but horrendous tactics. Basically too human like to be credible.
Although you may end up with unhuman play in the opposite direction: reasonably good positional play but horrendous tactics. Basically too human like to be credible.
Theory and practice sometimes clash. And when that happens, theory loses. Every single time.
-
- Posts: 10320
- Joined: Thu Mar 09, 2006 12:37 am
- Location: Tel-Aviv Israel
Re: LCZero update
lucasart wrote:Completely agree.hgm wrote:No one has a clue how much random play is. There just aren't enough intermediate opponents to bridge the gap between random movers and searching engines. The weak engines that seem to fit in there are all very buggy, and do not behave according to the Elo model: they have a fixed finite probability to lose against any opponent, no matter how weak, because they crash.
So random play could be 3000 Elo below Stockfish. Or 30,000 Elo. We just don't know.
The CCRL list in this region is a bit suspect. E.g. RAM is supposed to be a random mover. So how can it be at the level of NEG, ~300 Elo above Brutus RND? I am also pretty sure that NEG scores 99% against a random mover, much more than the Elo difference suggests. The rating of the random mover must be highly inflated because it gets free points from buggy engine higher in the list. You cannot make a sensible rating list with engines that hand out free points, irrespective of opponent performance.
I think the gap between random play and SF is immense. Much higher than what rating lists indicate.
I am looking at lczero.org now, and I can see that:
* game play is indistinguishable from random (to my untrained eye)
* yet, it is showing +3,100 elo vs. random play
By this metric, SF could easily be 30,000 elo.
Looking at the graph
http://162.217.248.187/
If we have rating=f(number of games) we have the following
f(24167)=0
f(33999)=436.43
f(72741)=988.52
f(122891)=1424.95
f(162953)=1555.89
f(213449)=1724.29
f(273967)=2039.64
f(286925)=2476.07
f(325551)=2764.14
f(384905)=2860.33
f(461652)=2825.47
f(483399)=2998.25
f(511445)=3100.76
I find the following surprising data:
1)LCZero needed 24167 to get to a level of random play
2)There was a regression from rating 2860.33 after 384905 games to rating of 2825.47 after 461652 games
3)The rating is calculated with 2 digits after the point(and I wonder what is the standard error of the rating and how people calculate it).
-
- Posts: 143
- Joined: Wed Jan 17, 2018 1:26 pm
Re: LCZero update
1) was caused by the fact that the first two networks ended up not being actually used for the reinforcement learning. They had some strange biases in winrate, and were also truncated after 75 moves, which as we found out at this point was too short. The pipeline was actually seeded by random games generated by Folkert Huizinga without using any neural network, where those issues were fixed.Uri Blass wrote:Looking at the graph
http://162.217.248.187/
If we have rating=f(number of games) we have the following
f(24167)=0
f(33999)=436.43
f(72741)=988.52
f(122891)=1424.95
f(162953)=1555.89
f(213449)=1724.29
f(273967)=2039.64
f(286925)=2476.07
f(325551)=2764.14
f(384905)=2860.33
f(461652)=2825.47
f(483399)=2998.25
f(511445)=3100.76
I find the following surprising data:
1)LCZero needed 24167 to get to a level of random play
2)There was a regression from rating 2860.33 after 384905 games to rating of 2825.47 after 461652 games
3)The rating is calculated with 2 digits after the point(and I wonder what is the standard error of the rating and how people calculate it).
2)This network lost its match against the previous net, but that was slightly before match support went active. It was uploaded anyway in the hope that it would help subsequent nets learn better. But for the immediate future, if a network cannot beat its predecessor in a distributed match, it will not be promoted anymore.
3)The standard error of the rating is much higher, you're right that keeping the decimals doesn't make much sense.
-
- Posts: 1142
- Joined: Thu Dec 28, 2017 4:06 pm
- Location: Argentina
Re: LCZero update
Here you go: client v3, latest network (gen 13) and latest lczero.exe (build 112)Nay Lin Tun wrote:Hey, Carlos, would you mind uploading the setup files in mediafire for Leelazero v3 to be able to run in my arena.
(i can setup client version 3 in the network with your previous files)
http://www.mediafire.com/file/ilm2ixxpl ... en%2013.7z
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls
-
- Posts: 708
- Joined: Mon Jan 16, 2012 6:34 am
Re: LCZero update
Thank you Carlos,
So, 1.I downloaded your files.
https://ibb.co/coLruc
2.I did test run my GPU, ok
https://ibb.co/hUD5LH
3. I try to install new engine in arena
https://ibb.co/dUc1Sx
And start arena, and Leele Zero is not running.
Am I missing something?
So, 1.I downloaded your files.
https://ibb.co/coLruc
2.I did test run my GPU, ok
https://ibb.co/hUD5LH
3. I try to install new engine in arena
https://ibb.co/dUc1Sx
And start arena, and Leele Zero is not running.
Am I missing something?