LCZero update

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Locked
Nay Lin Tun
Posts: 346
Joined: Mon Jan 16, 2012 5:34 am

Re: LCZero update

Post by Nay Lin Tun » Wed Mar 21, 2018 1:06 am

Hey, Carlos, would you mind uploading the setup files in mediafire for Leelazero v3 to be able to run in my arena.

(i can setup client version 3 in the network with your previous files)

gladius
Posts: 535
Joined: Tue Dec 12, 2006 9:10 am

Re: LCZero update

Post by gladius » Wed Mar 21, 2018 1:27 am

lucasart wrote:
hgm wrote:No one has a clue how much random play is. There just aren't enough intermediate opponents to bridge the gap between random movers and searching engines. The weak engines that seem to fit in there are all very buggy, and do not behave according to the Elo model: they have a fixed finite probability to lose against any opponent, no matter how weak, because they crash.

So random play could be 3000 Elo below Stockfish. Or 30,000 Elo. We just don't know.

The CCRL list in this region is a bit suspect. E.g. RAM is supposed to be a random mover. So how can it be at the level of NEG, ~300 Elo above Brutus RND? I am also pretty sure that NEG scores 99% against a random mover, much more than the Elo difference suggests. The rating of the random mover must be highly inflated because it gets free points from buggy engine higher in the list. You cannot make a sensible rating list with engines that hand out free points, irrespective of opponent performance.
Completely agree.

I think the gap between random play and SF is immense. Much higher than what rating lists indicate.

I am looking at lczero.org now, and I can see that:
* game play is indistinguishable from random (to my untrained eye)
* yet, it is showing +3,100 elo vs. random play

By this metric, SF could easily be 30,000 elo.
I agree the gap between random and SF is immense.

However, the gameplay is definitely not random, if you look at match games. If you look at training games, there is actually randomness deliberately introduced to ensure variety, so that will look more random :).

Check out some games from the recent promotion match: http://lczero.org/match/13

Or, for even more fun, a battle against the lichess SF at 3 ply, level 20: https://lichess.org/p7W6goyS (check out the beautiful queen windmill!).

Jhoravi
Posts: 243
Joined: Wed May 08, 2013 4:49 am

Re: LCZero update

Post by Jhoravi » Wed Mar 21, 2018 3:53 am

Nay Lin Tun wrote:Hey, Carlos, would you mind uploading the setup files in mediafire for Leelazero v3 to be able to run in my arena.

(i can setup client version 3 in the network with your previous files)
I want to know if LCZero in this mode still relies on GPU like in training mode.

nabildanial
Posts: 82
Joined: Thu Jun 05, 2014 3:29 am
Location: Malaysia

Re: LCZero update

Post by nabildanial » Wed Mar 21, 2018 4:06 am

gladius wrote:Or, for even more fun, a battle against the lichess SF at 3 ply, level 20: https://lichess.org/p7W6goyS (check out the beautiful queen windmill!).
Wow, that some human-alike moves right there!

User avatar
lucasart
Posts: 2991
Joined: Mon May 31, 2010 11:29 am
Full name: lucasart
Contact:

Re: LCZero update

Post by lucasart » Wed Mar 21, 2018 4:19 am

gladius wrote:
lucasart wrote:
hgm wrote:No one has a clue how much random play is. There just aren't enough intermediate opponents to bridge the gap between random movers and searching engines. The weak engines that seem to fit in there are all very buggy, and do not behave according to the Elo model: they have a fixed finite probability to lose against any opponent, no matter how weak, because they crash.

So random play could be 3000 Elo below Stockfish. Or 30,000 Elo. We just don't know.

The CCRL list in this region is a bit suspect. E.g. RAM is supposed to be a random mover. So how can it be at the level of NEG, ~300 Elo above Brutus RND? I am also pretty sure that NEG scores 99% against a random mover, much more than the Elo difference suggests. The rating of the random mover must be highly inflated because it gets free points from buggy engine higher in the list. You cannot make a sensible rating list with engines that hand out free points, irrespective of opponent performance.
Completely agree.

I think the gap between random play and SF is immense. Much higher than what rating lists indicate.

I am looking at lczero.org now, and I can see that:
* game play is indistinguishable from random (to my untrained eye)
* yet, it is showing +3,100 elo vs. random play

By this metric, SF could easily be 30,000 elo.
I agree the gap between random and SF is immense.

However, the gameplay is definitely not random, if you look at match games. If you look at training games, there is actually randomness deliberately introduced to ensure variety, so that will look more random :).

Check out some games from the recent promotion match: http://lczero.org/match/13

Or, for even more fun, a battle against the lichess SF at 3 ply, level 20: https://lichess.org/p7W6goyS (check out the beautiful queen windmill!).
Ok. Those games do look better. I'd say positionally reasonable, but tactics are the kryptonite of lczero.
Theory and practice sometimes clash. And when that happens, theory loses. Every single time.

User avatar
lucasart
Posts: 2991
Joined: Mon May 31, 2010 11:29 am
Full name: lucasart
Contact:

Re: LCZero update

Post by lucasart » Wed Mar 21, 2018 4:38 am

At least one advantage of this approach is that you can create human-like weak levels, which is not really possible with alpha beta. Snapshots of the NN weights along the way would give you that.

Although you may end up with unhuman play in the opposite direction: reasonably good positional play but horrendous tactics. Basically too human like to be credible.
Theory and practice sometimes clash. And when that happens, theory loses. Every single time.

Uri Blass
Posts: 8221
Joined: Wed Mar 08, 2006 11:37 pm
Location: Tel-Aviv Israel

Re: LCZero update

Post by Uri Blass » Wed Mar 21, 2018 5:10 am

lucasart wrote:
hgm wrote:No one has a clue how much random play is. There just aren't enough intermediate opponents to bridge the gap between random movers and searching engines. The weak engines that seem to fit in there are all very buggy, and do not behave according to the Elo model: they have a fixed finite probability to lose against any opponent, no matter how weak, because they crash.

So random play could be 3000 Elo below Stockfish. Or 30,000 Elo. We just don't know.

The CCRL list in this region is a bit suspect. E.g. RAM is supposed to be a random mover. So how can it be at the level of NEG, ~300 Elo above Brutus RND? I am also pretty sure that NEG scores 99% against a random mover, much more than the Elo difference suggests. The rating of the random mover must be highly inflated because it gets free points from buggy engine higher in the list. You cannot make a sensible rating list with engines that hand out free points, irrespective of opponent performance.
Completely agree.

I think the gap between random play and SF is immense. Much higher than what rating lists indicate.

I am looking at lczero.org now, and I can see that:
* game play is indistinguishable from random (to my untrained eye)
* yet, it is showing +3,100 elo vs. random play

By this metric, SF could easily be 30,000 elo.

Looking at the graph

http://162.217.248.187/



If we have rating=f(number of games) we have the following

f(24167)=0
f(33999)=436.43
f(72741)=988.52
f(122891)=1424.95
f(162953)=1555.89
f(213449)=1724.29
f(273967)=2039.64
f(286925)=2476.07
f(325551)=2764.14
f(384905)=2860.33
f(461652)=2825.47
f(483399)=2998.25
f(511445)=3100.76

I find the following surprising data:

1)LCZero needed 24167 to get to a level of random play

2)There was a regression from rating 2860.33 after 384905 games to rating of 2825.47 after 461652 games

3)The rating is calculated with 2 digits after the point(and I wonder what is the standard error of the rating and how people calculate it).

jkiliani
Posts: 138
Joined: Wed Jan 17, 2018 12:26 pm

Re: LCZero update

Post by jkiliani » Wed Mar 21, 2018 5:35 am

Uri Blass wrote:Looking at the graph

http://162.217.248.187/

If we have rating=f(number of games) we have the following

f(24167)=0
f(33999)=436.43
f(72741)=988.52
f(122891)=1424.95
f(162953)=1555.89
f(213449)=1724.29
f(273967)=2039.64
f(286925)=2476.07
f(325551)=2764.14
f(384905)=2860.33
f(461652)=2825.47
f(483399)=2998.25
f(511445)=3100.76

I find the following surprising data:

1)LCZero needed 24167 to get to a level of random play

2)There was a regression from rating 2860.33 after 384905 games to rating of 2825.47 after 461652 games

3)The rating is calculated with 2 digits after the point(and I wonder what is the standard error of the rating and how people calculate it).
1) was caused by the fact that the first two networks ended up not being actually used for the reinforcement learning. They had some strange biases in winrate, and were also truncated after 75 moves, which as we found out at this point was too short. The pipeline was actually seeded by random games generated by Folkert Huizinga without using any neural network, where those issues were fixed.

2)This network lost its match against the previous net, but that was slightly before match support went active. It was uploaded anyway in the hope that it would help subsequent nets learn better. But for the immediate future, if a network cannot beat its predecessor in a distributed match, it will not be promoted anymore.

3)The standard error of the rating is much higher, you're right that keeping the decimals doesn't make much sense.

User avatar
CMCanavessi
Posts: 730
Joined: Thu Dec 28, 2017 3:06 pm
Location: Argentina

Re: LCZero update

Post by CMCanavessi » Wed Mar 21, 2018 11:09 am

Nay Lin Tun wrote:Hey, Carlos, would you mind uploading the setup files in mediafire for Leelazero v3 to be able to run in my arena.

(i can setup client version 3 in the network with your previous files)
Here you go: client v3, latest network (gen 13) and latest lczero.exe (build 112)

http://www.mediafire.com/file/ilm2ixxpl ... en%2013.7z
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls

Nay Lin Tun
Posts: 346
Joined: Mon Jan 16, 2012 5:34 am

Re: LCZero update

Post by Nay Lin Tun » Wed Mar 21, 2018 11:41 am

Thank you Carlos,

So, 1.I downloaded your files.
https://ibb.co/coLruc

2.I did test run my GPU, ok
https://ibb.co/hUD5LH

3. I try to install new engine in arena

https://ibb.co/dUc1Sx

And start arena, and Leele Zero is not running.
Am I missing something?

Locked