LCZero update

[Nay Lin Tun]

Hey, Carlos, would you mind uploading the setup files in mediafire for Leelazero v3 to be able to run in my arena.

(i can setup client version 3 in the network with your previous files)

[gladius]

No one has a clue how much random play is. There just aren't enough intermediate opponents to bridge the gap between random movers and searching engines. The weak engines that seem to fit in there are all very buggy, and do not behave according to the Elo model: they have a fixed finite probability to lose against any opponent, no matter how weak, because they crash.

So random play could be 3000 Elo below Stockfish. Or 30,000 Elo. We just don't know.

The CCRL list in this region is a bit suspect. E.g. RAM is supposed to be a random mover. So how can it be at the level of NEG, ~300 Elo above Brutus RND? I am also pretty sure that NEG scores 99% against a random mover, much more than the Elo difference suggests. The rating of the random mover must be highly inflated because it gets free points from buggy engine higher in the list. You cannot make a sensible rating list with engines that hand out free points, irrespective of opponent performance.

Completely agree.

I think the gap between random play and SF is immense. Much higher than what rating lists indicate.

I am looking at lczero.org now, and I can see that:
* game play is indistinguishable from random (to my untrained eye)
* yet, it is showing +3,100 elo vs. random play

By this metric, SF could easily be 30,000 elo.

I agree the gap between random and SF is immense.

However, the gameplay is definitely not random, if you look at match games. If you look at training games, there is actually randomness deliberately introduced to ensure variety, so that will look more random .

Check out some games from the recent promotion match: http://lczero.org/match/13

Or, for even more fun, a battle against the lichess SF at 3 ply, level 20: https://lichess.org/p7W6goyS (check out the beautiful queen windmill!).

[Jhoravi]

Hey, Carlos, would you mind uploading the setup files in mediafire for Leelazero v3 to be able to run in my arena.

(i can setup client version 3 in the network with your previous files)

I want to know if LCZero in this mode still relies on GPU like in training mode.

[nabildanial]

Or, for even more fun, a battle against the lichess SF at 3 ply, level 20: https://lichess.org/p7W6goyS (check out the beautiful queen windmill!).

Wow, that some human-alike moves right there!

[lucasart]

No one has a clue how much random play is. There just aren't enough intermediate opponents to bridge the gap between random movers and searching engines. The weak engines that seem to fit in there are all very buggy, and do not behave according to the Elo model: they have a fixed finite probability to lose against any opponent, no matter how weak, because they crash.

So random play could be 3000 Elo below Stockfish. Or 30,000 Elo. We just don't know.

The CCRL list in this region is a bit suspect. E.g. RAM is supposed to be a random mover. So how can it be at the level of NEG, ~300 Elo above Brutus RND? I am also pretty sure that NEG scores 99% against a random mover, much more than the Elo difference suggests. The rating of the random mover must be highly inflated because it gets free points from buggy engine higher in the list. You cannot make a sensible rating list with engines that hand out free points, irrespective of opponent performance.

Completely agree.

I think the gap between random play and SF is immense. Much higher than what rating lists indicate.

I am looking at lczero.org now, and I can see that:
* game play is indistinguishable from random (to my untrained eye)
* yet, it is showing +3,100 elo vs. random play

By this metric, SF could easily be 30,000 elo.

I agree the gap between random and SF is immense.

However, the gameplay is definitely not random, if you look at match games. If you look at training games, there is actually randomness deliberately introduced to ensure variety, so that will look more random .

Check out some games from the recent promotion match: http://lczero.org/match/13

Or, for even more fun, a battle against the lichess SF at 3 ply, level 20: https://lichess.org/p7W6goyS (check out the beautiful queen windmill!).

Ok. Those games do look better. I'd say positionally reasonable, but tactics are the kryptonite of lczero.

[lucasart]

At least one advantage of this approach is that you can create human-like weak levels, which is not really possible with alpha beta. Snapshots of the NN weights along the way would give you that.

Although you may end up with unhuman play in the opposite direction: reasonably good positional play but horrendous tactics. Basically too human like to be credible.

[Uri Blass]

No one has a clue how much random play is. There just aren't enough intermediate opponents to bridge the gap between random movers and searching engines. The weak engines that seem to fit in there are all very buggy, and do not behave according to the Elo model: they have a fixed finite probability to lose against any opponent, no matter how weak, because they crash.

So random play could be 3000 Elo below Stockfish. Or 30,000 Elo. We just don't know.

The CCRL list in this region is a bit suspect. E.g. RAM is supposed to be a random mover. So how can it be at the level of NEG, ~300 Elo above Brutus RND? I am also pretty sure that NEG scores 99% against a random mover, much more than the Elo difference suggests. The rating of the random mover must be highly inflated because it gets free points from buggy engine higher in the list. You cannot make a sensible rating list with engines that hand out free points, irrespective of opponent performance.

Completely agree.

I think the gap between random play and SF is immense. Much higher than what rating lists indicate.

I am looking at lczero.org now, and I can see that:
* game play is indistinguishable from random (to my untrained eye)
* yet, it is showing +3,100 elo vs. random play

By this metric, SF could easily be 30,000 elo.

Looking at the graph

http://162.217.248.187/



If we have rating=f(number of games) we have the following

f(24167)=0
f(33999)=436.43
f(72741)=988.52
f(122891)=1424.95
f(162953)=1555.89
f(213449)=1724.29
f(273967)=2039.64
f(286925)=2476.07
f(325551)=2764.14
f(384905)=2860.33
f(461652)=2825.47
f(483399)=2998.25
f(511445)=3100.76

I find the following surprising data:

1)LCZero needed 24167 to get to a level of random play

2)There was a regression from rating 2860.33 after 384905 games to rating of 2825.47 after 461652 games

3)The rating is calculated with 2 digits after the point(and I wonder what is the standard error of the rating and how people calculate it).

[jkiliani]

Looking at the graph

http://162.217.248.187/

If we have rating=f(number of games) we have the following

f(24167)=0
f(33999)=436.43
f(72741)=988.52
f(122891)=1424.95
f(162953)=1555.89
f(213449)=1724.29
f(273967)=2039.64
f(286925)=2476.07
f(325551)=2764.14
f(384905)=2860.33
f(461652)=2825.47
f(483399)=2998.25
f(511445)=3100.76

I find the following surprising data:

1)LCZero needed 24167 to get to a level of random play

2)There was a regression from rating 2860.33 after 384905 games to rating of 2825.47 after 461652 games

3)The rating is calculated with 2 digits after the point(and I wonder what is the standard error of the rating and how people calculate it).

1) was caused by the fact that the first two networks ended up not being actually used for the reinforcement learning. They had some strange biases in winrate, and were also truncated after 75 moves, which as we found out at this point was too short. The pipeline was actually seeded by random games generated by Folkert Huizinga without using any neural network, where those issues were fixed.

2)This network lost its match against the previous net, but that was slightly before match support went active. It was uploaded anyway in the hope that it would help subsequent nets learn better. But for the immediate future, if a network cannot beat its predecessor in a distributed match, it will not be promoted anymore.

3)The standard error of the rating is much higher, you're right that keeping the decimals doesn't make much sense.

[CMCanavessi]

Hey, Carlos, would you mind uploading the setup files in mediafire for Leelazero v3 to be able to run in my arena.

(i can setup client version 3 in the network with your previous files)

Here you go: client v3, latest network (gen 13) and latest lczero.exe (build 112)

http://www.mediafire.com/file/ilm2ixxpl ... en%2013.7z

[Nay Lin Tun]

Thank you Carlos,

So, 1.I downloaded your files.
https://ibb.co/coLruc

2.I did test run my GPU, ok
https://ibb.co/hUD5LH

3. I try to install new engine in arena

https://ibb.co/dUc1Sx

And start arena, and Leele Zero is not running.
Am I missing something?