The real elo of lczero is in its name

[Uri Blass]

Till now it is Scorpio - Leela +5 =1 -3
Surprisingly not a bad performance by Leela. It has a -79 ELO performance compared to Scorpio.
And it should have been +3 =3 -3, since Leela lost 2 completely drawn games because she didn't want to force an easy perpetual.


It reports something like 2000 to 3000 nodes per second on the 43 CPUs which is rather very bad as on a good GPU will get easy 1000.
So on a top GPU(Titan V or GTX 1080 Ti) would be much stronger. So perhaps to see its true strength we need very very good hardware so all our tests for ELO are biased.

You can watch the 20 game match here: http://tcec.chessdom.com/bonus/live.php

I do not understand

I remember that not a long time ago I read the following by Carlos in another thread:

"If any1's interested, I'm doing a 100-game match between Leela and Scorpio, 30 min + 10 sec TC, ponder on (for scorpio), no TBs. Opening book is TCEC Season 10 Superfinal.

It might end 100-0, but it's just for fun. "

I remember that Scorpio was leading 18-0 in that match when probably they stopped the match.


How it is possible that Leela does significantly better in 15+10 time control in TCEC?

Leela did not make a big progress in the last days and I read that people even talk about regression so it cannot be better Leela.

Edit:I read in the last message that Scorpio is not the best Scorpio and in this case it is clearly misleading because I guess best Scorpio of today is used in TCEC in 4th division.

[CMCanavessi]

Till now it is Scorpio - Leela +5 =1 -3
Surprisingly not a bad performance by Leela. It has a -79 ELO performance compared to Scorpio.
And it should have been +3 =3 -3, since Leela lost 2 completely drawn games because she didn't want to force an easy perpetual.


It reports something like 2000 to 3000 nodes per second on the 43 CPUs which is rather very bad as on a good GPU will get easy 1000.
So on a top GPU(Titan V or GTX 1080 Ti) would be much stronger. So perhaps to see its true strength we need very very good hardware so all our tests for ELO are biased.

You can watch the 20 game match here: http://tcec.chessdom.com/bonus/live.php

I do not understand

I remember that not a long time ago I read the following by Carlos in another thread:

"If any1's interested, I'm doing a 100-game match between Leela and Scorpio, 30 min + 10 sec TC, ponder on (for scorpio), no TBs. Opening book is TCEC Season 10 Superfinal.

It might end 100-0, but it's just for fun. "

I remember that Scorpio was leading 18-0 in that match when probably they stopped the match.


How it is possible that Leela does significantly better in 15+10 time control in TCEC?

Leela did not make a big progress in the last days and I read that people even talk about regression so it cannot be better Leela.

Edit:I read in the last message that Scorpio is not the best Scorpio and in this case it is clearly misleading because I guess best Scorpio of today is used in TCEC in 4th division.

A couple of differences:

I was using latest version of Scorpio, 2.8.2 while TCEC was using last season's version, 2.7.9
I was running Leela in my GPU which is quite crap, getting around 1kn/s, and TCEC was running it on 43 CPU cores (real cores, not HT), getting around 3-4kn/s
Scorpio was using ponder ON in my match, TCEC does not use ponder


Anyways, the thing is people here on this forum were saying 3 weeks ago that Leela would never get past the 2500 elo barrier. Today, it's going toe to toe (on some unfavorable hardware, arguably) with one of last year's TCEC Division 4 participants, which is ranked 2987 in 40/4 CCRL... go figure.

Also note that Leela, with not that good endgame understanding yet, lost at least 2 games that were dead draw, avoiding 3-fold and ending up losing. Should get better soon.

[CMCanavessi]

Till now it is Scorpio - Leela +5 =1 -3
Surprisingly not a bad performance by Leela. It has a -79 ELO performance compared to Scorpio.
And it should have been +3 =3 -3, since Leela lost 2 completely drawn games because she didn't want to force an easy perpetual.

It is already +13=3-4 after 20 games i.e. +160Elo for Scorpio (current weaker version of it anyway).
Funny how early statistics can be misleading.

Scorpio 2.7.9 is 2987 on CCRL 40/4, 4CPU
By your own math, it would put Leela around 2800 elo mark... not that bad for a project less than 1 month old, even if trolls disagree

[Guenther]

Till now it is Scorpio - Leela +5 =1 -3
Surprisingly not a bad performance by Leela. It has a -79 ELO performance compared to Scorpio.
And it should have been +3 =3 -3, since Leela lost 2 completely drawn games because she didn't want to force an easy perpetual.

It is already +13=3-4 after 20 games i.e. +160Elo for Scorpio (current weaker version of it anyway).
Funny how early statistics can be misleading. ;)

Scorpio 2.7.9 is 2987 on CCRL 40/4, 4CPU
By your own math, it would put Leela around 2800 elo mark... not that bad for a project less than 1 month old, even if trolls disagree :roll:

Carlos, the Scorpio playing in that match is a much weaker version with mcts.

[CMCanavessi]

Till now it is Scorpio - Leela +5 =1 -3
Surprisingly not a bad performance by Leela. It has a -79 ELO performance compared to Scorpio.
And it should have been +3 =3 -3, since Leela lost 2 completely drawn games because she didn't want to force an easy perpetual.

It is already +13=3-4 after 20 games i.e. +160Elo for Scorpio (current weaker version of it anyway).
Funny how early statistics can be misleading.

Scorpio 2.7.9 is 2987 on CCRL 40/4, 4CPU
By your own math, it would put Leela around 2800 elo mark... not that bad for a project less than 1 month old, even if trolls disagree

Carlos, the Scorpio playing in that match is a much weaker version with mcts.

KM himself said that it was exactly the same config as last season, except he changed the smp algorithm from ABDABA to YBW.

I tested Leela vs a weaker version with MCTS (as per Daniel's instructions) and Leela won 9-1 in a 10 game match, finishing +8 =2 -0

[duncan]

Also note that Leela, with not that good endgame understanding yet, lost at least 2 games that were dead draw, avoiding 3-fold and ending up losing. Should get better soon.

what do you have to do to give it a good endgame understanding.?
also in the future does it have the potential for example to see mates in 30 for a bishop knight mate faster than regular software (without tablebase)?

[CMCanavessi]

Also note that Leela, with not that good endgame understanding yet, lost at least 2 games that were dead draw, avoiding 3-fold and ending up losing. Should get better soon.

what do you have to do to give it a good endgame understanding.?
also in the future does it have the potential for example to see mates in 30 for a bishop knight mate faster than regular software (without tablebase)?

You have to do nothing, just give her some time. AphaZero trained with 44 million games. Leela right now is just past 5 million.

The other question, I don't know... again, time will tell.

[Milos]

You have to do nothing, just give her some time. AphaZero trained with 44 million games. Leela right now is just past 5 million.

A0 started severely saturating below 20 million games with larger network.
Why do you think LC0 will continue improving for much longer?

[duncan]

You have to do nothing, just give her some time. AphaZero trained with 44 million games. Leela right now is just past 5 million.

A0 started severely saturating below 20 million games with larger network.
Why do you think LC0 will continue improving for much longer?

when leela saturates should it be close to elo of AphaZero?

[elpapa]

when leela saturates should it be close to elo of AphaZero?

How strong would Alpha Zero be on consumer hardware, though?