Strange Lc0 TCEC performance

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
corres
Posts: 2402
Joined: Wed Nov 18, 2015 10:41 am
Location: hungary

Re: Strange Lc0 TCEC performance

Post by corres » Wed Aug 15, 2018 4:31 pm

chrisw wrote:
Wed Aug 15, 2018 10:58 am
Poor score indicates something is broken. Which is always likely with entered late changes.

High draw rate indicates higher than "normal" width to depth ratio. In general depth finds interesting lines and possible wins. Width defends against overlooking stuff.

More width and less depth = safe, defensive play, but dull. High draw rate. Simples.

Could be broken somewhere, could be late changes conspired to alter the search profile, could be that the scaling to more nodes tends to reflect in width rather than depth. Could be random, but the operating assumption has to be that there is a problem.
I do not think LC0 developers made serious changes what drastically lessens the chess power of LC0.
The main evidence for it the difference between LC0 and DeusX does not change a lot in the games of TCEC 13 Division 3.
As I mentioned yet the weakness is in connection with NN namely the size of NN and the filling of NN.
The knowledge what is absent in NN the MCTS can not substitute fully.
The current state of Leela and DeusX is only good for STC games - that is all.

frankp
Posts: 216
Joined: Sun Mar 12, 2006 2:11 pm

Re: Strange Lc0 TCEC performance

Post by frankp » Thu Aug 16, 2018 7:06 am

There are some strange things going on with leela at TCEC.
The event has exposed 'bugs' and optimisations to be addressed, but that may not be the whole explanation.
Wonder if there are also hardware problems.
For example, compare the performance in TCEC with leela's performance in the test chess.com event - where it scored 4/5 against stronger opponents, albeit with different hardware, and played like leela. (or perhaps it is all a time control/nodes per second/depth-width effect)

User avatar
Laskos
Posts: 9988
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Strange Lc0 TCEC performance

Post by Laskos » Thu Aug 16, 2018 8:28 am

frankp wrote:
Thu Aug 16, 2018 7:06 am
There are some strange things going on with leela at TCEC.
The event has exposed 'bugs' and optimisations to be addressed, but that may not be the whole explanation.
Wonder if there are also hardware problems.
For example, compare the performance in TCEC with leela's performance in the test chess.com event - where it scored 4/5 against stronger opponents, albeit with different hardware, and played like leela. (or perhaps it is all a time control/nodes per second/depth-width effect)
What time control and hardware at chess.com event? The question to me is is whether Lc0 scales fine with time * hardware at lots of them. As for blitz games, the progress lately is remarkable. The nets in the region 10770-10780 are very strong. I got at 2' + 2'' on GTX 1060 against SF8 on 1 core the following:

Score of lc0_v16 10774 vs SF 8: +20 -10 =50 [0.563]
Elo difference: 43.66 +/- 46.78
80 of 80 games finished.

Probably in these conditions, Lc0 is already close to SF8 on 2 cores. When it will achieve the level of SF8 on 4 cores, I could casually say "Lc0 is the level of SF8", like people are used to say about engines, meaning on home reasonable balanced CPU/GPU hardware configuration and blitz time controls.

Uri Blass
Posts: 8730
Joined: Wed Mar 08, 2006 11:37 pm
Location: Tel-Aviv Israel

Re: Strange Lc0 TCEC performance

Post by Uri Blass » Thu Aug 16, 2018 8:32 am

corres wrote:
Wed Aug 15, 2018 4:31 pm
chrisw wrote:
Wed Aug 15, 2018 10:58 am
Poor score indicates something is broken. Which is always likely with entered late changes.

High draw rate indicates higher than "normal" width to depth ratio. In general depth finds interesting lines and possible wins. Width defends against overlooking stuff.

More width and less depth = safe, defensive play, but dull. High draw rate. Simples.

Could be broken somewhere, could be late changes conspired to alter the search profile, could be that the scaling to more nodes tends to reflect in width rather than depth. Could be random, but the operating assumption has to be that there is a problem.
I do not think LC0 developers made serious changes what drastically lessens the chess power of LC0.
The main evidence for it the difference between LC0 and DeusX does not change a lot in the games of TCEC 13 Division 3.
As I mentioned yet the weakness is in connection with NN namely the size of NN and the filling of NN.
The knowledge what is absent in NN the MCTS can not substitute fully.
The current state of Leela and DeusX is only good for STC games - that is all.
I do not see how the fact that the difference between LC0 and DeusX did not change by a lot is an evidence that LC0 developers did not do serious changes that reduce the power of LC0

I expected LC0 to perform clearly better than division 3 based on what I read about the new net so
I guess that they increased the power of LC0 by at least 100 elo by a new net but also reduced the power of it by at least 100 elo by other changes.

frankp
Posts: 216
Joined: Sun Mar 12, 2006 2:11 pm

Re: Strange Lc0 TCEC performance

Post by frankp » Thu Aug 16, 2018 8:58 am

Laskos wrote:
Thu Aug 16, 2018 8:28 am
frankp wrote:
Thu Aug 16, 2018 7:06 am
There are some strange things going on with leela at TCEC.
The event has exposed 'bugs' and optimisations to be addressed, but that may not be the whole explanation.
Wonder if there are also hardware problems.
For example, compare the performance in TCEC with leela's performance in the test chess.com event - where it scored 4/5 against stronger opponents, albeit with different hardware, and played like leela. (or perhaps it is all a time control/nodes per second/depth-width effect)
What time control and hardware at chess.com event? The question to me is is whether Lc0 scales fine with time * hardware at lots of them. As for blitz games, the progress lately is remarkable. The nets in the region 10770-10780 are very strong. I got at 2' + 2'' on GTX 1060 against SF8 on 1 core the following:

Score of lc0_v16 10774 vs SF 8: +20 -10 =50 [0.563]
Elo difference: 43.66 +/- 46.78
80 of 80 games finished.

Probably in these conditions, Lc0 is already close to SF8 on 2 cores. When it will achieve the level of SF8 on 4 cores, I could casually say "Lc0 is the level of SF8", like people are used to say about engines, meaning on home reasonable balanced CPU/GPU hardware configuration and blitz time controls.
https://www.chess.com/news/view/announc ... ampionship
GPU to be announced. I thought I heard 4 x v Tesla v100, but this may be wishful thinking. On the test leela was getting in the region of 70k n/s at the time I was watching.

corres
Posts: 2402
Joined: Wed Nov 18, 2015 10:41 am
Location: hungary

Re: Strange Lc0 TCEC performance

Post by corres » Thu Aug 16, 2018 9:11 am

Uri Blass wrote:
Thu Aug 16, 2018 8:32 am
corres wrote:
Wed Aug 15, 2018 4:31 pm
...
The current state of Leela and DeusX is only good for STC games - that is all.
I do not see how the fact that the difference between LC0 and DeusX did not change by a lot is an evidence that LC0 developers did not do serious changes that reduce the power of LC0

I expected LC0 to perform clearly better than division 3 based on what I read about the new net so
I guess that they increased the power of LC0 by at least 100 elo by a new net but also reduced the power of it by at least 100 elo by other changes.
The results what you see about new net is derived from more weaker (~very Short TC games on TCEC) systems than TCEC uses.

Milos
Posts: 3432
Joined: Wed Nov 25, 2009 12:47 am

Re: Strange Lc0 TCEC performance

Post by Milos » Thu Aug 16, 2018 12:03 pm

Laskos wrote:
Thu Aug 16, 2018 8:28 am
Probably in these conditions, Lc0 is already close to SF8 on 2 cores. When it will achieve the level of SF8 on 4 cores, I could casually say "Lc0 is the level of SF8", like people are used to say about engines, meaning on home reasonable balanced CPU/GPU hardware configuration and blitz time controls.
You keep repeating that but it doesn't make it true. A price-wise balanced CPU/GPU configuration is 16 threads on 8 cores Ryzen vs. 1060 (plus cheap CPU).
Power (TDP) balanced one is 32 threads (16 cores) Ryzen (Threadripper) vs 1060 (plus lower power CPU).

corres
Posts: 2402
Joined: Wed Nov 18, 2015 10:41 am
Location: hungary

Re: Strange Lc0 TCEC performance

Post by corres » Thu Aug 16, 2018 12:37 pm

I think at first we would know how scale LC0 on Nvidia GPUs.
We would make a lot of tests with the same net and 1xGTX 1060, 2xGTX 1 60, 1xGTX 1080 T1, 2xGTX 1080 Ti, etc. to take the picture of Leela scaling.
Basing this knowledge and knowledge of AB engine scaling we can decide what a GPU is needed by Leela to gain such a boost what AB engines get on TCEC hardware.

chrisw
Posts: 2923
Joined: Tue Apr 03, 2012 2:28 pm

Re: Strange Lc0 TCEC performance

Post by chrisw » Thu Aug 16, 2018 12:51 pm

Is there anybody who really actually knows with sufficient testing and good data how the LC0 MCTS algo scales up and beyond TCEC playing conditions? I don't think so.

jp
Posts: 1254
Joined: Mon Apr 23, 2018 5:54 am

Re: Strange Lc0 TCEC performance

Post by jp » Thu Aug 16, 2018 12:51 pm

Milos wrote:
Thu Aug 16, 2018 12:03 pm
A price-wise balanced CPU/GPU configuration is 16 threads on 8 cores Ryzen vs. 1060 (plus cheap CPU).
Power (TDP) balanced one is 32 threads (16 cores) Ryzen (Threadripper) vs 1060 (plus lower power CPU).
What sort of nps ratio Lc0:SF do these two balanced configs give?
(I know it depends on many things, but just something typical e.g. for larger NN, no TB hits, etc.)

Post Reply