Strange Lc0 TCEC performance

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
User avatar
kranium
Posts: 1824
Joined: Thu May 29, 2008 8:43 am

Re: Strange Lc0 TCEC performance

Post by kranium » Fri Aug 17, 2018 12:07 am

George Tsavdaris wrote:
Thu Aug 16, 2018 8:10 pm
Finally they have found the issue, apparently the GPUs were severely got overheated(the case was too small and had just 1 fan for 2 GTX 1080 Ti, 500 Watt with 1 fan in a small case!) and affected in a big way nodes per second(there were even cases with 75% reduction of normal nodes per second), performance anyway, of the NNs.
Problem was even more serious for Leela as she had this aggressive pruning thing setting that made the situation worse. Both NNs were crippled by this and Leela even more because of this aggressive thing.

In the last 6 games that they have fixed this(not that elegantly since they had to reduce speed of GPUS to 70% so NNs get 70% of the true power but at least there are not any more the throttling issues) Leela has 5.5/6 after the fix! In the previous 19 games she had just 2 wins.

Too bad for TCEC that they haven't fixed this sooner.
Let me get this straight...they're tweaking the hardware to help Lc0 while Div 3 is running? :shock:

fersbery
Posts: 8
Joined: Mon Aug 13, 2018 4:08 am
Full name: Daniel Uranga

Re: Strange Lc0 TCEC performance

Post by fersbery » Fri Aug 17, 2018 1:30 am

Yes, the video cards were over-heating resulting in weak play by Leela. After reducing their speed to 70% (so they don't overheat that much) Leela recovered it's normal level and won most matches greatly increasing it's qualifying chance. Maybe it's too late now, but at least the problem was found.

carldaman
Posts: 1687
Joined: Sat Jun 02, 2012 12:13 am

Re: Strange Lc0 TCEC performance

Post by carldaman » Fri Aug 17, 2018 3:30 am

fersbery wrote:
Fri Aug 17, 2018 1:30 am
Yes, the video cards were over-heating resulting in weak play by Leela. After reducing their speed to 70% (so they don't overheat that much) Leela recovered it's normal level and won most matches greatly increasing it's qualifying chance. Maybe it's too late now, but at least the problem was found.
Then the fair solution would be to replay those games where Leela was adversely affected by the faulty TCEC conditions.
Not that I expect it to happen.

fersbery
Posts: 8
Joined: Mon Aug 13, 2018 4:08 am
Full name: Daniel Uranga

Re: Strange Lc0 TCEC performance

Post by fersbery » Fri Aug 17, 2018 3:55 am

carldaman wrote:
Fri Aug 17, 2018 3:30 am
fersbery wrote:
Fri Aug 17, 2018 1:30 am
Yes, the video cards were over-heating resulting in weak play by Leela. After reducing their speed to 70% (so they don't overheat that much) Leela recovered it's normal level and won most matches greatly increasing it's qualifying chance. Maybe it's too late now, but at least the problem was found.
Then the fair solution would be to replay those games where Leela was adversely affected by the faulty TCEC conditions.
Not that I expect it to happen.
Imagine how people would react...
I think that it's ok as it is since this was an "experimental" season. There is no hurry for results, they will come alone once tournament organizers have more experience with GPU hardware and Leela project have better networks and bugs fixed.

corres
Posts: 1510
Joined: Wed Nov 18, 2015 10:41 am
Location: hungary

Re: Strange Lc0 TCEC performance

Post by corres » Fri Aug 17, 2018 6:45 am

fersbery wrote:
Fri Aug 17, 2018 3:55 am

I think that it's ok as it is since this was an "experimental" season. There is no hurry for results, they will come alone once tournament organizers have more experience with GPU hardware and Leela project have better networks and bugs fixed.
So you think system administrators of TCEC are inexperienced youngsters?
If this is the case really it is a serious problem for TCEC and for the participants.
I think even from an amateur PC builder can be expected to take care of the appropriate heating of its system.
In any case, we do not forget the real issues of Leele.
Every issues we can not imputed to the overheating, it is true.

User avatar
George Tsavdaris
Posts: 1559
Joined: Thu Mar 09, 2006 11:35 am

Re: Strange Lc0 TCEC performance

Post by George Tsavdaris » Fri Aug 17, 2018 7:04 am

kranium wrote:
Fri Aug 17, 2018 12:07 am
George Tsavdaris wrote:
Thu Aug 16, 2018 8:10 pm
Finally they have found the issue, apparently the GPUs were severely got overheated(the case was too small and had just 1 fan for 2 GTX 1080 Ti, 500 Watt with 1 fan in a small case!) and affected in a big way nodes per second(there were even cases with 75% reduction of normal nodes per second), performance anyway, of the NNs.
Problem was even more serious for Leela as she had this aggressive pruning thing setting that made the situation worse. Both NNs were crippled by this and Leela even more because of this aggressive thing.

In the last 6 games that they have fixed this(not that elegantly since they had to reduce speed of GPUS to 70% so NNs get 70% of the true power but at least there are not any more the throttling issues) Leela has 5.5/6 after the fix! In the previous 19 games she had just 2 wins.

Too bad for TCEC that they haven't fixed this sooner.
Let me get this straight...they're tweaking the hardware to help Lc0 while Div 3 is running? :shock:
Well same happened with Nemorino and instead of 43 cores and a big hash setting if i recall, they tweaked the settings in the middle of the tournament to use 16 cores and lower hash in order not to crash.
This is allowed.
Funny people stick on hardware tweaking(that is allowed) and not that NN engines were crippled in the first place.
After his son's birth they've asked him:
"Is it a boy or girl?"
YES! He replied.....

whereagles
Posts: 560
Joined: Thu Nov 13, 2014 11:03 am

Re: Strange Lc0 TCEC performance

Post by whereagles » Fri Aug 17, 2018 8:10 am

think it's in the rules that tweaks can be made within season if a play-affecting issue is found.

User avatar
George Tsavdaris
Posts: 1559
Joined: Thu Mar 09, 2006 11:35 am

Re: Strange Lc0 TCEC performance

Post by George Tsavdaris » Fri Aug 17, 2018 8:17 am

whereagles wrote:
Fri Aug 17, 2018 8:10 am
think it's in the rules that tweaks can be made within season if a play-affecting issue is found.
No. Tweaks to the engines are not allowed.
Tweaks to the hardware for engines are. E.g lowering cores.
After his son's birth they've asked him:
"Is it a boy or girl?"
YES! He replied.....

User avatar
Laskos
Posts: 9311
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Strange Lc0 TCEC performance

Post by Laskos » Fri Aug 17, 2018 9:46 am

Milos wrote:
Thu Aug 16, 2018 3:46 pm
Laskos wrote:
Thu Aug 16, 2018 12:54 pm
Milos wrote:
Thu Aug 16, 2018 12:03 pm
Laskos wrote:
Thu Aug 16, 2018 8:28 am
Probably in these conditions, Lc0 is already close to SF8 on 2 cores. When it will achieve the level of SF8 on 4 cores, I could casually say "Lc0 is the level of SF8", like people are used to say about engines, meaning on home reasonable balanced CPU/GPU hardware configuration and blitz time controls.
You keep repeating that but it doesn't make it true. A price-wise balanced CPU/GPU configuration is 16 threads on 8 cores Ryzen vs. 1060 (plus cheap CPU).
Power (TDP) balanced one is 32 threads (16 cores) Ryzen (Threadripper) vs 1060 (plus lower power CPU).
The Nvidia 10x series is 2 year old, and the release of significantly better 11x series is delayed only by crypto-maniacs digging fairy tales. It will be out anyway soon, a question of few months, so my flawed "reasonability" still seems better than your trolling.
You are the one who is trolling with your 5+ years old CPU (4 generations old) comparing to 2 years old GPU (current state-of-the-art).
When you have no arguments (to my spot on comparison) then you yell trolling and rent about crypto-maniacs and make hypothetical comparisons with hardware that is not yet out and with uncertain prices and performance.
Again spitting gibberish with some 5% here and there, while missing the general picture? Nvidia 11 series is just 2-3 months from now, and as we know, they will come with some massive improvements. And by the way, my OC i7 4790 is at least on par with say stock i7 7700. All in all, you nitpick (with misplaced wording) on the general picture that on fairly balanced CPU/GPU home hardware, Lc0 with a good net is on par with SF8 on 1-2 threads in blitz. And probably soon will be on par with SF8 on full CPU, and one could casually say "Lc0 is SF8 level". Generally, behave, and don't use misplaced language.

Werewolf
Posts: 1187
Joined: Thu Sep 18, 2008 8:24 pm

Re: Strange Lc0 TCEC performance

Post by Werewolf » Fri Aug 17, 2018 10:42 am

Laskos wrote:
Fri Aug 17, 2018 9:46 am
Nvidia 11 series is just 2-3 months from now, and as we know, they will come with some massive improvements. And by the way, my OC i7 4790 is at least on par with say stock i7 7700. All in all, you nitpick (with misplaced wording) on the general picture that on fairly balanced CPU/GPU home hardware, Lc0 with a good net is on par with SF8 on 1-2 threads in blitz. And probably soon will be on par with SF8 on full CPU, and one could casually say "Lc0 is SF8 level". Generally, behave, and don't use misplaced language.
The stats I've seen suggest "only" +~15%

But I was amazed Lc0 drew with Houdini 6 on 8 cores in a mini-match I ran yesterday on my 1060 card. It's an interesting time for computer chess...

Post Reply