Let me get this straight...they're tweaking the hardware to help Lc0 while Div 3 is running?George Tsavdaris wrote: ↑Thu Aug 16, 2018 10:10 pm Finally they have found the issue, apparently the GPUs were severely got overheated(the case was too small and had just 1 fan for 2 GTX 1080 Ti, 500 Watt with 1 fan in a small case!) and affected in a big way nodes per second(there were even cases with 75% reduction of normal nodes per second), performance anyway, of the NNs.
Problem was even more serious for Leela as she had this aggressive pruning thing setting that made the situation worse. Both NNs were crippled by this and Leela even more because of this aggressive thing.
In the last 6 games that they have fixed this(not that elegantly since they had to reduce speed of GPUS to 70% so NNs get 70% of the true power but at least there are not any more the throttling issues) Leela has 5.5/6 after the fix! In the previous 19 games she had just 2 wins.
Too bad for TCEC that they haven't fixed this sooner.
Strange Lc0 TCEC performance
Moderators: hgm, Rebel, chrisw
-
- Posts: 2129
- Joined: Thu May 29, 2008 10:43 am
Re: Strange Lc0 TCEC performance
-
- Posts: 8
- Joined: Mon Aug 13, 2018 6:08 am
- Full name: Daniel Uranga
Re: Strange Lc0 TCEC performance
Yes, the video cards were over-heating resulting in weak play by Leela. After reducing their speed to 70% (so they don't overheat that much) Leela recovered it's normal level and won most matches greatly increasing it's qualifying chance. Maybe it's too late now, but at least the problem was found.
-
- Posts: 2283
- Joined: Sat Jun 02, 2012 2:13 am
Re: Strange Lc0 TCEC performance
Then the fair solution would be to replay those games where Leela was adversely affected by the faulty TCEC conditions.fersbery wrote: ↑Fri Aug 17, 2018 3:30 am Yes, the video cards were over-heating resulting in weak play by Leela. After reducing their speed to 70% (so they don't overheat that much) Leela recovered it's normal level and won most matches greatly increasing it's qualifying chance. Maybe it's too late now, but at least the problem was found.
Not that I expect it to happen.
-
- Posts: 8
- Joined: Mon Aug 13, 2018 6:08 am
- Full name: Daniel Uranga
Re: Strange Lc0 TCEC performance
Imagine how people would react...carldaman wrote: ↑Fri Aug 17, 2018 5:30 amThen the fair solution would be to replay those games where Leela was adversely affected by the faulty TCEC conditions.fersbery wrote: ↑Fri Aug 17, 2018 3:30 am Yes, the video cards were over-heating resulting in weak play by Leela. After reducing their speed to 70% (so they don't overheat that much) Leela recovered it's normal level and won most matches greatly increasing it's qualifying chance. Maybe it's too late now, but at least the problem was found.
Not that I expect it to happen.
I think that it's ok as it is since this was an "experimental" season. There is no hurry for results, they will come alone once tournament organizers have more experience with GPU hardware and Leela project have better networks and bugs fixed.
-
- Posts: 3657
- Joined: Wed Nov 18, 2015 11:41 am
- Location: hungary
Re: Strange Lc0 TCEC performance
So you think system administrators of TCEC are inexperienced youngsters?
If this is the case really it is a serious problem for TCEC and for the participants.
I think even from an amateur PC builder can be expected to take care of the appropriate heating of its system.
In any case, we do not forget the real issues of Leele.
Every issues we can not imputed to the overheating, it is true.
-
- Posts: 1627
- Joined: Thu Mar 09, 2006 12:35 pm
Re: Strange Lc0 TCEC performance
Well same happened with Nemorino and instead of 43 cores and a big hash setting if i recall, they tweaked the settings in the middle of the tournament to use 16 cores and lower hash in order not to crash.kranium wrote: ↑Fri Aug 17, 2018 2:07 amLet me get this straight...they're tweaking the hardware to help Lc0 while Div 3 is running?George Tsavdaris wrote: ↑Thu Aug 16, 2018 10:10 pm Finally they have found the issue, apparently the GPUs were severely got overheated(the case was too small and had just 1 fan for 2 GTX 1080 Ti, 500 Watt with 1 fan in a small case!) and affected in a big way nodes per second(there were even cases with 75% reduction of normal nodes per second), performance anyway, of the NNs.
Problem was even more serious for Leela as she had this aggressive pruning thing setting that made the situation worse. Both NNs were crippled by this and Leela even more because of this aggressive thing.
In the last 6 games that they have fixed this(not that elegantly since they had to reduce speed of GPUS to 70% so NNs get 70% of the true power but at least there are not any more the throttling issues) Leela has 5.5/6 after the fix! In the previous 19 games she had just 2 wins.
Too bad for TCEC that they haven't fixed this sooner.
This is allowed.
Funny people stick on hardware tweaking(that is allowed) and not that NN engines were crippled in the first place.
After his son's birth they've asked him:
"Is it a boy or girl?"
YES! He replied.....
"Is it a boy or girl?"
YES! He replied.....
-
- Posts: 565
- Joined: Thu Nov 13, 2014 12:03 pm
Re: Strange Lc0 TCEC performance
think it's in the rules that tweaks can be made within season if a play-affecting issue is found.
-
- Posts: 1627
- Joined: Thu Mar 09, 2006 12:35 pm
Re: Strange Lc0 TCEC performance
No. Tweaks to the engines are not allowed.whereagles wrote: ↑Fri Aug 17, 2018 10:10 am think it's in the rules that tweaks can be made within season if a play-affecting issue is found.
Tweaks to the hardware for engines are. E.g lowering cores.
After his son's birth they've asked him:
"Is it a boy or girl?"
YES! He replied.....
"Is it a boy or girl?"
YES! He replied.....
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: Strange Lc0 TCEC performance
Again spitting gibberish with some 5% here and there, while missing the general picture? Nvidia 11 series is just 2-3 months from now, and as we know, they will come with some massive improvements. And by the way, my OC i7 4790 is at least on par with say stock i7 7700. All in all, you nitpick (with misplaced wording) on the general picture that on fairly balanced CPU/GPU home hardware, Lc0 with a good net is on par with SF8 on 1-2 threads in blitz. And probably soon will be on par with SF8 on full CPU, and one could casually say "Lc0 is SF8 level". Generally, behave, and don't use misplaced language.Milos wrote: ↑Thu Aug 16, 2018 5:46 pmYou are the one who is trolling with your 5+ years old CPU (4 generations old) comparing to 2 years old GPU (current state-of-the-art).Laskos wrote: ↑Thu Aug 16, 2018 2:54 pmThe Nvidia 10x series is 2 year old, and the release of significantly better 11x series is delayed only by crypto-maniacs digging fairy tales. It will be out anyway soon, a question of few months, so my flawed "reasonability" still seems better than your trolling.Milos wrote: ↑Thu Aug 16, 2018 2:03 pmYou keep repeating that but it doesn't make it true. A price-wise balanced CPU/GPU configuration is 16 threads on 8 cores Ryzen vs. 1060 (plus cheap CPU).Laskos wrote: ↑Thu Aug 16, 2018 10:28 am Probably in these conditions, Lc0 is already close to SF8 on 2 cores. When it will achieve the level of SF8 on 4 cores, I could casually say "Lc0 is the level of SF8", like people are used to say about engines, meaning on home reasonable balanced CPU/GPU hardware configuration and blitz time controls.
Power (TDP) balanced one is 32 threads (16 cores) Ryzen (Threadripper) vs 1060 (plus lower power CPU).
When you have no arguments (to my spot on comparison) then you yell trolling and rent about crypto-maniacs and make hypothetical comparisons with hardware that is not yet out and with uncertain prices and performance.
-
- Posts: 1797
- Joined: Thu Sep 18, 2008 10:24 pm
Re: Strange Lc0 TCEC performance
The stats I've seen suggest "only" +~15%Laskos wrote: ↑Fri Aug 17, 2018 11:46 am Nvidia 11 series is just 2-3 months from now, and as we know, they will come with some massive improvements. And by the way, my OC i7 4790 is at least on par with say stock i7 7700. All in all, you nitpick (with misplaced wording) on the general picture that on fairly balanced CPU/GPU home hardware, Lc0 with a good net is on par with SF8 on 1-2 threads in blitz. And probably soon will be on par with SF8 on full CPU, and one could casually say "Lc0 is SF8 level". Generally, behave, and don't use misplaced language.
But I was amazed Lc0 drew with Houdini 6 on 8 cores in a mini-match I ran yesterday on my 1060 card. It's an interesting time for computer chess...