George Tsavdaris wrote: ↑Thu Aug 16, 2018 10:10 pm
Finally they have found the issue, apparently the GPUs were severely got overheated(the case was too small and had just 1 fan for 2 GTX 1080 Ti, 500 Watt with 1 fan in a small case!) and affected in a big way nodes per second(there were even cases with 75% reduction of normal nodes per second), performance anyway, of the NNs.
Problem was even more serious for Leela as she had this aggressive pruning thing setting that made the situation worse. Both NNs were crippled by this and Leela even more because of this aggressive thing.
In the last 6 games that they have fixed this(not that elegantly since they had to reduce speed of GPUS to 70% so NNs get 70% of the true power but at least there are not any more the throttling issues) Leela has 5.5/6 after the fix! In the previous 19 games she had just 2 wins.
Too bad for TCEC that they haven't fixed this sooner.
Let me get this straight...they're tweaking the hardware to help Lc0 while Div 3 is running?
Yes, the video cards were over-heating resulting in weak play by Leela. After reducing their speed to 70% (so they don't overheat that much) Leela recovered it's normal level and won most matches greatly increasing it's qualifying chance. Maybe it's too late now, but at least the problem was found.
fersbery wrote: ↑Fri Aug 17, 2018 3:30 am
Yes, the video cards were over-heating resulting in weak play by Leela. After reducing their speed to 70% (so they don't overheat that much) Leela recovered it's normal level and won most matches greatly increasing it's qualifying chance. Maybe it's too late now, but at least the problem was found.
Then the fair solution would be to replay those games where Leela was adversely affected by the faulty TCEC conditions.
Not that I expect it to happen.
fersbery wrote: ↑Fri Aug 17, 2018 3:30 am
Yes, the video cards were over-heating resulting in weak play by Leela. After reducing their speed to 70% (so they don't overheat that much) Leela recovered it's normal level and won most matches greatly increasing it's qualifying chance. Maybe it's too late now, but at least the problem was found.
Then the fair solution would be to replay those games where Leela was adversely affected by the faulty TCEC conditions.
Not that I expect it to happen.
Imagine how people would react...
I think that it's ok as it is since this was an "experimental" season. There is no hurry for results, they will come alone once tournament organizers have more experience with GPU hardware and Leela project have better networks and bugs fixed.
fersbery wrote: ↑Fri Aug 17, 2018 5:55 am
I think that it's ok as it is since this was an "experimental" season. There is no hurry for results, they will come alone once tournament organizers have more experience with GPU hardware and Leela project have better networks and bugs fixed.
So you think system administrators of TCEC are inexperienced youngsters?
If this is the case really it is a serious problem for TCEC and for the participants.
I think even from an amateur PC builder can be expected to take care of the appropriate heating of its system.
In any case, we do not forget the real issues of Leele.
Every issues we can not imputed to the overheating, it is true.
George Tsavdaris wrote: ↑Thu Aug 16, 2018 10:10 pm
Finally they have found the issue, apparently the GPUs were severely got overheated(the case was too small and had just 1 fan for 2 GTX 1080 Ti, 500 Watt with 1 fan in a small case!) and affected in a big way nodes per second(there were even cases with 75% reduction of normal nodes per second), performance anyway, of the NNs.
Problem was even more serious for Leela as she had this aggressive pruning thing setting that made the situation worse. Both NNs were crippled by this and Leela even more because of this aggressive thing.
In the last 6 games that they have fixed this(not that elegantly since they had to reduce speed of GPUS to 70% so NNs get 70% of the true power but at least there are not any more the throttling issues) Leela has 5.5/6 after the fix! In the previous 19 games she had just 2 wins.
Too bad for TCEC that they haven't fixed this sooner.
Let me get this straight...they're tweaking the hardware to help Lc0 while Div 3 is running?
Well same happened with Nemorino and instead of 43 cores and a big hash setting if i recall, they tweaked the settings in the middle of the tournament to use 16 cores and lower hash in order not to crash.
This is allowed.
Funny people stick on hardware tweaking(that is allowed) and not that NN engines were crippled in the first place.
After his son's birth they've asked him:
"Is it a boy or girl?"
YES! He replied.....
Laskos wrote: ↑Thu Aug 16, 2018 10:28 am
Probably in these conditions, Lc0 is already close to SF8 on 2 cores. When it will achieve the level of SF8 on 4 cores, I could casually say "Lc0 is the level of SF8", like people are used to say about engines, meaning on home reasonable balanced CPU/GPU hardware configuration and blitz time controls.
You keep repeating that but it doesn't make it true. A price-wise balanced CPU/GPU configuration is 16 threads on 8 cores Ryzen vs. 1060 (plus cheap CPU).
Power (TDP) balanced one is 32 threads (16 cores) Ryzen (Threadripper) vs 1060 (plus lower power CPU).
The Nvidia 10x series is 2 year old, and the release of significantly better 11x series is delayed only by crypto-maniacs digging fairy tales. It will be out anyway soon, a question of few months, so my flawed "reasonability" still seems better than your trolling.
You are the one who is trolling with your 5+ years old CPU (4 generations old) comparing to 2 years old GPU (current state-of-the-art).
When you have no arguments (to my spot on comparison) then you yell trolling and rent about crypto-maniacs and make hypothetical comparisons with hardware that is not yet out and with uncertain prices and performance.
Again spitting gibberish with some 5% here and there, while missing the general picture? Nvidia 11 series is just 2-3 months from now, and as we know, they will come with some massive improvements. And by the way, my OC i7 4790 is at least on par with say stock i7 7700. All in all, you nitpick (with misplaced wording) on the general picture that on fairly balanced CPU/GPU home hardware, Lc0 with a good net is on par with SF8 on 1-2 threads in blitz. And probably soon will be on par with SF8 on full CPU, and one could casually say "Lc0 is SF8 level". Generally, behave, and don't use misplaced language.
Laskos wrote: ↑Fri Aug 17, 2018 11:46 am
Nvidia 11 series is just 2-3 months from now, and as we know, they will come with some massive improvements. And by the way, my OC i7 4790 is at least on par with say stock i7 7700. All in all, you nitpick (with misplaced wording) on the general picture that on fairly balanced CPU/GPU home hardware, Lc0 with a good net is on par with SF8 on 1-2 threads in blitz. And probably soon will be on par with SF8 on full CPU, and one could casually say "Lc0 is SF8 level". Generally, behave, and don't use misplaced language.
The stats I've seen suggest "only" +~15%
But I was amazed Lc0 drew with Houdini 6 on 8 cores in a mini-match I ran yesterday on my 1060 card. It's an interesting time for computer chess...
