Page 2 of 18
Re: how will Leela fare at the WCCC?
Posted: Thu Jul 12, 2018 9:17 pm
by Laskos
JJJ wrote: ↑Thu Jul 12, 2018 12:24 pm
I guess they rent a machine ? Because otherwise it cost 100K.
Also, I don't see Leela favorite at all even with that kind of support, but the game should be very interesting.
It will be very interesting, if they use a good lc0 engine and a good net. As we saw, Jonny on 2400 distributed cores and GridGinkgo on 224 are no better than Komodo on 48 Intel cores (several CPUs). If we take Komodo 12+ on 48 cores as the bench to be among the best, we can extrapolate and foresee a bit of how it will look like.
Komodo 12 on CCRL 40/4' is about 3450 Elo. Leela (lc0) on my GTX 1060 GPU with one of the latest nets in the main branch is at least 3300 Elo CCRL 40/4'.
Hardware:
Leela on 8 x Nvidia Tesla V100 gets maybe (depending on parallelization) about 20-30 the effective speed-up compared to my GPU.
Komodo on 48 Intel cores gets probably similarly 20 or so effective speed-up compared to one i7 core.
So, hardware-wise, we can compare simply Leela on my GPU to Komodo on one i7 core, as I did getting 3300+ CCRL 40/4' rating for Leela (lc0).
Time control:
Effective time control on i7 core for 4' CCRL is 2' (CCRL has as a bench a 2 times slower core).
Time control in WCCC is 120', or 60 times longer (6 doublings). Each doubling adds at this long time control about 30 Elo points to Leela compared to Komodo (Leela scales better), so the rating of Leela at 120' LTC is about an effective 3480, when compared to very comparable 3450 of Komodo.
All, in all, Leela in the best conditions, would be on par with the best there (probably Komodo and Jonny, but maybe Ginkgo and Shredder can come close too).
But this is from regular, solid, short openings I used. Leela has a property to be very weak on tactical puzzles and misses often in endgames easy wins or draws. While the endgames are addressed in this 3300 rating I get, my openings do not reflect the fact that in WCCC one can use own custom opening book. This can be huge in the case of Leela. If the people there will prepare anti-Leela tactically very sharp opening books, Leela can underperform by as much as 200 Elo points. I just performed a test where Leela (lc0) with a good later mainbranch net got slaughtered by Komodo 9.2 from tactically very sharp opening positions collected from ECM879 suite. While from quiet regular positions, they are pretty much on par in these condition (40/2'). The difference in performance was 200+ Elo points using these two different opening suites.
Re: how will Leela fare at the WCCC?
Posted: Thu Jul 12, 2018 10:12 pm
by JJJ
Laskos wrote: ↑Thu Jul 12, 2018 9:17 pm
JJJ wrote: ↑Thu Jul 12, 2018 12:24 pm
I guess they rent a machine ? Because otherwise it cost 100K.
Also, I don't see Leela favorite at all even with that kind of support, but the game should be very interesting.
It will be very interesting, if they use a good lc0 engine and a good net. As we saw, Jonny on 2400 distributed cores and GridGinkgo on 224 are no better than Komodo on 48 Intel cores (several CPUs). If we take Komodo 12+ on 48 cores as the bench to be among the best, we can extrapolate and foresee a bit of how it will look like.
Komodo 12 on CCRL 40/4' is about 3450 Elo. Leela (lc0) on my GTX 1060 GPU with one of the latest nets in the main branch is at least 3300 Elo CCRL 40/4'.
Hardware:
Leela on 8 x Nvidia Tesla V100 gets maybe (depending on parallelization) about 20-30 the effective speed-up compared to my GPU.
Komodo on 48 Intel cores gets probably similarly 20 or so effective speed-up compared to one i7 core.
So, hardware-wise, we can compare simply Leela on my GPU to Komodo on one i7 core, as I did getting 3300+ CCRL 40/4' rating for Leela (lc0).
Time control:
Effective time control on i7 core for 4' CCRL is 2' (CCRL has as a bench a 2 times slower core).
Time control in WCCC is 120', or 60 times longer (6 doublings). Each doubling adds at this long time control about 30 Elo points to Leela compared to Komodo (Leela scales better), so the rating of Leela at 120' LTC is about an effective 3480, when compared to very comparable 3450 of Komodo.
All, in all, Leela in the best conditions, would be on par with the best there (probably Komodo and Jonny, but maybe Ginkgo and Shredder can come close too).
But this is from regular, solid, short openings I used. Leela has a property to be very weak on tactical puzzles and misses often in endgames easy wins or draws. While the endgames are addressed in this 3300 rating I get, my openings do not reflect the fact that in WCCC one can use own custom opening book. This can be huge in the case of Leela. If the people there will prepare anti-Leela tactically very sharp opening books, Leela can underperform by as much as 200 Elo points. I just performed a test where Leela (lc0) with a good later mainbranch net got slaughtered by Komodo 9.2 from tactically very sharp opening positions collected from ECM879 suite. While from quiet regular positions, they are pretty much on par in these condition (40/2'). The difference in performance was 200+ Elo points using these two different opening suites.
very nice analyzis Kai. Thanks you. Well, let's see if Leela will failt at tactics or play brilliantly.
Re: how will Leela fare at the WCCC?
Posted: Fri Jul 13, 2018 12:34 am
by FWCC
I think she will end up in Top 3. She will be using 8×V100s which is insane she should do well.
Re: how will Leela fare at the WCCC?
Posted: Fri Jul 13, 2018 2:55 am
by Milos
Laskos wrote: ↑Thu Jul 12, 2018 9:17 pm
Leela on 8 x Nvidia Tesla V100 gets maybe (depending on parallelization) about 20-30 the effective speed-up compared to my GPU.
Seriously???
I understand that there are quite a few ppl without a clue about how multiplexing on CUDA works, but you can at least inform yourself.
You know what is a speed up from 1 to 2 V100? Less than 30%. You think going from 2 to 4 would bring even 30%???
8xV100 is 1.25^3x3=6x1060. And you are dreaming of 20-30x, gee...
Diminishing returns don't magically disappear just because you use NN and MCTS.
Ask Mark about it on MCTS Komodo. And I'm fairly convinced Komodo MCTS is scaling far better than LC0.
Re: how will Leela fare at the WCCC?
Posted: Fri Jul 13, 2018 3:41 am
by CMCanavessi
Current tests by Cyril (the guy who will be in charge of the games and the hardware) show around 100k NPS at the start of the search, but then it starts to drop to about 50-30k. Devs still investigating why and if it can be corrected, if it's indeed something wrong.
Re: how will Leela fare at the WCCC?
Posted: Fri Jul 13, 2018 6:13 am
by peter
FWCC wrote: ↑Fri Jul 13, 2018 12:34 am
I think she will end up in Top 3. She will be using 8×V100s which is insane she should do well.
Of course depending on the book mainly, I don't think she'll have real good chances against Ginko, Johnny, Shredder and komodo at that moment.
And I am curious, if Hiarcs will yet run on 64 bits once in a while again nowadays.
But of course anything can happen at such tournaments with so little numbers of games.
Anyhow there should be some interesting ones among those with good books and good hardware.
Re: how will Leela fare at the WCCC?
Posted: Fri Jul 13, 2018 7:10 am
by MikeB
dannyb wrote: ↑Tue Jul 10, 2018 11:23 pm
what's your opinion?
does anyone know what hardware will it run on?
I suspect she will do very well since she will be on the hardware she was designed for. It's going to create quite a stir, if she wins. The progress has been amazing, I used to be able to beat it ( albeit on very weak hardware - re: Raspberry Pi ), but no more. K would have be considered the favorite here, but in a tournament like this - just about anyone of the entrants could win with a little luck. You will have multiple engines over 3000, so luck, color draws and the opening book will be huge factors in a short tournament.
Re: how will Leela fare at the WCCC?
Posted: Fri Jul 13, 2018 8:51 am
by noobpwnftw
If it doesn't win, there must be something unfair.
Re: how will Leela fare at the WCCC?
Posted: Fri Jul 13, 2018 9:53 am
by Laskos
Milos wrote: ↑Fri Jul 13, 2018 2:55 am
Laskos wrote: ↑Thu Jul 12, 2018 9:17 pm
Leela on 8 x Nvidia Tesla V100 gets maybe (depending on parallelization) about 20-30 the effective speed-up compared to my GPU.
Seriously???
I understand that there are quite a few ppl without a clue about how multiplexing on CUDA works, but you can at least inform yourself.
You know what is a speed up from 1 to 2 V100? Less than 30%. You think going from 2 to 4 would bring even 30%???
8xV100 is 1.25^3x3=6x1060. And you are dreaming of 20-30x, gee...
Diminishing returns don't magically disappear just because you use NN and MCTS.
Ask Mark about it on MCTS Komodo. And I'm fairly convinced Komodo MCTS is scaling far better than LC0.
I do know that in Go MCTS scaled well, much better than the A/B scaling in Chess, at least up to latest Lazy SMP implementations. And here, using NNs, I see no reason MCTS to scale badly.
As for your numbers for CUDA, there seem to be a contradiction with the initial AlphaGo paper:
https://gogameguru.com/i/2016/03/deepmi ... ing-go.pdf
Look at Table 8. Observe the huge jump in Elo for 1 -> 4 GPUs.
Even if it flattens at around 4-8 GPUs, the total factor would be 4 (faster GPU) x 3 (parallelization) ~
12. I was maybe too optimistic giving 20-30, also because I thought that single V100 is about 5-6 times faster than my GTX 1060.
All in all, it seems to still be correct to say that at the present conditions Leela, Komodo and Jonny are pretty equal top in WCCC (from regular opening positions), with maybe Ginkgo and Shredder not very far away. And this smaller result of 12 of effective speed-up over my GTX 1060 compared to "my dream" of 20-30 is somehow compensated by the fact that combined results show Leela with the best nets at about 3330-3350 CCRL 40/4' Elo on my GTX 1060, which is a bit better than my earlier results.
The issue here is the book, and whether the opponents can exploit the atrocious performance of Leela on tactical puzzles even in openings. Leela can easily lose 200 Elo points if the opponents will use tactically very sharp openings.
Re: how will Leela fare at the WCCC?
Posted: Fri Jul 13, 2018 10:34 am
by whereagles
It's the first real test of NN outside the Tyrel... errr... Google Corporation.
Looking forward to it. My guess is top 4.