TPU is roughly equivalent to 10 1080Ti. So A0 would be roughly 40x weaker on 1080Ti, i.e. would have around 1.8knps on average, which considering A0 vs. LC0 network size is a very good estimate.elpapa wrote:How strong would Alpha Zero be on consumer hardware, though?duncan wrote:when leela saturates should it be close to elo of AphaZero?
The real elo of lczero is in its name
Moderators: hgm, Rebel, chrisw
-
- Posts: 4190
- Joined: Wed Nov 25, 2009 1:47 am
Re: Leela 43 cores Versus Scorpio 32 cores at TCEC
-
- Posts: 211
- Joined: Sun Jan 18, 2009 11:27 pm
- Location: Sweden
- Full name: Patrik Karlsson
Re: Leela 43 cores Versus Scorpio 32 cores at TCEC
log(40) / log (2) = 5.3. Does 100 Elo per doubling sound reasonable do you think? If so, 500-550 Elo weaker.Milos wrote:TPU is roughly equivalent to 10 1080Ti. So A0 would be roughly 40x weaker on 1080Ti, i.e. would have around 1.8knps on average, which considering A0 vs. LC0 network size is a very good estimate.elpapa wrote:How strong would Alpha Zero be on consumer hardware, though?duncan wrote:when leela saturates should it be close to elo of AphaZero?
-
- Posts: 4190
- Joined: Wed Nov 25, 2009 1:47 am
Re: Leela 43 cores Versus Scorpio 32 cores at TCEC
Depends on TC, on 1s/move yes, on 1min/move it is hardly 60Elo per doubling.elpapa wrote:log(40) / log (2) = 5.3. Does 100 Elo per doubling sound reasonable do you think? If so, 500-550 Elo weaker.Milos wrote:TPU is roughly equivalent to 10 1080Ti. So A0 would be roughly 40x weaker on 1080Ti, i.e. would have around 1.8knps on average, which considering A0 vs. LC0 network size is a very good estimate.elpapa wrote:How strong would Alpha Zero be on consumer hardware, though?duncan wrote:when leela saturates should it be close to elo of AphaZero?
-
- Posts: 1142
- Joined: Thu Dec 28, 2017 4:06 pm
- Location: Argentina
Re: Leela 43 cores Versus Scorpio 32 cores at TCEC
Just a detail, it would make it 40x slower, not 40x weaker. Not the same thing.Milos wrote:TPU is roughly equivalent to 10 1080Ti. So A0 would be roughly 40x weaker on 1080Ti, i.e. would have around 1.8knps on average, which considering A0 vs. LC0 network size is a very good estimate.elpapa wrote:How strong would Alpha Zero be on consumer hardware, though?duncan wrote:when leela saturates should it be close to elo of AphaZero?
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls
-
- Posts: 708
- Joined: Mon Jan 16, 2012 6:34 am
Re: Leela 43 cores Versus Scorpio 32 cores at TCEC
As a general guess, A0 would be 3300 elo on 1080Ti meanwhile stockfish on 8 cores CPU would be 3500 elo. ( Doubling processing power = 50 elo strength).
In those setting SF would crush A0 by 75-25 scores in 100 games match.
In those setting SF would crush A0 by 75-25 scores in 100 games match.
-
- Posts: 211
- Joined: Sun Jan 18, 2009 11:27 pm
- Location: Sweden
- Full name: Patrik Karlsson
Re: Leela 43 cores Versus Scorpio 32 cores at TCEC
I thought perhaps a doubling of speed/time would be worth more for Alpha0 as it would suffer less from diminishing returns than a traditional engine. But maybe that's not the case.
-
- Posts: 708
- Joined: Mon Jan 16, 2012 6:34 am
Re: Leela 43 cores Versus Scorpio 32 cores at TCEC
I tested Leela with challenging puzzles. Lol
I let Leela run for infinite analysis for 12 mins. Had 3.3 million visits and she could not solve .LOl
Solution. e5!!
https://ibb.co/gjruBn[/url]
I let Leela run for infinite analysis for 12 mins. Had 3.3 million visits and she could not solve .LOl
Solution. e5!!
https://ibb.co/gjruBn[/url]
-
- Posts: 4185
- Joined: Tue Mar 14, 2006 11:34 am
- Location: Ethiopia
Re: Leela 43 cores Versus Scorpio 32 cores at TCEC
What is astounding is that it lost 20-0 against standard scorpio on single core. Your GPU is getting about 1 kN/s that is 3x slower than TCEC's which was getting about 3-4 kN/s. Maybe lczero benefits more from scaling but it was clear it can be exploited tactically even then. In that regard the 80 kN/s of alpha zero using a 40 block res net is a huge hardware benefit.
About your test with scorpio-mcts-min (which i estimate to be 2300-2400 elos) was for you to gauge the elo of lczero and work upwards the list I provided with different versions of the MCTS. You tested against the weakest version and waived hooray scorpio-mcts is weak bye bye Moreover, scorpio-mcts was running on a single core, they are both using similar algorithms but on has the privilage of using a faster hardware.
I see this as the everlasting question in computer chess eval vs search ? Leela prefers to have a slower eval (which makes it positionally strong) but tactically weaker. The MCTS version of scorpio uses scorpio's evaluation that would be much faster than leela's. So this is a choice one has to make.
Example:
On my i7 CPU leela gets about 200 playouts per second
Scorpio gets about 21000 pps that is a 100x more playouts per second than leela's. But this is a fair setup since both using single core. Leela's nps is slower due to its preference for a slower eval using neural network -- nobody forced it to do so.
Note that the nps number of scorpio is not the proper metric to compare against leela's nps. It is the pps number i.e. 21684 which is more or less equivalent to leela's nps.
I will run a tournament with this setup and come back with the results. I will then keep on increasing the time control to see if leela's performance would scale better than scorpio's mcts.
Daniel
About your test with scorpio-mcts-min (which i estimate to be 2300-2400 elos) was for you to gauge the elo of lczero and work upwards the list I provided with different versions of the MCTS. You tested against the weakest version and waived hooray scorpio-mcts is weak bye bye Moreover, scorpio-mcts was running on a single core, they are both using similar algorithms but on has the privilage of using a faster hardware.
I see this as the everlasting question in computer chess eval vs search ? Leela prefers to have a slower eval (which makes it positionally strong) but tactically weaker. The MCTS version of scorpio uses scorpio's evaluation that would be much faster than leela's. So this is a choice one has to make.
Example:
On my i7 CPU leela gets about 200 playouts per second
Code: Select all
NNCache: 0/0 hits/lookups = 0.0% hitrate, 0 inserts, 0 size
info depth 6 nodes 2 nps 16 score cp -279 winrate 16.71% time 61 pv e4 c5
info depth 8 nodes 5 nps 45 score cp -110 winrate 30.88% time 88 pv e4 c5 c3 e6
info depth 9 nodes 9 nps 71 score cp -59 winrate 38.74% time 111 pv e4 c5 c3 e6 d4 d5
info depth 10 nodes 14 nps 87 score cp -37 winrate 42.74% time 148 pv e4 c5 c3 e6 d4 d5
info depth 11 nodes 26 nps 113 score cp -16 winrate 46.81% time 220 pv e4 c5 c3 e6 d4 d5
info depth 12 nodes 44 nps 133 score cp -4 winrate 49.07% time 322 pv e4 e6 d4 d5 e5 c5 c3 Qa5
info depth 13 nodes 77 nps 149 score cp 4 winrate 50.80% time 509 pv e4 e6 d4 d5 e5 c5 c3 Qa5
info depth 14 nodes 135 nps 165 score cp 7 winrate 51.53% time 811 pv e4 e6 d4 d5 e5 c5 c3 Qa5 dxc5 Qc7
info depth 15 nodes 236 nps 180 score cp 11 winrate 52.18% time 1301 pv e4 e6 d4 d5 e5 c5 c3 Nc6 Nf3 Nge7 Bd3
info depth 16 nodes 419 nps 202 score cp 14 winrate 52.75% time 2067 pv e4 e6 d4 d5 e5 c5 c3 cxd4 cxd4 Nc6 Nf3 Nge7 Bd3
info depth 16 nodes 580 nps 212 score cp 16 winrate 53.28% time 2733 pv e4 e6 d4 d5 e5 c5 c3 cxd4 cxd4 Nc6 Nf3 Nge7 Bd3
Code: Select all
[st = 5000ms, mt = 5000ms , hply = 0]
31 48 50 393728 d2-d4 d7-d5 Ng1-f3 Ng8-f6 Nb1-c3 Nb8-c6
31 23 101 856382 e2-e4 e7-e5 Nb1-c3 Ng8-f6 d2-d4 e5xd4 Qd1xd4
31 20 151 1314093 d2-d4 Ng8-f6 Ng1-f3 d7-d5 e2-e3 e7-e6 Nb1-c3
31 20 201 1782427 e2-e3 Nb8-c6 Ng1-f3 e7-e6 d2-d4 d7-d5 Nb1-c3
31 38 253 2256719 Ng1-f3 Nb8-c6 e2-e4 e7-e5 Bf1-d3 d7-d5 Ke1-g1 d5xe4 Bd3xe4
31 46 303 2743809 e2-e4 d7-d6 d2-d4 e7-e5 Ng1-f3 Ng8-f6 Nb1-c3 e5xd4 Nf3xd4
31 47 354 3195903 e2-e4 e7-e5 Nb1-c3 Bf8-d6 f2-f4 e5xf4 Ng1-f3 Nb8-c6 d2-d4 Ng8-f6
31 47 404 3635736 e2-e4 e7-e5 Nb1-c3 Nb8-c6 Bf1-c4 Ng8-f6 Ng1-f3 Bf8-c5 a2-a3 Ke8-g8 Ke1-g1 d7-d6 d2-d3 Bc8-g4
31 53 455 4115699 d2-d4 Ng8-f6 e2-e3 d7-d6 Bf1-d3 e7-e5 Nb1-c3 c7-c5 d4xc5 d6xc5
nodes = 9158608 <91% qnodes> time = 5010ms nps = 1828065
Tree: nodes = 3173095 depth = 24 pps = 21684 visits = 108641
qsearch_calls = 1568687 search_calls = 0
move d2d4
I will run a tournament with this setup and come back with the results. I will then keep on increasing the time control to see if leela's performance would scale better than scorpio's mcts.
Daniel
-
- Posts: 1142
- Joined: Thu Dec 28, 2017 4:06 pm
- Location: Argentina
Re: Leela 43 cores Versus Scorpio 32 cores at TCEC
I will make another duel, this was your original list:
Do you think that matching Leela vs. Scorpio ab6 or ab7 would be ok?
Code: Select all
1. scorpio-mcts-ab16 375 25 25 1209 85.1% -26 17.1%
2. scorpio 357 23 23 1310 83.9% -23 19.3%
3. scorpio-mcts-ab12 285 21 21 1478 77.9% -25 16.8%
4. scorpio-mcts-ab10 187 21 21 1309 68.1% -11 20.6%
5. scorpio-mcts-ab9 168 21 21 1308 66.3% -10 19.6%
6. scorpio-mcts-ab8 131 20 20 1308 62.6% -7 18.5%
7. scorpio-mcts-ab7 78 20 20 1308 57.2% -3 18.3%
8. scorpio-mcts-ab6 27 20 20 1308 52.3% 0 20.1%
9. scorpio-mcts-ab5 -50 20 20 1308 44.3% 5 18.1%
10. scorpio-mcts-ab4 -126 21 21 1308 36.9% 10 16.3%
11. scorpio-mcts-ab3 -177 21 21 1308 32.3% 14 14.2%
12. scorpio-mcts-min -236 22 22 1310 26.1% 18 18.5%
13. scorpio-mcts-ab2 -258 22 22 1308 24.7% 19 13.8%
14. scorpio-mcts-ab1 -365 25 25 1310 16.6% 27 11.0%
15. scorpio-mcts-ab -389 25 25 1310 14.6% 29 11.7%
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls
-
- Posts: 4185
- Joined: Tue Mar 14, 2006 11:34 am
- Location: Ethiopia
Re: Leela 43 cores Versus Scorpio 32 cores at TCEC
You should make the game against scorpio-mcts-min fairly first because it was using 1 core. If I assume that on the same hardware scorpio's evaluation is 100x faster than leela's (10x128 nn), then to be fair against a 1000 playouts leela on your GPU (i.e. if you have to run it on the GPU), the opponent engines should be using atleast 10 cores. That goes for the leela gauntlets you are doing with other engines too not only for scorpio.
P.S Anyone knows how to configure lczero on cutechess. I have this
But it says it can't find the weights.txt file
Does the working directory specification work for uci engines ?
P.S Anyone knows how to configure lczero on cutechess. I have this
Code: Select all
{
"name": "lczero",
"command": "../engines/leela-chess/build/lczero",
"workingDirectory": "../engines/leela-chess/build",
"protocol": "uci",
"options" : [
{
"name" : "Hash",
"value" : 64
}
]
},
Code: Select all
Using 2 thread(s).
Could not open weights file: weights.txt
lczero: /home/daniel/engines/leela-chess/src/Network.cpp:376: static void Network::initialize(): Assertion `m_format_version > 0' failed.
Aborted (core dumped)