The real elo of lczero is in its name

[Milos]

when leela saturates should it be close to elo of AphaZero?

How strong would Alpha Zero be on consumer hardware, though?

TPU is roughly equivalent to 10 1080Ti. So A0 would be roughly 40x weaker on 1080Ti, i.e. would have around 1.8knps on average, which considering A0 vs. LC0 network size is a very good estimate.

[elpapa]

when leela saturates should it be close to elo of AphaZero?

How strong would Alpha Zero be on consumer hardware, though?

TPU is roughly equivalent to 10 1080Ti. So A0 would be roughly 40x weaker on 1080Ti, i.e. would have around 1.8knps on average, which considering A0 vs. LC0 network size is a very good estimate.

log(40) / log (2) = 5.3. Does 100 Elo per doubling sound reasonable do you think? If so, 500-550 Elo weaker.

[Milos]

when leela saturates should it be close to elo of AphaZero?

How strong would Alpha Zero be on consumer hardware, though?

TPU is roughly equivalent to 10 1080Ti. So A0 would be roughly 40x weaker on 1080Ti, i.e. would have around 1.8knps on average, which considering A0 vs. LC0 network size is a very good estimate.

log(40) / log (2) = 5.3. Does 100 Elo per doubling sound reasonable do you think? If so, 500-550 Elo weaker.

Depends on TC, on 1s/move yes, on 1min/move it is hardly 60Elo per doubling.

[CMCanavessi]

when leela saturates should it be close to elo of AphaZero?

How strong would Alpha Zero be on consumer hardware, though?

TPU is roughly equivalent to 10 1080Ti. So A0 would be roughly 40x weaker on 1080Ti, i.e. would have around 1.8knps on average, which considering A0 vs. LC0 network size is a very good estimate.

Just a detail, it would make it 40x slower, not 40x weaker. Not the same thing.

[Nay Lin Tun]

As a general guess, A0 would be 3300 elo on 1080Ti meanwhile stockfish on 8 cores CPU would be 3500 elo. ( Doubling processing power = 50 elo strength).
In those setting SF would crush A0 by 75-25 scores in 100 games match.

[elpapa]

I thought perhaps a doubling of speed/time would be worth more for Alpha0 as it would suffer less from diminishing returns than a traditional engine. But maybe that's not the case.

[Nay Lin Tun]

I tested Leela with challenging puzzles. Lol



I let Leela run for infinite analysis for 12 mins. Had 3.3 million visits and she could not solve .LOl

Solution. e5!!

https://ibb.co/gjruBn[/url]

[Daniel Shawul]

What is astounding is that it lost 20-0 against standard scorpio on single core. Your GPU is getting about 1 kN/s that is 3x slower than TCEC's which was getting about 3-4 kN/s. Maybe lczero benefits more from scaling but it was clear it can be exploited tactically even then. In that regard the 80 kN/s of alpha zero using a 40 block res net is a huge hardware benefit.

About your test with scorpio-mcts-min (which i estimate to be 2300-2400 elos) was for you to gauge the elo of lczero and work upwards the list I provided with different versions of the MCTS. You tested against the weakest version and waived hooray scorpio-mcts is weak bye bye Moreover, scorpio-mcts was running on a single core, they are both using similar algorithms but on has the privilage of using a faster hardware.

I see this as the everlasting question in computer chess eval vs search ? Leela prefers to have a slower eval (which makes it positionally strong) but tactically weaker. The MCTS version of scorpio uses scorpio's evaluation that would be much faster than leela's. So this is a choice one has to make.

Example:
On my i7 CPU leela gets about 200 playouts per second

Code: Select all

NNCache: 0/0 hits/lookups = 0.0% hitrate, 0 inserts, 0 size
info depth 6 nodes 2 nps 16 score cp -279 winrate 16.71% time 61 pv e4 c5
info depth 8 nodes 5 nps 45 score cp -110 winrate 30.88% time 88 pv e4 c5 c3 e6
info depth 9 nodes 9 nps 71 score cp -59 winrate 38.74% time 111 pv e4 c5 c3 e6 d4 d5
info depth 10 nodes 14 nps 87 score cp -37 winrate 42.74% time 148 pv e4 c5 c3 e6 d4 d5
info depth 11 nodes 26 nps 113 score cp -16 winrate 46.81% time 220 pv e4 c5 c3 e6 d4 d5
info depth 12 nodes 44 nps 133 score cp -4 winrate 49.07% time 322 pv e4 e6 d4 d5 e5 c5 c3 Qa5
info depth 13 nodes 77 nps 149 score cp 4 winrate 50.80% time 509 pv e4 e6 d4 d5 e5 c5 c3 Qa5
info depth 14 nodes 135 nps 165 score cp 7 winrate 51.53% time 811 pv e4 e6 d4 d5 e5 c5 c3 Qa5 dxc5 Qc7
info depth 15 nodes 236 nps 180 score cp 11 winrate 52.18% time 1301 pv e4 e6 d4 d5 e5 c5 c3 Nc6 Nf3 Nge7 Bd3
info depth 16 nodes 419 nps 202 score cp 14 winrate 52.75% time 2067 pv e4 e6 d4 d5 e5 c5 c3 cxd4 cxd4 Nc6 Nf3 Nge7 Bd3

info depth 16 nodes 580 nps 212 score cp 16 winrate 53.28% time 2733 pv e4 e6 d4 d5 e5 c5 c3 cxd4 cxd4 Nc6 Nf3 Nge7 Bd3

Scorpio gets about 21000 pps that is a 100x more playouts per second than leela's. But this is a fair setup since both using single core. Leela's nps is slower due to its preference for a slower eval using neural network -- nobody forced it to do so.

Code: Select all

[st = 5000ms, mt = 5000ms , hply = 0]
31 48 50 393728  d2-d4 d7-d5 Ng1-f3 Ng8-f6 Nb1-c3 Nb8-c6
31 23 101 856382  e2-e4 e7-e5 Nb1-c3 Ng8-f6 d2-d4 e5xd4 Qd1xd4
31 20 151 1314093  d2-d4 Ng8-f6 Ng1-f3 d7-d5 e2-e3 e7-e6 Nb1-c3
31 20 201 1782427  e2-e3 Nb8-c6 Ng1-f3 e7-e6 d2-d4 d7-d5 Nb1-c3
31 38 253 2256719  Ng1-f3 Nb8-c6 e2-e4 e7-e5 Bf1-d3 d7-d5 Ke1-g1 d5xe4 Bd3xe4
31 46 303 2743809  e2-e4 d7-d6 d2-d4 e7-e5 Ng1-f3 Ng8-f6 Nb1-c3 e5xd4 Nf3xd4
31 47 354 3195903  e2-e4 e7-e5 Nb1-c3 Bf8-d6 f2-f4 e5xf4 Ng1-f3 Nb8-c6 d2-d4 Ng8-f6
31 47 404 3635736  e2-e4 e7-e5 Nb1-c3 Nb8-c6 Bf1-c4 Ng8-f6 Ng1-f3 Bf8-c5 a2-a3 Ke8-g8 Ke1-g1 d7-d6 d2-d3 Bc8-g4
31 53 455 4115699  d2-d4 Ng8-f6 e2-e3 d7-d6 Bf1-d3 e7-e5 Nb1-c3 c7-c5 d4xc5 d6xc5
nodes = 9158608 <91% qnodes> time = 5010ms nps = 1828065
Tree&#58; nodes = 3173095 depth = 24 pps = 21684 visits = 108641 
      qsearch_calls = 1568687 search_calls = 0
move d2d4

Note that the nps number of scorpio is not the proper metric to compare against leela's nps. It is the pps number i.e. 21684 which is more or less equivalent to leela's nps.

I will run a tournament with this setup and come back with the results. I will then keep on increasing the time control to see if leela's performance would scale better than scorpio's mcts.

Daniel

[CMCanavessi]

I will make another duel, this was your original list:

Code: Select all

 1.                       scorpio-mcts-ab16    375     25     25   1209    85.1%   -26    17.1%
 2.                                 scorpio    357     23     23   1310    83.9%   -23    19.3%
 3.                       scorpio-mcts-ab12    285     21     21   1478    77.9%   -25    16.8%
 4.                       scorpio-mcts-ab10    187     21     21   1309    68.1%   -11    20.6%
 5.                        scorpio-mcts-ab9    168     21     21   1308    66.3%   -10    19.6%
 6.                        scorpio-mcts-ab8    131     20     20   1308    62.6%    -7    18.5%
 7.                        scorpio-mcts-ab7     78     20     20   1308    57.2%    -3    18.3%
 8.                        scorpio-mcts-ab6     27     20     20   1308    52.3%     0    20.1%
 9.                        scorpio-mcts-ab5    -50     20     20   1308    44.3%     5    18.1%
10.                        scorpio-mcts-ab4   -126     21     21   1308    36.9%    10    16.3%
11.                        scorpio-mcts-ab3   -177     21     21   1308    32.3%    14    14.2%
12.                        scorpio-mcts-min   -236     22     22   1310    26.1%    18    18.5%
13.                        scorpio-mcts-ab2   -258     22     22   1308    24.7%    19    13.8%
14.                        scorpio-mcts-ab1   -365     25     25   1310    16.6%    27    11.0%
15.                         scorpio-mcts-ab   -389     25     25   1310    14.6%    29    11.7% 

Do you think that matching Leela vs. Scorpio ab6 or ab7 would be ok?

[Daniel Shawul]

You should make the game against scorpio-mcts-min fairly first because it was using 1 core. If I assume that on the same hardware scorpio's evaluation is 100x faster than leela's (10x128 nn), then to be fair against a 1000 playouts leela on your GPU (i.e. if you have to run it on the GPU), the opponent engines should be using atleast 10 cores. That goes for the leela gauntlets you are doing with other engines too not only for scorpio.

P.S Anyone knows how to configure lczero on cutechess. I have this

Code: Select all

    {
        "name": "lczero",
        "command": "../engines/leela-chess/build/lczero",
        "workingDirectory": "../engines/leela-chess/build",
        "protocol": "uci",
        "options" : [
         {
            "name" : "Hash",
            "value" : 64
         }
        ]
    },

But it says it can't find the weights.txt file

Code: Select all

Using 2 thread(s).
Could not open weights file: weights.txt
lczero: /home/daniel/engines/leela-chess/src/Network.cpp:376: static void Network::initialize(): Assertion `m_format_version > 0' failed.
Aborted (core dumped)

Does the working directory specification work for uci engines ?