The real elo of lczero is in its name

[CMCanavessi]

You should make the game against scorpio-mcts-min fairly first because it was using 1 core. If I assume that on the same hardware scorpio's evaluation is 100x faster than leela's (10x128 nn), then to be fair against a 1000 playouts leela on your GPU (i.e. if you have to run it on the GPU), the opponent engines should be using atleast 10 cores. That goes for the leela gauntlets you are doing with other engines too not only for scorpio.

P.S Anyone knows how to configure lczero on cutechess. I have this

Code: Select all

    {
        "name": "lczero",
        "command": "../engines/leela-chess/build/lczero",
        "workingDirectory": "../engines/leela-chess/build",
        "protocol": "uci",
        "options" : [
         {
            "name" : "Hash",
            "value" : 64
         }
        ]
    },

But it says it can't find the weights.txt file

Code: Select all

Using 2 thread(s).
Could not open weights file: weights.txt
lczero: /home/daniel/engines/leela-chess/src/Network.cpp:376: static void Network::initialize(): Assertion `m_format_version > 0' failed.
Aborted (core dumped)

Does the working directory specification work for uci engines ?

You need to specify -w <weight file>, you just missed the -w thing

[Daniel Shawul]

I tried adding it to the command line even specifiying the full path again but didn't work.
Anyway i though recent versions default to weights.txt.

I tried the following

Code: Select all

"command": "../engines/leela-chess/build/lczero",
"command": "../engines/leela-chess/build/lczero -w ID125",
"command": "../engines/leela-chess/build/lczero -w weights.txt",
"command": "../engines/leela-chess/build/lczero -w ../engines/leela-chess/build/weights.tx",

If I typed the last one directly on the command line it works, so it is probably an issue with the way cutechess handles working directories i think.

Code: Select all

daniel@dani-xp:~/cutechess-cli$ ./test.sh
Elo difference: nan +/- nan
Warning: Cannot start engine lczero:
Cannot execute command: ~/engines/leela-chess/build/lczero -w ~/engines/leela-chess/build/weights.txt
Finished match
daniel@dani-xp:~/cutechess-cli$ ~/engines/leela-chess/build/lczero -w ~/engines/leela-chess/build/weights.txt
Using 2 thread(s).
Detecting residual layers...v2...128 channels...10 blocks.
BLAS Core: Haswell

[Daniel Shawul]

The following worked

Code: Select all

    {
        "name": "lczero",
        "command": "./lczero",
        "workingDirectory": "../engines/leela-chess/build",
        "protocol": "uci",
        "options" : [
         {
            "name" : "Hash",
            "value" : 64
         }
        ]
    },

First result you are not going to like 90% for scorpio-mcts

Code: Select all

Finished game 23 (lczero vs scorpio-mcts-min): 0-1 {Black mates}
Score of lczero vs scorpio-mcts-min: 2 - 20 - 1  [0.109] 23
Started game 24 of 8000 (scorpio-mcts-min vs lczero)

I used a fast 20+0.1 and lczero is loosing almost all its games. I am not making any conclusions from this. My goal is to produce a plot by keep on doubing the time control until leela-chess starts to beat it. This may help to explain that leela is covering up its tactical weakness by conductng test games either at long time control (until it sees tactis that is trivial for full width searcher) or using a hardware that allows it to do many playouts to cover for its lack of tactical awareness.

Btw I am using ID 125 that is being used in tcec.

[Guenther]

What is astounding is that it lost 20-0 against standard scorpio on single core. Your GPU is getting about 1 kN/s that is 3x slower than TCEC's which was getting about 3-4 kN/s.l

The displayed speed in kN/s for Leela is wrong in TCEC.
See my other post.

[Daniel Shawul]

What is astounding is that it lost 20-0 against standard scorpio on single core. Your GPU is getting about 1 kN/s that is 3x slower than TCEC's which was getting about 3-4 kN/s.l

The displayed speed in kN/s for Leela is wrong in TCEC.
See my other post.

Ah, so it could be 3x more than what is displayed, so maybe 9-12x kN/s then.

20+0.1 is done now and it ended up a -420 for leela

Code: Select all

20+0.1

Score of lczero vs scorpio-mcts-min: 2 - 40 - 3  [0.078] 45
Elo difference: -429.59 +/- 252.08
SPRT: llr -3.01, lbound -2.94, ubound 2.94 - H0 was accepted
Finished match

40+0.1 , 80+0.1 etc will come next until we produce enough data points for a plot that leela's strength will cross scorpio-mcts-min's

Daniel

[Daniel Shawul]

20+0.1

Code: Select all

Score of lczero vs scorpio-mcts-min: 2 - 40 - 3  [0.078] 45 
Elo difference: -429.59 +/- 252.08 
SPRT: llr -3.01, lbound -2.94, ubound 2.94 - H0 was accepted 
Finished match 

40+0.1

Code: Select all

Score of lczero vs scorpio-mcts-min: 3 - 35 - 8  [0.152] 46
Elo difference: -298.39 +/- 125.93
SPRT: llr -3.03, lbound -2.94, ubound 2.94 - H0 was accepted
Finished match

Getting better. I am gonna cap of the rest of the matches at 50 games for the sake of time.

[CMCanavessi]

What is astounding is that it lost 20-0 against standard scorpio on single core. Your GPU is getting about 1 kN/s that is 3x slower than TCEC's which was getting about 3-4 kN/s.l

The displayed speed in kN/s for Leela is wrong in TCEC.
See my other post.

It is not wrong, what may be misleading is the total number of nodes, because of tree reuse. Those are not part of the real speed, it's like cache.

[Daniel Shawul]

80+0.2

Code: Select all

Finished game 36 (scorpio-mcts-min vs lczero): * {No result}
Score of lczero vs scorpio-mcts-min: 0 - 30 - 5  [0.071] 35
Elo difference: -445.58 +/- 206.80
Finished match

Stopped this one early as it seems it's gotten worse now. I m going for a 4x bigger time control now 320+0.4. But seeing Ferd's result with 12h+1min not optimistic that I would get this done soon. i.e. overpass the 2300-2400 elo scorpio-mcts-min on my "slow" i7 cpu that does 200 pps. A0 results could probably be explained by this massive hardware acceleration of its bulky NN eval to cover up for tactical weakness; just like an FGPA can be used to accelerate a massive evaluation. Strictly speaking from an algorithmic point of view however their MCTS sucks.

Note that

[carldaman]

I tested Leela with challenging puzzles. Lol



I let Leela run for infinite analysis for 12 mins. Had 3.3 million visits and she could not solve .LOl

Solution. e5!!

https://ibb.co/gjruBn[/url]

How are you able to run Leela in infinite analysis? My Leela (v0.6) runs analysis for about 1 sec and then it stops (running on cpu's).

Thanks,
CL

[Guenther]

What is astounding is that it lost 20-0 against standard scorpio on single core. Your GPU is getting about 1 kN/s that is 3x slower than TCEC's which was getting about 3-4 kN/s.l

The displayed speed in kN/s for Leela is wrong in TCEC.
See my other post.

It is not wrong, what may be misleading is the total number of nodes, because of tree reuse. Those are not part of the real speed, it's like cache.

If this is true I see no way to get a reliable bench with LCZero.
It can just print a fictional nps, because it does not show which nodes
are reused and which not.