Strong Leela nets competition

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
corres
Posts: 1802
Joined: Wed Nov 18, 2015 10:41 am
Location: hungary

Strong Leela nets competition

Post by corres » Fri Nov 08, 2019 8:13 pm

I made a competition among strong Leela nets.
Parameters
Used Leela: v.0.22.0
GPUs: 2 x RTX 2060 OC
Threads: 4 X 4GHz
Backend: Multiplexing
BackendOptions: backend=cudnn-fp16,(gpu=0),(gpu=1)
NNCacheSize: 10000000
MaxPrefetchBatch: 64
Others are Default
Time Control: 1 min + 2 sec / move
50 opening positions played with alternated colors

Results
T40B.4-160 : 42872 = 4 : 3 (93 draw) 100 games
42872 : J13B.2-188 = 9 : 6 (85 draw) 100 games
T40B.4-160 : J13B.2-188 = 9 : 3 (88 draw) 100 games

In Points
T40B.4-160 net 13 points
42872 net 12 points
J13B.2-188 net 9 points

I made test with 384x30-T40-970 "big net" with disappointing result
J13B.2-188 : 384x30-T40-970 = 17(!) : 3 (80 draw) 100 games.
Seeing the result I did not make further test with it.

User avatar
Laskos
Posts: 9545
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Strong Leela nets competition

Post by Laskos » Fri Nov 08, 2019 8:21 pm

corres wrote:
Fri Nov 08, 2019 8:13 pm
I made a competition among strong Leela nets.
Parameters
Used Leela: v.0.22.0
GPUs: 2 x RTX 2060 OC
Threads: 4 X 4GHz
Backend: Multiplexing
BackendOptions: backend=cudnn-fp16,(gpu=0),(gpu=1)
NNCacheSize: 10000000
MaxPrefetchBatch: 64
Others are Default
Time Control: 1 min + 2 sec / move
50 opening positions played with alternated colors

Results
T40B.4-160 : 42872 = 4 : 3 (93 draw) 100 games
42872 : J13B.2-188 = 9 : 6 (85 draw) 100 games
T40B.4-160 : J13B.2-188 = 9 : 3 (88 draw) 100 games

In Points
T40B.4-160 net 13 points
42872 net 12 points
J13B.2-188 net 9 points

I made test with 384x30-T40-970 "big net" with disappointing result
J13B.2-188 : 384x30-T40-970 = 17(!) : 3 (80 draw) 100 games.
Seeing the result I did not make further test with it.
J13B.4 is strong at LTC on strong hardware, or so it seems to me.

corres
Posts: 1802
Joined: Wed Nov 18, 2015 10:41 am
Location: hungary

Re: Strong Leela nets competition

Post by corres » Fri Nov 08, 2019 8:32 pm

Laskos wrote:
Fri Nov 08, 2019 8:21 pm
...
J13B.4 is strong at LTC on strong hardware, or so it seems to me.
Where we can see the results of J13b.4-xyz?

corres
Posts: 1802
Joined: Wed Nov 18, 2015 10:41 am
Location: hungary

Re: Strong Leela nets competition

Post by corres » Fri Nov 08, 2019 8:52 pm

corres wrote:
Fri Nov 08, 2019 8:13 pm
I made a competition among strong Leela nets.
Parameters
Used Leela: v.0.22.0
GPUs: 2 x RTX 2060 OC
Threads: 4 X 4GHz
Backend: Multiplexing
BackendOptions: backend=cudnn-fp16,(gpu=0),(gpu=1)
NNCacheSize: 10000000
MaxPrefetchBatch: 64
Others are Default
Time Control: 1 min + 2 sec / move
50 opening positions played with alternated colors

Results
T40B.4-160 : 42872 = 4 : 3 (93 draw) 100 games
42872 : J13B.2-188 = 9 : 6 (85 draw) 100 games
T40B.4-160 : J13B.2-188 = 9 : 3 (88 draw) 100 games

In Points
T40B.4-160 net 13 points
42872 net 12 points
J13B.2-188 net 9 points

I made test with 384x30-T40-970 "big net" with disappointing result
J13B.2-188 : 384x30-T40-970 = 17(!) : 3 (80 draw) 100 games.
Seeing the result I did not make further test with it.
You can download the .zip file
wikisend.com
File ID: 621504
Password: leela

shrapnel
Posts: 1248
Joined: Fri Nov 02, 2012 8:43 am
Location: New Delhi, India

Re: Strong Leela nets competition

Post by shrapnel » Sat Nov 09, 2019 11:15 am

Playing big NNs against small NNs is one thing. Playing those NNs against Stockfish is a whole new Ball Game .
You "MAY" get different results if you play them against Stockfish.
"MAY" because your GPUs are weak and will be unable to fully utilize the powere of large NNs.
So, they may again lose on your Setup.
That's the Problem with testing of NNs.
One has to take into account BOTH the NN being used AND the GPU... TWO Factors involved.
i7 5960X @ 4.1 Ghz, 64 GB G.Skill RipJaws RAM, Twin Asus ROG Strix OC 11 GB Geforce 2080 Tis

Damir
Posts: 2198
Joined: Mon Feb 11, 2008 2:53 pm
Location: Denmark
Full name: Damir Desevac

Re: Strong Leela nets competition

Post by Damir » Sat Nov 09, 2019 11:28 am

in TCEC Leela lost 2 games already…. it's play is crap, and seems no improvement has been made.. :) :) It is loser engine whose play only depends on strong GPU's... :) :)

corres
Posts: 1802
Joined: Wed Nov 18, 2015 10:41 am
Location: hungary

Re: Strong Leela nets competition

Post by corres » Sat Nov 09, 2019 12:04 pm

shrapnel wrote:
Sat Nov 09, 2019 11:15 am
Playing big NNs against small NNs is one thing. Playing those NNs against Stockfish is a whole new Ball Game .
You "MAY" get different results if you play them against Stockfish.
"MAY" because your GPUs are weak and will be unable to fully utilize the powere of large NNs.
So, they may again lose on your Setup.
That's the Problem with testing of NNs.
One has to take into account BOTH the NN being used AND the GPU... TWO Factors involved.
Sometimes you also should make some test, do you not think that?
I have no Twin RTX 2080 Ti only a twin RTX 2060 OC.
I put my openings on net and you should play two matches: a match (100 games) between the last Stockfish dev and your favorite big net as well as a match (100 games) between the last Stockfish dev and T40B.4-160 net with the same parameters. It takes about a day test, if you use 1 min + 2 sec / move TC (shorter time is bad!). If the big net will get better result against Stockfish than T40B.4-160 will, you are right.
It would be the exact evidence.
Any other is only ''bla-di-bla".

Nay Lin Tun
Posts: 530
Joined: Mon Jan 16, 2012 5:34 am

Re: Strong Leela nets competition

Post by Nay Lin Tun » Sat Nov 09, 2019 1:02 pm

Damir wrote:
Sat Nov 09, 2019 11:28 am
in TCEC Leela lost 2 games already…. it's play is crap, and seems no improvement has been made.. :) :) It is loser engine whose play only depends on strong GPU's... :) :)
I guess you are joking! Did you check what kind of custom positions they are testing for? Most of them are like that, I did not check all of them though!
Leela will probably play at 3000 rating! But not surprising!
NN are not trained for such custom positions that you wont see 1 in billion games.

Damir
Posts: 2198
Joined: Mon Feb 11, 2008 2:53 pm
Location: Denmark
Full name: Damir Desevac

Re: Strong Leela nets competition

Post by Damir » Sat Nov 09, 2019 1:14 pm

My response was to Shrapnel,,, since he keeps bashing all other engines and praise Leela... to show him that Lc0 is patzer engine too…. :) :)

User avatar
Nordlandia
Posts: 2479
Joined: Fri Sep 25, 2015 7:38 pm
Location: Sortland, Norway

Re: Strong Leela nets competition

Post by Nordlandia » Mon Nov 11, 2019 7:28 am

Try this position!



Lyudmil Tsvetkov claimed black wins. But Lc0 did not manage to win against SF Dev and SF 10.



Also he claimed that white wins here too.



5+5 Blitz | Lc0 (2080 Super) vs 7-core SF Dev


Post Reply