Something goes wrong with lc0 since yesterday?

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Karol Majewski
Posts: 94
Joined: Sat Jul 09, 2011 2:18 pm

Re: Something goes wrong with lc0 since yesterday?

Post by Karol Majewski » Fri Aug 10, 2018 5:08 pm

I'm still CPU guy, so I have question regarding GPU: is Leela's search on GPU deterministic, so that you can reproduce results on the same hardware, or is it non-deterministic, just like in paralell multi-CPU search?

User avatar
George Tsavdaris
Posts: 1557
Joined: Thu Mar 09, 2006 11:35 am

Re: Something goes wrong with lc0 since yesterday?

Post by George Tsavdaris » Sat Aug 11, 2018 7:55 am

Can't really believe this so far. :D
Lc0 10520 has so far a 3570 CCRL 40/4 Elo performance after 50 games. :shock: :shock:
Will continue till 100 games like with 10480.

Lc0 Test10 10520 after 50 games has 3571±89 CCRL 40/4 Elo

Code: Select all

  Program          CCRL Elo   Error(cl 95%)          Games            Score 
Lc0! Test10 10520   3571.0       ±88.5            50 (+37,=10,-3)     84.0 %

   vs.                      :  games (  +,  =, -),   (%) :    Diff,    SD, CFS (%)
   Stockfish 8              :     10 (  5,  4, 1),  70.0 :  +148.0,  45.2,   99.9
   Fire 7.1                 :     10 (  7,  2, 1),  80.0 :  +230.0,  45.2,  100.0
   Andscacs 9.3             :     10 (  9,  0, 1),  90.0 :  +364.0,  45.2,  100.0
   Gull 3                   :     10 (  7,  3, 0),  85.0 :  +378.0,  45.2,  100.0
   Texel 1.07               :     10 (  9,  1, 0),  95.0 :  +453.0,  45.2,  100.0
After his son's birth they've asked him:
"Is it a boy or girl?"
YES! He replied.....

Nay Lin Tun
Posts: 509
Joined: Mon Jan 16, 2012 5:34 am

Re: Something goes wrong with lc0 since yesterday?

Post by Nay Lin Tun » Sat Aug 11, 2018 8:33 am

George Tsavdaris wrote:
Sat Aug 11, 2018 7:55 am
Can't really believe this so far. :D
Lc0 10520 has so far a 3570 CCRL 40/4 Elo performance after 50 games. :shock: :shock:
Will continue till 100 games like with 10480.

Lc0 Test10 10520 after 50 games has 3571±89 CCRL 40/4 Elo

Code: Select all

  Program          CCRL Elo   Error(cl 95%)          Games            Score 
Lc0! Test10 10520   3571.0       ±88.5            50 (+37,=10,-3)     84.0 %

   vs.                      :  games (  +,  =, -),   (%) :    Diff,    SD, CFS (%)
   Stockfish 8              :     10 (  5,  4, 1),  70.0 :  +148.0,  45.2,   99.9
   Fire 7.1                 :     10 (  7,  2, 1),  80.0 :  +230.0,  45.2,  100.0
   Andscacs 9.3             :     10 (  9,  0, 1),  90.0 :  +364.0,  45.2,  100.0
   Gull 3                   :     10 (  7,  3, 0),  85.0 :  +378.0,  45.2,  100.0
   Texel 1.07               :     10 (  9,  1, 0),  95.0 :  +453.0,  45.2,  100.0
There is another +17 elo gain from ID 520 to ID 588. :D

User avatar
George Tsavdaris
Posts: 1557
Joined: Thu Mar 09, 2006 11:35 am

Re: Something goes wrong with lc0 since yesterday?

Post by George Tsavdaris » Sat Aug 11, 2018 12:54 pm

Another incredible result of Lc0 10520 testnet. (by Ankan in Leela's discord)

Leela Lc0 with 10520 net running on 1 GV100, versus Stockfish dev 18080112 running on 6 threads i7 5820k @4.2 GHz

TC was 2+2 and Arena perfect2010 opening book was used and the result was:
Leela 10520 - SF dev +8 -8 =69
0 ELo
difference with confidence interval [-32 , +32] (95% c.l).


I can give the PGN if someone wants them.
After his son's birth they've asked him:
"Is it a boy or girl?"
YES! He replied.....

Lion
Posts: 509
Joined: Fri Mar 31, 2006 11:26 am
Location: Switzerland

Re: Something goes wrong with lc0 since yesterday?

Post by Lion » Sat Aug 11, 2018 3:33 pm

I am interested in the pgn

thank you

User avatar
George Tsavdaris
Posts: 1557
Joined: Thu Mar 09, 2006 11:35 am

Re: Something goes wrong with lc0 since yesterday?

Post by George Tsavdaris » Sat Aug 11, 2018 3:52 pm

Lion wrote:
Sat Aug 11, 2018 3:33 pm
I am interested in the pgn

thank you
https://drive.google.com/open?id=1bOZGV ... 9YxBNDbeSu
After his son's birth they've asked him:
"Is it a boy or girl?"
YES! He replied.....

Lion
Posts: 509
Joined: Fri Mar 31, 2006 11:26 am
Location: Switzerland

Re: Something goes wrong with lc0 since yesterday?

Post by Lion » Sat Aug 11, 2018 4:09 pm

Thanks!

Just dl them and will look at them asap

rgds

User avatar
Laskos
Posts: 9244
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Something goes wrong with lc0 since yesterday?

Post by Laskos » Sat Aug 11, 2018 10:40 pm

George Tsavdaris wrote:
Sat Aug 11, 2018 12:54 pm
Another incredible result of Lc0 10520 testnet. (by Ankan in Leela's discord)

Leela Lc0 with 10520 net running on 1 GV100, versus Stockfish dev 18080112 running on 6 threads i7 5820k @4.2 GHz

TC was 2+2 and Arena perfect2010 opening book was used and the result was:
Leela 10520 - SF dev +8 -8 =69
0 ELo
difference with confidence interval [-32 , +32] (95% c.l).


I can give the PGN if someone wants them.
I have on my hardware: Lc0 ID10610 on GTX 1060 6GB GPU, SF8 on one i7 3.8 GHz core.

Time control 2' + 2'' (equivalent to CCRL 40/4' CPU hardware-time wise)

Score of lc0_v16 10610 vs SF 8: 9 - 10 - 21 [0.487]
Elo difference: -8.69 +/- 75.25

40 of 40 games finished.

Performance of 3420 CCRL 40/4' Elo points. At longer time control or stronger hardware it surely increases, as Lc0 scales better
The thing is the result would have been much better for Lc0 if not for stupid endgame blunders (if not bugs). Here I present 2 of them in those 40 games:


One won game by Lc0 which it managed to draw in what seems to be even a bug:





White Lc0 ID10610 to move.
Draw!


Another, a completely drawn late endgame, which Lc0 manages to lose, playing really miserably.





Black Lc0 ID10610 to move.
Black loses!

Sven
Posts: 3811
Joined: Thu May 15, 2008 7:57 pm
Location: Berlin, Germany
Full name: Sven Schüle
Contact:

Re: Something goes wrong with lc0 since yesterday?

Post by Sven » Sun Aug 12, 2018 6:56 pm

Laskos wrote:
Sat Aug 11, 2018 10:40 pm
One won game by Lc0 which it managed to draw in what seems to be even a bug:
[...]


White Lc0 ID10610 to move.
Draw!
Agreed, this must be a bug, why doesn't LC0 promote the pawn?
Laskos wrote:
Sat Aug 11, 2018 10:40 pm
Another, a completely drawn late endgame, which Lc0 manages to lose, playing really miserably.
[...]


Black Lc0 ID10610 to move.
Black loses!
Here I think this can be hard to find without EGTB. Strictly spoken the next 52 moves after the diagram were not played "miserably" since the position was still a TB draw until 126...Rf2? which leads to a mate in 16. Of course losing both pawns was absolutely not necessary but not losing them would not have changed the optimal outcome of the position, so losing them was not "incorrect".
Sven Schüle (engine author: Jumbo, KnockOut, Surprise)

tmokonen
Posts: 1014
Joined: Sun Mar 12, 2006 5:46 pm
Location: Vancouver

Re: Something goes wrong with lc0 since yesterday?

Post by tmokonen » Fri Aug 24, 2018 4:16 am

So... the main server nets are gone, never to return?

Post Reply