Lczero stalled in progress?

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Robert Flesher
Posts: 1211
Joined: Tue Aug 18, 2009 1:06 am

Lczero stalled in progress?

Post by Robert Flesher » Thu May 10, 2018 5:53 pm

I am just curious if the experts can explain why Lczero seems to have stalled in progress?

User avatar
Ozymandias
Posts: 1087
Joined: Sun Oct 25, 2009 12:30 am

Re: Lczero stalled in progress?

Post by Ozymandias » Thu May 10, 2018 6:27 pm

Robert Flesher wrote:
Thu May 10, 2018 5:53 pm
I am just curious if the experts can explain why Lczero seems to have stalled in progress?
When they say at the beginning of their main page, that "We are actively investigating the strength fluctuations, and improving the system. v0.9 helps here.", I'm guessing they can't... yet.

noobpwnftw
Posts: 327
Joined: Sun Nov 08, 2015 10:10 pm

Re: Lczero stalled in progress?

Post by noobpwnftw » Thu May 10, 2018 6:46 pm

Where are those people who think over-fitting can be "easily solved"?

The way I see this: probably the networks were over-fitted all along, a minor change to learning rates and more noise from resigning just made it obvious. There is an alternative method to prove, if it truly understands chess, then you are free to half or double the number of playouts during training and it should result in no regression.

Unfortunately their bible has no chapter on this matter, so it should "just work".

Albert Silver
Posts: 2829
Joined: Wed Mar 08, 2006 8:57 pm
Location: Rio de Janeiro, Brazil

Re: Lczero stalled in progress?

Post by Albert Silver » Thu May 10, 2018 7:31 pm

Robert Flesher wrote:
Thu May 10, 2018 5:53 pm
I am just curious if the experts can explain why Lczero seems to have stalled in progress?
A massive bug was revealed today, which has also affected the training. Leela will literally refuse to capture a hanging queen:

Image

https://groups.google.com/forum/#!topic ... iOlKkuhCGw
"Tactics are the bricks and sticks that make up a game, but positional play is the architectural blueprint."

Albert Silver
Posts: 2829
Joined: Wed Mar 08, 2006 8:57 pm
Location: Rio de Janeiro, Brazil

Re: Lczero stalled in progress?

Post by Albert Silver » Thu May 10, 2018 9:19 pm

A small update: the bug has been traced back to NN242 and is being discussed in the Issues section of the GitHub. The position is:

Image

[Event "leela bug"]
[Site "Rio de Janeiro, Brazil"]
[Date "2018.05.05"]
[Round "45"]
[White "Queen capture"]
[Black "?"]
[Result "*"]
[ECO "A29"]
[PlyCount "52"]
[EventDate "2018.05.04"]
[SourceVersionDate "2018.05.10"]

1. c4 e5 2. g3 Nf6 3. Bg2 d5 4. cxd5 Nxd5 5. Nc3 Nb6 6. Nf3 Nc6 7. a3 Be6 8. d3
Be7 9. O-O O-O 10. b4 a5 11. b5 Nd4 12. Nxe5 Bf6 13. f4 Bxe5 14. fxe5 Nb3 15.
Be3 Nxa1 16. Qxa1 Nd5 17. Bd4 Nxc3 18. Qxc3 Bd5 19. e4 Be6 20. Bc5 Re8 21. Be3
c5 22. bxc6 bxc6 23. d4 Qd7 24. d5 cxd5 25. exd5 Bxd5 26. Rd1 Bxg2 *

Obviously the move is 27. Qxd7 which Leela fails to play when given the history. If shown just the FEN, it finds it. Tests of previous neural network versions show the following probability to play Qxd7, as reported in the GitHub:

ID242: 91.7%
ID243: 17.8%
ID244: 14.39%
ID245: 0.83%

So now the developers are trying to determine where it came from, and as of what point the NN needs to be reset once fixed.
"Tactics are the bricks and sticks that make up a game, but positional play is the architectural blueprint."

Dann Corbit
Posts: 9849
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: Lczero stalled in progress?

Post by Dann Corbit » Thu May 10, 2018 10:06 pm

What is wrong is quite interesting.
More importantly, does anyone know why?
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

Robert Flesher
Posts: 1211
Joined: Tue Aug 18, 2009 1:06 am

Re: Lczero stalled in progress?

Post by Robert Flesher » Fri May 11, 2018 12:50 am

Dann Corbit wrote:
Thu May 10, 2018 10:06 pm
What is wrong is quite interesting.
More importantly, does anyone know why?
Agreed!

Milos
Posts: 3379
Joined: Wed Nov 25, 2009 12:47 am

Re: Lczero stalled in progress?

Post by Milos » Fri May 11, 2018 1:04 am

Dann Corbit wrote:
Thu May 10, 2018 10:06 pm
What is wrong is quite interesting.
More importantly, does anyone know why?
Because infinite growth is infeasible and saturation always happens sooner or later. ;)
What you are actually asking is why progress is really not in line with expectations of so many ppl, but that is kind of question for psychologist so you kind of posed it in a wrong forum ;).

Albert Silver
Posts: 2829
Joined: Wed Mar 08, 2006 8:57 pm
Location: Rio de Janeiro, Brazil

Re: Lczero stalled in progress?

Post by Albert Silver » Fri May 11, 2018 1:21 am

Milos wrote:
Fri May 11, 2018 1:04 am
Dann Corbit wrote:
Thu May 10, 2018 10:06 pm
What is wrong is quite interesting.
More importantly, does anyone know why?
Because infinite growth is infeasible and saturation always happens sooner or later. ;)
What you are actually asking is why progress is really not in line with expectations of so many ppl, but that is kind of question for psychologist so you kind of posed it in a wrong forum ;).
What do infinite growth and saturation have to do with it?
"Tactics are the bricks and sticks that make up a game, but positional play is the architectural blueprint."

syzygy
Posts: 4442
Joined: Tue Feb 28, 2012 10:56 pm

Re: Lczero stalled in progress?

Post by syzygy » Fri May 11, 2018 1:28 am

The problems started on 3 May, which was when v0.8 was released. So it seems rather likely that a severe bug was introduced in one of the about 100 commits between v0.7 and v0.8.

Since the bug has not yet been identified (as far as I can tell), it is practically excluded that v0.9 fixes it.

edit: it seems this is supposed to be a fix: https://github.com/glinscott/leela-chess/pull/577/files
However, if this fixes a bug at all, it seems to be fixing a bug that was introduced on April 8:
https://github.com/glinscott/leela-ches ... 4e15aed748
That was before v0.7 was released, so this new patch is not fixing any regression introduced between v0.7 and v0.8.

Post Reply