Lczero stalled in progress?

[Robert Flesher]

I am just curious if the experts can explain why Lczero seems to have stalled in progress?

[Ozymandias]

I am just curious if the experts can explain why Lczero seems to have stalled in progress?

When they say at the beginning of their main page, that "We are actively investigating the strength fluctuations, and improving the system. v0.9 helps here.", I'm guessing they can't... yet.

[noobpwnftw]

Where are those people who think over-fitting can be "easily solved"?

The way I see this: probably the networks were over-fitted all along, a minor change to learning rates and more noise from resigning just made it obvious. There is an alternative method to prove, if it truly understands chess, then you are free to half or double the number of playouts during training and it should result in no regression.

Unfortunately their bible has no chapter on this matter, so it should "just work".

[Albert Silver]

I am just curious if the experts can explain why Lczero seems to have stalled in progress?

A massive bug was revealed today, which has also affected the training. Leela will literally refuse to capture a hanging queen:



https://groups.google.com/forum/#!topic ... iOlKkuhCGw

[Albert Silver]

A small update: the bug has been traced back to NN242 and is being discussed in the Issues section of the GitHub. The position is:



[Event "leela bug"]
[Site "Rio de Janeiro, Brazil"]
[Date "2018.05.05"]
[Round "45"]
[White "Queen capture"]
[Black "?"]
[Result "*"]
[ECO "A29"]
[PlyCount "52"]
[EventDate "2018.05.04"]
[SourceVersionDate "2018.05.10"]

1. c4 e5 2. g3 Nf6 3. Bg2 d5 4. cxd5 Nxd5 5. Nc3 Nb6 6. Nf3 Nc6 7. a3 Be6 8. d3
Be7 9. O-O O-O 10. b4 a5 11. b5 Nd4 12. Nxe5 Bf6 13. f4 Bxe5 14. fxe5 Nb3 15.
Be3 Nxa1 16. Qxa1 Nd5 17. Bd4 Nxc3 18. Qxc3 Bd5 19. e4 Be6 20. Bc5 Re8 21. Be3
c5 22. bxc6 bxc6 23. d4 Qd7 24. d5 cxd5 25. exd5 Bxd5 26. Rd1 Bxg2 *

Obviously the move is 27. Qxd7 which Leela fails to play when given the history. If shown just the FEN, it finds it. Tests of previous neural network versions show the following probability to play Qxd7, as reported in the GitHub:

ID242: 91.7%
ID243: 17.8%
ID244: 14.39%
ID245: 0.83%

So now the developers are trying to determine where it came from, and as of what point the NN needs to be reset once fixed.

[Dann Corbit]

What is wrong is quite interesting.
More importantly, does anyone know why?

[Robert Flesher]

What is wrong is quite interesting.
More importantly, does anyone know why?

Agreed!

[Milos]

What is wrong is quite interesting.
More importantly, does anyone know why?

Because infinite growth is infeasible and saturation always happens sooner or later.
What you are actually asking is why progress is really not in line with expectations of so many ppl, but that is kind of question for psychologist so you kind of posed it in a wrong forum .

[Albert Silver]

What is wrong is quite interesting.
More importantly, does anyone know why?

Because infinite growth is infeasible and saturation always happens sooner or later.
What you are actually asking is why progress is really not in line with expectations of so many ppl, but that is kind of question for psychologist so you kind of posed it in a wrong forum .

What do infinite growth and saturation have to do with it?

[syzygy]

The problems started on 3 May, which was when v0.8 was released. So it seems rather likely that a severe bug was introduced in one of the about 100 commits between v0.7 and v0.8.

Since the bug has not yet been identified (as far as I can tell), it is practically excluded that v0.9 fixes it.

edit: it seems this is supposed to be a fix: https://github.com/glinscott/leela-chess/pull/577/files
However, if this fixes a bug at all, it seems to be fixing a bug that was introduced on April 8:
https://github.com/glinscott/leela-ches ... 4e15aed748
That was before v0.7 was released, so this new patch is not fixing any regression introduced between v0.7 and v0.8.