Bad day at the stock market?

[syzygy]

The current graph is a projection of Malthus' Dismal Theory.
Time to admit that something got broken, backing up to a version that is not broken, and restarting.
Or, perhaps, this is just an imitation of nature that is too realistic, and the food supply is running out.

I basically agree. But my strategy would still involve to isolate the point where it got broken. Throwing away some hundreds of source code commits would not only be a waste but might also decrease motivation for the developers. So I guess the primary goal should be to find and fix the bugs, assuming that bugs are the main reason for the current regression (which appears to be most likely for me).

What I find puzzling is that neither on the LC Zero forum nor on github there is much if any sense of "how are we going to fight the bugs". The mood seems to be "what goes down will go up again" as if LC Zero could fix the bugs itself.

What puzzles me is that according to the graph the regression did not start within 24 hours after a new release but more than three days after the release of v0.8. Network IDs 255, 256, 257 show the first symptoms of regression and were created on May 6th in the evening while v0.8 was released on May 3rd early in the morning. How is that possible? Contributing users play >100,000 games per day so there should be sufficient significance to rule out any weird statistical explanations.

I don't know if this is the answer, but it seems not all contributing users upgrade their clients immediately when a new version comes out.

The "Rxd7" bug started to show up around ID242 and probably earlier:
https://github.com/glinscott/leela-ches ... -388169418
This seems to coincide with v0.8.
On the other hand:
- this was fixed by changing code that had been committed on April 7 or so;
- it seems fixing it did not actually resolve LC's problems.

So this is again puzzling. If the bug was there since April 7, why did it not show up earlier? Was it triggered by some other change that was in v0.8 but not in v0.7?

Questions like this should get an answer. (Maybe it did get an answer, but I haven't seen it in any of the github threads.) Bugs don't just disappear by themselves but can be fixed only after having been understood.

Another thing I am wondering about: the sudden jump from ID226 to ID227. Larger network, they say, but does that actually explain the jump? I don't see how a larger network can lead to better play all by itself. It will first have to be trained. If the new larger network did not play better moves, then it seems the jump can only be explained by a calibration error. What was the larger network tested against? Is any of this fully understood by at least the core developers?

I still do not know enough about the project so maybe I misunderstand some of its concepts.

Same here

[Michel]

Contributing users play >100,000 games per day so there should be sufficient significance to rule out any weird statistical explanations.

I _think_ the training window is 500,000 games. So it may take a while before the effect of corrupted training data become visible.

[Michel]

I don't see how a larger network can lead to better play all by itself. It will first have to be trained. I

I think it was trained for a while (using the existing training data) before it was released.

[syzygy]

I don't see how a larger network can lead to better play all by itself. It will first have to be trained. I

I think it was trained for a while (using the existing training data) before it was released.

I see!

[Henk]

Probably they don't just don't add a block to the tower and retrain each and every time. For that task is similar and they mention a bug. For how could a bug appear if you follow the same procedure.

[Sven]

Contributing users play >100,000 games per day so there should be sufficient significance to rule out any weird statistical explanations.

I _think_ the training window is 500,000 games. So it may take a while before the effect of corrupted training data become visible.

On the list of existing networks I see new network IDs appearing roughly every 5 hours with about 30,000..40,000 new games per ID. Is that the number of training games or match games?

[Michel]

Contributing users play >100,000 games per day so there should be sufficient significance to rule out any weird statistical explanations.

I _think_ the training window is 500,000 games. So it may take a while before the effect of corrupted training data become visible.

On the list of existing networks I see new network IDs appearing roughly every 5 hours with about 30,000..40,000 new games per ID. Is that the number of training games or match games?

I am pretty sure it is training games. As far as I understand they are contributed to the window (but I do not understand things all that well).

The matches are only some 800 games (and the results are currently only used to make the graph).

[Sven]

I _think_ the training window is 500,000 games. So it may take a while before the effect of corrupted training data become visible.

On the list of existing networks I see new network IDs appearing roughly every 5 hours with about 30,000..40,000 new games per ID. Is that the number of training games or match games?

I am pretty sure it is training games. As far as I understand they are contributed to the window (but I do not understand things all that well).

The matches are only some 800 games (and the results are currently only used to make the graph).

Sounds right, matches are listed here.

[can00336]

What I find puzzling is that neither on the LC Zero forum nor on github there is much if any sense of "how are we going to fight the bugs". The mood seems to be "what goes down will go up again" as if LC Zero could fix the bugs itself.

You are probably confused since 90% of the project's discussion happens on our Discord server. Come join us here: https://discord.gg/pKujYxD

[syzygy]

What I find puzzling is that neither on the LC Zero forum nor on github there is much if any sense of "how are we going to fight the bugs". The mood seems to be "what goes down will go up again" as if LC Zero could fix the bugs itself.

You are probably confused since 90% of the project's discussion happens on our Discord server. Come join us here: https://discord.gg/pKujYxD

Yes, that is probably the reason.
Unfortunately I don't feel like signing up to Discord (whatever that might be exactly).