20x256 Leela seemed quite strong

[Dann Corbit]

Well, I think it was Dann who commented that the secret to Fruit's success was the sheer lack of bugs, proving it was more important than having clever ideas for algorithms. As such, in Discord today a rather glaring issue came up that is now the source of a serious hunt. Somehow, something happened precisely during the transition to 15x192.

The question is how Fruit got rid of bugs. What Leela needs is clever ideas for bug hunting.

Fabian is a true master of the assert();

[Milos]

Well, I think it was Dann who commented that the secret to Fruit's success was the sheer lack of bugs, proving it was more important than having clever ideas for algorithms. As such, in Discord today a rather glaring issue came up that is now the source of a serious hunt. Somehow, something happened precisely during the transition to 15x192.

The question is how Fruit got rid of bugs. What Leela needs is clever ideas for bug hunting.

Fabian is a true master of the assert();

You can't assert a ResNet. But keep dreaming on.

[Albert Silver]

The question is how Fruit got rid of bugs. What Leela needs is clever ideas for bug hunting.

Fabian is a true master of the assert();

You can't assert a ResNet. But keep dreaming on.

He was answering the question on how Fruit got rid of bugs, or avoided them.

[Milos]

Fabian is a true master of the assert();

You can't assert a ResNet. But keep dreaming on.

He was answering the question on how Fruit got rid of bugs, or avoided them.

That is true, but implication was (though not directly by Dann) that the similar approach could be used with LC0.

[Sven]

You can't assert a ResNet. But keep dreaming on.

He was answering the question on how Fruit got rid of bugs, or avoided them.

That is true, but implication was (though not directly by Dann) that the similar approach could be used with LC0.

Using assert() is always possible for the engine code itself (as far as I know they do not use it). Regarding the network I am not sure how this could be done, simply due to lack of sufficient NN knowledge. So let me ask:

1) Do you think that it would be doable and realistic to write some consistency checks for a network, in order to detect "illegal states"?

2) Do you think that it would be doable and realistic to formulate some "quality criteria" for a network that help to identify valid but undesired states, and write functions that calculate such quality scores?

If so, one could write a "network checker" that looks like this:

Code: Select all

assert(consistencyCheck01());
assert(consistencyCheck02());
...
cout << "quality score 01: " << qualityScore01() << endl;
cout << "quality score 02: " << qualityScore02() << endl;

[Albert Silver]

He was answering the question on how Fruit got rid of bugs, or avoided them.

That is true, but implication was (though not directly by Dann) that the similar approach could be used with LC0.

Using assert() is always possible for the engine code itself (as far as I know they do not use it). Regarding the network I am not sure how this could be done, simply due to lack of sufficient NN knowledge. So let me ask:

1) Do you think that it would be doable and realistic to write some consistency checks for a network, in order to detect "illegal states"?

2) Do you think that it would be doable and realistic to formulate some "quality criteria" for a network that help to identify valid but undesired states, and write functions that calculate such quality scores?

If so, one could write a "network checker" that looks like this:

Code: Select all

assert(consistencyCheck01());
assert(consistencyCheck02());
...
cout << "quality score 01: " << qualityScore01() << endl;
cout << "quality score 02: " << qualityScore02() << endl;

The entire project is open source so anyone can try their hand at modifications, and if you think it works in tests, submit it as a patch.

[Sven]

That is true, but implication was (though not directly by Dann) that the similar approach could be used with LC0.

Using assert() is always possible for the engine code itself (as far as I know they do not use it). Regarding the network I am not sure how this could be done, simply due to lack of sufficient NN knowledge. So let me ask:

1) Do you think that it would be doable and realistic to write some consistency checks for a network, in order to detect "illegal states"?

2) Do you think that it would be doable and realistic to formulate some "quality criteria" for a network that help to identify valid but undesired states, and write functions that calculate such quality scores?

If so, one could write a "network checker" that looks like this:

Code: Select all

assert(consistencyCheck01());
assert(consistencyCheck02());
...
cout << "quality score 01: " << qualityScore01() << endl;
cout << "quality score 02: " << qualityScore02() << endl;

The entire project is open source so anyone can try their hand at modifications, and if you think it works in tests, submit it as a patch.

You are certainly right. But my question to Milos was not meant as an attempt to modify the source but as a thought about something new that (to my knowledge) does not exist yet (a "network checker" as a separate tool). It was also mainly a reaction on the statement that using "assert" were not applicable in this project.

Of course that could be added in the usual way via Github. Before starting to investigate into such a project (for which I am not even sure I would have time for) I thought it would be best to ask someone who knows a lot more about NN than I do. Asking should be allowed

[Milos]

He was answering the question on how Fruit got rid of bugs, or avoided them.

That is true, but implication was (though not directly by Dann) that the similar approach could be used with LC0.

Using assert() is always possible for the engine code itself (as far as I know they do not use it). Regarding the network I am not sure how this could be done, simply due to lack of sufficient NN knowledge. So let me ask:

1) Do you think that it would be doable and realistic to write some consistency checks for a network, in order to detect "illegal states"?

2) Do you think that it would be doable and realistic to formulate some "quality criteria" for a network that help to identify valid but undesired states, and write functions that calculate such quality scores?

If so, one could write a "network checker" that looks like this:

Code: Select all

assert(consistencyCheck01());
assert(consistencyCheck02());
...
cout << "quality score 01: " << qualityScore01() << endl;
cout << "quality score 02: " << qualityScore02() << endl;

To my knowledge the only type of checks you could do is at output layer in the fully connected layer. You can just forget about convolutional layers. I don't think anyone knows you to check anything there even for far simpler CNNs and simple classification problems such as recognizing digits or even binary classification.