In defense of Pedro

[Rebel]

I do not know who imposed the idea that each engine should use its own evaluations to create the network and could not use evaluations from other engines, this was imposed as a must. Who decided it and why? If someone wants to organize a tournament under those circumstances I am fine with it. But I don't feel the need for someone to tell me how to make my engine.

Months ago I created a programmer poll on this issue and the feedback was disappointing, only 8 programmers responded to the 3 choices.

1. My preference is that I am allowed to use any existing network for my engine. [2 votes]
2. My preference is that I am allowed to use any game collection to create my own network. [4 votes]
3. My preference is that I am in favor that any programmer creates his own network either from human game collections or from his own engine. [2 votes]

While I personally favor option [3] I consider it unfair to starting programmers who want to do machine-learning and are forced to write a good HCE evaluation first and so my vote goes to [2], making it 5.

I heard opinions you don't need a good HCE evaluation for NNUE development, I invite them to read my experiments with 2 engines, Rodent giving 300 elo on a small NN of 400M positions and Crafty giving nothing. Crafty by the admission of the author himself saying his HCE is not so good. For NNUE development one needs a decent HCE. The links -
https://prodeo.actieforum.com/t728-rode ... evelopment
https://prodeo.actieforum.com/t764-craf ... evelopment

Remark - Regarding the importance of this issue it was disappointing to see only one author of the top-20 engines responded to the poll, is the issue too touchy?

[Modern Times]

Stockfish itself follows [2] - they use Lc0 games as well as their own to build their NNUE

So the idea that each engine should use ONLY its own evaluations to create the network and could not use evaluations from other engines - well it can't be one rule for Stockfish and one rule for everyone else.

[Frank Quisinsky]

Hi Ed,

Rodent with Stockfish and with Rebel net ...

Code: Select all

--. Rodent 1.0 NNSf             :  3250.14   1740    54.7   396  1111   233   951.5   63.9  10.23  3214.93  10.07   58.0
--. Rodent 1.1 NNRe             :  3073.46   1410    32.0   124   654   632   451.0   46.4  12.50  3223.91   9.62   47.0

Very aggressive with SF net.
In case of Rodent 1.0 / 1.1 ... a complete other playigng style.

Point 3:
I think with games from the own engines the style of engine is more in foreground.
The style of engines is for me most important. The features of engines as second.

And with a mix?
Can be point 4 in your poll.

75% games from the own engine and ...
25% other games

In my humble opinion more originality with an own net with games from the own engine. I can understand programmers working now around 2 years with a lot of energy on an own net. But ... in the long term different possibilities are conceivable.

To train an own net again and again is a lot of work, need many time and we know ... time can be use for other things?

I see a closer connection between "programmers" and "users".
A new job / order for users can be to create the net.

Example:
User "Hannibal Talk from Edland" is a specialist for strong nets.
Hannibal is working on different strong nets for programmers.

With a free protocol programmers can use the net / work from Hannibal.

Hannibal gave an exactly instruction / documentation from the work he do.

GUI programmers read that and thinking ... very nice, I can create new GUI features.
Net can be created from my GUI with interesting features.

That is an interesting way!
Programmers can create an own net or give it as "order" to others!

Best
Frank

An other very important point is:
We should thinking on further processing with our game material.
Should thinking on ressources ... fast mating, avaid 50-moves-rules with faster draws and to use things we have.

Point 2 isn't uninteresting!

[Uri Blass]

I do not know who imposed the idea that each engine should use its own evaluations to create the network and could not use evaluations from other engines, this was imposed as a must. Who decided it and why? If someone wants to organize a tournament under those circumstances I am fine with it. But I don't feel the need for someone to tell me how to make my engine.

Months ago I created a programmer poll on this issue and the feedback was disappointing, only 8 programmers responded to the 3 choices.

1. My preference is that I am allowed to use any existing network for my engine. [2 votes]
2. My preference is that I am allowed to use any game collection to create my own network. [4 votes]
3. My preference is that I am in favor that any programmer creates his own network either from human game collections or from his own engine. [2 votes]

While I personally favor option [3] I consider it unfair to starting programmers who want to do machine-learning and are forced to write a good HCE evaluation first and so my vote goes to [2], making it 5.

I heard opinions you don't need a good HCE evaluation for NNUE development, I invite them to read my experiments with 2 engines, Rodent giving 300 elo on a small NN of 400M positions and Crafty giving nothing. Crafty by the admission of the author himself saying his HCE is not so good. For NNUE development one needs a decent HCE. The links -
https://prodeo.actieforum.com/t728-rode ... evelopment
https://prodeo.actieforum.com/t764-craf ... evelopment

Remark - Regarding the importance of this issue it was disappointing to see only one author of the top-20 engines responded to the poll, is the issue too touchy?

I do not understand much about the reason for the failure of Crafty NNUE developement but
the question is if the reason for the failure of Crafty is the evaluation.

Reason may be also the search.
I wonder if somebody tried to use the same search with 2 different evaluations and found no improvmenent for evaluation A and 300 elo improvement for evaluation B.

[hgm]

It is always up to the organizer of a tournament to specify what he will allow and what not.

How would you like to enforce any such rules anyway? If people train their NNUE on a set of positions sampled from games, even requiring them to submit the set of training positions might not enable you to figure out where they came from. It is in general silly to specify rules that can in no way be enforced, as it is unavoidable that some people will just ignore them.

BTW, tuning of a HCE based on the evaluations of a top engine has been a popular passtime for more than a decade.

[Steve Maughan]

It is always up to the organizer of a tournament to specify what he will allow and what not.

How would you like to enforce any such rules anyway? If people train their NNUE on a set of positions sampled from games, even requiring them to submit the set of training positions might not enable you to figure out where they came from. It is in general silly to specify rules that can in no way be enforced, as it is unavoidable that some people will just ignore them.

BTW, tuning of a HCE based on the evaluations of a top engine has been a popular passtime for more than a decade.

Wise words from H.G.

I personally think copying code is wrong, and copying a net is wrong. These two can be verified relatively easily. Anything else is ok — it’s standing on the shoulders of giants, which we’ve been doing since the beginning.

To expand on this, suppose I compare the evaluation of my engine with that of Stockfish for a number of games, see where it’s radically different and then add terms or tune the parameters accordingly. How’s this philosophically different from training a neural network using output from Stockfish?

Steve

[smatovic]

It is always up to the organizer of a tournament to specify what he will allow and what not.
[...]

+1

Just stick to the licenses for data and code as programmer, the rest is up to the tournament/rating list organizers.

--
Srdja

[dkappe]

It is always up to the organizer of a tournament to specify what he will allow and what not.
[...]

+1

Just stick to the licenses for data and code as programmer, the rest is up to the tournament/rating list organizers.

--
Srdja

Easily done. If you create a network by training with lc0 data, you simply have to give credit. You can even distribute your network commercially if you wish.

https://opendatacommons.org/licenses/odbl/summary/

[Rebel]

Stockfish itself follows [2] - they use Lc0 games as well as their own to build their NNUE

So the idea that each engine should use ONLY its own evaluations to create the network and could not use evaluations from other engines - well it can't be one rule for Stockfish and one rule for everyone else.

Agree.

[Rebel]

It is always up to the organizer of a tournament to specify what he will allow and what not.

Agree.

How would you like to enforce any such rules anyway?

Enforcing is impossible but transparency is a good start.