CCRL 40/40, 40/4 and FRC lists updated (16th February 2019)

Discussion of computer chess matches and engine tournaments.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Modern Times
Posts: 2420
Joined: Thu Jun 07, 2012 9:02 pm

Re: CCRL 40/40, 40/4 and FRC lists updated (16th February 2019)

Post by Modern Times » Thu Feb 21, 2019 5:04 am

Rebel wrote:
Thu Feb 21, 2019 4:53 am

Learning is an unexplored area and as long as it is on the boycott list of rating lists it will remain that way.
Well, for ratings lists the engine that plays it's first game for the list must be the same engine that plays game number 1,000. Otherwise things get screwed up. Of course humans learn all the time, but this is computerchess.
.

Opinions expressed here are my own, and not necessarily those of the CCRL Group.

User avatar
Rebel
Posts: 4788
Joined: Thu Aug 18, 2011 10:04 am

Re: CCRL 40/40, 40/4 and FRC lists updated (16th February 2019)

Post by Rebel » Thu Feb 21, 2019 8:37 am

Modern Times wrote:
Thu Feb 21, 2019 5:04 am
Rebel wrote:
Thu Feb 21, 2019 4:53 am

Learning is an unexplored area and as long as it is on the boycott list of rating lists it will remain that way.
Well, for ratings lists the engine that plays it's first game for the list must be the same engine that plays game number 1,000. Otherwise things get screwed up. Of course humans learn all the time, but this is computerchess.
Already said the same.

So why not announce a rule change in your FAQ? Now that Lc0 is on the brink to overtake the crown of Stockfish give traditional engines the means to learn in return with the above limitation and maybe a few more.
90% of coding is debugging, the other 10% is writing bugs.

User avatar
hgm
Posts: 23790
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: CCRL 40/40, 40/4 and FRC lists updated (16th February 2019)

Post by hgm » Thu Feb 21, 2019 5:24 pm

Rebel wrote:
Thu Feb 21, 2019 4:53 am
hgm wrote:
Wed Feb 20, 2019 3:41 pm
No, I don't think that the file with NN parameters can be called a 'learn file' anymore than stockfish.exe can be called a learn file. They are just eval and search parameters that happen to be located in a separate file for convenience, rather than statically linked in the executable. LC0 has not done any position learning, which is what 'learn file' traditionally refers to: just a book that contains individual position and their evaluation or recommended best move.
There is much more you can do with learning than just a book in disguise, by head:

#1. The rebel 12 example I gave. In particular speeding up the search and often calculate one ply deeper.

#2. Evaluation of Move Sequences.

#3. Evaluation of Pawn Structures.

#4. The Rybka MIT technique.

Imagine a game that went as follows:

Move 1-10 : book
Move 11 : score +0.20
Move 15 : score +0.25
Move 20 : score -0.10
Move 25 : score -0.80
Move 30 : score -3.xx

The learning algorithm may decide to concentrate on move 15-20 and play something else.

Learning is an unexplored area and as long as it is on the boycott list of rating lists it will remain that way.
Not sure what all that means. In the rebel 12 example you seem to store data on individual positions. That is by definition position learning. It is this that is forbidden in CCRL testing.

It is not forbidden to tune your eval, or distribute the engine over several files (e.g. DLLs). It is absolutely insane to demand engines playing without their eval. This is on the same level as allowing Stockfish to participate only if its .exe file is not used.

User avatar
hgm
Posts: 23790
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: CCRL 40/40, 40/4 and FRC lists updated (16th February 2019)

Post by hgm » Thu Feb 21, 2019 7:23 pm

Rebel wrote:
Thu Feb 21, 2019 8:37 am
So why not announce a rule change in your FAQ? Now that Lc0 is on the brink to overtake the crown of Stockfish give traditional engines the means to learn in return with the above limitation and maybe a few more.
That is no rule change. It has always been allowed to train/tune engines, and submit the improved version for testing. But it would get a new version number, of course, to not confuse it with the older, less trained/tuned version, or contaminate its performance with the results of that previous one.

User avatar
Rebel
Posts: 4788
Joined: Thu Aug 18, 2011 10:04 am

Re: CCRL 40/40, 40/4 and FRC lists updated (16th February 2019)

Post by Rebel » Thu Feb 21, 2019 8:27 pm

hgm wrote:
Thu Feb 21, 2019 5:24 pm
Rebel wrote:
Thu Feb 21, 2019 4:53 am
hgm wrote:
Wed Feb 20, 2019 3:41 pm
No, I don't think that the file with NN parameters can be called a 'learn file' anymore than stockfish.exe can be called a learn file. They are just eval and search parameters that happen to be located in a separate file for convenience, rather than statically linked in the executable. LC0 has not done any position learning, which is what 'learn file' traditionally refers to: just a book that contains individual position and their evaluation or recommended best move.
There is much more you can do with learning than just a book in disguise, by head:

#1. The rebel 12 example I gave. In particular speeding up the search and often calculate one ply deeper.

#2. Evaluation of Move Sequences.

#3. Evaluation of Pawn Structures.

#4. The Rybka MIT technique.

Imagine a game that went as follows:

Move 1-10 : book
Move 11 : score +0.20
Move 15 : score +0.25
Move 20 : score -0.10
Move 25 : score -0.80
Move 30 : score -3.xx

The learning algorithm may decide to concentrate on move 15-20 and play something else.

Learning is an unexplored area and as long as it is on the boycott list of rating lists it will remain that way.
Not sure what all that means.
Just ask what's not clear.
hgm wrote:
Wed Feb 20, 2019 3:41 pm
In the rebel 12 example you seem to store data on individual positions. That is by definition position learning. It is this that is forbidden in CCRL testing.
Tell me what's so wrong on the following example:

Code: Select all

00:00:26  16.04  -0.52  1..Nb4 
00:00:48  16.04  -0.52  1..Nb4 
00:01:13  16.04   2.53  1..Nb4 2.cxb4 axb4 3.a3 Bf7 4.Qd1 b3 5.Nc4 Qxc4 6.Qc1 Qc2 
and via the learn file:

Code: Select all

00:00:50  19.00  1.46  1..Nb4 2.cxb4 axb4 3.a3 Bf6 4.Bd4 Bf7 5.Bxf6 
                          Qc2 6.Nc4 Bxc4 7.Rh2 Qb3 8.Qxc4 Qxc4 9.Rg2 
                          bxa3 10.b3 Qd5 11.Rxg7 Kf8 12.Rxc7 Qxf5 13.Be5 
3 plies deeper.

Why should this be forbidden, it's blocking progress.

hgm wrote:
Wed Feb 20, 2019 3:41 pm
It is not forbidden to tune your eval, or distribute the engine over several files (e.g. DLLs). It is absolutely insane to demand engines playing without their eval. This is on the same level as allowing Stockfish to participate only if its .exe file is not used.
In the above example search and eval are in full use and regarding AZ and LZ we both know for what the "Z" stands for, don't we? :lol:
90% of coding is debugging, the other 10% is writing bugs.

Modern Times
Posts: 2420
Joined: Thu Jun 07, 2012 9:02 pm

Re: CCRL 40/40, 40/4 and FRC lists updated (16th February 2019)

Post by Modern Times » Thu Feb 21, 2019 9:22 pm

hgm wrote:
Thu Feb 21, 2019 7:23 pm
Rebel wrote:
Thu Feb 21, 2019 8:37 am
So why not announce a rule change in your FAQ? Now that Lc0 is on the brink to overtake the crown of Stockfish give traditional engines the means to learn in return with the above limitation and maybe a few more.
That is no rule change. It has always been allowed to train/tune engines, and submit the improved version for testing. But it would get a new version number, of course, to not confuse it with the older, less trained/tuned version, or contaminate its performance with the results of that previous one.
Agreed.
.

Opinions expressed here are my own, and not necessarily those of the CCRL Group.

User avatar
hgm
Posts: 23790
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: CCRL 40/40, 40/4 and FRC lists updated (16th February 2019)

Post by hgm » Thu Feb 21, 2019 9:57 pm

Rebel wrote:
Thu Feb 21, 2019 8:27 pm
Just ask what's not clear.
What is 'valuation of move sequences'? What does MIT stand for, and what does it do?


Tell me what's so wrong on the following example:
...
Why should this be forbidden, it's blocking progress.
How do you expect anyone to see that from the output? It is what goes on inside that matters. So tell us what you did, and we can tell you whether it should be forbidden and why.


In the above example search and eval are in full use and regarding AZ and LZ we both know for what the "Z" stands for, don't we? :lol:
Yeah, and when Lance Armstrong was winning the Tour de France his muscles and lungs were also in full use. It were the other things that he used that got him into trouble. Likewise in the example above. That the engine can ride uphill faster doesn't mean it is clean...

One of us must be missing something. The "Z" indicates the engine was not tuned on any games that it did not generate itself. Why do you think that is funny?

User avatar
Rebel
Posts: 4788
Joined: Thu Aug 18, 2011 10:04 am

Re: CCRL 40/40, 40/4 and FRC lists updated (16th February 2019)

Post by Rebel » Thu Feb 21, 2019 10:00 pm

Modern Times wrote:
Thu Feb 21, 2019 9:22 pm
hgm wrote:
Thu Feb 21, 2019 7:23 pm
Rebel wrote:
Thu Feb 21, 2019 8:37 am
So why not announce a rule change in your FAQ? Now that Lc0 is on the brink to overtake the crown of Stockfish give traditional engines the means to learn in return with the above limitation and maybe a few more.
That is no rule change. It has always been allowed to train/tune engines, and submit the improved version for testing. But it would get a new version number, of course, to not confuse it with the older, less trained/tuned version, or contaminate its performance with the results of that previous one.
Agreed.
Under the current rules LZ has an unfair advantage.
90% of coding is debugging, the other 10% is writing bugs.

User avatar
hgm
Posts: 23790
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: CCRL 40/40, 40/4 and FRC lists updated (16th February 2019)

Post by hgm » Thu Feb 21, 2019 10:22 pm

So you say, but so far you have not offered a single valid argument why that would be. The argument that its evaluation resides in a different file doesn't impress anyone.

In fact the current CCRL rules give LC0 a tremendously unfair disadvantage, because the standard hardware does not include a GPU.

Modern Times
Posts: 2420
Joined: Thu Jun 07, 2012 9:02 pm

Re: CCRL 40/40, 40/4 and FRC lists updated (16th February 2019)

Post by Modern Times » Fri Feb 22, 2019 5:02 am

hgm wrote:
Thu Feb 21, 2019 10:22 pm

In fact the current CCRL rules give LC0 a tremendously unfair disadvantage, because the standard hardware does not include a GPU.
As was pointed out to us, we say we are a standard hardware list but we are not. We have 1CPU and 4CPU engines on there and they play each other to some extent so that the list is joined up. So a "GPU" could be just another variation of that. So the list would be "1CPU", "4CPU" and "GPU". I think CEGT have in fact set aside a couple of standard reference machines for GPU testing for a 2 year minimum period. From their postings, I think the time control for matches is set in the normal way based on the speed of the CPU, and then the GPU is what it is as the standard reference GPU. That seems completely clear and transparent to me.
.

Opinions expressed here are my own, and not necessarily those of the CCRL Group.

Post Reply