Well, for ratings lists the engine that plays it's first game for the list must be the same engine that plays game number 1,000. Otherwise things get screwed up. Of course humans learn all the time, but this is computerchess.
CCRL 40/40, 40/4 and FRC lists updated (16th February 2019)
Moderators: hgm, Rebel, chrisw
-
- Posts: 3546
- Joined: Thu Jun 07, 2012 11:02 pm
Re: CCRL 40/40, 40/4 and FRC lists updated (16th February 2019)
-
- Posts: 6991
- Joined: Thu Aug 18, 2011 12:04 pm
Re: CCRL 40/40, 40/4 and FRC lists updated (16th February 2019)
Already said the same.Modern Times wrote: ↑Thu Feb 21, 2019 6:04 amWell, for ratings lists the engine that plays it's first game for the list must be the same engine that plays game number 1,000. Otherwise things get screwed up. Of course humans learn all the time, but this is computerchess.
So why not announce a rule change in your FAQ? Now that Lc0 is on the brink to overtake the crown of Stockfish give traditional engines the means to learn in return with the above limitation and maybe a few more.
90% of coding is debugging, the other 10% is writing bugs.
-
- Posts: 27796
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: CCRL 40/40, 40/4 and FRC lists updated (16th February 2019)
Not sure what all that means. In the rebel 12 example you seem to store data on individual positions. That is by definition position learning. It is this that is forbidden in CCRL testing.Rebel wrote: ↑Thu Feb 21, 2019 5:53 amThere is much more you can do with learning than just a book in disguise, by head:hgm wrote: ↑Wed Feb 20, 2019 4:41 pm No, I don't think that the file with NN parameters can be called a 'learn file' anymore than stockfish.exe can be called a learn file. They are just eval and search parameters that happen to be located in a separate file for convenience, rather than statically linked in the executable. LC0 has not done any position learning, which is what 'learn file' traditionally refers to: just a book that contains individual position and their evaluation or recommended best move.
#1. The rebel 12 example I gave. In particular speeding up the search and often calculate one ply deeper.
#2. Evaluation of Move Sequences.
#3. Evaluation of Pawn Structures.
#4. The Rybka MIT technique.
Imagine a game that went as follows:
Move 1-10 : book
Move 11 : score +0.20
Move 15 : score +0.25
Move 20 : score -0.10
Move 25 : score -0.80
Move 30 : score -3.xx
The learning algorithm may decide to concentrate on move 15-20 and play something else.
Learning is an unexplored area and as long as it is on the boycott list of rating lists it will remain that way.
It is not forbidden to tune your eval, or distribute the engine over several files (e.g. DLLs). It is absolutely insane to demand engines playing without their eval. This is on the same level as allowing Stockfish to participate only if its .exe file is not used.
-
- Posts: 27796
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: CCRL 40/40, 40/4 and FRC lists updated (16th February 2019)
That is no rule change. It has always been allowed to train/tune engines, and submit the improved version for testing. But it would get a new version number, of course, to not confuse it with the older, less trained/tuned version, or contaminate its performance with the results of that previous one.
-
- Posts: 6991
- Joined: Thu Aug 18, 2011 12:04 pm
Re: CCRL 40/40, 40/4 and FRC lists updated (16th February 2019)
Just ask what's not clear.hgm wrote: ↑Thu Feb 21, 2019 6:24 pmNot sure what all that means.Rebel wrote: ↑Thu Feb 21, 2019 5:53 amThere is much more you can do with learning than just a book in disguise, by head:hgm wrote: ↑Wed Feb 20, 2019 4:41 pm No, I don't think that the file with NN parameters can be called a 'learn file' anymore than stockfish.exe can be called a learn file. They are just eval and search parameters that happen to be located in a separate file for convenience, rather than statically linked in the executable. LC0 has not done any position learning, which is what 'learn file' traditionally refers to: just a book that contains individual position and their evaluation or recommended best move.
#1. The rebel 12 example I gave. In particular speeding up the search and often calculate one ply deeper.
#2. Evaluation of Move Sequences.
#3. Evaluation of Pawn Structures.
#4. The Rybka MIT technique.
Imagine a game that went as follows:
Move 1-10 : book
Move 11 : score +0.20
Move 15 : score +0.25
Move 20 : score -0.10
Move 25 : score -0.80
Move 30 : score -3.xx
The learning algorithm may decide to concentrate on move 15-20 and play something else.
Learning is an unexplored area and as long as it is on the boycott list of rating lists it will remain that way.
Tell me what's so wrong on the following example:
[d]r5k1/2p1b1p1/6bp/p4P2/q3pP2/2PnB2P/PP1N4/KR3Q1R b - - bm Nb4;
Code: Select all
00:00:26 16.04 -0.52 1..Nb4
00:00:48 16.04 -0.52 1..Nb4
00:01:13 16.04 2.53 1..Nb4 2.cxb4 axb4 3.a3 Bf7 4.Qd1 b3 5.Nc4 Qxc4 6.Qc1 Qc2
Code: Select all
00:00:50 19.00 1.46 1..Nb4 2.cxb4 axb4 3.a3 Bf6 4.Bd4 Bf7 5.Bxf6
Qc2 6.Nc4 Bxc4 7.Rh2 Qb3 8.Qxc4 Qxc4 9.Rg2
bxa3 10.b3 Qd5 11.Rxg7 Kf8 12.Rxc7 Qxf5 13.Be5
Why should this be forbidden, it's blocking progress.
In the above example search and eval are in full use and regarding AZ and LZ we both know for what the "Z" stands for, don't we?
90% of coding is debugging, the other 10% is writing bugs.
-
- Posts: 3546
- Joined: Thu Jun 07, 2012 11:02 pm
Re: CCRL 40/40, 40/4 and FRC lists updated (16th February 2019)
Agreed.hgm wrote: ↑Thu Feb 21, 2019 8:23 pmThat is no rule change. It has always been allowed to train/tune engines, and submit the improved version for testing. But it would get a new version number, of course, to not confuse it with the older, less trained/tuned version, or contaminate its performance with the results of that previous one.
-
- Posts: 27796
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: CCRL 40/40, 40/4 and FRC lists updated (16th February 2019)
What is 'valuation of move sequences'? What does MIT stand for, and what does it do?
How do you expect anyone to see that from the output? It is what goes on inside that matters. So tell us what you did, and we can tell you whether it should be forbidden and why.Tell me what's so wrong on the following example:
...
Why should this be forbidden, it's blocking progress.
Yeah, and when Lance Armstrong was winning the Tour de France his muscles and lungs were also in full use. It were the other things that he used that got him into trouble. Likewise in the example above. That the engine can ride uphill faster doesn't mean it is clean...In the above example search and eval are in full use and regarding AZ and LZ we both know for what the "Z" stands for, don't we?
One of us must be missing something. The "Z" indicates the engine was not tuned on any games that it did not generate itself. Why do you think that is funny?
-
- Posts: 6991
- Joined: Thu Aug 18, 2011 12:04 pm
Re: CCRL 40/40, 40/4 and FRC lists updated (16th February 2019)
Under the current rules LZ has an unfair advantage.Modern Times wrote: ↑Thu Feb 21, 2019 10:22 pmAgreed.hgm wrote: ↑Thu Feb 21, 2019 8:23 pmThat is no rule change. It has always been allowed to train/tune engines, and submit the improved version for testing. But it would get a new version number, of course, to not confuse it with the older, less trained/tuned version, or contaminate its performance with the results of that previous one.
90% of coding is debugging, the other 10% is writing bugs.
-
- Posts: 27796
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: CCRL 40/40, 40/4 and FRC lists updated (16th February 2019)
So you say, but so far you have not offered a single valid argument why that would be. The argument that its evaluation resides in a different file doesn't impress anyone.
In fact the current CCRL rules give LC0 a tremendously unfair disadvantage, because the standard hardware does not include a GPU.
In fact the current CCRL rules give LC0 a tremendously unfair disadvantage, because the standard hardware does not include a GPU.
-
- Posts: 3546
- Joined: Thu Jun 07, 2012 11:02 pm
Re: CCRL 40/40, 40/4 and FRC lists updated (16th February 2019)
As was pointed out to us, we say we are a standard hardware list but we are not. We have 1CPU and 4CPU engines on there and they play each other to some extent so that the list is joined up. So a "GPU" could be just another variation of that. So the list would be "1CPU", "4CPU" and "GPU". I think CEGT have in fact set aside a couple of standard reference machines for GPU testing for a 2 year minimum period. From their postings, I think the time control for matches is set in the normal way based on the speed of the CPU, and then the GPU is what it is as the standard reference GPU. That seems completely clear and transparent to me.