Can I measure improvements of the engine by having both the current & previous version of the engine play a Gauntlet against a subset of the engines listed in CCRL or a similar list , merging the resulting PGN with the pure PGN of the CCRL ?
and if this could work would it be better to calculate the ELO by merging the 2 tournament PGN with the entire pure list or with the entries of the selected subsets only ?
and do I have to use the same time control or could I use something like 1 min/all with CCRL(Pure 40/4) ?
testing the engine using CCRL pgns and Gauntlets ?
Moderators: hgm, Rebel, chrisw
-
- Posts: 234
- Joined: Sat Jan 17, 2015 11:54 pm
-
- Posts: 27817
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: testing the engine using CCRL pgns and Gauntlets ?
You can measure improvement of your engine that way already without using the CCRL games, if all versions are playing the same gauntlet. The CCRL games would only be helpful if you want to have an absolute rating.
-
- Posts: 234
- Joined: Sat Jan 17, 2015 11:54 pm
Re: testing the engine using CCRL pgns and Gauntlets ?
But I meant as a higher quality reference instead of having the testing pool play each other first in a round robin tournament ?hgm wrote:You can measure improvement of your engine that way already without using the CCRL games, if all versions are playing the same gauntlet. The CCRL games would only be helpful if you want to have an absolute rating.
I may be missing something but my understanding is that I need to choose a set of engines close to my engines rating for example {A,B,C,D} , have them play a round robin to measure their relative strength , then have each version of my engine play a gauntlet against them , So if my understanding is right can I use the matches from the CCRL of the testing set even though it was on different hardware ?
-
- Posts: 855
- Joined: Sun May 23, 2010 1:32 pm
Re: testing the engine using CCRL pgns and Gauntlets ?
I think you have to find a subset of engine in ccrl rating list that are of the same strenght of your engine.
then you do a gauntlet of your old engine against them
and then you do a gauntlet of your new development engine against them
at the end you can see whether your new engine had a better or worst result compared against the same CCRL engines
then you do a gauntlet of your old engine against them
and then you do a gauntlet of your new development engine against them
at the end you can see whether your new engine had a better or worst result compared against the same CCRL engines
-
- Posts: 27817
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: testing the engine using CCRL pgns and Gauntlets ?
I don't think it would do anything for the quality of the improvement measurement. It just improves the Elo determination of the gauntlet engines. (Which you are not interested in.)MahmoudUthman wrote:But I meant as a higher quality reference instead of having the testing pool play each other first in a round robin tournament ?