CCRL update (6th November 2009)

[Graham Banks]

ok, since version 3.0 they are was buggy then fixed and updated quickly, but version 3.21a are stable i think or?.

how much games need to be established in your list?

is 100 games are not enough or even 50 ?

An engine must have at least 200 games in order to become established in our lists.

Cheers,
Graham.

I think 200 games is fine. But I also think there should be a minimum number of different opponents each with a minimum number of games.

200 games against 1 opponent just doesn't seem as meaningful as 200 games total where there are 20 opponents, each with a minimum of 4 games.

Agreed in principle.

[Engin]

hmmm, the rating list is become confuse with all 1-4 cpus and different version together in one pot.

can we exclude some engine like only 1cpu list engines or only best version and 1 cpu, and how do you will compare some engines are SMP but you dont using all possible cpus, reserved only for strongest top 20 engines ?

Tornado can use also 2, 4 cpus too !

[Dr.Wael Deeb]

hmmm, the rating list is become confuse with all 1-4 cpus and different version together in one pot.

can we exclude some engine like only 1cpu list engines or only best version and 1 cpu, and how do you will compare some engines are SMP but you dont using all possible cpus, reserved only for strongest top 20 engines ?

Tornado can use also 2, 4 cpus too !

And that's why in my rating list I test the engines with their biggest guns....x64 if available,4CPU for MP chess engines....
Why would I waste my time on testing a MP chess engine on 2CPU,then after that at 1 CPU....kinda of boring and wasting the hardware resources....I can understand that some people would like to know for example what is the strongest SP chess engine,well,I don't....
As I said,come with your biggest guns and let's fight
But hey,this is my way of seeing things....
Dr.D

[Graham Banks]

hmmm, the rating list is become confuse with all 1-4 cpus and different version together in one pot.

can we exclude some engine like only 1cpu list engines or only best version and 1 cpu, and how do you will compare some engines are SMP but you dont using all possible cpus, reserved only for strongest top 20 engines ?

Tornado can use also 2, 4 cpus too !

You can already select such rating lists. You can also construct customised lists to suit.

Cheers,
Graham.

[gerold]

hmmm, the rating list is become confuse with all 1-4 cpus and different version together in one pot.

can we exclude some engine like only 1cpu list engines or only best version and 1 cpu, and how do you will compare some engines are SMP but you dont using all possible cpus, reserved only for strongest top 20 engines ?

Tornado can use also 2, 4 cpus too !

Engin see the post above by Kirill Kryukow.

That may help you.

[rvida]

Hi Engin,

I'm not sure how many of the versions since Tornado 2.2 play FRC, but we started testing the following versions for the 40/40 list. Many of them had to be "killed" because a new version came out.

Tornado 2.2 (389 games - established on list)
Tornado 3.0 (26 games - killed))
Tornado 3.02 (76 games - killed)
Tornado 3.12 (50 games - killed)
Tornado 3.16 (81 games - killed)
Tornado 3.19a (79 games - killed)
Tornado 3.21a (149 games - will continue to get it established)

Not a criticism, but just explaining it from a tester's point of view.
Keep up the good work with Tornado.

Cheers,
Graham.

Engin, (and all engine authors)

I very much appreciate all the effort, time and computing power that all testers at CCRL / CEGT spend testing our engines. I think that "we" - engine authors - owe them some respect and should give them enough time to test our engines and not wreck their hard work with too frequent releases. Bugfixes (like when engine xx crashes under GUI yy) are O.K. But 3 versions per month are just overkill IMO. It is better to make a new release only once per quarter, or when there is significant (at least 25-30 ELO) improvement. Whichever comes first. Of course the latter is better

just my 2 eurocents

[Graham Banks]

Engin, (and all engine authors)

I very much appreciate all the effort, time and computing power that all testers at CCRL / CEGT spend testing our engines. I think that "we" - engine authors - owe them some respect and should give them enough time to test our engines and not wreck their hard work with too frequent releases. Bugfixes (like when engine xx crashes under GUI yy) are O.K. But 3 versions per month are just overkill IMO. It is better to make a new release only once per quarter, or when there is significant (at least 25-30 ELO) improvement. Whichever comes first. Of course the latter is better

just my 2 eurocents

Thanks for your kind words and support Richard.

I would never want to or feel comfortable dictating to engine authors what they should or shouldn't do. Additionally, I wouldn't like to see their enthusiasm or creativity stifled by the expectations of others.
I think the best thing is just for engine authors to realise that if they release a lot of versions over a short period of time, testing groups won't be able to test them all.

Cheers,
Graham.

[Engin]

ok, i understand that you need time to testing, but i think you have more waste of time to testing a buggy engine or even weaker version and that is not so good for me and you.

if i will wait only with your testing tempo ( all 3 month or even 6 month) then you are stopping me in my creations and ideas, if you are not the time dont test it, or download the latest one that is the one who less bugs have and may little improved then provous.

[Graham Banks]

ok, i understand that you need time to testing, but i think you have more waste of time to testing a buggy engine or even weaker version and that is not so good for me and you.

if i will wait only with your testing tempo ( all 3 month or even 6 month) then you are stopping me in my creations and ideas, if you are not the time dont test it, or download the latest one that is the one who less bugs have and may little improved then provous.

Hi Engin,

I don't expect you to wait.
We test for fun and do the best we can.

Cheers,
Graham.

[Kirill Kryukov]

ok, i understand that you need time to testing, but i think you have more waste of time to testing a buggy engine or even weaker version and that is not so good for me and you.

if i will wait only with your testing tempo ( all 3 month or even 6 month) then you are stopping me in my creations and ideas, if you are not the time dont test it, or download the latest one that is the one who less bugs have and may little improved then provous.

Hi Engin,

I think you misunderstand the role of the testing groups in computer chess. It's not our goal to provide an author (you) with test results necessary to improve his engine. Usually an author of a strong engine is supposed to have his own testing framework, where he will test various (and numerous) builds of his engine, searching for improvement. If the said author instead waits for the testing groups to test his engine, then indeed he will progress slowly. We like to believe that our results, and particularly games that we publish, are still helpful to the authors for improving their engines. However this is not our main goal.

Our main goal (though other testers may have different view) is to provide an independent objective strength comparison of various engines, under specific conditions. Together with tournaments, this forms a medium for a meaningful competition in computer chess. So, the real world logic is backwards to yours: FIRST you come up with a strong engine, THEN it gets evaluated by testing groups, not the other way around. Although of course updated versions are always welcome and will get tested in due time.

Some of the CCRL testers actually were so kind as to test more than a dozen versions of your engine in quick succession. (I was not one of them - I usually let a new engine "cool down" before I start testing it.) Many of those versions had to be "killed" from the list, as otherwise they would be left with 100 games or less. This means that our time was wasted testing those versions. It's amusing that you still think we don't do enough for you.

May be you don't realize that there are hundreds of other engines, many still waiting to have ANY version of them tested by us. Or may be you think only strong engines should be tested. While most of us will agree, we all have different ideas of what "strong" is. For me it's above about 2000 Elo.

As Graham said we try to do the best we can. Unfortunately "Best for Tornado" and "Best for computer chess" are not always the same thing.

Best wishes and good luck,
Kirill