Dark Basement Ratings List - The HIARCS 12 Interrogation...

[Spock]

Also the sequence in which you play the games probably affects the ratings too. If you play the weaker engines first, followed by the stronger ones, does that give a different rating than if you did it the other way round, i.e. the stronger ones first followed by the weaker ones ? As humans we learn, and FIDE ratings of course include learning. But a computer chess ratings list is altogether a different thing, and learning potentially "pollutes" the results. If you take the Hiarcs website ratings list - they are rightfully pretty pleased with their published 51-49 victory over Rybka 32-bit. But, was that match played on a brand new Hiarcs with no initial learning file, or was it played with a learning file already there from thousands or tens of thousands of games ? I tried to ask this on the Hiarcs forum but my questions were not exactly welcomed with open arms. The result is highly dependent on the status of the learning file at the time the match was played.
Of course Hiarcs also benefits from this being run on a 32-bit O/S which penalises the likes of Rybka and Zappa, whcih gain from 64- bit and maybe Naum. I see the argument, if the author spends the time making a wonderful learning feature, why penalise him by turning it off. Well, if an author goes to the time and trouble of writing code specifically optimised to get performance gains from increasingly common 64-bit OS, why penalise him by only using 32-bit....

In my view, learning just adds an additional margin of error into a ratings list dependent on sequence of games etc and status of the learning file when they were played. As long as that is understood, then fine

[Tony Thomas]

Learning does increase the margin of error. If you are part of the race for the top postion it changes everything. An engine with a better learning is penalized when its turned off. Most of us run tournaments with ponder off. True that the conditions are equal for all engines, but there are few top 50 engines that doesnt support pondering (Petir comes to mind). They would be rated lower on a rating list that plays ponder on matches. There isnt really any solutions to all these arguments, we are all boneheads and we will never come in to compromise. However, we can all deal in a civil manner and not end up hating each other.

Engines that gets penalized under learning off matches, I list a few with superior learning..
Romichess
Frenzee
Spike
Hiarcs
Shredder
Prodeo
Note that I didnt include chessbase only engines as my experience with their learning is rather old (Fritz 6, winboard adapter era).

If I am trying to find the improvement between versions (internal testing), then learning off is indeed better. Also, my future tests of Romi will be based on 15 fixed positions to reduce the randomness.

[Graham Banks]

There isnt really any solutions to all these arguments, we are all boneheads and we will never come in to compromise. However, we can all deal in a civil manner and not end up hating each other.

Quite correct. I value the testing efforts of all others. Variety is the spice of life.

[Tony Thomas]

There isnt really any solutions to all these arguments, we are all boneheads and we will never come in to compromise. However, we can all deal in a civil manner and not end up hating each other.

Quite correct. I value the testing efforts of all others. Variety is the spice of life.

I guess it is safe to assume that with learning on, and own book, Hiarcs is second only to Rybka. On the other hand, if learning is off and a general book is used Hiarcs is struggling to be in the top five..To me that means a huge tuning of books, were as the engine improvement was moderate.

[Graham Banks]

I guess it is safe to assume that with learning on, and own book, Hiarcs is second only to Rybka. On the other hand, if learning is off and a general book is used Hiarcs is struggling to be in the top five..To me that means a huge tuning of books, were as the engine improvement was moderate.

Possibly.

[Spock]

I guess it is safe to assume that with learning on, and own book, Hiarcs is second only to Rybka.

I don't think it is a very safe assumption necessarily

But I like the variety of lists. If everyone tested under exactly the same conditions it would be quite boring. I'd urge Erik to continue what he is doing, SSDF to continue etc. The more variety of testing conditions the better !! It adds to our understanding of an engine's behaviour

[Tony Thomas]

We are bunch of thread hijackers. When Erik comes back, he might get mad.

[Spock]

We are bunch of thread hijackers. When Erik comes back, he might get mad.

lol he might do.... but he may be happy there is so much interest in his list

[Tony Thomas]

We are bunch of thread hijackers. When Erik comes back, he might get mad.

lol he might do.... but he may be happy there is so much interest in his list

It has nothing to do with interest in his list. All hell break lose when someone uses the forbidden word/learning.




















































































........Just messing with ya Erik, we are interested in your list.

[Ovyron]

Engines that gets penalized under learning off matches, I list a few with superior learning..
Romichess
Frenzee
Spike
Hiarcs
Shredder
Prodeo

Yes, I also test with learning ON (even though I don't specify it, I will do it from now). My point is that if a new engine appears with some sort of "next generation" learning that is very good at learning how to beat its opponents, and after some time of learning it's capable of being the clear #1 engine, the other rating lists would not show that, and the only way to know is if some other independent testers test with learning ON, that's why I praise them when they do it.

I'm glad I decided to make my tourneys public instead of keeping them private, my last PGN had 5 downloads and I'd just need one to keep going for the person interested