A post by Don Dailey that I agree with

[Michael Sherwin]

Subject: Book learning and rating bias

Author: Don Dailey

Date: 10:58:29 05/01/98

Hi Everyone,

"I've been thinking a lot about the super book phenomenon we are seeing
and all the issues involved and would like to post some observations
and opinions. This seems like a good forum to do so.

Originally I started thinking about testing procedures to neutralize
the affect of heavily booked programs. But the more I thought about
it, the more I realized this would be impossible to do fairly. After
all, each program has it's own strengths and weakness and should play
openings compatible with it's own playing style (the same as we humans
do!) This implies that opening preparation is an integral part of how
each computer performs. This is also how it works with humans.

But then you get into the issue of a computer playing the same game
over and over. But just like us humans, if you allow yourself to get
beat the same way over and over again then shame on you! Artificial
techniques to prevent this abound, but I'm thinking they should not be
applied. Looked at it another way, why should I be penalized for
playing a move I know wins? YOU should be penalized for letting me do
this!

One very important factor is book learning and I do not know how this
is handled by the raters, hopefully it is handled correctly. The
issue is that if I have a program that learns from it's mistakes
(which I think is a very good thing,) then that program should never
be "reset" by the testing procedure. As an example, if I was a biased
tester, I could simply reset the learning mechanism frequently and
affect the results (perhaps) significantly. I might move the program
from machine to machine or whatever it takes to defeat the learning
mechanism.

Having several testers testing the same program on different machines
creates the same problem. I argue that the more computers you use to
test a program on, the more of a handicap you give to that program if
it utilizes learning mechanisms. I don't know the magnitude of the
error but it certainly would be a factor to consider. The only
solution I am aware of is to use the same machine to test the program
on. If you use other machines you must consider them separate
identities.

The other problem, which I believe is a pretty big factor is opponent
selection. From experiments I have done, this can have a large effect
on the results. I suspect it may be the single greatest source of
error the raters must face. I want to mention that I do not know how
they make these decisions and I know very little about their testing
methodology and am not criticizing them. I just bring this up as a
potential problem."

[Modern Times]

Should there be a rating list where engines that learn have their learning turned on?

There already is one - SSDF

[Michael Sherwin]

Should there be a rating list where engines that learn have their learning turned on?

There already is one - SSDF

I did not know that I'll check it out! Thanks.

P.S. I seem to remember that SSDF only rates higher strength engines? Anyway I will still take a look.

[jdart]

I agree that book knowledge and book learning are important components of overall chess performance.

For this reason, personally I am most interested in competitions and tournaments like SSDF where the book is enabled. I see programs play into bad opening lines all the time, even in something like TCEC were they are given a long time control and lots of cores. Remember, book knowledge includes knowledge about correspondence games, where players have days to make a move.

I do see though that there is a place for testing w fixed books and no learning. Learning on creates complications because now the engine's performance is history dependent, and over time its rating will go up, at least if it is paired with the same opponents.

--Jon

[Michael Sherwin]

Should there be a rating list where engines that learn have their learning turned on?

There already is one - SSDF

I did not know that I'll check it out! Thanks.

P.S. I seem to remember that SSDF only rates higher strength engines? Anyway I will still take a look.

Well RomiChess is not in their list so a non sequitur for me.