how to rate an engine

[AndrewShort]

what is the easiest way to get an estimate of the rating of my engine? My engine sits in a stand-alone windows application (which I created - I don't use winBoard), no automation interfaces or anything like that, so it's not clear to me how I could make my engine play hundreds of games without me actually moving the pieces in the user interface myself...

unless there is some magic set of puzzles that somehow give a reasonable estimate of rating?

[Aleks Peshkov]

If you do not want to create UCI or Winboard engine, you may learn your own GUI to accept foreign UCI/Winboard engines, like Fritz do.

[Volker Pittlik]

what is the easiest way to get an estimate of the rating of my engine?

Play much games against different opponents of similar strength. The strength of the opponents should be known.

...My engine sits in a stand-alone windows application (which I created - I don't use winBoard), no automation interfaces or anything like that,...

Maybe your design is a bit incomplete. Chess Engine Communication Protocol ("WinBoard") or UCI is not to complicated to implement

...so it's not clear to me how I could make my engine play hundreds of games without me actually moving the pieces in the user interface myself...

Maybe you can try AutoHotKey. When you created such a fine GUI why not implementing a xboard or UCI interface? There are fine and reliable GUIs which solved your problems (Winboard or Arena for example. They are even free of charge!).

...unless there is some magic set of puzzles that somehow give a reasonable estimate of rating?

There isn't. Take an easy test and use short time control. The more it solves the better it is.

vp

[hgm]

My engine sits in a stand-alone windows application (which I created - I don't use winBoard), no automation interfaces or anything like that, so it's not clear to me how I could make my engine play hundreds of games without me actually moving the pieces in the user interface myself...

So the easiest way by far is to make your engine WinBoard compatible. This can usually bedone in just a few hours of programming. (To play automated games you only need to implement the 'new', 'force', 'level', 'time' and 'go' commands as input (and simply ignore the rest), and print your move as 'move ....'. And you can even forget about 'level' and 'time' if you compile your engine for one fixed time control, and set WinBoard to approximately that same time for the benefit of the opponent, and then just play with /autoCallFlag=false.

And 'new' you can forget about too, if your engine always automatically sets up for a new game on startup, and you play with /reuse=false.

The only truly essential commands your engine has to understand are thus 'force' and 'go'. And an input move, of course.

The next simplest solution takes at least 100 times as much effort.

[AndrewShort]

thanks. very useful info.

by the way, does anyone have a rating estimate of ChessTitans that ships with Windows Vista? My engine trivially beats ChessTitans's level 10 (its highest level), which means either my engine is amazing (it isn't) or ChessTitan is terrible (more likely)...

[hgm]

I suppose the latter. I heard from people that hardly know the rules of Chess that they could beat it...

If you want a minimal serious opponent that does not lose because of bugs, you can play against micro-Max 1.6: this is hardly more than an alpha-beta search with material evaluation, and a little center-seeking for the light pieces.

[CRoberson]

The best idea has been stated (implement UCI and/or Winboard).

Aside from that, use a binary search!

You'll be the human interface for a series of matches.

You select engines from Leo's (or some other) rating list.
Leo has about 315 rated engines.

Code: Select all

   Start with #1 and play 2 games with alternating colors - you should lose
    and fairly quickly. 

   Next, play a 2 game match with the engine at position 157.
    if you lose to engine 157 
    { match with engine 78
      
        and so on      
    }
    else          
          if you win against engine 157 - match with engine 235
          {
               and so on
          }

Once you've gone through 5 engines, you can estimate a rating
using the standard provisional system. Probably shouldn't even
count the games against engine #1 if you had 2 losses to it. You
could skip #1 and start with #157.

I am assuming that you know what a binary search is.

[edlich]

Hi CR,

What I like is to use the Chessmaster Characters as they rate from 0 to about 2500. This can be done easily by hand and the rating is not so entirely wrong. It scores me about 1650 in blitz and that's correct.

Best
SE