A request to engine authors

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Tord Romstad
Posts: 1808
Joined: Wed Mar 08, 2006 8:19 pm
Location: Oslo, Norway

A request to engine authors

Post by Tord Romstad » Sat Apr 26, 2008 4:29 pm

Hi all,

I like to run test games at a fixed number of nodes per move, because it improves reproducibility, and because it enables me to run other CPU-intensive tasks on the computer without influencing the results of the running tests.

Unfortunately, it seems that almost no engine authors bother to support the UCI "go nodes ..." command, which means that I can currently only test against older versions of my own program. :-(

UCI engine authors, could you please implement "go nodes ..." in your next version? It's only a minute of work, after all. Every time you increment your move counter, check if it exceeds the maximum limit, and stop the search if it does.

Tord

User avatar
michiguel
Posts: 6389
Joined: Thu Mar 09, 2006 7:30 pm
Location: Chicago, Illinois, USA
Contact:

Re: A request to engine authors

Post by michiguel » Sat Apr 26, 2008 5:10 pm

Tord Romstad wrote:Hi all,

I like to run test games at a fixed number of nodes per move, because it improves reproducibility, and because it enables me to run other CPU-intensive tasks on the computer without influencing the results of the running tests.

Unfortunately, it seems that almost no engine authors bother to support the UCI "go nodes ..." command, which means that I can currently only test against older versions of my own program. :-(

UCI engine authors, could you please implement "go nodes ..." in your next version? It's only a minute of work, after all. Every time you increment your move counter, check if it exceeds the maximum limit, and stop the search if it does.

Tord
Gaviota supports fixed nodes/move but it is not UCI, it only supports winboard/xboard. Besides, Gaviota is not a strong engine. On the other hand, being a strong engine is not necessarily a requirement for testing as long as it is possible to play games with handicaps.

I think that movei can play with fixed number of nodes. What are the engines capable to do that. Uri?

Another advantage to play with fixed number of nodes is that we can used the debug versions to play once in while or we can choose to play really "fast" games without compromising the reliability of the test. There is a lot of things that can screw a game/2 seconds but a game played with 10,000 nodes/move is more reliable (or whatever number you wish).

Miguel

User avatar
hgm
Posts: 23793
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: A request to engine authors

Post by hgm » Sat Apr 26, 2008 5:22 pm

When we are at it, then, I would also want to encourage authors of WinBoard engines to implement a command 'sn N', which limits the search to N nodes, similar to the way 'st' limits the serach time and 'sd' limits the depth.

You might want to send a 'feature sn=1' to announce to WinBoard that you understand it, to prevent that WinBoard will warn the user that the engine might not support the command if the user tries to set the nodes through the WinBoard menu command I will supply for this purpose. It is safe to do this in protocol 2, as protocol 2 specifies that WinBoard will simply reject unimplemented features without further consequences.

The user can always force sending of an 'sd N' to the engine by making it part of the initString.

User avatar
michiguel
Posts: 6389
Joined: Thu Mar 09, 2006 7:30 pm
Location: Chicago, Illinois, USA
Contact:

Re: A request to engine authors

Post by michiguel » Sat Apr 26, 2008 8:26 pm

hgm wrote:When we are at it, then, I would also want to encourage authors of WinBoard engines to implement a command 'sn N', which limits the search to N nodes, similar to the way 'st' limits the serach time and 'sd' limits the depth.

You might want to send a 'feature sn=1' to announce to WinBoard that you understand it, to prevent that WinBoard will warn the user that the engine might not support the command if the user tries to set the nodes through the WinBoard menu command I will supply for this purpose. It is safe to do this in protocol 2, as protocol 2 specifies that WinBoard will simply reject unimplemented features without further consequences.

The user can always force sending of an 'sd N' to the engine by making it part of the initString.
Done :-)

Miguel

User avatar
Bill Rogers
Posts: 3562
Joined: Thu Mar 09, 2006 2:54 am
Location: San Jose, California

Re: A request to engine authors

Post by Bill Rogers » Sat Apr 26, 2008 11:47 pm

Yord
Being as not everyone counts nodes in exactly the same way why don't you just play them all to a fixed ply depth. It seems to be basically the same thing and everyone counts plys in the same way.
Bill

Robert Pope
Posts: 510
Joined: Sat Mar 25, 2006 7:27 pm

Re: A request to engine authors

Post by Robert Pope » Sun Apr 27, 2008 12:09 am

Bill Rogers wrote:Yord
Being as not everyone counts nodes in exactly the same way why don't you just play them all to a fixed ply depth. It seems to be basically the same thing and everyone counts plys in the same way.
Bill
But plies aren't equal either. One program does lots of extensions, and another has few extensions, and you get one program that gets 5x as much time as the other.

User avatar
Zach Wegner
Posts: 1922
Joined: Wed Mar 08, 2006 11:51 pm
Location: Earth
Contact:

Re: A request to engine authors

Post by Zach Wegner » Sun Apr 27, 2008 12:52 am

Robert Pope wrote:But plies aren't equal either. One program does lots of extensions, and another has few extensions, and you get one program that gets 5x as much time as the other.
Not only that, but reaching a certain depth gives a big distribution over both time and strength for the different phases of the game. E.g. reaching depth 10 in the opening and endgame will take different times and result in different levels of play.

bob
Posts: 20642
Joined: Mon Feb 27, 2006 6:30 pm
Location: Birmingham, AL

Re: A request to engine authors

Post by bob » Sun Apr 27, 2008 1:16 am

Tord Romstad wrote:Hi all,

I like to run test games at a fixed number of nodes per move, because it improves reproducibility, and because it enables me to run other CPU-intensive tasks on the computer without influencing the results of the running tests.

Unfortunately, it seems that almost no engine authors bother to support the UCI "go nodes ..." command, which means that I can currently only test against older versions of my own program. :-(

UCI engine authors, could you please implement "go nodes ..." in your next version? It's only a minute of work, after all. Every time you increment your move counter, check if it exceeds the maximum limit, and stop the search if it does.

Tord
Make a note that this can give highly biased results. A simple change to your eval changes the shape of the tree ever-so-slightly, but more than enough to change the game outcome even if the change is no better or worse than without...

I've played with this idea extensively and discarded it as being unusable...

User avatar
michiguel
Posts: 6389
Joined: Thu Mar 09, 2006 7:30 pm
Location: Chicago, Illinois, USA
Contact:

Re: A request to engine authors

Post by michiguel » Sun Apr 27, 2008 2:10 am

Bill Rogers wrote:Yord
Being as not everyone counts nodes in exactly the same way why don't you just play them all to a fixed ply depth. It seems to be basically the same thing and everyone counts plys in the same way.
Bill
I does not matter how you count them. It is just approximately proportional to time. As an example, rather than playing a game 1 second a move, you play N nodes a move (when N equals approx, NPS of the the engine). Since NPS increase on endgames, the game will slightly speed up at the end, which is not bad.

Miguel

Uri Blass
Posts: 8611
Joined: Wed Mar 08, 2006 11:37 pm
Location: Tel-Aviv Israel

Re: A request to engine authors

Post by Uri Blass » Sun Apr 27, 2008 2:20 am

michiguel wrote:
Tord Romstad wrote:Hi all,

I like to run test games at a fixed number of nodes per move, because it improves reproducibility, and because it enables me to run other CPU-intensive tasks on the computer without influencing the results of the running tests.

Unfortunately, it seems that almost no engine authors bother to support the UCI "go nodes ..." command, which means that I can currently only test against older versions of my own program. :-(

UCI engine authors, could you please implement "go nodes ..." in your next version? It's only a minute of work, after all. Every time you increment your move counter, check if it exceeds the maximum limit, and stop the search if it does.

Tord
Gaviota supports fixed nodes/move but it is not UCI, it only supports winboard/xboard. Besides, Gaviota is not a strong engine. On the other hand, being a strong engine is not necessarily a requirement for testing as long as it is possible to play games with handicaps.

I think that movei can play with fixed number of nodes. What are the engines capable to do that. Uri?

Another advantage to play with fixed number of nodes is that we can used the debug versions to play once in while or we can choose to play really "fast" games without compromising the reliability of the test. There is a lot of things that can screw a game/2 seconds but a game played with 10,000 nodes/move is more reliable (or whatever number you wish).

Miguel
Movei can do it as uci engine
I used it mainly to test against older versions.

I know that rybka also supports it.
I do not know much about the other engines that support it because I did not test much.

Uri

Post Reply