Hi all,
I like to run test games at a fixed number of nodes per move, because it improves reproducibility, and because it enables me to run other CPU-intensive tasks on the computer without influencing the results of the running tests.
Unfortunately, it seems that almost no engine authors bother to support the UCI "go nodes ..." command, which means that I can currently only test against older versions of my own program.
UCI engine authors, could you please implement "go nodes ..." in your next version? It's only a minute of work, after all. Every time you increment your move counter, check if it exceeds the maximum limit, and stop the search if it does.
Tord
A request to engine authors
Moderators: hgm, Rebel, chrisw
-
- Posts: 1808
- Joined: Wed Mar 08, 2006 9:19 pm
- Location: Oslo, Norway
-
- Posts: 6401
- Joined: Thu Mar 09, 2006 8:30 pm
- Location: Chicago, Illinois, USA
Re: A request to engine authors
Gaviota supports fixed nodes/move but it is not UCI, it only supports winboard/xboard. Besides, Gaviota is not a strong engine. On the other hand, being a strong engine is not necessarily a requirement for testing as long as it is possible to play games with handicaps.Tord Romstad wrote:Hi all,
I like to run test games at a fixed number of nodes per move, because it improves reproducibility, and because it enables me to run other CPU-intensive tasks on the computer without influencing the results of the running tests.
Unfortunately, it seems that almost no engine authors bother to support the UCI "go nodes ..." command, which means that I can currently only test against older versions of my own program.
UCI engine authors, could you please implement "go nodes ..." in your next version? It's only a minute of work, after all. Every time you increment your move counter, check if it exceeds the maximum limit, and stop the search if it does.
Tord
I think that movei can play with fixed number of nodes. What are the engines capable to do that. Uri?
Another advantage to play with fixed number of nodes is that we can used the debug versions to play once in while or we can choose to play really "fast" games without compromising the reliability of the test. There is a lot of things that can screw a game/2 seconds but a game played with 10,000 nodes/move is more reliable (or whatever number you wish).
Miguel
-
- Posts: 27825
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: A request to engine authors
When we are at it, then, I would also want to encourage authors of WinBoard engines to implement a command 'sn N', which limits the search to N nodes, similar to the way 'st' limits the serach time and 'sd' limits the depth.
You might want to send a 'feature sn=1' to announce to WinBoard that you understand it, to prevent that WinBoard will warn the user that the engine might not support the command if the user tries to set the nodes through the WinBoard menu command I will supply for this purpose. It is safe to do this in protocol 2, as protocol 2 specifies that WinBoard will simply reject unimplemented features without further consequences.
The user can always force sending of an 'sd N' to the engine by making it part of the initString.
You might want to send a 'feature sn=1' to announce to WinBoard that you understand it, to prevent that WinBoard will warn the user that the engine might not support the command if the user tries to set the nodes through the WinBoard menu command I will supply for this purpose. It is safe to do this in protocol 2, as protocol 2 specifies that WinBoard will simply reject unimplemented features without further consequences.
The user can always force sending of an 'sd N' to the engine by making it part of the initString.
-
- Posts: 6401
- Joined: Thu Mar 09, 2006 8:30 pm
- Location: Chicago, Illinois, USA
Re: A request to engine authors
Donehgm wrote:When we are at it, then, I would also want to encourage authors of WinBoard engines to implement a command 'sn N', which limits the search to N nodes, similar to the way 'st' limits the serach time and 'sd' limits the depth.
You might want to send a 'feature sn=1' to announce to WinBoard that you understand it, to prevent that WinBoard will warn the user that the engine might not support the command if the user tries to set the nodes through the WinBoard menu command I will supply for this purpose. It is safe to do this in protocol 2, as protocol 2 specifies that WinBoard will simply reject unimplemented features without further consequences.
The user can always force sending of an 'sd N' to the engine by making it part of the initString.
Miguel
-
- Posts: 3562
- Joined: Thu Mar 09, 2006 3:54 am
- Location: San Jose, California
Re: A request to engine authors
Yord
Being as not everyone counts nodes in exactly the same way why don't you just play them all to a fixed ply depth. It seems to be basically the same thing and everyone counts plys in the same way.
Bill
Being as not everyone counts nodes in exactly the same way why don't you just play them all to a fixed ply depth. It seems to be basically the same thing and everyone counts plys in the same way.
Bill
-
- Posts: 558
- Joined: Sat Mar 25, 2006 8:27 pm
Re: A request to engine authors
But plies aren't equal either. One program does lots of extensions, and another has few extensions, and you get one program that gets 5x as much time as the other.Bill Rogers wrote:Yord
Being as not everyone counts nodes in exactly the same way why don't you just play them all to a fixed ply depth. It seems to be basically the same thing and everyone counts plys in the same way.
Bill
-
- Posts: 1922
- Joined: Thu Mar 09, 2006 12:51 am
- Location: Earth
Re: A request to engine authors
Not only that, but reaching a certain depth gives a big distribution over both time and strength for the different phases of the game. E.g. reaching depth 10 in the opening and endgame will take different times and result in different levels of play.Robert Pope wrote:But plies aren't equal either. One program does lots of extensions, and another has few extensions, and you get one program that gets 5x as much time as the other.
-
- Posts: 20943
- Joined: Mon Feb 27, 2006 7:30 pm
- Location: Birmingham, AL
Re: A request to engine authors
Make a note that this can give highly biased results. A simple change to your eval changes the shape of the tree ever-so-slightly, but more than enough to change the game outcome even if the change is no better or worse than without...Tord Romstad wrote:Hi all,
I like to run test games at a fixed number of nodes per move, because it improves reproducibility, and because it enables me to run other CPU-intensive tasks on the computer without influencing the results of the running tests.
Unfortunately, it seems that almost no engine authors bother to support the UCI "go nodes ..." command, which means that I can currently only test against older versions of my own program.
UCI engine authors, could you please implement "go nodes ..." in your next version? It's only a minute of work, after all. Every time you increment your move counter, check if it exceeds the maximum limit, and stop the search if it does.
Tord
I've played with this idea extensively and discarded it as being unusable...
-
- Posts: 6401
- Joined: Thu Mar 09, 2006 8:30 pm
- Location: Chicago, Illinois, USA
Re: A request to engine authors
I does not matter how you count them. It is just approximately proportional to time. As an example, rather than playing a game 1 second a move, you play N nodes a move (when N equals approx, NPS of the the engine). Since NPS increase on endgames, the game will slightly speed up at the end, which is not bad.Bill Rogers wrote:Yord
Being as not everyone counts nodes in exactly the same way why don't you just play them all to a fixed ply depth. It seems to be basically the same thing and everyone counts plys in the same way.
Bill
Miguel
-
- Posts: 10317
- Joined: Thu Mar 09, 2006 12:37 am
- Location: Tel-Aviv Israel
Re: A request to engine authors
Movei can do it as uci enginemichiguel wrote:Gaviota supports fixed nodes/move but it is not UCI, it only supports winboard/xboard. Besides, Gaviota is not a strong engine. On the other hand, being a strong engine is not necessarily a requirement for testing as long as it is possible to play games with handicaps.Tord Romstad wrote:Hi all,
I like to run test games at a fixed number of nodes per move, because it improves reproducibility, and because it enables me to run other CPU-intensive tasks on the computer without influencing the results of the running tests.
Unfortunately, it seems that almost no engine authors bother to support the UCI "go nodes ..." command, which means that I can currently only test against older versions of my own program.
UCI engine authors, could you please implement "go nodes ..." in your next version? It's only a minute of work, after all. Every time you increment your move counter, check if it exceeds the maximum limit, and stop the search if it does.
Tord
I think that movei can play with fixed number of nodes. What are the engines capable to do that. Uri?
Another advantage to play with fixed number of nodes is that we can used the debug versions to play once in while or we can choose to play really "fast" games without compromising the reliability of the test. There is a lot of things that can screw a game/2 seconds but a game played with 10,000 nodes/move is more reliable (or whatever number you wish).
Miguel
I used it mainly to test against older versions.
I know that rybka also supports it.
I do not know much about the other engines that support it because I did not test much.
Uri