Tools for running match/tournament/tests

[mathmoi]

Hi,

To test new versions of my engine I use a bash script that run matchs between my engine and pre-defined opponents using xboard endlessly (until I kill it). This method has some drawbacks.

• If I stop the process in the middle of a run and start it later the numbers of games against each opponent migh not be the same.
  There is no way to make it play more than one game at a time. Except by running the script two or more times.
  There is no way to distribute the match (or test) to more than one computer.
  Since xboard does not allow games faster than 1 minutes + 1 seconds, I can't run really fast games like Dr. Hyatt is doing (10s + .1s).

Is there some free/open-source tools that would allow me to run tests/match/tournaments in theses conditions?

If I can't find any, I might create one. I would do something in python with a command line interface only and open source it. I already put some tough into this, but I don't want to reinvent the wheel.

[Volker Pittlik]

...

If I stop the process in the middle of a run and start it later the numbers of games against each opponent migh not be the same.

...
Since xboard does not allow games faster than 1 minutes + 1 seconds, I can't run really fast games like Dr. Hyatt is doing (10s + .1s).
...

If I can't find any, I might create one. I would do something in python with a command line interface only and open source it. I already put some tough into this, but I don't want to reinvent the wheel.

I working on something similar. I also didn't find something usable (different tournament types, restart...) on the net. Mine is written as pure script for the bash.

To restart a test/tournament I'm thinking about that the script should write the state of the tourney into a file (which engines are playing, number of round and so on), reads that file in case a restart should be done and makes the setting then.

Xboard can use that time control just write tc as -tc $minutes:$seconds
The problem is that some engine get in severe time trouble because it seems time control faster than 1 minute is often not implemented.

I'm going to open source it when it satisfies my needs what may take some more time.

Volker

[hgm]

As Volker points out, XBoard allows time controls down to 1 sec. If you want to go sub-second (e.g. for having a more precisely tiuned incremment, e.g. 6 sec + 0.1 sec/move you can achieve that by specifying a time-odds factor.

If you give both engines time-odds factor 100, you would specify the above as 10+10. So you can specify the TC as precisely as you like. Internally WB accounts in msec, though, so nothing would be achieved by specifying a time-odds factor above 1000.

The engine gets the remaining time rounded to centi-seconds, so having less than 1 centi-second per move is also not recommended. In that case it would be better to play by nodes-per-second.

In principle, communication delays should be short enough to play pretty fast games, if you don't animate the moves. The fastest I ever tried was 24 sec sudden-death for Joker80 with WinBoard, when it was playing an opponent that had 36 min. Joker80 still did not suffer any time losses, and was actually winning most of these games. (I did not giveit a time odds factor of 90 for no reason...)

[michiguel]

Hi,

To test new versions of my engine I use a bash script that run matchs between my engine and pre-defined opponents using xboard endlessly (until I kill it). This method has some drawbacks.

• If I stop the process in the middle of a run and start it later the numbers of games against each opponent migh not be the same.
  There is no way to make it play more than one game at a time. Except by running the script two or more times.
  There is no way to distribute the match (or test) to more than one computer.
  Since xboard does not allow games faster than 1 minutes + 1 seconds, I can't run really fast games like Dr. Hyatt is doing (10s + .1s).

Is there some free/open-source tools that would allow me to run tests/match/tournaments in theses conditions?

If I can't find any, I might create one. I would do something in python with a command line interface only and open source it. I already put some tough into this, but I don't want to reinvent the wheel.

I am starting to do something very simple in Ruby, which is a language I am starting to experiment/learn. Very easy to use. Python will be comparable in this respect. I really hate bash. I found it difficult to read and awkward to program. What I have done is very simple, but I doubt I will need more. I do not have it with me right now, but I will post it tonight.
With a flexible language like ruby (or python), it is possible to do certain things adjusted to your immediate needs. For instance, I replay games that were lost on time.

Miguel

[michiguel]

As Volker points out, XBoard allows time controls down to 1 sec. If you want to go sub-second (e.g. for having a more precisely tiuned incremment, e.g. 6 sec + 0.1 sec/move you can achieve that by specifying a time-odds factor.

If you give both engines time-odds factor 100, you would specify the above as 10+10. So you can specify the TC as precisely as you like. Internally WB accounts in msec, though, so nothing would be achieved by specifying a time-odds factor above 1000.

The engine gets the remaining time rounded to centi-seconds, so having less than 1 centi-second per move is also not recommended. In that case it would be better to play by nodes-per-second.

In principle, communication delays should be short enough to play pretty fast games, if you don't animate the moves. The fastest I ever tried was 24 sec sudden-death for Joker80 with WinBoard, when it was playing an opponent that had 36 min. Joker80 still did not suffer any time losses, and was actually winning most of these games. (I did not giveit a time odds factor of 90 for no reason...)

Another way to specify fast games is time/x moves, without the need to go subsecond for the increment. I am currently experimenting with 20 seconds/ 40 moves.

Yes, animating moves is not a good idea. I measured the delay to be 150 ms on average, but sometimes it goes way higher than that.
http://www.open-aurec.com/wbforum/viewtopic.php?t=49855

Without animate move, the delay is between 10-50 ms. Later, I am going to measure it in "blind" mode to see if there is a difference.

Miguel

[krazyken]

I use a bash script, which has the option to run round robin or gauntlet style matches. To have it go endlessly you could just surround it in while true loop.

it also has the option to run simultaneous games. based on a flag I set for each engine. You can find it here.

[mathmoi]

As Volker points out, XBoard allows time controls down to 1 sec. If you want to go sub-second (e.g. for having a more precisely tiuned incremment, e.g. 6 sec + 0.1 sec/move you can achieve that by specifying a time-odds factor.

If you give both engines time-odds factor 100, you would specify the above as 10+10. So you can specify the TC as precisely as you like. Internally WB accounts in msec, though, so nothing would be achieved by specifying a time-odds factor above 1000.

The engine gets the remaining time rounded to centi-seconds, so having less than 1 centi-second per move is also not recommended. In that case it would be better to play by nodes-per-second.

In principle, communication delays should be short enough to play pretty fast games, if you don't animate the moves. The fastest I ever tried was 24 sec sudden-death for Joker80 with WinBoard, when it was playing an opponent that had 36 min. Joker80 still did not suffer any time losses, and was actually winning most of these games. (I did not giveit a time odds factor of 90 for no reason...)

Hi H.G.,

Is the time odd factor a functionnality you added in the latest Winboard version? If it's the case is it avaible as a xboard port?

[mathmoi]

Hi Kenny,

I think I'll write my own script in Python, but I'll take a look at yours thanks.

[hgm]

Is the time odd factor a functionnality you added in the latest Winboard version? If it's the case is it avaible as a xboard port?

It is an addition of mine, but it has been around for some time already, even in the previous XBoard version (4.3.14).

[Don]

Hi,

To test new versions of my engine I use a bash script that run matchs between my engine and pre-defined opponents using xboard endlessly (until I kill it). This method has some drawbacks.

• If I stop the process in the middle of a run and start it later the numbers of games against each opponent migh not be the same.
  There is no way to make it play more than one game at a time. Except by running the script two or more times.
  There is no way to distribute the match (or test) to more than one computer.
  Since xboard does not allow games faster than 1 minutes + 1 seconds, I can't run really fast games like Dr. Hyatt is doing (10s + .1s).

Is there some free/open-source tools that would allow me to run tests/match/tournaments in theses conditions?

If I can't find any, I might create one. I would do something in python with a command line interface only and open source it. I already put some tough into this, but I don't want to reinvent the wheel.

Before you try to create one - consider the following - I have some ideas here that might help:

You can run remotely just by shelling out to program with rsh or ssh. Very few computers use rsh these days because of security issues but ssh is the way to go. You must have ssh server on the other end of course. Not sure if this is possible in windows but it probably is. Look for openssh and see if it's available for Windows. Or maybe there is another ssh thingy for windows. You need both a server and a client.

So with ssh you just invoke a program on a remote computer like this:

ssh greencheeks.homelinux.org gnuchess

this would run gnuchess from my computer if your secure key was set up on my machine and you could pretend it was on your machine. So you don't really need special remote considerations if you have the right tools. Even though Windows is awkward with useful abstractions like this I'm pretty sure it's possible to set this up and you wouldn't regret it because it serves you for many things, not just this.

I have my own tester that is a tcl program. It is command line based and does not give you a GUI. It is crufty because I designed it only for myself and you must configure it with a configuration file. I keep telling myself I am going to make it better. Here is what it can and cannot do - you are free to have a copy if you want:

1. It has almost 8000 opening hard coded to 5 moves exactly (10 ply)
2. it can play round robin.
3. it can play "free" where it schedules games attempting to not mismatch opponents.
4. You can set fischer or fixed depth time control. Perhaps others, I haven't looked in a while.
5. It can make a pgn file of all games but it's terribly slow if you have thousands of games. It used to generate them as it played but I like to test super hyper bullet like game in 1 second etc and I turned this off - you have to generate off line. It's real fast if you are not trying to produce PGN files while you play. However it still produces a game record in a simple format that you could probably easily convert to PGN - it just doesn't do the SAN thing by default because it's slow.
6. It wants you to have bayeselo in your path but it's not required. There is a mode where it make a rating chart.
7. all results are in a sqlite3 database so you can do your own processing of it - make html pages, etc. (I'm not sure, but it may already produce an html crosstable - or maybe that is the go tester I don't remember.)
8. It only works with UCI - would not be difficult to also make it work with Xboard and I'm considering this so that I can use Crafty and others.

You configure it via a configuration file, don't have to know sql but it helps.

If you are not a command line junkie you won't like it.

You can also purge games (without knowing sql) and add players. If you add a new play in, for instance, round robin mode it will catch up this new player. so when trying things I add new players and remove old ones that don't cut the mustard in my testing.

Here is something you and other might be interested in:

I have most of the code for a java autotester too. Routines to talk to chess program, create PGN and understand rules. Right now it is working great for problem testing but this is currently proprietary and not public. I am in control of it however so it's only proprietary because I say it is. I'm using it for a project with Larry Kaufman for running problems and taking statistics from the results, etc. I do plan to turn this into a really practical auto tester system that does it all (UCI, xboard, any kind of tournament or match, even remote playing perhaps.) I hate Java completely, only saving grace is GCJ and it's reasonably fast - I like being able to produce native executables. Anybody want to do simple GUI for this autotester in Java? Let me know and we can do something with this.

If anyone wants the tcl tester let me know - I'm not going to give you any support but I will try to get you started. It does work quite well.

I think I have a prototype of an older verison of the tcl tester that shows the game being played with a tk gui and it looks nice. May integrate it but I'm a command line junkie. Would make it an option. Sometimes I would like to watch the games.

Sample output:

Code: Select all

PLAYER            ELO       TIME     +     -      GAMES
--------------   ----   --------   ----  ----  --------
s-246-0-7        2023      7.517     25    25      371
spike-6          2000      2.967     26    25      369
s-246-2-7        1912      2.536     25    25      370
s-246-1-7        1912      2.496     25    25      370

Various version of my program vs Spike at fixed depth searches. I usually test with several other programs but in this particular test I'm trying to see how far I have to go to catch Spike.