elpapa wrote:
Oh dear, I think Tony will explode this time. He didn't make a point of using Arena. He was just explaining what the time settings meant.
Yeah, what he said.... 'twas a copy and paste from an output log, nothing more. Lucas does have a point with using command line tools, though. Haven't tried CLOP yet for myself, but it looks like an interesting concept. Maybe I could use it to try tune material values for variants.
If you use Windows, there's LittleBlitzer. It can play very fast games and in parralel too. The only thing that it doesn't have compared to cutechess-cli is that it's still a GUI, and it's not a fully scriptable command line tool. So you couldn't use it with CLOP for example, but for self testing of playing tournaments at fast time controls, it's certainly a good choice. On a positive side, some people might find it more user friendly than a command line tool.
Anyway, the most important thing is to have the right tool for the job, which ever it is. And the job is to play a lot of games (yes 1000 is generally a good choice although it depends on the elo difference you need to measure, I prefer a more systematic approach with sequential probability ratio test). So with a given CPU time constraint, the most games will be achieved by playing parralel games at fast time controls. When you have to do a trade off between playing lots of games very fast and playing less games slowly (for better quality of the games), it's best to choose the former in (almost?) all situations
I do use Windows, and I am OK with command line tools. There is a Windows compile of cutechess-cli, and it sounds like it's the best option for integrating with CLOP, so I think I will go that route. cutechess-cli may not be the friendliest tool, but according to some I'm not the friendliest guy, so it should suit me just fine.
tmokonen wrote:I do use Windows, and I am OK with command line tools. There is a Windows compile of cutechess-cli, and it sounds like it's the best option for integrating with CLOP, so I think I will go that route. cutechess-cli may not be the friendliest tool, but according to some I'm not the friendliest guy, so it should suit me just fine.
Good luck with cutechess-cli and CLOP. I'm currently running a piece value optimisation with CLOP
lucasart wrote:For fast and parralel testing, cutechess-cli is beats the crap out of any GUI (precisely because it is not a *G*UI), and it also works easily with shell or python scripting. A great example is the combinaison of cutechess-cli and CLOP
Is it really that much faster than WinBoard (or XBoard) when you run the latter with the option-noGUI? I never did any precice timing measurements, but from looking at the code I don't see any places where WinBoard seems to waste any time, so it is hard to believe the bare necessities could be done much faster...
lucasart wrote:For fast and parralel testing, cutechess-cli is beats the crap out of any GUI (precisely because it is not a *G*UI), and it also works easily with shell or python scripting. A great example is the combinaison of cutechess-cli and CLOP
Is it really that much faster than WinBoard (or XBoard) when you run the latter with the option-noGUI? I never did any precice timing measurements, but from looking at the code I don't see any places where WinBoard seems to waste any time, so it is hard to believe the bare necessities could be done much faster...
I don't know, as I don't use winboard. But I'm glad to hear that there's a no GUI option that is fast. The important thing is also to be able to play several games at the same time. If you have a quad, with the same amount of time (assuming a single threaded program) you can play 4 times more games! And that's where cutechess-cli and littleblitzer really shine
Oh, but concurrent play is also a standard feature of the WinBoard built-in tournament manager.
Not that it seems very important: even before WinBoard had a tournament manager, (i.e. still relied on an external one like PSWBTM), I was simply playing two gauntlets in parallel on my dual. Either of two versions of my own engine I wanted to test both, or with the same engine with each half the set of opponents. You could let the games be written to the same PGN file, or concatenate them later.
I did a small test, by making a modified version of Fairy-Max, which prints the millisec timestamp before it sends a move, and after it receives a move, and then let two instances of it play against each other under XBoard (ponder off with -noGUI on a single-core machine, a 1.3GHz Pentium M running Ubuntu 10.04). A trace from the debug file below shows that there typically only elapses 1 msec between the sending of the move and the opponent getting it. (Sometimes I see 2 msec.)
So it should not be much of a problem to do 1-sec games. (Faster could be problematic for WB engines anyway, as WB protocol has only centi-sec resolution.)
hgm wrote:I did a small test, by making a modified version of Fairy-Max, which prints the millisec timestamp before it sends a move, and after it receives a move, and then let two instances of it play against each other under XBoard (ponder off with -noGUI on a single-core machine, a 1.3GHz Pentium M running Ubuntu 10.04). A trace from the debug file below shows that there typically only elapses 1 msec between the sending of the move and the opponent getting it. (Sometimes I see 2 msec.)
So it should not be much of a problem to do 1-sec games. (Faster could be problematic for WB engines anyway, as WB protocol has only centi-sec resolution.)
It would be a good idea to have engines sending such timing info to the GUI for inclusion in the log file, for testing GUIs. A UCI engine could send it as an info string. UCI2WB again passes everyting it receives from the engine to the GUI, so it would appear in the WinBoard debug file as well. Polyglot might write it in its own log (so you have to go through the drudgery of merging logs...) Of course GUIs that support UCI directly will always include the UCI output in their log.
Your engine is UCI, isn't it? Can't you put that in? I have no sources of UCI engines ready for compiling...
