Looking for automatic Engine Testing Software

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
OliverBr
Posts: 334
Joined: Tue Dec 18, 2007 8:38 pm
Location: Munich, Germany
Full name: Dr. Oliver Brausch
Contact:

Looking for automatic Engine Testing Software

Post by OliverBr » Sun Jul 19, 2020 12:24 pm

Hello together,

I am looking for a piece of software to test engines most comfortably. It should be able to run automatically many (>1000) games against other versions/engines and yields a reliable and reproductive result. Dependent of the number of cores it should be able to run games simultaneously in order get sooner results. Linux would be a nice to have.

I am quite sure that the Stockfishs and Komodos use something like this if they test some changes.

Thank you very much!
Chess Engine OliThink: http://brausch.org/home/chess

chrisw
Posts: 3366
Joined: Tue Apr 03, 2012 2:28 pm

Re: Looking for automatic Engine Testing Software

Post by chrisw » Sun Jul 19, 2020 12:50 pm

OliverBr wrote:
Sun Jul 19, 2020 12:24 pm
Hello together,

I am looking for a piece of software to test engines most comfortably. It should be able to run automatically many (>1000) games against other versions/engines and yields a reliable and reproductive result. Dependent of the number of cores it should be able to run games simultaneously in order get sooner results. Linux would be a nice to have.

I am quite sure that the Stockfishs and Komodos use something like this if they test some changes.

Thank you very much!
PET, Ed Schroeder bat files, cutechess,

mhouppin
Posts: 60
Joined: Wed Feb 12, 2020 4:00 pm
Full name: Morgan Houppin

Re: Looking for automatic Engine Testing Software

Post by mhouppin » Sun Jul 19, 2020 1:08 pm

OliverBr wrote:
Sun Jul 19, 2020 12:24 pm
Hello together,

I am looking for a piece of software to test engines most comfortably. It should be able to run automatically many (>1000) games against other versions/engines and yields a reliable and reproductive result. Dependent of the number of cores it should be able to run games simultaneously in order get sooner results. Linux would be a nice to have.

I am quite sure that the Stockfishs and Komodos use something like this if they test some changes.

Thank you very much!
OpenBench is a quite powerful tool for testing engine modifications: https://github.com/AndyGrant/OpenBench

jdart
Posts: 3954
Joined: Fri Mar 10, 2006 4:23 am
Location: http://www.arasanchess.org

Re: Looking for automatic Engine Testing Software

Post by jdart » Sun Jul 19, 2020 1:16 pm

If you are running on a single machine then cutechess-cli is really all you need.

Several engine developers are using OpenBench https://github.com/AndyGrant/OpenBench, developed by Andy Grant, author of Ethereal. It is similar in functionality to Fishtest (https://github.com/glinscott/fishtest, which is what the Stockfish team is using. These support tests run on multiple machines, possibly in mulitiple locations (WAN). OpenBench makes some assumptions, for example that an engine supports the "bench" command.

I use a combination of Python and shell scripts, together with cutechess-cli, to run matches on multiple machines, monitor the results, and do the SPRT test to determine when to stop the test. This assumes all machines are on a LAN. I haven't released all the pieces of this setup, but some parts of it are in the Arasan tool directory (https://github.com/jdart1/arasan-chess/ ... ster/tools) on Github: in particular, match.py, match_status.py and monitor.py.

--Jon

Modern Times
Posts: 2569
Joined: Thu Jun 07, 2012 9:02 pm

Re: Looking for automatic Engine Testing Software

Post by Modern Times » Sun Jul 19, 2020 2:48 pm

Yes cutechess for me, although I use the GUI as I do like to see the games in progress from time to time.
.

Opinions expressed here are my own, and not necessarily those of the CCRL Group.

OliverBr
Posts: 334
Joined: Tue Dec 18, 2007 8:38 pm
Location: Munich, Germany
Full name: Dr. Oliver Brausch
Contact:

Re: Looking for automatic Engine Testing Software

Post by OliverBr » Sun Jul 19, 2020 3:58 pm

Thank you for your tips!

As far as I can tell, there are two possibilities: Cutechess or OpenBench. Both seem to provide:

- Cross platform
- Concurrent Games on multiple CPU Cores
- Xboard Protocol

What are the main differences between those two?
jdart wrote:
Sun Jul 19, 2020 1:16 pm
OpenBench makes some assumptions, for example that an engine supports the "bench" command.
Interesting. Where can I find more about those assumptions?
Chess Engine OliThink: http://brausch.org/home/chess

jdart
Posts: 3954
Joined: Fri Mar 10, 2006 4:23 am
Location: http://www.arasanchess.org

Re: Looking for automatic Engine Testing Software

Post by jdart » Sun Jul 19, 2020 4:20 pm

Cutechess-cli provides the functionality to run matches, with many options.

OpenBench uses cutechess-cli for that layer of functionality but adds a UI on top and coordination across multiple machines, among other features. There is some basic documentation for it on the Github page. See "Engine Compliance with Openbench" section for what requirements are placed on engines.

--Jon

chrisw
Posts: 3366
Joined: Tue Apr 03, 2012 2:28 pm

Re: Looking for automatic Engine Testing Software

Post by chrisw » Sun Jul 19, 2020 5:04 pm

jdart wrote:
Sun Jul 19, 2020 4:20 pm
Cutechess-cli provides the functionality to run matches, with many options.

OpenBench uses cutechess-cli for that layer of functionality but adds a UI on top and coordination across multiple machines, among other features. There is some basic documentation for it on the Github page. See "Engine Compliance with Openbench" section for what requirements are placed on engines.

--Jon
I don’t think you can have a private engine on that system. Which means basically you are forced into some kind if GPL licence which some people may not want to do

jdart
Posts: 3954
Joined: Fri Mar 10, 2006 4:23 am
Location: http://www.arasanchess.org

Re: Looking for automatic Engine Testing Software

Post by jdart » Sun Jul 19, 2020 5:41 pm

I don’t think you can have a private engine on that system. Which means basically you are forced into some kind if GPL licence which some people may not want to do
Not so. You can run your own private server. And in any case, you are not linking with OpenBench, or distributing your engine along with it, so its license provisions would not apply to your engine, IMO.

--Jon

chrisw
Posts: 3366
Joined: Tue Apr 03, 2012 2:28 pm

Re: Looking for automatic Engine Testing Software

Post by chrisw » Sun Jul 19, 2020 6:21 pm

jdart wrote:
Sun Jul 19, 2020 5:41 pm
I don’t think you can have a private engine on that system. Which means basically you are forced into some kind if GPL licence which some people may not want to do
Not so. You can run your own private server. And in any case, you are not linking with OpenBench, or distributing your engine along with it, so its license provisions would not apply to your engine, IMO.

--Jon
I thought the idea was that engines were spread out over participating users PC and ran on those. A puts his engine on the system for testing and (unknown?) users XYZ have the engine on their system at home. Seems a pretty good way to get your engine “distributed”, willing or not. Or am I missing something.

Post Reply