Looking for automatic Engine Testing Software

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
OliverBr
Posts: 334
Joined: Tue Dec 18, 2007 8:38 pm
Location: Munich, Germany
Full name: Dr. Oliver Brausch
Contact:

Re: Looking for automatic Engine Testing Software

Post by OliverBr » Sun Jul 19, 2020 10:40 pm

For Mac Users:

Code: Select all

brew install qt
echo 'export PATH="/usr/local/opt/qt/bin:$PATH"' >> ~/.bash_profile
then new Terminal Window:

Code: Select all

git clone https://github.com/cutechess/cutechess.git
cd cutechess
qmake -spec macx-g++ -config static
make
cp projects/cli/cutechess-cli /usr/local/bin/
let them play:

Code: Select all

cutechess-cli -engine cmd=olithink557a -engine cmd=../olithink557 -each proto=xboard tc=40/60 -rounds 10 -concurrency 2
Many thanks and great respect for the creators of cutechess. Open source rules!
Chess Engine OliThink: http://brausch.org/home/chess

User avatar
hgm
Posts: 24651
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: Looking for automatic Engine Testing Software

Post by hgm » Mon Jul 20, 2020 6:45 am

If you want to be able to watch the games, you could also use XBoard. (Available for Linux and Mac.)

User avatar
xr_a_y
Posts: 1184
Joined: Sat Nov 25, 2017 1:28 pm
Location: France

Re: Looking for automatic Engine Testing Software

Post by xr_a_y » Mon Jul 20, 2020 9:24 am

chrisw wrote:
Sun Jul 19, 2020 6:21 pm
jdart wrote:
Sun Jul 19, 2020 5:41 pm
I don’t think you can have a private engine on that system. Which means basically you are forced into some kind if GPL licence which some people may not want to do
Not so. You can run your own private server. And in any case, you are not linking with OpenBench, or distributing your engine along with it, so its license provisions would not apply to your engine, IMO.

--Jon
I thought the idea was that engines were spread out over participating users PC and ran on those. A puts his engine on the system for testing and (unknown?) users XYZ have the engine on their system at home. Seems a pretty good way to get your engine “distributed”, willing or not. Or am I missing something.
You can host your own openbench server and client on your own set of machines. Nobody else will see your dev or fill your hardware, but you also won't be able to use others resources ...

For Minic I've build a docker image for this : https://github.com/tryingsomestuff/Mini ... ter/Docker

OliverBr
Posts: 334
Joined: Tue Dec 18, 2007 8:38 pm
Location: Munich, Germany
Full name: Dr. Oliver Brausch
Contact:

Re: Looking for automatic Engine Testing Software

Post by OliverBr » Mon Jul 20, 2020 3:22 pm

hgm wrote:
Mon Jul 20, 2020 6:45 am
If you want to be able to watch the games, you could also use XBoard. (Available for Linux and Mac.)
Thank you for this tip. I am using xboard, but it can't give me the numbers.
Cutechess works well. Today, I have ordered a 32core server :)
Chess Engine OliThink: http://brausch.org/home/chess

OliverBr
Posts: 334
Joined: Tue Dec 18, 2007 8:38 pm
Location: Munich, Germany
Full name: Dr. Oliver Brausch
Contact:

Re: Looking for automatic Engine Testing Software

Post by OliverBr » Mon Jul 20, 2020 3:38 pm

Hello again,

now I am looking for a nice pgn file with a lot of diversified openings. Do you have tips?

Thank you!

EDIT: I found already the Silver Suite, it's very very nice for tests of 100-200 matches. I have created something similar also with 50 openings, it was a lot of work.
How can I attach this here? I have just added this pgn file to my binaries of Version 5.5.8.

Is there something similar, but larger, suitable for 1000-2000 games?
Last edited by OliverBr on Mon Jul 20, 2020 3:54 pm, edited 2 times in total.
Chess Engine OliThink: http://brausch.org/home/chess

User avatar
xr_a_y
Posts: 1184
Joined: Sat Nov 25, 2017 1:28 pm
Location: France

Re: Looking for automatic Engine Testing Software

Post by xr_a_y » Mon Jul 20, 2020 3:49 pm

OliverBr wrote:
Mon Jul 20, 2020 3:38 pm
Hello again,

now I am looking for a nice pgn file with a lot of diversified openings. Do you have tips?

Thank you!
viewtopic.php?f=7&t=71469&p=808135&hili ... et#p808135

brianr
Posts: 422
Joined: Thu Mar 09, 2006 2:01 pm

Re: Looking for automatic Engine Testing Software

Post by brianr » Mon Jul 20, 2020 4:24 pm

OliverBr wrote:
Mon Jul 20, 2020 3:22 pm
Thank you for this tip. I am using xboard, but it can't give me the numbers.
If you are serious about testing enough to order a 32 core box, suggest using Ordo in addition to other tools.
Sample command:
./ordo -Q -N 0 -D -a 0 -A "nameofengineusedforzeroElo" -W -n4 -s500 -U "0,1,2,3,4,5,7,8,9,10,6" -p your.pgn

If you stop/restart cutechess-cli matches before normal completion, there are games with '*' results, which Ordo does not like.
For that, just use SCID. Import pgn, search headers to exclude '*' result games, then export pgn.

brianr
Posts: 422
Joined: Thu Mar 09, 2006 2:01 pm

Re: Looking for automatic Engine Testing Software

Post by brianr » Mon Jul 20, 2020 4:33 pm

Here are some books I like:
book_3moves_zz4032.zip in the Leela Discord pinned in #test-discuss under zz4032 has breakdowns by SF evals sorted by cp ranges.
The 8moves_v3.pgn is from Fishtest, IIRC.


AndrewGrant
Posts: 588
Joined: Tue Apr 19, 2016 4:08 am
Location: U.S.A
Full name: Andrew Grant
Contact:

Re: Looking for automatic Engine Testing Software

Post by AndrewGrant » Mon Jul 20, 2020 7:33 pm

chrisw wrote:
Sun Jul 19, 2020 5:04 pm
jdart wrote:
Sun Jul 19, 2020 4:20 pm
Cutechess-cli provides the functionality to run matches, with many options.

OpenBench uses cutechess-cli for that layer of functionality but adds a UI on top and coordination across multiple machines, among other features. There is some basic documentation for it on the Github page. See "Engine Compliance with Openbench" section for what requirements are placed on engines.

--Jon
I don’t think you can have a private engine on that system. Which means basically you are forced into some kind if GPL licence which some people may not want to do
Anyone is free to run a private instance of OpenBench. The only possible snag is that OpenBench pulls from Github. So you will need to copy paste some code for authenticating credentials and use a private repository. Alternatively, host your own git server as well.

Post Reply