Page 1 of 3

SIMEX 2.1

Posted: Sun Sep 01, 2019 5:44 pm
by Rebel
Made an update, changes:

1. New MEA that fixes a speed issue when running under Win10. MEA was extremely slow, is OK now and even runs faster on Win 7 systems.

2. Cosmetic changes to the HTML output that makes it better readable. It will no longer mark versions of the same engine as orange (similarity 60-64%) or red (65% or higher) by checking the first 2 characters of engines.

Example: http://rebel13.nl/html/ccrl.html

41 high rated CCRL engines and no single 60% found!

One sane list.

My congratulations to the current chess programmers of these engines, especially the new kids on the block while so many strong open source engines are available on Github.

Download SIMEX 2.1 - http://rebel13.nl/dl/simex2.7z

Documentation - http://rebel13.nl/misc/simex.html

Re: SIMEX 2.1

Posted: Mon Sep 02, 2019 10:16 am
by Ovyron
Rebel wrote: Sun Sep 01, 2019 5:44 pm41 high rated CCRL engines and no single 60% found!
I see similarities that go beyond that on the Stockfish versions, yet they're not colored yellow as the bottom of the page seem to claim they should be?

Re: SIMEX 2.1

Posted: Mon Sep 02, 2019 11:56 am
by Guenther
Rebel wrote: Sun Sep 01, 2019 5:44 pm Made an update, changes:

1. New MEA that fixes a speed issue when running under Win10. MEA was extremely slow, is OK now and even runs faster on Win 7 systems.

2. Cosmetic changes to the HTML output that makes it better readable. It will no longer mark versions of the same engine as orange (similarity 60-64%) or red (65% or higher) by checking the first 2 characters of engines.

Example: http://rebel13.nl/html/ccrl.html

...
Something seems wrong with the results of Pedone 1.8. I guess it did not obey the search time/depth?
Senpai 1 vs. 2 looks strange too and a few other results also seem systematically too low?

Re: SIMEX 2.1

Posted: Mon Sep 02, 2019 4:58 pm
by Rebel
Guenther wrote: Mon Sep 02, 2019 11:56 am
Rebel wrote: Sun Sep 01, 2019 5:44 pm Made an update, changes:

1. New MEA that fixes a speed issue when running under Win10. MEA was extremely slow, is OK now and even runs faster on Win 7 systems.

2. Cosmetic changes to the HTML output that makes it better readable. It will no longer mark versions of the same engine as orange (similarity 60-64%) or red (65% or higher) by checking the first 2 characters of engines.

Example: http://rebel13.nl/html/ccrl.html

...
Something seems wrong with the results of Pedone 1.8. I guess it did not obey the search time/depth?
Senpai 1 vs. 2 looks strange too
I double checked both Pedone and Senpai, the (odd looking) numbers are correct. There are 2 ways to check, the log-file of each or even more simple the created epd in the EPD_OUT folder. EPD's should have "acd 1". If not the engine in question ignores the depth=1 setting. For instance Laser and Booot, they shouldn't be in the overview, I noticed too late.

Re: SIMEX 2.1

Posted: Mon Sep 02, 2019 5:00 pm
by Rebel
Ovyron wrote: Mon Sep 02, 2019 10:16 am
Rebel wrote: Sun Sep 01, 2019 5:44 pm41 high rated CCRL engines and no single 60% found!
I see similarities that go beyond that on the Stockfish versions, yet they're not colored yellow as the bottom of the page seem to claim they should be?
There are some irregularities, I will post them later.

Re: SIMEX 2.1

Posted: Mon Sep 02, 2019 5:07 pm
by jorose
Does Winter already support such a tool? What do I need to add to an engine in order to support a tool like SIMEX?

I'm kind of curious what engines Winter (especially Winter dev) is related to :D

EDIT: I'm trying things out in WINE now. I get the following error at the moment:

Can't recognize 'sort DATASIMEX.DATA /rec 65000 /o DATASIMEX.DATA' as an internal or external command, or batch script.

Re: SIMEX 2.1

Posted: Mon Sep 02, 2019 5:29 pm
by chrisw
Rebel wrote: Sun Sep 01, 2019 5:44 pm Made an update, changes:

1. New MEA that fixes a speed issue when running under Win10. MEA was extremely slow, is OK now and even runs faster on Win 7 systems.

2. Cosmetic changes to the HTML output that makes it better readable. It will no longer mark versions of the same engine as orange (similarity 60-64%) or red (65% or higher) by checking the first 2 characters of engines.

Example: http://rebel13.nl/html/ccrl.html

41 high rated CCRL engines and no single 60% found!

One sane list.

My congratulations to the current chess programmers of these engines, especially the new kids on the block while so many strong open source engines are available on Github.

Download SIMEX 2.1 - http://rebel13.nl/dl/simex2.7z

Documentation - http://rebel13.nl/misc/simex.html
what is the origin of the testing suite? who composed it, or did it just grow, and when was it composed? Do we know how it was composed?

Re: SIMEX 2.1

Posted: Mon Sep 02, 2019 7:16 pm
by Rebel
jorose wrote: Mon Sep 02, 2019 5:07 pm Does Winter already support such a tool? What do I need to add to an engine in order to support a tool like SIMEX?

I'm kind of curious what engines Winter (especially Winter dev) is related to :D

EDIT: I'm trying things out in WINE now. I get the following error at the moment:

Can't recognize 'sort DATASIMEX.DATA /rec 65000 /o DATASIMEX.DATA' as an internal or external command, or batch script.
I tried Winter, did not work. I also suspect MicroSoft sort.exe won't work under WINE, but I am no expert.

Re: SIMEX 2.1

Posted: Mon Sep 02, 2019 7:41 pm
by Rebel
chrisw wrote: Mon Sep 02, 2019 5:29 pmwhat is the origin of the testing suite? who composed it, or did it just grow, and when was it composed? Do we know how it was composed?
Don isn't longer among us, but maybe Larry can tell.

My best guess is that the 8238 positions were derived from comp-comp games. If you scroll through the simex epd's in the epd_out folder scores reasonable well fit a -1.00/+1.00 window. Also I tried a dozen position searches with 2.9 million games, no matches except known openings.

Re: SIMEX 2.1

Posted: Mon Sep 02, 2019 10:49 pm
by chrisw
Rebel wrote: Mon Sep 02, 2019 7:41 pm
chrisw wrote: Mon Sep 02, 2019 5:29 pmwhat is the origin of the testing suite? who composed it, or did it just grow, and when was it composed? Do we know how it was composed?
Don isn't longer among us, but maybe Larry can tell.

My best guess is that the 8238 positions were derived from comp-comp games. If you scroll through the simex epd's in the epd_out folder scores reasonable well fit a -1.00/+1.00 window. Also I tried a dozen position searches with 2.9 million games, no matches except known openings.
There's quite a skew in the test positions, many have all the pieces on the board. Take a look, histograms of piece count, and piece weight for the epd set.
simex-8352-histogram-piece-counts.png
simex-8352-histogram-piece-weights.png
was it Don who chose them? Why early game positions?

Edit: I should learn to label things correct! 8328 it should say.