Opening Test Suites releases

Discussion of anything and everything relating to chess playing software and machines.

Moderator: Ras

chrisw
Posts: 4638
Joined: Tue Apr 03, 2012 4:28 pm
Location: Midi-Pyrénées
Full name: Christopher Whittington

Opening Test Suites releases

Post by chrisw »

https://github.com/ChrisWhittington/Chess-Openings-Data

Suitable for testers. Lines are active, usually with both sides having active play/counterplay. Selection criteria are highest non-draw result rate (W+L)/(W+D+L) from a large set of chess rating list computer-computer games. The idea being that no matter what the static evaluation of a line is, the important detail is the engine-engine result after playing the line. White bias is shown for each PGN, ideally we would like whitebias=0.5, but it is usually greater. Lines with whitebias > 0.7 are culled.
In practical play between strong engines I don't vouch for a much reduced draw rate, but I think you'll find the games are much more unbalanced in a play-counterplay sense. What an engine pair makes of that is up to the engine pair.

These PGNs:
require at least 50 examples each
are ranked on non-draw rate
use a whitebias filter of 0.7
use an eval filter of 200 (SF15 analysis at 1 core and 1 second)
are 11 ply deep

Complaints, criticisms, support, general comments - Use Github - Issues, that's what it's for.
chrisw
Posts: 4638
Joined: Tue Apr 03, 2012 4:28 pm
Location: Midi-Pyrénées
Full name: Christopher Whittington

Re: Opening Test Suites releases

Post by chrisw »

chrisw wrote: Sun Sep 03, 2023 1:05 pm https://github.com/ChrisWhittington/Chess-Openings-Data

Suitable for testers. Lines are active, usually with both sides having active play/counterplay. Selection criteria are highest non-draw result rate (W+L)/(W+D+L) from a large set of chess rating list computer-computer games. The idea being that no matter what the static evaluation of a line is, the important detail is the engine-engine result after playing the line. White bias is shown for each PGN, ideally we would like whitebias=0.5, but it is usually greater. Lines with whitebias > 0.7 are culled.
In practical play between strong engines I don't vouch for a much reduced draw rate, but I think you'll find the games are much more unbalanced in a play-counterplay sense. What an engine pair makes of that is up to the engine pair.

These PGNs:
require at least 50 examples each
are ranked on non-draw rate
use a whitebias filter of 0.7
use an eval filter of 200 (SF15 analysis at 1 core and 1 second)
are 11 ply deep

Complaints, criticisms, support, general comments - Use Github - Issues, that's what it's for.
added two more opening suites, at depth 12
gordonr
Posts: 235
Joined: Thu Aug 06, 2009 8:04 pm
Location: UK

Re: Opening Test Suites releases

Post by gordonr »

Thanks for sharing these files :)
bastiball
Posts: 5360
Joined: Tue Oct 20, 2020 4:18 am
Full name: Basti Dangca

Re: Opening Test Suites releases

Post by bastiball »

chrisw wrote: Sun Sep 03, 2023 1:05 pm https://github.com/ChrisWhittington/Chess-Openings-Data

Suitable for testers. Lines are active, usually with both sides having active play/counterplay. Selection criteria are highest non-draw result rate (W+L)/(W+D+L) from a large set of chess rating list computer-computer games. The idea being that no matter what the static evaluation of a line is, the important detail is the engine-engine result after playing the line. White bias is shown for each PGN, ideally we would like whitebias=0.5, but it is usually greater. Lines with whitebias > 0.7 are culled.
In practical play between strong engines I don't vouch for a much reduced draw rate, but I think you'll find the games are much more unbalanced in a play-counterplay sense. What an engine pair makes of that is up to the engine pair.

These PGNs:
require at least 50 examples each
are ranked on non-draw rate
use a whitebias filter of 0.7
use an eval filter of 200 (SF15 analysis at 1 core and 1 second)
are 11 ply deep

Complaints, criticisms, support, general comments - Use Github - Issues, that's what it's for.
Thanks a lot!
Basti Dangca
CCRL testing group
chrisw
Posts: 4638
Joined: Tue Apr 03, 2012 4:28 pm
Location: Midi-Pyrénées
Full name: Christopher Whittington

Re: Opening Test Suites releases

Post by chrisw »

For Testers and Testing groups:

On Github is a 100 line opening suite which is same as before, except it's now filtered to remove lines with an SF15 eval < 50 cp.

low-drawrate-openings-v2-q50-n100-d11-Wbiasfilter0.7-Evalfilter50-100-comp-comp.pgn

Tentative results (at TC=20+0.05, so very fast games) against my usual pool of top engines are:

Using random book, drawrate = 0.60
Using UHO book, drawrate = 0.50
Using above book, drawrate = 0.48

which indicates an improvement. Obviously nowhere near enough data, but looks like this one may be worth testing ...
If it looks any good, I'll try and generate a 1000-liner as well.

Comments welcome