How to properly test new/changed opening books?

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
phhnguyen
Posts: 286
Joined: Wed Apr 21, 2010 2:58 am
Location: Australia
Full name: Nguyen Hong Pham
Contact:

How to properly test new/changed opening books?

Post by phhnguyen » Wed Jun 13, 2018 11:07 am

Please share some your experiences/methodologies to test new/twisted opening books.

I have just created a new opening book from carefully-selected games but not sure if it’s better than old ones.

Thanks

Ferdy
Posts: 3620
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: How to properly test new/changed opening books?

Post by Ferdy » Wed Jun 13, 2018 2:28 pm

phhnguyen wrote:
Wed Jun 13, 2018 11:07 am
Please share some your experiences/methodologies to test new/twisted opening books.

I have just created a new opening book from carefully-selected games but not sure if it’s better than old ones.

Thanks
You can try to play some games say 100 games or more. Get 2 same engines say sf9, one using the old book and the other using the new book. If new book performs above 50% then it is better.

Verify it by conductiong additional game tests using other book or books.
sf9_newbook vs sf9_otherbook, 100 games or more
sf9_oldbook vs sf9_otherbook, 100 games or more

if sf9_newbook performs better than sf9_oldbook against common different book then it is really better.

You may test it on two diffeent TC (Time Control), say 1m+1s and 5m+1s, there can be book lines that are good only on fast TC.

phhnguyen
Posts: 286
Joined: Wed Apr 21, 2010 2:58 am
Location: Australia
Full name: Nguyen Hong Pham
Contact:

Re: How to properly test new/changed opening books?

Post by phhnguyen » Thu Jun 14, 2018 12:04 am

Thanks for your answer.
Ferdy wrote:
Wed Jun 13, 2018 2:28 pm
You can try to play some games say 100 games or more. Get 2 same engines say sf9, one using the old book and the other using the new book. If new book performs above 50% then it is better.
AFAIK, many engines such as sf (and mine) don't use their own books. In other hand, many popular GUI such as XBoard, Cutechess... don't allow to use more than one opening books.

How do you organise the test for two books?

I may add more code to my engines to use books, but it takes time, labor and I don't think my engine is good enough to confirm a given book is good or not.
Ferdy wrote:
Wed Jun 13, 2018 2:28 pm
Verify it by conductiong additional game tests using other book or books.
sf9_newbook vs sf9_otherbook, 100 games or more
sf9_oldbook vs sf9_otherbook, 100 games or more
I don't have experience of book testing, but I am confused about the number 100 since it is too small to for a test. Are you sure it is enough?
Ferdy wrote:
Wed Jun 13, 2018 2:28 pm
if sf9_newbook performs better than sf9_oldbook against common different book then it is really better.

You may test it on two diffeent TC (Time Control), say 1m+1s and 5m+1s, there can be book lines that are good only on fast TC.
Some issues / questions still confuse me, need some suggestions:
- Results of games for a given opening line are enough to confirm that opening line is good? I am confused since for very long games it is hard to say how their opening lines affect to results.
- If I always select the best answers from openings, look like I can test only one opening line. If I select randomly (to test many opening lines), the results may have too much noise to make any conclusion

Ferdy
Posts: 3620
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: How to properly test new/changed opening books?

Post by Ferdy » Thu Jun 14, 2018 4:30 am

phhnguyen wrote:
Thu Jun 14, 2018 12:04 am
Thanks for your answer.
Ferdy wrote:
Wed Jun 13, 2018 2:28 pm
You can try to play some games say 100 games or more. Get 2 same engines say sf9, one using the old book and the other using the new book. If new book performs above 50% then it is better.
AFAIK, many engines such as sf (and mine) don't use their own books. In other hand, many popular GUI such as XBoard, Cutechess... don't allow to use more than one opening books.

How do you organise the test for two books?
SF can use polyglot book, most if not all uci engines can use polyglot book. In my reply I assume that you use polyglot book sorry. Most GUI's support 2 modes of opening book. (1) GUI opening book and (2) Engine opening book. You can disable (1) and use book via (2).

You can setup polyglot to act as an engine and use a book of your own choice. You need polyglot.ini, polyglot.exe and uci engine say SF9

Sample polyglot.ini

Code: Select all

[PolyGlot]
EngineCommand=SF9.exe
EngineDir=.
Book=true
BookFile=myoldbook.bin
[Engine]
Hash=128
In that setting, SF9 will use myoldbook.bin book.

1. Create a dir and name it engine1 and copy polyglot.exe, polyglot.ini and SF9.exe
2. Create another dir and name it engine2 and copy polyglot.exe, polyglot.ini and SF9.exe
In this case use a different polyglot.ini entry.
BookFile=mynewbook.bin
You can then install the engine by selecting polyglot.exe.
phhnguyen wrote: I may add more code to my engines to use books, but it takes time, labor and I don't think my engine is good enough to confirm a given book is good or not.
Ferdy wrote:
Wed Jun 13, 2018 2:28 pm
Verify it by conductiong additional game tests using other book or books.
sf9_newbook vs sf9_otherbook, 100 games or more
sf9_oldbook vs sf9_otherbook, 100 games or more
I don't have experience of book testing, but I am confused about the number 100 since it is too small to for a test. Are you sure it is enough?
You can run test games and run ordo or bayeselo program to see if the lead of new book is statistically significant.
phhnguyen wrote:
Ferdy wrote:
Wed Jun 13, 2018 2:28 pm
if sf9_newbook performs better than sf9_oldbook against common different book then it is really better.

You may test it on two diffeent TC (Time Control), say 1m+1s and 5m+1s, there can be book lines that are good only on fast TC.
Some issues / questions still confuse me, need some suggestions:
- Results of games for a given opening line are enough to confirm that opening line is good? I am confused since for very long games it is hard to say how their opening lines affect to results.
Go for the basic thing, If newbook won in a given specific line then it is good because it won that line. If you dig deeper, check the game and examine each move score, analyze each position, does its opponent blunders? For long TC game, the book line is more reliable, because each engine has more time to think on the move.
phhnguyen wrote:- If I always select the best answers from openings, look like I can test only one opening line.
Correct, this is why it is recommended to test your best lines against other books, not just your oldbook. Those other books may have different line continuation that might equalize or worst defeat your best line.
phhnguyen wrote: If I select randomly (to test many opening lines), the results may have too much noise to make any conclusion
Running more test games is better here to see performance of your book when run on random mode.

fantasmadel50
Posts: 112
Joined: Thu Apr 30, 2015 5:36 pm

Re: How to properly test new/changed opening books?

Post by fantasmadel50 » Wed Jun 27, 2018 2:00 pm

Image




Would this be correct, so that my engine would take the book .bin?

The folder is located in c: / desktop / New folder


Thanks for your help.

fantasmadel50
Posts: 112
Joined: Thu Apr 30, 2015 5:36 pm

Re: How to properly test new/changed opening books?

Post by fantasmadel50 » Wed Jun 27, 2018 2:03 pm

This is the .INI file

Image

User avatar
hgm
Posts: 22572
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: How to properly test new/changed opening books?

Post by hgm » Wed Jun 27, 2018 2:05 pm

When an engine is UCI, you can run it through Polyglot, and then let Polyglot handle the book. That way the book is only used for a single engine.

If you use WinBoard/XBoard you can set the Polyglot book for the individual engines through their Engine #N Settings dialog. You would only have a problem if both engines are WB, and both engines do not support an own book. (Traditionally WB engines hanle their books themselves.)

Post Reply