Please share some your experiences/methodologies to test new/twisted opening books.
I have just created a new opening book from carefully-selected games but not sure if it’s better than old ones.
Thanks
How to properly test new/changed opening books?
Moderators: hgm, Rebel, chrisw
-
- Posts: 1437
- Joined: Wed Apr 21, 2010 4:58 am
- Location: Australia
- Full name: Nguyen Hong Pham
How to properly test new/changed opening books?
https://banksiagui.com
The most features chess GUI, based on opensource Banksia - the chess tournament manager
The most features chess GUI, based on opensource Banksia - the chess tournament manager
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: How to properly test new/changed opening books?
You can try to play some games say 100 games or more. Get 2 same engines say sf9, one using the old book and the other using the new book. If new book performs above 50% then it is better.
Verify it by conductiong additional game tests using other book or books.
sf9_newbook vs sf9_otherbook, 100 games or more
sf9_oldbook vs sf9_otherbook, 100 games or more
if sf9_newbook performs better than sf9_oldbook against common different book then it is really better.
You may test it on two diffeent TC (Time Control), say 1m+1s and 5m+1s, there can be book lines that are good only on fast TC.
-
- Posts: 1437
- Joined: Wed Apr 21, 2010 4:58 am
- Location: Australia
- Full name: Nguyen Hong Pham
Re: How to properly test new/changed opening books?
Thanks for your answer.
How do you organise the test for two books?
I may add more code to my engines to use books, but it takes time, labor and I don't think my engine is good enough to confirm a given book is good or not.
- Results of games for a given opening line are enough to confirm that opening line is good? I am confused since for very long games it is hard to say how their opening lines affect to results.
- If I always select the best answers from openings, look like I can test only one opening line. If I select randomly (to test many opening lines), the results may have too much noise to make any conclusion
AFAIK, many engines such as sf (and mine) don't use their own books. In other hand, many popular GUI such as XBoard, Cutechess... don't allow to use more than one opening books.
How do you organise the test for two books?
I may add more code to my engines to use books, but it takes time, labor and I don't think my engine is good enough to confirm a given book is good or not.
I don't have experience of book testing, but I am confused about the number 100 since it is too small to for a test. Are you sure it is enough?
Some issues / questions still confuse me, need some suggestions:
- Results of games for a given opening line are enough to confirm that opening line is good? I am confused since for very long games it is hard to say how their opening lines affect to results.
- If I always select the best answers from openings, look like I can test only one opening line. If I select randomly (to test many opening lines), the results may have too much noise to make any conclusion
https://banksiagui.com
The most features chess GUI, based on opensource Banksia - the chess tournament manager
The most features chess GUI, based on opensource Banksia - the chess tournament manager
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: How to properly test new/changed opening books?
SF can use polyglot book, most if not all uci engines can use polyglot book. In my reply I assume that you use polyglot book sorry. Most GUI's support 2 modes of opening book. (1) GUI opening book and (2) Engine opening book. You can disable (1) and use book via (2).
You can setup polyglot to act as an engine and use a book of your own choice. You need polyglot.ini, polyglot.exe and uci engine say SF9
Sample polyglot.ini
Code: Select all
[PolyGlot]
EngineCommand=SF9.exe
EngineDir=.
Book=true
BookFile=myoldbook.bin
[Engine]
Hash=128
1. Create a dir and name it engine1 and copy polyglot.exe, polyglot.ini and SF9.exe
2. Create another dir and name it engine2 and copy polyglot.exe, polyglot.ini and SF9.exe
In this case use a different polyglot.ini entry.
BookFile=mynewbook.bin
You can then install the engine by selecting polyglot.exe.
You can run test games and run ordo or bayeselo program to see if the lead of new book is statistically significant.phhnguyen wrote: I may add more code to my engines to use books, but it takes time, labor and I don't think my engine is good enough to confirm a given book is good or not.
I don't have experience of book testing, but I am confused about the number 100 since it is too small to for a test. Are you sure it is enough?
Go for the basic thing, If newbook won in a given specific line then it is good because it won that line. If you dig deeper, check the game and examine each move score, analyze each position, does its opponent blunders? For long TC game, the book line is more reliable, because each engine has more time to think on the move.phhnguyen wrote:Some issues / questions still confuse me, need some suggestions:
- Results of games for a given opening line are enough to confirm that opening line is good? I am confused since for very long games it is hard to say how their opening lines affect to results.
Correct, this is why it is recommended to test your best lines against other books, not just your oldbook. Those other books may have different line continuation that might equalize or worst defeat your best line.phhnguyen wrote:- If I always select the best answers from openings, look like I can test only one opening line.
Running more test games is better here to see performance of your book when run on random mode.phhnguyen wrote: If I select randomly (to test many opening lines), the results may have too much noise to make any conclusion
-
- Posts: 112
- Joined: Thu Apr 30, 2015 7:36 pm
Re: How to properly test new/changed opening books?
Would this be correct, so that my engine would take the book .bin?
The folder is located in c: / desktop / New folder
Thanks for your help.
-
- Posts: 112
- Joined: Thu Apr 30, 2015 7:36 pm
Re: How to properly test new/changed opening books?
This is the .INI file
-
- Posts: 27811
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: How to properly test new/changed opening books?
When an engine is UCI, you can run it through Polyglot, and then let Polyglot handle the book. That way the book is only used for a single engine.
If you use WinBoard/XBoard you can set the Polyglot book for the individual engines through their Engine #N Settings dialog. You would only have a problem if both engines are WB, and both engines do not support an own book. (Traditionally WB engines hanle their books themselves.)
If you use WinBoard/XBoard you can set the Polyglot book for the individual engines through their Engine #N Settings dialog. You would only have a problem if both engines are WB, and both engines do not support an own book. (Traditionally WB engines hanle their books themselves.)