Do you test engines with or without using a book?

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Dann Corbit, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
JohnW
Posts: 246
Joined: Wed Nov 21, 2012 11:20 pm
Location: New Hampshire

Do you test engines with or without using a book?

Post by JohnW » Wed Nov 25, 2020 3:20 pm

Just curious how most people test engines when running a match or tournament. I usually run them without.

Terje
Posts: 302
Joined: Tue Nov 19, 2019 3:34 am
Location: https://github.com/TerjeKir/weiss
Full name: Terje Kirstihagen

Re: Do you test engines with or without using a book?

Post by Terje » Wed Nov 25, 2020 3:31 pm

With no book you're just testing a handful of lines from startpos. If your engine picks moves somewhat randomly in the opening then you avoid this problem, but this is the job of an opening book, not the engine really.

JohnW
Posts: 246
Joined: Wed Nov 21, 2012 11:20 pm
Location: New Hampshire

Re: Do you test engines with or without using a book?

Post by JohnW » Wed Nov 25, 2020 4:01 pm

Terje wrote:
Wed Nov 25, 2020 3:31 pm
With no book you're just testing a handful of lines from startpos. If your engine picks moves somewhat randomly in the opening then you avoid this problem, but this is the job of an opening book, not the engine really.
But if you use a book aren't you testing from a situation that could give one side an advantage without ever putting any thought into it?

Terje
Posts: 302
Joined: Tue Nov 19, 2019 3:34 am
Location: https://github.com/TerjeKir/weiss
Full name: Terje Kirstihagen

Re: Do you test engines with or without using a book?

Post by Terje » Wed Nov 25, 2020 4:23 pm

JohnW wrote:
Wed Nov 25, 2020 4:01 pm
Terje wrote:
Wed Nov 25, 2020 3:31 pm
With no book you're just testing a handful of lines from startpos. If your engine picks moves somewhat randomly in the opening then you avoid this problem, but this is the job of an opening book, not the engine really.
But if you use a book aren't you testing from a situation that could give one side an advantage without ever putting any thought into it?
Both engines (should) get to play each side of each position. Also startpos already gives white an advantage without the engines doing anything :)

Dann Corbit
Posts: 12012
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: Do you test engines with or without using a book?

Post by Dann Corbit » Wed Nov 25, 2020 5:08 pm

JohnW wrote:
Wed Nov 25, 2020 4:01 pm
Terje wrote:
Wed Nov 25, 2020 3:31 pm
With no book you're just testing a handful of lines from startpos. If your engine picks moves somewhat randomly in the opening then you avoid this problem, but this is the job of an opening book, not the engine really.
But if you use a book aren't you testing from a situation that could give one side an advantage without ever putting any thought into it?
That depends on whether you put any thought into the book or not.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

User avatar
Ovyron
Posts: 4410
Joined: Tue Jul 03, 2007 2:30 am

Re: Do you test engines with or without using a book?

Post by Ovyron » Tue Jan 05, 2021 5:23 am

Terje wrote:
Wed Nov 25, 2020 4:23 pm
Both engines (should) get to play each side of each position. Also startpos already gives white an advantage without the engines doing anything :)
This is a myth believed by most testers and rating lists. In reality humans aren't forced to play the same openings with both colors, they play the openings they know well, and the same should happen with engines, building for them a book that makes them go into positions they understand better, to show their true strength.

This was critical on the days where Stockfish was still weaker than Rybka, Rybka was only stronger on these "repeated openings" set up, but Stockfish was clearly stronger if you guided both engines to the positions they played best (you could always force the game into positions that Stockfish played better than Rybka - even with black, but you couldn't force the game into positions where Rybka was better than Stockfish, even though on a random sample of generic openings Rybka would score better.)

Nowadays Stockfish 12+ will come up on top no matter what method you use, so the point is moot (but there could theoretically be a strongest chess entity that would be engine +book that would beat Stockfish 12+, and it'd appear weaker than that in the rating lists because of the generic openings.)

Gabor Szots
Posts: 756
Joined: Sat Jul 21, 2018 5:43 am
Location: Szentendre, Hungary
Full name: Gabor Szots

Re: Do you test engines with or without using a book?

Post by Gabor Szots » Tue Jan 05, 2021 7:39 am

We test with books (short ones) to inroduce some variety to the openings.

To play the same opening with both colours is not important, when you play hundreds of games that will even out.
Gabor Szots
CCRL testing group

supersharp77
Posts: 991
Joined: Sat Jul 05, 2014 5:54 am
Location: Southwest USA

Re: Do you test engines with or without using a book?

Post by supersharp77 » Tue Jan 05, 2021 8:47 pm

JohnW wrote:
Wed Nov 25, 2020 3:20 pm
Just curious how most people test engines when running a match or tournament. I usually run them without.
Well my opinion is you should use a book...you must anyway because many of these engines have their own books anyway (ex. Deep Junior, Arasan Pedone Crafty) so you need a book sometimes...Chessmaster (The King) plays quite well without a book some engines (Deep Junior, Open Tal Crafty) don't play their proper style with out the proper book...attacking engines need a sharp book...positional engines do better with a quiet book...LC0 plays ok without a book but plays a bit better with a appropriate book (D-4 lines) my opinion is that these books line should not go too deep (8-9 moves) plus playing the same lines over and over is quite boring to watch...The sharper and more complex the opening lines are the better it is and the more we can learn....AR :) :wink:


ps You can mix and match and create opening books in the Fritz, Chess Ok and Arena GUI without any problems..

User avatar
hgm
Posts: 25857
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: Do you test engines with or without using a book?

Post by hgm » Tue Jan 05, 2021 9:44 pm

In Fairy-Max I randomize the first few moves quite heavily. (Every move that isn't more than 50cP below the best is eligible.) So I can test it without book. That was important for me, as in most variants it plays no theory exist. And I often start with meterial odds, so that even when theory would exist, it would not apply.

If you play many opening lines, it isn't really important to play them with reversed colors. As long as you randomly assign the lines, and they are not extremely unbalanced. You don't want the latter anyway; playing games that are a certain win is just a waste of time. And if you play them with reversed colors a double waste of time.

Michel
Posts: 2198
Joined: Sun Sep 28, 2008 11:50 pm

Re: Do you test engines with or without using a book?

Post by Michel » Wed Jan 06, 2021 5:56 am

Replaying games with reversed colors reduces the variance of the test outcome (one should use the pentanomial model to correctly estimate this variance). So you need fewer games to reach a decision. This effect is quite substantial. Fishtest (which is the gold standard in engine testing) uses a very balanced book and there is still a 5% saving. With their previous slightly less balanced book it was 10%. With very unbalanced books it is much more.

The math is discussed here.

http://hardy.uhasselt.be/Fishtest/accou ... entity.pdf

This document is actually about comparing the trinomial and the pentanomial model, but this is the same problem.
Ideas=science. Simplification=engineering.
Without ideas there is nothing to simplify.

Post Reply