On Opening books in 2015

[Peter Berger]

I have thought about this for a considerable amount of time theoretically, and I've also invested some time and effort by now to verify that my thoughts cannot be falsified (too) easily.

This is about slow time controls and conditions similar to those in an ICGA WCCC event - sth like 3min/move on reasonably fast hardware ( aka your Laptop ).

Simplified claim: Opening books are pretty useless these days. E.g. Stockfish don't need no book at slow time controls.

All my personal games and tests were done with current/recent versions of Stockfish against various opponents. I lack the resources to reach anything statistically significant this way, but I still want to share my impressions as this somehow goes against intuition and might be interesting - and there are others who do better testing work than me.

Kai Laskos shared some of his findings on opening book performance at very fast time controls in the WCCC 2015 thread in "Tournaments and Matches" , and they suggest that there still is a significant book impact on engine performance.

I personally think this will turn out to be just an illusion if verified at slower time controls. We need some variety from game to game to deal with aggressive learners of course, but downloading new development versions of Stockfish as they come out seems to provide enough of that.

I can/could elaborate on this, but then I'd never get this posted, maybe more later.

Peter

[gerold]

For a long time i have used very small opening books. Just so i get many different openings. I like H.S. old short books. They still work good for me.

I do not use endgame tablebase. I like to watch the engines play their own endings.

[Dann Corbit]

I have thought about this for a considerable amount of time theoretically, and I've also invested some time and effort by now to verify that my thoughts cannot be falsified (too) easily.

This is about slow time controls and conditions similar to those in an ICGA WCCC event - sth like 3min/move on reasonably fast hardware ( aka your Laptop ).

Simplified claim: Opening books are pretty useless these days. E.g. Stockfish don't need no book at slow time controls.

All my personal games and tests were done with current/recent versions of Stockfish against various opponents. I lack the resources to reach anything statistically significant this way, but I still want to share my impressions as this somehow goes against intuition and might be interesting - and there are others who do better testing work than me.

Kai Laskos shared some of his findings on opening book performance at very fast time controls in the WCCC 2015 thread in "Tournaments and Matches" , and they suggest that there still is a significant book impact on engine performance.

I personally think this will turn out to be just an illusion if verified at slower time controls. We need some variety from game to game to deal with aggressive learners of course, but downloading new development versions of Stockfish as they come out seems to provide enough of that.

I can/could elaborate on this, but then I'd never get this posted, maybe more later.

Peter

If there were a test included in this list with no opening book, then we would know the answer:
http://www.chess2u.com/t7735p60-bookswa ... ated#64195

[jdart]

I disagree. I have looked at TCEC games for example (long time control) and I see a lot of mistakes in the opening, some of them serious.

For example take this position from the Marshall Gambit:

[d] r1b2rk1/5p1p/p1pb4/1p1n2pq/3PR3/1BP3P1/PP1N1P1P/R1B2QK1 b - - 0 17

The main line goes Bf5! f3 Nf6!. f5 is usually what engines choose and it is playable, but usually considered inferior after Bd1!. Engines don't like Bf5 at all.

Marshall games are usually won or lost in the endgame and the opening is just full of decision points where you get a good endgame if you go the right way and a bad endgame if you don't. So I think book knowledge is still important, at least if you are not in correspondence where your search time is in days.

--Jon

[Uri Blass]

<snipped>

I have thought about this for a considerable amount of time theoretically, and I've also invested some time and effort by now to verify that my thoughts cannot be falsified (too) easily.

This is about slow time controls and conditions similar to those in an ICGA WCCC event - sth like 3min/move on reasonably fast hardware ( aka your Laptop ).

Simplified claim: Opening books are pretty useless these days. E.g. Stockfish don't need no book at slow time controls.

Peter

I disagree and if you build opening book correctly the book can be useful also in 3 minutes per move.

I have no idea how people build opening book but the logical way to do it is simply to give program like stockfish or komodo a lot of time in every position that you want to add to the book and add the move to the opening book.

people already did the work of using a lot of time in let is check and you only need to collect the data
It is possible to use the following site for this information because it has analysis with a lot of time for many positions so you can decide simply to add every position with stockfish and depth of at least 45 and komodo and depth of at least 40 and probably get many positions in the book but we need somebody to do it manually(unfortunately the information that I see include only depth and not number of nodes but I guess that usually you cannot get depth above 45 for stockfish or above 40 for komodo in opening moves when you use 3 minutes on fast hardware).

http://database.chessbase.com/js/apps/database/

starting to write the book manually

For white
1.d4 stockfish301114 0.13/54
1.d4 Nf6 2.c4 stockfish6 0.1/47
1.d4 d5 2.c4 stockfish6 0.22/48
1.d4 e6 2.e4 stockfish6 0.13/47
1.d4 d6 2.Nf3 stockfish5 0.37/42(depth smaller than 45 so maybe better not to include it)
1.d4 f5 2.Bf4 komodo8 0.32/36(depth for komodo smaller than 40 so maybe better not to include it)
1.d4 Nf6 2.c4 e6 3.Nf3 stockfish 020215 0.14/48
1.d4 Nf6 2.c4 g6 3.Nc3 stockfish 140215 0.28/47

for black

1.e4 c5
stockfish210215 0.08/50
1.d4 Nf6
stockfish301114 0.18/52

[Laskos]

Simplified claim: Opening books are pretty useless these days. E.g. Stockfish don't need no book at slow time controls.

I once went from game in 1'+1'' on one core to game in 5'+3'' on 4 cores, and although differences between books get a bit smaller, it's due as usual to increasing draw rate. One has to keep in mind that books are not simply successions of positions analyzed by engines, and it doesn't matter how long the positions are analyzed. People use game databases to build books, for example PlayChess engine room game databases. Engines as of now are unable to discern by analysis of an opening position (even very long analysis) what that line leads to, and some statistic of played outcomes has to be used. Also, engine analysis doesn't care what sort of intermediate position is reached, tactical with a lot of possible traps, closed, fortress having weird evals, and so on. Books can be tuned to take into account all that. Monte Carlo search can offset somewhat the advantage of the books, but even where MCTS is applied (Go), books are very useful in the openings, and some Go authors spend large amounts of time building them, either by hand or via automated process.

[Lyudmil Tsvetkov]

Jon is right: engines play openings worst of all other stages, as openings are most complex.

The main point is however different. What is our main goal?

If our main goal is to just hide as well as possible imperfect/weak engine opening play, then a long and very long book is OK.

If our primary goal is to on the other hand make the engine learn how to play well/perfectly different opening lines even without a book, then the obvious option is staying with just a short or very short book, some 2,3,5-moves long, just to ensure us of the required variety of play and avoid doubles, and try gradually tuning as well as possible all out search and eval-related parameters, adding more and more knowledge if necessary in the process.

I think the second goal is much more important: not veiling engine difficulties in the opening, but, on the contrary, try to eliminate them with purposeful efforts.
It might take time, even a lot of time, in the meantime the engine will play worse in the opening than otherwise, but sure at some point that will bring its benefit with the engine being able to better assess and play openings self-sufficiently.

That is why I have always been a strong supporter of short and very short opening books: not that the engines play perfectly and do not need any help in this stage, but rather that such an approach is very helpful in the development process of the engine, which I think is what we all desire.

[Uri Blass]

I disagree with lyudmil and I think that at least in 99% of the cases engines play the right move if you give them a long time to calculate.

if there are some cases when they do mistakes you can find it by the evaluation and if white play the right move based on long analysis in every move but black get the advantage after move 15 or move 20(at least 0.1 pawns for black) then it is better to do analysis to see where white went wrong in order to fix the book.

Starting point if we build a book for white is simply play the moves the engines suggest after a long analysis for white and moves that are in the database for black.

Same idea if we build an opening book for black but in the last case I suggest to try to fix only cases when the evaluation for white is at least 0.5 pawn for white because small advantage for white does not mean that black did a mistake.

[Guenther]

I have thought about this for a considerable amount of time theoretically, and I've also invested some time and effort by now to verify that my thoughts cannot be falsified (too) easily.

This is about slow time controls and conditions similar to those in an ICGA WCCC event - sth like 3min/move on reasonably fast hardware ( aka your Laptop ;) ).

Simplified claim: Opening books are pretty useless these days. E.g. Stockfish don't need no book at slow time controls.

All my personal games and tests were done with current/recent versions of Stockfish against various opponents. I lack the resources to reach anything statistically significant this way, but I still want to share my impressions as this somehow goes against intuition and might be interesting - and there are others who do better testing work than me.

Kai Laskos shared some of his findings on opening book performance at very fast time controls in the WCCC 2015 thread in "Tournaments and Matches" , and they suggest that there still is a significant book impact on engine performance.

I personally think this will turn out to be just an illusion if verified at slower time controls. We need some variety from game to game to deal with aggressive learners of course, but downloading new development versions of Stockfish as they come out seems to provide enough of that.

I can/could elaborate on this, but then I'd never get this posted, maybe more later.

Peter

I have to disagree. I am sure that if you would play a tournament at 40/2h or similar and the opponents would know you would use NO book
they would cook opening lines which would make it very hard for you to win a single game(if the opponents were not far weaker of course).
The job of a bookmaker for standard time tournaments with may be 9 or 11 rounds is to analyze lines which are very hard to find by pure search
even in LTC games.
OTH I am quite sure that programs made a lot of progress also in the opening stage in the last years.

Guenther

[Lyudmil Tsvetkov]

I disagree with lyudmil and I think that at least in 99% of the cases engines play the right move if you give them a long time to calculate.

if there are some cases when they do mistakes you can find it by the evaluation and if white play the right move based on long analysis in every move but black get the advantage after move 15 or move 20(at least 0.1 pawns for black) then it is better to do analysis to see where white went wrong in order to fix the book.

Starting point if we build a book for white is simply play the moves the engines suggest after a long analysis for white and moves that are in the database for black.

Same idea if we build an opening book for black but in the last case I suggest to try to fix only cases when the evaluation for white is at least 0.5 pawn for white because small advantage for white does not mean that black did a mistake.

Engines might play the right move in 99% of cases in simple endgame positions, or simple middlegame positions, but not in the opening, as such positions are usually very complex, both for humans and engines to understand thoroughly.

Give current top engines not 1,2 or 5 hours thinking time, but 10 days thinking time on standard opening positions, and they still will not find the right move in at least half of the cases, I think in 70-80% of cases, based on what I have seen from engine play in openings.

It simply can not be another way: it took humans decades, and even centuries of analysis to ascertain with a fairly high degree of certainly what is the best move in specific opening positions, so how would you want engines to solve the puzzle in an extremely limited time in comparison?

Openings are very difficult to play, very complex, not so specific middlegame and endgame positions with reduced material and play opportunities.