On Opening books in 2015

[Ozymandias]

I know that chess engines are tested in games and not in going forward and backward so it is logical to suspect that there are going to be bugs if you go forward and backward.

Maybe it would be "logical" to expect those bugs, but my experience showed me, that the only thing you face, when going forward and backward, are hash related inconveniences you can easily get used to. Those inconveniences don't affect automated mini-maxing, you're just encountering engines weaknesses this way. It's the operator's "job" to compensate for those weaknesses, that's the knowledge he's adding to the game, his contribution. It's not a small thing.

[Uri Blass]

Another funny thing is: you test your engine's performance and improvement with long books, say 8,10,15 moves-long or even longer.

All the relevant search and eval parameters, but also the time management, are tuned to perform well after move 8 or 15. Not before that.

So how would you expect an engine, whose search and eval is tuned to statistically perform well starting late into the opening, to also perform well early into the opening, on move 3,5,6? The specific chess conditions early in the opening and in late opening are not quite the same, so an engine could play well into the early opening and find good moves, only if it has tuned its search and eval parameters with very short, 2-5 moves book, preferably shorter, for years.

How many engines out there have tuned their parameters with a very short book for years? I am afraid none.

So, when SF says d4 is the best move, I strongly doubt it, as there are much better chances it is c4 or e4.

Same from the black side: if SF suggests the French is a good option on e4, I strongly disbelieve that, as Sf has never tuned its parameters to perform well starting from the very first move.

A lot of strong GM's practically play 1.d4 so I wonder why do you think that you know better than them.

Stockfish also say that 1...c5 is the best move against 1.e4 and not 1...e6

http://database.chessbase.com/js/apps/database/

The analysis that suggest 1...e6 is at depth 48 when the analysis that suggest 1...c5 is at depth 50

And the analysis that suggests e4 is better than d4 is at depth 70.

Of course all of these estimates assume that the entire plan is followed exactly and there are no deviations in correctness of the plan at any point.

The fact that 50 ply analysis likes 1.d4 above 1.e4 seems to indicate to me that if 1.e4 is superior, then the difference is so small that no one can see perfectly far enough ahead to gain a real advantage of one over the other. In practical terms, I guess the difference is 1/100 of a pawn or so, and most chess players will lose that amount through inaccuracy in every game. So it becomes nothing more than random noise.

Now, it is true that someone familiar with 1.e4 and not familiar with 1.d4 openings will fare better with 1.e4 on average.

Perhaps that is why Fischer did so well with 1.e4 and Berliner did so well with 1.d4 -- because they liked those and were the most thoroughly versed in them.

That 50-ply analysis is inaccurate at best, for the simple reason that:

- it is done with engines
- engines do not understand the KID
- the single line that equalises for black fully after d4 is the black KID

So such an analysis could be absolutely discarded, until engines learn how to play the KID.

It is not a matter of who knows better his openings, e4 is simply better than d4, objectively, i.e. it provides white with larger advantage with a search that is not infinite.

About Fischer and Berliner, interesting why almost everyone knows who Fischer is and almost no one, apart from the specialised circles, knows who Berliner is?

I found on the internet a single game where Berliner drew Fischer, maybe there are more, please advise.

At the same time, I also found those:

http://www.chessgames.com/perl/chessgame?gid=1044152

http://www.chessgames.com/perl/chessgame?gid=1044112

http://www.chessgames.com/perl/chessgame?gid=1008418

All wins by Fischer, convincing at that.

Seems to indicate Berliner was nowhere near Fischer in chess understanding, even in Fischer's youth, at the time when Fischer still did not know if e4 or d4 is better.

If you look more carefully at the games, you will notice, that Fischer won both white games with e4, and also his black game against Berliner's d4, which should have given white comfortable advantage.

So, when I make statements, I like to stick to authorities.

No evidence that the KID equalize for black

People win games with white against the KID in correspondence games and I did not play the KID against 1.d4 in correspondence games and got a draw.

Here is an example for a correspondence game when black lost inspite of starting 1.d4 Nf6 2.c4 g6
https://www.iccf.com/game?id=417239

If the KID equalize then you could expect GM's to play it but many GM's choose different defence against 1.d4

[jefk]

[quote="Uri Blass"]
in big majority of cases chess engines have a right move after long analysis. (..)

If you think that there are many holes in a book that you build in this way when every move is at least result of one hour of analysis by stockfish then show me the holes
[/quote]

well it's up to you, but if you take an hour for every position you'll
get a very narrow book.
Although it will be solid, and indicating a repertoire but not
the very best. Because you still discard known theory, example:

lets say you have 1.d4 Nf6 2.c4 for White, and
then your opponent plays 2..g6, the Gruenfeld.
Because you expected 2...e6 you probably already
are out of book, so the engine then takes only about
a minute -in a typical online 'standard'game;
and comes up with 3.Nc3 or 3.Nf3. Not bad, but
not the best , which would be 3.f3.

Ok you say, but i also will analyze for one hour
the move after 2...g6, because this is a known
variation, the Gruenfeld. Fine, then you probably
get 3.Nc3, even an hour of analysis probably is
not enough to find the best move, 3.f3.

No big deal you say, i include also 3.f3, and then
analyze the response move for one hour, and so on.
Fine, if you proceed in such a way, you're aiming
at a book of lets say hundredthousand positions.

Now an hour for each position will take you
quite some time doesnt it.

Wouldn't it be more effective to take a database
of recent standard computer games on playchess
both comps above 2700 ? And add eg a database with
GMs one at least above 2600, and one above 2700
combine them, and then look at positions occurred
at least twice(*) in a first attempt to filter out the
weird moves. Then export only the endnodes, in Epd(*),
analyze those at eg ten minutes, or a bit more, do
some backsolving(*) and voila, you got a solid book.

In much faster time, and much better as well.
But even that would only be a start, opening
theory is still being discussed eg in the NIC
yearbooks, and new lines are continuously being
developed. Only by playing real games, analyzing
the results, and where necessary improving the
'standard' book will give you a good result.

But as engine programmer you probably can better
leave such an effort to a bookmaker if you can find one,
otherwise look for some good books elsewhere, i agree,
not those blitz books in .ctg but some good books for
standard games. And if they are broad/big enough such
'standard' books also play well in blitz, obviously.

good luck
jef

PS the tasks indicated above with (*) can be done
with an old version of bookbuilder, which still for a while is
downloadable as a tool on my site www.bookbuilder.nl
(download page, look at bb36 download); or direct link:
http://home.kpn.nl/jefkaan/DreamHC/bb36.zip
no expiration date, no spam, completely free.
NB i don't say that this method is perfect, you have
to check and finetune the result after minimax,
because one or a few odd moves or sidelines can spoil
the quality of the whole book, but in the end its
a better method than statistics.

[Lyudmil Tsvetkov]

Another funny thing is: you test your engine's performance and improvement with long books, say 8,10,15 moves-long or even longer.

All the relevant search and eval parameters, but also the time management, are tuned to perform well after move 8 or 15. Not before that.

So how would you expect an engine, whose search and eval is tuned to statistically perform well starting late into the opening, to also perform well early into the opening, on move 3,5,6? The specific chess conditions early in the opening and in late opening are not quite the same, so an engine could play well into the early opening and find good moves, only if it has tuned its search and eval parameters with very short, 2-5 moves book, preferably shorter, for years.

How many engines out there have tuned their parameters with a very short book for years? I am afraid none.

So, when SF says d4 is the best move, I strongly doubt it, as there are much better chances it is c4 or e4.

Same from the black side: if SF suggests the French is a good option on e4, I strongly disbelieve that, as Sf has never tuned its parameters to perform well starting from the very first move.

A lot of strong GM's practically play 1.d4 so I wonder why do you think that you know better than them.

Stockfish also say that 1...c5 is the best move against 1.e4 and not 1...e6

http://database.chessbase.com/js/apps/database/

The analysis that suggest 1...e6 is at depth 48 when the analysis that suggest 1...c5 is at depth 50

And the analysis that suggests e4 is better than d4 is at depth 70.

Of course all of these estimates assume that the entire plan is followed exactly and there are no deviations in correctness of the plan at any point.

The fact that 50 ply analysis likes 1.d4 above 1.e4 seems to indicate to me that if 1.e4 is superior, then the difference is so small that no one can see perfectly far enough ahead to gain a real advantage of one over the other. In practical terms, I guess the difference is 1/100 of a pawn or so, and most chess players will lose that amount through inaccuracy in every game. So it becomes nothing more than random noise.

Now, it is true that someone familiar with 1.e4 and not familiar with 1.d4 openings will fare better with 1.e4 on average.

Perhaps that is why Fischer did so well with 1.e4 and Berliner did so well with 1.d4 -- because they liked those and were the most thoroughly versed in them.

That 50-ply analysis is inaccurate at best, for the simple reason that:

- it is done with engines
- engines do not understand the KID
- the single line that equalises for black fully after d4 is the black KID

So such an analysis could be absolutely discarded, until engines learn how to play the KID.

It is not a matter of who knows better his openings, e4 is simply better than d4, objectively, i.e. it provides white with larger advantage with a search that is not infinite.

About Fischer and Berliner, interesting why almost everyone knows who Fischer is and almost no one, apart from the specialised circles, knows who Berliner is?

I found on the internet a single game where Berliner drew Fischer, maybe there are more, please advise.

At the same time, I also found those:

http://www.chessgames.com/perl/chessgame?gid=1044152

http://www.chessgames.com/perl/chessgame?gid=1044112

http://www.chessgames.com/perl/chessgame?gid=1008418

All wins by Fischer, convincing at that.

Seems to indicate Berliner was nowhere near Fischer in chess understanding, even in Fischer's youth, at the time when Fischer still did not know if e4 or d4 is better.

If you look more carefully at the games, you will notice, that Fischer won both white games with e4, and also his black game against Berliner's d4, which should have given white comfortable advantage.

So, when I make statements, I like to stick to authorities.

No evidence that the KID equalize for black

People win games with white against the KID in correspondence games and I did not play the KID against 1.d4 in correspondence games and got a draw.

Here is an example for a correspondence game when black lost inspite of starting 1.d4 Nf6 2.c4 g6
https://www.iccf.com/game?id=417239

If the KID equalize then you could expect GM's to play it but many GM's choose different defence against 1.d4

Probably not the time for me to comment anymore on this, as everything already seems to be said, but:

- what kind of GMs, 2600-elo players those are? Murad Akdag, Turkey, vs Wolfgang Brodda, Germany, really, never heard of them
- on move 12, black plays the incredible Ne4, opening the game, against all right KID principles; instead, h6, followed by f4, gives black some advantage
- because of the above constatation, and the fact that engines would definitely prefer Ne4, opening the game, in place of f4, building a longer chain, it is almost certain those players massively used engines to assist them

PS. I do not of a single case, when a black KID player played fe4, and proceeded to score well; this is a standard position, in which black either keeps its pawn on f5, advances f4, or, when f4 is played, pushes e4, which is not so good, but capturing ef4...

[Uri Blass]

Another funny thing is: you test your engine's performance and improvement with long books, say 8,10,15 moves-long or even longer.

All the relevant search and eval parameters, but also the time management, are tuned to perform well after move 8 or 15. Not before that.

So how would you expect an engine, whose search and eval is tuned to statistically perform well starting late into the opening, to also perform well early into the opening, on move 3,5,6? The specific chess conditions early in the opening and in late opening are not quite the same, so an engine could play well into the early opening and find good moves, only if it has tuned its search and eval parameters with very short, 2-5 moves book, preferably shorter, for years.

How many engines out there have tuned their parameters with a very short book for years? I am afraid none.

So, when SF says d4 is the best move, I strongly doubt it, as there are much better chances it is c4 or e4.

Same from the black side: if SF suggests the French is a good option on e4, I strongly disbelieve that, as Sf has never tuned its parameters to perform well starting from the very first move.

A lot of strong GM's practically play 1.d4 so I wonder why do you think that you know better than them.

Stockfish also say that 1...c5 is the best move against 1.e4 and not 1...e6

http://database.chessbase.com/js/apps/database/

The analysis that suggest 1...e6 is at depth 48 when the analysis that suggest 1...c5 is at depth 50

And the analysis that suggests e4 is better than d4 is at depth 70.

Of course all of these estimates assume that the entire plan is followed exactly and there are no deviations in correctness of the plan at any point.

The fact that 50 ply analysis likes 1.d4 above 1.e4 seems to indicate to me that if 1.e4 is superior, then the difference is so small that no one can see perfectly far enough ahead to gain a real advantage of one over the other. In practical terms, I guess the difference is 1/100 of a pawn or so, and most chess players will lose that amount through inaccuracy in every game. So it becomes nothing more than random noise.

Now, it is true that someone familiar with 1.e4 and not familiar with 1.d4 openings will fare better with 1.e4 on average.

Perhaps that is why Fischer did so well with 1.e4 and Berliner did so well with 1.d4 -- because they liked those and were the most thoroughly versed in them.

That 50-ply analysis is inaccurate at best, for the simple reason that:

- it is done with engines
- engines do not understand the KID
- the single line that equalises for black fully after d4 is the black KID

So such an analysis could be absolutely discarded, until engines learn how to play the KID.

It is not a matter of who knows better his openings, e4 is simply better than d4, objectively, i.e. it provides white with larger advantage with a search that is not infinite.

About Fischer and Berliner, interesting why almost everyone knows who Fischer is and almost no one, apart from the specialised circles, knows who Berliner is?

I found on the internet a single game where Berliner drew Fischer, maybe there are more, please advise.

At the same time, I also found those:

http://www.chessgames.com/perl/chessgame?gid=1044152

http://www.chessgames.com/perl/chessgame?gid=1044112

http://www.chessgames.com/perl/chessgame?gid=1008418

All wins by Fischer, convincing at that.

Seems to indicate Berliner was nowhere near Fischer in chess understanding, even in Fischer's youth, at the time when Fischer still did not know if e4 or d4 is better.

If you look more carefully at the games, you will notice, that Fischer won both white games with e4, and also his black game against Berliner's d4, which should have given white comfortable advantage.

So, when I make statements, I like to stick to authorities.

No evidence that the KID equalize for black

People win games with white against the KID in correspondence games and I did not play the KID against 1.d4 in correspondence games and got a draw.

Here is an example for a correspondence game when black lost inspite of starting 1.d4 Nf6 2.c4 g6
https://www.iccf.com/game?id=417239

If the KID equalize then you could expect GM's to play it but many GM's choose different defence against 1.d4

Probably not the time for me to comment anymore on this, as everything already seems to be said, but:

- what kind of GMs, 2600-elo players those are? Murad Akdag, Turkey, vs Wolfgang Brodda, Germany, really, never heard of them
- on move 12, black plays the incredible Ne4, opening the game, against all right KID principles; instead, h6, followed by f4, gives black some advantage
- because of the above constatation, and the fact that engines would definitely prefer Ne4, opening the game, in place of f4, building a longer chain, it is almost certain those players massively used engines to assist them

PS. I do not of a single case, when a black KID player played fe4, and proceeded to score well; this is a standard position, in which black either keeps its pawn on f5, advances f4, or, when f4 is played, pushes e4, which is not so good, but capturing ef4...

These are ICCF GM's but I also looked at games of other GM's like
Carlsen Kramnik or Caruana and they often choose e6 after
1. d4 Nf6 2. c4

[jefk]

[quote="Uri Blass"]
These are ICCF GM's but I also looked at games of other GM's like
Carlsen Kramnik or Caruana and they often choose e6 after
1. d4 Nf6 2. c4[/quote]

often, yes, but 2...c6 also is possible. Carlsen played 2...g6 against Anand.
And it not always leads to KID, better is first ...d5 instead of ...Bg7
and then aiming for a Gruenfeld. Maybe there is an antidote
against 3.f3, will require quite some analysis.

jef

PS when i wrote in previous message about exporting
endnodes of a pgn i meant obviously a pgn with many
games, all cutoff at a certain ply eg 40 or so.

PS2 not everybody is doing it in my way, which was geared
more for the Chesspartner interface and not Cbase

So here's some more info about bookmaking:

http://horizonchess.com/FAQ/Winboard/openingbook.html
http://www.spaghettichess.com/Dagh%20Nielsen_tips.txt
http://www.chess2u.com/t6350-excellent- ... bookmaking
(the .pdf link, with a comprehensive article
http://www.autochess.com/forum/tuning-a ... t1209.html

so you (U.B) see it's not all so easy.
unless you want to play games with one hour per move
which is more like correspondence chess but even
then i can assure you that depending on the engine
eval and the knowledge of your opponent you will
not be able to win with only Sfish analysis.

[Laskos]

I disagree with lyudmil and I think that at least in 99% of the cases engines play the right move if you give them a long time to calculate.

if there are some cases when they do mistakes you can find it by the evaluation and if white play the right move based on long analysis in every move but black get the advantage after move 15 or move 20(at least 0.1 pawns for black) then it is better to do analysis to see where white went wrong in order to fix the book.

Starting point if we build a book for white is simply play the moves the engines suggest after a long analysis for white and moves that are in the database for black.

Same idea if we build an opening book for black but in the last case I suggest to try to fix only cases when the evaluation for white is at least 0.5 pawn for white because small advantage for white does not mean that black did a mistake.

That seems wrong. Engines' PV in a position, especially in the openings, are useless after 2-3 displayed half-moves. Even identical engines will play the expected reply only 70%-80% of the time, the next in PV 50%-60%, and so on decreasing. Engines have only a very fragmented outlook of the game, changing randomly "plans" and PV lines. Books need a trustworthy PV of 20 or so half-moves, a thing engines don't give.

The books are not built by leaving long analyzes of openings. Most lines are chosen by outcome statistics. Most games used are fast, for example PlayChess 1' and 3' per game.

I did not say to trust engine pv's except the first half move.
The idea is to analyze every position and pick the first move in the pv.

That's what I was saying, engine hops from old PV to new PV, a new PV which it didn't expect in the old PV, often ending during the opening in a bad position against a book. Lyudmil offered a good early position which SF evaluates at 16-26 cp after a long analysis, while really it was 45-50 cp as expected score goes. I don't think an early opening can be analyzed well having a crappy PV at the second-third half-move and the outcome being very far away. Maybe a combination of human planning the opening and an engine for blunder check is worth pursuing.

I think that in big majority of cases chess engines have a right move after long analysis.
Right score at the early stage is not important.

I do not say it is going to be 100% right but using the evaluation of the engine not in the early stage can help to discover mistakes in the book.

more than 0.1 pawn for black(in the book for stockfish as white) or more than 0.5 pawn for white(in the book for stockfish as black) strongly suggest that some move in the line was not correct and you may try longer analysis to see what came wrong but I believe that in most cases you will not have one of these cases even if you use 0.0 pawns and 0.4 pawns instead of 0.1 pawn for black and 0.5 pawns for white.

If you think that there are many holes in a book that you build in this way when every move is at least result of one hour of analysis by stockfish then show me the holes and not a short line but some long lines when every move is based on the choice of the engine after at least one hour and the evaluation at the end of the line suggest that the choice of the program is not good.

I can add that the example of some opening that starts with 1.e4 e5 is not relevant assuming stockfish does not play 1.e4 as white and does not play 1.e4 e5 as black because stockfish will never get this opening when I generate a book based on stockfish's moves.

More common is the case when engine without book thinks at the beginning that it's steadily improving its position, say from -0.20 at the move 1 as black to 0.00 at the move 5-8, then, when book engine exits the book, say at move 15-18, the non-book engine hopping from one PV to another unrelated PV all the time, finds itself in at -0.50 disadvantage. That is common at say 1 min/move games, and my guess it would happen too at 1 hour/move. Unreliable PV in the openings, not valid even for the first 2-3 half-moves at any reasonable time control in the openings makes engine analysis helpless against a book. And books are not built by analyzing 1 hour/move each position in a tree, because of both:

a. Practically building such books is not feasible due to limited time.
b. Even if it would be possible, the book of jumping PVs at each position of the tree would be weaker than a regular book built on outcome statistics.

The problem is more stringent in the openings because it's very hard to have a reliable PV several plies deep, and the hidden mechanisms of openings would need something like 20 trustworthy plies in PV. One cannot jump from unrelated PV to unrelated PV expecting to beat a regular good book. In combination with a strong human, engine can be useful even in early opening phase, provided the human has a good database.

[Uri Blass]

I disagree with lyudmil and I think that at least in 99% of the cases engines play the right move if you give them a long time to calculate.

if there are some cases when they do mistakes you can find it by the evaluation and if white play the right move based on long analysis in every move but black get the advantage after move 15 or move 20(at least 0.1 pawns for black) then it is better to do analysis to see where white went wrong in order to fix the book.

Starting point if we build a book for white is simply play the moves the engines suggest after a long analysis for white and moves that are in the database for black.

Same idea if we build an opening book for black but in the last case I suggest to try to fix only cases when the evaluation for white is at least 0.5 pawn for white because small advantage for white does not mean that black did a mistake.

That seems wrong. Engines' PV in a position, especially in the openings, are useless after 2-3 displayed half-moves. Even identical engines will play the expected reply only 70%-80% of the time, the next in PV 50%-60%, and so on decreasing. Engines have only a very fragmented outlook of the game, changing randomly "plans" and PV lines. Books need a trustworthy PV of 20 or so half-moves, a thing engines don't give.

The books are not built by leaving long analyzes of openings. Most lines are chosen by outcome statistics. Most games used are fast, for example PlayChess 1' and 3' per game.

I did not say to trust engine pv's except the first half move.
The idea is to analyze every position and pick the first move in the pv.

That's what I was saying, engine hops from old PV to new PV, a new PV which it didn't expect in the old PV, often ending during the opening in a bad position against a book. Lyudmil offered a good early position which SF evaluates at 16-26 cp after a long analysis, while really it was 45-50 cp as expected score goes. I don't think an early opening can be analyzed well having a crappy PV at the second-third half-move and the outcome being very far away. Maybe a combination of human planning the opening and an engine for blunder check is worth pursuing.

I think that in big majority of cases chess engines have a right move after long analysis.
Right score at the early stage is not important.

I do not say it is going to be 100% right but using the evaluation of the engine not in the early stage can help to discover mistakes in the book.

more than 0.1 pawn for black(in the book for stockfish as white) or more than 0.5 pawn for white(in the book for stockfish as black) strongly suggest that some move in the line was not correct and you may try longer analysis to see what came wrong but I believe that in most cases you will not have one of these cases even if you use 0.0 pawns and 0.4 pawns instead of 0.1 pawn for black and 0.5 pawns for white.

If you think that there are many holes in a book that you build in this way when every move is at least result of one hour of analysis by stockfish then show me the holes and not a short line but some long lines when every move is based on the choice of the engine after at least one hour and the evaluation at the end of the line suggest that the choice of the program is not good.

I can add that the example of some opening that starts with 1.e4 e5 is not relevant assuming stockfish does not play 1.e4 as white and does not play 1.e4 e5 as black because stockfish will never get this opening when I generate a book based on stockfish's moves.

More common is the case when engine without book thinks at the beginning that it's steadily improving its position, say from -0.20 at the move 1 as black to 0.00 at the move 5-8, then, when book engine exits the book, say at move 15-18, the non-book engine hopping from one PV to another unrelated PV all the time, finds itself in at -0.50 disadvantage. That is common at say 1 min/move games, and my guess it would happen too at 1 hour/move. Unreliable PV in the openings, not valid even for the first 2-3 half-moves at any reasonable time control in the openings makes engine analysis helpless against a book. And books are not built by analyzing 1 hour/move each position in a tree, because of both:

a. Practically building such books is not feasible due to limited time.
b. Even if it would be possible, the book of jumping PVs at each position of the tree would be weaker than a regular book built on outcome statistics.

The problem is more stringent in the openings because it's very hard to have a reliable PV several plies deep, and the hidden mechanisms of openings would need something like 20 trustworthy plies in PV. One cannot jump from unrelated PV to unrelated PV expecting to beat a regular good book. In combination with a strong human, engine can be useful even in early opening phase, provided the human has a good database.

In many cases there is already analysis of more than one hour in let is check so you do not need to do the analysis by yourself and it is enough to collect the data.

I saw no evidence that book like it is worse than book based on statistics.

[Zenmastur]

I disagree with lyudmil and I think that at least in 99% of the cases engines play the right move if you give them a long time to calculate. snip ...snip...

Engines might play the right move in 99% of cases in simple endgame positions, or simple middlegame positions, but not in the opening, as such positions are usually very complex, both for humans and engines to understand thoroughly.

Give current top engines not 1,2 or 5 hours thinking time, but 10 days thinking time on standard opening positions, and they still will not find the right move in at least half of the cases, I think in 70-80% of cases, based on what I have seen from engine play in openings. snip...snip...

I think Lyudmil is exaggerating quite a bit.

Playing the opening is not a bunch of individual moves its a well choreographed series of moves that can be quite long. Making a single mistake in this series of moves can ruin your chances in a game. If the opening line is 40 plies long, then each side has 20 chances to blow it. If your error rate is 5% (95% accuracy) on every move your chances of making it out of the opening in good shape is about 36%. Which means that in 64% of the games you will be at a disadvantage against a perfect opponent. If your error rate is 1% on every move, your chances of making it out of the opening in good shape is about 82%. Which means you will be at a disadvantage 18% of the time against an opponent that plays perfect openings. On the other hand if Lyudmil is right and your accuracy is only 50% even if the opening line is only 16 plies (8-moves) long your chances of getting out of the opening in good shape is a scant 0.4% so in 99.6% of your games you will be at a disadvantage out of the opening. For 4 moves (8-plies) this goes up to 6.25%.

So I have a few question for Lyudmil:

When was the last time you saw Stockfish come out of the opening 94 times out of 100 with a distinct disadvantage? Enough said!

Since you seem to think that over the board play is everything, and Fischer is a god, why don't you think Stockfish or Komodo are gods since no human, INCLUDING Fischer, could beat Stockfish or Komodo in an extended match on TCEC hardware and standard time controls (40/120 repeating)?

That seems wrong. Engines' PV in a position, especially in the openings, are useless after 2-3 displayed half-moves. Even identical engines will play the expected reply only 70%-80% of the time, the next in PV 50%-60%, and so on decreasing. Engines have only a very fragmented outlook of the game, changing randomly "plans" and PV lines. Books need a trustworthy PV of 20 or so half-moves, a thing engines don't give.

The books are not built by leaving long analyzes of openings. Most lines are chosen by outcome statistics. Most games used are fast, for example PlayChess 1' and 3' per game.

In any case, Stockfish's evaluation based on its search is currently

Code: Select all

info depth 42 seldepth 67 multipv 1 score cp -16 nodes 46315755063 nps 17942433 hashfull 999 tbhits 0 time 2581353 pv d7d5 e4d5 c6a5 d2d3 [/quote]

depth 42 looks like an impressive search, but it illustrates again
you cannot look deep into the pv.
At ply 4 it already goes wrong, in my book Bb5+ is  better 
than d3, and also if you check on the chessbase  online database 

 snip...snip...

jef[/quote]


"Engine's PVs in a position, especially in the openings, are useless after 2-3 displayed half-moves." I would say this is definitely true for Stockfish. They claimed they fixed inaccurate PV's. Right afterwards it wasn't perfect but it was "better". Since then this "fix" appears to be broken. I've seen lots of PV's where the second move is wrong (mostly with search depths greater than 35 and in a few cases as deep as 60.  

But...

Just because the displayed PV doesn't correspond to the score given doesn't mean the score is flawed. The scores given are always correct within the given search depth as far as I have been able to determine. (I know because I have hand checked a huge number of them! Some of these were deep and it took a long time to reconstruct a PV that matched the score given. In a few cases I gave up due to time constraints.) It's just that the PV displayed doesn't correspond. So, only use the first half-move of the PV. Each half move after that has to be independently verified  with a separate search. So, on a 20 move PV, which isn't uncommon, you need to run 39 additional searches to get the "real" PV that corresponds to the given score. If you want to talk about how current programs waste the end users time by failing to provide them accurate information, this would be a good place to start. I'm not sure why they bother to generate a PV for output. They should either fix it OR, if they don't want to do that, just output the move they are going to play and leave it at that. Displaying an inaccurate PV under the pretense that it's accurate doesn't help anyone including the developers and generally wastes peoples time. End of rant.

So the point is, just because the displayed PV is NOT a valid line of play (because it doesn't correspond to the score given for the root move) doesn't mean that the score for the root move is wrong! It simply means the PV is unreliable and has to be created from scratch. This fact has absolutely no effect on the soundness of the selected move. The PV displayed  is simply window dressing, and nothing more!
 
[quote="Laskos"] I think Lyudmil is right. From this position, I built an opening book depth=6, and played 400 fast self-play games with Stockfish. The outcome was 132 white wins, 205 draws, 63 black wins. Performance ~59%, much higher than an average opening (53%). The Stockfish eval of this position, according to my data of performance versus eval should be ~45 cp.[/quote]

The problem here is that the results of quick games tells you little about the game theoretical value of any position because you are discarding almost all of the useful information you gained during the game. Until you stop burning CPU cycles and then throwing away all the information you gained you are very unlikely to make any significant progress on opening books. In my opinion, the current methods used to build books is a complete waste of a valuable resource, namely a humans time.

Anyone care to guess how many games have been played by the stockfish development team, or by Bob Hyatt's testing? I'm sure its in the hundreds of millions of games. I suppose it could be over a billion games if we count all the game played by the entire chess community. I counted over a million games on the first page of test for stockfish and noted that there are 132 more pages. The fact is, people are throwing away thousands and thousands of hours of CPU time. What if they used these games to collect information about openings? I suspect that we would have very accurate opening books with very little human intervention. 

Some may argue that because these are quick time control games that very little useful information could be gained by collecting information from them. This is absolutely not true! The exact same method has been used with Monte Carlo play out. i.e. no thinking time or analysis at all. The opening-book problem is isomorphic to the many armed-bandit problem. There have been many papers written on optimal algorithms that achieve controlled exploration of the search space with a logarithmic regret function. I'm going to collectively call these algorithms Bandit algorithms. 

You only need to add a few rules to build a book.

1.) The standard starting position is a member of the book positions.
2.) Only nodes that have been seen more than the book minimum_times_seen constant may add a child node. 
3.) Any position that is not in the book and is the child of a node in the book that has been seen more than minimum_times_seen will be added as a leaf node. 

These are the only rules needed to build the book. These rules say nothing about the information stored in the book (other than the number of times each node has been seen) or the rules that govern the use of the information in the book. e.g. which nodes should be  played and when "NEW" moves should be tried to gain more information about them. "NEW" in this context means a node/move that doesn't meet the user criteria for being played. They only govern when nodes are added.

Just because a node is stored in the book doesn't make it a playable move. This is unlike current books which only store playable moves or responses that have been seen many times but are considered substandard and will never be played by the book user. This type of book contains lots of nodes that aren't part of the normally played book. The playable nodes are those that meet certain criteria set by the book user. The rest of the nodes are there to collect information and may be played when it's determined by the bandit algorithm that further exploration of the position is warranted. The bandit algorithm could also select a move that is not in the book. e.g. a move from a node (that has the right to add leaf nodes) that produces a position not seen before. There is no point to making a new move from a node that doesn't have the right to add a leaf node because nothing can be learned from this since the new node can't be added to the book.

It would be nice to see any of the stronger engines implement this type of book with a published format or even better published code so that any engine developer could use it. But, I'm not holding my breath.

For those that want to read more about the many-armed bandit problems there are links on the chess wiki or you can Google it.

Regards,

Zen

[reflectionofpower]

I disagree. I have looked at TCEC games for example (long time control) and I see a lot of mistakes in the opening, some of them serious.

For example take this position from the Marshall Gambit:

[d] r1b2rk1/5p1p/p1pb4/1p1n2pq/3PR3/1BP3P1/PP1N1P1P/R1B2QK1 b - - 0 17

The main line goes Bf5! f3 Nf6!. f5 is usually what engines choose and it is playable, but usually considered inferior after Bd1!. Engines don't like Bf5 at all.
--Jon

Just every engine I use chooses Bf5. Bh3 came up from Deep Junior 10 (psycho) so I thought because I have it running very aggressive that maybe it it's an iffy move, SugaR & SF6 choose the same move.