Computer based Opening theory

[Evert]

I would find this very useful - especially for chess variants, where you may not know any reliable openings.

I've used a similar approach to generate candidate positions, expanding the tree for 4 or 6 plies (don't remember exactly) and eliminating moves that are more likely to cause unbalanced positions. Still generated too many candidate positions to weed out the unbalanced ones by eye and didn't have time to run a deep analysis on them.

It's clearly not perfect, but a sensible thing to do to begin with.

[Uri Blass]

I don't think even that is going to help. There are some very complex openings where in some cases the latest theory comes from correspondence games where 1 hour/move is considered a fast time control. I could give many examples, but some are the Botvinnik Semi-Slav, the Bishop sac in the 6. Ne5 Slav, some lines of the King's Gambit (John Shaw has a very detailed new book on this with deep analysis), etc.

Better IMO to take a decent book as a starting point, evaluate the leaf positions, and minimax back up to to the root. You are not going to discover the truth about openings starting from the root and working forward. The branching factor is too large and there are too many cases where only very deep search reveals the right way to go.

--Jon

I think that 1 hour per move may be sometimes above the level of correspondence games considering the fast improvement of stockfish.

It may be possible later to compare with opening book and see if there are common moves that the program does not suggest.

[Uri Blass]

this way, you'd get useful lines without the rather dubious 1. f3, 1.h4 stuff

Why these lines are doubious? Because all of us would never play such moves doesn't mean they are not a good start...

probably almost all: 1. e4, 1. d4, 1.c4, 1. Nf3, 1. g3, etc.
my guess is that the engines would mostly enter main lines of human opening theory
(Ruy Lopez, Queens Gambit, etc.)
and quite possibly play novelties or rarities once in awhile, thereby enhancing/enriching what we already know.

Engines will never give a pawn in the opening, so remove every gambit from the list....

Why do you think that engines will never play a gambit?

I think that you are wrong about it and engines can sacrifice for positional reasons.

For example many engines play after 1.e4 d5 2.exd5 not Qxd5 but Nf6

[mvk]

Engines will never give a pawn in the opening, so remove every gambit from the list....

Why do you think that engines will never play a gambit?
I think that you are wrong about it and engines can sacrifice for positional reasons.

Funny. The main reason I use a book is to avoid gambits.

[Sergei S. Markoff]

In SmarThink I have two book options — DISCOVER and KILL (see ini-file).
I have a stats for each node — number of times played and cumulative score. DISCOVER means to play variation with least times played count, while KILL prefers node with highest relative score.
Also there is bound value for KILL mode.

Anyway I think that the most statiscially correct way is to play variation with highest value of upper bound of confidence interval of win probability (based on score and games played). I think that using set of strongest engines and "self-expanding" book you can achieve great results.

[Sergei S. Markoff]

Moreover, it's possible to just limit number of nodes in your book and prune-off lines with lowest value of upper bound of confidence interval of win probability.

[lucasart]

How would you avoid convergence to the same position after one, or a few, moves into the game?

I detect transpositions using a hash table. This was quite a big speed gain. Even at 3 or 4 plies:, there are *lots* of transpositions already.

But I do not the hash table, as it can miss some due to hash overwriting. So I filter the unique positions after every step of the generation as follows:

Code: Select all

cut -d\  -f1,2,3,4 ./bookN|sort -u>./bookN.epd

The 'cut' part trims off the move counter and 50-move counter at the end of the FEN. This is important because the same positions can appear twice with different counters. Them a simple 'sort -u' (sort unique).

What is ensured is that my bookN.epd contains unique positions, reached after up-to N plies, not N plies exactly. But that doesn't matter. On the contrary the shallower the better for me.

[mvk]

How would you avoid convergence to the same position after one, or a few, moves into the game?

I detect transpositions using a hash table. This was quite a big speed gain. Even at 3 or 4 plies:, there are *lots* of transpositions already.

Ah, I should express my question more clearly: you intend to use these positions for testing. For that you need divergence: line A and line B should not naturally flow into the same continuation after one or a few moves out of 'book'. If you have a 'dense' opening selection, that effect might spoil a lot of your effort (getting duplicate games nonetheless).

Code: Select all

            N
            :
            :
            A
          / : \
         /  :  \
Starting    :    Oops
         \  :  /
          \ : /
            B
            :
    Book    :   Engine
            :

[lucasart]

Ah, I should express my question more clearly: you intend to use these positions for testing. For that you need divergence: line A and line B should not naturally flow into the same continuation after one or a few moves out of 'book'. If you have a 'dense' opening selection, that effect might spoil a lot of your effort (getting duplicate games nonetheless).

I see. Yes, that is something I thought about, but unfortunately I don't see any simple way to find out. This is a problem every book potentially has. As you say, the denser it gets (high branching factor) the more likely it will happen. My branching factor is not so high now, with the more restrictive conditions (45cp threashold and prune quiet moves that do not improve the static eval). Hopefully the eval improving condition limits that kind of thing as moves tend to go "forward".

[lucasart]

But another more interesting approach would be to perform a multi PV of 3 on each position with depth 12 and filter all the positions for which at least one of the PV lines is below threshold. In this way you drop all the playable but forced positions and just keep the ones that allow the engine to freely choose how to continue.

Yes, multi-PV would be an interesting approach. I think my goal is different:

• for generating starting positions (leaves of the tree) I do not care about the moves that were played along the way to reach these positions, and how absurd these moves may positionally. The only thing that matters is to create a shallow and diversified opening set. In the end I will filter the resulting positions set with a deeper search (using Critter which I trust best to find tactical errors in subsecond analyses, also it does not show inflated positional scores like SF).
• for generating a proper book, moves matter, not just leaf positions. So yes, that's where Multi-PV would be the best approach. Generating random moves so long as they are not outright blunders would produce a pretty bad book. If both sides play with that same book it's ok, but the opponent engine playing without book or with a better book would generally come out on top, and that's no good.