Engine learning and the development of opening repertoire

[ozziejoe]

I've been a bit obsessive about this topic lately, because I have been so disatisfied with what is out there. I've come to a few conclusions, and again some suggestions, based on hitting my head repeatedly against the wall.

I wanted to understand the best way to responding to particular positions, in the opening and middle game. For example, I'd like to know the best way to respond to white C4 in the following line : 1) E4, C6, 2) D4, d5; 3) e5, c5; 4) c4!?. This opening looks sound and dangerous for black (and I play this side), and there are few gm games (61 in my hugebase = 3.5 million).

I want to use the empirical method for evaluating this position and finding the best line. this means I want to run a butload of games and see what is emerges

Some conclusions so far:

Rybka Randomizer seems ok, but does deliberately select suboptimal lines. It seemed to not give you accurate assessments of a position, based purely on winning percentage. It does seem better if you base assessment on the best lines.

Engine learning: This looks very promising, as it should allow you to hone in on a good line. Romi is particularly good at this. the one limitation is the tendency to "satisfice." That is, Learning will find a line that is just good enough to win (not a line that is necessarily optimal). Another limitation of engine learning is the engines that are best at it (romi) are not necessarily strong enough, compared to rybka, hiarcs, etc. Rybka shoudl change this in the next version....However, if rybka is much better than others, then it will not have to identify the ideal opening response.

Ok, so i have a suggestion (i know you folks are getting tired of these, but i can't help myself) for evaluating opening positions, developing ideal responses to openings, and finding novelties. I think you do it in a series of steps.

1) Download all human gm games from the given opening position. Create pgn1, which will later be used to build a book. I have pgn of 61 games in the above positions
2) Run a massive engine tournement from the opening position and ply set to, say, 7. The quality of the games is not so important. We just want variety of opening responses. Use the games to create pgn2. I'll create over three hundred games from the opening position
3) Use PGN1 and PGN2 to create an opening book with ply of X. In the above example, I may go to 12 half moves.
4) Now run a tournement with engines using the new opening book and opening book set at its highest variety. This will force good and bad engines to play every line (and thus unconfounds quality of lines with quality of engines).
5) Collect maybe 1000 games of engines playing each other (after removing duplicates). Use these games to build a new statistics tree, a new book, etc. Basically , the 1000 games should show which lines are most promising

I think this whole process could be fairly automated e.g., the gui runs step 2. It forms the opening test bookt step 3 (maybe asking if you would like to importat some human games). Then it runs the massive tournemant.

I am testing this idea now.

best
J

[Ovyron]

I want to use the empirical method for evaluating this position and finding the best line. this means I want to run a butload of games and see what is emerges

But, the game has only 3 results, win, lose or draw. And all lines are equally good as long as they don't worsen the game result (Draw > Lose | Win > Draw.) So, a "best" move against one opponent isn't going to work against another, and playing this move from engine A isn't going to work for engine B.

So, the best you can do is try to know what kind of positions work better for an engine, and what positions work better against an engine, but those "best" moves are only going to be best in this context.

Meaning, that there's no good concept of "Best Move" since it's very dependent of the situation (d4 may be best for you, but e4 may be best for me.) All Rybka is good at is at not playing bad moves, while all the other engines play them more often and it causes them to have a lower rating.

About your idea, the Randomizer is supposed to be good at this, as Rybka Ply 7 random 1 is going to play stronger and faster than any other 7ply engine, so the quality of the games is greater and faster and you should be able to have a high quality result earlier.