Creation of Opening books -- free to the community

[okoli]

I am currently packaging Open Source chess engines for the debian GNU/Linux operating system.

See: http://oko00.hostsharing.net/debian/

You will see packages for fruit, toga, glaurung, hoichess and sloppy.

My Problem is to provide any opening book for them, because the opening books come in binary format, that is not re-distributable in terms of the debian Free Software Guideline.

I thought of providing one for them all to be used with polyglot. Because then I have a .pgn that I can provide as readble source. And "compile" it with polyglot to .bin.

Do I have to start on my own or is anybody knowing of people that would share their experience or better .pgns for this matter.

Thanks in advance,

Oliver
(I know Dr. Hyatts crafty opening book, but I wonder if it is still on the edge of time.)
Please move this topic if it is too technical for the General Topics!

[ilari]

Sloppy is an Xboard engine, not UCI. So it can't use and doesn't need Polyglot.

Sloppy's opening book was built from this pgn file:
http://cvs.mandriva.com/cgi-bin/viewvc. ... gm2600.pgn

To get the exact same book I'm hosting on my site you'll just need to copy the pgn file to the Sloppy folder, run Sloppy and enter this command "readpgn gm2600.pgn". So maybe you can provide gm2600.pgn as the "source code".

[okoli]

Thanks for the hint. That will make it easy for me to provide the "original" Opening Book for sloppy as a package.

I am in doubt if other chess engine enthusiasts would not blame me to provide opening books, that are not specially tuned or at least meet a very high quality standard.

I myself see it: better I have a book than none ;o)

So the question was meant to be more general:

1.
Is there an opening book basis in .pgn that can be recommended to be used.
2.
Or is there a group of people sharing their expertise in opening book creation to provide a common opening book that meet "general" principles and a high quality.

Oliver

[Ratta]

You are doing a very good thing packaging them, at least linux users will easily have an alternative to the usual combo xboard+gnuchess, maybe it would also be nice to find a way to make programs user to use polyglot (such as a fruitx, or togax, that will work will a simple polyglot.ini, more advanced users will be able to edit their own file).
You may also consider scorpio and arasan (i don't know if their licence is too restictive for debian), bbchess is also a very strong GPL engine.
My engine rattatechess is also GPL, but the last released version is about as strong as gnuchess, so i don't know if it would be worth. You may want to reconsider when the version 2.0 will be released

[okoli]

As I am not a good programmer packaging is a way to help. I have rattatechess, arasan and scorpio on my mind, but the packages I already do, come first. And thats not a matter of strength.

I love hoichess, for the features and the fun...

But the topic was: my lack of information regarding strong opening books, do you have additional information on that?

Thanks in advance,

Oliver

I already know:
http://horizonchess.com/FAQ/Winboard/openingbook.html

[ilari]

I thought about this a bit more, and I don't understand why it would be a problem to add the binary books to the repositories, assuming that the books really are free. Gnuchess-book and Crafty's books are already there. These are data packages, not programs, so it shouldn't be a problem that they're binary-only.

[okoli]

I thought about this a bit more, and I don't understand why it would be a problem to add the binary books to the repositories, assuming that the books really are free. Gnuchess-book and Crafty's books are already there. These are data packages, not programs, so it shouldn't be a problem that they're binary-only.

If this data file is just binary you will never get it officially in the debian archive, if you dont provide the "source" for the binary and the "way" how to build them.

The binary packages of gnuchess and crafty are created out of "TEXT" files. PGN and some other information. The source packages are binary free. (and the binarys can be recreated by any time)

Thats something I would like to do also: Have a set of chess moves in .pgn and create the packages out of this "public domain", readble information.

Almost any good chess engine provides commands, you can use for book creation. For the UCI engines you can use the book commands of polyglot.

Both officially known books (crafty and gnuchess) have as source some pgn or other information of Dr. Rober Hyatts website and might be very, very old.

My question was: Is there a source known that provides readable files to create opening books and has a qood (UpToDate) quality.

Creators of Opening books seem not to share their expertise, knowledge and pgn files, because they fear that recreation would be possible and if everybody looks inside the "sources" they would face a lot of discussion.

Why not having a community project for creation of opening books. So that everybody can start with some basics and has not to start a point sero.

Comments and feedback welcome,

Oliver

[Marc Lacrosse]

(...)

My Problem is to provide any opening book for them, because the opening books come in binary format, that is not re-distributable in terms of the debian Free Software Guideline.

I thought of providing one for them all to be used with polyglot. Because then I have a .pgn that I can provide as readble source. And "compile" it with polyglot to .bin.

(...)

There are several freely distributable good quality polyglot opening books.
"performance.bin" and "varied.bin" (by myself) and Dann Corbit's large book are freely available at various places including my own chess file repository here or at WBEC, Superchessengine.com, etc.

For what regards my own books, you won't get the PGN. Both the PGN itself and the cooking "recipe" (Polyglot parameters for building the books) are private.

Maybe Dann or other book-makers would give you their native PGN base and the parameters for building the book.

Books like "varied.bin" or Dann's large book are mostly based on good quality human games. THere are many good freely available bases that could suit for building such books. Among them Norman Pollock's ones are available onJim Ablett's pages.

If the goal is more to build a "competitive" book (a book with which an engine will try to beat other engines or strong humans) then you also need good quality computer games played with "neutral" books at slow timing. This is a bit more difficult to find ....

Marc

[okoli]

There are several freely distributable good quality polyglot opening books.
"performance.bin" and "varied.bin" (by myself) and Dann Corbit's large book are freely available at various places including my own chess file repository here or at WBEC, Superchessengine.com, etc.

Yes, I know them and have already used them.

For what regards my own books, you won't get the PGN. Both the PGN itself and the cooking "recipe" (Polyglot parameters for building the books) are private.

I really understand this, I started on my own and I know how much work lies in stripping a collection from useless sources, not mentioning collecting games, maybe looking up books. Not mentioning the testing, looking for blunders.

Books like "varied.bin" or Dann's large book are mostly based on good quality human games. THere are many good freely available bases that could suit for building such books. Among them Norman Pollock's ones are available onJim Ablett's pages.

Thats helping, thanks for the hint.

If the goal is more to build a "competitive" book (a book with which an engine will try to beat other engines or strong humans) then you also need good quality computer games played with "neutral" books at slow timing. This is a bit more difficult to find ....

Good hint also. Self creation ;o)

But as I watch this posts I think what I (and maybe more people) really need is the following:

A website, blog whatsoever community effort that provides:

- Sources of high quality proofed material that fits the goal (Grandmaster Games, Correspondance chess,HQ Computer Chess)
- Recipes how to "cook" books from this sources
- Updating of these Recipes and Sources on a regulary basis

What do you think?

Oliver

[Marc Lacrosse]

(...)
But as I watch this posts I think what I (and maybe more people) really need is the following:

A website, blog whatsoever community effort that provides:

- Sources of high quality proofed material that fits the goal (Grandmaster Games, Correspondance chess,HQ Computer Chess)
- Recipes how to "cook" books from this sources
- Updating of these Recipes and Sources on a regulary basis

What do you think?

Oliver

There is a continuum going from A "good general book with varied openings and no gross blunder" to B "highly optimised book with hand-tuned lines for a given engine trying to beat a given opponent".

As you go from "A" to "B", more and more work is involved and those who take time to produce these "B" books will generally keep them private and use them in some sort of competition.

So I fear that you won't get the real high-end recipes and sources for your intended site.

Even my already old and basic performance.bin had a complicated building process that involved "merging" no less than 6 different books. Each of these 6 books was produced with different parameters (min-score, minimum number of games etc). There was even a little hand-tuning involved ...

But don't worry too much : as long as you simply use PG's powerful heuristic to build a generic book from a careful selection of games you will end with something valuable.

See for example this little test . In fact my best-performing private "generic" book is only 25 Elo points better than performance.bin .

... but when you go into hand-tuned selected-lines books you may hope for much better performances but this is in fact really another world.

Marc