Good size for Opening Book?

[stevemulligan]

What is a good size for an opening book? I found about 2 million games in PGN format and extracted the first 12 half-moves from winning players. It takes up about 31megs compressed. It consists of ~4.1 million board positions.

Is that way to big? What size opening book should I be going for?

Edit: My max size total is 150megs. EGTB's take about 93megs now because I'm only using a few 5-man tables. I'm considering shrinking the OpeningBook to 25% it's current size to fit more EGTB's in my final build.

[cms271828]

Thats interesting,

Maybe you were looking at same pgn files as me...
I've put 1.7 million games to a depth of 26 half moves into database.

Takes up almost 10,000,000 records, each record contains the top 4(or less) best moves played for that position, along with a weight which corresponds to the frequency of which it was played over the 1.7 million games.

This allows for some randomness in the computer play, occasionally playing the less frequently seen opening lines if needed.

Its online, so any engine needs to connect to internet to use it.
www.colin-java.co.uk then select chess opening book.

[stevemulligan]

Ahh with 10M records I think my book is quite small in comparison. Good to know it's not overkill.

I had not thought of only storing the top N most popular moves for each position. I could probably save a lot of space doing that. Going 13 moves deep would be nice but I don't think I can afford the space.

I got some of my PGN files from the site you referenced on your website, also from http://www.chessgames.com

I assume that games in PGN format are public domain. I can't find any license agreements on sites hosting pgn files. Most sites call them public domain however they also admit they were "collected" over the years. If they are not original works then I'm basically trusting the person hosting the files that they know the real terms of use.

Do any PGN files have usage restrictions? or all they all considered public domain given the finite state space?

[lucasart]

What is a good size for an opening book? I found about 2 million games in PGN format and extracted the first 12 half-moves from winning players. It takes up about 31megs compressed. It consists of ~4.1 million board positions.

Is that way to big? What size opening book should I be going for?

Edit: My max size total is 150megs. EGTB's take about 93megs now because I'm only using a few 5-man tables. I'm considering shrinking the OpeningBook to 25% it's current size to fit more EGTB's in my final build.

IMO it's much better to keep the book small, and accurate. better to have, let's say, an 8 ply book, where each position was tested (perhaps a sec search or sth), than a huge dump of PGN where you can never be sure what you get.

[cms271828]

I'm not sure, they were free to download, so I downloaded them.

Its not easy creating the database, I think one needs to create hashamp with the zobrist keys before converting into sql database. I tried storing java objects containing position and moves played in the hashmap (array), but its seemed to slow down and run out of memory. So I converted all the data into strings and that worked.
Then I sorted then with quicksort I think before making database.

I'm interested in egtbs, I have about 1-1.5G space to spare. Would it be possible to store all 5 men egtbs with this?

[stevemulligan]

I'm interested in egtbs, I have about 1-1.5G space to spare. Would it be possible to store all 5 men egtbs with this?

I'm using Gaviota EGTB's, about 7 gigs for all the 5 man tables. I'm only using a few 5 man tables (kppkp, kbbkr, kbbkb, knnkb, knnkn)

I heard someone was porting to Java but I'm not sure what development stage they're at.

perhaps a sec search or sth

I'm not sure what those are? Do you mean to a deep search for each position to make sure it has a good score?

[mike_bike_kite]

I'm using Gaviota EGTB's, about 7 gigs for all the 5 man tables. I'm only using a few 5 man tables (kppkp, kbbkr, kbbkb, knnkb, knnkn)

I heard someone was porting to Java but I'm not sure what development stage they're at.

Let's hope it's not a Java applet