EGTB generator

[Dann Corbit]

This generator creates very tiny EGTB files:
http://generatorchess.qsh.ru/Default.aspx

For example:
01/29/2009 07:27 PM 803,649 5_KNN-KN.tb
01/29/2009 07:27 PM 1,624 5_KNN-KN.txt

Contents of 5_KNN-KN.txt:

C:\chess\winboard\RetroChess\Bases>type 5_KNN-KN.txt

______________________________________________
All Positions. WHITE to move.

ALL: 44118240
WHITE Lost: 56
DRAWS: 44082657
WHITE Win: 35527

DETAILS:

-#1: 56
Draws: 44082657
+#7: 38
+#6: 757
+#5: 3887
+#4: 5955
+#3: 9382
+#2: 10535
+#1: 4973
______________________________________________
All Positions. BLACK to move.

ALL: 48096060
BLACK Lost: 5092
DRAWS: 48090675
BLACK Win: 293

DETAILS:

+#1: 964
+#2: 1007
+#3: 1404
+#4: 726
+#5: 796
+#6: 177
+#7: 18
Draws: 48090675
-#1: 293
______________________________________________
NORMAL Positions for WHITE.
(WHITE to move and no white figures attacked)

ALL: 26712282
WHITE Lost: 0 (0.0 %)
DRAWS: 26685466 (99.8 %)
WHITE Win: 26816 (0.1 %)

DETAILS:

Draws: 26685466
+#7: 37
+#6: 660
+#5: 3071
+#4: 3754
+#3: 7221
+#2: 8425
+#1: 3648
______________________________________________
NORMAL Positions for BLACK.
(BLACK to move and no black figures attacked)

ALL: 30129635
BLACK Lost: 3298 (0.0 %)
DRAWS: 30126136 (99.9 %)
BLACK Win: 201 (0.0 %)

DETAILS:

+#2: 936
+#3: 1077
+#4: 527
+#5: 586
+#6: 156
+#7: 16
Draws: 30126136
-#1: 201
______________________________________________

[Dirt]

For example:
01/29/2009 07:27 PM 803,649 5_KNN-KN.tb
01/29/2009 07:27 PM 1,624 5_KNN-KN.txt

The size in bytes is 803,649, right? I have the size of the two knn-kn Nalimov files on my system totaling 164,995.

[Dann Corbit]

Yes, but this is their decompressed size:
48,096,060 knnkn.nbb
44,118,240 knnkn.nbw

[hgm]

Well, so the EGTBs made by retrochess are compressed as well, but just not as good?

[Edsel Apostol]

If I understand correctly the translation from Russian the author says that the compression is somewhat a little bigger than Nalimov TBs.

http://www.google.com/search?hl=en&as_q ... afe=images

[Bill Rogers]

I don't care if we are talking about EGTB or books what I would like someone to do is explain the encoding techniques to me. I know that on square on a chess board can be represented by one byte but don't know exactly how it is done.
A few days ago someone posted a link to a program that would create books for PGN's, but I stil don't know the encoding tech.
Bill

[hgm]

This is completely different for EGTBs and opening books, and even within each class it is completely different for various formats.

The Polyglot opening-book format, for instance, stores (position, move) pairs. So different moves from the same position are independent entries. Each entry is 16 bytes, an 8-byte 'hash key' to identify the position, two bytes for the move, two bytes for a weight, and 4 bytes for learning information. All entries are simply behind each other, in order of increasing hash key.


For details, see:

http://alpha.uhasselt.be/Research/Algeb ... ormat.html

[Bill Rogers]

Thanks H.G.
At least I now understand how one book is created. It seems to waste a lot of space though. My books are created just like the ones used in TSCP
program. Although it takes 4 bytes per move I can shorten it to only 2 bytes per move thus allowing bigger books that take up less space.
I have not given up on learning how other books are created I just wondered how they are coded.
Bill

[Edsel Apostol]

Thanks H.G.
At least I now understand how one book is created. It seems to waste a lot of space though. My books are created just like the ones used in TSCP
program. Although it takes 4 bytes per move I can shorten it to only 2 bytes per move thus allowing bigger books that take up less space.
I have not given up on learning how other books are created I just wondered how they are coded.
Bill

I think the Polyglot book format is still much efficient in terms of accessing it and in storage compared to that of TSCP book format.

For example in TSCP:

e2e4 e7e5 g1f3 b8c6 f1b5
e2e4 e7e5 g1f3 b8c6 f1c4
e2e4 e7e5 g1f3 b8c6 f1d3
e2e4 e7e5 g1f3 b8c6 f1e2

In Polyglot:

hashkey for position: 0xabcdef1234567890 move: f1b5 weight: etc..
hashkey for position: 0xabcdef1234567890 move: f1c4 weight: etc..
hashkey for position: 0xabcdef1234567890 move: f1d3 weight: etc..
hashkey for position: 0xabcdef1234567890 move: f1e2 weight: etc..

Each entry in Polyglot is only 16 bytes. In the TSCP book format an entry above is 25 bytes including the newline character. Imagine a book line of length 20 using TSCP book format, I guess it would take up around 100 bytes per entry whereas the Polyglot book format will still use 16 bytes.

When you access the book, the Polyglot format will use a binary search whereas in TSCP book format you will have to compare the game moves string to the book moves string line by line.

[hgm]

The Polyglot format could still be improved on a lot on, though: not every move from the same position would have to repeat the hash key. (An unused bit in the move field could indicate that the field normally containing a hash key would contain a second (move, weight, learn) combination in stead. This would pretty much cut the book size in half. And omitting the learn field, which is apparently never used, would cut it in half again...