An idea for a new WCCC format - what do you think?

[Gian-Carlo Pascutto]

What you can't do is export Aquarium format back to .ctg.

Vas

But can you export Aquarium format to something else?

[bob]

There is no benefit to the chess world in having me or anyone else roll his own EGTB implementation.

Vas

That's like saying Nalimov can't be improved upon. Think 6-men bitbases, for example.

Once you reach a certain size, a 10-fold size reduction begins to not be so interesting. 10% of infinity is still infinity, as it were...

Bitbases were more than a 10-fold reduction, I think...

(It's true they don't have the same info, but I don't think that really changes the point)

IIRC they collapse each entry to about 1.5 bits. most 4-5 piece files are 8 bit entries, some go tto 16 and the 6's are mainly 16 bit entries. Which gives roughly 10x size reduction before compression.

[Gian-Carlo Pascutto]

IIRC they collapse each entry to about 1.5 bits. most 4-5 piece files are 8 bit entries, some go tto 16 and the 6's are mainly 16 bit entries. Which gives roughly 10x size reduction before compression.

Yes, but they compress a lot better, and nobody's using uncompressed 6-man tables... 5 men is already 7G (Nalimov) to 150Mb (Shredder), a worthy improvement IMHO

[bob]

IIRC they collapse each entry to about 1.5 bits. most 4-5 piece files are 8 bit entries, some go tto 16 and the 6's are mainly 16 bit entries. Which gives roughly 10x size reduction before compression.

Yes, but they compress a lot better, and nobody's using uncompressed 6-man tables... 5 men is already 7G (Nalimov) to 150Mb (Shredder), a worthy improvement IMHO

OK, a factor of 40. Still one order of magnitude... For all the 6 pieces, we need almost 2 Terabytes. Or, at that same scale, just 50 gigs of ram or so?

[michiguel]

There is no benefit to the chess world in having me or anyone else roll his own EGTB implementation.

Vas

That's like saying Nalimov can't be improved upon. Think 6-men bitbases, for example.

Once you reach a certain size, a 10-fold size reduction begins to not be so interesting. 10% of infinity is still infinity, as it were...

Bitbases were more than a 10-fold reduction, I think...

(It's true they don't have the same info, but I don't think that really changes the point)

IIRC they collapse each entry to about 1.5 bits. most 4-5 piece files are 8 bit entries, some go tto 16 and the 6's are mainly 16 bit entries. Which gives roughly 10x size reduction before compression.

Just for the record, entries of 8 bits are enough for any 5-piece EGTB without using any compression. I do not know what Nalimov did, but if you are going to compress it, it may not matter at all.

For many of the 6-pc TBs you may need 16 bit entries. But this is true for DTM, not for DTZ50. You may only need 8 bit entries for DTZ50.

My own 5-pc TBs are not compressed, 8 bits, 38 Gb, and they work just fine with the cache I designed. I had several ideas how to compress them, but it turned out that I do not think I need that. They are fast enough already.

My little experience writing my own TBs told me that there is a huge room for improvement in this area, which we may never see because all the people are just happy with a uniform "pseudo-democratic" EGTB probing code.

There is also a lot of room for improvement in how to generate them. In fact, it is quite possible that some may be generated on the fly once the position is reached. Some of the retrograde algorithms, if they become fast enough, can be used for analysis of other type of positions (with blocked pawns, for instance).

Miguel

[Vasik Rajlich]

What you can't do is export Aquarium format back to .ctg.

Vas

But can you export Aquarium format to something else?

You can export to things like .html, but not directly to another tree format. Convekta is going to rely on their UCI-compliant wrapper to fill the gaps - this wrapper plays from the book when a book move exists and invokes the 'real' engine otherwise.

Vas

[Jeroen]

In the last WCCC in Beijing there were specific rules concerning the use of opening books. The rules stated (if I recall correctly) that each book author can only be part of one team (= program+GUI+book). To prevent other chess programs using the same book, each team had to reveil its book author before the start of the tournament, who also had to give permission that his book was being used.

I asked the ICGA about this issue as soon as my Rybka 3 book had become available through ChessBase. The former rules didn't prevent the situation that a chess programmer would buy the Rybka 3 book and use it in the WCCC.

Of course the new rules would not rule out everything, but IMO they were a clear improvement.

[CRoberson]

In the last WCCC in Beijing there were specific rules concerning the use of opening books. The rules stated (if I recall correctly) that each book author can only be part of one team (= program+GUI+book). To prevent other chess programs using the same book, each team had to reveil its book author before the start of the tournament, who also had to give permission that his book was being used.

I asked the ICGA about this issue as soon as my Rybka 3 book had become available through ChessBase. The former rules didn't prevent the situation that a chess programmer would buy the Rybka 3 book and use it in the WCCC.

Of course the new rules would not rule out everything, but IMO they were a clear improvement.

I don't know how far back the book rules go, but there was such
a rule in 2002 and there was much debate on how to interpret it.

[Vasik Rajlich]

In the last WCCC in Beijing there were specific rules concerning the use of opening books. The rules stated (if I recall correctly) that each book author can only be part of one team (= program+GUI+book). To prevent other chess programs using the same book, each team had to reveil its book author before the start of the tournament, who also had to give permission that his book was being used.

I asked the ICGA about this issue as soon as my Rybka 3 book had become available through ChessBase. The former rules didn't prevent the situation that a chess programmer would buy the Rybka 3 book and use it in the WCCC.

Of course the new rules would not rule out everything, but IMO they were a clear improvement.

I don't know how far back the book rules go, but there was such
a rule in 2002 and there was much debate on how to interpret it.

In my view the rule is good and not that ambiguous. Books should be treated like engine code.

The issue (which isn't really a problem) is that book authors tend to form clusters and work together, exchanging their books regularly inside their small group. It's like if two or more programmers were exchanging source code. There is nothing wrong with this.

Vas

[Marc Lacrosse]

book authors tend to form clusters and work together, exchanging their books regularly inside their small group (...)

I am not so sure this is common practice among top engine book authors ...

Sure there are book exchanges and competitions, specially on rybka's forum, among amateurs who all use rybka 3 on playchess' server and try to reach a high rating there.

But I never heard that the official book authors of Naum, Zap, Hiarcs, Shredder, Junior, Fritz or even Rybka to name a few top ones exchange anything regarding favored book lines.

Marc