New Cerebellum Library

[Dann Corbit]

Here is a link for the PGN built using the technique described using polyglot:

https://drive.google.com/open?id=19IhT8 ... 6PoJw5DoRC

Even though it is enormously larger, it looks to me like both colors were truncated at around 75,000 games.
So I guess the conversion is not complete.

[clumma]

Here is a link for the PGN built using the technique described using polyglot:

https://drive.google.com/open?id=19IhT8 ... 6PoJw5DoRC

Even though it is enormously larger, it looks to me like both colors were truncated at around 75,000 games.
So I guess the conversion is not complete.

Thanks again.

There's a lot of redundancy in such an approach, and it seems we should avoid the step of generating pseudo-games if possible.

We want something that can parse the polyglot format directly and display a tree. It would need to have a move generator of some kind.

For the demo on Zipproth's website, the GUI looks up FENs in a JSON file. I've placed the last version of that file I was able to archive here:

https://we.tl/t-J10Sv0j8kN

(File is 2.8 MB. The host is Wetransfer and is not too annoying.)

The polyglot format spec is here:

http://hgm.nubati.net/book_format.html

Brainfish needs the weights, right? Looking at Cerebellum_Light_Poly.bin in a hex editor just now, all the weights appear to be set to 00 FF!

I know Brainfish has an option to randomly play 2nd-best moves... it needs access to the weights to do that. Am I parsing this file wrong?

-Carl

[Ferdy]

Here is a link for the PGN built using the technique described using polyglot:

https://drive.google.com/open?id=19IhT8 ... 6PoJw5DoRC

Even though it is enormously larger, it looks to me like both colors were truncated at around 75,000 games.
So I guess the conversion is not complete.

Thanks again.

There's a lot of redundancy in such an approach, and it seems we should avoid the step of generating pseudo-games if possible.

We want something that can parse the polyglot format directly and display a tree. It would need to have a move generator of some kind.

For the demo on Zipproth's website, the GUI looks up FENs in a JSON file. I've placed the last version of that file I was able to archive here:

https://we.tl/t-J10Sv0j8kN

(File is 2.8 MB. The host is Wetransfer and is not too annoying.)

That one would be easier to parse and with lots of info already, but of course this is for demo purposes file is not too big. Only the author can produce that json data.

The polyglot format spec is here:

http://hgm.nubati.net/book_format.html

Brainfish needs the weights, right? Looking at Cerebellum_Light_Poly.bin in a hex editor just now, all the weights appear to be set to 00 FF!

I know Brainfish has an option to randomly play 2nd-best moves... it needs access to the weights to do that. Am I parsing this file wrong?

In Cerebellum_Light_Poly.bin there are positions where it offers 2 moves. Example from startpos.



And after e4 we have.

There's a lot of redundancy in such an approach, and it seems we should avoid the step of generating pseudo-games if possible.

Dumping of book moves indeed has lots of book lines. This is good as all transpositions will be captured. What probably is needed is to make a selective extraction, for example we want a nimzo-indian, we can use pgn-extract and tell it to output only those positions that starts with d4 Nf6 c4 e6 Nc3 Bb4 using the dump pgn as the source.

[Ferdy]

Last night, Thomas Zipproth released a new Cerebellum Library for Brainfish. First new release since the middle of 2018(!)

Download Cerebellum and latest Brainfish here:

http://www.zipproth.de/#Brainfish_download


I started the testrun... Result not before next Monday on my website (https://www.sp-cc.de).
Stay tuned!

What is the average analysis time per position used in cerebellum?

[JollyJoker]

Are these evaluated using Stockfish? It would be interesting to see a comparison with a Leela version. Even more interesting if the AplhaZero team made a book with a trillion positions or so

[Ferdy]

Are these evaluated using Stockfish?

I think by Brainfish with Stockfish latest dev.

[Dann Corbit]

Of the 669,691 distinct board positions in the extracted part of the subset of the Cerebellum library, I have all positions analyzed, but the vast majority are weakly analyzed (my definition is less than 36 plies).

654,569 are less than 36 plies
616,896 are less than 30 plies
It will take more than a year to analyze all of them to 36 plies or better.

On the other hand, I may have side table analysis for a majority of them. (All the contests like CCRL, CEGT, TCEC, etc. are decomposed into analyzed EPD records in tables named after the contests.). So the workload might not really be that daunting.

Some of the contests (e.g. Sedat's book contests) do not have analysis embedded, but only the PGN {and in a few cases time used}. So those are good for statistics, but not helpful for engine analysis.

To me, "make this move 100%" or "make this move 33%" is not all that helpful. I want to know why.

[Dann Corbit]

Here is a link to all of the distinct positions in the extracted part of the subset of the Cerebellum library:
https://drive.google.com/open?id=1HgwCF ... Xga02YfNV0

Remember, that this is "Cerebellum Light" so not all of the stuff is inside it.

As to the missing evaluation data, keep in mind that they switched to polyglot book format, which does not have evaluation data in it.
I guess that the polyglot book is the final output, but the intermediate book format will have the evaluation data and the real statistics in it.

[Jonathan003]

I would be very interested in hiring a chess programmer to rip the book into a format that could be used with something like Chess Position Trainer. Paying work. Please spread the word and PM me if interested.

-Carl

It is possible in Scid vs PC to export a bin book to a big merged pgn game. http://scidvspc.sourceforge.net/
There is a limitation of 3000 moves to export a book to pgn.
Changing the line "exportMax 3000" in scid/scid.gui is what is needed. The scid.gui is in the installation directory of Scid vs PC

I just had to open the file scid.gui in a text editor and change "exportMax 3000" to something like "exportMax3000000000" for exemple.
Scid can not handle merged games with more than 3000 moves. But you can save the merged pgn game and import it in Chess Position Trainer.

I didn't tried it yet with really big bin books. But maybe it will work if you let it run for nights and days. Only the moves will be in the merged pgn game, an no weights.

I try to find out how I can do the same for Scid 4.7 https://sourceforge.net/projects/scid/f ... cid%204.7/

I think this version is faster for exporting bin books to pgn. But I can't find this scid.gui file in the insallation directory.
Maybe someone can help me with this?

[Jonathan003]

I tried it with some book and come to the conclusion that not all lines are there. Scid vs PC export 3500 moves from a bin book I used with my own game. Than it quits and I don't get a warning. So I need an other way to convert a bin book to pgn. I think I will divide the bin book to small parts. So the limit of 3000 moves is no exceeded.