Knowledge Base 1.0 is the work of months carefully building a quality Polyglot database with analyzed positions by top engines, mainly by Stockfish 12 and Stockfish 13 covering about 95% of the general opening theory. The estimated total computer time roughly is about 8 years 24/7 single cpu time limited to 5 real months 24/7 due to the use 2 powerful PC's with many cores. The average quality of of the 94 million analyzed positions is around ~3300 elo. The database contains a wealth of information for opening study and for correspondence players. Knowledge Base is made available for SCID for easy browsing through the Polyglot database by clicking on the moves.
....
http://rebel13.nl/home/scid-with-knowledge-base.html
Knowledge Base 1.0
Moderators: hgm, Rebel, chrisw
-
- Posts: 6991
- Joined: Thu Aug 18, 2011 12:04 pm
Knowledge Base 1.0
90% of coding is debugging, the other 10% is writing bugs.
-
- Posts: 7
- Joined: Tue Apr 13, 2021 10:04 pm
- Full name: Josh Ryan
Re: Knowledge Base 1.0
Wow this is awesome. Is there a way to query this database? I'm not familiar with Polyglot. I'm building a similar database with Stockfish 13 and storing the evaluations in a MongoDB database. This would make a great resource for training neural networks. Maybe consider making it available through Kaggle? https://www.kaggle.com/
-
- Posts: 6808
- Joined: Wed Nov 18, 2009 7:16 pm
- Location: Gutweiler, Germany
- Full name: Frank Quisinsky
Re: Knowledge Base 1.0
Hi Ed,
thank you again for another good work after ...
- Somu 1.7a
- a new forum
- Polyglot 1.7a
- PGN Annotator 1.2
and so many other things you do for us so many years!!
An additional hint for people like SCID!
A good source for a big database for SCID GUI is:
http://caissabase.co.uk/ (4.7 Million games)
Another good source for databases:
http://billwall.phpwebhosting.com/ (Bill Wall).
Both I found last months (new for me).
So I am looking for strong databases for a new FEOBOS project I will start with Klaus Wlotzka soon.
During my search I found these pages.
Back to Ed ...
The PGN Annotator and Polyglot works just fine.
I like it a lot!!
I will looking in the knowledge base 1.0 soon.
Thanks again and Best
Frank
All the links I added in my Link selection!
PS: ECO key from SCID and Chessbase are not really good (what I dislike).
Better key can be found in Chesspad!
And best key's can be found on DVDs from Informator!
thank you again for another good work after ...
- Somu 1.7a
- a new forum
- Polyglot 1.7a
- PGN Annotator 1.2
and so many other things you do for us so many years!!
An additional hint for people like SCID!
A good source for a big database for SCID GUI is:
http://caissabase.co.uk/ (4.7 Million games)
Another good source for databases:
http://billwall.phpwebhosting.com/ (Bill Wall).
Both I found last months (new for me).
So I am looking for strong databases for a new FEOBOS project I will start with Klaus Wlotzka soon.
During my search I found these pages.
Back to Ed ...
The PGN Annotator and Polyglot works just fine.
I like it a lot!!
I will looking in the knowledge base 1.0 soon.
Thanks again and Best
Frank
All the links I added in my Link selection!
PS: ECO key from SCID and Chessbase are not really good (what I dislike).
Better key can be found in Chesspad!
And best key's can be found on DVDs from Informator!
-
- Posts: 3186
- Joined: Sat Feb 16, 2008 7:38 am
- Full name: Peter Martan
Re: Knowledge Base 1.0
Thanks for the great work!Rebel wrote: ↑Wed Apr 14, 2021 9:30 pm Knowledge Base 1.0 is the work of months carefully building a quality Polyglot database with analyzed positions by top engines, mainly by Stockfish 12 and Stockfish 13 covering about 95% of the general opening theory. The estimated total computer time roughly is about 8 years 24/7 single cpu time limited to 5 real months 24/7 due to the use 2 powerful PC's with many cores. The average quality of of the 94 million analyzed positions is around ~3300 elo. The database contains a wealth of information for opening study and for correspondence players. Knowledge Base is made available for SCID for easy browsing through the Polyglot database by clicking on the moves.
....
http://rebel13.nl/home/scid-with-knowledge-base.html
Just to be sure, not to download something again, that I maybe already have, what's the main differenc to Knowledge.bin as a book? Can scores of the stored moves be seen now with Scid too like it worked with BareTail and the Knowledge.bin- book already?
Peter.
-
- Posts: 626
- Joined: Sun Jul 25, 2010 10:07 pm
- Location: the Netherlands
- Full name: Jef Kaan
Re: Knowledge Base 1.0
killing two birds with one message;
yep i've combined the knowledge.bin (not bad) and
the cerebellum-merged (with lco analysis) into one book.
(was still possible <2Gb book)
but i'm still puzzled about this polyglot update(s)
and i don't see a need for two options coz sofar both
options still imho look like a little but imperfect
(sure there was a problem (e4 first book etc) but for
my earlier work it was not such a problem in practice).
Yet now upon inspection (mergebooks by Ed works quite well
just like earlier versions) i noticed that it's not mergin
book moves from book1 100 pct and 2 100 pct into both
50 pct (or 70/30 or so' its now 100/0 thats all). yes you
need to be a nerd to notice that but i like gambits etc
and maybe Ed Rebel can also release his polygot (*) on
github or otherwise sourceforge like i do sometimes
https://sourceforge.net/projects/chess-gambiteer/
default engine set to 1430
Whiz kids readers here can try to beat the program, set the
gambit engine to eg 2000 (or higher) and try to beat it.
(*) fork from HGM polyglot but then x64 and compiled release
PS chess is solved (draw) but imho (repeating myself) we need
an extra parameter, sharpness (stronger engines then can crush
weaker engines (or human) when using a 'sharper' book),
for humans even GM there's still stuff to learn (even when
you cannot beat the top engine)
yep i've combined the knowledge.bin (not bad) and
the cerebellum-merged (with lco analysis) into one book.
(was still possible <2Gb book)
but i'm still puzzled about this polyglot update(s)
and i don't see a need for two options coz sofar both
options still imho look like a little but imperfect
(sure there was a problem (e4 first book etc) but for
my earlier work it was not such a problem in practice).
Yet now upon inspection (mergebooks by Ed works quite well
just like earlier versions) i noticed that it's not mergin
book moves from book1 100 pct and 2 100 pct into both
50 pct (or 70/30 or so' its now 100/0 thats all). yes you
need to be a nerd to notice that but i like gambits etc
and maybe Ed Rebel can also release his polygot (*) on
github or otherwise sourceforge like i do sometimes
https://sourceforge.net/projects/chess-gambiteer/
default engine set to 1430
Whiz kids readers here can try to beat the program, set the
gambit engine to eg 2000 (or higher) and try to beat it.
(*) fork from HGM polyglot but then x64 and compiled release
PS chess is solved (draw) but imho (repeating myself) we need
an extra parameter, sharpness (stronger engines then can crush
weaker engines (or human) when using a 'sharper' book),
for humans even GM there's still stuff to learn (even when
you cannot beat the top engine)
-
- Posts: 7
- Joined: Tue Apr 13, 2021 10:04 pm
- Full name: Josh Ryan
Re: Knowledge Base 1.0
"PS chess is solved (draw) but imho (repeating myself) we need
an extra parameter, sharpness (stronger engines then can crush
weaker engines (or human) when using a 'sharper' book),
for humans even GM there's still stuff to learn (even when
you cannot beat the top engine)"
No way. Chess is solved when we know, for each position, the guaranteed outcome if the best moves are played from that point forward.
an extra parameter, sharpness (stronger engines then can crush
weaker engines (or human) when using a 'sharper' book),
for humans even GM there's still stuff to learn (even when
you cannot beat the top engine)"
No way. Chess is solved when we know, for each position, the guaranteed outcome if the best moves are played from that point forward.
-
- Posts: 6991
- Joined: Thu Aug 18, 2011 12:04 pm
Re: Knowledge Base 1.0
I think I have about 25 million analyzed Stockfish 13 (opening) positions in EPD format. It looks like this:jryandx wrote: ↑Wed Apr 14, 2021 9:47 pm Wow this is awesome. Is there a way to query this database? I'm not familiar with Polyglot. I'm building a similar database with Stockfish 13 and storing the evaluations in a MongoDB database. This would make a great resource for training neural networks. Maybe consider making it available through Kaggle? https://www.kaggle.com/
Code: Select all
r2r2k1/ppp1nppp/3p1q2/8/2N1PP2/2P2Q2/PP4PP/R4RK1 b - - bm d5; ce -3; acd 20;
1r2k2r/p1pqbppp/5n2/3p4/8/8/PPP1QPPP/RNB1R1K1 b k - bm O-O; ce 85; acd 20;
2krr3/p1pp2pp/b1pb1n2/R7/2P4P/PqN1B3/1P2QPP1/R3KB2 b Q - bm Rxe3; ce 564; acd 20;
2kr3r/ppp2pp1/2pq4/2b1n2p/4P1b1/1BN1B1Q1/PPP2PPP/R4RK1 w - - bm f4; ce 174; acd 20;
Q1bk1b1r/p1p1qppp/3p1n2/4P3/2P5/1P6/PB3PPP/RN2KB1R b KQ - bm Nd7; ce -917; acd 20;
r3r1k1/2p2ppp/1p1q1n2/3p4/6b1/2P3P1/PPQN1PBP/R3R1K1 w - - bm a4; ce 0; acd 20;
r1bq1rk1/bpp2ppp/p2p4/P2Pn3/5R2/1N6/1PP1BPPP/2BQ1RK1 w - - bm Be3; ce -4; acd 20;
90% of coding is debugging, the other 10% is writing bugs.
-
- Posts: 6991
- Joined: Thu Aug 18, 2011 12:04 pm
Re: Knowledge Base 1.0
That's the whole point of moving to SCID, instead of the annoying Baretail moves, scores and depth are now visible in SCID and also one now can easily browse through openings by clicking on book moves.peter wrote: ↑Thu Apr 15, 2021 12:20 amThanks for the great work!Rebel wrote: ↑Wed Apr 14, 2021 9:30 pm Knowledge Base 1.0 is the work of months carefully building a quality Polyglot database with analyzed positions by top engines, mainly by Stockfish 12 and Stockfish 13 covering about 95% of the general opening theory. The estimated total computer time roughly is about 8 years 24/7 single cpu time limited to 5 real months 24/7 due to the use 2 powerful PC's with many cores. The average quality of of the 94 million analyzed positions is around ~3300 elo. The database contains a wealth of information for opening study and for correspondence players. Knowledge Base is made available for SCID for easy browsing through the Polyglot database by clicking on the moves.
....
http://rebel13.nl/home/scid-with-knowledge-base.html
Just to be sure, not to download something again, that I maybe already have, what's the main differenc to Knowledge.bin as a book? Can scores of the stored moves be seen now with Scid too like it worked with BareTail and the Knowledge.bin- book already?
90% of coding is debugging, the other 10% is writing bugs.
-
- Posts: 7
- Joined: Tue Apr 13, 2021 10:04 pm
- Full name: Josh Ryan
Re: Knowledge Base 1.0
Yeah, it's definitely usable for many purposes imo. I mean, anyone can generate it, but it takes a lot of time like you say. I'm recording games of Stockfish 13 with search depth set to 20, play against search depths 1,2,3....20 and I've built a GUI to play though these games as a way to try and learn chess (www.thechesstrainer.com) But I also want to train transformer neural networks to predict these evals, so the more data that's already available for training the better.Rebel wrote: ↑Thu Apr 15, 2021 7:37 am
I think I have about 25 million analyzed Stockfish 13 (opening) positions in EPD format. It looks like this:I can make it available for download if you think it is usable.Code: Select all
r2r2k1/ppp1nppp/3p1q2/8/2N1PP2/2P2Q2/PP4PP/R4RK1 b - - bm d5; ce -3; acd 20; 1r2k2r/p1pqbppp/5n2/3p4/8/8/PPP1QPPP/RNB1R1K1 b k - bm O-O; ce 85; acd 20; 2krr3/p1pp2pp/b1pb1n2/R7/2P4P/PqN1B3/1P2QPP1/R3KB2 b Q - bm Rxe3; ce 564; acd 20; 2kr3r/ppp2pp1/2pq4/2b1n2p/4P1b1/1BN1B1Q1/PPP2PPP/R4RK1 w - - bm f4; ce 174; acd 20; Q1bk1b1r/p1p1qppp/3p1n2/4P3/2P5/1P6/PB3PPP/RN2KB1R b KQ - bm Nd7; ce -917; acd 20; r3r1k1/2p2ppp/1p1q1n2/3p4/6b1/2P3P1/PPQN1PBP/R3R1K1 w - - bm a4; ce 0; acd 20; r1bq1rk1/bpp2ppp/p2p4/P2Pn3/5R2/1N6/1PP1BPPP/2BQ1RK1 w - - bm Be3; ce -4; acd 20;
-
- Posts: 6991
- Joined: Thu Aug 18, 2011 12:04 pm
Re: Knowledge Base 1.0
http://rebel13.nl/dl/sf13-epd.7zjryandx wrote: ↑Thu Apr 15, 2021 6:29 pmYeah, it's definitely usable for many purposes imo. I mean, anyone can generate it, but it takes a lot of time like you say. I'm recording games of Stockfish 13 with search depth set to 20, play against search depths 1,2,3....20 and I've built a GUI to play though these games as a way to try and learn chess (www.thechesstrainer.com) But I also want to train transformer neural networks to predict these evals, so the more data that's already available for training the better.Rebel wrote: ↑Thu Apr 15, 2021 7:37 am
I think I have about 25 million analyzed Stockfish 13 (opening) positions in EPD format. It looks like this:I can make it available for download if you think it is usable.Code: Select all
r2r2k1/ppp1nppp/3p1q2/8/2N1PP2/2P2Q2/PP4PP/R4RK1 b - - bm d5; ce -3; acd 20; 1r2k2r/p1pqbppp/5n2/3p4/8/8/PPP1QPPP/RNB1R1K1 b k - bm O-O; ce 85; acd 20; 2krr3/p1pp2pp/b1pb1n2/R7/2P4P/PqN1B3/1P2QPP1/R3KB2 b Q - bm Rxe3; ce 564; acd 20; 2kr3r/ppp2pp1/2pq4/2b1n2p/4P1b1/1BN1B1Q1/PPP2PPP/R4RK1 w - - bm f4; ce 174; acd 20; Q1bk1b1r/p1p1qppp/3p1n2/4P3/2P5/1P6/PB3PPP/RN2KB1R b KQ - bm Nd7; ce -917; acd 20; r3r1k1/2p2ppp/1p1q1n2/3p4/6b1/2P3P1/PPQN1PBP/R3R1K1 w - - bm a4; ce 0; acd 20; r1bq1rk1/bpp2ppp/p2p4/P2Pn3/5R2/1N6/1PP1BPPP/2BQ1RK1 w - - bm Be3; ce -4; acd 20;
40 million.
90% of coding is debugging, the other 10% is writing bugs.