Database snapshot

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
noobpwnftw
Posts: 362
Joined: Sun Nov 08, 2015 10:10 pm

Re: Database snapshot

Post by noobpwnftw » Wed Jul 31, 2019 7:48 am

Ferdy wrote:
Wed Jul 31, 2019 3:55 am
noobpwnftw wrote:
Wed Jul 31, 2019 3:10 am
finally there exists a "Discover" program people can run, which loads any UCI-compatible engine and play games, contributing moves encountered during the process.
I tried to compile ChessDBDiscover and it seems to work. Created an engine dir and put engine.exe, a renamed stockfish dev engine, then run ChessDBDiscover.exe with default config values.

Image

So what does the ChessDBDiscover.exe is doing? Does it play engine vs engine matches based on StartFEN in config file? If so how many games? Does it send the results to the cloud?
Yes, it keeps playing games and submitting results until you close the program.

Damir
Posts: 2196
Joined: Mon Feb 11, 2008 2:53 pm
Location: Denmark
Full name: Damir Desevac

Re: Database snapshot

Post by Damir » Wed Jul 31, 2019 8:20 am

noobpwnftw wrote:
Sat Jul 27, 2019 9:54 pm
For those who want to probe my database locally or for other unspecified reasons, here is a full database snapshot of my book project as of today:

ftp://ftp.chessdb.cn/pub/chessdb/data-s ... 190728.tar

The database contains about 3 billion unique chess positions, mostly connected to startpos, analyzed by Stockfish with no less than 22 plies at terminal node and has a very wide multi-pv exploration, the scores been back-propagated using a weighted averaging function, also for most of the positions there is a special field(encoded as 'a0a0') marking known shortest distance of the position from startpos.

Using this database snapshot is as simple as putting the data files under your database folder and launch the server, yet still, I'd recommend you to use the online API and make feature requests if you need any, since it is getting updated constantly and I have no plans to make such kind of snapshots very frequently(while waiting for a contributor to make incremental snapshots possible).

This database snapshot is released into the public domain.
Is it possible to download your database in PGN as PGN format ?

Damir
Posts: 2196
Joined: Mon Feb 11, 2008 2:53 pm
Location: Denmark
Full name: Damir Desevac

Re: Database snapshot

Post by Damir » Wed Jul 31, 2019 8:24 am

or somehow convert it with a tool to PGN?

Ferdy
Posts: 4113
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Database snapshot

Post by Ferdy » Wed Jul 31, 2019 9:40 am

Updated the probe_chessdb_book.exe at https://drive.google.com/file/d/1_KM89V ... sp=sharing

Positive mate score with missing winrate is now fixed. winrate is now calculated based on score returned using logistic function. So a mate in 1 has a 100% winrate now.

Code: Select all

enter fen or epd or q to quit? r1bqkb1r/ppppnppp/5n2/4p1N1/2B1P3/8/PPPP1PPP/RNBQK2R w KQkq - 6 5
(1) Move Stats:
Move    Score   Winrate
Bxf7#   29999   100.00 

(2) PV info of top move:
29999/1 pv Bxf7#

(3) Database stats:
DB Size ( Storage / Queue ) : 64.41 GB / 23.60 GB

Ferdy
Posts: 4113
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Database snapshot

Post by Ferdy » Wed Jul 31, 2019 12:50 pm

noobpwnftw wrote:
Wed Jul 31, 2019 7:48 am
Ferdy wrote:
Wed Jul 31, 2019 3:55 am
noobpwnftw wrote:
Wed Jul 31, 2019 3:10 am
finally there exists a "Discover" program people can run, which loads any UCI-compatible engine and play games, contributing moves encountered during the process.
I tried to compile ChessDBDiscover and it seems to work. Created an engine dir and put engine.exe, a renamed stockfish dev engine, then run ChessDBDiscover.exe with default config values.

Image

So what does the ChessDBDiscover.exe is doing? Does it play engine vs engine matches based on StartFEN in config file? If so how many games? Does it send the results to the cloud?
Yes, it keeps playing games and submitting results until you close the program.
I tried to run ChessDBDiscover with the position in the image for not less 2 hours. This position is the last without book info.

Image

How do I know if positions from the run are saved in cloud?

noobpwnftw
Posts: 362
Joined: Sun Nov 08, 2015 10:10 pm

Re: Database snapshot

Post by noobpwnftw » Thu Aug 01, 2019 5:28 am

Ferdy wrote:
Wed Jul 31, 2019 12:50 pm
How do I know if positions from the run are saved in cloud?
It will be there, just currently the DB has a very long queue, need a while to process them.

Ferdy
Posts: 4113
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Database snapshot

Post by Ferdy » Fri Aug 02, 2019 11:26 am

Ferdy wrote:
Tue Jul 30, 2019 8:40 am
Ovyron wrote:
Mon Jul 29, 2019 10:54 pm
Ferdy wrote:
Mon Jul 29, 2019 10:47 pm
Use fen and not epd.
If you only have epd, just add "0 1" at the end to make it a fen.
Thanks, now it works.

This is really cool, I kind of feel like I downloaded some huge database that weighs just 9MB :)
Also working on simple html interface, I hope to release it next week.

Image
Html interface is released at https://github.com/fsmosca/ChessDB-Online-Book

Download the whole thing and open index.html.

Or try it at https://fsmosca.github.io/ChessDB-Online-Book/
Make sure to enable javascript.

zullil
Posts: 5684
Joined: Mon Jan 08, 2007 11:31 pm
Location: PA USA
Full name: Louis Zulli

Re: Database snapshot

Post by zullil » Fri Aug 02, 2019 12:34 pm

Ferdy wrote:
Fri Aug 02, 2019 11:26 am

Html interface is released at https://github.com/fsmosca/ChessDB-Online-Book

Download the whole thing and open index.html.

Or try it at https://fsmosca.github.io/ChessDB-Online-Book/
Make sure to enable javascript.
Works fine if I download and open index.html. But the version online you've linked to doesn't work for me.

[EDIT] OK, Chrome is saying "Insecure content blocked. This page is trying to load scripts from unauthenticated sources." If I allow Chrome to "load unsafe scripts", then it works.

zullil
Posts: 5684
Joined: Mon Jan 08, 2007 11:31 pm
Location: PA USA
Full name: Louis Zulli

Re: Database snapshot

Post by zullil » Fri Aug 02, 2019 1:10 pm

How was the "winrate" for each position determined?

Ferdy
Posts: 4113
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Database snapshot

Post by Ferdy » Fri Aug 02, 2019 1:57 pm

zullil wrote:
Fri Aug 02, 2019 1:10 pm
How was the "winrate" for each position determined?
I suspect it is logistic function based on score. The score is from
"the scores been back-propagated using a weighted averaging function" see first post of this thread.

In javascipt when winrate is missing (by design) for mate scores, I use

Code: Select all

// 2019-08-02: ChessDB does not return winrate when score is mate.
// So we will calculate winrate by logistic function
if (typeof winrate === "undefined") {
  const K = 1.2;
  winrate = 100 * 1/(1 + (10**(-K*score/400)));
  winrate = winrate.toFixed(2);
}
Even for non-mate scores, this function is close to the winrate from server.

Post Reply