OpenchessDB + New Wasm site.

[jshriver]

Good Morning,

I've setup a site for my openchessdb project so people can contribute by simply viewing the website. It uses a Stockfish 16 nnue wasm build and automates getting work units and sending back results for the collection. Still planning to release a native client but my next 2 tasks is to create a login system. Idea is to mimic boinc a bit so each user can keep stats of how many work units they process. The actual viewer itself to probe the database is nearly done and aiming to get it and the new bootstrap UI out there soon.

If interested in donating some CPU time feel free to load this site on any device for as long or little as you want. All contributions are appreciated.

https://openchessdb.org/wasm

Happy holidays everyone.
-Josh

[jshriver]

Future plans are to release the client I've been using the past 3 years. I'm finishing up the core functionality of the web UI for the main viewer so people can do look ups either by FEN input or by setting the board up themselves. Planning to create a user account system to make it more boinc like. Contributors like to know statistics on their contributions and also rewards be it achievements for work done and ranking. Saved this for last but feel it will ultimately help get more people interested in contributing. Those are the two main things left till the site goes fully public.

The goal is something like lichess's analysis page but has pre-computed information you can pull up. Want to see what an engine has to say just input a position. Unlike other offerings, this gives full PV and my inputs aim to be mid-level deep, right now depth 28 for stockfish and 12 for lc0. I keep engines current so data is current. My client uses a json file so users can define various things lie hash size, core count, even the engine itself. I'm aiming for the strongest engine so been using stockfish and lc0. Though early on I did do a good burn using Dragon 2 when I bought it. Really wish Torch had a Linux build for sell would buy even just to use for the seeding 62k eco positions.

Another item on my to do list is that I took the ECO and exported all possible FEN positions and turns out its around 62k unique opening positions. I used various top engines to deep dive each one. Stockfish alone took me 4 months with 15.1 and redid with 16 in 3 months. Including lc0, Dragon 2 and one or two other engines. So openings will be most fruitful for data and also multiple engine analysis. My todo is to add that data to the core database.


I'm also working on something akin to lichess's WDL opening. Code is done but sadly I either need to limit it's depth or pruning or rent a box with over 100gigs+ of ram. Ran it a few days multiple times in various code changes and hitting a huge curve due to memory lookup as table counts explode. Think I found a best approach then sadly my dev box died. Anyway everything is backed up. Hoping to resume in Jan. Using only GM rated games from a mix of Caissebase + updated with TWIC. Believe the total count of useful data in unique FEN count is just over 120gigs in raw form (nice size memory table of uniq FENs)

Going to try as a side note to bring back my old oics.olympuschess.com torrent site. The old one had a very outdated php based server with security issues. I'd like to host that again under the openchessdb.org site.

I'm also open to suggestions! I realize this data might be useful for people training neural nets and willing to share the entire database. Still trying to figure out the proper license. But will definitely be open source (hence the name).

Thanks to those who have joined!
It's not public (till bots farm this) but you can view your workload as it shows up here. Ignore nodes those are my personal boxes I currently track when I do release my client I will not include such data other than uuid and maybe OS version. Even for myself I purge every 5min its only to get a feel for how many nodes are currently working (not applied to wasm site).

https://openchessdb.org/new/workloadwasm.html

-Josh

P.S. This project means a lot to me so I greatly appreciate recommendations and criticisms. I know its already been done but not to this detail. Been working on it for just over 3 years, so it helped me through covid (smile). I will never make a top notch engine, but I love data analysis so seemed like a marriage made in heaven.

[jdart]

Chessbase has an online database and opening book, but there was another existing freeware site with online browsable DB and REST API - I don't remember the details. I haven't visited it for a while.

IMO it is important to record WLD. One of the issues with Polyglot books is that they don't have that, just weights. Arasan uses some fairly complex logic to select opening moves from its own book, and WLD is one of the inputs, but not the only one.

Another issue is that to intelligent select moves, you really need to do minimax. For example, a move that has a high winning percentage might be refuted by a line that has few examples in the db, but gives a good score to the opponent.

Correspondence games, which are really very high-depth computer games these days, are a very good source of opening knowledge. Games from ICCF are available for download (you have to register). My opening book makes use of these among other sources. Frequently you'll find that a certain line used to be popular, but if the correspondence players stop using it, that is a pretty sure sign there is something wrong with it.

[jshriver]

Thanks for the info. I'm a member of ICCF however I've not actually played a match yet. Will dig around their site for their PGNs, thanks for the heads up.

[Shaun]

Hi Josh,

once logged in this should give you the games

https://www.iccf.com/message?message=454

Shaun