Database snapshot

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
OneTrickPony
Posts: 95
Joined: Mon Apr 29, 2013 11:29 pm

Re: Database snapshot

Post by OneTrickPony » Mon Jul 29, 2019 6:42 pm

Very interesting, thank you very much.
One question: When I find a leaf in the tree (position after last given move not found) does it mean that this was evaluated with Stockfish depth 22 and no other analysis starting from that position was performed?

OneTrickPony
Posts: 95
Joined: Mon Apr 29, 2013 11:29 pm

Re: Database snapshot

Post by OneTrickPony » Mon Jul 29, 2019 7:15 pm

It unfortunately wants to play a losing line in Vienna with black (and Vienna is its first choice). It's the same problem Leela has searching from the position after 4th move:



First choices of the book so far for black (with the transposition of 1...Nf6/e6/d5) and this position it only gives:
Bxd5 0 50.00

Unfortunately for black 12.Ng5+ is a forced win for white (which btw my SF sees at depth 22 already)

As an anecdote: I played a Lichess blitz game vs a cheater with white in this line and they have fallen into it (probably were using SF on shallow depths as beginners don't know how to change the default settings/use more threads etc.). No human goes into that line as it just looks very risky. They were banned by Lichess a while later.

noobpwnftw
Posts: 362
Joined: Sun Nov 08, 2015 10:10 pm

Re: Database snapshot

Post by noobpwnftw » Mon Jul 29, 2019 8:11 pm

OneTrickPony wrote:
Mon Jul 29, 2019 6:42 pm
Very interesting, thank you very much.
One question: When I find a leaf in the tree (position after last given move not found) does it mean that this was evaluated with Stockfish depth 22 and no other analysis starting from that position was performed?
Yes.
OneTrickPony wrote:
Mon Jul 29, 2019 7:15 pm
As an anecdote: I played a Lichess blitz game vs a cheater with white in this line and they have fallen into it (probably were using SF on shallow depths as beginners don't know how to change the default settings/use more threads etc.). No human goes into that line as it just looks very risky. They were banned by Lichess a while later.
Of course, there must be infinite number of holes, or chess would be solved. :D

I have a Lichess bot which uses this database as its opening book, so it could be tested on whether the self-improving method works in practice.

User avatar
Ovyron
Posts: 2830
Joined: Tue Jul 03, 2007 2:30 am

Re: Database snapshot

Post by Ovyron » Mon Jul 29, 2019 10:06 pm

OneTrickPony wrote:
Mon Jul 29, 2019 7:15 pm
It unfortunately wants to play a losing line in Vienna with black (and Vienna is its first choice). It's the same problem Leela has searching from the position after 4th move:



First choices of the book so far for black (with the transposition of 1...Nf6/e6/d5) and this position it only gives:
Bxd5 0 50.00

Unfortunately for black 12.Ng5+ is a forced win for white (which btw my SF sees at depth 22 already)
Thanks! This was exactly the thing I would have tried to find first by querying the database :D

My analysis deviates with 9...b5 with backsolved score of 0.36, but in general, this is a line black doesn't want to play (4...Be7!! is 0.00). I'm glad 1000 times less positions analyzed can hold.
Great spirits have always encountered violent opposition from mediocre minds.

Ferdy
Posts: 4113
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Database snapshot

Post by Ferdy » Mon Jul 29, 2019 10:47 pm

Ovyron wrote:
Mon Jul 29, 2019 5:41 pm
Thanks, I downloaded your exe, ran it in some directory and pasted a fen. I got this error:

Traceback (most recent call last):
File "book_probe2.py", line 55, in <module>
File "book_probe2.py", line 50, in main
File "book_probe2.py", line 26, in probe_book
IndexError: list index out of range
[172] Failed to execute script book_probe2
Use fen and not epd.
If you only have epd, just add "0 1" at the end to make it a fen.

User avatar
Ovyron
Posts: 2830
Joined: Tue Jul 03, 2007 2:30 am

Re: Database snapshot

Post by Ovyron » Mon Jul 29, 2019 10:54 pm

Ferdy wrote:
Mon Jul 29, 2019 10:47 pm
Use fen and not epd.
If you only have epd, just add "0 1" at the end to make it a fen.
Thanks, now it works.

This is really cool, I kind of feel like I downloaded some huge database that weighs just 9MB :)
Great spirits have always encountered violent opposition from mediocre minds.

Ferdy
Posts: 4113
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Database snapshot

Post by Ferdy » Tue Jul 30, 2019 8:40 am

Ovyron wrote:
Mon Jul 29, 2019 10:54 pm
Ferdy wrote:
Mon Jul 29, 2019 10:47 pm
Use fen and not epd.
If you only have epd, just add "0 1" at the end to make it a fen.
Thanks, now it works.

This is really cool, I kind of feel like I downloaded some huge database that weighs just 9MB :)
Also working on simple html interface, I hope to release it next week.

Image

User avatar
Ovyron
Posts: 2830
Joined: Tue Jul 03, 2007 2:30 am

Re: Database snapshot

Post by Ovyron » Tue Jul 30, 2019 1:37 pm

Ferdy wrote:
Mon Jul 29, 2019 10:47 pm
Ovyron wrote:
Mon Jul 29, 2019 5:41 pm
Thanks, I downloaded your exe, ran it in some directory and pasted a fen. I got this error:

Traceback (most recent call last):
File "book_probe2.py", line 55, in <module>
File "book_probe2.py", line 50, in main
File "book_probe2.py", line 26, in probe_book
IndexError: list index out of range
[172] Failed to execute script book_probe2
Use fen and not epd.
If you only have epd, just add "0 1" at the end to make it a fen.
Hi Ferdy, I tried that for this position:



But entering:

r1bqkb1r/pppp1ppp/2n2n2/4p1N1/2B1P3/8/PPPP1PPP/RNBQK2R b KQkq - 0 1

Produces errors and it exits.
Great spirits have always encountered violent opposition from mediocre minds.

Ferdy
Posts: 4113
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Database snapshot

Post by Ferdy » Tue Jul 30, 2019 2:45 pm

Ovyron wrote:
Tue Jul 30, 2019 1:37 pm
Ferdy wrote:
Mon Jul 29, 2019 10:47 pm
Ovyron wrote:
Mon Jul 29, 2019 5:41 pm
Thanks, I downloaded your exe, ran it in some directory and pasted a fen. I got this error:

Traceback (most recent call last):
File "book_probe2.py", line 55, in <module>
File "book_probe2.py", line 50, in main
File "book_probe2.py", line 26, in probe_book
IndexError: list index out of range
[172] Failed to execute script book_probe2
Use fen and not epd.
If you only have epd, just add "0 1" at the end to make it a fen.
Hi Ferdy, I tried that for this position:



But entering:

r1bqkb1r/pppp1ppp/2n2n2/4p1N1/2B1P3/8/PPPP1PPP/RNBQK2R b KQkq - 0 1

Produces errors and it exits.
You found a bug. The exe expects to have a winrate but there is a move from the database that does not give a win rate and this is:

Code: Select all

{'uci': 'c6e7', 'san': 'Ne7', 'score': -29998, 'rank': 0, 'note': '? (01-01)'}
See there is no 'winrate'. Its winrate should have been 0.0 assuming winrate has a range [0, 100], %.

I can modify the exe to handle such error and not exit the program, but lets wait for noobpwnftw to fix it by sending the winrate even if it is zero.


The html interface handles it by itself.

Image

Ferdy
Posts: 4113
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Database snapshot

Post by Ferdy » Tue Jul 30, 2019 11:51 pm

Ovyron wrote:
Tue Jul 30, 2019 1:37 pm
Ferdy wrote:
Mon Jul 29, 2019 10:47 pm
Ovyron wrote:
Mon Jul 29, 2019 5:41 pm
Thanks, I downloaded your exe, ran it in some directory and pasted a fen. I got this error:

Traceback (most recent call last):
File "book_probe2.py", line 55, in <module>
File "book_probe2.py", line 50, in main
File "book_probe2.py", line 26, in probe_book
IndexError: list index out of range
[172] Failed to execute script book_probe2
Use fen and not epd.
If you only have epd, just add "0 1" at the end to make it a fen.
Hi Ferdy, I tried that for this position:



But entering:

r1bqkb1r/pppp1ppp/2n2n2/4p1N1/2B1P3/8/PPPP1PPP/RNBQK2R b KQkq - 0 1

Produces errors and it exits.
New version of exe
https://drive.google.com/file/d/1_KM89V ... sp=sharing

You can now just input epd.

python source code.

Code: Select all

#!/usr/bin/env python3
"""
probe_chessdb_book.py

Probe book moves in ChessDB

Requirements:
    * Python 3
    * Requests
        pip install requests
        
Dev log:
    v0.1
    * Use http
    * Add pv of bestmove results
    * Add Database stats results
    * Use fen directly after ...board=
    * epd can now be entered
    * Handle errors in querying all&board
"""

import requests
import json
import time


def probe_book(fen):
    data = []
    
    # Query all moves
    base = 'http://www.chessdb.cn/cdb.php?action=queryall&json=1&board='
    url = base + fen
    
    try:
        r = requests.get(url)
        r.raise_for_status()
    except requests.exceptions.HTTPError as errh:
        print("Http Error:",errh)
        return
    except requests.exceptions.ConnectionError as errc:
        print("Error Connecting:",errc)
        return
    except requests.exceptions.Timeout as errt:
        print("Timeout Error:",errt)
        return
    except requests.exceptions.RequestException as err:
        print("OOps: Something Else",err)
        return
        
    jdata = r.text
    
    d = json.loads(jdata)
    if d['status'] == 'ok':
        
        for n in d['moves']:
            try:
                data.append([n['san'], int(n['score']), float(n['winrate'])])
            except KeyError:
                pass
                
    else:
        print('Position is not found!')
        
    if len(data) > 0:
        print('(1) Move Stats:')
        print('{:6s}  {:<5s}  {}'.format('Move', 'Score', 'Winrate'))
        for d in data:
            print('{:6s}  {:<5d}  {:0.2f}'.format(d[0], d[1], d[2]))
    
    # Query pv
    base = 'http://www.chessdb.cn/cdb.php?action=querypv&json=1&board='
    url = base + fen
    r = requests.get(url)
        
    jdata = r.text
    d = json.loads(jdata)

    if d['status'] == 'ok':
        score = d['score']
        depth = d['depth']
        pv = ' '.join(d['pvSAN'])
        
        print()
        print('(2) PV info of top move:')
        print('{}/{} pv {}'.format(score, depth, pv))
        
    # Query database stats
    print()
    print('(3) Database stats:')
    url = 'http://www.chessdb.cn/statsc.php?lang=1'
    try:
        r = requests.get(url)
        res = r.text
        sp = res.split('<table class="stats"><tr><td>DB Size ( Storage / Queue ) :</td><td style="text-align: right;">')[1]
        sp = sp.split('</td>')
        value = sp[0].strip()
        print('DB Size ( Storage / Queue ) : {}'.format(value))
    except Exception as err:
        print(err)
    
    
def main():  
    print('Probe ChessDB Book v0.1')
    while True:
        fen = input('enter fen or epd or q to quit? ')
        if fen.lower() == 'q':
            break
        probe_book(fen)
        time.sleep(2)
    

if __name__ == '__main__':
    main()

Sample run:

Code: Select all

Probe ChessDB Book v0.1
enter fen or epd or q to quit? r1bqkb1r/pppp1ppp/2n2n2/4p1N1/2B1P3/8/PPPP1PPP/RN
BQK2R b KQkq -
(1) Move Stats:
Move    Score  Winrate
d5      -5     49.62
Bc5     -175   37.04
Nxe4    -248   32.05
Qe7     -290   29.34
Nd5     -365   24.86
b5      -375   24.30
d6      -418   21.98
g6      -453   20.22
Na5     -461   19.83
Ba3     -473   19.26
a5      -487   18.61
Bb4     -509   17.62
Bd6     -512   17.49
a6      -526   16.88
b6      -589   14.37
Rb8     -599   14.00
Nd4     -612   13.53
Nb4     -612   13.53
Nb8     -636   12.71
Rg8     -651   12.21
Ng8     -721   10.11
Ke7     -725   10.00
Be7     -730   9.87
h6      -780   8.60
Ng4     -796   8.23
h5      -805   8.02
Nh5     -885   6.41

(2) PV info of top move:
-5/30 pv d5 exd5 Na5 Bb5+ c6 dxc6 bxc6 Bd3 Nd5 Nf3 Bd6 O-O O-O Re1 f5 Nxe5 Qf6 N
f3 g5 g3 f4 Nc3 fxg3 hxg3 Qxf3 Qxf3 Rxf3 Be4 Rf7 Nxd5

(3) Database stats:
DB Size ( Storage / Queue ) : 63.92 GB / 17.60 GB
enter fen or epd or q to quit?

Post Reply