Modular opening book SF analysed 87417 pos., beta-1

Frank Quisinsky · Post by **Frank Quisinsky** » Sun Feb 26, 2017 8:54 am

Hi Juan,

interesting is ...

People likes Stockfish, Komodo and Houdini engines only can used Beta 3 of my files. Means the Basic database from Beta 3 after Houdini analysis.

People like other engines too ... can used the final version.

I would like to say that to each step all files will be included in my download files.

You can be sure that after Fizbo a lot will recject, not important 6 other engines calculated before the same again and again.

For that reason I try that the Basics (3-moves after ECO Code was formed) is very big. It's quiet clear that engines have differnet opinions in eval.

Best
Frank

Hope I am thinking on all the possible cases in engine testing. An other important perspective are all the fast draw games with 3-fold repetition. In 80% of my Basics such positions for fast 3-fold repetition is reject. Because games based on FCP Live Book. So the opening book later will not have holes ... possible that main lines will give us a 3-fold Repetition and aren't included in the database. So not a hole ... a feature!

Again ... all should be a good Basic for engine tourneys and Rating list systems. A database with a lot of balanced positions as opening book with all the 500 ECO codes. In my opinion perfect for testing engines!

Ozymandias · Post by **Ozymandias** » Sun Feb 26, 2017 9:03 am

Frank Quisinsky wrote:Biggest problem:
Varied interpretation of engines in eval.

Indeed, which brings us to the much broader question, is eval a good metric for book building? I always favoured game stats.

Frank Quisinsky · Post by **Frank Quisinsky** » Sun Feb 26, 2017 9:23 am

Hi Juan,

that is indeed a good question!

Game stats are in my opinion bad because from newer theory not enough games are in the book stats and such games played rarely in comp-comp matches. So you can see, the idea with game stats is old and today in my opinion not good enough.

Not good enough for more variability.

But indeed what you wrote in right also.
I can't say I reject lines if allways (not important which engine analysed) the eval is more as 0.50 or -0.30.

Houdini gave very often (of course without contempt) around + or -0.25 more as Komodo and Stockfish.

In case of Houdini I reject I think with more as 0.65 or -0.35.
Not sure, must see the first analysis from Houdini.

What I wrote are more knowledge after checking all the FCP Rating List games. I know which engines from TOP-41 gave higher evals or not. So many hours I am working on FCP Live Book in the past.

All will be good if ready!

Think so ...

Best
Frank

PS:
If we have a clear database game stats are not important. Means if all lines checked again and again. Advantage: The book is much smaler ... if I am ready my book with balanced lines will be very small. Thinking on hundrets of MB People downloading for big books with all the bad book stats. And if People used book stats ... on the end of the line ... a small mistake and all the book stats are ineffective.

Again I am not a fan of book stats.
Checking so many books before ...

But honestly ...what the Hiarcs book programmer do here is fantastic. In my opinion really the best opening book with book stats I ever saw.

Frank Quisinsky · Post by **Frank Quisinsky** » Sun Feb 26, 2017 9:40 am

r4rk1/2pqbppp/p1np1nb1/1p1Bp3/3PP1P1/2P2N1P/PP3P2/RNBQR1K1 w - - c0 7; ; ce -42; acd 27; acs 30; acn 293273771; pv dxe5 Sxd5 ;

You can see ...
-0.42 ... a position with 7 move transpositions in main database (82.704 with or 26.629 without move transpositions).

7 times not seen by Stockfish .... more for Stockfish OK.
Very complicated position!

And this position, sure here you will have in your book stat for 7 or more times, can't be good ... I believe Komodo is more right. Position is bad for white in my opinion.

But all in all ... position should not included in database ... reject with the next update.

And such things I will see here ... from the complete playable theory, 3 moves after ECO was formed.

Now after the first engine calculated all the positions ... the next weeks are really very exciting.

Best
Frank

Vinvin · Post by **Vinvin** » Sun Feb 26, 2017 10:52 am

With diagram :

Frank Quisinsky wrote:[d]r4rk1/2pqbppp/p1np1nb1/1p1Bp3/3PP1P1/2P2N1P/PP3P2/RNBQR1K1 w - - c0 7; ; ce -42; acd 27; acs 30; acn 293273771; pv dxe5 Sxd5 ;

You can see ...
-0.42 ... a position with 7 move transpositions in main database (82.704 with or 26.629 without move transpositions).

7 times not seen by Stockfish .... more for Stockfish OK.
Very complicated position!
...
Frank

Ferdy · Post by **Ferdy** » Sun Feb 26, 2017 1:15 pm

Frank Quisinsky wrote: Biggest problem:
Varied interpretation of engines in eval.

I have a suggestion on how to best understand and interpret the evaluation of each engine. Take a range of evaluation of each engine and calculate its performance that is score/game or (wins + draws/2) / (wins + loses + draws)

Case 1:
Downloaded white and black games of Komodo 10.2 x64 C15, and calculate its performace for every positions in every game from moves 13 to 24 and check its eval from -1.0 pawn to +1.0 pawn units or score window [-1.0, 1.0] and see if in that game it won, lost or draws.

Here are some results.

Code: Select all

Summary:
                 players,      min,      max,     Pcnt,     Wcnt,     Lcnt,     Dcnt,  perf(%),
     Komodo 10.2 x64 C15,    -1.00,    -0.90,        6,        0,        2,        4,   33.33,
     Komodo 10.2 x64 C15,    -0.89,    -0.79,        8,        0,        3,        5,   31.25,
     Komodo 10.2 x64 C15,    -0.78,    -0.68,       10,        0,        2,        8,   40.00,
     Komodo 10.2 x64 C15,    -0.67,    -0.57,       10,        3,        5,        2,   40.00,
     Komodo 10.2 x64 C15,    -0.56,    -0.46,       57,       17,        9,       31,   57.02,
     Komodo 10.2 x64 C15,    -0.45,    -0.35,      160,       83,       13,       64,   71.88,
     Komodo 10.2 x64 C15,    -0.34,    -0.24,      385,      196,       14,      175,   73.64,
     Komodo 10.2 x64 C15,    -0.23,    -0.13,     1040,      512,       26,      502,   73.37,
     Komodo 10.2 x64 C15,    -0.12,    -0.02,     1562,      873,       36,      653,   76.79,
     Komodo 10.2 x64 C15,    -0.01,     0.09,     5184,     3034,       80,     2070,   78.49,
     Komodo 10.2 x64 C15,     0.10,     0.20,     3333,     2273,       18,     1042,   83.83,
     Komodo 10.2 x64 C15,     0.21,     0.31,     3449,     2556,       15,      878,   86.84,
     Komodo 10.2 x64 C15,     0.32,     0.42,     3061,     2424,       12,      625,   89.40,
     Komodo 10.2 x64 C15,     0.43,     0.53,     2490,     2106,       22,      362,   91.85,
     Komodo 10.2 x64 C15,     0.54,     0.64,     1829,     1620,       10,      199,   94.01,
     Komodo 10.2 x64 C15,     0.65,     0.75,     1221,     1147,        3,       71,   96.85,
     Komodo 10.2 x64 C15,     0.76,     0.86,      809,      768,        2,       39,   97.34,
     Komodo 10.2 x64 C15,     0.87,     0.97,      624,      607,        3,       14,   98.40,
     Komodo 10.2 x64 C15,     0.98,     1.08,      390,      380,        1,        9,   98.59,

Total real games: 2450
Eval window: -1.00 to 1.00
Move range: 13 to 24
Elapsed time: 2.75m

Table interpretation.
When Komodo has an eval of from -0.01 to 0.09 (pawn unit), it has a performance of 78.49%.

Code: Select all

                 players,      min,      max,     Pcnt,     Wcnt,     Lcnt,     Dcnt,  perf(%),
     Komodo 10.2 x64 C15,    -0.01,     0.09,     5184,     3034,       80,     2070,   78.49,

Wcnt = count of pos at given eval window and move range from the games whose result is a win by Komodo = 3034
Lcnt = from the games whose result is a loss by Komodo = 80
Dcnt =from the games whose result is a draw by Komodo = 2070
Pcnt = total positions evaluated = Wcnt + Lcnt + Dcnt = 5184

pts = Wcnt + Dcnt/2
perf = pts/Pcnt

The perft is high because most engines are way too weak for Komodo.
Game source:

Code: Select all

Results from file komodo.pgn:

No. Name                           Win Draw Loss Unf.  Score Games       %
--------------------------------------------------------------------------
  1 Komodo 10.2 x64 C15           +1818 =607  -25   *0 2121.5  2450   86.6%
  2 SF 18Sep2016 BMI2 x64 C10       +9  =33   -8   *0   25.5    50   51.0%
  3 Houdini 4 STD B x64             +4  =25  -21   *0   16.5    50   33.0%
  4 Fire 4 x64                      +0  =27  -23   *0   13.5    50   27.0%
  5 Critter 1.6a x64                +1  =23  -26   *0   12.5    50   25.0%
  6 Equinox 3.30 x64                +2  =21  -27   *0   12.5    50   25.0%
  7 Nirvanachess 2.3 POP x64        +1  =23  -26   *0   12.5    50   25.0%
  8 GullChess 3.0 BMI2 x64          +0  =24  -26   *0   12.0    50   24.0%
  9 iCE 3.0 v658 POP x64            +2  =18  -30   *0   11.0    50   22.0%
 10 Fizbo 1.8 BMI2 x64              +1  =19  -30   *0   10.5    50   21.0%
 11 Protector 1.9.0 x64             +1  =18  -31   *0   10.0    50   20.0%
 12 Andscacs 0.872 BMI2 x64         +0  =18  -32   *0    9.0    50   18.0%
 13 Fritz 15 x64                    +0  =18  -32   *0    9.0    50   18.0%
 14 Shredder 12 x64                 +1  =16  -33   *0    9.0    50   18.0%
 15 Vajolet2 2.2 POP x64            +0  =18  -32   *0    9.0    50   18.0%
 16 Hannibal 1.7 x64                +0  =17  -33   *0    8.5    50   17.0%
 17 Chiron 3 x64                    +0  =15  -35   *0    7.5    50   15.0%
 18 SmarThink 1.96 SSE4 x64         +1  =13  -36   *0    7.5    50   15.0%
 19 Booot 6.0.2 POP x64             +1  =12  -37   *0    7.0    50   14.0%
 20 Texel 1.06 x64                  +0  =14  -36   *0    7.0    50   14.0%
 21 Hakkapeliitta TCEC v2 x64       +0  =12  -38   *0    6.0    50   12.0%
 22 Naum 4.6 x64                    +0  =12  -38   *0    6.0    50   12.0%
 23 Senpai 1.0 SSE42 x64            +0  =12  -38   *0    6.0    50   12.0%
 24 Spike 1.4 Leiden w32            +0  =12  -38   *0    6.0    50   12.0%
 25 Deuterium 14.3.34.130 POP x64   +0  =10  -40   *0    5.0    50   10.0%
 26 Sjeng c't 2010 w32              +0  =10  -40   *0    5.0    50   10.0%
 27 TogaII 280513 Intel w32         +0  =10  -40   *0    5.0    50   10.0%
 28 Zappa Mexico II x64             +0  =10  -40   *0    5.0    50   10.0%
 29 Cheng 4.39 x64                  +0   =9  -41   *0    4.5    50    9.0%
 30 DiscoCheck 5.2.1 x64            +0   =9  -41   *0    4.5    50    9.0%
 31 Gaviota 1.0 AVX x64             +0   =9  -41   *0    4.5    50    9.0%
 32 Junior 13.3.00 x64              +1   =7  -42   *0    4.5    50    9.0%
 33 Pedone 1.4.1 BMI2 x64           +0   =9  -41   *0    4.5    50    9.0%
 34 Rodent II 0.9.64 x64            +0   =9  -41   *0    4.5    50    9.0%
 35 Arasan 19.1 POP x64             +0   =8  -42   *0    4.0    50    8.0%
 36 Atlas 3.80 x64                  +0   =8  -42   *0    4.0    50    8.0%
 37 Bobcat 7.6 x64                  +0   =8  -42   *0    4.0    50    8.0%
 38 Murka 3 x64                     +0   =8  -42   *0    4.0    50    8.0%
 39 Nemorino 1.02 POP x64           +0   =8  -42   *0    4.0    50    8.0%
 40 Dirty 03NOV2015 POP x64         +0   =7  -43   *0    3.5    50    7.0%
 41 Wasp 1.25 BMI2 x64              +0   =7  -43   *0    3.5    50    7.0%
 42 Hiarcs 14 WCSC w32              +0   =6  -44   *0    3.0    50    6.0%
 43 Octochess r5506 DC x64          +0   =6  -44   *0    3.0    50    6.0%
 44 Quazar 0.4 x64                  +0   =6  -44   *0    3.0    50    6.0%
 45 EXchess 7.92b x64               +0   =5  -45   *0    2.5    50    5.0%
 46 Spark 1.0 x64                   +0   =5  -45   *0    2.5    50    5.0%
 47 Crafty 25.0.1 JA POP x64        +0   =4  -46   *0    2.0    50    4.0%
 48 DisasterArea 1.65 POP x64       +0   =4  -46   *0    2.0    50    4.0%
 49 Nemo 1.01 Beta POP x64          +0   =3  -47   *0    1.5    50    3.0%
 50 Laser 1.2 POP x64               +0   =2  -48   *0    1.0    50    2.0%

Total Games:    2450
White Wins:      986 (40.2%)
Black Wins:      857 (35.0%)
Draws:           607 (24.8%)
Unfinished:        0 (0.0%)

Against these opponents if Komodo's eval is -0.45 ot -0.35 from move 13 to move 24 in every game it got 71.88%

Code: Select all

Komodo 10.2 x64 C15,    -0.45,    -0.35,      160,       83,       13,       64,   71.88,

It would be interesting to see how it performs vs opp close to its strength like SF, but games here is a bit low only 50.

Code: Select all

  2 SF 18Sep2016 BMI2 x64 C10       +9  =33   -8   *0   25.5    50   51.0%

Case 2:
I will run ICE next.

Frank Quisinsky · Post by **Frank Quisinsky** » Sun Feb 26, 2017 1:22 pm

Hi Ferdinand,

very interesting.
Also the idea with ice ...

Note:
Fizbo and Smarthink also Texel are really give often other evals.
And ICE is very hard here with clearly higher evals.

I reject a lot of lines in FCP Live Book ... Comes from ICE games.

For me interesting:
How I can find out the best way for recject games for the main database for my new Project.

For Stockfish ASM and Komodo 10.3 I am using ...
0.50 / -0.30

For Houdini I think I should use ...
0.60 / -0.35 or 0.65 / - 0.40
Not sure ... must look in the first analysis.

Maybe possible to calculate the differents in eval ...

What I have is my experience from checking the eval ... 3 moves after my FCP Live Book ended with your tool (you create for me) for my older FCP Rating List.

Now I have to study your idea in detail.
If I understand all ... my bad english ... very nice!

Best
Frank

Leto · Post by **Leto** » Sun Feb 26, 2017 2:43 pm

Ozymandias wrote:
Frank Quisinsky wrote:Biggest problem:
Varied interpretation of engines in eval.
Indeed, which brings us to the much broader question, is eval a good metric for book building? I always favoured game stats.

For my correspondence chess which I take very seriously I never rely on statistics, there just isn't enough games played at a high level, and to me human (and I mean any human, even Carlsen and Kasparov) games don't qualify as high level.

Ferdy · Post by **Ferdy** » Sun Feb 26, 2017 3:15 pm

Frank Quisinsky wrote:Hi Ferdinand,

very interesting.
Also the idea with ice ...

Note:
Fizbo and Smarthink also Texel are really give often other evals.
And ICE is very hard here with clearly higher evals.

I reject a lot of lines in FCP Live Book ... Comes from ICE games.

For me interesting:
How I can find out the best way for recject games for the main database for my new Project.

For Stockfish ASM and Komodo 10.3 I am using ...
0.50 / -0.30

For Houdini I think I should use ...
0.60 / -0.35 or 0.65 / - 0.40
Not sure ... must look in the first analysis.

Maybe possible to calculate the differents in eval ...

What I have is my experience from checking the eval ... 3 moves after my FCP Live Book ended with your tool (you create for me) for my older FCP Rating List.

Now I have to study your idea in detail.
If I understand all ... my bad english ... very nice!

Best
Frank

Results from ICE, also re-run komodo, due to eval range changes.

Code: Select all

Summary:
                 players,      min,      max,     Pcnt,     Wcnt,     Lcnt,     Dcnt,  perf(%),
     Komodo 10.2 x64 C15,    -1.00,    -0.91,        6,        0,        2,        4,   33.33,
     Komodo 10.2 x64 C15,    -0.90,    -0.81,        6,        0,        3,        3,   25.00,
     Komodo 10.2 x64 C15,    -0.80,    -0.71,        9,        0,        1,        8,   44.44,
     Komodo 10.2 x64 C15,    -0.70,    -0.61,        8,        2,        2,        4,   50.00,
     Komodo 10.2 x64 C15,    -0.60,    -0.51,       28,       11,        8,        9,   55.36,
     Komodo 10.2 x64 C15,    -0.50,    -0.41,       99,       38,       10,       51,   64.14,
     Komodo 10.2 x64 C15,    -0.40,    -0.31,      187,      100,       15,       72,   72.73,
     Komodo 10.2 x64 C15,    -0.30,    -0.21,      525,      263,       14,      248,   73.71,
     Komodo 10.2 x64 C15,    -0.20,    -0.11,     1044,      536,       26,      482,   74.43,
     Komodo 10.2 x64 C15,    -0.10,    -0.01,     1364,      766,       28,      570,   77.05,
     Komodo 10.2 x64 C15,    -0.00,     0.09,     5086,     2977,       76,     2033,   78.52,
     Komodo 10.2 x64 C15,     0.10,     0.19,     2949,     2008,       16,      925,   83.77,
     Komodo 10.2 x64 C15,     0.20,     0.29,     3087,     2246,       17,      824,   86.10,
     Komodo 10.2 x64 C15,     0.30,     0.39,     2792,     2210,       10,      572,   89.40,
     Komodo 10.2 x64 C15,     0.40,     0.49,     2331,     1934,       16,      381,   91.14,
     Komodo 10.2 x64 C15,     0.50,     0.59,     1912,     1652,       16,      244,   92.78,
     Komodo 10.2 x64 C15,     0.60,     0.69,     1343,     1233,        5,      105,   95.72,
     Komodo 10.2 x64 C15,     0.70,     0.79,      937,      898,        1,       38,   97.87,
     Komodo 10.2 x64 C15,     0.80,     0.89,      690,      656,        1,       33,   97.46,
     Komodo 10.2 x64 C15,     0.90,     1.00,      579,      561,        3,       15,   98.19,

Total real games: 2450
Eval window: -1.00 to 1.00
Move range: 13 to 24
Elpased time: 2.74m

Code: Select all

Summary:
                 players,      min,      max,     Pcnt,     Wcnt,     Lcnt,     Dcnt,  perf(%),
    iCE 3.0 v658 POP x64,    -1.00,    -0.91,      503,       70,      234,      199,   33.70,
    iCE 3.0 v658 POP x64,    -0.90,    -0.81,      664,      113,      284,      267,   37.12,
    iCE 3.0 v658 POP x64,    -0.80,    -0.71,      798,      141,      361,      296,   36.22,
    iCE 3.0 v658 POP x64,    -0.70,    -0.61,     1048,      190,      379,      479,   40.98,
    iCE 3.0 v658 POP x64,    -0.60,    -0.51,     1442,      290,      506,      646,   42.51,
    iCE 3.0 v658 POP x64,    -0.50,    -0.41,     1680,      341,      529,      810,   44.40,
    iCE 3.0 v658 POP x64,    -0.40,    -0.31,     1890,      455,      569,      866,   46.98,
    iCE 3.0 v658 POP x64,    -0.30,    -0.21,     2072,      570,      560,      942,   50.24,
    iCE 3.0 v658 POP x64,    -0.20,    -0.11,     2175,      591,      533,     1051,   51.33,
    iCE 3.0 v658 POP x64,    -0.10,    -0.01,     1820,      538,      418,      864,   53.30,
    iCE 3.0 v658 POP x64,    -0.00,     0.09,     7973,     2274,     1566,     4133,   54.44,
    iCE 3.0 v658 POP x64,     0.10,     0.19,     2641,      926,      514,     1201,   57.80,
    iCE 3.0 v658 POP x64,     0.20,     0.29,     2889,     1105,      474,     1310,   60.92,
    iCE 3.0 v658 POP x64,     0.30,     0.39,     3185,     1314,      516,     1355,   62.53,
    iCE 3.0 v658 POP x64,     0.40,     0.49,     2829,     1212,      382,     1235,   64.67,
    iCE 3.0 v658 POP x64,     0.50,     0.59,     2750,     1272,      308,     1170,   67.53,
    iCE 3.0 v658 POP x64,     0.60,     0.69,     2313,     1077,      280,      956,   67.23,
    iCE 3.0 v658 POP x64,     0.70,     0.79,     1983,      932,      226,      825,   67.80,
    iCE 3.0 v658 POP x64,     0.80,     0.89,     1722,      898,      173,      651,   71.05,
    iCE 3.0 v658 POP x64,     0.90,     1.00,     1726,      954,      153,      619,   73.20,

Total real games: 4750
Eval window: -1.00 to 1.00
Move range: 13 to 24
Elpased time: 4.06m

With a max of 0.49 comparison 91.14% for Komodo and 64.67% for iCE.

Code: Select all

Komodo 10.2 x64 C15,     0.40,     0.49,     2331,     1934,       16,      381,   91.14,

Code: Select all

iCE 3.0 v658 POP x64,     0.40,     0.49,     2829,     1212,      382,     1235,   64.67

,

With your current setting.

Code: Select all

For Stockfish ASM and Komodo 10.3 I am using ... 
0.50 / -0.30

0.50 could be too high for Komodo based from 91.14% perf

Code: Select all

Komodo 10.2 x64 C15,     0.40,     0.49,     2331,     1934,       16,      381,   91.14,

It would crash most of the opponents in your rating list easily.
Reducing 0.50 to 0.29 or 0.30 could be reasonable based from data below.

Code: Select all

Komodo 10.2 x64 C15,     0.20,     0.29,     3087,     2246,       17,      824,   86.10,

Later I will run Komodo vs other opponents that is close in strength to see what eval is at 45% to 55% perf.

Ozymandias · Post by **Ozymandias** » Sun Feb 26, 2017 3:31 pm

Frank Quisinsky wrote:Game stats are in my opinion bad because from newer theory not enough games are in the book stats and such games played rarely in comp-comp matches. So you can see, the idea with game stats is old and today in my opinion not good enough.

Not good enough for more variability.

Leto wrote:For my correspondence chess which I take very seriously I never rely on statistics, there just isn't enough games played at a high level, and to me human (and I mean any human, even Carlsen and Kasparov) games don't qualify as high level.

You both mention the problem with any DB, lack of a good quantity/quality ratio, for a given position. So we "just" need more games.

With evals, the absence of a clear correlation between a given value and the expected score (Kai did some work on this a while back) is made worse, by the reliability of said evaluation value. It can be mitigated by a big depth, but not always. It's unclear to me, at what point an eval will be enough to discard a line, but a few high quality game results, are all I ever needed.

Modular opening book SF analysed 87417 pos., beta-1

Re: Interesting is ...

Re: Modular opening book SF analysed 87417 pos., beta-1

Re: Modular opening book SF analysed 87417 pos., beta-1

Re: Have a look here ... Komodo analysis!

Re: Have a look here ... Komodo analysis!

Re: Modular opening book SF analysed 87417 pos., beta-1

Re: Modular opening book SF analysed 87417 pos., beta-1

Re: Modular opening book SF analysed 87417 pos., beta-1

Re: Modular opening book SF analysed 87417 pos., beta-1

Re: Modular opening book SF analysed 87417 pos., beta-1