program style, risk aversion

[Ajedrecista]

Hello Don:

I'm trying a test where Houdini is handicapped to the point that it is clearly weaker, then I will try the opposite - idea is to see if the numbers are roughly equivalent.

Don

Clarification: The idea is to see if the numbers produced by the formula Jesús proposes makes any sense.

Thank you very much for your interest! I stay tuned for the results. Honestly, I will be surprised if these numbers make sense.

------------

Going off-topic: I take a look to Distributed Perft(14) Calculation almost daily and I notice that updates are frozen for weeks! Is there any problem with the project? Thanks in advance for your attention.

Regards from Spain.

ajedrecista.

[Don]

Hello Don:

I'm trying a test where Houdini is handicapped to the point that it is clearly weaker, then I will try the opposite - idea is to see if the numbers are roughly equivalent.

Don

Clarification: The idea is to see if the numbers produced by the formula Jesús proposes makes any sense.

Thank you very much for your interest! I stay tuned for the results. Honestly, I will be surprised if these numbers make sense.

------------

Going off-topic: I take a look to Distributed Perft(14) Calculation almost daily and I notice that updates are frozen for weeks! Is there any problem with the project? Thanks in advance for your attention.

Regards from Spain.

ajedrecista.

That project is still on, but we decided to re-engineer it to make the calculation more efficient. It was set up with about 9 million unique perft(6) positions and the idea was to do a perft(8) from all of these positions.

But it's more efficient to do it just the opposite and let one client work on a large set of positions. So the new scheme will start with all the unique perft(8) positions and the clients will do 1000 of them. The total amount of CPU power to do it this way will be substantially less. Unfortunately, that requires me to restructure the whole thing. I have all the unique 8 ply positions right now and compressed that is 5 gig. But I have to fix up the clients and the server to work with this new system.

Don

[Don]

Thank you very much for your interest! I stay tuned for the results. Honestly, I will be surprised if these numbers make sense.

It doesn't make sense for me to test this by increase or decreasing the strength of Houdini. In either case it only increases the number of decisive games for Houdini. It either starts winning more games if you increase it, or it starts losing more games but in both cases you decisive game rate is lowest only when your wins and losses are balanced out.

So instead I decided to see if I could make Stockfish look like the most decisive program by weakening it. This makes more sense since it already consuming the most testing resources to match Komodo and Houdini.

I only have a couple of hundred games so I need a lot more, but this appears to drop the Stockfish draw rate enough to be below Houdini's. And your formula is not "fooled" either, it still see's Houdini as the most draw fearing program. I'll give a full report when I have a few thousand games.

If this works, we may be able to run any programs without messing with time adjustments, just as you set out to do. It probably wouldn't hurt to at least make a rough estimated adjustment anyway because I have a feeling there are still second order errors in your way - but I admit I don't understand it completely. For example maybe it works better if you get all programs within 100 ELO of each other.

Don

[Ajedrecista]

Hello:

http://ls-ratinglist.beepworld.de/

Code: Select all

The LightSpeed ratinglist.
14th December, 2012.

 Rank Name                   Elo    +    - games score oppo. draws 
   1 Houdini 3 x64          3152    4    4 16000   69%  3017   41% (€)
   2 Houdini 3 tactical     3119    5    5 10000   63%  3027   44% (€)(set)
   3 Houdini 2.0c x64       3104    5    5 11000   62%  3014   42% (€)
   4 Houdini 1.5a x64       3083    5    5 10000   59%  3016   44% (best freeware)
   5 Strelka 5.5 x64        3070    4    4 20000   55%  3032   53% (sc)
   6 Critter 1.6a x64       3069    4    4 20000   55%  3032   53% 
   7 Komodo 5 x64           3061    4    4 20000   54%  3033   43% (€)(sc)
   8 Ivanhoe 50kQ x64s      3039    5    5 11000   50%  3040   57% (best open source)(cp)
   9 Robbolito 0.21Q x64s   3033    4    4 14000   49%  3041   58% 
  10 Bouquet 1.6 x64s       3026    5    5 10000   47%  3044   56% 
  11 Ivanhoe 46h x64        3020    4    4 18000   48%  3035   54% 
  12 Bouquet 1.5 x64s       3018    4    4 15000   47%  3038   57% 
  13 Robbolito 0.10 x64s    3018    4    4 15000   48%  3032   55% 
  14 Rybka 4.1 x64s         3011    4    4 20000   47%  3035   46% (€)
  15 Akkad 0.52b x64s       3004    4    4 13000   45%  3037   55% 
  16 Robbolito 0.085g3 x64  3000    4    4 20000   45%  3036   54% (sc)(Ippolit 2009)
  17 Stockfish 2.3.1 x64s   2998    4    4 18000   45%  3036   45% 
  18 Stockfish 2.2.2 x64s   2994    5    5 11000   45%  3032   44% 
  19 Saros 3.0 x64          2986    4    4 18000   43%  3037   48% 
  20 Gull 2 beta2 x64       2984    5    5 11000   42%  3037   53% 
  21 Bouquet 1.4 x64s       2928    5    5 13000   34%  3044   42%

Thank you very much to Stefan Pohl and all the programmers. I know that each engine do not have played the same number of games.

Code: Select all

 Rank Name                     µ      D   (0.5 + |µ - 0.5|)*D

   1 Houdini 3 x64            69%    41%        0.2829
   2 Houdini 3 tactical       63%    44%        0.2772
   3 Houdini 2.0c x64         62%    42%        0.2604
   4 Houdini 1.5a x64         59%    44%        0.2596
   5 Strelka 5.5 x64          55%    53%        0.2915
   6 Critter 1.6a x64         55%    53%        0.2915
   7 Komodo 5 x64             54%    43%        0.2322
   8 Ivanhoe 50kQ x64s        50%    57%        0.285
   9 Robbolito 0.21Q x64s     49%    58%        0.2958
  10 Bouquet 1.6 x64s         47%    56%        0.2968
  11 Ivanhoe 46h x64          48%    54%        0.2808
  12 Bouquet 1.5 x64s         47%    57%        0.3021
  13 Robbolito 0.10 x64s      48%    55%        0.286
  14 Rybka 4.1 x64s           47%    46%        0.2438
  15 Akkad 0.52b x64s         45%    55%        0.3025
  16 Robbolito 0.085g3 x64    45%    54%        0.297
  17 Stockfish 2.3.1 x64s     45%    45%        0.2475
  18 Stockfish 2.2.2 x64s     45%    44%        0.242
  19 Saros 3.0 x64            43%    48%        0.2736
  20 Gull 2 beta2 x64         42%    53%        0.3074
  21 Bouquet 1.4 x64s         34%    42%        0.2772

I hope no typos although I went too fast with my calculator. I do not want to interpret these results very deeply because I am not sure about the validity of the last column (a higher number means a higher trend to draw, and viceversa).

Regards from Spain.

Ajedrecista.

[Don]

I sorted the data for you. I'm guessing that the data makes no sense unless the games are played round robin style but for what it's worth here is what we get:

Code: Select all

       7 Komodo 5 x64             54%    43%        0.2322 
      18 Stockfish 2.2.2 x64s     45%    44%        0.242 
      14 Rybka 4.1 x64s           47%    46%        0.2438 
      17 Stockfish 2.3.1 x64s     45%    45%        0.2475 
       4 Houdini 1.5a x64         59%    44%        0.2596 
       3 Houdini 2.0c x64         62%    42%        0.2604 
      19 Saros 3.0 x64            43%    48%        0.2736 
      21 Bouquet 1.4 x64s         34%    42%        0.2772
       2 Houdini 3 tactical       63%    44%        0.2772 
      11 Ivanhoe 46h x64          48%    54%        0.2808 
       1 Houdini 3 x64            69%    41%        0.2829 
       8 Ivanhoe 50kQ x64s        50%    57%        0.285 
      13 Robbolito 0.10 x64s      48%    55%        0.286 
       5 Strelka 5.5 x64          55%    53%        0.2915 
       6 Critter 1.6a x64         55%    53%        0.2915 
       9 Robbolito 0.21Q x64s     49%    58%        0.2958 
      10 Bouquet 1.6 x64s         47%    56%        0.2968 
      16 Robbolito 0.085g3 x64    45%    54%        0.297 
      12 Bouquet 1.5 x64s         47%    57%        0.3021 
      15 Akkad 0.52b x64s         45%    55%        0.3025 
      20 Gull 2 beta2 x64         42%    53%        0.3074 

Hello:

http://ls-ratinglist.beepworld.de/

Code: Select all

The LightSpeed ratinglist.
14th December, 2012.

 Rank Name                   Elo    +    - games score oppo. draws 
   1 Houdini 3 x64          3152    4    4 16000   69%  3017   41% (€)
   2 Houdini 3 tactical     3119    5    5 10000   63%  3027   44% (€)(set)
   3 Houdini 2.0c x64       3104    5    5 11000   62%  3014   42% (€)
   4 Houdini 1.5a x64       3083    5    5 10000   59%  3016   44% (best freeware)
   5 Strelka 5.5 x64        3070    4    4 20000   55%  3032   53% (sc)
   6 Critter 1.6a x64       3069    4    4 20000   55%  3032   53% 
   7 Komodo 5 x64           3061    4    4 20000   54%  3033   43% (€)(sc)
   8 Ivanhoe 50kQ x64s      3039    5    5 11000   50%  3040   57% (best open source)(cp)
   9 Robbolito 0.21Q x64s   3033    4    4 14000   49%  3041   58% 
  10 Bouquet 1.6 x64s       3026    5    5 10000   47%  3044   56% 
  11 Ivanhoe 46h x64        3020    4    4 18000   48%  3035   54% 
  12 Bouquet 1.5 x64s       3018    4    4 15000   47%  3038   57% 
  13 Robbolito 0.10 x64s    3018    4    4 15000   48%  3032   55% 
  14 Rybka 4.1 x64s         3011    4    4 20000   47%  3035   46% (€)
  15 Akkad 0.52b x64s       3004    4    4 13000   45%  3037   55% 
  16 Robbolito 0.085g3 x64  3000    4    4 20000   45%  3036   54% (sc)(Ippolit 2009)
  17 Stockfish 2.3.1 x64s   2998    4    4 18000   45%  3036   45% 
  18 Stockfish 2.2.2 x64s   2994    5    5 11000   45%  3032   44% 
  19 Saros 3.0 x64          2986    4    4 18000   43%  3037   48% 
  20 Gull 2 beta2 x64       2984    5    5 11000   42%  3037   53% 
  21 Bouquet 1.4 x64s       2928    5    5 13000   34%  3044   42%

Thank you very much to Stefan Pohl and all the programmers. I know that each engine do not have played the same number of games.

Code: Select all

 Rank Name                     µ      D   (0.5 + |µ - 0.5|)*D

   1 Houdini 3 x64            69%    41%        0.2829
   2 Houdini 3 tactical       63%    44%        0.2772
   3 Houdini 2.0c x64         62%    42%        0.2604
   4 Houdini 1.5a x64         59%    44%        0.2596
   5 Strelka 5.5 x64          55%    53%        0.2915
   6 Critter 1.6a x64         55%    53%        0.2915
   7 Komodo 5 x64             54%    43%        0.2322
   8 Ivanhoe 50kQ x64s        50%    57%        0.285
   9 Robbolito 0.21Q x64s     49%    58%        0.2958
  10 Bouquet 1.6 x64s         47%    56%        0.2968
  11 Ivanhoe 46h x64          48%    54%        0.2808
  12 Bouquet 1.5 x64s         47%    57%        0.3021
  13 Robbolito 0.10 x64s      48%    55%        0.286
  14 Rybka 4.1 x64s           47%    46%        0.2438
  15 Akkad 0.52b x64s         45%    55%        0.3025
  16 Robbolito 0.085g3 x64    45%    54%        0.297
  17 Stockfish 2.3.1 x64s     45%    45%        0.2475
  18 Stockfish 2.2.2 x64s     45%    44%        0.242
  19 Saros 3.0 x64            43%    48%        0.2736
  20 Gull 2 beta2 x64         42%    53%        0.3074
  21 Bouquet 1.4 x64s         34%    42%        0.2772

I hope no typos although I went too fast with my calculator. I do not want to interpret these results very deeply because I am not sure about the validity of the last column (a higher number means a higher trend to draw, and viceversa).

Regards from Spain.

Ajedrecista.

[Don]

I decided to normalize the risk style numbers - so basically I use your values but use them as the demonintor where the numerator is the total. This makes it possible to compare directly any 2 runs, even when the program are at different time controls.

The first is the big run I did at fast time controls:

Code: Select all

 Percent   Percent   Percent   Percent      Risk 
Decisive      Wins    Losses     Draws     Style  Player
--------  --------  --------  --------  --------  -------------------
   66.92     33.54     33.38     33.08   3.22696  hou3
   63.59     32.21     31.37     36.41   2.91158  kdev-4518.00
   63.31     31.15     32.15     36.69   2.88471  sf23

This is the run where I crippled Stockfish significantly in order to lower it's draw rate to be below to lower it's draw rate. As you see the numbers are extremely similar.

Code: Select all

 Percent   Percent   Percent   Percent      Risk 
Decisive      Wins    Losses     Draws     Style  Player
--------  --------  --------  --------  --------  -------------------
   70.42     46.92     23.50     29.58   3.24712  hou3
   72.04     12.75     59.29     27.96   2.89315  sf23
   66.64     44.86     21.77     33.36   2.88691  kdev-4518.00

I'm satisfied with this experiment and believe that your formula probably works acceptably although possibly it could be improved. At the 4x longer time control the differences compress a bit:

Code: Select all

 Percent   Percent   Percent   Percent      Risk 
Decisive      Wins    Losses     Draws     Style  Player
--------  --------  --------  --------  --------  -------------------
   62.40     30.77     31.63     37.60   3.15478  hou3
   60.25     31.11     29.14     39.75   2.95121  sf23
   59.28     29.08     30.19     40.72   2.90549  kdev-4518.00

[Don]

I'm running another test where komodo is given a high contempt factor and Houdini's is set to zero.

I only have a few hundred games each, but this appears to upset the balance a bit. Houdini gets stronger because contempt 1 is ridiculous against evenly matched opponents and Komodo gets weaker for the same reason but they are all within about 15 ELO of each other. I used 23 contempt in Komodo and it appears to have a much smaller effect on the results than changing it does for Houdini, probably due to the king safety issue Richard mentioned.

It appears from the data so far that Houdini is not particularly dynamic - the draw aversion was primarily a result of the contempt factor. It did not change Komodo very much.

Code: Select all

 Percent   Percent   Percent   Percent      Risk 
Decisive      Wins    Losses     Draws     Style  Player
--------  --------  --------  --------  --------  -------------------
   60.56     29.47     31.09     39.44   3.07371  sf23
   61.49     28.70     32.80     38.51   3.07323  kdev-4518.00
   59.30     32.50     26.79     40.70   2.86312  hou3

Please take all of this with a grain of salt. I'm not sure of the significance of any of this. I do have a hypothesis though. The hypothesis is that no strong program is going to be particularly draw fearing. To play really exiting "go for broke" chess you have to have a somewhat unsound evaluation function and strong programs do not have that. Maybe you can do some things to make them more "fun" but if you want your program to play soundly you cannot just sacrifice material left and right.

[Laskos]

I'm running another test where komodo is given a high contempt factor and Houdini's is set to zero.

I only have a few hundred games each, but this appears to upset the balance a bit. Houdini gets stronger because contempt 1 is ridiculous against evenly matched opponents and Komodo gets weaker for the same reason but they are all within about 15 ELO of each other. I used 23 contempt in Komodo and it appears to have a much smaller effect on the results than changing it does for Houdini, probably due to the king safety issue Richard mentioned.

It appears from the data so far that Houdini is not particularly dynamic - the draw aversion was primarily a result of the contempt factor. It did not change Komodo very much.

Code: Select all

 Percent   Percent   Percent   Percent      Risk 
Decisive      Wins    Losses     Draws     Style  Player
--------  --------  --------  --------  --------  -------------------
   60.56     29.47     31.09     39.44   3.07371  sf23
   61.49     28.70     32.80     38.51   3.07323  kdev-4518.00
   59.30     32.50     26.79     40.70   2.86312  hou3

Please take all of this with a grain of salt. I'm not sure of the significance of any of this. I do have a hypothesis though. The hypothesis is that no strong program is going to be particularly draw fearing. To play really exiting "go for broke" chess you have to have a somewhat unsound evaluation function and strong programs do not have that. Maybe you can do some things to make them more "fun" but if you want your program to play soundly you cannot just sacrifice material left and right.

Have you finished the test (a few thousand games)? For now it seems that the contempt factor was the primary cause of Houdini "aversion" to draws. My guess is that all three (Houdini, Komodo, Stockfish) are in Fruit or Fruit/Rybka branch of engines, also artistically known as "dull" engines. When Fritz 11 (Fruit/Strelka/Rybka) appeared, many observed its dullness compared to the "brilliant" Fritz 10. Could you make pretty equal strength Junior 13 or Hiarcs 14 by adjusting their time controls, and compare one of them to those 3 super-engines from above? They are not in Fruit/Rybka branch, maybe we will see differences.

Kai

[Don]

I'm running another test where komodo is given a high contempt factor and Houdini's is set to zero.

I only have a few hundred games each, but this appears to upset the balance a bit. Houdini gets stronger because contempt 1 is ridiculous against evenly matched opponents and Komodo gets weaker for the same reason but they are all within about 15 ELO of each other. I used 23 contempt in Komodo and it appears to have a much smaller effect on the results than changing it does for Houdini, probably due to the king safety issue Richard mentioned.

It appears from the data so far that Houdini is not particularly dynamic - the draw aversion was primarily a result of the contempt factor. It did not change Komodo very much.

Code: Select all

 Percent   Percent   Percent   Percent      Risk 
Decisive      Wins    Losses     Draws     Style  Player
--------  --------  --------  --------  --------  -------------------
   60.56     29.47     31.09     39.44   3.07371  sf23
   61.49     28.70     32.80     38.51   3.07323  kdev-4518.00
   59.30     32.50     26.79     40.70   2.86312  hou3

Please take all of this with a grain of salt. I'm not sure of the significance of any of this. I do have a hypothesis though. The hypothesis is that no strong program is going to be particularly draw fearing. To play really exiting "go for broke" chess you have to have a somewhat unsound evaluation function and strong programs do not have that. Maybe you can do some things to make them more "fun" but if you want your program to play soundly you cannot just sacrifice material left and right.

Have you finished the test (a few thousand games)? For now it seems that the contempt factor was the primary cause of Houdini "aversion" to draws. My guess is that all three (Houdini, Komodo, Stockfish) are in Fruit or Fruit/Rybka branch of engines, also artistically known as "dull" engines. When Fritz 11 (Fruit/Strelka/Rybka) appeared, many observed its dullness compared to the "brilliant" Fritz 10. Could you make pretty equal strength Junior 13 or Hiarcs 14 by adjusting their time controls, and compare one of them to those 3 super-engines from above? They are not in Fruit/Rybka branch, maybe we will see differences.

Kai

I don't have Junior but I do have Hiarcs, but I was unable to get it working due the key issue. I did purchase it so it's just a matter of trying to make it work. I'm using Linux but I do have it working on a different machine but not one that I would use for this test. I'll look into getting it working.

Here is the final result of my last test - I set Komodo contempt to 23 and Houdini to 0 (which still is something like 13 but I don't know exactly how it works.)

Code: Select all

drd@odie ~/autotest $ rate ds06

Rank    ELO     +/-    Games    Score  Player
---- ------- ------ -------- --------  ----------------------------
   1  3006.3   10.7     2625   51.371  hou3         
   2  2999.8   10.7     2625   49.962  kdev-4518.00 
   3  2993.8   10.7     2624   48.666  sf23         

w/l/d: 1345 1078 1514    38.46 percent draws


      TIME       RATIO    log(r)     NODES    log(r)  ave DEPTH    GAMES   PLAYER
 ---------  ----------  --------  --------  --------  ---------  -------   ------------
    0.1988       1.000     0.000     0.247     0.000    12.6294     2625   hou3
    0.3116       1.568     0.450     0.243    -0.015    12.5835     2625   kdev-4518.00
    0.5774       2.905     1.066     0.450     0.602    15.5249     2624   sf23

drd@odie ~/autotest $ 
drd@odie ~/autotest $ cat ds06.pgn | crosstable.tcl

White         Black            winP       games     drawP
------------  ------------  -------  ----------  --------
hou3          kdev-4518.00    51.49        1313     39.15
hou3          sf23            51.26        1312     40.62
kdev-4518.00  hou3            48.51        1313     39.15
kdev-4518.00  sf23            51.41        1312     35.59
sf23          hou3            48.74        1312     40.62
sf23          kdev-4518.00    48.59        1312     35.59


 Percent   Percent   Percent   Percent      Risk 
Decisive      Wins    Losses     Draws     Style  Player
--------  --------  --------  --------  --------  -------------------
   62.63     31.28     31.35     37.37   3.14188  kdev-4518.00
   61.89     29.61     32.28     38.11   3.00324  sf23
   60.11     31.43     28.69     39.89   2.86742  hou3

[Ajedrecista]

Hello Don:

I'm running another test where komodo is given a high contempt factor and Houdini's is set to zero.

I only have a few hundred games each, but this appears to upset the balance a bit. Houdini gets stronger because contempt 1 is ridiculous against evenly matched opponents and Komodo gets weaker for the same reason but they are all within about 15 ELO of each other. I used 23 contempt in Komodo and it appears to have a much smaller effect on the results than changing it does for Houdini, probably due to the king safety issue Richard mentioned.

It appears from the data so far that Houdini is not particularly dynamic - the draw aversion was primarily a result of the contempt factor. It did not change Komodo very much.

Code: Select all

 Percent   Percent   Percent   Percent      Risk 
Decisive      Wins    Losses     Draws     Style  Player
--------  --------  --------  --------  --------  -------------------
   60.56     29.47     31.09     39.44   3.07371  sf23
   61.49     28.70     32.80     38.51   3.07323  kdev-4518.00
   59.30     32.50     26.79     40.70   2.86312  hou3

Please take all of this with a grain of salt. I'm not sure of the significance of any of this. I do have a hypothesis though. The hypothesis is that no strong program is going to be particularly draw fearing. To play really exiting "go for broke" chess you have to have a somewhat unsound evaluation function and strong programs do not have that. Maybe you can do some things to make them more "fun" but if you want your program to play soundly you cannot just sacrifice material left and right.

I do not know how did you compute 'risk style' column this time. A higher number in this column means more likelihood of drawing, or less? I think that a higher number in your 'risk style' column means less likelihood to draw. Anyway, I get the following column c (rounding up to 0.0001):

Code: Select all

 Percent   Percent   Percent   Percent      
Decisive      Wins    Losses     Draws     c      Player
--------  --------  --------  --------  --------  -------------------
   60.56     29.47     31.09     39.44   0.2004   sf23
   61.49     28.70     32.80     38.51   0.2005   kdev-4518.00
   59.30     32.50     26.79     40.70   0.2151   hou3

SF: (0.5 + |µ - 0.5|)*D = (0.5 + |0.4919 - 0.5|)*0.3944 ~ 0.2004
Komodo: (0.5 + |µ - 0.5|)*D ~ (0.5 + |0.47955 - 0.5|)*0.3851 ~ 0.2005
Houdini: (0.5 + |µ - 0.5|)*D ~ (0.5 + |0.5285 - 0.5|)*0.407 ~ 0.2151

A higher number c means more likelihood to draw, and viceversa. In this case with three engines, Houdini is the least 'draw fearing' engine.

Regards from Spain.

Ajedrecista.