N4 vs. Rybka 3 different cont 16_1

[M ANSARI]

I have been doing a lot of tourneys to see how N4 does against R3. I will eventually tabulate all of them and make a report. In the meantime here is some results from 200 games using Quadcore ... as you can imagine it takes quite a long time to generate so many games at slower TC's. Book used is Perfect 12 truncated to 10 moves a side to try and get some interesting positions. It seems that moving Rybka 3 to zero contempt at this TC is beneficial. I will look into the games and post some results. Eventually I plan to post all the data I get. Interesting will be how Rybka and Naum do in a 30_1 tourney on Octa at 4Ghz I am now running which could be considered as 60_1 on Quadcore. So far after 100 games both Rybka 3's seem to be doing very well in that tourney scoring more than 70% ... although the book there is only 7 move HS book which could also be a factor.

N4_16_1_new_gauntlet-1 2008

Naum 4 - Rybka 3 40.0 - 60.0 +11/-31/=58 40.00%
Naum 4 - Rybka 3_no_cont 35.0 - 65.0 +10/-40/=50 35.00%

[Marc Lacrosse]

Naum 4 - Rybka 3 40.0 - 60.0 +11/-31/=58 40.00%
Naum 4 - Rybka 3_no_cont 35.0 - 65.0 +10/-40/=50 35.00%

Very interesting.
By the way, do you know that Remi Coulom not only authored Bayeselo but also produced a montecarlo statistical evaluator that addresses this typical situation : what is the likelihood of program A being better than program B, based on the results of two matches played against a common opponent.

Using this tool, we get :

Code: Select all

F:\chess\0 chess work tools>MonteCarlo-TwoEvaluatedAgainstCommonOpponent.exe
  This program evaluates the likelihood that program A is better than
program B, based on the result of two matches played against the same
opponent (or set of opponents). The number of games played in each of
these matches does not have to be the same. If playing against a set
of opponents, the proportion of each opponent should be the same in each
match.
  The likelihood is estimated by Bayesian inference, assuming an uniform
prior distribution of the probabilites of losing and winning.
  The resulting integral is estimated with a Monte-Carlo method. It may
take a long time to converge when the number of games is large (>100).
The computation can be interrupted at any time with Ctrl-C.

A wins   = 40
A losses = 10
A draws  = 50

B wins   = 31
B losses = 11
B draws  = 58

^CA>B) = 0.864434 (501000000 iterations)

So a reasonable probability (0.86) that contempt = 0 is better against Naum in these testing conditions.
By the way there is still a 13.6% probability that contempt = 15 is superior to contempt = 0 with the data that you got. :

Code: Select all

A wins   = 31
A losses = 11
A draws  = 58

B wins   = 40
B losses = 10
B draws  = 50

^CA>B) = 0.135654

Marc

[M ANSARI]

More testing is needed ... hopefully more people will pick-up on this and do some more tests with different contempt. At 1_1 time controls I did a 1000 game tourney where it seems that the best contempt factor is "30" ... which is double the contempt of the default. I think it has to do with Rybka 3 being an incredibly faster searcher and outsearching N4 at fast time controls. Results for the 1_1 where almost 90% for the highest contempt and around 82% to 86% for the "15" and "8" and "0" contempt following a very predictable linear scale. As you increase TC N4 seems to gain ... but at 30_1 TC on Octa things seems to change again.