Komodo_Ivanhoe- 2 Nice Final Matches!

[geots]

Hi George,

Thanks for running those matches. I'm sorry if I came on too strong in my previous comments. I also ran a match to 1000 games under similar conditions. I am surprised at the result, even though Komodo won it was by a smaller margin than I expected. Here are my results:

Code: Select all

   Hardware:   i7-2630QM CPU @ 2.00GHz  (notebook)
         OS:   Linux
         TC:   40 / 3 minutes repeating
       Hash:   64 meg
     Ponder:   OFF
       Book:   private 35,553 line 10 ply 

 Rank Name       Elo      +      -    games   score   oppo.   draws 
   1 Komodo4  3020.4   20.1   20.1    1025   53.3%  3000.0   46.3% 
   2 IvanHoe  3000.0   20.1   20.1    1025   46.7%  3020.4   46.3% 


      TIME       RATIO    log(r)     NODES    log(r)  ave DEPTH    GAMES   PLAYER
 ---------  ----------  --------  --------  --------  ---------  -------   -------
    4.7201       0.992    -0.008     3.074    -0.679    17.1057     1025   Komodo4
    4.7564       1.000     0.000     6.059     0.000    17.9695     1025   IvanHoe

I am going to duplicate this match with the latest development version of Komodo.

Don

Here are 2 nice matches to wind up the "Ivanhoe 12" in 64bit test. Their opponent needed to be strong, but not Houdini in that direction, and not Hiarcs or Shredder in the other. Very competitive. I picked Komodo because of its incredible stability for me- as I have mentioned more than once and it had played most all these engines for me in 32bit.

At any rate, here are the last 2 matches, and I thank the good Dr. Deeb for the offer of an Ivanhoe version he said to just trust him on. I did- and the good Dr. did not lead me astray. It is in the 2nd and last match posted here:


Intel i5 w/4TCs
Fritz 11 gui
1CPU/64bit
128MB hash
Bases=NONE
Ponder_Learning=OFF
Perfect 12.32 book w/12-move limit
40/3 Repeating
Match=50 games



Komodo64 SSE Ver. 4:

Code: Select all

1   Komodo64 SSE Version 4   +49    +14/-7/=29   57.00%   28.5/50
2   Ivanhoe B50kBx64p        -49    +7/-14/=29   43.00%   21.5/50

Komodo64 SSE Ver. 4: v Dr. Deeb's Ivanhoe ver.

Code: Select all

1   Ivanhoe-B PP-0.1 x64     +14    +12/-10/=28   52.00%   26.0/50
2   Komodo64 SSE Version 4   -14    +10/-12/=28   48.00%   24.0/50

Final Standings:



Komodo64 SSE Version 4:

Code: Select all

1 Ivanhoe B46e x64      +92    +21/-8/=21  OR- +35   +12/-7/=31 (run 4/7)
2 Ivanhoe 46h x64       +14    +16/-14/=20
3 Ivanhoe-B PP-0.1 x64  +14    +12/-10/=28
4 Ivanhoe B46a x64      +7     +12/-11/=27
5 Ivanhoe B47cCx64      +7     +14/-13/=23
6 Ivanhoe B46fC x64      0     +11/-11/=28
7 IvanHoe T52E SSE4.2    0     +9/-9/=32  
8 Ivanhoe B47f0.2 x64   -28    +12/-16/=22
9 Ivanhoe B47e x64      -28    +11/-15/=24
10Ivanhoe B47d x64      -49    +9/-16/=25
11Ivanhoe B50kBx64p     -49    +7/-14/=29
12IvanHoe B50hCx64p     -63    +8/-17/=25   

For Ivanhoe B46e, I added to the right a match with Komodo run April 7. I am quite sure it is much more palatable and much less questionable.

This concludes my effort. I hope it was interesting and enjoyable to some.



Best,

george

Don, don't worry. We were both trying to protect something. It would have never become a problem if I had checked with the author before I "USED" his commercial engine as the base for a litmus test. That was the biggest mistake that started everything.

I was also amazed at the difference in Komodo's competitive game in relation to all others on 64bit. You had told me- but I was still caught off guard by the difference.

One thing you are doing concerns me. You keep showing "Ivanhoe"- but which Ivanhoe? It can make enough of a difference to invalidate all you are doing. Please let me send you the correct version. Then you will be good to go. You need to be using B46e x64. And NOT 9.46e. You might as well be using a weaker engine. If you don't have it- let me send it. That is the strongest ag. Komodo on 64 bit. (And you need these PGNs from both B46e matches)


Best,

george

I am using 999946e since that is the one you were reporting. And it's the linux 64 bit compile. I can also run windows compiles if you have a version to send me I will check it out.

Don

STOP!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! I lost your address and I never got dropbox. Just send something to my address and I can reply with B46e. Trust me- you don't want 9. anything. That particular 9 version is at least 28 elo weaker.

[Don]

And I cannot argue that your choice for 40/3 repeating is probably better, but if you want to mimic mine- switch to 128MB hash. I don't have a good reason for it- except it is just what I use.

That and being sure you have the correct version are critical if you want matching conditions.


Best,

george

I just checked my configuration file and I am using 128 meg Hash, what I typed before was a misprint.

----[ config file ]-----

cpus = 4

player = Komodo4
invoke = /home/drd/u/kom/komodo/VERSIONS/4354.00/kse
tc = 40 180
Hash = 128

player = IvanHoe
invoke = ./IvanHoe999946e
tc = 40 180
Threads = 1
Hash = 128

[geots]

And I cannot argue that your choice for 40/3 repeating is probably better, but if you want to mimic mine- switch to 128MB hash. I don't have a good reason for it- except it is just what I use.

That and being sure you have the correct version are critical if you want matching conditions.


Best,

george

I just checked my configuration file and I am using 128 meg Hash, what I typed before was a misprint.

----[ config file ]-----

cpus = 4

player = Komodo4
invoke = /home/drd/u/kom/komodo/VERSIONS/4354.00/kse
tc = 40 180
Hash = 128

player = IvanHoe
invoke = ./IvanHoe999946e
tc = 40 180
Threads = 1
Hash = 128

Please stop with 9.46e. Weaker by far than any Izak compile that begins with B. I don't know why everyone thinks that is the site for official releases. There are no official releases- only modified source codes. All yoou got is Franklin's compile- nothing official. I am giving you a compile that starts with a B_ which means instead of just param. changes- he made changes also in the thinking process. I have compared each 46 release ag. 6 diff. common opponents 4 times over. You are making a big mistake. When I say 28 elo- I may be too kind. The closest I ever saw was twice- 14elo. Look, Izak has the talent. How I don't know. The kid is good.

[geots]

If you could and had gone public with Komodo, and Izak was going to join your team the first of the week- when the market opened Monday I woubuy every Komodo stock I could get my hands on and steal what I couldn't buy.


gts

[geots]

If you want to be technical, and you usually like everything right- if you use a chessbase gui- always check "below normal" priority for all Ivanhoe engines. There is normal, below normal, and then low. I might consider running "low" if it was an option.

gts

[Ajedrecista]

Hello Don:

Code: Select all

   Hardware:   i7-2630QM CPU @ 2.00GHz  (notebook) 
         OS:   Linux 
         TC:   40 / 3 minutes repeating 
       Hash:   64 meg 
     Ponder:   OFF 
       Book:   private 35,553 line 10 ply 

 Rank Name       Elo      +      -    games   score   oppo.   draws 
   1 Komodo4  3020.4   20.1   20.1    1025   53.3%  3000.0   46.3% 
   2 IvanHoe  3000.0   20.1   20.1    1025   46.7%  3020.4   46.3% 


      TIME       RATIO    log(r)     NODES    log(r)  ave DEPTH    GAMES   PLAYER 
 ---------  ----------  --------  --------  --------  ---------  -------   ------- 
    4.7201       0.992    -0.008     3.074    -0.679    17.1057     1025   Komodo4 
    4.7564       1.000     0.000     6.059     0.000    17.9695     1025   IvanHoe

Thanks for the match! It is interesting. I thought that in a direct match of N games between two engines, the rating difference should be 400·log(score_Komodo/score_IvanHoe). If this is not the case, please try to explain in simple words where I am wrong. According to the data posted by you, I get: +309 -241 =475 (please correct me if I am wrong). With this data I get ~ +23.1 Elo for Komodo, and not +20.4 Elo. Are you using BayesElo or other programme? I also get different error bars: for what confidence level are yours? I post here what I get with my own programme:

Code: Select all

 Elo_uncertainties_calculator, © 2012.

 Calculation of Elo uncertainties in a match between two engines:
 ----------------------------------------------------------------

 (The input and output data is referred to the first engine).

 Please write down non-negative integers.

 Write down the number of wins:

309

 Write down the number of loses:

241

 Write down the number of draws:

475

 ***************************************
 1-sigma confidence ~ 68.27% confidence.
 2-sigma confidence ~ 95.45% confidence.
 3-sigma confidence ~ 99.73% confidence.
 ***************************************

 -----------------------------------------------------------------------

 Confidence interval for            1-sigma:

 Elo rating difference:      23.083289669143689    Elo

 Lower rating difference:      15.142280756656772    Elo
 Upper rating difference:      31.048458042726799    Elo

 Lower bound uncertainty:     -7.9410089124869165    Elo
 Upper bound uncertainty:      7.9651683735831103    Elo
 Average error: +-     7.9530886430350134    Elo

 K = (average error)*[sqrt(n)] =      254.62307326334710

 Elo interval: ]     15.142280756656772    ,     31.048458042726799    [
 -----------------------------------------------------------------------

 Confidence interval for            2-sigma:

 Elo rating difference:      23.083289669143689    Elo

 Lower rating difference:      7.2170537956172484    Elo
 Upper rating difference:      39.046316237476915    Elo

 Lower bound uncertainty:     -15.866235873526440    Elo
 Upper bound uncertainty:      15.963026568333227    Elo
 Average error: +-     15.914631220929833    Elo

 K = (average error)*[sqrt(n)] =      509.51680450270673

 Elo interval: ]     7.2170537956172484    ,     39.046316237476915    [
 -----------------------------------------------------------------------

 Confidence interval for            3-sigma:

 Elo rating difference:      23.083289669143689    Elo

 Lower rating difference:    -0.70066962596180779    Elo
 Upper rating difference:      47.085603654880218    Elo

 Lower bound uncertainty:     -23.783959295105496    Elo
 Upper bound uncertainty:      24.002313985736530    Elo
 Average error: +-     23.893136640421013    Elo

 K = (average error)*[sqrt(n)] =      764.95361165287328

 Elo interval: ]   -0.70066962596180779    ,     47.085603654880218    [
 -----------------------------------------------------------------------

 Number of games of the match:         1025
 Score:      53.317073170731707    %
 Elo rating difference:      23.083289669143689    Elo
 Draw ratio:      46.341463414634146    %

 *****************************************************************
 1-sigma:      1.1393024995912142    % of the points of the match.
 2-sigma:      2.2786049991824283    % of the points of the match.
 3-sigma:      3.4179074987736425    % of the points of the match.
 *****************************************************************

 End of the calculations.

 Thanks for using Elo_uncertainties_calculator. Press Enter to exit.

For 2-sigma confidence (~ 95.45% confidence), I get ~ ± 15.9 Elo; as you see, I also calculate a thing that I call K, where K = |(average uncertainty)·[sqrt(number of games)]|. In this case, I get K ~ 509.5, which is very reasonable for my patzer views and little experience. This K decreases when the draw ratio grows, and typical values for an even match (like the one you post) are K ~ 580, K ~ 600, ... for draw ratios of around 30%. If I calculate K with your data, K ~ 20.1 · sqrt(1025) ~ 643.5, which is very high for 2-sigma confidence (it corresponds with draw ratios of around 15% in an even match, and not over 45%). Everything changes if you are calculating those intervals with other confidence level, around 99% in a fast calculation by me with pencil and paper (always using my method, which of course will not be the best).

I have more questions regarding the data you provide: what do you refer with TIME? The average time (in minutes) spent by each engine in each game? I understand RATIO, where you normalize TIME respect to IvanHoe 46e; I also understand ln(RATIO), although I do not understand the need of taking logarithms (it surely be better, but I am puzzled). Regarding NODES, I have the same doubt as in TIME: can it be the average number of millions of nodes searched in each move of a game? Again, I understand ln(NODES), having the same doubt that in ln(RATIO). I suppose that average depth refers to the average depth that each engine reached in each move.

Finally, I have the more interesting question for the majority of readers of the forum: how goes the progress in Komodo MP? I hope that well, and I wish you good luck. I hope that the new Komodo will be released before the end of June of this year. Thanks for your attention and your patience for many questions.

Regards from Spain.

Ajedrecista.

[Dr.Wael Deeb]

Hi George,

Thanks for running those matches. I'm sorry if I came on too strong in my previous comments. I also ran a match to 1000 games under similar conditions. I am surprised at the result, even though Komodo won it was by a smaller margin than I expected. Here are my results:

Code: Select all

   Hardware:   i7-2630QM CPU @ 2.00GHz  (notebook)
         OS:   Linux
         TC:   40 / 3 minutes repeating
       Hash:   64 meg
     Ponder:   OFF
       Book:   private 35,553 line 10 ply 

 Rank Name       Elo      +      -    games   score   oppo.   draws 
   1 Komodo4  3020.4   20.1   20.1    1025   53.3%  3000.0   46.3% 
   2 IvanHoe  3000.0   20.1   20.1    1025   46.7%  3020.4   46.3% 


      TIME       RATIO    log(r)     NODES    log(r)  ave DEPTH    GAMES   PLAYER
 ---------  ----------  --------  --------  --------  ---------  -------   -------
    4.7201       0.992    -0.008     3.074    -0.679    17.1057     1025   Komodo4
    4.7564       1.000     0.000     6.059     0.000    17.9695     1025   IvanHoe

I am going to duplicate this match with the latest development version of Komodo.

Don

Here are 2 nice matches to wind up the "Ivanhoe 12" in 64bit test. Their opponent needed to be strong, but not Houdini in that direction, and not Hiarcs or Shredder in the other. Very competitive. I picked Komodo because of its incredible stability for me- as I have mentioned more than once and it had played most all these engines for me in 32bit.

At any rate, here are the last 2 matches, and I thank the good Dr. Deeb for the offer of an Ivanhoe version he said to just trust him on. I did- and the good Dr. did not lead me astray. It is in the 2nd and last match posted here:


Intel i5 w/4TCs
Fritz 11 gui
1CPU/64bit
128MB hash
Bases=NONE
Ponder_Learning=OFF
Perfect 12.32 book w/12-move limit
40/3 Repeating
Match=50 games



Komodo64 SSE Ver. 4:

Code: Select all

1   Komodo64 SSE Version 4   +49    +14/-7/=29   57.00%   28.5/50
2   Ivanhoe B50kBx64p        -49    +7/-14/=29   43.00%   21.5/50

Komodo64 SSE Ver. 4: v Dr. Deeb's Ivanhoe ver.

Code: Select all

1   Ivanhoe-B PP-0.1 x64     +14    +12/-10/=28   52.00%   26.0/50
2   Komodo64 SSE Version 4   -14    +10/-12/=28   48.00%   24.0/50

Final Standings:



Komodo64 SSE Version 4:

Code: Select all

1 Ivanhoe B46e x64      +92    +21/-8/=21  OR- +35   +12/-7/=31 (run 4/7)
2 Ivanhoe 46h x64       +14    +16/-14/=20
3 Ivanhoe-B PP-0.1 x64  +14    +12/-10/=28
4 Ivanhoe B46a x64      +7     +12/-11/=27
5 Ivanhoe B47cCx64      +7     +14/-13/=23
6 Ivanhoe B46fC x64      0     +11/-11/=28
7 IvanHoe T52E SSE4.2    0     +9/-9/=32  
8 Ivanhoe B47f0.2 x64   -28    +12/-16/=22
9 Ivanhoe B47e x64      -28    +11/-15/=24
10Ivanhoe B47d x64      -49    +9/-16/=25
11Ivanhoe B50kBx64p     -49    +7/-14/=29
12IvanHoe B50hCx64p     -63    +8/-17/=25   

For Ivanhoe B46e, I added to the right a match with Komodo run April 7. I am quite sure it is much more palatable and much less questionable.

This concludes my effort. I hope it was interesting and enjoyable to some.



Best,

george

Don, don't worry. We were both trying to protect something. It would have never become a problem if I had checked with the author before I "USED" his commercial engine as the base for a litmus test. That was the biggest mistake that started everything.

I was also amazed at the difference in Komodo's competitive game in relation to all others on 64bit. You had told me- but I was still caught off guard by the difference.

One thing you are doing concerns me. You keep showing "Ivanhoe"- but which Ivanhoe? It can make enough of a difference to invalidate all you are doing. Please let me send you the correct version. Then you will be good to go. You need to be using B46e x64. And NOT 9.46e. You might as well be using a weaker engine. If you don't have it- let me send it. That is the strongest ag. Komodo on 64 bit. (And you need these PGNs from both B46e matches)


Best,

george

George,
If you are a customer of Komodo and you paid for it,you can use it in matches and tournaments HOWEVER you want and WHENEVER you want as along as this happens on your personal computer at home....

You can also publish the results WHEREVER & WHENEVER you want....
So you don't have to appologize....it must be the other way around....
Dr.D

[geots]

Hello Don:

Code: Select all

   Hardware:   i7-2630QM CPU @ 2.00GHz  (notebook) 
         OS:   Linux 
         TC:   40 / 3 minutes repeating 
       Hash:   64 meg 
     Ponder:   OFF 
       Book:   private 35,553 line 10 ply 

 Rank Name       Elo      +      -    games   score   oppo.   draws 
   1 Komodo4  3020.4   20.1   20.1    1025   53.3%  3000.0   46.3% 
   2 IvanHoe  3000.0   20.1   20.1    1025   46.7%  3020.4   46.3% 


      TIME       RATIO    log(r)     NODES    log(r)  ave DEPTH    GAMES   PLAYER 
 ---------  ----------  --------  --------  --------  ---------  -------   ------- 
    4.7201       0.992    -0.008     3.074    -0.679    17.1057     1025   Komodo4 
    4.7564       1.000     0.000     6.059     0.000    17.9695     1025   IvanHoe

Thanks for the match! It is interesting. I thought that in a direct match of N games between two engines, the rating difference should be 400·log(score_Komodo/score_IvanHoe). If this is not the case, please try to explain in simple words where I am wrong. According to the data posted by you, I get: +309 -241 =475 (please correct me if I am wrong). With this data I get ~ +23.1 Elo for Komodo, and not +20.4 Elo. Are you using BayesElo or other programme? I also get different error bars: for what confidence level are yours? I post here what I get with my own programme:

Code: Select all

 Elo_uncertainties_calculator, © 2012.

 Calculation of Elo uncertainties in a match between two engines:
 ----------------------------------------------------------------

 (The input and output data is referred to the first engine).

 Please write down non-negative integers.

 Write down the number of wins:

309

 Write down the number of loses:

241

 Write down the number of draws:

475

 ***************************************
 1-sigma confidence ~ 68.27% confidence.
 2-sigma confidence ~ 95.45% confidence.
 3-sigma confidence ~ 99.73% confidence.
 ***************************************

 -----------------------------------------------------------------------

 Confidence interval for            1-sigma:

 Elo rating difference:      23.083289669143689    Elo

 Lower rating difference:      15.142280756656772    Elo
 Upper rating difference:      31.048458042726799    Elo

 Lower bound uncertainty:     -7.9410089124869165    Elo
 Upper bound uncertainty:      7.9651683735831103    Elo
 Average error: +-     7.9530886430350134    Elo

 K = (average error)*[sqrt(n)] =      254.62307326334710

 Elo interval: ]     15.142280756656772    ,     31.048458042726799    [
 -----------------------------------------------------------------------

 Confidence interval for            2-sigma:

 Elo rating difference:      23.083289669143689    Elo

 Lower rating difference:      7.2170537956172484    Elo
 Upper rating difference:      39.046316237476915    Elo

 Lower bound uncertainty:     -15.866235873526440    Elo
 Upper bound uncertainty:      15.963026568333227    Elo
 Average error: +-     15.914631220929833    Elo

 K = (average error)*[sqrt(n)] =      509.51680450270673

 Elo interval: ]     7.2170537956172484    ,     39.046316237476915    [
 -----------------------------------------------------------------------

 Confidence interval for            3-sigma:

 Elo rating difference:      23.083289669143689    Elo

 Lower rating difference:    -0.70066962596180779    Elo
 Upper rating difference:      47.085603654880218    Elo

 Lower bound uncertainty:     -23.783959295105496    Elo
 Upper bound uncertainty:      24.002313985736530    Elo
 Average error: +-     23.893136640421013    Elo

 K = (average error)*[sqrt(n)] =      764.95361165287328

 Elo interval: ]   -0.70066962596180779    ,     47.085603654880218    [
 -----------------------------------------------------------------------

 Number of games of the match:         1025
 Score:      53.317073170731707    %
 Elo rating difference:      23.083289669143689    Elo
 Draw ratio:      46.341463414634146    %

 *****************************************************************
 1-sigma:      1.1393024995912142    % of the points of the match.
 2-sigma:      2.2786049991824283    % of the points of the match.
 3-sigma:      3.4179074987736425    % of the points of the match.
 *****************************************************************

 End of the calculations.

 Thanks for using Elo_uncertainties_calculator. Press Enter to exit.

For 2-sigma confidence (~ 95.45% confidence), I get ~ ± 15.9 Elo; as you see, I also calculate a thing that I call K, where K = |(average uncertainty)·[sqrt(number of games)]|. In this case, I get K ~ 509.5, which is very reasonable for my patzer views and little experience. This K decreases when the draw ratio grows, and typical values for an even match (like the one you post) are K ~ 580, K ~ 600, ... for draw ratios of around 30%. If I calculate K with your data, K ~ 20.1 · sqrt(1025) ~ 643.5, which is very high for 2-sigma confidence (it corresponds with draw ratios of around 15% in an even match, and not over 45%). Everything changes if you are calculating those intervals with other confidence level, around 99% in a fast calculation by me with pencil and paper (always using my method, which of course will not be the best).

I have more questions regarding the data you provide: what do you refer with TIME? The average time (in minutes) spent by each engine in each game? I understand RATIO, where you normalize TIME respect to IvanHoe 46e; I also understand ln(RATIO), although I do not understand the need of taking logarithms (it surely be better, but I am puzzled). Regarding NODES, I have the same doubt as in TIME: can it be the average number of millions of nodes searched in each move of a game? Again, I understand ln(NODES), having the same doubt that in ln(RATIO). I suppose that average depth refers to the average depth that each engine reached in each move.

Finally, I have the more interesting question for the majority of readers of the forum: how goes the progress in Komodo MP? I hope that well, and I wish you good luck. I hope that the new Komodo will be released before the end of June of this year. Thanks for your attention and your patience for many questions.

Regards from Spain.

Ajedrecista.

The biggest problem is when you mention "46e". He did not know, and used 9.46e instead of "B46e"- which would likely boost the Ivanhoe engine, being ultra-conservative- 21 elo. 28 to 40 would be no surprise at all. I do admit it could be as low as 14 elo- but that only happened 2 out of 10 times.


gts

[beram]

...

Code: Select all

   Hardware:   i7-2630QM CPU @ 2.00GHz  (notebook)
         OS:   Linux
         TC:   40 / 3 minutes repeating
       Hash:   64 meg
     Ponder:   OFF
       Book:   private 35,553 line 10 ply 

 Rank Name       Elo      +      -    games   score   oppo.   draws 
   1 Komodo4  3020.4   20.1   20.1    1025   53.3%  3000.0   46.3% 
   2 IvanHoe  3000.0   20.1   20.1    1025   46.7%  3020.4   46.3% 


      TIME       RATIO    log(r)     NODES    log(r)  ave DEPTH    GAMES   PLAYER
 ---------  ----------  --------  --------  --------  ---------  -------   -------
    4.7201       0.992    -0.008     3.074    -0.679    17.1057     1025   Komodo4
    4.7564       1.000     0.000     6.059     0.000    17.9695     1025   IvanHoe

I am going to duplicate this match with the latest development version of Komodo.

Don

Hi Don, I am very curious about the results. Thx for sharing.

grts Bram

[geots]

Anyone need anything from me, they need to get me in the next 45 min. or so. After that I will be having sweet dreams of Salma Hayek till 10:00 PM CDT, USA. (Watching the end of "Desperado" for the nth time)

gts