Toga II 3.0 released

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
geots
Posts: 4790
Joined: Fri Mar 10, 2006 11:42 pm

Re: For CCRL Tester/s- One More.....

Post by geots » Mon Mar 25, 2013 5:07 am

Would you happen to know how CEGT would handle the same scenario as above?



And thanks again in advance-

george

User avatar
Eelco de Groot
Posts: 4162
Joined: Sun Mar 12, 2006 1:40 am
Location: Groningen

Re: Toga II 3.0 released

Post by Eelco de Groot » Mon Mar 25, 2013 5:24 am

Eelco de Groot wrote: I see that Teemu Pudas (Vempele) writes in the same thread that he adapted the code so that would imply it is already in one of the CMLX versions. I did not remember that... He probably placed the code better, my version was placed somewhere as one of of the 'win' recognizers/situations early in eval.cpp but I don't know how Teemu did it.
It arrived not exactly later that day but one year later there it was: Toga CMLX 1.4.5e. Teemu's Toga CMLX 1.4.5e version is hopefully still downloadable from this 2009 thread on Rybka forum.

Eelco
Debugging is twice as hard as writing the code in the first
place. Therefore, if you write the code as cleverly as possible, you
are, by definition, not smart enough to debug it.
-- Brian W. Kernighan

User avatar
tjfroh
Posts: 8078
Joined: Thu Mar 09, 2006 8:51 pm
Location: Near the Intel Plant in the Land of Manana
Full name: Timothy Frohlick
Contact:

Re: Toga II 3.0 released

Post by tjfroh » Mon Mar 25, 2013 5:51 am

Toga II 3.0 is going head to head with Stockfish 2.3. on all my dual processor machines. Each have two wins and one draw with a total of five games. Very small sample size but if 1,000,000 players get the same results then that is statistically significant.

I am impressed.


TJF

PS I haven't installed Houdini 3 yet.. I am waiting for my eight core machine with 32 GB of RAM to arrive.
A New Way Comes Upon Earth.
God is an infinitely variable Constant.
Man marks his ground with ideologies.
Galaxies are the dreidels of God.
War is a punishment for implacability.
Peace flows from forgiveness of sins.

IWB
Posts: 1539
Joined: Thu Mar 09, 2006 1:02 pm

Re: Toga II 3.0 released

Post by IWB » Mon Mar 25, 2013 6:23 am

geots wrote:Ok- my mistake. I guess Ingo is running Z Mexico 64bit.
Yes I do run the 64bit version. If someone would bother to read my conditions it becomes very clear. Unfortunately you are not the only one who doesn't care and just look to the rank, rating and if in good mood to the time control. That other conditions are at least as important seems to be ignored by many ... at least according to the emails I get from my web site ... :-(
geots wrote: ... And don't listen to anyone who says 200 games mean nothing and you need over 1000. Stats are what they are, but if under 1000 meant nothing, you could throw out about 50% or more of CCRLs ratings.
Interesting thought :-)! But not what I meant. I meant 150(or 200) games against ONE opponent. That IS irrelevant!
This is an exceprt from the latest Toga run:

Code: Select all

Toga II 3.0 32b - Zappa Mexico II (2703)		63.0	-	87.0		42.00%		Perf=2647
Toga II 3.0 32b - Chiron 1.5 (2845)		57.5	-	92.5		38.33%		Perf=2763
That is a difference of 116 Elo* - which cant be explained by statistics! This is simply the difference that occurs because of the playing style suits better or not. So:
1. You can't draw conclusions out of 150 games against ONE opponent
2. You only get a valid average result if you run against a higher number of opponents. (which makes lists with many times the same opponent and just a few others doubtfull)

Of course, if you are just interested in how does the engine perform against one particular engine you might consider the result interesting ...

Bye
Ingo

*some engines even have a bigger performance gap than 116 Elo between opponents

jd1
Posts: 266
Joined: Wed Oct 24, 2012 12:07 am

Re: Toga II 3.0 released

Post by jd1 » Mon Mar 25, 2013 6:40 am

Eelco de Groot wrote:
Eelco de Groot wrote: I see that Teemu Pudas (Vempele) writes in the same thread that he adapted the code so that would imply it is already in one of the CMLX versions. I did not remember that... He probably placed the code better, my version was placed somewhere as one of of the 'win' recognizers/situations early in eval.cpp but I don't know how Teemu did it.
It arrived not exactly later that day but one year later there it was: Toga CMLX 1.4.5e. Teemu's Toga CMLX 1.4.5e version is hopefully still downloadable from this 2009 thread on Rybka forum.

Eelco
Ok thanks Eelco!

I'll have a look when I start to work on Toga again.

Jerry

User avatar
geots
Posts: 4790
Joined: Fri Mar 10, 2006 11:42 pm

Re: Toga II 3.0 released

Post by geots » Mon Mar 25, 2013 6:47 am

IWB wrote:
geots wrote:Ok- my mistake. I guess Ingo is running Z Mexico 64bit.
Yes I do run the 64bit version. If someone would bother to read my conditions it becomes very clear. Unfortunately you are not the only one who doesn't care and just look to the rank, rating and if in good mood to the time control. That other conditions are at least as important seems to be ignored by many ... at least according to the emails I get from my web site ... :-(
geots wrote: ... And don't listen to anyone who says 200 games mean nothing and you need over 1000. Stats are what they are, but if under 1000 meant nothing, you could throw out about 50% or more of CCRLs ratings.
Interesting thought :-)! But not what I meant. I meant 150(or 200) games against ONE opponent. That IS irrelevant!
This is an exceprt from the latest Toga run:

Code: Select all

Toga II 3.0 32b - Zappa Mexico II (2703)		63.0	-	87.0		42.00%		Perf=2647
Toga II 3.0 32b - Chiron 1.5 (2845)		57.5	-	92.5		38.33%		Perf=2763
That is a difference of 116 Elo* - which cant be explained by statistics! This is simply the difference that occurs because of the playing style suits better or not. So:
1. You can't draw conclusions out of 150 games against ONE opponent
2. You only get a valid average result if you run against a higher number of opponents. (which makes lists with many times the same opponent and just a few others doubtfull)

Of course, if you are just interested in how does the engine perform against one particular engine you might consider the result interesting ...

Bye
Ingo

*some engines even have a bigger performance gap than 116 Elo between opponents




You have no clue what I looked at and care about or don't care about. FYI I saw 32b, but I mistakenly was putting it with the engine on the right. Have you ever considered that in your info you have "32b" highlighted in red. A lot of people might very well think all they have to see is the red and when they don't might assume it is 64. I would highlight it in both places- or in neither- if it were me. But it isn't me, so whatever.


So if I can't draw conclusions from 150 games ag. ONE opponent, then I don't suppose I can draw conclusions from 50 games ag. ONE opponent. Bullshit. I have a 50 game match where a lower tier Ivanhoe version beat Fritz 13 (wins and losses only) 29-2. I have another (2) 50 game matches where Strelka 5.6 beat the strongest Ivanhoe in existence at the moment, 32-4 and 31-6. If I were you, I would add a caveat to your statement. The results are true- if I sounded a bit sarcastic- that is probably true as well, as I'm not at the moment in the mood for testing lessons. But no harm- no foul.


Best,

User avatar
geots
Posts: 4790
Joined: Fri Mar 10, 2006 11:42 pm

Please IGNORE Both "Help" Threads- Sorry!

Post by geots » Mon Mar 25, 2013 7:43 am

I needed to go ahead and run the matches- I will live with 40/3.


Best,

Uri Blass
Posts: 8594
Joined: Wed Mar 08, 2006 11:37 pm
Location: Tel-Aviv Israel

Re: Toga II 3.0 released

Post by Uri Blass » Mon Mar 25, 2013 11:28 am

IWB wrote:
geots wrote:Ok- my mistake. I guess Ingo is running Z Mexico 64bit.
Yes I do run the 64bit version. If someone would bother to read my conditions it becomes very clear. Unfortunately you are not the only one who doesn't care and just look to the rank, rating and if in good mood to the time control. That other conditions are at least as important seems to be ignored by many ... at least according to the emails I get from my web site ... :-(
geots wrote: ... And don't listen to anyone who says 200 games mean nothing and you need over 1000. Stats are what they are, but if under 1000 meant nothing, you could throw out about 50% or more of CCRLs ratings.
Interesting thought :-)! But not what I meant. I meant 150(or 200) games against ONE opponent. That IS irrelevant!
This is an exceprt from the latest Toga run:

Code: Select all

Toga II 3.0 32b - Zappa Mexico II (2703)		63.0	-	87.0		42.00%		Perf=2647
Toga II 3.0 32b - Chiron 1.5 (2845)		57.5	-	92.5		38.33%		Perf=2763
That is a difference of 116 Elo* - which cant be explained by statistics! This is simply the difference that occurs because of the playing style suits better or not. So:
1. You can't draw conclusions out of 150 games against ONE opponent
2. You only get a valid average result if you run against a higher number of opponents. (which makes lists with many times the same opponent and just a few others doubtfull)

Of course, if you are just interested in how does the engine perform against one particular engine you might consider the result interesting ...

Bye
Ingo

*some engines even have a bigger performance gap than 116 Elo between opponents
I think that it is possible to get conclusions based on 150 games against a single opponent and the question is what are the conclusions.

A program may perform better against one opponent because of playing style but there is a limit for it.

If I see for example result like 120-30 I can be practically sure that the winner is better.
If I see for example result like 90-60 I cannot be practically sure that the winner is better but I can be practically sure that the winner is not more than 100 elo worse than the loser.

These conclusions are based on common sense and previous experience and I was relatively careful here.

I believe that there is no single practical case when A score even 70% against B in 150 games when A score worse than B against C.

Note that C should have a similiar rating to the average of A and B(not more than 100 elo difference).

If you think that I am wrong then I would like to see a single case including the names of the programs A,B,C so everybody can reproduce the results(with possible small differences).

Note that I do not claim that it is impossible to build programs A,B,C when it happens but only that it practically does not happen.

User avatar
geots
Posts: 4790
Joined: Fri Mar 10, 2006 11:42 pm

Re: Toga II 3.0 released

Post by geots » Mon Mar 25, 2013 11:50 am

Uri Blass wrote:
IWB wrote:
geots wrote:Ok- my mistake. I guess Ingo is running Z Mexico 64bit.
Yes I do run the 64bit version. If someone would bother to read my conditions it becomes very clear. Unfortunately you are not the only one who doesn't care and just look to the rank, rating and if in good mood to the time control. That other conditions are at least as important seems to be ignored by many ... at least according to the emails I get from my web site ... :-(
geots wrote: ... And don't listen to anyone who says 200 games mean nothing and you need over 1000. Stats are what they are, but if under 1000 meant nothing, you could throw out about 50% or more of CCRLs ratings.
Interesting thought :-)! But not what I meant. I meant 150(or 200) games against ONE opponent. That IS irrelevant!
This is an exceprt from the latest Toga run:

Code: Select all

Toga II 3.0 32b - Zappa Mexico II (2703)		63.0	-	87.0		42.00%		Perf=2647
Toga II 3.0 32b - Chiron 1.5 (2845)		57.5	-	92.5		38.33%		Perf=2763
That is a difference of 116 Elo* - which cant be explained by statistics! This is simply the difference that occurs because of the playing style suits better or not. So:
1. You can't draw conclusions out of 150 games against ONE opponent
2. You only get a valid average result if you run against a higher number of opponents. (which makes lists with many times the same opponent and just a few others doubtfull)

Of course, if you are just interested in how does the engine perform against one particular engine you might consider the result interesting ...

Bye
Ingo

*some engines even have a bigger performance gap than 116 Elo between opponents
I think that it is possible to get conclusions based on 150 games against a single opponent and the question is what are the conclusions.

A program may perform better against one opponent because of playing style but there is a limit for it.

If I see for example result like 120-30 I can be practically sure that the winner is better.
If I see for example result like 90-60 I cannot be practically sure that the winner is better but I can be practically sure that the winner is not more than 100 elo worse than the loser.

These conclusions are based on common sense and previous experience and I was relatively careful here.

I believe that there is no single practical case when A score even 70% against B in 150 games when A score worse than B against C.

Note that C should have a similiar rating to the average of A and B(not more than 100 elo difference).

If you think that I am wrong then I would like to see a single case including the names of the programs A,B,C so everybody can reproduce the results(with possible small differences).

Note that I do not claim that it is impossible to build programs A,B,C when it happens but only that it practically does not happen.



You are NOT wrong, my friend. I could not have said it any better!



Best,

george

IWB
Posts: 1539
Joined: Thu Mar 09, 2006 1:02 pm

Re: Toga II 3.0 released

Post by IWB » Mon Mar 25, 2013 3:28 pm

Uri Blass wrote:
I think that it is possible to get conclusions based on 150 games against a single opponent and the question is what are the conclusions.

A program may perform better against one opponent because of playing style but there is a limit for it.

If I see for example result like 120-30 I can be practically sure that the winner is better.
If I see for example result like 90-60 I cannot be practically sure that the winner is better but I can be practically sure that the winner is not more than 100 elo worse than the loser.

These conclusions are based on common sense and previous experience and I was relatively careful here.

I believe that there is no single practical case when A score even 70% against B in 150 games when A score worse than B against C.

Note that C should have a similiar rating to the average of A and B(not more than 100 elo difference).

If you think that I am wrong then I would like to see a single case including the names of the programs A,B,C so everybody can reproduce the results(with possible small differences).

Note that I do not claim that it is impossible to build programs A,B,C when it happens but only that it practically does not happen.
I have no idea why people are so nitpicking. Your example of 70% is about 150 Elo difference (or 105 to 45 in a 150 game match) while in the given case we have a 50 Elo difference.
So yes, there are case where you can draw conclusions,
1. Where you know the result in advance and
2. Where you have differences which are huge.

The Toga example shows quite nice a shown 50 Elo difference is in reality 0 and that goes in both directions. Where Toga underperformed roughly 70 Elo against Zappa it overperformed ~40 Elo vs Chiron. If you guys are happy with a 150 Elo (70%) accuracy, than fine you can draw conclusions, for myself this is not good enough.

Best
Ingo

EDIT: All the data of the IPON is online, you can search for any best/worst perfomance there, and I am sure there are examples with more than 116 Elo ... (Toga will be updated in a few hours)

Post Reply