Does Rybka 2.3.2a 32-bit cheat when playing ply games ?

[Spacious_Mind]

I think it is also unreasonable perhaps if for example the C64 1MHz gets say 1800 seconds to move and the the engine 1 second as an example because the differences in speed would amount to something like this.

Most programs have no difficulty playing at 0.1 sec/move, so even with the extreme factor you mention you might be able to get into the realm of reasonable times/move.

It is very easy to try this; WinBoard allows you to enter time-odds factors for each engine separately in the time-control dialog. Of course when you enter the moves by hand, it will consider it an engine vs Human game, but in that case the time-odds given for the second engine is used for the Human (irrespective of which color you play).

I had considered this possibility, but unfortunately I think this method puts the engine at a bigger dissadvantage. Probably, because of the exact reasons explained by Bob Hyatt in his earlier post. Staying with the example of a speed differenz of 1800. The formula I think would be as you have shown 1/1800*180s = 0.1 seconds per move for the engine whereas the C64 plays 180 seconds per move. Well in the example shown in my earlier post the average thinking time for 5 ply from Stockfish 1.5.1. was @ 0.3 seconds per move (100 moves in 30 seconds). This would make me think that the engine is at a clear disatvantage with this method. Or am I missing something?

[hgm]

It seems like it, although an average time per move might not mean a whole lot. All the time might have been eaten by just a few moves, while the blunder that cost the game was hardly thought about. This is a problem inherent in playing fixed depth; the time needed for a move often is extremely variable.

[Spacious_Mind]

Hi Nick and many greetings to you across the Atlantic from Germany. I hope we two can become chessfriends in future because we have so many similarities except the only difference of our origine of culture. But psychologically we are twin brothers in a way. Because when I wrote my first messages back in 1996 I started with a similar attack against a real top number one at the time (Ed Schroder NL), topic was business tricks, while he was at the same time active in questions against another number one guy (Marty Hirsch USA) for hidden opening tricks that brought him allegedly unfair advantages. Now you entered with your researches directly into the middle of the top debate if a "Rybka", the actual top winner program, with questioning if it "cheated".

The problem is this. After the battle with hidden hardware mega irons is over the USA, who was practically leader in that competition, lost all top places/ranks IMO because in your region different to Europe you follow a principle that you somehow underestimate or disrespect the software ideas because similar to what China does, you dismantle every interesting producht in chess software and publish the secrets as if there were no personal ownership for the inventor.

Hi Rolf,
Thank you for response to my Post. I know Germany very well, I was born there and although I grew up in England, I also studied in Germany and I also speak German very well. I now live in the U.S but my roots are European and both my parents are Yugoslav. I also worked in Brazil and my girlfriend lives there. I am pointing this out because I have no axes to grind. I treat the World as my home because otherwise whereever I go I would be a foreigner

Nick, do you really think that a software cheated if your chosen setting of technical details brings you to unexpected results in a statistical sense? Isnt a software designed to play the best chess it can? Where is the cheat, please tell me, as researcher.

What do you think about the apparent lack of commercial engines in the USA? And why is it that all top progs reside in Europe? Nick, do you see the poissible reason in the open publication of all the secrets a software could have? Isnt it also telling that you didnt present your astonishing results on the Rybkaforum?

I do not think that you read my question seriously, because I sense that you are trying to read between the lines. First of all, a fellow dedicated chess computer enthusiast recommended this Forum because of the interesting topics that are posted here by many knowledgeable and well known people from the computer chess community.

My question was very specific to Rybka v2.3.2a.w32 which I tested playing some 5 ply games and very simple coming from someone who does not program and who uses chess software and engines to play games. The program does not behave as other programs do when playing 5 ply. It uses almost half the time compared to other engines that I have tried so far, but at the same time it seems to be a lot stronger. Therefore, I am interested in finding out why because as it stands at the moment the behaviour makes me think that it does not follow the setup rules for PLY games with Ponder Off, it seems to be doing something more then that. Threrefore for my Tests in Ply games this engine is useless for comparison purposes, unless someone can confirm that it behaves perfectly correctly as it was told to do in the SETUP (In otherwords not sneak in and steal some of the opponents CPU time when it is not it's turn). It also now makes me wonder if it behaves correctly in engine matches where both engines play from the same computer.

I do not think that the program cheats if you are playing against a human (why should it?) neither do I think there is anything wrong if it plays against other engines, so long as each engine runs from a different computer. The program itself is great and very powerful and wonderful for evaluation purposes, but it does not seem to follow certain SETUP RULES, which in my world makes it not suitable for these purposes.

Is it not allowed to ask questions when you are unclear about something? Something that you see happening with your own eyes? I have never posted in the Rybka Forum, I am not even sure if I have ever bothered to register there, but trust me, I would have no hesitation in asking exactly the same questions

Ok, let's talk about Free Speech and freedom to research anything you want. What is the reason for you (in the USA) to destilate in public the details of commercial software that runs under European copyright laws?

Why then it's not allowed to debate the science violations in the DB2 vs Kasparov match in 1997? Why the double standard?

Are you game in a new friendship based on hard debates?

I am certainly interested in hard debates but please no Politics or Religion!

In Europe we have different copyright laws. Look, for me it would be a no go to take Stockfish which is already a clone of Rybka IMO. More, it is absolutely inexplainable how you could dare to ask if a software cheated, while it's your private business to research such details whereas we in Europe enjoy Rybka as the actually top program. World Champion.

I am not disputing Rybka's position in the World at all Neither can I make a comment about Stockfish. A CLONE to me (in my dedicated world) is if a program replays exactly the same moves as another program would. I don't know if Stockfish does this or not. If you are asking if Stockfish originates from Rybka then again I cannot say because I do not know. Do not all chess programs start from the same root beginnings ? In otherwords does not each chess program have something that originated in another program?

All the best

Nick

[Spacious_Mind]

Here's the basic issues. There is old/new hardware, and old/new software. Nothing says old software on new hardware needs any sort of "handicap" for comparison. If both programs use equal hardware, what you are doing is getting a good measure of how much software progress has been made since the old software version you are testing was written. You have upgraded it's hardware, so all that is left is the programming.

I don't think that there is any reasonable way to handicap further, since all you can really do is artificially weaken the newer program until you make it play at the level of the old program. This would tell you that the new software is equivalent to a Nx hardware speedup. Say you have to give the old program a 20:1 time handicap advantage to make the old and new play equally on the same hardware. Then the software improvements would be worth about 20x. This is not quite a valid comparison, because most likely new software on old hardware would really play badly since the search depth would be so limited the reductions and null-move would cause grave tactical issues or else be completely ineffectual.

You have captured the problem that has eluded my search for a fair method to compare software from the past with software from today and thereby chart the progress made over the years, because many of the softwares that interest me are either inside a dedicated chess computer or on an old Home Computer. An Emulator would perhaps work for Home Computers but even then the comparison would have it's flaws I suspect.

On a lighter note out of pure gaming curiosity I am seriously thinking of starting a new tournament with teams from old Home Computers, Dedicated Manufacturers and engines (some modern and some a little older) all play 5 ply which seems to be a great equalizer, and just see what happens

5 Ply interests me because it seems to be barely manageable with old Home Computers (timewise) and also about the right strength where most average chess players can follow the games and enjoy the combinations or mistakes with their own eyes and without too much stress.

I still suspect that an engine team would end up winning the tournament, but the games would be a lot closer and the outcome unknown. If Rybka 2.3.2a 32-bit plays then I might just have to handicap it with 4 ply

hmmmmmm... i wonder if Crafty should play..........

Best regards

Nick

[Uri Blass]

I think it is also unreasonable perhaps if for example the C64 1MHz gets say 1800 seconds to move and the the engine 1 second as an example because the differences in speed would amount to something like this.

Most programs have no difficulty playing at 0.1 sec/move, so even with the extreme factor you mention you might be able to get into the realm of reasonable times/move.

It is very easy to try this; WinBoard allows you to enter time-odds factors for each engine separately in the time-control dialog. Of course when you enter the moves by hand, it will consider it an engine vs Human game, but in that case the time-odds given for the second engine is used for the Human (irrespective of which color you play).

I had considered this possibility, but unfortunately I think this method puts the engine at a bigger dissadvantage. Probably, because of the exact reasons explained by Bob Hyatt in his earlier post. Staying with the example of a speed differenz of 1800. The formula I think would be as you have shown 1/1800*180s = 0.1 seconds per move for the engine whereas the C64 plays 180 seconds per move. Well in the example shown in my earlier post the average thinking time for 5 ply from Stockfish 1.5.1. was @ 0.3 seconds per move (100 moves in 30 seconds). This would make me think that the engine is at a clear disatvantage with this method. Or am I missing something?

My comments:
1)I do not believe that stockfish needs 0.3 seconds per move at 5 plies and I guess that most of the time is eaten by the interface.

stockfish is a fast searcher

2)I believe that you are interested in evaluation improvement and not in search improvement but even in this case fixed depth is no way to find if stockfish's evaluation is better than the opponent because the engines do not search the same tree in fixed depth and do not do the same pruning and extensions.

Competition between evaluation when you force both programs to search the same tree may be possible if you have the evaluation code of both programs but I suspect that the evaluation of the dedicated computers is a secret and without getting the code for the evaluation of the dedicated computer it is impossible to compare by that way.

There is another problem in competition between evaluation and the winner may dependent on the type of the tree that is searched.

For example
a program may not be designed to evaluate correctly positions when the king is under threat because the program extends these positions in the search.

It is possible to have a tree when you simply search 5 plies forward and do qsearch only of captures and it is also possible to have a tree when you simply search 5 plies forward and do qsearch when you always get out of check.

Uri

[Spacious_Mind]

My comments:
1)I do not believe that stockfish needs 0.3 seconds per move at 5 plies and I guess that most of the time is eaten by the interface.

stockfish is a fast searcher

2)I believe that you are interested in evaluation improvement and not in search improvement but even in this case fixed depth is no way to find if stockfish's evaluation is better than the opponent because the engines do not search the same tree in fixed depth and do not do the same pruning and extensions.

Competition between evaluation when you force both programs to search the same tree may be possible if you have the evaluation code of both programs but I suspect that the evaluation of the dedicated computers is a secret and without getting the code for the evaluation of the dedicated computer it is impossible to compare by that way.

There is another problem in competition between evaluation and the winner may dependent on the type of the tree that is searched.

For example
a program may not be designed to evaluate correctly positions when the king is under threat because the program extends these positions in the search.

It is possible to have a tree when you simply search 5 plies forward and do qsearch only of captures and it is also possible to have a tree when you simply search 5 plies forward and do qsearch when you always get out of check.

Uri

Hi Uri

Very interesting. Are you suggesting that I should contemplate a tournament as suggested by H.G Muller starting as per the following example:

Engine 0.01 second per move (exact start dependent on PC speed)
C64 (1 MHz) at 180 seconds per move
Amiga (8 MHz) at 22 or 23 seconds per move
Mephisto Lyon (12 MHz) at 15 seconds

and so on..

Do you feel that an engine (Movei) would play better under this setting using Winboard compared to a ply format. Would this be fairer for every platform playing instead of using Ply ?

regards

Nick

[Uri Blass]

My comments:
1)I do not believe that stockfish needs 0.3 seconds per move at 5 plies and I guess that most of the time is eaten by the interface.

stockfish is a fast searcher

2)I believe that you are interested in evaluation improvement and not in search improvement but even in this case fixed depth is no way to find if stockfish's evaluation is better than the opponent because the engines do not search the same tree in fixed depth and do not do the same pruning and extensions.

Competition between evaluation when you force both programs to search the same tree may be possible if you have the evaluation code of both programs but I suspect that the evaluation of the dedicated computers is a secret and without getting the code for the evaluation of the dedicated computer it is impossible to compare by that way.

There is another problem in competition between evaluation and the winner may dependent on the type of the tree that is searched.

For example
a program may not be designed to evaluate correctly positions when the king is under threat because the program extends these positions in the search.

It is possible to have a tree when you simply search 5 plies forward and do qsearch only of captures and it is also possible to have a tree when you simply search 5 plies forward and do qsearch when you always get out of check.

Uri

Hi Uri

Very interesting. Are you suggesting that I should contemplate a tournament as suggested by H.G Muller starting as per the following example:

Engine 0.01 second per move (exact start dependent on PC speed)
C64 (1 MHz) at 180 seconds per move
Amiga (8 MHz) at 22 or 23 seconds per move
Mephisto Lyon (12 MHz) at 15 seconds

and so on..

Do you feel that an engine (Movei) would play better under this setting using Winboard compared to a ply format. Would this be fairer for every platform playing instead of using Ply ?

regards

Nick

1)I cannot say that an engine is going to play better with 0.01 seconds per move and not with ply format and I cannot say the opposite.

2)There are engines that do some initializations in the beginning of the search.

The initialization may take only 0.02 seconds on fast computers and is not significant in practical game when they lose only 1 second per game for initializations but it can make them too slow to play at 0.01 second per move.

Movei clearly does some things that I could avoid in case that the target was to tune it for conditions of very fast time control but I think that it can play at 0.01 seconds per move(I did not test recently so I am not 100% sure about it).

[Spacious_Mind]

Well I am getting excited about this. My mouth is starting to water at the prospect of a new tournament

So, the decision to play ply or level (with or without average time) Ponder Off, stays open. If the configurations work then I might start with the base of C64 at 6 mins per move which gives the engines 0.2 secs per move.

But subject to change (after tweaking) this particular World Cup (Soccer format) would/might look like this:

Team C64
Colossus 4
Sargon III
Chessmaster 2100

Team ATIGA QL
Psion
Chess Player 2175
Complete Chess System Tal

Team SAB
Sinclair Superchess 3.5
Atari 800 Colossus 3.0
BBC White Knight MK 12

Team Mephisto 16
Berlin 68000
Almeria 68000
Portorose 68000

Mephisto 32
Lyon 68020
Vacouver 68020
Berlin Pro 68020

Team Novag
Star Diamond
Sapphire II
Super Expert C

Team Saitek
Brute Force
Risc 2500
Master

Mephisto 8 Bit
Nigel Short
Academy
Amsterdam

Fidelity
Designer Mach IV
Designer Mach III
Designer 2100

SciSys
Corona
Turbostar 432
Turboking

Tasc
CM 16 Madrid 3.1
CM 16 Gideon 3.0
CM 16 King 2.54

Resurrection
Toga 1.2
Fruit 2.1
Ruffian 1.5

Engine Team USA
XXX
XXX
XXX

Engine Team Germany
XXX
XXX
XXX

Engine Team Holland
XXX
XXX
XXX

Engine Team ROW 1
XXX
XXX
XXX


This is assuming 4 groups of 4 teams. If it grows to 32 teams then it will be 8 groups following the world cup format. Followed by 1/4 final (or final 16) etc.

The engine teams are open to discussion if anyone has an interest to see particular engines play in this challenge If there is no interest then I will just pick my own

After tweaking due to play levels some of the above dedicates may change.

What do you think? would this be an interesting tournament or challenge?

The tournament rules I would still have to define in more detail.

I would play this tournament for the sheer fun of the unknown and experience it brings... there is no intention here to further anyones interests or to demean them. But this tournament could be so cool

All the best

Nick

[Rolf]

I treat the World as my home because otherwise whereever I go I would be a foreigner

Hi Nick, yes that is a good description of what I tried to criticise in your approach, see the first message. What you confuse is your approach and the responsibility of something you approached, here the software. Your approach and your views are all yours, but not the one of the software. If you create a setting how can you conclude that a software cheats on your setting? IMO the software is completely innocent.

BTW it's not only you here but many others too who create a sort of wrongdoing on Rybka side on the base of certain findings that others got
themselves acting with illegal wrongdoings. Know what I mean?

I do not think that you read my question seriously, because I sense that you are trying to read between the lines. The program itself is great and very powerful and wonderful for evaluation purposes, but it does not seem to follow certain SETUP RULES, which in my world makes it not suitable for these purposes.

There you have it what I mean. Your approach also in reading is your personal way but you are unfamiliar with my method. It's nothinmg personal of me because it's just what one learns in studies of psychology. The interpretation of communication is normal after such time spent with the topics of social sciences. It's based on logic. And by force I discover this illogical reasoning. Where did you learn to believe that a chessprogram should follow your Setup Rules? Isnt science research linked to appropriate choices of methods before you even start to measure something?

Look I am very astonished that programmers never prevented certain testing routines by amateur "testers" in computerchess before. But if I would program or if I would test I would be aware of the different possibilities to influence the results of naive testers. Therefore I cant be accused of cheating IMO. Therefore I felt offended siding with Marty when he was accused of cheats with his opening book tricks. I was outraged because I saw no violation of any truth or science. The years have passed since then so that I am perhaps allowed to publish what Marty wrote to me in dispair because in his eyes the help from a total lay like me didnt look optimal also when I was quarreling with the leader figure Ed Schroeder, what nobody really could understand. He wrote. Yes, thanks very much, Rolf, but please do no longer support me in public... which was quite a shock for me at the time because I knew for sure that I was right and Marty did nothing wrong. But on the other hand I knew also that DB2 team of IBM violated scientific fundamentals when they created new aspects that influenced the outcome when allegedly the machine play alone should be researched against the best human player. It's so basic that it was another big shock to see even veritable scientist here who defended IBM and insulted Kasparov.

Is it not allowed to ask questions when you are unclear about something? Something that you see happening with your own eyes? I have never posted in the Rybka Forum, I am not even sure if I have ever bothered to register there, but trust me, I would have no hesitation in asking exactly the same questions

I would hate you if you would NOT speak out all the questions you have. That is why this forum is existing for. Only without me on response if it's coming to tech details. If however the science aspects of testing or of experimental design are on top of the day, then I am happy if I can help out. The truth is however that asking questions is not the most natural behavior in our community. Most people confuse questions with showing failures and disadvantages, a basic psychological fault.

BTW I must correct a false usage of an engine as an example. I wrote Stockfish when I should have written Shark. A typo.

A final idea for you. If I were you with these ideas for tournaments with little ply, please do concentrate on The Dutch Dr. Muller. He is that sort of guy who is in love with minimalism instead of huge power. I saw that he already answered you.

All the best again from my side for your new life over there.
-Rolf

[Spacious_Mind]

Hi Nick, yes that is a good description of what I tried to criticise in your approach, see the first message. What you confuse is your approach and the responsibility of something you approached, here the software. Your approach and your views are all yours, but not the one of the software. If you create a setting how can you conclude that a software cheats on your setting? IMO the software is completely innocent.

BTW it's not only you here but many others too who create a sort of wrongdoing on Rybka side on the base of certain findings that others got
themselves acting with illegal wrongdoings. Know what I mean?

There you have it what I mean. Your approach also in reading is your personal way but you are unfamiliar with my method. It's nothinmg personal of me because it's just what one learns in studies of psychology. The interpretation of communication is normal after such time spent with the topics of social sciences. It's based on logic. And by force I discover this illogical reasoning. Where did you learn to believe that a chessprogram should follow your Setup Rules? Isnt science research linked to appropriate choices of methods before you even start to measure something?

Look I am very astonished that programmers never prevented certain testing routines by amateur "testers" in computerchess before. But if I would program or if I would test I would be aware of the different possibilities to influence the results of naive testers. Therefore I cant be accused of cheating IMO. Therefore I felt offended siding with Marty when he was accused of cheats with his opening book tricks. I was outraged because I saw no violation of any truth or science. The years have passed since then so that I am perhaps allowed to publish what Marty wrote to me in dispair because in his eyes the help from a total lay like me didnt look optimal also when I was quarreling with the leader figure Ed Schroeder, what nobody really could understand. He wrote. Yes, thanks very much, Rolf, but please do no longer support me in public... which was quite a shock for me at the time because I knew for sure that I was right and Marty did nothing wrong. But on the other hand I knew also that DB2 team of IBM violated scientific fundamentals when they created new aspects that influenced the outcome when allegedly the machine play alone should be researched against the best human player. It's so basic that it was another big shock to see even veritable scientist here who defended IBM and insulted Kasparov.

I would hate you if you would NOT speak out all the questions you have. That is why this forum is existing for. Only without me on response if it's coming to tech details. If however the science aspects of testing or of experimental design are on top of the day, then I am happy if I can help out. The truth is however that asking questions is not the most natural behavior in our community. Most people confuse questions with showing failures and disadvantages, a basic psychological fault.

BTW I must correct a false usage of an engine as an example. I wrote Stockfish when I should have written Shark. A typo.

A final idea for you. If I were you with these ideas for tournaments with little ply, please do concentrate on The Dutch Dr. Muller. He is that sort of guy who is in love with minimalism instead of huge power. I saw that he already answered you.

All the best again from my side for your new life over there.
-Rolf

Hi Rolf,
Thanks for your response. One of the nice things about Forums is reading comments and reactions from the posters and I see already that this seems to be one of the attractions for you as well.

Sure my first post here in TalkChess was Grandstanding and deliberate, but already it has provoked some thoughts and responses from people who I value very much because of their achievements and dedication to chess programming including all engines and game platforms, and it has already helped me to move a big step forward in a goal that I have set for myself for a long time before I even joined this Forum. I think my next tournament goal can already clearly be seen from my posts.

But here is where I think and you will of course correct me in my error in judgement of thinking that you are wrong in this. Your defense of Rybka is wrong. Rybka does not require your defense at all, it can stand up on it's own two feet, just like any other program.

Rybka started as an engine (I think) no different to any other engine, it was given to the world to play and test and compete against all other engines in order to prove it's prowess. It wants to be the best and IS. But, it did not have it's own platform until recently, it used Winboard, Arena, Chessbase products for this to show its mettle and strength. But here is the rub, if you intend to use other platforms for your base and of course use other engines as your competition and become a Victor, then at least in my humble opinion use the settings provided correctly whether it be ponder on or off, random, ply etc so that you can play against Rybka in a way that these platform settings were intended. I genuinely mean this for every chess engine after all do not all programmers obtain their greatest satisfaction from seeing their creations however strong or weak be used and enjoyed without question by other people? Everyone here in this Forum I think has their own reasons for playing games with chess engines. I think mine have already been outlined.

If for example Rybka had clear disclaimers stating that for whatever uses it is not suitable then I would not have a problem with this at all because then I as a consumer could make my choice accordingly.

Rolf I have asked a specific question to settings in Fritz 9 playing Rybka with 5 ply. Sofar with regards to this question only Robert Hyatt responded with some kind of explanation (perhaps with tongue in cheek ) but still I respected the comment and I have adjusted my opinion accordingly for the uses of the engine in the ways that I have in mind. Btw this amateur testers comment from you is scary... because if I was to rely on your results after your defense of the program then I would perhaps not get the complete picture, afterall it is easy for all of us to make statistics look the way we want to sell it The question thank god is do other people buy into the sale?

Look I am very astonished that programmers never prevented certain testing routines by amateur "testers" in computerchess before. But if I would program or if I would test I would be aware of the different possibilities to influence the results of naive testers.

Do you not think this is a delusion? Does this not sound like "First I want to play with the big kids in the block but now that I am bigger I don't have time for them anymore?

Rolf thanks again for your interesting post which I am enjoying very much, and I hope to share many many more with you.

Kindest regards from Sweet Home Alabama....

Nick