Is Naum 4 a serious challenge for Rybka

[M ANSARI]

I think things will be pretty clear soon ... I have been doing a 300 game 60 minute +1sec on 3 Quads for almost a week. I am using neutral 7 move book. I am testing N4 against both Rybka 3_default contempt and Rybka 3_0 contempt. So far with 70% of the games played it is clear that R3 is quite a bit stronger ... but it is also seems that R3_0_cont is scoring higher. On the one computer I have at work where I am typing this post this is the score so far

N4_60_1_gaunt 2008

Naum 4 - Rybka 3_no_cont 10.5 - 21.5 +2/-13/=17 32.81%
Naum 4 - Rybka 3 14.5 - 17.5 +6/-9/=17 45.31%


Let me say that the reason I started testing contempt was when I was going through the games that R3 lost ... chesswise it was obvious that R3 was pushing too hard and that in many games it would lose simply because it had overreached. N4 is simply too strong and once R3's initiative evaporated, N4 had enough technique and strength to punish Rybka for speculative play. In many lost games I could not really find an evaluation mistake (although in many there were). This is what led me to believe that contempt was hurting Rybka's performance and I have a feeling this hunch will turn out to be correct.


I will collect all 300 games once the tourney is finished and tabulate them again. Still this is a very good performance for N4 but to test R3 at LTC it seems it will perform better at 0 contempt. By the way I have also tested the same thing at 16_1 time controls and on 30_1 time controls ... the results seem pretty much the same. The only time normal Rybka performed as good as Rybka 0 contempt was when I used Perfect 15 book. But I don't think that was an accurate test since 198 of the 200 games turned out to be B90 (the other two games were B92). Neutral books give a better idea of performance of an engine. I also did a 1000 game 1_1 test on Octa 4ghz using differnt cont settings ... the 30 cont and 15 cont scored almost 90% with lower 0 cont and 8 cont and -8 cont scoring in the 80's%. So contempt has a positive effect on R3 at fast time controls and this seems to decline linearily as TC go higher.

[M ANSARI]

I have very clear that the answer to the title of this post is a big YES! when we talk about long time controls.

Tomorrow I will post the final results of mi test of 200 games at 60 minutes in a Quad 6700 between both engines. The victory of Naum4 has been impressive.

My feeling is that at long time controls and fast hardware, Naum4 is slightly better than Rybka3.

Regards from Barcelona.

Tom.

What book are you using and how long are the book lines? If you are using one of the deep Playchess tuned books please check how many of your games are B90. To test an engine it is better to use neutral books or set conditions. Ofcourse you don't have to do that and you might want to test certain books for fun, but that really doesn't give an accurate assessment of engine strength

[Tomcass]

I am conscious that the books I use are decisive for the results I get. Most of games are B80 and B90. I will eliminate repeated games, although by now I have found only 2 of them repeated.

Obviously I agree with you that a neutral set of conditions would give a more reliable result. Can you tell me, please, how can I use one of these reliable sets?.

The books I am using are

In the first 100 games: RybkaII for R3 and Perfect15 for Naum4.

In games 101 to 150: Perfect15 for R3 and Storm for Naum4.

In games 151 to 200: Perfect15 for R3 and Magnificient for N4.

Since I think that my test will finish tonight, I give you the CUMULATIVE RESULTS after 192 games:

Tomcass, Blitz:60'


1 Naum 4(x4) +53/-25/=114 57.29% 110.0/192
2 Rybka 3 32-bit +25/-53/=114 42.71% 82.0/192

You will see now why I said that the victory of Naum4 over Rybka3 has been really IMPRESSIVE.

Regards from Barcelona.

Tom.

[M ANSARI]

You can try using the books you are using and allow it to only choose say 10 moves from book and then the engine plays. But even so I did such a tournament with Perfect 15 book and over 90% of the games were B90. So now I choose Harry's 7 move book ... which is very neutral and gives a variety of different openings. There is a Noomen set of positions which is also good to test and the CB GUI can allow you to play each position with colors reversed ... but honestly I have never tried them. Still a lot can be learned by using the books you have used, I for one found out that in some positions out of B90 Rybka misevaluates the strength of a bishop pair and will lose what it used to think was a better position for it as black. But really such tests are useless for reaching a conclusion of the strength of an engine.

[Uri Blass]

You can try using the books you are using and allow it to only choose say 10 moves from book and then the engine plays. But even so I did such a tournament with Perfect 15 book and over 90% of the games were B90. So now I choose Harry's 7 move book ... which is very neutral and gives a variety of different openings. There is a Noomen set of positions which is also good to test and the CB GUI can allow you to play each position with colors reversed ... but honestly I have never tried them. Still a lot can be learned by using the books you have used, I for one found out that in some positions out of B90 Rybka misevaluates the strength of a bishop pair and will lose what it used to think was a better position for it as black. But really such tests are useless for reaching a conclusion of the strength of an engine.

I wonder what happened to the book learning.
I do not give chess engines to play with books but if rybka has problems against naum in B90 then common sense tells me that learning is going to tell it to play other lines.

Uri

[M ANSARI]

Rybka has no problems with B90 scoring almost 70% against N4 in B90. But if you put different books with different lines of B90 against each other then things are not predictable. You have to remember that 2 different books played against each using same Rybka engine have differed in some cases more than 300 ELO points. That is why to test engine performance you need to use same book for both sides ... otherwise the different strengths of books can pollute the final outcome.

[swami]

I wonder if the latest Rybka supports learning function the way Romichess and Learninglemming does? Or is it still in Vas's To-do list?

[M ANSARI]

I am not sure but I do see a book learning box in the CB GUI, however I never turn it on.

[Dr.Wael Deeb]

I wonder if the latest Rybka supports learning function the way Romichess and Learninglemming does? Or is it still in Vas's To-do list?

Isn't the "Preserve hash" function in the Rybka UCI configuration file some kind of learning feature

[Jeroen]

You are seriously handicapping Rybka 3 in these matches:

Perfect15 is a much better book than RybkaII (which is already 2 years old, and probably more than 150 elo weaker).

Storm is a much better book than Perfect15, the same for Magnificent.

For equal conditions for both programs you could use one of my test suites, f.e. Noomen Testsuite 2008. You can download it here:

http://www.rybkachess.com/index.php?auswahl=Downloads

If you let both programs play the white side and the black side from each position, none of the two has a book advantage.