Is Naum 4 a serious challenge for Rybka

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Dann Corbit, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
M ANSARI
Posts: 3500
Joined: Thu Mar 16, 2006 6:10 pm

Re: Is Naum 4 a serious challenge for Rybka

Post by M ANSARI » Sun Jan 04, 2009 6:12 am

I think things will be pretty clear soon ... I have been doing a 300 game 60 minute +1sec on 3 Quads for almost a week. I am using neutral 7 move book. I am testing N4 against both Rybka 3_default contempt and Rybka 3_0 contempt. So far with 70% of the games played it is clear that R3 is quite a bit stronger ... but it is also seems that R3_0_cont is scoring higher. On the one computer I have at work where I am typing this post this is the score so far

N4_60_1_gaunt 2008

Naum 4 - Rybka 3_no_cont 10.5 - 21.5 +2/-13/=17 32.81%
Naum 4 - Rybka 3 14.5 - 17.5 +6/-9/=17 45.31%


Let me say that the reason I started testing contempt was when I was going through the games that R3 lost ... chesswise it was obvious that R3 was pushing too hard and that in many games it would lose simply because it had overreached. N4 is simply too strong and once R3's initiative evaporated, N4 had enough technique and strength to punish Rybka for speculative play. In many lost games I could not really find an evaluation mistake (although in many there were). This is what led me to believe that contempt was hurting Rybka's performance and I have a feeling this hunch will turn out to be correct.


I will collect all 300 games once the tourney is finished and tabulate them again. Still this is a very good performance for N4 but to test R3 at LTC it seems it will perform better at 0 contempt. By the way I have also tested the same thing at 16_1 time controls and on 30_1 time controls ... the results seem pretty much the same. The only time normal Rybka performed as good as Rybka 0 contempt was when I used Perfect 15 book. But I don't think that was an accurate test since 198 of the 200 games turned out to be B90 (the other two games were B92). Neutral books give a better idea of performance of an engine. I also did a 1000 game 1_1 test on Octa 4ghz using differnt cont settings ... the 30 cont and 15 cont scored almost 90% with lower 0 cont and 8 cont and -8 cont scoring in the 80's%. So contempt has a positive effect on R3 at fast time controls and this seems to decline linearily as TC go higher.

User avatar
M ANSARI
Posts: 3500
Joined: Thu Mar 16, 2006 6:10 pm

Re: Is Naum 4 a serious challenge for Rybka

Post by M ANSARI » Sun Jan 04, 2009 6:45 am

I have very clear that the answer to the title of this post is a big YES! when we talk about long time controls.

Tomorrow I will post the final results of mi test of 200 games at 60 minutes in a Quad 6700 between both engines. The victory of Naum4 has been impressive.

My feeling is that at long time controls and fast hardware, Naum4 is slightly better than Rybka3.

Regards from Barcelona.

Tom.


What book are you using and how long are the book lines? If you are using one of the deep Playchess tuned books please check how many of your games are B90. To test an engine it is better to use neutral books or set conditions. Ofcourse you don't have to do that and you might want to test certain books for fun, but that really doesn't give an accurate assessment of engine strength

Tomcass
Posts: 786
Joined: Sun Apr 16, 2006 7:09 pm

Re: Is Naum 4 a serious challenge for Rybka

Post by Tomcass » Sun Jan 04, 2009 10:04 am

I am conscious that the books I use are decisive for the results I get. Most of games are B80 and B90. I will eliminate repeated games, although by now I have found only 2 of them repeated.

Obviously I agree with you that a neutral set of conditions would give a more reliable result. Can you tell me, please, how can I use one of these reliable sets?.

The books I am using are

In the first 100 games: RybkaII for R3 and Perfect15 for Naum4.

In games 101 to 150: Perfect15 for R3 and Storm for Naum4.

In games 151 to 200: Perfect15 for R3 and Magnificient for N4.

Since I think that my test will finish tonight, I give you the CUMULATIVE RESULTS after 192 games:

Tomcass, Blitz:60'


1 Naum 4(x4) +53/-25/=114 57.29% 110.0/192
2 Rybka 3 32-bit +25/-53/=114 42.71% 82.0/192

You will see now why I said that the victory of Naum4 over Rybka3 has been really IMPRESSIVE. :wink:

Regards from Barcelona.

Tom.

User avatar
M ANSARI
Posts: 3500
Joined: Thu Mar 16, 2006 6:10 pm

Re: Is Naum 4 a serious challenge for Rybka

Post by M ANSARI » Sun Jan 04, 2009 12:22 pm

You can try using the books you are using and allow it to only choose say 10 moves from book and then the engine plays. But even so I did such a tournament with Perfect 15 book and over 90% of the games were B90. So now I choose Harry's 7 move book ... which is very neutral and gives a variety of different openings. There is a Noomen set of positions which is also good to test and the CB GUI can allow you to play each position with colors reversed ... but honestly I have never tried them. Still a lot can be learned by using the books you have used, I for one found out that in some positions out of B90 Rybka misevaluates the strength of a bishop pair and will lose what it used to think was a better position for it as black. But really such tests are useless for reaching a conclusion of the strength of an engine.

Uri Blass
Posts: 8948
Joined: Wed Mar 08, 2006 11:37 pm
Location: Tel-Aviv Israel

Re: Is Naum 4 a serious challenge for Rybka

Post by Uri Blass » Sun Jan 04, 2009 1:10 pm

M ANSARI wrote:You can try using the books you are using and allow it to only choose say 10 moves from book and then the engine plays. But even so I did such a tournament with Perfect 15 book and over 90% of the games were B90. So now I choose Harry's 7 move book ... which is very neutral and gives a variety of different openings. There is a Noomen set of positions which is also good to test and the CB GUI can allow you to play each position with colors reversed ... but honestly I have never tried them. Still a lot can be learned by using the books you have used, I for one found out that in some positions out of B90 Rybka misevaluates the strength of a bishop pair and will lose what it used to think was a better position for it as black. But really such tests are useless for reaching a conclusion of the strength of an engine.
I wonder what happened to the book learning.
I do not give chess engines to play with books but if rybka has problems against naum in B90 then common sense tells me that learning is going to tell it to play other lines.

Uri

User avatar
M ANSARI
Posts: 3500
Joined: Thu Mar 16, 2006 6:10 pm

Re: Is Naum 4 a serious challenge for Rybka

Post by M ANSARI » Sun Jan 04, 2009 2:13 pm

Rybka has no problems with B90 scoring almost 70% against N4 in B90. But if you put different books with different lines of B90 against each other then things are not predictable. You have to remember that 2 different books played against each using same Rybka engine have differed in some cases more than 300 ELO points. That is why to test engine performance you need to use same book for both sides ... otherwise the different strengths of books can pollute the final outcome.

swami
Posts: 6546
Joined: Thu Mar 09, 2006 3:21 am

Re: Is Naum 4 a serious challenge for Rybka

Post by swami » Sun Jan 04, 2009 2:28 pm

I wonder if the latest Rybka supports learning function the way Romichess and Learninglemming does? Or is it still in Vas's To-do list?

User avatar
M ANSARI
Posts: 3500
Joined: Thu Mar 16, 2006 6:10 pm

Re: Is Naum 4 a serious challenge for Rybka

Post by M ANSARI » Sun Jan 04, 2009 5:34 pm

I am not sure but I do see a book learning box in the CB GUI, however I never turn it on.

User avatar
Dr.Wael Deeb
Posts: 9773
Joined: Wed Mar 08, 2006 7:44 pm
Location: Amman,Jordan

Re: Is Naum 4 a serious challenge for Rybka

Post by Dr.Wael Deeb » Sun Jan 04, 2009 5:52 pm

swami wrote:I wonder if the latest Rybka supports learning function the way Romichess and Learninglemming does? Or is it still in Vas's To-do list?
Isn't the "Preserve hash" function in the Rybka UCI configuration file some kind of learning feature :!: :?:
_No one can hit as hard as life.But it ain’t about how hard you can hit.It’s about how hard you can get hit and keep moving forward.How much you can take and keep moving forward….

Jeroen
Posts: 501
Joined: Wed Mar 08, 2006 8:49 pm

Re: Is Naum 4 a serious challenge for Rybka

Post by Jeroen » Sun Jan 04, 2009 6:12 pm

You are seriously handicapping Rybka 3 in these matches:

Perfect15 is a much better book than RybkaII (which is already 2 years old, and probably more than 150 elo weaker).

Storm is a much better book than Perfect15, the same for Magnificent.

For equal conditions for both programs you could use one of my test suites, f.e. Noomen Testsuite 2008. You can download it here:

http://www.rybkachess.com/index.php?auswahl=Downloads

If you let both programs play the white side and the black side from each position, none of the two has a book advantage.

Post Reply