M ANSARI wrote:I think things will be pretty clear soon ... I have been doing a 300 game 60 minute +1sec on 3 Quads for almost a week. I am using neutral 7 move book. I am testing N4 against both Rybka 3_default contempt and Rybka 3_0 contempt. So far with 70% of the games played it is clear that R3 is quite a bit stronger ... but it is also seems that R3_0_cont is scoring higher. On the one computer I have at work where I am typing this post this is the score so far
Let me say that the reason I started testing contempt was when I was going through the games that R3 lost ... chesswise it was obvious that R3 was pushing too hard and that in many games it would lose simply because it had overreached. N4 is simply too strong and once R3's initiative evaporated, N4 had enough technique and strength to punish Rybka for speculative play. In many lost games I could not really find an evaluation mistake (although in many there were). This is what led me to believe that contempt was hurting Rybka's performance and I have a feeling this hunch will turn out to be correct.
I will collect all 300 games once the tourney is finished and tabulate them again. Still this is a very good performance for N4 but to test R3 at LTC it seems it will perform better at 0 contempt. By the way I have also tested the same thing at 16_1 time controls and on 30_1 time controls ... the results seem pretty much the same. The only time normal Rybka performed as good as Rybka 0 contempt was when I used Perfect 15 book. But I don't think that was an accurate test since 198 of the 200 games turned out to be B90 (the other two games were B92). Neutral books give a better idea of performance of an engine. I also did a 1000 game 1_1 test on Octa 4ghz using differnt cont settings ... the 30 cont and 15 cont scored almost 90% with lower 0 cont and 8 cont and -8 cont scoring in the 80's%. So contempt has a positive effect on R3 at fast time controls and this seems to decline linearily as TC go higher.
Still not enough games, 2 sigma (95%) error bars are about 10% in your results, so things may change. I am surprised at the huge difference between contempt 15 and 0, as, if I understand it, it matters only when Rybka is between 0.00 and -0.15. Can Rybka's play change seriously, as you describe it, by a contempt of 15?
Kai
ps better use an opening set of positions (Noomen 30 maybe, so 60 games), switching black and white, with books my tests are wildly oscillating, and I don't know even how to estimate the error margins.
From my experience, you can usually get a good idea of what to expect after about 300 games ... especially if this is followed by several hundred games at different time controls. But really to me test games generally give an idea of what to expect ... and you can get a feel of the pluses and minuses by watching many games. I enjoy looking at test games and trying to figure out where they went wrong. I have a compilation of around 15 games where I have analyzed in detail the games and so I feel I have much better grasp of the weaknesses of Rybka and Naum 4 than before.
Another thing ... it takes forever to compile 300 games at long time controls. I have had 3 Quads running almost a week now 24/7 and figure I need another couple of days to finish the games. This is at 60 mintues + 1second time controls. Imagine at say 120 minutes. Still with computers, time is a constant that can be greately reduced by simply increasing the hardware. So a 30minute game on an Octa at 4 Ghz can easily surpass the quality of a 120 minute game on a dual core and maybe even some slower Quadcores.
If you let both programs play the white side and the black side from each position, none of the two has a book advantage.
Thanks for your comments
I wonder what book you recommend for correspondence games when part of them are in the opening stage.
I am not going to follow blindly a single book but books can certainly help to decide which moves to analyze and I wonder if the move with the bigger number in the fact colum is the move that is considered to be best based on the storm book.
I understood based on the result that storm is probably better than perfect15 but it clearly surprised me because I earlier did not know that there are free books that are supposed to be better than your book for rybka3 except the perfect15.ctg book
I wonder how people generate books that are better and better for long time control when they have no time to test a lot of moves not in blitz.(60 minutes per game is not blitz).
I have no interest in handicaping any of this two programmes. After my first 100 games, I realized that Perfect15 is much better than RybkaII. For this reason I gave Perfect15 to Rybka3 in the following 100 games. To my surprise Storm and Magnificient have also proven to be excellent books.
Thank you very much also for your link to your test suite. I think it will be an excellent tool for testing programmes. For obvious reasons other than Rybka.
I have decided to test again both programmes in another 100 games. It will be fair enough, IMO, to give one commercial book to one of the programmes and other free book to the other for 50 games and then change the books. All with the same quad 6700 hardware, settings, GUI and 60 minutes game.
Congratulations, Jeroen, for the exccellent work, after a lot of years, buiding top quality books.
Tomcass wrote:I am conscious that the books I use are decisive for the results I get. Most of games are B80 and B90. I will eliminate repeated games, although by now I have found only 2 of them repeated.
Obviously I agree with you that a neutral set of conditions would give a more reliable result. Can you tell me, please, how can I use one of these reliable sets?.
The books I am using are
In the first 100 games: RybkaII for R3 and Perfect15 for Naum4.
In games 101 to 150: Perfect15 for R3 and Storm for Naum4.
In games 151 to 200: Perfect15 for R3 and Magnificient for N4.
Since I think that my test will finish tonight, I give you the CUMULATIVE RESULTS after 192 games:
You will see now why I said that the victory of Naum4 over Rybka3 has been really IMPRESSIVE.
Regards from Barcelona.
Tom.
This is completely skewed as you are testing books as much as engines. In the end you can draw no conclusion this way and calling it a victory for Naum 4 is bullocks. My test so far shows an advantage of Rybka 3 over Naum 4. Due to the limited number of games I won't conclude how much.
It very much depends upon the opening lines you want to play. Basically the Playchess books are 75%-80% Najdorf, with other Sicilians and other replies to 1.e4 being a bit neglected.
Furthermore, these books are made of mostly (fast) Rybka-Rybka games and based on statistics, which is quite dangerous to follow blindly IMO. In correspondence chess I still value the influence of the strong human player, as I do think that the combination Rybka+human is stronger than Rybka alone. This view is supported by the fact that the Freestyle tournaments are won by centaurs and not 'engine only' participants.
In any case, my advice as always is: use common sense and don't follow computer suggestions blindly. It is the same with opening books.
The big problem with these Playchess books is the fact that your match is turning into an openingbook fight rather than an engine-engine fight. It is very likely that the engine with the (much) better book will win, even if the opponent is Rybka.
To rule out any book contest, you could use the neutral test suites, or a general book for both (both the same book), f.e. limited to 6, 8 or 10 moves. As the Playchess books are rather unbalanced (a lot of Sicilian Najdorfs), I would prefer a more neutral book like the Fritz books, limited to 6, 8 or 10 moves.
I understood based on the result that storm is probably better than perfect15 but it clearly surprised me because I earlier did not know that there are free books that are supposed to be better than your book for rybka3 except the perfect15.ctg book
There is a simple rule for Playchess books: the younger it is, the better it will score against older books.
But don't forget that these books are mostly build to score heavily in 3' blitz matches between Rybka and Rybka.
R3.ctg is already half a year old, which is really old in Playchess terms . But Playchess is not the only part of the chess world, I still think there are many ideas in R3.ctg that are still not well discovered, at least not at Playchess.
The big problem with these Playchess books is the fact that your match is turning into an openingbook fight rather than an engine-engine fight. It is very likely that the engine with the (much) better book will win, even if the opponent is Rybka.
To rule out any book contest, you could use the neutral test suites, or a general book for both (both the same book), f.e. limited to 6, 8 or 10 moves. As the Playchess books are rather unbalanced (a lot of Sicilian Najdorfs), I would prefer a more neutral book like the Fritz books, limited to 6, 8 or 10 moves.
Kind regards, Jeroen
Still, suites are much better IMO, because you can play both white/black on the same position, and you can choose some representative repertoire, as your Noomen 30. Also, I don't want to test the book learning.
The big problem with these Playchess books is the fact that your match is turning into an openingbook fight rather than an engine-engine fight. It is very likely that the engine with the (much) better book will win, even if the opponent is Rybka.
To rule out any book contest, you could use the neutral test suites, or a general book for both (both the same book), f.e. limited to 6, 8 or 10 moves. As the Playchess books are rather unbalanced (a lot of Sicilian Najdorfs), I would prefer a more neutral book like the Fritz books, limited to 6, 8 or 10 moves.
Kind regards, Jeroen
Still, suites are much better IMO, because you can play both white/black on the same position, and you can choose some representative repertoire, as your Noomen 30. Also, I don't want to test the book learning.
Regards,
Kai
Under the ChessBase GUI you can use one opening book for both engines playing both as white and black the same opening line....this is even better than using test suits....
_No one can hit as hard as life.But it ain’t about how hard you can hit.It’s about how hard you can get hit and keep moving forward.How much you can take and keep moving forward….
. But Playchess is not the only part of the chess world, I still think there are many ideas in R3.ctg that are still not well discovered, at least not at Playchess.