Rybka's knowledge

[Mike S.]

Thanks for the explanations, which are plausible and much simpler than I expected actually. Yesterday I seemed to search a miracle where there is none.

There must be a typical depth (for the moment, ignoring problems of depth definition and -display) where the general strength gain from even more depth is very small. But this would be only a rough average because engines are so different.

I think it is true that the standard Rybka 2.3.2a is somewhat slower than many other good engines... I didn't test all, but in my tactics test with 24 combos where knowledge should nearly not matter (or not at all), it did not achieve a top result. Good, but by top standards mediocre. For comparison, it solved 17/24 like Fritz 7 did, and the test rating including the time consumption is only slightly better than Naum 2.0's which solved 19.

That suggests that the default Rybka does only the "necessary" for tactics which is required for the top rank, and almost all the superiority is caused by the positional play due to better knowledge / quality of the evaluation. (But OTOH, Rybka WinFinder 2.2 is my new number one in that test! )

The endgame may have a small share too, although I think Rybka is not famous for it, except one very convincing test suite result in the E-E-T.

Btw. the Quicktest gave me the impression that in the sector of combinative speed, there wasn't much progress in recent years, if any. In my results, #2 is Hiarcs 9 and #3 is a simple The King 3.23 setting. Maybe newer Hiarcs versions, or Zappa, are faster than the WinFinder but I don't have them.

Another quick combinator is Spike 1.2. Only WinFinder 2.2, H9 and Spike 1.2 solved 21/24.

http://members.aon.at/computerschach/quick/quickxls.zip

[Michael Sherwin]

There must be a typical depth (for the moment, ignoring problems of depth definition and -display) where the general strength gain from even more depth is very small. But this would be only a rough average because engines are so different.

We are nowhere near the point that deeper searches will not pay big dividends. Tactical sequences are always just over the horizon. If truth be told, all chess programs are stupid and slow, compared to the best programs twenty years from now. People just have not yet figured out how to write a really good chess program. When they do, Rybka will look like a really weak example of ancient chess programming ideas!

[Uri Blass]

I think that test suites are misleading here and if a program is better in playing and weaker in test suites it does not mean that it has a better evaluation.

A better program with the same evaluation may be weaker in test suites because of improvement in the search that simply reduce sacrifices that usually are not good in normal games but often good in test suites.

Uri

[Mike S.]

I think that test suites are misleading here and if a program is better in playing and weaker in test suites it does not mean that it has a better evaluation..

In theory, you may be right in some cases (where I would expect smaller differences). In practise, and speaking about a totally superior engine when games are played, I think "better knowledge/eval" is the more realistic assumption.

A better program with the same evaluation may be weaker in test suites because of improvement in the search that simply reduce sacrifices that usually are not good in normal games but often good in test suites.

I have mentioned WinFinder, where obviously the reverse effect takes place. That makes the engine a very good choice to search for (suspected) deep tactical wins, in analysis. - Nevertheless, there are some counter-examples where the default Rybka solved faster, but in almost all of them the side to move was in the defender's position, somehow. I think that illustrates a big asymmetry in WinFinder's logic which it probably has, causing that combinative speed-up - but mostly for the winning side to move, only.

(I did ignore the "detail" that WinFinder is based on version 2.2 but I compared it with 2.3.2a )

This engine isn't intended for games, but I found one rating for the x64/2 CPU version in the CEGT blitz list. The rating is ~200 Elo lower than of the normal 2.2 x64/2 CPU version.

[Vempele]

A better program with the same evaluation may be weaker in test suites because of improvement in the search that simply reduce sacrifices that usually are not good in normal games but often good in test suites.

I have mentioned WinFinder, where obviously the reverse effect takes place. That makes the engine a very good choice to search for (suspected) deep tactical wins, in analysis.

Why do you believe Winfinder is a better program than Rybka 2.2? They use the same evaluation, and Winfinder's search is obviously more tactically oriented (for the side to move, anyway).

[Michael Sherwin]

I think that test suites are misleading here and if a program is better in playing and weaker in test suites it does not mean that it has a better evaluation.

A better program with the same evaluation may be weaker in test suites because of improvement in the search that simply reduce sacrifices that usually are not good in normal games but often good in test suites.

Uri

I only look at test suites very rarely and then only out of curiosity to see if there has been a noticeable change.

The last release version of RomiChess finds 83 of 100 in a certain test suite. I was very shocked to find earlier today that RomiChess is now finding 96 of 100. My search has not deepened much if at all. So now I am wondering why the increased performance.

[Uri Blass]

I think that test suites are misleading here and if a program is better in playing and weaker in test suites it does not mean that it has a better evaluation..

In theory, you may be right in some cases (where I would expect smaller differences). In practise, and speaking about a totally superior engine when games are played, I think "better knowledge/eval" is the more realistic assumption.

A better program with the same evaluation may be weaker in test suites because of improvement in the search that simply reduce sacrifices that usually are not good in normal games but often good in test suites.

I have mentioned WinFinder, where obviously the reverse effect takes place. That makes the engine a very good choice to search for (suspected) deep tactical wins, in analysis. - Nevertheless, there are some counter-examples where the default Rybka solved faster, but in almost all of them the side to move was in the defender's position, somehow. I think that illustrates a big asymmetry in WinFinder's logic which it probably has, causing that combinative speed-up - but mostly for the winning side to move, only.

(I did ignore the "detail" that WinFinder is based on version 2.2 but I compared it with 2.3.2a )

This engine isn't intended for games, but I found one rating for the x64/2 CPU version in the CEGT blitz list. The rating is ~200 Elo lower than of the normal 2.2 x64/2 CPU version.

My opinion is based on practical experience as a programmer.

I know that there were search changes that made movei weaker in test suites and stronger in games.

Movei use pruning based on evaluation and other factors.

This idea may cause movei to discover sacrifices few plies later but in games these sacrifices usually do not happen and movei get an additional ply to its search depth.

Uri

[Mike S.]

Why do you believe Winfinder is a better program than Rybka 2.2?

Why do you think I would believe that? I don't. I wrote: "That makes the engine a very good choice to search for (suspected) deep tactical wins, in analysis." For that specific purpose it was designed for, it works great. I didn't write that it is a better choice than 2.x for anything else... I was even quoting a -200 Elo CEGT rating list result myself, so it is really a miracle to me why you assume I'd think that it is a better program than 2.2.

[swami]

In other words Larry admit that there are chess programs with more chess knowledge than rybka2.3.2a

I was surprised to read it because I expected the rybka team not to admit that there are programs with better knowledge than rybka.

I remember the times when vasik claimed that he believes that the main advantage of rybka beta is her evaluation(he did not deny that rybka has a good search but claimed that the main advantage relative to top programs is the evaluation).

Hi Uri,

It seems that this kind of post is more appropriate for Programming forum, I don't have any problem with it.

Just general observation.Since most of us in general forum do not give a damn about what evaluation, knowledge or search in engine is.

[Matthias Gemuh]

As you keep adding more and more time, the gap between the number of plys searched keeps dropping. Agree ? That is if they were going 10 plys and 13 plys each and you quadruple the time, the engines may go
15 plys and 17 plys. This a great gain for the shalloer knowledge-based engine.
As you linearly increase time, it becomes exponentially tougher to search an extra ply, so the shallower engine gradually catches up.


RDChess outsearches my BigLion consistently at blitz, but add more time and RDChess gets crucified consistently.

Best,
Matthias.