Question to Larry Kaufman about Rybka

[Uri Blass]

Definitely not. If I did, either the ideas would already be in Komodo and it would be as strong as Robbo, or else I wouldn't have worked on Komodo at all as a matter of principle. But of course I do understand some of ideas that are now talked about on this forum.

Thx for sincere answer. I also don't know what makes Rybka search so efficient since I haven't looked at Rybka source code, and didn't spend time trying to disassemble it. I can only assume what it is (there I agree with Uri's assumption).
However, my firm belief is that publishing of Ippo sources doesn't brings us any closer to the key of Rybka strength.

My guess is that Ippo probably use similiar ideas to rybka

I clearly verified that rybka at depth -2(that is the minimal depth that she can use) can find the capture a4xb3 with positive score

uci
position fen 1rb1k2r/5pp1/2p1pb1p/2Q4P/p3qP2/1P2B3/P1P3P1/1K1R1B1R b k - 0 19

go depth -2


uci
...
uciok
position fen 1rb1k2r/5pp1/2p1pb1p/2Q4P/p3qP2/1P2B3/P1P3P1/1K1R1B1R b k - 0 19
go depth -2
info depth 1 score cp -574 time 1 nodes 0 nps 0 pv e4e3
info depth 1 score cp 48 time 1 nodes 1 nps 1024 pv a4b3
info depth -2 time 1 nodes 1 nps 1024
bestmove a4b3 ponder a2b3


Rybka3 see 0.48 for a4b3 at depth -2 and 2.42 pawns for black at depth -1

Uri

[lkaufman]

It has a better tuned eval function and superior qsearch.

A better eval would show up more at higher depths, not at one or two ply where the results are dominated by tactics. A superior qsearch is a part of the answer, but only a small part, as our tests indicate that even a very slow, inclusive qsearch would only gain a tiny fraction of the missing 200 Elo even if it were for free.

[Uri Blass]

It has a better tuned eval function and superior qsearch.

A better eval would show up more at higher depths, not at one or two ply where the results are dominated by tactics. A superior qsearch is a part of the answer, but only a small part, as our tests indicate that even a very slow, inclusive qsearch would only gain a tiny fraction of the missing 200 Elo even if it were for free.

Maybe your inclusive qsearch is not good enough.

If Komodo with inclusive qsearch is losing against Robbolito at 1 plies or 2 plies then you should look at the games to understand why does it lose.

Assuming that it is not because of some better evaluation then it has to be because of better qsearch or because of more extensions.

Note that better evaluation can help significantly also at 1 ply assuming that the evaluation see tactical ideas(and it is possible to see things like trapped piece or forks but I do not think that it is the advantage of robbolito)

Uri

[Don]

I also know that the pruning in Robbo, Stockfish, and Komodo are not so drastically different;

Well, perhaps Komodo is similar to Stockfish, but I can assure you that search in Robbo is very different from SF. There are a good number of fundamental differences that I won't enumerate (in case ask Don that has for sure already done this for himself) but any average programmer that has spent few hours on Robbo could see them without a big effort.

We spent a significant amount of time looking for ideas from the source code of stockfish 1.4 and I even implemented a version of doch that had the stockfish search as an experiment. The program ended up significantly weaker and we could not explain why, as this version of stockfish was clearly stronger than doch at the time.

As it turns out, there were not that many differences anyway (our code was remarkably similar), but there were some things that were nothing like doch such as the "history pruning" idea which stockfish did on the last several ply. I was excited when I saw that, thinking that the idea might be a big strength boost. I remember Tord explaining it in one of the forum posts a few months back. But it weakened our program.

In the end, in order to restore the strength of doch I removed almost every difference from stockfish and thus doch reverted mostly to it's old form! To me this was a remarkable example of how not every good idea works the same (of course I already knew this.)

We did benefit significantly however from the using the <pce><to> indexing for history - which is only used for move ordering in doch. Stockfish makes much greater use of this information. This was a remarkable improvement which I can attribute to stockfish, even though it's hardly an original idea. Also, the idea of refreshing the tables periodically (by reducing each entry) I took from stockfish.

This is pretty much the only idea kept from stockfish that had any measurable impact on the strength of doch and since then I have improved on this too.

Doch's search is certainly not a subset of the stockfish search either. There are some ideas that we consider secret that is not in stockfish that accounts for a surprising improvement in strength for doch. Somehow I feel that if they were implemented in stockfish it wouldn't help!

So what happened is that we went through this phase of implementing a very stockfish-like search, then backing out almost all the differences, then adding back our own stuff and the program was much weaker for many weeks while performing this failed experiment.

In the end it's rather hard to compare these 2 searches and say that they are "similar" as they are both pretty much classic PVS search which makes them similar, but they each have some ideas that look very much different.

By the way, I implemented the hash table singular extension in doch when it was discussed on this forum. When I looked at how stockfish implemented it later in stockfish 1.6, it was almost identical - with minor differences.

Every good program out there is basically built on a zillion ideas that are in the "public domain" with a little "secret sauce" thrown in. Once a good idea is revealed and becomes public, it's ludicrous not to use it if it works for you.

The evaluation of doch is my own original creation with the help of Larry of course. It's much like our work of over a decade ago, and I consider Rybka just a better version of our (Larry and I) work on evaluation.

The idea of copying other peoples programs is very distasteful to me. I don't enjoy looking at the source code of other programs even when it's open - it's not the fun part of chess programming. I don't think it's immoral or anything like that, I just do it reluctantly. I would rather spend most of my time working on my own innovations - and just being a copycat is not going to push you ahead - one must blaze new trails. Unfortunately, like in any other feel one must also "keep up" with the latest ideas.

but no twiddling with parameters or details will make one program a hundred Elo stronger.

I don't agree here: Sf 1.5.1 is more then 100 ELO stronger then Glaurung just by means of with what you call " twiddling with parameters or details". The difference between a strong and a weak engine is at 90% _only_ in the details, but in a chess engine there are hundreds, thousands of details that at the end of the day make a big difference. I can state this with confidence because I have seen it directly with the Galurung <-> Stockfish case.

In short, why are there no programs that search like Robbo but have totally unrelated evaluation functions?

In chess engines copy and paste does not work. Robbo is _so_ different from the open sources engines known up to few months ago that it will take years, not months, to completely assimilate the underlying ideas by this community in what will be considered common knwoledge, for instance what today could be null move search or king safety.

[mcostalba]

The idea of copying other peoples programs is very distasteful to me. I don't enjoy looking at the source code of other programs even when it's open - it's not the fun part of chess programming. I don't think it's immoral or anything like that, I just do it reluctantly. I would rather spend most of my time working on my own innovations - and just being a copycat is not going to push you ahead - one must blaze new trails. Unfortunately, like in any other feel one must also "keep up" with the latest ideas.

With all the greatest respect, I think you realize that the above statement from someone who wants to go commercial sounds strange. Not because of you, it is just a general consideration.

When it is your job and you get money from producing the best possible engine then you _should_ try every possible _legal_ way to improve what is not only your engine, but becomes your company and your source of revenues. This is something everybody _accepts_ from a commercial developer and there is no shame in this.

If you do or do not look at other sources is something personal, something that is about your personal opinions, but trying to advertise this in public is like hoping to have the cake and eat it too.

I think it is important to accept the _role_ that someone chooses for himself, with coherence and with the humility of avoiding ethical considerations when not required.

This is a difficult concept to express in what is not my language, if something is offending for you I apologize in advance, this is not the intention, it is just an incorrect use of English from my own.

[lkaufman]

Maybe your inclusive qsearch is not good enough.

If Komodo with inclusive qsearch is losing against Robbolito at 1 plies or 2 plies then you should look at the games to understand why does it lose.

Assuming that it is not because of some better evaluation then it has to be because of better qsearch or because of more extensions.

Note that better evaluation can help significantly also at 1 ply assuming that the evaluation see tactical ideas(and it is possible to see things like trapped piece or forks but I do not think that it is the advantage of robbolito)

Uri

My impression is that the reason Robbolito beats Komodo and all other "normal" programs at one ply so badly is that it includes many positional moves in the qsearch, which raises the positional standard of play greatly without helping so much in tactics. The question is how Robbo does this without a major slowdown.

[lkaufman]

[quote So does anyone have a clue as to why Robbo is so much stronger than standard engines at 1 or 2 ply but not at 5 ply?

"Perhaps because is not at 1 or 2 ply this is only for the other engine but not for Robbo.

hint: how do you take in account extensions ?....ask Don for the answer [/quote]"

Certainly we must consider extensions. However normal extensions like check or capture extensions or Stockfish's mate threat extension are either in other programs or are only worth a handful of Elo points on a one ply search. Robbolito wins by a huge amount at one ply, but still not by enough to be worth a full extra ply. So if the explanation is an extension, it must be one that kicks in in most or all positions, such as simply extending the PV by a move always. But I don't think this is what Robbo does, nor do I think it is a good algorithm.

Since you seem to know what extension causes this behavior, but apparently do not use it in Stockfish, is this just another example of something that works well in one chess program but not in another, of which we have found numerous examples with Komodo?

[Uri Blass]

Maybe your inclusive qsearch is not good enough.

If Komodo with inclusive qsearch is losing against Robbolito at 1 plies or 2 plies then you should look at the games to understand why does it lose.

Assuming that it is not because of some better evaluation then it has to be because of better qsearch or because of more extensions.

Note that better evaluation can help significantly also at 1 ply assuming that the evaluation see tactical ideas(and it is possible to see things like trapped piece or forks but I do not think that it is the advantage of robbolito)

Uri

My impression is that the reason Robbolito beats Komodo and all other "normal" programs at one ply so badly is that it includes many positional moves in the qsearch, which raises the positional standard of play greatly without helping so much in tactics. The question is how Robbo does this without a major slowdown.

I do not know but it seems that robbolito can smell forks at depth 1
so I guess that the reason is combination of evaluation that can smell tactical things and extensions.

Here is an example that I composed

[D]1k6/6p1/8/b1r5/8/P1p5/1PK3P1/2R5 w - - 0 1

RobboLito 0.085g3 w32 - so k
1k6/6p1/8/b1r5/8/P1p5/1PK3P1/2R5 w - - 0 1

Analysis by RobboLito 0.085g3 w32:

1.b2xc3 Ba5xc3
-+ (-2.23) Depth: 1/8 00:00:00
1.g2-g4 c3xb2+ 2.Kc2xb2 Rc5xc1 3.Kb2xc1
-+ (-2.22) Depth: 1/9 00:00:00
1.Kc2-b1 Kb8-c7
-+ (-2.19) Depth: 1/9 00:00:00
1.b2-b4
³ (-0.64) Depth: 1/9 00:00:00
1.b2-b4 Rc5-h5 2.b4xa5 Rh5xa5
² (0.46) Depth: 2/11 00:00:00


You can see a tactical jump for b4 at depth 1 and the question is how does robbolito get it.

Note that it does not see the full evaluation that it see at depth 2 so maybe the evaluation has tactical motives.

Another example

New game - RobboLito 0.085g3 w32
[D]1kq5/2pp4/6p1/b1r5/8/P1p5/1PK3PP/2R4Q w - - 0 1

Analysis by RobboLito 0.085g3 w32:

1.b2xc3 Rc5xc3+ 2.Kc2-b2 Qc8-b7+ 3.Kb2-a2 Rc3xc1 4.Qh1xc1 Qb7xg2+ 5.Qc1-b2+ Qg2xb2+ 6.Ka2xb2
-+ (-4.55) Depth: 1/2 00:00:00
1.h2-h4 c3xb2+ 2.Kc2xb2 Qc8-b7+ 3.Kb2-a2 Rc5xc1 4.Qh1xc1 Qb7xg2+ 5.Ka2-b3
-+ (-4.35) Depth: 1/12 00:00:00
1.g2-g4 Qc8-a6 2.Kc2-b1 Qa6-d3+ 3.Kb1-a1 c3xb2+ 4.Ka1xb2
-+ (-4.32) Depth: 1/12 00:00:00
1.g2-g3 c3xb2+ 2.Kc2xb2 Qc8-h8+ 3.Kb2-b1
-+ (-3.99) Depth: 1/12 00:00:00
1.Kc2-b1 Qc8-b7
-+ (-3.68) Depth: 1/12 00:00:00
1.b2-b4
-+ (-3.11) Depth: 1/12 00:00:00
1.b2-b4 Ba5xb4 2.a3xb4
µ (-1.12) Depth: 2/13 00:00:00


You can see evaluation jump at depth 1

There may also be extensions of threats at depth 2 that are not threat against the king and the pv at depth 2 in the following position has 3 plies

New game, 5'/40+5'/40+5'/40
[D]1kq5/2pp4/6p1/b1r5/8/P1p5/1PK3P1/2R4Q w - - 0 1

Analysis by RobboLito 0.085g3 w32:

1.b2xc3 Rc5xc3+ 2.Kc2-b2 Qc8-b7+ 3.Kb2-a2 Rc3xc1 4.Qh1xc1 Qb7xg2+ 5.Ka2-b3
-+ (-5.17) Depth: 1/12 00:00:00
1.g2-g4 d7-d5
-+ (-4.50) Depth: 1/12 00:00:00
1.Kc2-b1 Rc5-b5
-+ (-4.42) Depth: 1/12 00:00:00
1.b2-b4
-+ (-3.84) Depth: 1/12 00:00:00
1.b2-b4 Rc5-h5 2.Qh1-e1
-+ (-4.08) Depth: 2/12 00:00:00

I do not see the same extension at depth 1 in this case

New game - RobboLito 0.085g3 w32
[D]1kq5/2pp4/6p1/b1r5/1P6/P1p5/2K3P1/2R4Q b - - 0 1

Analysis by RobboLito 0.085g3 w32:

1...Ba5xb4 2.a3xb4
-+ (-1.85) Depth: 1/3 00:00:00
1...Rc5-h5
-+ (-4.22) Depth: 1/8 00:00:00
1...Rc5-h5 2.Qh1-e1
-+ (-4.08) Depth: 2/8 00:00:00

[Uri Blass]

I can add that when I test the original position with rybka3 it also show a jump for b4 at depth=-2 and my conclusion is that probably Vas did not tell larry about the tactical parts of rybka3's evaluation.

Edit:Another option is that rybka does some selective search and does not trust the score of the selective search so it does some type of average between the score and the score after selective search and I guess that people can find what happens by looking at the code.

uci
id name Rybka 3 1-cpu 32-bit
id author Vasik Rajlich, Larry Kaufman
option name UCI_Query type button
option name Hash type spin min 2 max 4096 default 32
option name Max CPUs type spin min 1 max 1 default 1
option name NalimovPath type string default <empty>
option name NalimovCache type spin min 1 max 256 default 1
option name NalimovUsage type combo default Rarely var Frequently var Normally v
ar Rarely var Never
option name Preserve Analysis type check default false
option name Clear Hash type button
option name Saved Hash File type string default <empty>
option name Save Hash type button
option name Load Hash type button
option name Persistent Hash Enabled type check default false
option name Persistent Hash File type string default <empty>
option name Persistent Hash Play Depth type spin default 64 min 8 max 64
option name Persistent Hash Write Depth type spin default 10 min 8 max 64
option name Persistent Hash Size type spin default 16 min 8 max 65536
option name Persistent Hash Reset type button
option name Persistent Hash Resize type button
option name Persistent Hash Merge File type string default <empty>
option name Persistent Hash Do Merge type button
option name Ponder type check default false
option name MultiPV type spin default 1 min 1 max 100
option name MultiPV_cp type spin default 32768 min 0 max 32768
option name Contempt Play type spin default 15 min -250 max 250
option name Contempt Analyze type spin default 0 min -250 max 250
option name UCI_LimitStrength type check default false
option name UCI_Elo type spin default 1200 min 1200 max 2400
option name UCI_EngineAbout type string default www.rybkachess.com
option name Time Buffer type spin default 0 min 0 max 900
uciok
position fen 1k6/6p1/8/b1r5/8/P1p5/1PK3P1/2R5 w - - 0 1
go depth -2
info depth 1 score cp -193 time 1 nodes 4 nps 4096 pv b2c3
info depth 1 score cp -53 time 1 nodes 5 nps 5120 pv b2b4
info depth -2 time 17 nodes 7 nps 421
bestmove b2b4 ponder a5b4

[metax]

Another option is that rybka does some selective search and does not trust the score of the selective search so it does some type of average between the score and the score after selective search and I guess that people can find what happens by looking at the code.

This idea seems very strange...