testing engines with fixed depth

[Togga]

do you get then other result? different ratinglist?

[Jim Walker]

I haven't tried it but I'm willing to bet you will get very different results because not everyone counts plies the same. Also different programs do not reach the same depth when given the same amount of time. Therefore making them play equal plies if it was possible would naturally cause different results.
Jim

[Togga]

i made one test:

fixed depth = 9

Philou 3.1.2 - RobboLito : 5,5/10 4-3-3 (00=10=1=11) 55% +35



timecontrol: 2min+2sec

Philou 3.1.2 - RobboLito : 0,5/10 0-9-1 (=000000000) 5% -512

[Tord Romstad]

I haven't tried it but I'm willing to bet you will get very different results because not everyone counts plies the same.

It's far worse than that: Counting plies just isn't possible, except for the length of a single line in the search tree. The search tree of a modern chess program is far too heterogeneous (because of all sorts of extensions, reductions, pruning and qsearch) to have a clearly defined search depth.

The term "search depth" is therefore a misnomer. The "depths" reported by chess programs are not really search depths at all, but merely iteration counters. For a single program, most lines of search will be a little deeper at iteration n+1 than at iteration n, but they won't all be exactly one ply deeper. In fact, for most modern programs, the average lengths of all lines at iteration n+1 and at iteration n will differ by far less than one ply. It would be better if we all stopped using the term "search depth" entirely, because it is misleading and only causes confusion. Let's start saying that "program X finds move Y during the Zth iteration" rather than "program X finds move Y at depth Z". Or better still, let's just state the time taken or number of nodes taken to find the move, as the iteration counter doesn't really tell us anything interesting anyway.

Testing engines with fixed depth isn't possible, because fixed depth simply doesn't exist. What Paul tests actually do is to put a limit on the number of iterations performed in the iterative deepening loop in the two programs. Because the difference between what is searched in two successive iterations is quite arbitrary and can vary greatly between different programs, such a test his is about as meaningful as limiting the number of times the program can poll for keyboard input during a search. Testing at a fixed number of iterations doesn't tell us which program is stronger, which program has the better evaluation, which program has the better search, which program is faster, or anything else worth knowing.

[Togga]

timecontrol 2min + 2sec:

RobboLito 0.085d2 w32 - Stockfish-1.6.3-32-ja : 8,5/10 7-0-3 (=1=11111=1) 85% +301


fixed depth=12 half moves:

RobboLito 0.085d2 w32 - Stockfish-1.6.3-32-ja : 5,5/10 4-3-3 (111==000=1) 55% +35




Philou the new, still slow, chess star??

when it gets the speed of robbolito

[Togga]

i am using the depth settings of arena 1.1.

i dont know what the chess engines do with this settings internal.

ps. : now stockfish tested.

[Florian]

i am using the depth settings of arena 1.1.

i dont know what the chess engines do with this settings internal.

ps. : now stockfish tested.

What you doing is ridiculous!
It is known that many engines show completely different the depth.

[Togga]

fixed depth= 12 half moves

Stockfish-1.6.3-32-ja - Glaurung22_win32_ja : 7,5/10 5-0-5 (1=1====111) 75% +191

[Togga]

wow, Florian your 22 post at my post??:) thanks very much:)

i dont think that : robbolito, philou, stockfish, glaurung show and use wrong depth.

Those really used the setting depth and did not go farther than the given depth: here depth 12 and 9.

ask the author of glaurung, stockfish, philou.

the engines are going wrong way (like rybka?) when these dont used the setting depth and dont show the true depth:(

[Florian]

wow, Florian your 22 post at my post??:) thanks very much:)

i dont think that : robbolito, philou, stockfish, glaurung show and use wrong depth.

Those really used the setting depth and did not go farther than the given depth: here depth 12 and 9.

ask the author of glaurung, stockfish, philou.

the engines are going wrong way (like rybka?) when these dont used the setting depth and dont show the true depth:(

Hi Paul,
That what you doing is good as amusement,it hasn't statistical worth!
You can never so determine the difference in strenght between the engines.
Regards,
Florian