Fixed depth - Komodo 10.4 vs Stockfish 8

[fastgm]

.
Conditions: Each Engine 128 MB Hash, default settings, 3000 games per match, 1500 opening positions, no ponder, no learning, no tablebases, no draw and resign rules
Elo rating from the point of view of Komodo, Elostat Start Elo = 3000

GUI: Cutechess-Cli
Linux - Ubuntu Server 16.04 LTS
Intel E7-8880v3 @ 2.3 GHz - Amazon AWS x1.32xlarge - 64 CPUs - 4 NUMA nodes - Hyperthreading disabled
Stockfish 8 (profile-build ARCH=x86-64-bmi2)

D = depth, T = thread(s)





Download PDF: http://www.fastgm.de/schach/Fixed_Depth ... fish_8.pdf

[Waschbaer]

I don't see the reason why to test different engines on fixed depth.

An engine has to "understand" the position, it can be done by

1) use lot of knowledge by not searching as deep as other engines
2) search fast and deep

Each engine has it's own style of balancing knowledge and depth

Or do you compare the power of a truck and a ferrari by a given number for RPM?

Sure not

[yurikvelo]

ferrari by a given number for RPM?

Sure not

Comparing ferrari and lamborghini RPM, but not crankshaft RPM.
Some arbitrary (unknown) axle RPM, somewhere in gear box

[fastgm]

An engine has to "understand" the position, it can be done by

1) use lot of knowledge by not searching as deep as other engines
2) search fast and deep

Each engine has it's own style of balancing knowledge and depth

You're absolutly right.

Or do you compare the power of a truck and a ferrari by a given number for RPM?

Sure not

Of course not, but when comparing Komodo and Stockfish the difference is not as big as with a truck and a ferrari.

For example:
With 1 thread and from depth 10 to depth 20, they are approximately equal, Komodo is here even better than Stockfish.
But with more threads the situation changes in favor of Stockfish. The more threads, the bigger the difference. The difference is also more pronounced with lower search depths. In case of higher search depths, Komodo regains again.
Perhaps this test is of interest for a developer and programmer of an engine.

[shrapnel]

Its OK if you are comparing Komodo 10.4 to SF8.
But don't compare it to latest Stockfish development version which is simply LIGHT YEARS ahead of Komodo !....no matter WHAT Depth you use.
Don't even go there !
Play in the Shallows, don't go into the Deep Water !

[mjlef]

An engine has to "understand" the position, it can be done by

1) use lot of knowledge by not searching as deep as other engines
2) search fast and deep

Each engine has it's own style of balancing knowledge and depth

You're absolutly right.

Or do you compare the power of a truck and a ferrari by a given number for RPM?

Sure not

Of course not, but when comparing Komodo and Stockfish the difference is not as big as with a truck and a ferrari.

For example:
With 1 thread and from depth 10 to depth 20, they are approximately equal, Komodo is here even better than Stockfish.
But with more threads the situation changes in favor of Stockfish. The more threads, the bigger the difference. The difference is also more pronounced with lower search depths. In case of higher search depths, Komodo regains again.
Perhaps this test is of interest for a developer and programmer of an engine.

Once again, a very interesting experiment. Stockfish uses variable search depths in its Lazy SMP implementation, which should include the quality of many moves put into the hash table and used by Lazy SMP. But it is interesting that this benefit seems to decline more with search depth than I would have expected. Larry ans I have noticed the one thread hump" too against Stockfish that dwindles with depth, so you are confirming what we have seen. Thanks Andreas.