comparing different Glaurung 2.0.1 compiles

[Murx]

Hello there

I am testing different compiles of Glaurung 2.0.1 64 bit.
I got 5 different compiles all 64 bit versions:
Glaurung 2.0.1 JA from Jim Ablett (from superchessengine.com)
Glaurung 2.0.1 std not shure but think from Bryan Hofmann
Glaurung 2.0.1 SSE not shure but think from Bryan Hofmann
Glaurung 2.0.1 SSE2 not shure but think from Bryan Hofmann

All this versions I test with default settings AND with variations in the settings. I use the chessbase GUI and the Nunn2 testset(50 games).
As I need verry much games, I decided to run a verry short time control for the first runs. So I hope to find out a close circle of Glaurung versions for tests with different time controls.
Sparring partner for the first runs is Deep Fritz 10.1.
450 games played yet.
at that time Jim Abletts compile with default settings is the leader.
Followed by JA compile with "thread depth 5" setting.

1)Glaurung 2.0.1 ja def
2)Glaurung 2.0.1 ja td5
3)Glaurung 2.0.1 sse2 def
4)Glaurung 2.0.1 sse td5
5)Glaurung 2.0.1 sse def
6)Glaurung 2.0.1std def
7)Glaurung 2.0.1 std td5

Test is continued with td 5 and sel 7 settings. Seems a verry good combination.

Results follow.

Regards, Clemens Keck

[ernest]

As I need verry much games,

Isn't it much easier/simpler to compare the times obtained on infinite analysis of a few selected positions
(the PVs should all be the same for a given position, same number of nodes... only different times)

[Murx]

Hello Ernest

I do somthing like that. Nunn 2 testset is 25 possitions to start a game with. So I get results by the played games, not by test positions.
I think its nearly the same. I like the active way in real games.
Kind regards, Clemens

btw. : JA compile with td5 and sel 7 plays verry promising. Maybe it is the best right now.

[Mike S.]

But I think Ernest is right... for finding a pure speed difference of compiles based on the same source, playing test games is just a waste of time. Even worse, I think results from games may be misleading, because statistical uncertainty may result in the fastest engine NOT being on top.

For example, if you would play matches between two comps with two identical engines each, but one comp being 5% faster, the slower computer will probably win a couple of matches. Because +5% speed will often not be enough to compensate for a bad move.

I mean, if you simply compare time for depth X in some positions (of different nature), what sense would it make to consider that the fastest in this test in not the best at the same time, from a set of engines based on the identical source code? Really, games make no sense here to test that.

[Norm Pollock]

But I think Ernest is right... for finding a pure speed difference of compiles based on the same source, playing test games is just a waste of time. Even worse, I think results from games may be misleading, because statistical uncertainty may result in the fastest engine NOT being on top.

For example, if you would play matches between two comps with two identical engines each, but one comp being 5% faster, the slower computer will probably win a couple of matches. Because +5% speed will often not be enough to compensate for a bad move.

I mean, if you simply compare time for depth X in some positions (of different nature), what sense would it make to consider that the fastest in this test in not the best at the same time, from a set of engines based on the identical source code? Really, games make no sense here to test that.

I concur.

[Murx]

Hello Mike

I want no theoretical discussions, but I am not testing any speed differences! Speed is not the same as performance. I try to find the best glaurung on my machine. My testplatform is 3 Ghz quadcore machine, so I thing +5% speed makes no diff.
I think game results (always the same start pos.) is a good way to see if engine compiles and/or settings improve playing strength.

best regards, Clemens

[Vempele]

Hello Mike

I want no theoretical discussions, but I am not testing any speed differences! Speed is not the same as performance. I try to find the best glaurung on my machine. My testplatform is 3 Ghz quadcore machine, so I thing +5% speed makes no diff.
I think game results (always the same start pos.) is a good way to see if engine compiles and/or settings improve playing strength.

best regards, Clemens

Then why didn't you just pick one build at random and test settings on it? The only way they differ _at all_ from each other is speed.

[pedrox]

Hello Ernest

I do somthing like that. Nunn 2 testset is 25 possitions to start a game with. So I get results by the played games, not by test positions.
I think its nearly the same. I like the active way in real games.
Kind regards, Clemens

btw. : JA compile with td5 and sel 7 plays verry promising. Maybe it is the best right now.

Hello Clemens,

Everything said by Ernest is true.

Maybe your way of thinking is equal to that which I had previously, if the starting positions are the same then the same games, but this it is not true, in a computer because the operating system (antivirus...) can be lost in milliseconds that makes in equal positions the responses are different. My thought is that with longer times this happen less, but this is a thought of mine untested.

The forum programming has also been a discussion about the number of games needed to determine if a version is better than another, and as Bob Hyatt said the number of games is approximately 2500 per engine, with 3000 games you may have an error more than 10 points ELO, if you want you can see this in the list of CCRL blitz. For your work is valid maybe you have to play that number of games.

Pedro

[pedrox]

Hello Mike

I want no theoretical discussions, but I am not testing any speed differences! Speed is not the same as performance. I try to find the best glaurung on my machine. My testplatform is 3 Ghz quadcore machine, so I thing +5% speed makes no diff.
I think game results (always the same start pos.) is a good way to see if engine compiles and/or settings improve playing strength.

best regards, Clemens

Jim Ablett compiled with the compiler Intel v.10 and Bryan Hofmann using Microsoft Visual studio 2005 (or superior), I think that prior to version 10 of Intel compiler Microsoft was better for both Intel and AMD computers, perhaps the version 10 of Intel is now better for computers carrying an Intel (or AMD also or at least to the most modern)

Pedro