TCEC resurrection - need to decide between ponder ON or OFF

[Martin Thoresen]

Very glad TCEC is back! I'd prefer ponder off, but why not run some matches/tournaments with ponder on as well once in a while? That way, there will be something for everyone.

Regards,
CL

Thanks Carl.

Your suggestion is perfectly valid, but I want to keep a coherent setup so that people know what to expect. Would be too much micro-managing to have two different configurations and making sure people know which one the current tournament is using.

[Martin Thoresen]

The computer that is going to be used is an Intel Core i7 3770k @ 4400 MHz.

What happened to your other machine Martin ?

I've sold it off as parts because I was not satisfied with the stability.

My plan though is to get a more powerful computer sometime in 2013 which then can be used for the new TCEC.

But my focus now is stability and having to spend as little time on this as possible. Too much management was one of the major factors that got me to shut down TCEC in the first place.

[Martin Thoresen]

That would surely be the best solution if no much use is done of the PC, however in the case he wants to use some other application that requires CPU use not just limited to little spikes but semi-permanent, using 4 cores with HT can produce some unpredictability as Windows will switch physical cores around (and probably set some threads to logical processes instead), especially if the engines are set with lower priority. Using affinities will prevent this, but in that case it will be just like using the full 4 cores without HT, and could produce the same problems that would arise in that case.

So it all depends on what use is made of the PC. If it is only surfing etc. then 4 cores/HT would probably be the best solution, but if instead a more resourceful use of the PC is done (as an example using Photoshop or similar applications that require CPU time always) then it's better to use 3 cores, in that way you are sure that in any case physical cores are used by the engines.

Fabio, spot on. That's exactly why I am limiting the number of cores to 3 physical instead of all 4/HT.

[IGarcia]

That would surely be the best solution if no much use is done of the PC, however in the case he wants to use some other application that requires CPU use not just limited to little spikes but semi-permanent, using 4 cores with HT can produce some unpredictability as Windows will switch physical cores around (and probably set some threads to logical processes instead), especially if the engines are set with lower priority. Using affinities will prevent this, but in that case it will be just like using the full 4 cores without HT, and could produce the same problems that would arise in that case.

So it all depends on what use is made of the PC. If it is only surfing etc. then 4 cores/HT would probably be the best solution, but if instead a more resourceful use of the PC is done (as an example using Photoshop or similar applications that require CPU time always) then it's better to use 3 cores, in that way you are sure that in any case physical cores are used by the engines.

Fabio, spot on. That's exactly why I am limiting the number of cores to 3 physical instead of all 4/HT.

Happy new year!
Great news with the competition coming back, thanks for all your efforts.

On the topic, I'm very much agree with Graham. See, you make a question (poll) and soon people start fighting accusing each other in the "idiot way to do... (something)", or the "weirdness" of a number.

So, better go with the selection, Any option you select will be ok. Spectators (we) are always free to follow if we like your match, or the opposite.

[S.Taylor]

...and the choices are:

1: 3 cores ponder OFF
2: 1 core ponder ON

The computer that is going to be used is an Intel Core i7 3770k @ 4400 MHz.

With ponder OFF the engines will search faster/deeper with 3 cores as opposed to just 1.

On the other hand, with ponder ON it doesn't matter if the engine is SMP capable or not as all engines have equal terms.

What do you prefer?

Wasn't your hardware and cpu's of 2 years ago, faster?
(I'm interested to know, because i myself now have a i7 3770 and i thought it is less powerful than what you used then).

[Martin Thoresen]

Wasn't your hardware and cpu's of 2 years ago, faster?
(I'm interested to know, because i myself now have a i7 3770 and i thought it is less powerful than what you used then).

Yes, the 980x should be a little faster with 6 cores compared to the 3770k with 4 cores.

[hgm]

Many say that PB is the best since engines are meant to be used that way (the time management etc. is meant for PB), ...

That is a bit hard to belief. Almost every engine developer tests his engine with ponder off. So why would the use a time management that sucks.

Both WB and UCI engines know whether pondering is on. (The latter through the Ponder option.) So a serious engine developer would make the time manageemnt dependent on that. And if not, one expects it to be tuned for ponder off, because then it would perform best in the testing.

[lkaufman]

I think the only way you can get decent sample sizes at decent time limits with 3 available cores is to run 3 instances of the test on single core with ponder off. If you run with ponder on and only one instance, or if you run with 3 cpus per engine, you can only play one game at a time, which is reallly pretty hopeless if you want quality games.
In my opinion testing with ponder on is not really worthwhile. First of all, I think all top engines use a fairly similar rule for ponder on, so the results will be very similar with or without it. Furthermore, since I imagine that most people who care about the engine strength do so because they use it for analysis (even Carlsen doesn't need full strength of engine for practice games), ponder on results are actually less relevant for most users.
As for testing MP, it seems that all the strong engines that support MP use very similar algorithms, and so the elo gain from using any given number of cores is probably not that variable, except to the extent that engines scale differently. If you run SP you can use 3x as long a time limit as if you don't (to get the same sample size), so this is a better test of how engines scale than running MP. Of course, since Komodo is not yet MP it is more interesting for us right now if you test SP, but since I expect this will change quite soon this is not the reason for my opinion. The overall quality of your test will be best if you run SP, because using 3 times longer time limit raises the quality much more than running 3 core MP.
Good luck with the project!

[michiguel]

I think the only way you can get decent sample sizes at decent time limits with 3 available cores is to run 3 instances of the test on single core with ponder off. If you run with ponder on and only one instance, or if you run with 3 cpus per engine, you can only play one game at a time, which is reallly pretty hopeless if you want quality games.
In my opinion testing with ponder on is not really worthwhile. First of all, I think all top engines use a fairly similar rule for ponder on, so the results will be very similar with or without it. Furthermore, since I imagine that most people who care about the engine strength do so because they use it for analysis (even Carlsen doesn't need full strength of engine for practice games), ponder on results are actually less relevant for most users.
As for testing MP, it seems that all the strong engines that support MP use very similar algorithms, and so the elo gain from using any given number of cores is probably not that variable, except to the extent that engines scale differently. If you run SP you can use 3x as long a time limit as if you don't (to get the same sample size), so this is a better test of how engines scale than running MP. Of course, since Komodo is not yet MP it is more interesting for us right now if you test SP, but since I expect this will change quite soon this is not the reason for my opinion. The overall quality of your test will be best if you run SP, because using 3 times longer time limit raises the quality much more than running 3 core MP.
Good luck with the project!

The purpose of TCEC was not about testing engines and statistic accuracy, it was about chess entertainment and competition. Games have been played one at a time so the spectators can follow, comment, chat etc. It had been a lot of fun.

I would like to have both ponder and SMP, but given the option stated in the poll, SMP with three threads would be better.

Ideally, if Martin were a millionaire , would be to have two identical computers and each engine can use fully the resources w/o interfering with the opponent.

Miguel

[Graham Banks]

The purpose of TCEC was not about testing engines and statistic accuracy, it was about chess entertainment and competition. Games have been played one at a time so the spectators can follow, comment, chat etc. It had been a lot of fun.

I wish him the best with that, because I've found that not many spectators tend to chat with each other when they've been watching the live broadcasts I've been running. Fingers crossed.