Problem with UCI engines hash in Arena

[carldaman]

I've discovered a problem in Arena 3.0, where UCI engines end up playing with a very limited hash table (2-5 MB), even though they were configured to use more (256MB for example).

When running a tournament or match, in the first game the UCI engine will use the proper amount of allocated hash, but in subsequent games, the amount will drop drastically, to the aforementioned 2-5 MB. Unless Arena is restarted before each new game, the UCI engine will continue to use minimal hash, unfortunately. This is totally unacceptable for serious testing, of course, since the UCI egines are being adversely impacted.

I'm not yet positive that this affects all UCI engines, but I've not witnessed this problem with Winboard engines, so it only seems to apply to UCI engines.

Is anyone aware of this problem/bug, or of any good way around it that would allow one to continue using Arena?

(Unfortunately, Arena development has stalled indefinitely, and it's not likely to see a 'bug-fix' anytime soon. Please correct me if I'm wrong.)

Thanks,
Carl

[Michael Diosi]

Hi,

your source is Pravda? Didn't you learn yet not to listen to them?

... and yes we are checking this..

Michael
http://www.playwitharena.com

[Sylwy]

Hi,

your source is Pravda? Didn't you learn yet not to listen to them?

... and yes we are checking this..

Michael
http://www.playwitharena.com

Hello Miky !

Nice to read you again ! A true pleasure !

Speaking about Pravda (Narodnaia Gazeta) I have found a very interesting article in the online edition:

http://english.pravda.ru/society/sex/25 ... _heaven-0/


Regarding Arena 3.0 GUI . I think a big team must hardly works some years for a (near) free bugs chess GUI. I can recommend you Mr. HGM/The Netherlands - a great expert in good chess GUIs !

Take a look below - please :

Image number one:



Image number two:



Both engines (BeRoChess 1.00 - an UCI one , and Project Invincible 2.01 - a pure WB chess engine , were setted to use 64 MB hash). My match- on Arena 3.0 GUI - is a true fiasco !!! I'm extremely nervous Miky ( in the 3-rd Christmas day ) !!!


SilvianR

[Modern Times]

I do not use Arena for this very reason. But I know other CCRL testers have a workaround for this bug.

[Lavir]

Have you tried with the option "restart engines after each game" in the tournament section?

Anyway, since we are here, there are a couple of things I want to say that could make Arena better.

The arrows when pondering don't work well (for example if you enable the arrow to point out the move before is made, as in the Shredder GUI, when a ponder hit the arrow doesn't appear).

It's badly needed a way to set cores indipendently from every time setting it in engines (as it happens with ChessGUI and Winboard: concerning this last, while it is good to incorporate it, it should work for ANY engine, but with engines that don't have the typical "threads" option - for example those that have "maxthread" or "NumCPUs" or even "NumThreads" - it doesn't work).

A way to set options of engines in the tournament/match and without necessarily save them (as it happens for example in the Fritz GUI).

Make all GUIs have all OPTIONS for what it concerns openings etc. Arena doesn't support repeating lines with opening books, for example.

There are others but these are the most important.

[hgm]

... concerning this last, while it is good to incorporate it, it should work for ANY engine, but with engines that don't have the typical "threads" option - for example those that have "maxthread" or "NumCPUs" or even "NumThreads" - it doesn't work).

The problem is that the option to set this parameter is not explicitly standardized in the UCI specs, which encourages engine authors to think up the most crazy names for this option. (Rather than accepting that in such a case the way Shedder does it should be accepted as the standard.)

So this is basically a losing battle for GUI programmers; No matter how much work you would put into it to catalog all existing engines and investigate which option or combination of options they use fo this, and then implement them all, the next day there can emerge a new engine that uses yet something else. Sometimes I get the idea that engine authors do this out of spite, or see it as a blemish on the originality of their engine to use an option name that has already been used by someone else. Of couse one also has to combat the misconception that users would actually care if the option name accuately reflects the technical implementation. Like refusing to use 'Threads' because the engine happens to use processes rather than threads to implement SMP, etc.

Of course UCI handling in WinBoard is delegated to Polyglot, and the latter actually does make a fair attempt to support a variety of names, put in their by the last active Polyglot developer:

Code: Select all

// Hopefully the following confusion is temporary
// Normally we should check for the engine name but this is a hack anyway
// Some of there where provided by Marc Lacrosse

const char * thread_options[]={
  "number of threads",        // toga
  "number threads",           // Deep Learning Toga
  "threads",                  // glaurung, zappa, cyclone, grapefruit,
                              // Deep Shredder, Deep Junior, bright
  "core threads",             // HIARCS
  "max cpus",                 // rybka
  "cpus",                     // Deep Sjeng, Fruit2.3.5
  "maxthreads",               // Naum 
  NULL
};

Personally I have my severe doubts whether it is wise to go that road. The world would be much better served if GUI authors would make a firm stance, and create a de-facto standard where the official specs fail to provide guidance. Basically saying: "OK, this is the supported set from now on", and if engine authors want to wreck correct operation of their engine by picking something else, the consequences will be on them, and if users don't like it, they should complain to the engine author!". (And preferably the set of supported options would have only a single member...)

Unfortunately petty rivalry between GUI authors ("My GUI is better than yous, because it can run this crappy and highly non-compliant engine without problem, when yours rejects it!") is unlikely to ever make this happen. Much to the detriment of everyone...

[carldaman]

Hi Ray, glad a serious tester has noticed this as well. I don't mean to pick on Arena, as I like the software, which came free, to boot , but some of the glitches can really get in the way of good testing.

My comments were meant as constructive criticism. Sorry if I was not aware of recent work on Arena.

Regards,
CL

[Matthias Gemuh]

... concerning this last, while it is good to incorporate it, it should work for ANY engine, but with engines that don't have the typical "threads" option - for example those that have "maxthread" or "NumCPUs" or even "NumThreads" - it doesn't work).

The problem is that the option to set this parameter is not explicitly standardized in the UCI specs, which encourages engine authors to think up the most crazy names for this option. (Rather than accepting that in such a case the way Shedder does it should be accepted as the standard.)

...

ChessGUI supports

Code: Select all

    char *Threads[] = { "CPUs", "Core Threads", "Cores", "MAX_CPU", "Max CPUs", "MaxThreads", "Max_Threads", "Number Of Threads", "NumberOfProcessors", "NumCPUs", "Processes", "Threads", "TitanicCPUsSizings", "TitanicCPUsSizings (Capitalist!)" };

... see the last in the list !!!!

ChessGUI shall not support any new crazy "Thread" pseudonyms.

[Lavir]

Personally I have my severe doubts whether it is wise to go that road. The world would be much better served if GUI authors would make a firm stance, and create a de-facto standard where the official specs fail to provide guidance.

I understand what you say, and I agree with you. Authors should really conform to a standard and simply follow it. All would be much easier.

I think however, given that things sadly are in this way, that you should at last support the major branches that are "maxthreads", "max cpus" and "numthreads" (written in different ways) at last (for example Bouquet 1.6 doesn't work because it has "max_threads" instead of "maxthreads" in your example).

As an advice for Winboard: you should work a little to refine the tournament/matches section to give more options, as for example the possibility of changing engines parameters also there (as the threads, hash, options etc). Also a possibility to end the match after a certain number of games are played would be good (instead of manually aborting) etc.

If you like I could make a detailed list of the improvements I think that could really improve Winboard as an engine vs engine interface in private.

[hgm]

I think however, given that things sadly are in this way, that you should at last support the major branches that are "maxthreads", "max cpus" and "numthreads" (written in different ways) at last (for example Bouquet 1.6 doesn't work because it has "max_threads" instead of "maxthreads" in your example).

I am not yet completely convinced. If it is just Bouquet that uses this variety, it would be better that Bouquet changes the option to the already-supported 'maxthreads', rather than expanding what Polyglot supports even further. Bouquet is still actively developed, and its author posts here regularly. I think we should first make a serious attempt to weed out all the forms that are hardly used (especially those used by only one engine), and only add new forms to the supported list if that really is not possible. (E.g. because a commercial engine considers it against its interest to be compatible with other GUIs.)

As an advice for Winboard: you should work a little to refine the tournament/matches section to give more options, as for example the possibility of changing engines parameters also there (as the threads, hash, options etc).

I am not sure I completely understand what you mean here. The Tournament Options dialog of WinBoard already has buttons where you can call up the Time Control and Common Engine dialogs, to change the things you mention without having to leave the Tournament Options dialog. Would you want to have the individual controls of the latter dialogs as duplicates in the Tounament Options, rather than a quick way to go there? I am afraid this would overload the dialog, which is already quite full. I am also not sure what you mean by 'options'. The General Options?

Also a possibility to end the match after a certain number of games are played would be good (instead of manually aborting) etc.

It does already do that, not? If you start a match from the menu, that match will stop after 'default match games', setable from the Common Engine dialog (or, as 'games per pairing', from the Tournament Options dialog). Unless you mean something else than I think.

If you like I could make a detailed list of the improvements I think that could really improve Winboard as an engine vs engine interface in private.

Yes, please. I always appreciate feedback. My e-mail address is h DOT g DOT muller ATSIGN hccnet DOT nl. I cannot promise that I will always agree with you, but it always very useful to hear what users want, even if in the end it would only be a minority opinion. (Sometimes irreconcilable differences in user demands can be solved by putting features under option control.)