NICE vs CUTE

[RubiChess]

Hi Ed.
That looks very interesting.
I have rewritten my engine today to expose all the serveral hundreds evaluation parameters as uci options and I have a question before running some tests with NICE:
Is it possible for mea.exe to start the engine with commandline parameter like

Code: Select all

set EXE=engines\RubiChess.exe -evaloptions
...
mea.exe --engine "%EXE%" --name %NAME% ...

The evaluation parameters are only exposed to uci interface if engine is started with "-evaloptions".
If this is not possible I need to compile a binary with hardcoded -evaloptions enabled.

Thanks, Andreas

[Rebel]

Hi Andreas,

I see 2 possibilities.

set EXE="engines\Rubi.exe -evaloptions"

or via OPTIONS

set OPTIONS="MultiPV=1,Mobility = 75"

or

set OPTIONS="MultiPV=1,Mobility = 75,Passed Pawns = 125"

[RubiChess]

Hi Andreas,

I see 2 possibilities.

set EXE="engines\Rubi.exe -evaloptions"

or via OPTIONS

set OPTIONS="MultiPV=1,Mobility = 75"

or

set OPTIONS="MultiPV=1,Mobility = 75,Passed Pawns = 125"

Thanks. I did a first test by passing a very bad value for some parameter and it "failed" by getting the same score as the version with default values. Another problem might be that my eval options are given in the format "Value(20,40)" for midgame and endgame evaluation. The comma may confuse the parser of Mea so I will change it to "Value(20/40)".

Andreas

[Rebel]

Hi Andreas,

I see 2 possibilities.

set EXE="engines\Rubi.exe -evaloptions"

or via OPTIONS

set OPTIONS="MultiPV=1,Mobility = 75"

or

set OPTIONS="MultiPV=1,Mobility = 75,Passed Pawns = 125"

Thanks. I did a first test by passing a very bad value for some parameter and it "failed" by getting the same score as the version with default values. Another problem might be that my eval options are given in the format "Value(20,40)" for midgame and endgame evaluation. The comma may confuse the parser of Mea so I will change it to "Value(20/40)".

I am pretty sure that is the case.

[RubiChess]

I did some tests now and so far I just see noise.

Code: Select all

MEA
A. Processor:
Brand: AMD Ryzen 7 3700X 8-Core Processor
Arch: X86_64
Cores: 16

B. EPD test set:
Filename: lcx-2.epd

Rank	Engine	Rating	Top1	MaxTop1	Top1Rate	Score	MaxScore	ScoreRate	MoveTime(ms)	Hash(MB)	Threads
1	Rubi-default	2500	4850	10000	0.485	73815	100000	0.738	100	128	1
2	Rubi-ppt-50-23	2500	4862	10000	0.486	73521	100000	0.735	100	128	1
3	Rubi-ppt-0-33	2500	4825	10000	0.482	73207	100000	0.732	100	128	1
4	Rubi-ppt-30-23	2500	4849	10000	0.485	73193	100000	0.732	100	128	1
5	Rubi-ppt-40-43	2500	4831	10000	0.483	73170	100000	0.732	100	128	1
6	Rubi-ppt-0-13	2500	4780	10000	0.478	73117	100000	0.731	100	128	1
7	Rubi-ppt-40-33	2500	4813	10000	0.481	73012	100000	0.730	100	128	1
8	Rubi-ppt-10-13	2500	4856	10000	0.486	71778	100000	0.718	100	128	1
9	Rubi-ppt-0-13	2500	4825	10000	0.482	71604	100000	0.716	100	128	1
10	Rubi-ppt-30-23	2500	4882	10000	0.488	71586	100000	0.716	100	128	1
11	Rubi-ppt-10-13	2500	4934	10000	0.493	71472	100000	0.715	100	128	1
12	Rubi-ppt-40-33	2500	4842	10000	0.484	71455	100000	0.715	100	128	1
13	Rubi-ppt-40-43	2500	4883	10000	0.488	71443	100000	0.714	100	128	1
14	Rubi-ppt-0-33	2500	4848	10000	0.485	71291	100000	0.713	100	128	1
15	Rubi-ppt-50-23	2500	4830	10000	0.483	71118	100000	0.711	100	128	1
16	Rubi-default	2500	4839	10000	0.484	70997	100000	0.710	100	128	1

This is the result of two runs with variations of my Pawnpushthreatbonus parameter. The two runs used different epd sets (lcx-1.epd in the first, lcx-2.epd in the second), everything else was the same.
The default version finished last in the first run and first in the second. And also the other version are almost mirrored in the score board.

I took your a (a1...a8) example and ran 8 threads parallel on a Ryzen 3700x (8 cores, 16 threads with HT).
I also tried some other parameters with the same no-result.

I'm quite sure that this disappointing noisy result is related to some problems of unbalanced cpu time or whatever that I also reported here http://talkchess.com/forum3/viewtopic.php?f=7&t=73299

So my idea is not to use a constant movetime of 100ms but a constant number of nodes. This would eliminate the unbalanced cpu problemes but it seems that mea doesn't support that?

Regards, Andreas

[Rebel]

Not sure if MEA supports nodes (I don't think so) but it does support depth.

set OPTIONS="MultiPV=1,depth=20"

Secondly so far I have only used the SFX epd's after noticing they gave me better results.

[Rebel]

I'm quite sure that this disappointing noisy result is related to some problems of unbalanced cpu time or whatever that I also reported here http://talkchess.com/forum3/viewtopic.php?f=7&t=73299

What definitely helps is managing Affinity when you have x single threads running, pinning each run to a specific core.

http://rebel13.nl/misc/scheduling.html

[RubiChess]

Not sure if MEA supports nodes (I don't think so) but it does support depth.

set OPTIONS="MultiPV=1,depth=20"

Secondly so far I have only used the SFX epd's after noticing they gave me better results.

Meanwhile I have some better (more reliable) results with fixed depth search (18 or 20) that showed default settings constantly with top position or at least head-to-head with the top position. So the Texel-tuning did a good job and your tool is also useful confirming these Texel-tuned parameters.

Regards, Andreas

[Rebel]

Excellent news.

If I may ask, did you use the sfx or lcx positions?

BTW, I split the sfx.epd 80,000 positions into 7 parts for specific tuning:

Code: Select all

  1. Midgame           46.262 positions (opening plus midgame)
  2. Early-endgame      7.194 positions (midgame minus queens)
  3. Endgame           21.264 positions (material white & black <14 each following the 1,3,3,5,9 rule.
  4. Queen-endings        418 positions
  5. Rook-endings       1.481 positions
  6. Late-endgame       5.799 positions (Bishop versus Knight endings)
  7. Pawn-endgame         146 positions

One need to (re)download.

[RubiChess]

Excellent news.

If I may ask, did you use the sfx or lcx positions?

The fixed-depth tests used the sfx. You said you had better results with them so I used them too.
Andreas