NICE vs CUTE

[Rebel]

NICE is a tool meant to significantly shorten long lasting engine testing like eng-eng matches with CUTE-chess-cli of 10,000, 15,000 ... 30,000 bullet games. However NICE can never replace the final arbiter CUTE.

NICE is a further developed version of Temere that is reasonable able to produce a reliable ranking list of chess engins within an error bar of -25/+25 elo. NICE goes one step further and tries to keep the elo error bar within 5-10 elo.

...

http://rebel13.nl/misc/nice.html

[pohl4711]

Some strange lines in sfx.epd...

Example:
r2q1rk1/p1pn1ppp/bp1bpn2/3pN3/2PP4/1P4P1/PB2PPBP/RN1Q1RK1 b - - id sf10x.exe; bm c7c5; c0 "Bb7=10, Re8=6, Ne4=6, c5=6";

1) id sf10.exe is only in some lines written in others not
2) bm c7c5 in full notation? In other lines there are pawn-moves like c4 (short notation)
3) The best move is only worth 6 points??? If Bb7 gives 10 points, why is not Bb7 the best move?

(the line below the example from above looks better:)
r2q1rk1/p1pnb1pp/1p2p3/3b1p2/8/1P3NP1/PBQPPPBP/R3R1K1 b - - bm a5; bm a5; c0 "a5=10, c5=8, Bc5=7, Bf6=2";


Strange, too:
in the lcx.epd file, I found that line:
r1r3k1/3Qnp1p/p2ppbp1/P1p2q2/2PP4/R4N2/1P1B1PPP/4R1K1 w - - id sf10x.exe; bm d7d6; bm Bc3; c0 "h4=10, Bc3=8, d5=6, Qxd6=3";

1) sf10.exe in lcx-file ???
2) 2 best moves (d7d6 and Bc3)?
3) 1 best move (d7d6) gives 0 points???
4) Why is h4 not the best move, when it gives 10 points?

[Rebel]

r2q1rk1/p1pn1ppp/bp1bpn2/3pN3/2PP4/1P4P1/PB2PPBP/RN1Q1RK1 b - - id sf10x.exe; bm c7c5; c0 "Bb7=10, Re8=6, Ne4=6, c5=6";

What matters in MEA EPD's are the moves plus points in the "c0" tag.

[JVMerlino]

Excellent tool, Ed (and Ferdinand). Does it only support UCI engines?

[Rebel]

Excellent tool, Ed (and Ferdinand). Does it only support UCI engines?

Winboard also, see the MEA help.txt file.

Or use WB2UCI.

[pohl4711]

r2q1rk1/p1pn1ppp/bp1bpn2/3pN3/2PP4/1P4P1/PB2PPBP/RN1Q1RK1 b - - id sf10x.exe; bm c7c5; c0 "Bb7=10, Re8=6, Ne4=6, c5=6";

What matters in MEA EPD's are the moves plus points in the "c0" tag.

Ah. OK. I understand. Thanx for explanation.

[JVMerlino]

Excellent tool, Ed (and Ferdinand). Does it only support UCI engines?

Winboard also, see the MEA help.txt file.

Or use WB2UCI.

Terrific! I've given it a try to get a base level, and here's the output that comes back very quickly.

> .\mea --engine engines\Myrddin-087-64.exe --name Myrddin --protocol xboard --epd epd\sfx-1.epd --movetime 100 --log
epd 1 / 10000
Traceback (most recent call last):
File "mea.py", line 1007, in <module>
File "mea.py", line 979, in main
File "mea.py", line 178, in run
File "mea.py", line 628, in run_xb_engine
UnboundLocalError: local variable 'movesan' referenced before assignment
[7568] Failed to execute script mea

I assume I'm doing something wrong....

[Rebel]

Excellent tool, Ed (and Ferdinand). Does it only support UCI engines?

Winboard also, see the MEA help.txt file.

Or use WB2UCI.

Terrific! I've given it a try to get a base level, and here's the output that comes back very quickly.

> .\mea --engine engines\Myrddin-087-64.exe --name Myrddin --protocol xboard --epd epd\sfx-1.epd --movetime 100 --log
epd 1 / 10000
Traceback (most recent call last):
File "mea.py", line 1007, in <module>
File "mea.py", line 979, in main
File "mea.py", line 178, in run
File "mea.py", line 628, in run_xb_engine
UnboundLocalError: local variable 'movesan' referenced before assignment
[7568] Failed to execute script mea

I assume I'm doing something wrong....

If I remember correctly for Winboard MEA needs a few extra settings, try to figure it out via the MEA help.txt file, else this is a question for Ferdy.

My own is also Winboard and I took the easy way out using the WB2UCI adapter and you can stick to the UCI settings, runs flawlessly.

So:

mea --engine engines\wb2uci.exe --name Myrddin --protocol uci --epd epd\sfx-1.epd --movetime 100 --log

[Rebel]

r2q1rk1/p1pn1ppp/bp1bpn2/3pN3/2PP4/1P4P1/PB2PPBP/RN1Q1RK1 b - - id sf10x.exe; bm c7c5; c0 "Bb7=10, Re8=6, Ne4=6, c5=6";

What matters in MEA EPD's are the moves plus points in the "c0" tag.

Ah. OK. I understand. Thanx for explanation.

The 80,000 EPD's come from multiple sources with the intend to be as random as possible, hence not all deserve the beauty prize

[pohl4711]

The 80,000 EPD's come from multiple sources with the intend to be as random as possible, hence not all deserve the beauty prize

Thats clear. I use the first 50000 sf-epd and the scorerate of the first NNs, I retested (1 node per position), is clearly lower (from 67% down to 60%), than using your old epd-files. Thats good, because there is more room for improved NNs...