NICE 1.2

[bob]

I came to the conclusion that NICE is 95% programmer tool, hence I post it here.

Version 1.2 is a significant improvement.

1. NICE now can make predictions with with a precision of 1 elo. The idea to invest 20 minutes before you put an engine change into cute-chess looks attractive to me when NICE can report an elo gain or regression with a precision of 1 elo. It's to hope NICE turns out to be a good prophet

2. Reporting errors - Some engines (in fact a bit more than some) have the habit to surpress the mainlines of the early plies, some hide the first x plies, some hide the main lines the first 100ms and there are even those who use 1 full second. It's probably inspired by a speed gain of 1-2 millieseconds. When there is no main line NICE will report the number of cases found in the error column of the html and the result will be unreliable.

3. The time-bonus feature has been dropped. Someone said, it only adds noise. After a good look I had to conclude it is even worse, without it the results are more reliable.

http://rebel13.nl/misc/nice.html

It isn't about speed. It is about "noise". IE when I play or work with Crafty, I do it in a console window. Run fine #70 and you will see why I added the "noise <n>" feature. Crafty, one thread, on my MacBook hits depth 55 in < 1 second. Do you REALLY want to see all of that, plus the dozens of iterations beyond one second? I don't. But you can always use "noise 0" in the crafty.rc/.craftyrc config file if you want it all.

So for me, it was about usability. IE what if I play someone a game locally where I prefer to use a console window? Crafty announces a move, tells me which move it is pondering, and then promptly scrolls it off the top of the screen, even if the window contains 60-80 lines. No idea that it made a move, no idea what move it made, no idea what it is pondering... Noise makes that sensible. For blitz games (using Cray Blitz) I always used noise 100. That way you get no analysis except for the final PV before it announces a move. No chatty-kathy output that makes it hard to pick out its move and make it.

[Rebel]

It works - AND I found a (possible) bug in Myrddin! It does not accept "st" command values less than 1, because the protocol says it is used to "set the exact number of seconds per move". I guess, when I implemented that more than 10 years ago, I never considered decimal values. But the text of the protocol (and the example, which only uses the number 30) should probably be updated.

However, trying to do a Split Run via Myrddin.bat in NICE fails with "Error in line 8".

Thanks very much!

Split Run has an obliged format -

Myrddin.bat

Code: Select all

del debug*.*

set MT=1000
set HASH=128
set THREADS=1
set PROTOCOL=uci
set EPD=epd\sts.epd
set OPTIONS=MultiPV=1
set MRL=x1

set EXE=engines\wb2uci.exe
set NAME=Myrddin_0.87

mea.exe --engine %EXE% --name %NAME% --hash %HASH% --threads %THREADS% --protocol %PROTOCOL% --epd %EPD% --movetime %MT% --log --eoption %OPTIONS%
csv %MRL% %EPD% %OPTIONS% %NAME% %MT% %CCRL%
mrl %MRL% %MRL% %EPD% %MT%

exit

[Rebel]

I came to the conclusion that NICE is 95% programmer tool, hence I post it here.

Version 1.2 is a significant improvement.

1. NICE now can make predictions with with a precision of 1 elo. The idea to invest 20 minutes before you put an engine change into cute-chess looks attractive to me when NICE can report an elo gain or regression with a precision of 1 elo. It's to hope NICE turns out to be a good prophet

2. Reporting errors - Some engines (in fact a bit more than some) have the habit to surpress the mainlines of the early plies, some hide the first x plies, some hide the main lines the first 100ms and there are even those who use 1 full second. It's probably inspired by a speed gain of 1-2 millieseconds. When there is no main line NICE will report the number of cases found in the error column of the html and the result will be unreliable.

3. The time-bonus feature has been dropped. Someone said, it only adds noise. After a good look I had to conclude it is even worse, without it the results are more reliable.

http://rebel13.nl/misc/nice.html

It isn't about speed. It is about "noise". IE when I play or work with Crafty, I do it in a console window. Run fine #70 and you will see why I added the "noise <n>" feature. Crafty, one thread, on my MacBook hits depth 55 in < 1 second. Do you REALLY want to see all of that, plus the dozens of iterations beyond one second? I don't. But you can always use "noise 0" in the crafty.rc/.craftyrc config file if you want it all.

So for me, it was about usability. IE what if I play someone a game locally where I prefer to use a console window? Crafty announces a move, tells me which move it is pondering, and then promptly scrolls it off the top of the screen, even if the window contains 60-80 lines. No idea that it made a move, no idea what move it made, no idea what it is pondering... Noise makes that sensible. For blitz games (using Cray Blitz) I always used noise 100. That way you get no analysis except for the final PV before it announces a move. No chatty-kathy output that makes it hard to pick out its move and make it.

That's all fine, but why not release your source code with optimized settings, or include an optimized crafty.rc

BTW, I ran Crafty 25.1 under MEA using the wb2uci adapter at 1000ms for the Speedy Rating List and it used much to much time.

http://rebel13.nl/mea/crafty.html

Normal is 11 hours and +/- 30 minutes.

Is there a workaround?

[Terje]

Where do the engines using too much time spend it? I assume they give a move in time, so not on search?

[Rebel]

Where do the engines using too much time spend it? I assume they give a move in time, so not on search?

Engines are feeded with "go movetime 1000".

Some play too fast, some use too much time.

It's up the programmer to fix it.

[Terje]

Where do the engines using too much time spend it? I assume they give a move in time, so not on search?

Engines are feeded with "go movetime 1000".

Some play too fast, some use too much time.

It's up the programmer to fix it.

Oh, they actually search too long?

[bob]

I came to the conclusion that NICE is 95% programmer tool, hence I post it here.

Version 1.2 is a significant improvement.

1. NICE now can make predictions with with a precision of 1 elo. The idea to invest 20 minutes before you put an engine change into cute-chess looks attractive to me when NICE can report an elo gain or regression with a precision of 1 elo. It's to hope NICE turns out to be a good prophet

2. Reporting errors - Some engines (in fact a bit more than some) have the habit to surpress the mainlines of the early plies, some hide the first x plies, some hide the main lines the first 100ms and there are even those who use 1 full second. It's probably inspired by a speed gain of 1-2 millieseconds. When there is no main line NICE will report the number of cases found in the error column of the html and the result will be unreliable.

3. The time-bonus feature has been dropped. Someone said, it only adds noise. After a good look I had to conclude it is even worse, without it the results are more reliable.

http://rebel13.nl/misc/nice.html

It isn't about speed. It is about "noise". IE when I play or work with Crafty, I do it in a console window. Run fine #70 and you will see why I added the "noise <n>" feature. Crafty, one thread, on my MacBook hits depth 55 in < 1 second. Do you REALLY want to see all of that, plus the dozens of iterations beyond one second? I don't. But you can always use "noise 0" in the crafty.rc/.craftyrc config file if you want it all.

So for me, it was about usability. IE what if I play someone a game locally where I prefer to use a console window? Crafty announces a move, tells me which move it is pondering, and then promptly scrolls it off the top of the screen, even if the window contains 60-80 lines. No idea that it made a move, no idea what move it made, no idea what it is pondering... Noise makes that sensible. For blitz games (using Cray Blitz) I always used noise 100. That way you get no analysis except for the final PV before it announces a move. No chatty-kathy output that makes it hard to pick out its move and make it.

That's all fine, but why not release your source code with optimized settings, or include an optimized crafty.rc

BTW, I ran Crafty 25.1 under MEA using the wb2uci adapter at 1000ms for the Speedy Rating List and it used much to much time.

Normal is 11 hours and +/- 30 minutes.

Is there a workaround?

What is an "optimized version?" That is what I always release. The default noise level is set to 1 second. So any output prior to one second is not displayed. I can't imagine anyone wanting to play a game and get that first second's worth of output that could easily be 100 lines of output. But if you do want it, just add "noise 0" to your crafty.rc/.craftyrc file. Or if you compile your own, edit data.c and change the initialization for "noise_level" to zero rather than 100 (my time resolution is in 1/100ths of a second.

Next question. How are you setting the time limit? Normal would be "st=1" or "st=1.0" and it will nail that almost exactly (usually +/- .01 seconds. If you are using some other way, that might be causing the issue. Other ways to set the time let Crafty take extra time when the score drops, and less time when it does not. Probably not what you want.

Here's a quick run on WAC, st=1.00. (BTW I just noticed this is not a recent version with all the cooks included. In looking at the output, the only one Crafty fails to get right is #230.)

Code: Select all

total positions searched..........         300
number right......................         287
number wrong......................          13
percentage right..................          95
percentage wrong..................           4
total nodes searched..............  1333598067
average search depth..............        18.0
nodes per second..................     5870743
total time........................        3:47

227 seconds. Should be 300. Only issue here is that on simple mates, it will early terminate, something I am going to remove since it doesn't quite work right anyway. I used to be able to look at depth and (say) if the score was mate in 4, which takes 7 plies to see, once it finished depth=7 it would terminate since no shorter mate could be found. Now with LMR, adaptive null-move, LMP and such, that cutoff really doesn't work and it often quits before finding the shortest mate. But that won't make it go longer. This might well be some incompatibility with the UCI protocol sent from WBtoUCI.

If you can send me a few thousand lines of a log file, I can try to figure out what is getting garbled... But with a "st=1" crafty is going to take 1 second per move, +/- 1/100th (due to sampling noise).

[bob]

Where do the engines using too much time spend it? I assume they give a move in time, so not on search?

Engines are feeded with "go movetime 1000".

Some play too fast, some use too much time.

It's up the programmer to fix it.

And if it is not the program's issue? Crafty does this perfectly. In console mode or with xboard. Not a clue what UCITOWB does/sends. A log file (at least a few thousand lines) would help. The "go movetime 1000" would certainly be an issue with Crafty if 1000 is passed to it. That is 10 seconds in crafty's 1/100th of a second time resolution.

[Rebel]

Where do the engines using too much time spend it? I assume they give a move in time, so not on search?

Engines are feeded with "go movetime 1000".

Some play too fast, some use too much time.

It's up the programmer to fix it.

And if it is not the program's issue? Crafty does this perfectly. In console mode or with xboard. Not a clue what UCITOWB does/sends. A log file (at least a few thousand lines) would help. The "go movetime 1000" would certainly be an issue with Crafty if 1000 is passed to it. That is 10 seconds in crafty's 1/100th of a second time resolution.

Got it to work with the wb2uci adapter. The trick to use "go movetime 1" and Crafty plays at one second.

It's odd, mine is WB too and and for one second I have to use the normal "go movetime 1000".

Well, as long as it works, even in mysterious ways.

[Dann Corbit]

I think the mystery is this:
When crafty starts, it is not in winboard/xboard mode.
It understands st 3 to mean set time to 3 seconds, or sd 5 to set depth to 5 plies.
After you issue the xboard command, crafty is in xboard mode and the commands are processed normally.

One of my tweaks was to set xboard mode on by default so you did not have to issue the xboard command, since I only used xboard commands with it anyway