Rebel wrote: ↑Tue May 26, 2020 6:30 am
bob wrote: ↑Mon May 25, 2020 8:20 pm
Rebel wrote: ↑Tue May 19, 2020 10:09 am
I came to the conclusion that NICE is 95% programmer tool, hence I post it here.
Version 1.2 is a significant improvement.
1. NICE now can make predictions with with a precision of 1 elo. The idea to invest 20 minutes before you put an engine change into cute-chess looks attractive to me when NICE can report an elo gain or regression with a precision of 1 elo. It's to hope NICE turns out to be a good prophet
2. Reporting errors - Some engines (in fact a bit more than some) have the habit to surpress the mainlines of the early plies, some hide the first x plies, some hide the main lines the first 100ms and there are even those who use 1 full second. It's probably inspired by a speed gain of 1-2 millieseconds. When there is no main line NICE will report the number of cases found in the error column of the html and the result will be unreliable.
3. The time-bonus feature has been dropped. Someone said, it only adds noise. After a good look I had to conclude it is even worse, without it the results are more reliable.
http://rebel13.nl/misc/nice.html
It isn't about speed. It is about "noise". IE when I play or work with Crafty, I do it in a console window. Run fine #70 and you will see why I added the "noise <n>" feature. Crafty, one thread, on my MacBook hits depth 55 in < 1 second. Do you REALLY want to see all of that, plus the dozens of iterations beyond one second? I don't. But you can always use "noise 0" in the crafty.rc/.craftyrc config file if you want it all.
So for me, it was about usability. IE what if I play someone a game locally where I prefer to use a console window? Crafty announces a move, tells me which move it is pondering, and then promptly scrolls it off the top of the screen, even if the window contains 60-80 lines. No idea that it made a move, no idea what move it made, no idea what it is pondering... Noise makes that sensible. For blitz games (using Cray Blitz) I always used noise 100. That way you get no analysis except for the final PV before it announces a move. No chatty-kathy output that makes it hard to pick out its move and make it.
That's all fine, but why not release your source code with optimized settings, or include an optimized crafty.rc
BTW, I ran Crafty 25.1 under MEA using the wb2uci adapter at 1000ms for the Speedy Rating List and it used much to much time.
Normal is 11 hours and +/- 30 minutes.
Is there a workaround?
What is an "optimized version?" That is what I always release. The default noise level is set to 1 second. So any output prior to one second is not displayed. I can't imagine anyone wanting to play a game and get that first second's worth of output that could easily be 100 lines of output. But if you do want it, just add "noise 0" to your crafty.rc/.craftyrc file. Or if you compile your own, edit data.c and change the initialization for "noise_level" to zero rather than 100 (my time resolution is in 1/100ths of a second.
Next question. How are you setting the time limit? Normal would be "st=1" or "st=1.0" and it will nail that almost exactly (usually +/- .01 seconds. If you are using some other way, that might be causing the issue. Other ways to set the time let Crafty take extra time when the score drops, and less time when it does not. Probably not what you want.
Here's a quick run on WAC, st=1.00. (BTW I just noticed this is not a recent version with all the cooks included. In looking at the output, the only one Crafty fails to get right is #230.)
Code: Select all
total positions searched.......... 300
number right...................... 287
number wrong...................... 13
percentage right.................. 95
percentage wrong.................. 4
total nodes searched.............. 1333598067
average search depth.............. 18.0
nodes per second.................. 5870743
total time........................ 3:47
227 seconds. Should be 300. Only issue here is that on simple mates, it will early terminate, something I am going to remove since it doesn't quite work right anyway. I used to be able to look at depth and (say) if the score was mate in 4, which takes 7 plies to see, once it finished depth=7 it would terminate since no shorter mate could be found. Now with LMR, adaptive null-move, LMP and such, that cutoff really doesn't work and it often quits before finding the shortest mate. But that won't make it go longer. This might well be some incompatibility with the UCI protocol sent from WBtoUCI.
If you can send me a few thousand lines of a log file, I can try to figure out what is getting garbled... But with a "st=1" crafty is going to take 1 second per move, +/- 1/100th (due to sampling noise).