DiscoCheck performing 200+ ELO better on ONE machine ??

[AndrewGrant]

So I use DiscoCheck to test patches for my own engine, Ethereal.

On an i5-win64 box, DiscoCheck has about a 55% winrate against Ethereal
On an ryzen-linux box, DiscoCheck has about a 55% winrate against Ethereal
On an xeonE5-linux box, DiscoCheck has about a 90% winrate against Ethereal

Other matchups, like Pawny1.2 vs Ethereal, give similar results across all 3 machines. DiscoCheck seems to be an outlier, and I have NO idea why.

I have tried playing games (through cutechess) with concurrency=4, =11, and =22, and the results are about the same 90% WR.

Any ideas?

[AndrewGrant]

Also, I'll note that I scale time control relative to each machine

Also, DiscoCheck gets about the same NPS on all three machines, +-150K (which is less than a 20% difference, not worth hundreds of ELO)

[cdani]

Are you playing many games?
Have you verified this various times?

[lucasart]

So I use DiscoCheck to test patches for my own engine, Ethereal.

On an i5-win64 box, DiscoCheck has about a 55% winrate against Ethereal
On an ryzen-linux box, DiscoCheck has about a 55% winrate against Ethereal
On an xeonE5-linux box, DiscoCheck has about a 90% winrate against Ethereal

Other matchups, like Pawny1.2 vs Ethereal, give similar results across all 3 machines. DiscoCheck seems to be an outlier, and I have NO idea why.

I have tried playing games (through cutechess) with concurrency=4, =11, and =22, and the results are about the same 90% WR.

Any ideas?

DiscoCheck (like any other engine) runs faster on Linux that on Windows. But that should be the same for Ethereal, so it can't be the explanation.

Between the 2 Linux machines, there shouldn't be such a big difference (relative to Ethereal).

Are you sure you compiled it the same way on both Linux machines ?

The way to compile it is as follows:

Code: Select all

g++ ./src/*.cc -o ./discocheck -std=c++11 -DNDEBUG -O3 -msse4.2 -fno-rtti -flto -s

You could try clang also, if it works (depending on your linux distro, clang LTO may not work).

[Ras]

Other matchups, like Pawny1.2 vs Ethereal, give similar results across all 3 machines.

What about the other way round? Does Discocheck only score that high against Ethereal on the Xeon machine, or also against other engines in the same ballpark? How many games, i.e. what is the error margin?

[AndrewGrant]

500 games 3 times, all resulted in ~90%

Played a 2k game as well, also around ~90%

As for Ethereal vs Discocheck on the non xeon machines,
Ethereal has a 47.38% winrate over the course of 6000 games.

Also, a slight tweak of Ethereal vs Disco check on the non xeon machines
Ethereal has a 47.15% winrate over the course of 5000 games.

[AndrewGrant]

Your compile string is just about the way I compiled it.

Even so, compile flags should simply affect the speed of the engine. Something which I am already accounting for.

[AndrewGrant]

Here is how cutechess is seeding its RNG

QTime(0,0,0).msecsTo(QTime::currentTime())

My only thought left is that I am EXTREMELY unlucky in the first thousand or so games played againt Ethereal and DiscoCheck.

OR, for some reason cutechess is using the same seed everytime, and I am again VERY unlucky.

Tomorrow I will repeat this test, but passing a seed to the cutechess executable...

[lucasart]

I understand you're building some sort of fishtest. Might be worth keeping track of info like: number of time losses and number of crashes. Perhaps it's Ethereal that has problems on this machine.

[AndrewGrant]

You would be correct. Ethereal is at fault, and I have found the culprit.

At the top of every alphabeta node I do this:

Code: Select all

    // Step 1. Check to see if search time has expired
    if (       Info->searchIsTimeLimited 
        &&  getRealTime() >= Info->endTime2){
        Info->terminateSearch = 1;
        return board->turn == EvaluatingPlayer ? -MATE : MATE;
    }

Code: Select all

double getRealTime(){
#if defined(_WIN32) || defined(_WIN64)
    return (double)(GetTickCount());
#else
    struct timeval tv;
    double secsInMilli, usecsInMilli;
    
    gettimeofday(&tv, NULL);
    secsInMilli = ((double)tv.tv_sec) * 1000;
    usecsInMilli = tv.tv_usec / 1000;
    
    return secsInMilli + usecsInMilli;
#endif
}

On this system it seems that getRealTime() is VERY VERY slow.

For Ethereal-master:

position startpos
go wtime 60000 winc 600
info depth 14 score cp 20 time 1179 nodes 1346799 nps 1141000 hashfull 86 pv e2e4 e7e5 g1f3 b8c6 f1b5 g8f6 e1g1 f8d6 b1c3 e8g8 d2d4 e5d4 f3d4 f8e8
bestmove e2e4

For Ethereal-master:

position startpos
go depth 14
info depth 14 score cp 20 time 658 nodes 1346799 nps 2043000 hashfull 86 pv e2e4 e7e5 g1f3 b8c6 f1b5 g8f6 e1g1 f8d6 b1c3 e8g8 d2d4 e5d4 f3d4 f8e8
bestmove e2e4

Only difference is the nodes per second. When we search to a set depth, i set info->searchIsTimeLimited to false, and getRealTime() is never called.


I ran a 1000 game test against Ethereal-every4096 and Ethereal-master...
Score of every4096 vs master : 792 - 57 - 151 [0.868] 1000
I think that proves it. I'm currently playing games against DiscoCheck now to confirm, and it looks like I'm back to the expected 45% winrate.