DiscoCheck performing 200+ ELO better on ONE machine ??

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
AndrewGrant
Posts: 321
Joined: Tue Apr 19, 2016 4:08 am
Contact:

DiscoCheck performing 200+ ELO better on ONE machine ??

Post by AndrewGrant » Mon Nov 13, 2017 6:58 pm

So I use DiscoCheck to test patches for my own engine, Ethereal.

On an i5-win64 box, DiscoCheck has about a 55% winrate against Ethereal
On an ryzen-linux box, DiscoCheck has about a 55% winrate against Ethereal
On an xeonE5-linux box, DiscoCheck has about a 90% winrate against Ethereal

Other matchups, like Pawny1.2 vs Ethereal, give similar results across all 3 machines. DiscoCheck seems to be an outlier, and I have NO idea why.

I have tried playing games (through cutechess) with concurrency=4, =11, and =22, and the results are about the same 90% WR.

Any ideas?

AndrewGrant
Posts: 321
Joined: Tue Apr 19, 2016 4:08 am
Contact:

Re: DiscoCheck performing 200+ ELO better on ONE machine ??

Post by AndrewGrant » Mon Nov 13, 2017 7:09 pm

Also, I'll note that I scale time control relative to each machine

Also, DiscoCheck gets about the same NPS on all three machines, +-150K (which is less than a 20% difference, not worth hundreds of ELO)

User avatar
cdani
Posts: 2047
Joined: Sat Jan 18, 2014 9:24 am
Location: Andorra
Contact:

Re: DiscoCheck performing 200+ ELO better on ONE machine ??

Post by cdani » Mon Nov 13, 2017 11:09 pm

Are you playing many games?
Have you verified this various times?

User avatar
lucasart
Posts: 2957
Joined: Mon May 31, 2010 11:29 am
Contact:

Re: DiscoCheck performing 200+ ELO better on ONE machine ??

Post by lucasart » Mon Nov 13, 2017 11:19 pm

AndrewGrant wrote:So I use DiscoCheck to test patches for my own engine, Ethereal.

On an i5-win64 box, DiscoCheck has about a 55% winrate against Ethereal
On an ryzen-linux box, DiscoCheck has about a 55% winrate against Ethereal
On an xeonE5-linux box, DiscoCheck has about a 90% winrate against Ethereal

Other matchups, like Pawny1.2 vs Ethereal, give similar results across all 3 machines. DiscoCheck seems to be an outlier, and I have NO idea why.

I have tried playing games (through cutechess) with concurrency=4, =11, and =22, and the results are about the same 90% WR.

Any ideas?
DiscoCheck (like any other engine) runs faster on Linux that on Windows. But that should be the same for Ethereal, so it can't be the explanation.

Between the 2 Linux machines, there shouldn't be such a big difference (relative to Ethereal).

Are you sure you compiled it the same way on both Linux machines ?

The way to compile it is as follows:

Code: Select all

g++ ./src/*.cc -o ./discocheck -std=c++11 -DNDEBUG -O3 -msse4.2 -fno-rtti -flto -s
You could try clang also, if it works (depending on your linux distro, clang LTO may not work).
Theory and practice sometimes clash. And when that happens, theory loses. Every single time.

Ras
Posts: 952
Joined: Tue Aug 30, 2016 6:19 pm
Contact:

Re: DiscoCheck performing 200+ ELO better on ONE machine ??

Post by Ras » Tue Nov 14, 2017 12:14 am

AndrewGrant wrote:Other matchups, like Pawny1.2 vs Ethereal, give similar results across all 3 machines.
What about the other way round? Does Discocheck only score that high against Ethereal on the Xeon machine, or also against other engines in the same ballpark? How many games, i.e. what is the error margin?

AndrewGrant
Posts: 321
Joined: Tue Apr 19, 2016 4:08 am
Contact:

Re: DiscoCheck performing 200+ ELO better on ONE machine ??

Post by AndrewGrant » Tue Nov 14, 2017 2:15 am

500 games 3 times, all resulted in ~90%

Played a 2k game as well, also around ~90%

As for Ethereal vs Discocheck on the non xeon machines,
Ethereal has a 47.38% winrate over the course of 6000 games.

Also, a slight tweak of Ethereal vs Disco check on the non xeon machines
Ethereal has a 47.15% winrate over the course of 5000 games.

AndrewGrant
Posts: 321
Joined: Tue Apr 19, 2016 4:08 am
Contact:

Re: DiscoCheck performing 200+ ELO better on ONE machine ??

Post by AndrewGrant » Tue Nov 14, 2017 2:17 am

Your compile string is just about the way I compiled it.

Even so, compile flags should simply affect the speed of the engine. Something which I am already accounting for.

AndrewGrant
Posts: 321
Joined: Tue Apr 19, 2016 4:08 am
Contact:

Re: DiscoCheck performing 200+ ELO better on ONE machine ??

Post by AndrewGrant » Tue Nov 14, 2017 2:26 am

Here is how cutechess is seeding its RNG

QTime(0,0,0).msecsTo(QTime::currentTime())

My only thought left is that I am EXTREMELY unlucky in the first thousand or so games played againt Ethereal and DiscoCheck.

OR, for some reason cutechess is using the same seed everytime, and I am again VERY unlucky.

Tomorrow I will repeat this test, but passing a seed to the cutechess executable...

User avatar
lucasart
Posts: 2957
Joined: Mon May 31, 2010 11:29 am
Contact:

Re: DiscoCheck performing 200+ ELO better on ONE machine ??

Post by lucasart » Tue Nov 14, 2017 6:19 am

I understand you're building some sort of fishtest. Might be worth keeping track of info like: number of time losses and number of crashes. Perhaps it's Ethereal that has problems on this machine.
Theory and practice sometimes clash. And when that happens, theory loses. Every single time.

AndrewGrant
Posts: 321
Joined: Tue Apr 19, 2016 4:08 am
Contact:

Re: DiscoCheck performing 200+ ELO better on ONE machine ??

Post by AndrewGrant » Tue Nov 14, 2017 7:14 pm

You would be correct. Ethereal is at fault, and I have found the culprit.

At the top of every alphabeta node I do this:

Code: Select all

    // Step 1. Check to see if search time has expired
    if (       Info->searchIsTimeLimited 
        &&  getRealTime() >= Info->endTime2){
        Info->terminateSearch = 1;
        return board->turn == EvaluatingPlayer ? -MATE : MATE;
    }

Code: Select all

double getRealTime(){
#if defined(_WIN32) || defined(_WIN64)
    return (double)(GetTickCount());
#else
    struct timeval tv;
    double secsInMilli, usecsInMilli;
    
    gettimeofday(&tv, NULL);
    secsInMilli = ((double)tv.tv_sec) * 1000;
    usecsInMilli = tv.tv_usec / 1000;
    
    return secsInMilli + usecsInMilli;
#endif
}
On this system it seems that getRealTime() is VERY VERY slow.

For Ethereal-master:
position startpos
go wtime 60000 winc 600
info depth 14 score cp 20 time 1179 nodes 1346799 nps 1141000 hashfull 86 pv e2e4 e7e5 g1f3 b8c6 f1b5 g8f6 e1g1 f8d6 b1c3 e8g8 d2d4 e5d4 f3d4 f8e8
bestmove e2e4
For Ethereal-master:
position startpos
go depth 14
info depth 14 score cp 20 time 658 nodes 1346799 nps 2043000 hashfull 86 pv e2e4 e7e5 g1f3 b8c6 f1b5 g8f6 e1g1 f8d6 b1c3 e8g8 d2d4 e5d4 f3d4 f8e8
bestmove e2e4
Only difference is the nodes per second. When we search to a set depth, i set info->searchIsTimeLimited to false, and getRealTime() is never called.


I ran a 1000 game test against Ethereal-every4096 and Ethereal-master...
Score of every4096 vs master : 792 - 57 - 151 [0.868] 1000
I think that proves it. I'm currently playing games against DiscoCheck now to confirm, and it looks like I'm back to the expected 45% winrate.

Post Reply