Crafty on Cray Blitz vs. Rybka3

Discussion of anything and everything relating to chess playing software and machines.

Moderator: Ras

bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Crafty on Cray Blitz vs. Rybka3

Post by bob »

Eizenhammer wrote:
bob wrote:
BubbaTough wrote:
Or not...
Hehe, cannot argue with that. Trying to translate speed into ELO is a pretty funny business.

-Sam
My point was a bit subtle for most. A cluster is _not_ the best platform to run on. I have run on machines with 64 cores and shared memory. That is _far_ faster than a cluster with 64 cores and message passing, even if using something like Infiniband. Testers don't have a "good book" for Crafty. Too many variables to take a rating list and use that to estimate actual Elo differences in a real tournament.
I dont see how all this guessing could change the fact that rybka is about as strong as Uri suggests. Everybody who has both programs can see it, i ran a match on equal hardware, old hardware, ok, and in 50 games crafty got a draw, or maybe two, nothing more: Now, by using long books and effective learning one can obfuscate this difference, of course, but it does hardly change the basic facts. I would really be surprised if Crafty could win a long match, using the Silver suite or similar neutral suites, on all its 64 cores against a decent 64Bit single version of rybka ...
sorry, but sometimes one has to say the truth, even if it hurts.

Peter
If you run crafty on a 64 core system, and Rybka on one, the truth will be far different than what you imagine.
CThinker
Posts: 388
Joined: Wed Mar 08, 2006 10:08 pm

Re: Crafty on Cray Blitz vs. Rybka3

Post by CThinker »

BubbaTough wrote:
Or not...
Hehe, cannot argue with that. Trying to translate speed into ELO is a pretty funny business.

-Sam
I don't have data on Rybka3 vs Crafty22.1 with 1:30 H/W difference.

What I do have now is Rybka2.2 (free version) vs Crafty22.1 with 1:5 H/W difference, and the result so far is that Rybka is way ahead. So, I'm speculating that Rybka3 (being much better than Rybka2.2) can afford to be still better than Crafty22.1 even with a 10:1 H/W difference (or 20:1, 30:1, if I can find the machine pair with that much gap to test on).

My test rig involves an old 1ghz P3M Vaio notebook, running Rybka2.2, vs a 2.4ghz q6600 HP Optiplex, running Crafty22.1. Crafty bench shows 0.45M nps on the Vaio, and 2.35M nps (2 threads) on the HP. That translates to more than 5 to 1 HW difference. Games are for the 20 Nunn positions, and time control is 0 10 2, with pondering enabled.

Tournament UI is Arena, running on a third machine. There is no chess UI running with Rybka or Crafty - just RemoteThinker.

After 8 games, it is 6-0-2, in favor or Rybka2.2. The test is still running...
Uri Blass
Posts: 10900
Joined: Thu Mar 09, 2006 12:37 am
Location: Tel-Aviv Israel

Re: Crafty on Cray Blitz vs. Rybka3

Post by Uri Blass »

CThinker wrote:
BubbaTough wrote:
Or not...
Hehe, cannot argue with that. Trying to translate speed into ELO is a pretty funny business.

-Sam
I don't have data on Rybka3 vs Crafty22.1 with 1:30 H/W difference.

What I do have now is Rybka2.2 (free version) vs Crafty22.1 with 1:5 H/W difference, and the result so far is that Rybka is way ahead. So, I'm speculating that Rybka3 (being much better than Rybka2.2) can afford to be still better than Crafty22.1 even with a 10:1 H/W difference (or 20:1, 30:1, if I can find the machine pair with that much gap to test on).

My test rig involves an old 1ghz P3M Vaio notebook, running Rybka2.2, vs a 2.4ghz q6600 HP Optiplex, running Crafty22.1. Crafty bench shows 0.45M nps on the Vaio, and 2.35M nps (2 threads) on the HP. That translates to more than 5 to 1 HW difference. Games are for the 20 Nunn positions, and time control is 0 10 2, with pondering enabled.

Tournament UI is Arena, running on a third machine. There is no chess UI running with Rybka or Crafty - just RemoteThinker.

After 8 games, it is 6-0-2, in favor or Rybka2.2. The test is still running...
I am not surprised.

My guess is that Crafty may need 100:1 hardware advantage to
make a match against rybka3 interesting when slower time control is going to help rybka.

The difference for rybka2.2 may be something like 20:1 or 25:1 because I guess that rybka3 is about 4-5 times better than free rybka.

Uri
CThinker
Posts: 388
Joined: Wed Mar 08, 2006 10:08 pm

Re: Crafty on Cray Blitz vs. Rybka3

Post by CThinker »

Uri Blass wrote:
CThinker wrote:
BubbaTough wrote:
Or not...
Hehe, cannot argue with that. Trying to translate speed into ELO is a pretty funny business.

-Sam
I don't have data on Rybka3 vs Crafty22.1 with 1:30 H/W difference.

What I do have now is Rybka2.2 (free version) vs Crafty22.1 with 1:5 H/W difference, and the result so far is that Rybka is way ahead. So, I'm speculating that Rybka3 (being much better than Rybka2.2) can afford to be still better than Crafty22.1 even with a 10:1 H/W difference (or 20:1, 30:1, if I can find the machine pair with that much gap to test on).

My test rig involves an old 1ghz P3M Vaio notebook, running Rybka2.2, vs a 2.4ghz q6600 HP Optiplex, running Crafty22.1. Crafty bench shows 0.45M nps on the Vaio, and 2.35M nps (2 threads) on the HP. That translates to more than 5 to 1 HW difference. Games are for the 20 Nunn positions, and time control is 0 10 2, with pondering enabled.

Tournament UI is Arena, running on a third machine. There is no chess UI running with Rybka or Crafty - just RemoteThinker.

After 8 games, it is 6-0-2, in favor or Rybka2.2. The test is still running...
I am not surprised.

My guess is that Crafty may need 100:1 hardware advantage to
make a match against rybka3 interesting when slower time control is going to help rybka.

The difference for rybka2.2 may be something like 20:1 or 25:1 because I guess that rybka3 is about 4-5 times better than free rybka.

Uri
I stoped the test after 18 games (from 9 Nunn positions). Crafty crashed at game 11. At game 13, Rybka is up be a pawn, but it was a drawn endgame, so, I adjudicated it as such.

The result looks like this:

Code: Select all

                  Score     123456789012345678
----------------------------------------------
 1: Rybka2.2    13.5 / 18   11=1=1111011=10110
 2: Crafty22.1   4.5 / 18   00=0=0000100=01001
----------------------------------------------
If I find the time to dust off an even older notebook (500mhz) and get Crafty to use 4 threads withouth crashing, then we might see a 1:20 test.
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Crafty on Cray Blitz vs. Rybka3

Post by bob »

CThinker wrote:
Uri Blass wrote:
CThinker wrote:
BubbaTough wrote:
Or not...
Hehe, cannot argue with that. Trying to translate speed into ELO is a pretty funny business.

-Sam
I don't have data on Rybka3 vs Crafty22.1 with 1:30 H/W difference.

What I do have now is Rybka2.2 (free version) vs Crafty22.1 with 1:5 H/W difference, and the result so far is that Rybka is way ahead. So, I'm speculating that Rybka3 (being much better than Rybka2.2) can afford to be still better than Crafty22.1 even with a 10:1 H/W difference (or 20:1, 30:1, if I can find the machine pair with that much gap to test on).

My test rig involves an old 1ghz P3M Vaio notebook, running Rybka2.2, vs a 2.4ghz q6600 HP Optiplex, running Crafty22.1. Crafty bench shows 0.45M nps on the Vaio, and 2.35M nps (2 threads) on the HP. That translates to more than 5 to 1 HW difference. Games are for the 20 Nunn positions, and time control is 0 10 2, with pondering enabled.

Tournament UI is Arena, running on a third machine. There is no chess UI running with Rybka or Crafty - just RemoteThinker.

After 8 games, it is 6-0-2, in favor or Rybka2.2. The test is still running...
I am not surprised.

My guess is that Crafty may need 100:1 hardware advantage to
make a match against rybka3 interesting when slower time control is going to help rybka.

The difference for rybka2.2 may be something like 20:1 or 25:1 because I guess that rybka3 is about 4-5 times better than free rybka.

Uri
I stoped the test after 18 games (from 9 Nunn positions). Crafty crashed at game 11. At game 13, Rybka is up be a pawn, but it was a drawn endgame, so, I adjudicated it as such.

The result looks like this:

Code: Select all

                  Score     123456789012345678
----------------------------------------------
 1: Rybka2.2    13.5 / 18   11=1=1111011=10110
 2: Crafty22.1   4.5 / 18   00=0=0000100=01001
----------------------------------------------
If I find the time to dust off an even older notebook (500mhz) and get Crafty to use 4 threads withouth crashing, then we might see a 1:20 test.
What do you mean "crashed"???

Crafty doesn't normally crash, _ever_. Last weekend I played over 1/2 million games on our cluster with no crashes whatsoever. Is this windows or linux???
CThinker
Posts: 388
Joined: Wed Mar 08, 2006 10:08 pm

Re: Crafty on Cray Blitz vs. Rybka3

Post by CThinker »

bob wrote: What do you mean "crashed"???

Crafty doesn't normally crash, _ever_. Last weekend I played over 1/2 million games on our cluster with no crashes whatsoever. Is this windows or linux???
The Watson dialog came up with the usual options of "terminate" or "debug". This is on a Windows 2008 Server. The UI is Arena. I should have saved the crash dump.

The next time I try out Crafty, I will build it myself. That way, I can attach a debugger to it (with symbols and all) if it crashes.
glorfindel

Re: Crafty on Cray Blitz vs. Rybka3

Post by glorfindel »

bob wrote:What do you mean "crashed"???

Crafty doesn't normally crash, _ever_. Last weekend I played over 1/2 million games on our cluster with no crashes whatsoever. Is this windows or linux???
I have also had a problem with Crafty. Crafty used to be stable for me, but 22.1 occasionally crashes when playing, and it cannot communicate properly with XBoard in analysis mode after I take back a move.

I have sent you and Tracy a detailed e-mail on Sept 27, but perhaps it got lost in a spam filter? If you want I can send it again.
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Crafty on Cray Blitz vs. Rybka3

Post by bob »

glorfindel wrote:
bob wrote:What do you mean "crashed"???

Crafty doesn't normally crash, _ever_. Last weekend I played over 1/2 million games on our cluster with no crashes whatsoever. Is this windows or linux???
I have also had a problem with Crafty. Crafty used to be stable for me, but 22.1 occasionally crashes when playing, and it cannot communicate properly with XBoard in analysis mode after I take back a move.

I have sent you and Tracy a detailed e-mail on Sept 27, but perhaps it got lost in a spam filter? If you want I can send it again.
By all means. There is a known issue with smpnice=1, ponder=off, with mt set to something > 1. I think the new version will work just fine since I have gone back to posix threads since the development of the NPTL stuff for linux eliminated my old complaints...
glorfindel

Re: Crafty on Cray Blitz vs. Rybka3

Post by glorfindel »

bob wrote:By all means. There is a known issue with smpnice=1, ponder=off, with mt set to something > 1. I think the new version will work just fine since I have gone back to posix threads since the development of the NPTL stuff for linux eliminated my old complaints...
I have resent the e-mail, although it was exactly the kind of problem you described. I didn't know it was a known issue.

If I set smpnice=0 the problem goes away, but this is not the ideal solution, because e.g. if I am analyzing a tablebase position on a quad, one core is almost idle (the one that looks up the tablebase I guess) but the other three are at 100% usage not doing anything useful.
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Crafty on Cray Blitz vs. Rybka3

Post by bob »

glorfindel wrote:
bob wrote:By all means. There is a known issue with smpnice=1, ponder=off, with mt set to something > 1. I think the new version will work just fine since I have gone back to posix threads since the development of the NPTL stuff for linux eliminated my old complaints...
I have resent the e-mail, although it was exactly the kind of problem you described. I didn't know it was a known issue.

If I set smpnice=0 the problem goes away, but this is not the ideal solution, because e.g. if I am analyzing a tablebase position on a quad, one core is almost idle (the one that looks up the tablebase I guess) but the other three are at 100% usage not doing anything useful.
Someone actually fixed the problem (it was simple to fix) and released a patched version of 22.1 that did not do that. We should have 22.2 ready for release right after the ACCA event in November. It will play significantly better and also not have this problem at all...