Crafty on Cray Blitz vs. Rybka3

[Uri Blass]

What do you mean "crashed"???

Crafty doesn't normally crash, _ever_. Last weekend I played over 1/2 million games on our cluster with no crashes whatsoever. Is this windows or linux???

The Watson dialog came up with the usual options of "terminate" or "debug". This is on a Windows 2008 Server. The UI is Arena. I should have saved the crash dump.

The next time I try out Crafty, I will build it myself. That way, I can attach a debugger to it (with symbols and all) if it crashes.

Note that Arena can be used for matches with unequal time control with the same hardware(ponder off games)

Here are the instructions:

Engine->manage->details->strength->5% mean 20:1 time handicap

Uri

[CThinker]

Note that Arena can be used for matches with unequal time control with the same hardware(ponder off games)

Here are the instructions:

Engine->manage->details->strength->5% mean 20:1 time handicap
Uri

With pondering enabled, enequal time control is not the same as unequal hardware.

To illustrate this, consider giving Rybka 30 seconds per move, and Crafty 10 minutes per move.

Now, when Crafty thinks for 10 minutes, Rybka also ponders for 10 minutes (not 30 seconds).

I think that the only way to test unequal hardware conditions, is to use unequal hardware.

[CThinker]

What do you mean "crashed"???

Crafty doesn't normally crash, _ever_. Last weekend I played over 1/2 million games on our cluster with no crashes whatsoever. Is this windows or linux???

Got another crash. This time with a game against Thinker. Interestingly, Crafty crashed from the same Nunn position that it crashed when playing against Rybka.

I could not debug it because by the time I saw the Watson dialog, several more games have passed, and Arena has re-started Crafty. All I could get was the Watson info, which does not mean much.

Code: Select all

Problem signature:
  Problem Event Name:	APPCRASH
  Application Name:	crafty-22.1-win64.exe
  Application Version:	0.0.0.0
  Application Timestamp:	487c1ccd
  Fault Module Name:	crafty-22.1-win64.exe
  Fault Module Version:	0.0.0.0
  Fault Module Timestamp:	487c1ccd
  Exception Code:	c0000005
  Exception Offset:	000000000009c351
  OS Version:	6.0.6001.2.1.0.274.10
  Locale ID:	1033
  Additional Information 1:	fcd0
  Additional Information 2:	1a944c15a617e01c0054cdfe708a517a
  Additional Information 3:	aac7
  Additional Information 4:	e283361f19c81f3f08b864288701933d

On the same 5:1 hardware advantage for crafty, after 22 games (11 Nunn positions) the result this time against Thinker is:

Code: Select all

               Score     1234567890123456789012
-----------------------------------------------
 1: Thinker  13.0 / 22   0=110011011=110=11010=
 2: Crafty    9.0 / 22   1=001100100=001=00101=
-----------------------------------------------

[Event "Arena tournament"]
[Site "KERWINM-007"]
[Date "2008.10.08"]
[Round "11"]
[White "RelayCrafty"]
[Black "RelayThinker"]
[Result "0-1"]
[Time "01:39:15"]
[TimeControl "1200+2"]
[SetUp "1"]
[Termination "time forfeit"]
[PlyCount "62"]
[WhiteType "program"]
[BlackType "program"]
[FEN "2kr3r/ppqn1pp1/2pbp2p/7P/3PQ3/5NP1/PPPB1P2/2KR3R w - - 0 16"]
16.Qe2 Kb8 17.c4 c5 18.Bc3 Rhg8 19.Kb1 cxd4 20.Nxd4 a6 21.Nb3 Qc6 22.Rhe1 Nf6 23.Bd4 Bb4 24.Rh1 Ka8 25.c5 Ne4 26.Rh4 f5 27.a3 e5 28.axb4 exd4 29.Rf4 Qe6 30.Qc2 d3 31.Rxd3 Rxd3 32.Qxd3 Nxf2 33.Rxf2 Qe1+ 34.Ka2 Qxf2 35.Na5 Rb8 36.c6 Qb6 37.cxb7+ Rxb7 38.Nxb7 Kxb7 39.Qd7+ Qc7 40.Qxc7+ Kxc7 41.Kb3 Kd7 42.Ka4 Ke6 43.b5 axb5+ 44.Kxb5 g6 45.Kc4 gxh5 46.Kd4 f4 0-1

[Event "Arena tournament"]
[Site "KERWINM-007"]
[Date "2008.10.07"]
[Round "11"]
[White "RelayCrafty"]
[Black "RelayRybka"]
[Result "0-1"]
[Time "14:20:04"]
[TimeControl "600+2"]
[SetUp "1"]
[Termination "time forfeit"]
[PlyCount "46"]
[WhiteType "program"]
[BlackType "program"]
[FEN "2kr3r/ppqn1pp1/2pbp2p/7P/3PQ3/5NP1/PPPB1P2/2KR3R w - - 0 16"]
16.Kb1 Nf6 17.Qe2 Kb8 18.Ne5 Bxe5 19.dxe5 Nd5 20.c4 Ne7 21.Bc3 c5 22.Rd6 Nf5 23.Rhd1 Kc8 24.R6d3 Rxd3 25.Qxd3 Rd8 26.Qxd8+ Qxd8 27.Rxd8+ Kxd8 28. Kc2 Ne7 29.Kd3 Kd7 30.g4 Kc8 31.Ke4 Kd7 32.Bd2 b6 33.f3 Kd8 34.f4 g6 35.g5 Nf5 36.hxg6 fxg6 37.gxh6 Nxh6 38.Ke3 Nf5+ 0-1

[bob]

What do you mean "crashed"???

Crafty doesn't normally crash, _ever_. Last weekend I played over 1/2 million games on our cluster with no crashes whatsoever. Is this windows or linux???

Got another crash. This time with a game against Thinker. Interestingly, Crafty crashed from the same Nunn position that it crashed when playing against Rybka.

I could not debug it because by the time I saw the Watson dialog, several more games have passed, and Arena has re-started Crafty. All I could get was the Watson info, which does not mean much.

Code: Select all

Problem signature:
  Problem Event Name:	APPCRASH
  Application Name:	crafty-22.1-win64.exe
  Application Version:	0.0.0.0
  Application Timestamp:	487c1ccd
  Fault Module Name:	crafty-22.1-win64.exe
  Fault Module Version:	0.0.0.0
  Fault Module Timestamp:	487c1ccd
  Exception Code:	c0000005
  Exception Offset:	000000000009c351
  OS Version:	6.0.6001.2.1.0.274.10
  Locale ID:	1033
  Additional Information 1:	fcd0
  Additional Information 2:	1a944c15a617e01c0054cdfe708a517a
  Additional Information 3:	aac7
  Additional Information 4:	e283361f19c81f3f08b864288701933d

On the same 5:1 hardware advantage for crafty, after 22 games (11 Nunn positions) the result this time against Thinker is:

Code: Select all

               Score     1234567890123456789012
-----------------------------------------------
 1: Thinker  13.0 / 22   0=110011011=110=11010=
 2: Crafty    9.0 / 22   1=001100100=001=00101=
-----------------------------------------------

[Event "Arena tournament"]
[Site "KERWINM-007"]
[Date "2008.10.08"]
[Round "11"]
[White "RelayCrafty"]
[Black "RelayThinker"]
[Result "0-1"]
[Time "01:39:15"]
[TimeControl "1200+2"]
[SetUp "1"]
[Termination "time forfeit"]
[PlyCount "62"]
[WhiteType "program"]
[BlackType "program"]
[FEN "2kr3r/ppqn1pp1/2pbp2p/7P/3PQ3/5NP1/PPPB1P2/2KR3R w - - 0 16"]
16.Qe2 Kb8 17.c4 c5 18.Bc3 Rhg8 19.Kb1 cxd4 20.Nxd4 a6 21.Nb3 Qc6 22.Rhe1 Nf6 23.Bd4 Bb4 24.Rh1 Ka8 25.c5 Ne4 26.Rh4 f5 27.a3 e5 28.axb4 exd4 29.Rf4 Qe6 30.Qc2 d3 31.Rxd3 Rxd3 32.Qxd3 Nxf2 33.Rxf2 Qe1+ 34.Ka2 Qxf2 35.Na5 Rb8 36.c6 Qb6 37.cxb7+ Rxb7 38.Nxb7 Kxb7 39.Qd7+ Qc7 40.Qxc7+ Kxc7 41.Kb3 Kd7 42.Ka4 Ke6 43.b5 axb5+ 44.Kxb5 g6 45.Kc4 gxh5 46.Kd4 f4 0-1

[Event "Arena tournament"]
[Site "KERWINM-007"]
[Date "2008.10.07"]
[Round "11"]
[White "RelayCrafty"]
[Black "RelayRybka"]
[Result "0-1"]
[Time "14:20:04"]
[TimeControl "600+2"]
[SetUp "1"]
[Termination "time forfeit"]
[PlyCount "46"]
[WhiteType "program"]
[BlackType "program"]
[FEN "2kr3r/ppqn1pp1/2pbp2p/7P/3PQ3/5NP1/PPPB1P2/2KR3R w - - 0 16"]
16.Kb1 Nf6 17.Qe2 Kb8 18.Ne5 Bxe5 19.dxe5 Nd5 20.c4 Ne7 21.Bc3 c5 22.Rd6 Nf5 23.Rhd1 Kc8 24.R6d3 Rxd3 25.Qxd3 Rd8 26.Qxd8+ Qxd8 27.Rxd8+ Kxd8 28. Kc2 Ne7 29.Kd3 Kd7 30.g4 Kc8 31.Ke4 Kd7 32.Bd2 b6 33.f3 Kd8 34.f4 g6 35.g5 Nf5 36.hxg6 fxg6 37.gxh6 Nxh6 38.Ke3 Nf5+ 0-1

About all that tells me is that I would not be using that version of Crafty for testing. over the past two weeks, I have played over 3,000,000 games, with not a single crash. About 400,000 of the games had crafty using two CPUs. No idea what is causing the problem, but something certainly seems wrong. Whether it is hardware, compiler or program I have no idea... but certainly something is broken, based on the more normal results I have been seeing. I have not seed crafty crash in years except for the smpnice=1 fiasco. Which now works perfectly on 22.2 by the way...

[bob]

What do you mean "crashed"???

Crafty doesn't normally crash, _ever_. Last weekend I played over 1/2 million games on our cluster with no crashes whatsoever. Is this windows or linux???

Got another crash. This time with a game against Thinker. Interestingly, Crafty crashed from the same Nunn position that it crashed when playing against Rybka.

I could not debug it because by the time I saw the Watson dialog, several more games have passed, and Arena has re-started Crafty. All I could get was the Watson info, which does not mean much.

Code: Select all

Problem signature:
  Problem Event Name:	APPCRASH
  Application Name:	crafty-22.1-win64.exe
  Application Version:	0.0.0.0
  Application Timestamp:	487c1ccd
  Fault Module Name:	crafty-22.1-win64.exe
  Fault Module Version:	0.0.0.0
  Fault Module Timestamp:	487c1ccd
  Exception Code:	c0000005
  Exception Offset:	000000000009c351
  OS Version:	6.0.6001.2.1.0.274.10
  Locale ID:	1033
  Additional Information 1:	fcd0
  Additional Information 2:	1a944c15a617e01c0054cdfe708a517a
  Additional Information 3:	aac7
  Additional Information 4:	e283361f19c81f3f08b864288701933d

On the same 5:1 hardware advantage for crafty, after 22 games (11 Nunn positions) the result this time against Thinker is:

Code: Select all

               Score     1234567890123456789012
-----------------------------------------------
 1: Thinker  13.0 / 22   0=110011011=110=11010=
 2: Crafty    9.0 / 22   1=001100100=001=00101=
-----------------------------------------------

[Event "Arena tournament"]
[Site "KERWINM-007"]
[Date "2008.10.08"]
[Round "11"]
[White "RelayCrafty"]
[Black "RelayThinker"]
[Result "0-1"]
[Time "01:39:15"]
[TimeControl "1200+2"]
[SetUp "1"]
[Termination "time forfeit"]
[PlyCount "62"]
[WhiteType "program"]
[BlackType "program"]
[FEN "2kr3r/ppqn1pp1/2pbp2p/7P/3PQ3/5NP1/PPPB1P2/2KR3R w - - 0 16"]
16.Qe2 Kb8 17.c4 c5 18.Bc3 Rhg8 19.Kb1 cxd4 20.Nxd4 a6 21.Nb3 Qc6 22.Rhe1 Nf6 23.Bd4 Bb4 24.Rh1 Ka8 25.c5 Ne4 26.Rh4 f5 27.a3 e5 28.axb4 exd4 29.Rf4 Qe6 30.Qc2 d3 31.Rxd3 Rxd3 32.Qxd3 Nxf2 33.Rxf2 Qe1+ 34.Ka2 Qxf2 35.Na5 Rb8 36.c6 Qb6 37.cxb7+ Rxb7 38.Nxb7 Kxb7 39.Qd7+ Qc7 40.Qxc7+ Kxc7 41.Kb3 Kd7 42.Ka4 Ke6 43.b5 axb5+ 44.Kxb5 g6 45.Kc4 gxh5 46.Kd4 f4 0-1

[Event "Arena tournament"]
[Site "KERWINM-007"]
[Date "2008.10.07"]
[Round "11"]
[White "RelayCrafty"]
[Black "RelayRybka"]
[Result "0-1"]
[Time "14:20:04"]
[TimeControl "600+2"]
[SetUp "1"]
[Termination "time forfeit"]
[PlyCount "46"]
[WhiteType "program"]
[BlackType "program"]
[FEN "2kr3r/ppqn1pp1/2pbp2p/7P/3PQ3/5NP1/PPPB1P2/2KR3R w - - 0 16"]
16.Kb1 Nf6 17.Qe2 Kb8 18.Ne5 Bxe5 19.dxe5 Nd5 20.c4 Ne7 21.Bc3 c5 22.Rd6 Nf5 23.Rhd1 Kc8 24.R6d3 Rxd3 25.Qxd3 Rd8 26.Qxd8+ Qxd8 27.Rxd8+ Kxd8 28. Kc2 Ne7 29.Kd3 Kd7 30.g4 Kc8 31.Ke4 Kd7 32.Bd2 b6 33.f3 Kd8 34.f4 g6 35.g5 Nf5 36.hxg6 fxg6 37.gxh6 Nxh6 38.Ke3 Nf5+ 0-1

About all that tells me is that I would not be using that version of Crafty for testing. over the past two weeks, I have played over 3,000,000 games, with not a single crash. About 400,000 of the games had crafty using two CPUs. No idea what is causing the problem, but something certainly seems wrong. Whether it is hardware, compiler or program I have no idea... but certainly something is broken, based on the more normal results I have been seeing. I have not seed crafty crash in years except for the smpnice=1 fiasco. Which now works perfectly on 22.2 by the way... I have played several hundred thousand games using mt=2 and smpnice=1 with ponder=off to make sure nothing hangs or fails, and I have seen zero crashes or losses on time which shows nothing is hanging up either.

[CThinker]

This time, I have a 10:1 hardware advantage for Crafty.

Code: Select all

              Score     123456789012345678901234567890
------------------------------------------------------
 1: Rybka   22.5 / 30   0111==1101=11111=1=1=1=101=11=
 2: Crafty   7.5 / 30   1000==0010=00000=0=0=0=010=00=
------------------------------------------------------

Rybka 2.2 (free version) ran on an old Vaio Notebook (P3M) while Crafty 22.1 ran on a quadcore HP Optiplex (Q6600) using all cores (4 threads).

Time control is 15+2, on 15 Nunn positions.

[bob]

This time, I have a 10:1 hardware advantage for Crafty.

Code: Select all

              Score     123456789012345678901234567890
------------------------------------------------------
 1: Rybka   22.5 / 30   0111==1101=11111=1=1=1=101=11=
 2: Crafty   7.5 / 30   1000==0010=00000=0=0=0=010=00=
------------------------------------------------------

Rybka 2.2 (free version) ran on an old Vaio Notebook (P3M) while Crafty 22.1 ran on a quadcore HP Optiplex (Q6600) using all cores (4 threads).

Time control is 15+2, on 15 Nunn positions.

Remember that this is not exactly 10:1. Given the choice of a 10ghz processor, or 10 1ghz processors, I would take the single 10ghz every time. Crafty loses about 30% for each additional processor. So four really acts like three (this is probably true of most SMP programs). It is not as easy to compare this way. If I compare I either always use single-cpu for both, or multiple cpu for both. Then the speed comparisons make sense. Otherwise the numbers are off by a significant amount (25% in this case).

[Eizenhammer]

This time, I have a 10:1 hardware advantage for Crafty.

Code: Select all

              Score     123456789012345678901234567890
------------------------------------------------------
 1: Rybka   22.5 / 30   0111==1101=11111=1=1=1=101=11=
 2: Crafty   7.5 / 30   1000==0010=00000=0=0=0=010=00=
------------------------------------------------------

Rybka 2.2 (free version) ran on an old Vaio Notebook (P3M) while Crafty 22.1 ran on a quadcore HP Optiplex (Q6600) using all cores (4 threads).

Time control is 15+2, on 15 Nunn positions.

thanks for testing this, btw. , it is always good to see something more solid than wild guessing and exuberant imagination

[glorfindel]

This time, I have a 10:1 hardware advantage for Crafty.

Code: Select all

              Score     123456789012345678901234567890
------------------------------------------------------
 1: Rybka   22.5 / 30   0111==1101=11111=1=1=1=101=11=
 2: Crafty   7.5 / 30   1000==0010=00000=0=0=0=010=00=
------------------------------------------------------

Rybka 2.2 (free version) ran on an old Vaio Notebook (P3M) while Crafty 22.1 ran on a quadcore HP Optiplex (Q6600) using all cores (4 threads).

Time control is 15+2, on 15 Nunn positions.

Remember that this is not exactly 10:1. Given the choice of a 10ghz processor, or 10 1ghz processors, I would take the single 10ghz every time. Crafty loses about 30% for each additional processor. So four really acts like three (this is probably true of most SMP programs). It is not as easy to compare this way. If I compare I either always use single-cpu for both, or multiple cpu for both. Then the speed comparisons make sense. Otherwise the numbers are off by a significant amount (25% in this case).

But Lance has said previously that "Crafty bench shows 0.45M nps on the Vaio, and 2.35M nps (2 threads) on the HP" and this is how he had calculated that Crafty had a 5:1 advantage in the previous match. I assume the same method (a benchmark) was used in this case too, and it is really a 10:1 advantage.

[bob]

This time, I have a 10:1 hardware advantage for Crafty.

Code: Select all

              Score     123456789012345678901234567890
------------------------------------------------------
 1: Rybka   22.5 / 30   0111==1101=11111=1=1=1=101=11=
 2: Crafty   7.5 / 30   1000==0010=00000=0=0=0=010=00=
------------------------------------------------------

Rybka 2.2 (free version) ran on an old Vaio Notebook (P3M) while Crafty 22.1 ran on a quadcore HP Optiplex (Q6600) using all cores (4 threads).

Time control is 15+2, on 15 Nunn positions.

Remember that this is not exactly 10:1. Given the choice of a 10ghz processor, or 10 1ghz processors, I would take the single 10ghz every time. Crafty loses about 30% for each additional processor. So four really acts like three (this is probably true of most SMP programs). It is not as easy to compare this way. If I compare I either always use single-cpu for both, or multiple cpu for both. Then the speed comparisons make sense. Otherwise the numbers are off by a significant amount (25% in this case).

But Lance has said previously that "Crafty bench shows 0.45M nps on the Vaio, and 2.35M nps (2 threads) on the HP" and this is how he had calculated that Crafty had a 5:1 advantage in the previous match. I assume the same method (a benchmark) was used in this case too, and it is really a 10:1 advantage.

No it isn't. Crafty's NPS scaling is very good. But parallel search efficiency does not scale as well as NPS for _any_ program. 4 cpus at X gigahertz is roughly the same as one cpu at 3X gigahertz. Even though the NPS will be almost 4x faster. But the trees searched will be 33% larger.

As I said, it is closer to 7x faster...