Stockfish 1.6s is a disaster, compared to 1.6

[George Tsavdaris]

Stockfish Jim Ablett compiles (x64), against Rybka 3 (x64)
100 games, 2'+1", no ponder, 2-cpu, book PB-5moves
Fritz 11 GUI, XP Pro x64.

First stockfish-16-x64-ja.exe (23/12/2009, 267 KB)
Stockfish 1.6 JA - Rybka 3: +20 -41 =39, score 39.5-60.5 (-76 Elo)

Then stockfish-16s-x64-ja.exe (29/12/09, 273 KB)
Stockfish 1.6s JA - Rybka 3: +14 -51 =35, score 31.5-68.5 (-137 Elo)

Only 200 games, but the difference is big enough to call the "s" version a failure!

I now tend to believe that only the 32bit version(the first one i don't know about the second) is an improvement.
See some results for it here for example. It seems an improvement.

But then again the situation seems a bit odd since in the Silver match they(S1.6 32bit and S1.6s 32bit) are almost equal after 44 games.

And even more crazy is this start of another test. Each engine will play 40 games so it's just the beginning, but nevertheless result is a bit strange.

Code: Select all

Stockfish 1.6 JA      ****** 110½01 1111½½   8.5/12   
Rybka 3 1-cpu 32-bit  001½10 ****** 10110½   6.0/12  
Stockfish 1.6s        0000½½ 01001½ ******   3.5/12

And yet i see this! or this....

So how more confusing can it be?

[royb]

Stockfish Jim Ablett compiles (x64), against Rybka 3 (x64)
100 games, 2'+1", no ponder, 2-cpu, book PB-5moves
Fritz 11 GUI, XP Pro x64.

First stockfish-16-x64-ja.exe (23/12/2009, 267 KB)
Stockfish 1.6 JA - Rybka 3: +20 -41 =39, score 39.5-60.5 (-76 Elo)

Then stockfish-16s-x64-ja.exe (29/12/09, 273 KB)
Stockfish 1.6s JA - Rybka 3: +14 -51 =35, score 31.5-68.5 (-137 Elo)

Only 200 games, but the difference is big enough to call the "s" version a failure!

I now tend to believe that only the 32bit version(the first one i don't know about the second) is an improvement.
See some results for it here for example. It seems an improvement.

But then again the situation seems a bit odd since in the Silver match they(S1.6 32bit and S1.6s 32bit) are almost equal after 44 games.

And even more crazy is this start of another test. Each engine will play 40 games so it's just the beginning, but nevertheless result is a bit strange.

Code: Select all

Stockfish 1.6 JA      ****** 110½01 1111½½   8.5/12   
Rybka 3 1-cpu 32-bit  001½10 ****** 10110½   6.0/12  
Stockfish 1.6s        0000½½ 01001½ ******   3.5/12

And yet i see this! or this....

So how more confusing can it be?

I am running a test with Stockfish-1.6.2-official and Stockfish-1.6.2sdc and Rybka-3 at 60sec+1sec. After 378 games, Stockfish-1.6.2-sdc scores 54.8% vs Stockfish-1.6.2 and 34.7% vs Rybka-3. Stockfish-1.6.2-official scores 40.5% vs Rybka-3 and 45.2% vs Stockfish-1.6.2-sdc.

There may be a bit of an ELO increase for Stockfish-1.6.2sdc but it would seem to not be very large at this time. Of course many more games are needed to clarify and I intend to let this match continue for another day or two.

Roy

[ernest]

After 378 games, Stockfish-1.6.2-sdc scores 54.8% vs Stockfish-1.6.2 and 34.7% vs Rybka-3. Stockfish-1.6.2-official scores 40.5% vs Rybka-3 and 45.2% vs Stockfish-1.6.2-sdc.Roy

Well, against Rybka 3,
Stockfish-1.6.2-sdc scores 34.7%
Stockfish-1.6.2 scores 40.5%
That looks almost as bad for "sdc" as in my games!

I finally made 100 games (same conditions as in my previous games) between the 64-bit Stockfish 1.6s JA and the 64-bit Stockfish 1.6 JA:
result was practically (and statistically) even:
Stockfish 1.6s JA - Stockfish 1.6 JA : +18 -22 =60, score 48-52 (-14 Elo)

[royb]

After 378 games, Stockfish-1.6.2-sdc scores 54.8% vs Stockfish-1.6.2 and 34.7% vs Rybka-3. Stockfish-1.6.2-official scores 40.5% vs Rybka-3 and 45.2% vs Stockfish-1.6.2-sdc.Roy

Well, against Rybka 3,
Stockfish-1.6.2-sdc scores 34.7%
Stockfish-1.6.2 scores 40.5%
That looks almost as bad for "sdc" as in my games!

I finally made 100 games (same conditions as in my previous games) between the 64-bit Stockfish 1.6s JA and the 64-bit Stockfish 1.6 JA:
result was practically (and statistically) even:
Stockfish 1.6s JA - Stockfish 1.6 JA : +18 -22 =60, score 48-52 (-14 Elo)

Now with 628 games played, Stockfish-1.6.2-sdc has scored 35.1% against Rybka-3 and 51.6% against Stockfish-1.6.2-official.

The two flavors of Stockfish are very close in strength it would seem. I will continue to let games run for the next day or so and see where things end up with something close to 1000 games.