Stockfish 2.3.1 weaker than 2.2.2?

Peter Skinner · Post by **Peter Skinner** » Sat Sep 29, 2012 4:49 pm

Graham Banks wrote:
Sorry. I have no need for the linux binaries, so I didn't extract them at the time. Perhaps somebody else will be kind enough to help you out.

No problem. Thanks for the download you did supply. I can use that on my Windows machine until I find the other package. It's better than nothing

Peter

Graham Banks · Post by **Graham Banks** » Sat Sep 29, 2012 7:53 pm

Peter Skinner wrote:
Graham Banks wrote:
Sorry. I have no need for the linux binaries, so I didn't extract them at the time. Perhaps somebody else will be kind enough to help you out.
No problem. Thanks for the download you did supply. I can use that on my Windows machine until I find the other package. It's better than nothing

Peter

Here is the complete package, courtesy of Adam Hair:
http://kirill-kryukov.com/chess/discuss ... p?id=27599

Adam Hair · Post by **Adam Hair** » Sat Sep 29, 2012 7:56 pm

Peter Skinner wrote:
Graham Banks wrote:
Sorry. I have no need for the linux binaries, so I didn't extract them at the time. Perhaps somebody else will be kind enough to help you out.
No problem. Thanks for the download you did supply. I can use that on my Windows machine until I find the other package. It's better than nothing

Peter

Peter,

I just added my Stockfish 2.2.2 zip file from Jim to Graham's thread at the CCRL forum. It contains the Linux compiles.

Adam

Joerg Oster · Post by **Joerg Oster** » Sat Sep 29, 2012 11:26 pm

Hi Eelco,

if you can provide the changes (I only could see 1 change in evaluate.cpp) I will run a testmatch with cutechess-cli under Linux.

Eelco de Groot · Post by **Eelco de Groot** » Sat Sep 29, 2012 11:53 pm

Hi Joerg,

That would be great! I am still running a match here, but at 2 min + 10 seconds Fischerbonus it is really taking all day

I am just half way through a short match, the object is just 60 games but have not reached 30 yet. But this is comparatively long time control so that is also good to have. Joerg I just copied the changes from the link that Gary provided without whitespaces added: https://github.com/RyanTaker/RyanFish/c ... ndgame?w=1

Can you see the changes there? I have not actually added them to the pure Stockfish files yet, my evaluate.cpp is slightly different from the standard one. Maybe it is a good idea to post the source I used somewhere later, on the TDBB maybe. But from Gary's link I think it is little work to copy those changes over? You can also see the full files I think with the view file @ links on that page, and if necessary copy the file to clipboard.

It would be great to have some more significant data!

Eelco

Joerg Oster · Post by **Joerg Oster** » Sun Sep 30, 2012 3:37 pm

Hi Eelco,

thank you for providing the necessary changes.

Below the output of a first testmatch between Stockfish 2.3.1 default and Stockfish 2.3.1 50MR (50-move-rule). Time control was 80 sec + 0.5 sec increment.

./cutechess-cli.sh -engine name=stockfish cmd=./stockfish proto=uci option.Threads=1 option.Hash=256 tc=80+0.5 -engine name=stockfish-50mr cmd=./stockfish-50mr proto=uci option.Threads=1 option.Hash=256 tc=80+0.5 -pgnin klo_25_eco_a00-e97_variations.pgn -repeat -games 50 -concurrency 4 -wait 50 -pgnout 80+0.5_SF_SF-50MR_50.pgn
Started game 1 of 50 (stockfish vs stockfish-50mr)
Started game 2 of 50 (stockfish-50mr vs stockfish)
Started game 3 of 50 (stockfish vs stockfish-50mr)
Started game 4 of 50 (stockfish-50mr vs stockfish)
Finished game 1 (stockfish vs stockfish-50mr): 0-1 {Black mates}
Score of stockfish vs stockfish-50mr: 0 - 1 - 0 [0.00] 1
Started game 5 of 50 (stockfish vs stockfish-50mr)
Finished game 2 (stockfish-50mr vs stockfish): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 0 - 1 - 1 [0.25] 2
Started game 6 of 50 (stockfish-50mr vs stockfish)
Finished game 4 (stockfish-50mr vs stockfish): 1/2-1/2 {Draw by insufficient mating material}
Score of stockfish vs stockfish-50mr: 0 - 1 - 2 [0.33] 3
Started game 7 of 50 (stockfish vs stockfish-50mr)
Finished game 3 (stockfish vs stockfish-50mr): 1/2-1/2 {Draw by insufficient mating material}
Score of stockfish vs stockfish-50mr: 0 - 1 - 3 [0.38] 4
Started game 8 of 50 (stockfish-50mr vs stockfish)
Finished game 7 (stockfish vs stockfish-50mr): 0-1 {Black mates}
Score of stockfish vs stockfish-50mr: 0 - 2 - 3 [0.30] 5
Started game 9 of 50 (stockfish vs stockfish-50mr)
Finished game 6 (stockfish-50mr vs stockfish): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 0 - 2 - 4 [0.33] 6
Started game 10 of 50 (stockfish-50mr vs stockfish)
Finished game 8 (stockfish-50mr vs stockfish): 1-0 {White mates}
Score of stockfish vs stockfish-50mr: 0 - 3 - 4 [0.29] 7
Started game 11 of 50 (stockfish vs stockfish-50mr)
Finished game 5 (stockfish vs stockfish-50mr): 1-0 {White mates}
Score of stockfish vs stockfish-50mr: 1 - 3 - 4 [0.38] 8
Started game 12 of 50 (stockfish-50mr vs stockfish)
Finished game 10 (stockfish-50mr vs stockfish): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 1 - 3 - 5 [0.39] 9
Started game 13 of 50 (stockfish vs stockfish-50mr)
Finished game 12 (stockfish-50mr vs stockfish): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 1 - 3 - 6 [0.40] 10
Started game 14 of 50 (stockfish-50mr vs stockfish)
Finished game 11 (stockfish vs stockfish-50mr): 0-1 {Black mates}
Score of stockfish vs stockfish-50mr: 1 - 4 - 6 [0.36] 11
Started game 15 of 50 (stockfish vs stockfish-50mr)
Finished game 9 (stockfish vs stockfish-50mr): 1/2-1/2 {Draw by insufficient mating material}
Score of stockfish vs stockfish-50mr: 1 - 4 - 7 [0.38] 12
Started game 16 of 50 (stockfish-50mr vs stockfish)
Finished game 13 (stockfish vs stockfish-50mr): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 1 - 4 - 8 [0.38] 13
Started game 17 of 50 (stockfish vs stockfish-50mr)
Finished game 14 (stockfish-50mr vs stockfish): 1/2-1/2 {Draw by insufficient mating material}
Score of stockfish vs stockfish-50mr: 1 - 4 - 9 [0.39] 14
Started game 18 of 50 (stockfish-50mr vs stockfish)
Finished game 15 (stockfish vs stockfish-50mr): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 1 - 4 - 10 [0.40] 15
Started game 19 of 50 (stockfish vs stockfish-50mr)
Finished game 16 (stockfish-50mr vs stockfish): 1-0 {White mates}
Score of stockfish vs stockfish-50mr: 1 - 5 - 10 [0.38] 16
Started game 20 of 50 (stockfish-50mr vs stockfish)
Finished game 18 (stockfish-50mr vs stockfish): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 1 - 5 - 11 [0.38] 17
Started game 21 of 50 (stockfish vs stockfish-50mr)
Finished game 17 (stockfish vs stockfish-50mr): 1/2-1/2 {Draw by insufficient mating material}
Score of stockfish vs stockfish-50mr: 1 - 5 - 12 [0.39] 18
Started game 22 of 50 (stockfish-50mr vs stockfish)
Finished game 19 (stockfish vs stockfish-50mr): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 1 - 5 - 13 [0.39] 19
Started game 23 of 50 (stockfish vs stockfish-50mr)
Finished game 21 (stockfish vs stockfish-50mr): 0-1 {Black mates}
Score of stockfish vs stockfish-50mr: 1 - 6 - 13 [0.38] 20
Started game 24 of 50 (stockfish-50mr vs stockfish)
Finished game 20 (stockfish-50mr vs stockfish): 1/2-1/2 {Draw by insufficient mating material}
Score of stockfish vs stockfish-50mr: 1 - 6 - 14 [0.38] 21
Started game 25 of 50 (stockfish vs stockfish-50mr)
Finished game 23 (stockfish vs stockfish-50mr): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 1 - 6 - 15 [0.39] 22
Started game 26 of 50 (stockfish-50mr vs stockfish)
Finished game 22 (stockfish-50mr vs stockfish): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 1 - 6 - 16 [0.39] 23
Started game 27 of 50 (stockfish vs stockfish-50mr)
Finished game 25 (stockfish vs stockfish-50mr): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 1 - 6 - 17 [0.40] 24
Started game 28 of 50 (stockfish-50mr vs stockfish)
Finished game 28 (stockfish-50mr vs stockfish): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 1 - 6 - 18 [0.40] 25
Started game 29 of 50 (stockfish vs stockfish-50mr)
Finished game 24 (stockfish-50mr vs stockfish): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 1 - 6 - 19 [0.40] 26
Started game 30 of 50 (stockfish-50mr vs stockfish)
Finished game 27 (stockfish vs stockfish-50mr): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 1 - 6 - 20 [0.41] 27
Started game 31 of 50 (stockfish vs stockfish-50mr)
Finished game 26 (stockfish-50mr vs stockfish): 0-1 {Black mates}
Score of stockfish vs stockfish-50mr: 2 - 6 - 20 [0.43] 28
Started game 32 of 50 (stockfish-50mr vs stockfish)
Finished game 30 (stockfish-50mr vs stockfish): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 2 - 6 - 21 [0.43] 29
Started game 33 of 50 (stockfish vs stockfish-50mr)
Finished game 29 (stockfish vs stockfish-50mr): 1-0 {White mates}
Score of stockfish vs stockfish-50mr: 3 - 6 - 21 [0.45] 30
Started game 34 of 50 (stockfish-50mr vs stockfish)
Finished game 31 (stockfish vs stockfish-50mr): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 3 - 6 - 22 [0.45] 31
Started game 35 of 50 (stockfish vs stockfish-50mr)
Finished game 32 (stockfish-50mr vs stockfish): 1-0 {White mates}
Score of stockfish vs stockfish-50mr: 3 - 7 - 22 [0.44] 32
Started game 36 of 50 (stockfish-50mr vs stockfish)
Finished game 33 (stockfish vs stockfish-50mr): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 3 - 7 - 23 [0.44] 33
Started game 37 of 50 (stockfish vs stockfish-50mr)
Finished game 34 (stockfish-50mr vs stockfish): 1-0 {White mates}
Score of stockfish vs stockfish-50mr: 3 - 8 - 23 [0.43] 34
Started game 38 of 50 (stockfish-50mr vs stockfish)
Finished game 36 (stockfish-50mr vs stockfish): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 3 - 8 - 24 [0.43] 35
Started game 39 of 50 (stockfish vs stockfish-50mr)
Finished game 35 (stockfish vs stockfish-50mr): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 3 - 8 - 25 [0.43] 36
Started game 40 of 50 (stockfish-50mr vs stockfish)
Finished game 37 (stockfish vs stockfish-50mr): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 3 - 8 - 26 [0.43] 37
Started game 41 of 50 (stockfish vs stockfish-50mr)
Finished game 39 (stockfish vs stockfish-50mr): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 3 - 8 - 27 [0.43] 38
Started game 42 of 50 (stockfish-50mr vs stockfish)
Finished game 38 (stockfish-50mr vs stockfish): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 3 - 8 - 28 [0.44] 39
Started game 43 of 50 (stockfish vs stockfish-50mr)
Finished game 40 (stockfish-50mr vs stockfish): 0-1 {Black mates}
Score of stockfish vs stockfish-50mr: 4 - 8 - 28 [0.45] 40
Started game 44 of 50 (stockfish-50mr vs stockfish)
Finished game 43 (stockfish vs stockfish-50mr): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 4 - 8 - 29 [0.45] 41
Started game 45 of 50 (stockfish vs stockfish-50mr)
Finished game 41 (stockfish vs stockfish-50mr): 1/2-1/2 {Draw by insufficient mating material}
Score of stockfish vs stockfish-50mr: 4 - 8 - 30 [0.45] 42
Started game 46 of 50 (stockfish-50mr vs stockfish)
Finished game 42 (stockfish-50mr vs stockfish): 0-1 {Black mates}
Score of stockfish vs stockfish-50mr: 5 - 8 - 30 [0.47] 43
Started game 47 of 50 (stockfish vs stockfish-50mr)
Finished game 44 (stockfish-50mr vs stockfish): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 5 - 8 - 31 [0.47] 44
Started game 48 of 50 (stockfish-50mr vs stockfish)
Finished game 45 (stockfish vs stockfish-50mr): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 5 - 8 - 32 [0.47] 45
Started game 49 of 50 (stockfish vs stockfish-50mr)
Finished game 46 (stockfish-50mr vs stockfish): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 5 - 8 - 33 [0.47] 46
Started game 50 of 50 (stockfish-50mr vs stockfish)
Finished game 47 (stockfish vs stockfish-50mr): 1/2-1/2 {Draw by insufficient mating material}
Score of stockfish vs stockfish-50mr: 5 - 8 - 34 [0.47] 47
Finished game 48 (stockfish-50mr vs stockfish): 1/2-1/2 {Draw by 3-fold repetition}
Score of stockfish vs stockfish-50mr: 5 - 8 - 35 [0.47] 48
Finished game 49 (stockfish vs stockfish-50mr): 0-1 {Black mates}
Score of stockfish vs stockfish-50mr: 5 - 9 - 35 [0.46] 49
Finished game 50 (stockfish-50mr vs stockfish): 1-0 {White mates}
Score of stockfish vs stockfish-50mr: 5 - 10 - 35 [0.45] 50
ELO difference: -35
Finished match

70% draw ratio is high, but in self-play to be expected.
Remarkable: no draws by 50-move-rule! Many (too many?) draws by 3-fold-repetition! IIRC in former tests the percentage of draws by 3-fold-repetition were not as high. But I will check this.
Next I will look over the games to see, if there are any oddities. If everything looks ok, I will run a 500-game-match this night and report tomorrow.

@Marco or Joona, if you have any suggestions, you're welcome.

Joerg.

mcostalba · Post by **mcostalba** » Sun Sep 30, 2012 7:13 pm

Joerg Oster wrote: If everything looks ok, I will run a 500-game-match this night and report tomorrow.

@Marco or Joona, if you have any suggestions, you're welcome.

Hi Joerg,

thanks for testing. I will look forward your result on 500 games, if good I will verify on a longer test and hopefully commit the change.

Thanks
Marco

Joerg Oster · Post by **Joerg Oster** » Mon Oct 01, 2012 11:14 am

mcostalba wrote:
Joerg Oster wrote: If everything looks ok, I will run a 500-game-match this night and report tomorrow.

@Marco or Joona, if you have any suggestions, you're welcome.
Hi Joerg,

thanks for testing. I will look forward your result on 500 games, if good I will verify on a longer test and hopefully commit the change.

Thanks
Marco

Hi Marco,

test is done.
Result is (+78 -74 =348) in favor of Stockfish default. Which is rather meaningless with such a big draw rate. Both versions seem to be equally strong.
The modification doesn't seem to help or hurt either.

But I wonder if this is also true against different opponents.
So I decided to run a gauntlet for each version against 5 opponents.
I will post the results in this thread.

Joerg.

Joerg Oster · Post by **Joerg Oster** » Mon Oct 01, 2012 1:25 pm

OK, two Test gauntlets, 50 games each, are finished.
First, Stockfish default:

Code: Select all

                   Score     Stockfish  Komodo3    Critter1.6 Hiarcs13.2 Gaviota0.8 Rybka4.1  
-----------------------------------------------------------------------------------------------
 1: Stockfish    33.0 / 50   XXXXXXXXXX =01=10000= 0====1==0= 111=101011 1=11=1=111 1111111110  (+26 -10 =14)

 2: Komodo3       6.5 / 10   =10=01111= XXXXXXXXXX .......... .......... .......... ..........  (+5 -2 =3)

 3: Critter1.6    5.5 / 10   1====0==1= .......... XXXXXXXXXX .......... .......... ..........  (+2 -1 =7)

 4: Hiarcs13.2    2.5 / 10   000=010100 .......... .......... XXXXXXXXXX .......... ..........  (+2 -7 =1)

 5: Gaviota0.84   1.5 / 10   0=00=0=000 .......... .......... .......... XXXXXXXXXX ..........  (+0 -7 =3)

 6: Rybka4.1      1.0 / 10   0000000001 .......... .......... .......... .......... XXXXXXXXXX  (+1 -9 =0)
-----------------------------------------------------------------------------------------------
50 games: +21 =14 -15

Then Stockfish-50MR:

Code: Select all

                      Score     Stockfish- Critter1.6 Komodo3    Rybka4.1   Hiarcs13.2 Gaviota0.8
--------------------------------------------------------------------------------------------------
 1: Stockfish-50MR  34.0 / 50   XXXXXXXXXX 0=01=00=00 0000==0=1= 1111111011 1=11111111 1111111111  (+30 -12 =8)

 2: Critter1.6       7.5 / 10   1=10=11=11 XXXXXXXXXX .......... .......... .......... ..........  (+6 -1 =3)

 3: Komodo3          7.0 / 10   1111==1=0= .......... XXXXXXXXXX .......... .......... ..........  (+5 -1 =4)

 4: Rybka4.1         1.0 / 10   0000000100 .......... .......... XXXXXXXXXX .......... ..........  (+1 -9 =0)

 5: Hiarcs13.2       0.5 / 10   0=00000000 .......... .......... .......... XXXXXXXXXX ..........  (+0 -9 =1)

 6: Gaviota0.84      0.0 / 10   0000000000 .......... .......... .......... .......... XXXXXXXXXX  (+0 -10 =0)
--------------------------------------------------------------------------------------------------
50 games: +21 =8 -21

Cross tables from Scid vs. PC

This test was mainly done to see, if the gauntlet mode works as expected resp. if my command line was OK. Seems it was. It seems the modified version produces less draws. But too less games, of course.
Btw. thanks Ilarij and Arto for such a great tool!

@Marco: Test gauntlets with 500 games each will start in a few minutes. Results on Wednesday.

mcostalba · Post by **mcostalba** » Mon Oct 01, 2012 7:26 pm

Joerg Oster wrote: @Marco: Test gauntlets with 500 games each will start in a few minutes. Results on Wednesday.

Great! Thanks a lot.

Stockfish 2.3.1 weaker than 2.2.2?

Re: Stockfish 2.3.1 weaker than 2.2.2?

Re: Stockfish 2.3.1 weaker than 2.2.2?

Re: Stockfish 2.3.1 weaker than 2.2.2?

Re: Stockfish 2.3.1 weaker than 2.2.2?

Re: Stockfish 2.3.1 weaker than 2.2.2?

Re: Stockfish 2.3.1 weaker than 2.2.2?

Re: Stockfish 2.3.1 weaker than 2.2.2?

Re: Stockfish 2.3.1 weaker than 2.2.2?

Re: Stockfish 2.3.1 weaker than 2.2.2?

Re: Stockfish 2.3.1 weaker than 2.2.2?