Questions for the Stockfish team

[Daniel Shawul]

Wesley you are a genius !!
Indeed that is weird. How can it avoid all hanging piece situations with mobility eval ?? The _root move ordering _ must some how persist thorught where they are sorted by Qsearch, captures first etc ...
I changed the order of captures and non-captures in root.c, and set value = 0 for Qsearch . I also set skill = 0 in eval. I did this by removing the part of the code in eval.c because it doesn't allow you to set skill 0.

And look what i got on my first try.. Crap game that i get from scorpio_random... I don't want to be excited about this as I don't know internals of crafty very well and could have made mistake somewhere , but if this is really turns out as the culprit then the Beal effect bla bla crap savings material smartly through mobility is utter crap..

Code: Select all

[Event "?"]
[Site "?"]
[Date "2010.07.22"]
[Round "2"]
[White "XboardEngine"]
[Black "Crafty-23.1"]
[Result "1-0"]
[PlyCount "73"]
[TimeControl "40/30+1"]

1. e4 {book} c5 {book} 2. Nf3 {book} e6 {book} 3. d4 {book} cxd4 {book}
4. Nxd4 {book} Nc6 {book} 5. Nb5 {book} Nf6 {book} 6. N1c3 {book} d6 {book}
7. Bf4 {book} e5 {book} 8. Bg5 {book} a6 {book} 9. Na3 {book} b5 {book}
10. Bxf6 {book} gxf6 {book} 11. Nd5 {book} f5 {book} 12. c3 {book} Bg7 {book}
13. exf5 {book} Bxf5 {book} 14. Be2 {+0.35/4 1.4s} Be4 {+0.56/8 2.3s}
15. Bf3 {+0.43/5 1.4s} Qh4 {+0.54/8 2.3s} 16. Nc7+ {+2.43/5 1.4s}
Kd7 {+0.57/9 2.3s} 17. Nxa8 {+2.43/5 1.4s} Nd4 {+0.56/8 2.4s}
18. Nb6+ {+5.82/5 1.4s} Ke7 {+0.55/8 2.3s} 19. cxd4 {+6.93/5 1.4s}
Kf8 {+0.60/8 4.6s} 20. dxe5 {+8.30/5 1.4s} Bxe5 {+0.56/8 2.0s}
21. Nd7+ {+8.07/5 1.3s} Kg7 {+0.61/10 2.2s} 22. Nxe5 {+8.40/5 1.3s}
b4 {+0.67/9 3.2s} 23. g3 {+11.49/5 1.3s} Qe7 {+0.68/8 1.6s}
24. Bxe4 {+11.39/5 1.3s} Qg5 {+0.77/9 2.0s} 25. f4 {+14.77/5 1.3s}
Re8 {+0.88/9 2.3s} 26. fxg5 {+18.51/5 1.3s} Rxe5 {+0.97/10 1.6s}
27. Nc2 {+18.40/5 1.3s} Rxe4+ {+0.95/10 2.2s} 28. Kf2 {+18.86/5 1.3s}
b3 {+0.93/9 1.5s} 29. axb3 {+19.65/5 1.3s} Kg8 {+0.96/9 1.8s}
30. Qxd6 {+20.77/5 1.3s} Re8 {+0.99/10 1.5s} 31. Rxa6 {+20.87/4 1.2s}
Rc8 {+327.53/10 1.7s} 32. Qd3 {+21.08/4 1.2s} Re8 {+0.98/10 1.7s}
33. Rh6 {+22.31/4 1.2s} Kf8 {+327.55/10 1.6s} 34. Rxh7 {+22.46/4 1.2s}
Kg8 {+327.57/10 1.6s} 35. g6 {+23.06/4 1.2s} Kf8 {+327.59/8 1.6s}
36. Qd6+ {+30.31/4 1.2s} Kg8 {+327.61/9 1.7s} 37. gxf7+ {+32.13/4 1.2s}
{Black resigns} 1-0

[bob]

Wesley you are a genius !!
Indeed that is weird. How can it avoid all hanging piece situations with mobility eval ?? The _root move ordering _ must some how persist thorught where they are sorted by Qsearch, captures first etc ...
I changed the order of captures and non-captures in root.c, and set value = 0 for Qsearch . I also set skill = 0 in eval. I did this by removing the part of the code in eval.c because it doesn't allow you to set skill 0.

First, it doesn't avoid losing material at ply=2. Tried it an hour ago and it hung pieces right and left. _IF_ you use skill=1 _AND_ you compile with -DSKILL.

Root moves are ordered by calling Quiesce(). Which returns, as we know, purely random evaluations. If you want to see them, just type "disp ply1" and then either "go" or enter a move. Here is an example. I set sd=2, typed "display ply1" and then "e4". Here is the ply-1 move list as it is sorted, with the scores returned by quiesce():
move score
e5 -2
Nf6 -9
e6 -11
d6 -12
a6 -16
f5 -21
f6 -28
c5 -30
Na6 -35
c6 -41
g6 -47
d5 -51
b5 -60
a5 -64
Nc6 -66
b6 -66
h5 -78
Nh6 -82
g5 -88
h6 -98

Notice the move f5? Hangs a pawn. Not first. Yet that is what it played here with sd=2:

time surplus 0.00 time limit 30.00 (+0.00) (3:00)
depth time score variation (1)
2-> 0.14 0.90 1. ... f5 2. Be2
time=0.14 mat=0 n=488 fh=5% nps=3K
extensions=3 qchecks=2 reduced=0 pruned=0
predicted=0 evals=429 50move=0 EGTBprobes=0 hits=0
SMP-> splits=0 aborts=0 data=0/128 elap=0.14
Black(1): f5


Notice the ply1 move list has negative scores between -100 and 0. Because when I call quiesce, it negates the numbers since the side to move is reversed.

I don't know what you are seeing, or what you are getting, but if you do skill=1 and sd=2, it will _not_ play well. It hangs pieces right and left. Whatever you are talking about above, I have no idea. You _will_ get crap moves with sd=2. And skill 1. But if you use something like st=1, you will not. Note that this _must_ be done with 23.2... 23.3 has a time waster loop that at st=1 reduces the nps to under 5000 which will make it play poorly. You can comment that loop out (first reference to SKILL in evaluate.c, or else use 23.2 to avoid using the new version that seems to solve the strong skill=1 play.

So first, let's get on the right version, which is 23.2. Second, make sure it is compiled with -DSKILL and run with st=1 or so and skill=1. And see how often it hangs material. It still will, but very infrequently. If you do the test properly.

If you insist on using 23.3, you have to comment out the loop at the top of evaluate.c so it won't slow down and reach very shallow depths which ruins the mobility effect...

More once you are clear on what you actually tested... BTW root move ordering simply makes each root move, calls Quiesce() and saves the returned value. The moves are then sorted based on these values, from high to low. No captures first or any other such thing. Once you get to iteration 2, moves are sorted by node counts which with this kind of evaluation is also very random.

And look what i got on my first try.. Crap game that i get from scorpio_random... I don't want to be excited about this as I don't know internals of crafty very well and could have made mistake somewhere , but if this is really turns out as the culprit then the Beal effect bla bla crap savings material smartly through mobility is utter crap..

Code: Select all

[Event "?"]
[Site "?"]
[Date "2010.07.22"]
[Round "2"]
[White "XboardEngine"]
[Black "Crafty-23.1"]
[Result "1-0"]
[PlyCount "73"]
[TimeControl "40/30+1"]

1. e4 {book} c5 {book} 2. Nf3 {book} e6 {book} 3. d4 {book} cxd4 {book}
4. Nxd4 {book} Nc6 {book} 5. Nb5 {book} Nf6 {book} 6. N1c3 {book} d6 {book}
7. Bf4 {book} e5 {book} 8. Bg5 {book} a6 {book} 9. Na3 {book} b5 {book}
10. Bxf6 {book} gxf6 {book} 11. Nd5 {book} f5 {book} 12. c3 {book} Bg7 {book}
13. exf5 {book} Bxf5 {book} 14. Be2 {+0.35/4 1.4s} Be4 {+0.56/8 2.3s}
15. Bf3 {+0.43/5 1.4s} Qh4 {+0.54/8 2.3s} 16. Nc7+ {+2.43/5 1.4s}
Kd7 {+0.57/9 2.3s} 17. Nxa8 {+2.43/5 1.4s} Nd4 {+0.56/8 2.4s}
18. Nb6+ {+5.82/5 1.4s} Ke7 {+0.55/8 2.3s} 19. cxd4 {+6.93/5 1.4s}
Kf8 {+0.60/8 4.6s} 20. dxe5 {+8.30/5 1.4s} Bxe5 {+0.56/8 2.0s}
21. Nd7+ {+8.07/5 1.3s} Kg7 {+0.61/10 2.2s} 22. Nxe5 {+8.40/5 1.3s}
b4 {+0.67/9 3.2s} 23. g3 {+11.49/5 1.3s} Qe7 {+0.68/8 1.6s}
24. Bxe4 {+11.39/5 1.3s} Qg5 {+0.77/9 2.0s} 25. f4 {+14.77/5 1.3s}
Re8 {+0.88/9 2.3s} 26. fxg5 {+18.51/5 1.3s} Rxe5 {+0.97/10 1.6s}
27. Nc2 {+18.40/5 1.3s} Rxe4+ {+0.95/10 2.2s} 28. Kf2 {+18.86/5 1.3s}
b3 {+0.93/9 1.5s} 29. axb3 {+19.65/5 1.3s} Kg8 {+0.96/9 1.8s}
30. Qxd6 {+20.77/5 1.3s} Re8 {+0.99/10 1.5s} 31. Rxa6 {+20.87/4 1.2s}
Rc8 {+327.53/10 1.7s} 32. Qd3 {+21.08/4 1.2s} Re8 {+0.98/10 1.7s}
33. Rh6 {+22.31/4 1.2s} Kf8 {+327.55/10 1.6s} 34. Rxh7 {+22.46/4 1.2s}
Kg8 {+327.57/10 1.6s} 35. g6 {+23.06/4 1.2s} Kf8 {+327.59/8 1.6s}
36. Qd6+ {+30.31/4 1.2s} Kg8 {+327.61/9 1.7s} 37. gxf7+ {+32.13/4 1.2s}
{Black resigns} 1-0

[Daniel Shawul]

First of all look at the game I posted. It tells you it is
- crafty version 23.1
- tc = 40/30 + 1 increment . I used that since I heard it fails in time sometimes but this made the games last long.
- I did not use sd=2. You can see crafty reached depth 9 on average.
I used -DSKILL too.

I was testing Wesely's idea when I run the game so I thought it came from the move ordering thing, because I was under the impression crafty avoids hanging pieces with full randomness due to Beal effect ,no ?? Just change this in crafty 23.1

Code: Select all

#if defined(SKILL)
  //if (skill < 100)
	score = PAWN_VALUE * (BITBOARD) Random32() / 0x100000000ULL;
/*
    score =
        skill * score / 100 + ((100 -
            skill) * PAWN_VALUE * (BITBOARD) Random32() / 0x100000000ULL) /
        100;
*/
#endif

This is an always skill=0 modification so that I don't need to set in crafty.rc.
It now becomes same as scorpio_random in everything. Lots of games crafty gives up material looking for mates. It seems more successful than scorpio (maybe due to better mating capabilties) but it throws away material and does the same horrible thing.

Results:

40/30 : TSCP 260 elo better than crafty

Code: Select all

Num. Name          games   score 
   0 Crafty-23.1     300    59.5 
   1 XboardEngine    300   240.5 
Rank Name           Elo    +    - games score oppo. draws 
   1 XboardEngine   130   22   21   300   80%  -130    8% 
   2 Crafty-23.1   -130   21   22   300   20%   130    8% 

40/30+1 : TSCP 216 elo better than crafty

Code: Select all

Num. Name          games   score 
   0 Crafty-23.1     300      70 
   1 XboardEngine    300     230 
Rank Name           Elo    +    - games score oppo. draws 
   1 XboardEngine   108   21   20   300   77%  -108   11% 
   2 Crafty-23.1   -108   20   21   300   23%   108   11% 

Download source,games etc here http://sites.google.com/site/dshawul/cr ... ects=0&d=1

Report back with the eval.c change only. Forget the root move ordering changes for now. I thought crafty didnot produce hanging pieces at skill 0 ?? If it does, I ask what is the differnce between scorpio_random and crafty_0. They both don't seem to have a clue of material eval at skill 0.

[Daniel Shawul]

There seem to be a big difference between skill=1 and skill=0. crafty scored much better almost +200elo !! Never underestimate anything..Thats why I objected to it strongly in the first place!

Code: Select all

Num. Name          games   score 
   0 Crafty-23.1     300     122 
   1 XboardEngine    300     178 
Rank Name           Elo    +    - games score oppo. draws 
   1 XboardEngine    33   18   18   300   59%   -33   18% 
   2 Crafty-23.1    -33   18   18   300   41%    33   18% 

Games http://sites.google.com/site/dshawul/cr ... ects=0&d=1

It will be good if someone does the same tests as I did for confrimation..
I did the test for skill 0 again and the result is the same 80-20 % for TSCP...

[bob]

First of all look at the game I posted. It tells you it is
- crafty version 23.1
- tc = 40/30 + 1 increment . I used that since I heard it fails in time sometimes but this made the games last long.
- I did not use sd=2. You can see crafty reached depth 9 on average.
I used -DSKILL too.

That doesn't answer what I asked. Did you enter the command "skill=1" before you played the game? -DSKILL by itself only enables the skill feature, but the default is 100.

I was testing Wesely's idea when I run the game so I thought it came from the move ordering thing, because I was under the impression crafty avoids hanging pieces with full randomness due to Beal effect ,no ?? Just change this in crafty 23.1

Code: Select all

#if defined(SKILL)
  //if (skill < 100)
	score = PAWN_VALUE * (BITBOARD) Random32() / 0x100000000ULL;
/*
    score =
        skill * score / 100 + ((100 -
            skill) * PAWN_VALUE * (BITBOARD) Random32() / 0x100000000ULL) /
        100;
*/
#endif

This is an always skill=0 modification so that I don't need to set in crafty.rc.
It now becomes same as scorpio_random in everything. Lots of games crafty gives up material looking for mates. It seems more successful than scorpio (maybe due to better mating capabilties) but it throws away material and does the same horrible thing.

Let me say this one more time. "use skill = 1". As I have explained, this has other effects on the program. It disables search extensions, it disables null-move, it disables LMR, etc.

So please do what I ask, as opposed to what you want. If you want to reproduce my experimental results, that is. If you just want to produce random noise in the discussion, there are hundreds of different things to try. But how about, for once, just doing what has been discussed _from the beginning_ and using "skill=1" And explicitly on version 23.2, which is the version being discussed in Volker's original post in the general forum... According to his comments, something changed in 23.2... what that is I do not know, except that it has nothing to do with the random evaluation stuff...

So. 23.2, -DSKILL, and skill=1. Otherwise none of this means a thing.

Results:

40/30 : TSCP 260 elo better than crafty

Code: Select all

Num. Name          games   score 
   0 Crafty-23.1     300    59.5 
   1 XboardEngine    300   240.5 
Rank Name           Elo    +    - games score oppo. draws 
   1 XboardEngine   130   22   21   300   80%  -130    8% 
   2 Crafty-23.1   -130   21   22   300   20%   130    8% 

40/30+1 : TSCP 216 elo better than crafty

Code: Select all

Num. Name          games   score 
   0 Crafty-23.1     300      70 
   1 XboardEngine    300     230 
Rank Name           Elo    +    - games score oppo. draws 
   1 XboardEngine   108   21   20   300   77%  -108   11% 
   2 Crafty-23.1   -108   20   21   300   23%   108   11% 

Download source,games etc here http://sites.google.com/site/dshawul/cr ... ects=0&d=1

Report back with the eval.c change only. Forget the root move ordering changes for now. I thought crafty didnot produce hanging pieces at skill 0 ?? If it does, I ask what is the differnce between scorpio_random and crafty_0. They both don't seem to have a clue of material eval at skill 0.

You do understand the difference between "never" and "anything except never?"??? I did not say it _never_ hung a piece. I said it _rarely_ hangs a piece. Once you run the experiment properly, you may see different results than what you are getting with the things you are trying that do not match what has been discussed.

[bob]

There seem to be a big difference between skill=1 and skill=0. crafty scored much better almost +200elo !! Never underestimate anything..Thats why I objected to it strongly in the first place!

Code: Select all

Num. Name          games   score 
   0 Crafty-23.1     300     122 
   1 XboardEngine    300     178 
Rank Name           Elo    +    - games score oppo. draws 
   1 XboardEngine    33   18   18   300   59%   -33   18% 
   2 Crafty-23.1    -33   18   18   300   41%    33   18% 

Games http://sites.google.com/site/dshawul/cr ... ects=0&d=1

It will be good if someone does the same tests as I did for confrimation..
I did the test for skill 0 again and the result is the same 80-20 % for TSCP...

This really gets old. Please clarify how you get "skill=0" and "skill=1" in crafty? skill=0 is _not_ an option. If you just make your change to evaluate.c, that is _not_ the same thing as doing skill=0. If you really used skill=1, then you have changed _everything.

The word "thick-headed" comes to mind. The "skill=1" command not only makes the evaluation 99% random, it also turns off LMR, NULL-MOVE, extensions, pruning, etc. Just changing the eval doesn't change any of those _other_ things. You are comparing apples and oranges.. Again...

Please do it right, or else don't do it at all. This is just producing a ton of random noise.

[Daniel Shawul]

Well it is not my problem if you keep on changing what you think it in your replies. I did what you asked for got you the results.

Turnoff null_move,lmr,extensions etc etc to get an equal depth search so that
the alpha-beta won't be screwed. You add any selectivity to get different depths , you are screwed.
But you don't LISTEN. I said from the beginning there is no way this thing works in normal search if it breaks the "perfect game information" assumption of chess. It evaluates one side's score so your only hope is to get all the leaves at the same depth.
VERY VERY unrealistic seearc tree indeed. All engines are selective and search to different depths .....
I can shape my tree in a weird way and make anything work... Thats not interesting at all. How many times did metnion Minimax will not work if you mix scores from ply to another.

Well your " myth " is finally solved. You probably forgot to make sure that everything gets searched to equal depth in your previous crafty versions.

[Daniel Shawul]

Well you asked for random eval only and only that...
How many times did you say I should add rand() to the beginning of eval
and I should see the beal effect. Did that in scorpio,glaurung and in your own _crafty_. Don't say I screwed up the test BECAUSE you asked for random eval and only eval.

NO that won't work.
You have to shape your tree in a UNIQUE way..bullshit.
This is modifying anything is required to get the desired result...
You do the experiment, and analyze the resutl. Not the other way round.
In any modern selective engine that won't work at all...

[bob]

Well you asked for random eval only and only that...
How many times did you say I should add rand() to the beginning of eval
and I should see the beal effect. Did that in scorpio,glaurung and in your own _crafty_. Don't say I screwed up the test BECAUSE you asked for random eval and only eval.

No I didn't. I have stated, at least a dozen times, that skill=1 does two things... It uses 99% random, 1% real score, and all the pruning and stuff is turned completely off. I also stated that logically, the "Beal effect" is probably better on short bushy trees, rather than deep selective ones, particularly when some moves go way deep, and some barely go anywhere thanks to LMR.

You were of the opinion that that 1% real score was the reason it was playing better, I disagreed and suggested a way to check it, by simply using a completely random score. I _never_ said to not do all the other things that skill=1 does. Just that allowing a "skill=0" would test your hypothesis cleanly.

So don't say I "only asked for a random eval". I suggested that to get you off the claim that that 1% real score was enough. I have _never_ suggested not changing the other things.

NO that won't work.
You have to shape your tree in a UNIQUE way..bullshit.
This is modifying anything is required to get the desired result...
You do the experiment, and analyze the resutl. Not the other way round.
In any modern selective engine that won't work at all...

All you have to do is to do what I have said at least a dozen times. You have already proven that skill=1 is stronger than TSCP. You said that was impossible with a random eval. So use skill=1, but modify the eval to use pure random and see what you get. The results will be the same as the original skill=1 test.

As a lesson in science, if you see an experiment you don't trust (Volker's claim that skill=1 is almost 1800 Elo based on his testing) then you try to run that _same_ experiment yourself and measure the results. You don't run _different_ experiments and then claim that those refute the original experiment. You have to start with the _same_ experiment. It really is that simple. And you have finally done that one time when you tested skill=1. It only took a hundred posts and two days to get it right.

Now if you want to take skill=1 (even with the 100% random score change) and try to figure out why _your_ experiments failed, go right ahead. It will be interesting. Does stopping null-move make a big difference? Does eliminating LMR do the trick? what's the effect of disabling check extensions? All of which might be interesting to someone, me included. But none of which has one thing to do with the original discussion about Crafty playing too strong with a random evaluation. I suppose you at least now see that what Volker said and I verified is now no longer "hot air and bullshit?" Hopefully you see that...

[bob]

Well it is not my problem if you keep on changing what you think it in your replies. I did what you asked for got you the results.

When you tried skill=1 you did do what I asked. But nowhere else did you do so. You just _assumed_ you were doing what I had suggested. Even though I have mentioned a dozen times that skill=1 turns all the selective stuff off as well.

Turnoff null_move,lmr,extensions etc etc to get an equal depth search so that
the alpha-beta won't be screwed. You add any selectivity to get different depths , you are screwed.
But you don't LISTEN. I said from the beginning there is no way this thing works in normal search if it breaks the "perfect game information" assumption of chess. It evaluates one side's score so your only hope is to get all the leaves at the same depth.

Crafty's search, with skill=1 _IS_ a "normal search." Selectivity is not good when relying on the Beal effect. random eval does not break the "perfect game information assumption" since there is no such assumption in chess. If you mean zero-sum, that is not what this is about. The evaluation is no longer relevant. The only thing that is interesting is that we base a move on the probability of getting a large (or small at min nodes) evaluation, not on the number itself. The number means nothing. Doesn't break alpha/beta since we make the number work in the alpha/beta framework by using the normal assumption that the bigger the number, the better the position for us.

You simply stated that random evaluation would not work, that the "beal effect" was hot air or bullshit. Now time to backtrack a bit...

VERY VERY unrealistic seearc tree indeed. All engines are selective and search to different depths .....
I can shape my tree in a weird way and make anything work... Thats not interesting at all. How many times did metnion Minimax will not work if you mix scores from ply to another.

I don't even know what that means. But minimax works. And random eval works so long as you provide a search framework that can use the probability distribution of the eval to quantify the goodness of the position. Clearly the "beal effect" will not work if you go 500 plies deep, forward prune all but the first move at every node, and use that to choose between root moves. Doesn't take advantage of the probability component of the eval, which is _all_ we have to go on.

Just because you didn't read the paper, and didn't listen to explanations, doesn't mean it doesn't work. And finally you actually tried skill=1 and saw the result. Hopefully we can now move on.

Well your " myth " is finally solved. You probably forgot to make sure that everything gets searched to equal depth in your previous crafty versions.

SKILL has not changed. But I have improved the selective stuff quite a bit. Might be we failed to turn everything off somehow, I'll try to look. But that was the only interesting thing in this, and I have solved the problem in 23.3 in a way that will likely give much lower ratings, which was the goal. There is no "myth" however. Perhaps you meant "mystery"? The "myth" would be that the "Beal effect" is garbage. It most definitely is not, done correctly. Unexpected? Yes. But bogus? Hardly.