Stockfish 1.9 JA update available

[Uri Blass]

In my own testing it seems like every version that is a little safer plays worse at longer time controls but I have never tried to measure this with low enough error margins. As you point out this requires a LOT of CPU power to do well.

stockfish(including stockfish1.9) clearly outsearch Komodo so it seems that you still use very safe pruning relative to stockfish.

It is possible that more safe pruning is better for stockfish when less safe pruning is better for komodo.

based on testing at fixed depth the difference in pruning between stockfish1.9 and stockfish1.8 is very small at small depths

It is possible that Stockfish1.9 does not like the noomen test but here are my result in these positions at fixed depth

depth 2:
Stockfish1.8-stockfish1.9 55:45

depth 7:
Stockfish1.8-stockfish1.9 52:48

depth 12:
Stockfish1.9-Stockfish1.8 50.5:49.5

Hello Uri, what about the time used to reach this depths ? Is one version faster ?

Thx,
Vincent

I believe that the difference is very small.

they often get depth 12 in less than a second but even when I played some games at depth 17 and I looked at the times I usually saw that the difference is not big(and usually stockfish1.9 needs more time but less than twice more time even at this depth).

Uri

[Don]

In my own testing it seems like every version that is a little safer plays worse at longer time controls but I have never tried to measure this with low enough error margins. As you point out this requires a LOT of CPU power to do well.

stockfish(including stockfish1.9) clearly outsearch Komodo so it seems that you still use very safe pruning relative to stockfish.

Since this discussion started we have been analyzing this issue with a new test program we have.

The term "out-search" is a term that I am not sure of - what does it mean?

If you start stockish 1.8 (for example) and komodo 1.2 and do a search on some given position, stockfish will get up to 15 ply more quickly than Komodo. Is that what you mean by outsearch?

Another definition is that it will find tactics more quickly in general (regardless of depth) - tactics that have nothing to do with the positional evaluation function.

We have a version of komodo that we have been working on which has a very low branching factor and outsearches komodo 1.2 by 2 ply even at fast time controls. By the first definition I gave above - it also outsearches Stockfish and has a lower branching factor. However it doesn't play better than Stockfish or even Komodo 1.2. We did not set out to make this program especially fast, we just implemented things that seemed to work in making it stronger.

We discovered that this new version has a lower branching factor but in test games it scales worse than komodo 1.2. When I say "scales worse" I mean that it plays just as well at fast time controls, but with deeper searches it falls behind and the deeper you search, the more of a superiority komodo 1.2 has.

So we tried being a bit more conservative with this version (with LMR specifically) in order to try to understand what was going on. When we did this, the program gets weaker and the time/elo scalability does not improve. It's just a slower program that does not scale as well.

So there is something going on that we are not understanding. The programs are very similar, but lots of minor differences. And it doesn't appear to be a simple matter of being a little more conservative or a little less conservative.

It is possible that more safe pruning is better for stockfish when less safe pruning is better for komodo.

based on testing at fixed depth the difference in pruning between stockfish1.9 and stockfish1.8 is very small at small depths

It is possible that Stockfish1.9 does not like the noomen test but here are my result in these positions at fixed depth

depth 2:
Stockfish1.8-stockfish1.9 55:45

depth 7:
Stockfish1.8-stockfish1.9 52:48

depth 12:
Stockfish1.9-Stockfish1.8 50.5:49.5

[Uri Blass]

The term "out-search" is a term that I am not sure of - what does it mean?

If you start stockish 1.8 (for example) and komodo 1.2 and do a search on some given position, stockfish will get up to 15 ply more quickly than Komodo. Is that what you mean by outsearch?

[/quote]

Yes

[NATIONAL12]

any chance of JA giving us a 12cpu compile.

[alpha123]

any chance of JA giving us a 12cpu compile.

My GTB version should be able to use up to 24 threads. I recommend Dann Corbit's compile (http://cap.connx.com/chess-engines/new- ... -19-gtb.7z), as Jim's doesn't seem to have enabled that. Admittedly, you have to specify a lot of -DUSE_XXX stuff when compiling (but that's in the makefile by default), so maybe he missed it.

Peter

[UncombedCoconut]

I am most impressed by the improvements in code clarity and readability. They go significantly beyond the mere deletions of unnecessary code.
Thanks for the update, and for inadvertently teaching me about the ios_base extensible array. (I still don't understand why it was invented, though! )

[mcostalba]

I am most impressed by the improvements in code clarity and readability. They go significantly beyond the mere deletions of unnecessary code.
Thanks for the update, and for inadvertently teaching me about the ios_base extensible array. (I still don't understand why it was invented, though! )

Is a way to add some 'state' to the stream, for instance in the below case

Code: Select all

cout << second;
cout << first << second;

it would be possible to make the output of 'second' to depend on 'first', so that 'second' is printed in a different way in the two cases. It is a kind of customized stream manipulator (http://www.cplusplus.com/reference/iost ... ipulators/).

In the specific case of SF, we use this trick to properly print the moves in Chess960 case. To print a move we use:

Code: Select all

cout << move;

But it happens that is not enough because in Chess960 castling moves are special formatted, so we use the below:

Code: Select all

 // When formatting a move for std::cout we must know if we are in Chess960
 // or not. To keep using the handy operator<<() on the move the trick is to
 // embed this flag in the stream itself. Function-like named enum set960 is
 // used as a custom manipulator and the stream internal general-purpose array,
 // accessed through ios_base::iword(), is used to pass the flag to the move's
 // operator<<() that will use it to properly format castling moves.

enum set960 {};

std::ostream& operator<< (std::ostream& os, const set960& m) {
  os.iword(0) = int(m);
  return os;
}


/// Overload the << operator, to make it easier to print moves.

std::ostream& operator << (std::ostream& os, Move m) {

  bool chess960 = (os.iword(0) != 0); // See set960()
  return os << move_to_string(m, chess960);
}

The custom manipulator set960 is used as following:

Code: Select all

    // Print RootMoveList startup scoring to the standard output,
    // so to output information also for iteration 1.
    cout << set960(p.is_chess960()) // Is enough to set once at the beginning
         << "info depth " << 1
         << "\ninfo depth " << 1
         << " score " << value_to_uci(rml.move_score(0))
         << " time " << current_search_time()
         << " nodes " << ThreadsMgr.nodes_searched()
         << " nps " << nps()
         << " pv " << rml.move(0) << "\n";

[mcostalba]

So we tried being a bit more conservative with this version (with LMR specifically) in order to try to understand what was going on. When we did this, the program gets weaker and the time/elo scalability does not improve. It's just a slower program that does not scale as well.

It is incredible that, although universally used, LMR is still a kind of unkonw land in many ways. Scalability of LMR in particular is still far from being completely understood

[Uri Blass]

We have a version of komodo that we have been working on which has a very low branching factor and outsearches komodo 1.2 by 2 ply even at fast time controls. By the first definition I gave above - it also outsearches Stockfish and has a lower branching factor. However it doesn't play better than Stockfish or even Komodo 1.2. We did not set out to make this program especially fast, we just implemented things that seemed to work in making it stronger.

We discovered that this new version has a lower branching factor but in test games it scales worse than komodo 1.2. When I say "scales worse" I mean that it plays just as well at fast time controls, but with deeper searches it falls behind and the deeper you search, the more of a superiority komodo 1.2 has.

So we tried being a bit more conservative with this version (with LMR specifically) in order to try to understand what was going on. When we did this, the program gets weaker and the time/elo scalability does not improve. It's just a slower program that does not scale as well.

So there is something going on that we are not understanding. The programs are very similar, but lots of minor differences. And it doesn't appear to be a simple matter of being a little more conservative or a little less conservative.

I do not understand what is the reason that the new version has a lower branching factor than komodo in the first place.

I understand that it cannot be because of more aggresive LMR(otherwise
it could mean that you can get komodo by being more conservative with LMR)

[Don]

We have a version of komodo that we have been working on which has a very low branching factor and outsearches komodo 1.2 by 2 ply even at fast time controls. By the first definition I gave above - it also outsearches Stockfish and has a lower branching factor. However it doesn't play better than Stockfish or even Komodo 1.2. We did not set out to make this program especially fast, we just implemented things that seemed to work in making it stronger.

We discovered that this new version has a lower branching factor but in test games it scales worse than komodo 1.2. When I say "scales worse" I mean that it plays just as well at fast time controls, but with deeper searches it falls behind and the deeper you search, the more of a superiority komodo 1.2 has.

So we tried being a bit more conservative with this version (with LMR specifically) in order to try to understand what was going on. When we did this, the program gets weaker and the time/elo scalability does not improve. It's just a slower program that does not scale as well.

So there is something going on that we are not understanding. The programs are very similar, but lots of minor differences. And it doesn't appear to be a simple matter of being a little more conservative or a little less conservative.

I do not understand what is the reason that the new version has a lower branching factor than komodo in the first place.

I understand that it cannot be because of more aggresive LMR(otherwise
it could mean that you can get komodo by being more conservative with LMR)

We do not understand what is going on either, but we are trying to get to the bottom of it as we feel it will help us learn how to make a stronger program.