asmFish

[Stephen Ham]

For those who are interested...

I'm stopping the Stockfish (SF) vs. asmFish match after one week at 100 games. The score is now 15:6 in favor of SF after 99 games (game 100 is 40 moves deep and equal). That's nearly a 3x greater victory factor! Here quality was a greater factor than calculating speed in a match that's virtually designed for asmFish's superior speed.

SF's score could have been higher as SF played weakly in a couple of endgames, failing to convert wins. I only use 5-man TableBases, and the engines were otherwise too stupid to have the technique to convert wins with more than 5 pieces. Also, a couple of times SF incorrectly simplified a dominant middlegame into an endgame of R + RP + BP versus R. It incorrectly saw the two pawn advantage as winning, not knowing that particular endgame is almost always drawn. Perhaps this missing endgame knowledge might be something that should be added to SF's codes.

For some, 100 games is too few for the result to have statistical value. I disagree as these games ran nearly 3 hours each. So, the engines had sufficient time/move to demonstrate their qualities. Again, this match was meant to be more of a tactics test than overall game test.

I'll audit each game when game #100 is completed.

All the best,
-Steve-

[MikeB]

For those who are interested...

I'm stopping the Stockfish (SF) vs. asmFish match after one week at 100 games. The score is now 15:6 in favor of SF after 99 games (game 100 is 40 moves deep and equal). That's nearly a 3x greater victory factor! Here quality was a greater factor than calculating speed in a match that's virtually designed for asmFish's superior speed.

SF's score could have been higher as SF played weakly in a couple of endgames, failing to convert wins. I only use 5-man TableBases, and the engines were otherwise too stupid to have the technique to convert wins with more than 5 pieces. Also, a couple of times SF incorrectly simplified a dominant middlegame into an endgame of R + RP + BP versus R. It incorrectly saw the two pawn advantage as winning, not knowing that particular endgame is almost always drawn. Perhaps this missing endgame knowledge might be something that should be added to SF's codes.

For some, 100 games is too few for the result to have statistical value. I disagree as these games ran nearly 3 hours each. So, the engines had sufficient time/move to demonstrate their qualities. Again, this match was meant to be more of a tactics test than overall game test.

I'll audit each game when game #100 is completed.

All the best,
-Steve-

amsFish is thankless job. It was pretty cool when they were able to keep current and you would get the little speed bump. Maybe if someone can come up with away to make it less tedious, it would be more sustainable. I do tip my hat for their efforts.

edit - btw , my results were very similar to yours with respect to asmfish lagging behind cur-dev-SF.

[Stephen Ham]

Hi Mike,

Thanks for your interest. Like you, I wish the asmFish effort was sustainable since conceptually, its superior speed would make it the strongest Stockfish (SF).

I'm auditing the games now and something interesting is apparent. It was my perception while watching the games that while asmFish was 1,000-2,000 kN/s faster, SF seemed to search deeper. I just shrugged this notion off, thinking "That's impossible!" After all, asmFish IS Stockfish, just an outdated version that's much faster. So if one SF version is faster than the other, surely it's searching to greater overall depths too. But now after auditing the first three games (OK, I've got 97 more to go, so this data is preliminary), I have hard evidence supporting my perception.

In game #1, SF's speed average was 6,309 kN/s versus 7,887 for asmFish. But SF searched to an average depth of 40.5 plies, versus 37.0 for asmFish.

In game #2, asmFish searched at an average of 7,419 kN/s to an average depth of 42.5 plies. But SF's numbers are: 6,018 kN/s and 49.6 plies.

In game #3, asmFish searched at an average of 6,344 kN/s to average depth of 36.9 plies. But SF's numbers are: 5,131 kN/s and 39.5 plies. I'll bet the remaining 97 games confirm my perception from watching some latter games.

Mike, does this suggest that recent SF coding results in more radical pruning to reach greater search depth?

All the very best,
-Steve-

[syzygy]

It would not be a superfluous test because Asmfish is not an asm-copy of Stockfish and the difference between them more then the speed. There was a real copy named pedantFish, but this version ended in the past.

The old asmFIsh was replaced with pedantFish when it was found that pedantFish was about 2 Elo better than asmFish (despite being very slightly slower in nps). asmFish has been functionally identical to Stockfish ever since. So the difference between asmFish and the corresponding version of Stockfish is speed and only speed.

The speed of PedantFish was about 10 % higher than Stockfish but this 10 % was important in very short games only.

Being 10% faster in nps is equivalent to having 10% more thinking time. This makes a difference at both short and long time controls.

Of course the Elo improvements made to Stockfish since September 2018 (or to whatever point asmFish has been updated) may well exceed the Elo gained by 10-25% more speed.

[syzygy]

I'm auditing the games now and something interesting is apparent. It was my perception while watching the games that while asmFish was 1,000-2,000 kN/s faster, SF seemed to search deeper. I just shrugged this notion off, thinking "That's impossible!" After all, asmFish IS Stockfish, just an outdated version that's much faster. So if one SF version is faster than the other, surely it's searching to greater overall depths too. But now after auditing the first three games (OK, I've got 97 more to go, so this data is preliminary), I have hard evidence supporting my perception.

So you have learned that the logic behind your "surely" is flawed.

So nps is not everything. The next thing you may learn is that a comparison of search depths reached by different engines is not everything either. (Yes, SF of today is a different engine than SF of last year.)

[corres]

...
asmFish has been functionally identical to Stockfish ever since. So the difference between asmFish and the corresponding version of Stockfish is speed and only speed.

If you are right what was the sense to make asmFish AND pedantFish simultaneously?

Being 10% faster in nps is equivalent to having 10% more thinking time. This makes a difference at both short and long time controls.

I agree.
But in an STC match the Elo difference becomes obvious after some tens games meanwhile in a LTC match you need many thousand games to see the difference. An average engine user surely does not play such a lot of games.

Of course the Elo improvements made to Stockfish since September 2018 (or to whatever point asmFish has been updated) may well exceed the Elo gained by 10-25% more speed.

I agree.

[Guenther]

...
asmFish has been functionally identical to Stockfish ever since. So the difference between asmFish and the corresponding version of Stockfish is speed and only speed.

If you are right what was the sense to make asmFish AND pedantFish simultaneously?

Being 10% faster in nps is equivalent to having 10% more thinking time. This makes a difference at both short and long time controls.

I agree.
But in an STC match the Elo difference becomes obvious after some tens games...

Huh? Still on the strange side...

[syzygy]

...
asmFish has been functionally identical to Stockfish ever since. So the difference between asmFish and the corresponding version of Stockfish is speed and only speed.

If you are right what was the sense to make asmFish AND pedantFish simultaneously?

asmFish included a few speed optimizations that caused node counts to be different from those of Stockfish because they caused some changes in move ordering.
I think the main differences were:
- the implementation in do_move() of castling, and
- the elimination of piece lists.

Because different node counts make it really difficult to find bugs in asmFish, asmFish's source also implemented SF's way of doing castling and piece lists. asmFish and pedantFish were "compiled" (assembled) from the same code, with either the asmFIsh or the pedantFish implementations for castling and piece lists enabled.

So asmFish development would have been impossible without pedantFish. But since the higher speed of asmFish did not translate into more Elo, there was no reason to keep it. Therefore asmFish was removed and pedantFish was renamed into asmFish.

Being 10% faster in nps is equivalent to having 10% more thinking time. This makes a difference at both short and long time controls.

I agree.
But in an STC match the Elo difference becomes obvious after some tens games meanwhile in a LTC match you need many thousand games to see the difference. An average engine user surely does not play such a lot of games.

I don't think so. STC matches are more noisy (= lower draw rate) than LTC matches, so you need more games rather than less. (Of course playing thousands of games is much easier at STC than at LTC.)

[corres]

...
asmFish has been functionally identical to Stockfish ever since. So the difference between asmFish and the corresponding version of Stockfish is speed and only speed.

If you are right what was the sense to make asmFish AND pedantFish simultaneously?

asmFish included a few speed optimizations that caused node counts to be different from those of Stockfish because they caused some changes in move ordering.
I think the main differences were:
- the implementation in do_move() of castling, and
- the elimination of piece lists.
Because different node counts make it really difficult to find bugs in asmFish, asmFish's source also implemented SF's way of doing castling and piece lists. asmFish and pedantFish were "compiled" (assembled) from the same code, with either the asmFIsh or the pedantFish implementations for castling and piece lists enabled.
So asmFish development would have been impossible without pedantFish. But since the higher speed of asmFish did not translate into more Elo, there was no reason to keep it. Therefore asmFish was removed and pedantFish was renamed into asmFish.

Being 10% faster in nps is equivalent to having 10% more thinking time. This makes a difference at both short and long time controls.

I agree.
But in an STC match the Elo difference becomes obvious after some tens games meanwhile in a LTC match you need many thousand games to see the difference. An average engine user surely does not play such a lot of games.

I don't think so. STC matches are more noisy (= lower draw rate) than LTC matches, so you need more games rather than less. (Of course playing thousands of games is much easier at STC than at LTC.)

Maybe it would be better to keep the name "pedantFish" - for the shake of unanimity, I think.

[Leo]

I'm auditing the games now and something interesting is apparent. It was my perception while watching the games that while asmFish was 1,000-2,000 kN/s faster, SF seemed to search deeper. I just shrugged this notion off, thinking "That's impossible!" After all, asmFish IS Stockfish, just an outdated version that's much faster. So if one SF version is faster than the other, surely it's searching to greater overall depths too. But now after auditing the first three games (OK, I've got 97 more to go, so this data is preliminary), I have hard evidence supporting my perception.

So you have learned that the logic behind your "surely" is flawed.

So nps is not everything. The next thing you may learn is that a comparison of search depths reached by different engines is not everything either. (Yes, SF of today is a different engine than SF of last year.)

What is everything?