For those who are interested...
I'm stopping the Stockfish (SF) vs. asmFish match after one week at 100 games. The score is now 15:6 in favor of SF after 99 games (game 100 is 40 moves deep and equal). That's nearly a 3x greater victory factor! Here quality was a greater factor than calculating speed in a match that's virtually designed for asmFish's superior speed.
SF's score could have been higher as SF played weakly in a couple of endgames, failing to convert wins. I only use 5-man TableBases, and the engines were otherwise too stupid to have the technique to convert wins with more than 5 pieces. Also, a couple of times SF incorrectly simplified a dominant middlegame into an endgame of R + RP + BP versus R. It incorrectly saw the two pawn advantage as winning, not knowing that particular endgame is almost always drawn. Perhaps this missing endgame knowledge might be something that should be added to SF's codes.
For some, 100 games is too few for the result to have statistical value. I disagree as these games ran nearly 3 hours each. So, the engines had sufficient time/move to demonstrate their qualities. Again, this match was meant to be more of a tactics test than overall game test.
I'll audit each game when game #100 is completed.
All the best,
-Steve-
asmFish
Moderators: hgm, Rebel, chrisw
-
- Posts: 2488
- Joined: Wed Mar 08, 2006 9:40 pm
- Location: Eden Prairie, Minnesota
- Full name: Stephen Ham
-
- Posts: 4889
- Joined: Thu Mar 09, 2006 6:34 am
- Location: Pen Argyl, Pennsylvania
Re: asmFish
amsFish is thankless job. It was pretty cool when they were able to keep current and you would get the little speed bump. Maybe if someone can come up with away to make it less tedious, it would be more sustainable. I do tip my hat for their efforts.Stephen Ham wrote: ↑Sun May 05, 2019 2:56 am For those who are interested...
I'm stopping the Stockfish (SF) vs. asmFish match after one week at 100 games. The score is now 15:6 in favor of SF after 99 games (game 100 is 40 moves deep and equal). That's nearly a 3x greater victory factor! Here quality was a greater factor than calculating speed in a match that's virtually designed for asmFish's superior speed.
SF's score could have been higher as SF played weakly in a couple of endgames, failing to convert wins. I only use 5-man TableBases, and the engines were otherwise too stupid to have the technique to convert wins with more than 5 pieces. Also, a couple of times SF incorrectly simplified a dominant middlegame into an endgame of R + RP + BP versus R. It incorrectly saw the two pawn advantage as winning, not knowing that particular endgame is almost always drawn. Perhaps this missing endgame knowledge might be something that should be added to SF's codes.
For some, 100 games is too few for the result to have statistical value. I disagree as these games ran nearly 3 hours each. So, the engines had sufficient time/move to demonstrate their qualities. Again, this match was meant to be more of a tactics test than overall game test.
I'll audit each game when game #100 is completed.
All the best,
-Steve-
edit - btw , my results were very similar to yours with respect to asmfish lagging behind cur-dev-SF.
-
- Posts: 2488
- Joined: Wed Mar 08, 2006 9:40 pm
- Location: Eden Prairie, Minnesota
- Full name: Stephen Ham
Re: asmFish
Hi Mike,
Thanks for your interest. Like you, I wish the asmFish effort was sustainable since conceptually, its superior speed would make it the strongest Stockfish (SF).
I'm auditing the games now and something interesting is apparent. It was my perception while watching the games that while asmFish was 1,000-2,000 kN/s faster, SF seemed to search deeper. I just shrugged this notion off, thinking "That's impossible!" After all, asmFish IS Stockfish, just an outdated version that's much faster. So if one SF version is faster than the other, surely it's searching to greater overall depths too. But now after auditing the first three games (OK, I've got 97 more to go, so this data is preliminary), I have hard evidence supporting my perception.
In game #1, SF's speed average was 6,309 kN/s versus 7,887 for asmFish. But SF searched to an average depth of 40.5 plies, versus 37.0 for asmFish.
In game #2, asmFish searched at an average of 7,419 kN/s to an average depth of 42.5 plies. But SF's numbers are: 6,018 kN/s and 49.6 plies.
In game #3, asmFish searched at an average of 6,344 kN/s to average depth of 36.9 plies. But SF's numbers are: 5,131 kN/s and 39.5 plies. I'll bet the remaining 97 games confirm my perception from watching some latter games.
Mike, does this suggest that recent SF coding results in more radical pruning to reach greater search depth?
All the very best,
-Steve-
Thanks for your interest. Like you, I wish the asmFish effort was sustainable since conceptually, its superior speed would make it the strongest Stockfish (SF).
I'm auditing the games now and something interesting is apparent. It was my perception while watching the games that while asmFish was 1,000-2,000 kN/s faster, SF seemed to search deeper. I just shrugged this notion off, thinking "That's impossible!" After all, asmFish IS Stockfish, just an outdated version that's much faster. So if one SF version is faster than the other, surely it's searching to greater overall depths too. But now after auditing the first three games (OK, I've got 97 more to go, so this data is preliminary), I have hard evidence supporting my perception.
In game #1, SF's speed average was 6,309 kN/s versus 7,887 for asmFish. But SF searched to an average depth of 40.5 plies, versus 37.0 for asmFish.
In game #2, asmFish searched at an average of 7,419 kN/s to an average depth of 42.5 plies. But SF's numbers are: 6,018 kN/s and 49.6 plies.
In game #3, asmFish searched at an average of 6,344 kN/s to average depth of 36.9 plies. But SF's numbers are: 5,131 kN/s and 39.5 plies. I'll bet the remaining 97 games confirm my perception from watching some latter games.
Mike, does this suggest that recent SF coding results in more radical pruning to reach greater search depth?
All the very best,
-Steve-
-
- Posts: 5563
- Joined: Tue Feb 28, 2012 11:56 pm
Re: asmFish
The old asmFIsh was replaced with pedantFish when it was found that pedantFish was about 2 Elo better than asmFish (despite being very slightly slower in nps). asmFish has been functionally identical to Stockfish ever since. So the difference between asmFish and the corresponding version of Stockfish is speed and only speed.
Being 10% faster in nps is equivalent to having 10% more thinking time. This makes a difference at both short and long time controls.The speed of PedantFish was about 10 % higher than Stockfish but this 10 % was important in very short games only.
Of course the Elo improvements made to Stockfish since September 2018 (or to whatever point asmFish has been updated) may well exceed the Elo gained by 10-25% more speed.
-
- Posts: 5563
- Joined: Tue Feb 28, 2012 11:56 pm
Re: asmFish
So you have learned that the logic behind your "surely" is flawed.Stephen Ham wrote: ↑Sun May 05, 2019 4:55 am I'm auditing the games now and something interesting is apparent. It was my perception while watching the games that while asmFish was 1,000-2,000 kN/s faster, SF seemed to search deeper. I just shrugged this notion off, thinking "That's impossible!" After all, asmFish IS Stockfish, just an outdated version that's much faster. So if one SF version is faster than the other, surely it's searching to greater overall depths too. But now after auditing the first three games (OK, I've got 97 more to go, so this data is preliminary), I have hard evidence supporting my perception.
So nps is not everything. The next thing you may learn is that a comparison of search depths reached by different engines is not everything either. (Yes, SF of today is a different engine than SF of last year.)
-
- Posts: 3657
- Joined: Wed Nov 18, 2015 11:41 am
- Location: hungary
Re: asmFish
If you are right what was the sense to make asmFish AND pedantFish simultaneously?
I agree.
But in an STC match the Elo difference becomes obvious after some tens games meanwhile in a LTC match you need many thousand games to see the difference. An average engine user surely does not play such a lot of games.
I agree.
-
- Posts: 4606
- Joined: Wed Oct 01, 2008 6:33 am
- Location: Regensburg, Germany
- Full name: Guenther Simon
Re: asmFish
-
- Posts: 5563
- Joined: Tue Feb 28, 2012 11:56 pm
Re: asmFish
asmFish included a few speed optimizations that caused node counts to be different from those of Stockfish because they caused some changes in move ordering.
I think the main differences were:
- the implementation in do_move() of castling, and
- the elimination of piece lists.
Because different node counts make it really difficult to find bugs in asmFish, asmFish's source also implemented SF's way of doing castling and piece lists. asmFish and pedantFish were "compiled" (assembled) from the same code, with either the asmFIsh or the pedantFish implementations for castling and piece lists enabled.
So asmFish development would have been impossible without pedantFish. But since the higher speed of asmFish did not translate into more Elo, there was no reason to keep it. Therefore asmFish was removed and pedantFish was renamed into asmFish.
I don't think so. STC matches are more noisy (= lower draw rate) than LTC matches, so you need more games rather than less. (Of course playing thousands of games is much easier at STC than at LTC.)I agree.
But in an STC match the Elo difference becomes obvious after some tens games meanwhile in a LTC match you need many thousand games to see the difference. An average engine user surely does not play such a lot of games.
-
- Posts: 3657
- Joined: Wed Nov 18, 2015 11:41 am
- Location: hungary
Re: asmFish
Maybe it would be better to keep the name "pedantFish" - for the shake of unanimity, I think.syzygy wrote: ↑Sun May 05, 2019 6:52 pmasmFish included a few speed optimizations that caused node counts to be different from those of Stockfish because they caused some changes in move ordering.
I think the main differences were:
- the implementation in do_move() of castling, and
- the elimination of piece lists.
Because different node counts make it really difficult to find bugs in asmFish, asmFish's source also implemented SF's way of doing castling and piece lists. asmFish and pedantFish were "compiled" (assembled) from the same code, with either the asmFIsh or the pedantFish implementations for castling and piece lists enabled.
So asmFish development would have been impossible without pedantFish. But since the higher speed of asmFish did not translate into more Elo, there was no reason to keep it. Therefore asmFish was removed and pedantFish was renamed into asmFish.I don't think so. STC matches are more noisy (= lower draw rate) than LTC matches, so you need more games rather than less. (Of course playing thousands of games is much easier at STC than at LTC.)I agree.
But in an STC match the Elo difference becomes obvious after some tens games meanwhile in a LTC match you need many thousand games to see the difference. An average engine user surely does not play such a lot of games.
-
- Posts: 1080
- Joined: Fri Sep 16, 2016 6:55 pm
- Location: USA/Minnesota
- Full name: Leo Anger
Re: asmFish
What is everything?syzygy wrote: ↑Sun May 05, 2019 3:02 pmSo you have learned that the logic behind your "surely" is flawed.Stephen Ham wrote: ↑Sun May 05, 2019 4:55 am I'm auditing the games now and something interesting is apparent. It was my perception while watching the games that while asmFish was 1,000-2,000 kN/s faster, SF seemed to search deeper. I just shrugged this notion off, thinking "That's impossible!" After all, asmFish IS Stockfish, just an outdated version that's much faster. So if one SF version is faster than the other, surely it's searching to greater overall depths too. But now after auditing the first three games (OK, I've got 97 more to go, so this data is preliminary), I have hard evidence supporting my perception.
So nps is not everything. The next thing you may learn is that a comparison of search depths reached by different engines is not everything either. (Yes, SF of today is a different engine than SF of last year.)
Advanced Micro Devices fan.