pedantfish vs SF8

[Werewolf]

This may be a silly question, but I'd like to ask it as it has has come up on several forums.

SF8 has been released.
Shortly afterwards, PedantFishW 2016-11-04 (Nov 4th) was also released.

On my machine it's about 10% faster than SF8. My results however (LTC Game in 3 hours + 90 sec) slightly favour SF8, but there were only 200 games.

I would be very happy for someone to tell me that's just because I haven't played enough games, but can I check:

1) PedantFishW 2016-11-04 is functionally EXACTLY the same as Stockfish 8, except slightly faster?

2) Given enough games we could demonstrate PedantFishW 2016-11-04 will ALWAYS be stronger than Stockfish 8 at ALL time controls?

And can I just ask: what is the "W" at the end of the name for?
Thanks.

[syzygy]

The W stands for Windows. L for Linux.

200 games is not much.

As far as I know pedantFish is functionally identical in single-threaded search. This is relatively easy to check by comparing node counts with SF for a number of long single-threaded searches on various positions.

It is not impossible that there are differences (bugs) in time management. Comparing node counts won't uncover such differences.

It is not impossible that there are difference in the smp implementation. This is not very likely, since lazy smp has such a simple implementation, but single-threaded node counts obviously do not show that multi-threaded search is identical. (And multi-threaded node counts won't show this either because every multi-threaded run will be slightly different anyway.)

[Werewolf]

Thanks, that's really helpful.

So for my intended use - IDeA analysis in Aquarium, where it is single thread usage with a fixed amount of time being spent on each move - I think I can say with CERTAINTY that PedantFish will be stronger.

Correct?

[syzygy]

Thanks, that's really helpful.

So for my intended use - IDeA analysis in Aquarium, where it is single thread usage with a fixed amount of time being spent on each move - I think I can say with CERTAINTY that PedantFish will be stronger.

Correct?

Correct.

Or you could decrease the fixed amount of time per move (e.g. by 15% if pedantFish is 15% faster) and get the same results but more quickly.

[Werewolf]

Brilliant, thanks.

[MikeB]

This may be a silly question, but I'd like to ask it as it has has come up on several forums.

SF8 has been released.
Shortly afterwards, PedantFishW 2016-11-04 (Nov 4th) was also released.

On my machine it's about 10% faster than SF8. My results however (LTC Game in 3 hours + 90 sec) slightly favour SF8, but there were only 200 games.

I would be very happy for someone to tell me that's just because I haven't played enough games, but can I check:

1) PedantFishW 2016-11-04 is functionally EXACTLY the same as Stockfish 8, except slightly faster?

2) Given enough games we could demonstrate PedantFishW 2016-11-04 will ALWAYS be stronger than Stockfish 8 at ALL time controls?

And can I just ask: what is the "W" at the end of the name for?
Thanks.

A 10% pickup would at best ~5 ELO in self play. ( lower in random opponent play, maybe 3.5 ELO). It would take thousands of games to verify such a gain. If it is otherwise identical in all respects , it would be a given that it is stronger. They had locked down SF 8 on 10/30 - officially naming it SF 8 on 11/1. It was not reopened for development until 11/5. So based on those dates - what you have should be identical to SF 8 in all respects.

[Eelco de Groot]

I think that is debatable. The best result from asmFish is almost 25 Elo above Stockfish 8 in Stefan Pohl's list.

Code: Select all

   5 asmFish 161004 x64       : 3414    8    8  7000    81.5 %   3147   29.2 %
   6 asmFish 160830 x64       : 3397    7    7  7000    80.0 %   3147   29.6 %
   7 Stockfish 161025 x64     : 3394    7    7  7000    79.6 %   3147   31.1 %
   8 Stockfish 8 161101       : 3390    7    7  7000    78.9 %   3150   31.8 % (new)

With very long time controls, and different openings without many opposite castling to get more decisive results, there will be more draws. But this was a question about use for iDea.

It does likely that the development of asmFish will be on hold though. That was effectively the first act the computer chess community has felt from Donald Trump as president elect, even before the final results of all the states of the 8 november election were in. Probably Iran could use this dangerous AI for military uses. Just like Belle from Ken Thompson I believe, wasn't there some story about that? I don't quite remember.

[MikeB]

Since you replied to my post , please clarify what is debateable. Thanks

[Eelco de Groot]

Well. I think if asmFish is consistently about 15 points over the Stockfish development version of the same time, not exactly same time because they are not usually tested at the same time, but I think that you will get more than 5 Elo even at say game in 2 hours per player. That is still testable. Above that we just don't have enough information. Self play not considering, as Stefan's list is against other engines. So PedantFish might be much weaker, but I think you could get 15 Elo with AsmFish up to a very decent timecontrol.

[MikeB]

Well. I think if asmFish is consistently about 15 points over the Stockfish development version of the same time, not exactly same time because they are not usually tested at the same time, but I think that you will get more than 5 Elo even at say game in 2 hours per player. That is still testable. Above that we just don't have enough information. Self play not considering, as Stefan's list is against other engines. So PedantFish might be much weaker, but I think you could get 15 Elo with AsmFish up to a very decent timecontrol.

Well a 15 ELO increase , with everything else being equal, implies a 30 % in speed - with random oppenents it would be slightly lower at this level. Its more or less accepted doubling of speed at this level is worth perhaps a 35 point ELO gain. Naturally self play with exact same engines would inflate the ELO as the faster engine sees everything the slower identical engine sees plus a little bit more. Totally different scenario when you play against random opponents. But to tout a 25 ELO gain with a 10% increase in speed with what the original poster referenced would be incorrect.