Alpha Zero vs Stockfish 8 tournament conditions.
Moderators: hgm, Rebel, chrisw
-
- Posts: 6340
- Joined: Mon Mar 13, 2006 2:34 pm
- Location: Acworth, GA
Re: Alpha Zero vs Stockfish 8 tournament conditions.
I think people place to much 'Stock' in Stockfish. I see it here, and also witnessed it while viewing the TCEC chat of season 10. I think we are seeing too-big-to-fail syndrome happening.
"Good decisions come from experience, and experience comes from bad decisions."
__________________________________________________________________
Ted Summers
__________________________________________________________________
Ted Summers
-
- Posts: 165
- Joined: Tue Dec 02, 2014 1:29 am
Re: Alpha Zero vs Stockfish 8 tournament conditions.
to make it more practical: i want to know the price of 64core st hardware and the price of alpha zero hardware pls anyone? to make a more fair comparison
-
- Posts: 1339
- Joined: Fri Nov 02, 2012 9:43 am
- Location: New Delhi, India
Re: Alpha Zero vs Stockfish 8 tournament conditions.
+1.Evert wrote:This is irrelevant, and besises the point. People arguing about hardware or number of cores are equally missing the point.
Even discussions of which is stronger under what conditions, or whether Stockfish dev would be stronger or not, miss the point.
The astounding thing here is that Alpha Zero is in the same ballpark as Stockfish. Especially given the claim of how little time was spent training Alpha Zero.
This is meaning of the Proverb, "Missing the Wood for the Trees ".
i7 5960X @ 4.1 Ghz, 64 GB G.Skill RipJaws RAM, Twin Asus ROG Strix OC 11 GB Geforce 2080 Tis
-
- Posts: 89
- Joined: Sat Sep 13, 2014 4:12 pm
- Location: Zagreb, Croatia
- Full name: Branko Radovanović
Re: Alpha Zero vs Stockfish 8 tournament conditions.
Price would, of course, be a good criterion for comparison, but from an engineering standpoint it would be interesting to compare power consumption of these two systems. If Google's TPU system does not draw (much) more power than a 64-core PC, then it might be argued that it is not (much) more powerful, both in a literal and in a metaphorical sense. Does anyone know the figures?stavros wrote:to make it more practical: i want to know the price of 64core st hardware and the price of alpha zero hardware pls anyone? to make a more fair comparison
For those who believe the matchup was not fair on the account of (supposedly or not) much stronger hardware used by AlphaZero, consider the following question: how many cores on a conventional PC running Stockfish 8 would it take to outplay a 64-core PC also running Stockfish 8 by as much as 100 Elo? 256 would certainly not suffice, and my guess is that even 512 would not be enough. It's not just that AlphaZero is stronger - it's by what margin, and I'm not sure this could be explained simply by having more powerful hardware.
-
- Posts: 165
- Joined: Tue Dec 02, 2014 1:29 am
Re: Alpha Zero vs Stockfish 8 tournament conditions.
whats does that means practicaly? even a supercomputer before aplha zero can makeEvert wrote:This is irrelevant, and besises the point. People arguing about hardware or number of cores are equally missing the point.
Even discussions of which is stronger under what conditions, or whether Stockfish dev would be stronger or not, miss the point.
The astounding thing here is that Alpha Zero is in the same ballpark as Stockfish. Especially given the claim of how little time was spent training Alpha Zero.
a 80% score vs st8
its not about chess its about NN revolution with special chips(TPU-tensor cpu),thats the missed point.
-
- Posts: 2071
- Joined: Thu May 04, 2006 3:40 am
- Location: Dune
Re: Alpha Zero vs Stockfish 8 tournament conditions.
No it can't. Hydra died because Rybka running on a puny computer was able to outplay it despite Hydra running on expensive hardware.stavros wrote:whats does that means practicaly? even a supercomputer before aplha zero can makeEvert wrote:This is irrelevant, and besises the point. People arguing about hardware or number of cores are equally missing the point.
Even discussions of which is stronger under what conditions, or whether Stockfish dev would be stronger or not, miss the point.
The astounding thing here is that Alpha Zero is in the same ballpark as Stockfish. Especially given the claim of how little time was spent training Alpha Zero.
a 80% score vs st8
its not about chess its about NN revolution with special chips(TPU-tensor cpu),thats the missed point.
Johnny runs on these crazy 2000+ core servers and it's no match for Stockfish. There's only so much that hardware can do for software.
Stockfish 8 probably doesn't get much of an elo increase after 32 cores. You can give Stockfish 8 two billion cores and it wouldn't beat AlphaZero.
-
- Posts: 273
- Joined: Wed Aug 24, 2016 9:49 pm
Re: Alpha Zero vs Stockfish 8 tournament conditions.
This.Branko Radovanovic wrote:Price would, of course, be a good criterion for comparison, but from an engineering standpoint it would be interesting to compare power consumption of these two systems. If Google's TPU system does not draw (much) more power than a 64-core PC, then it might be argued that it is not (much) more powerful, both in a literal and in a metaphorical sense. Does anyone know the figures?stavros wrote:to make it more practical: i want to know the price of 64core st hardware and the price of alpha zero hardware pls anyone? to make a more fair comparison
For those who believe the matchup was not fair on the account of (supposedly or not) much stronger hardware used by AlphaZero, consider the following question: how many cores on a conventional PC running Stockfish 8 would it take to outplay a 64-core PC also running Stockfish 8 by as much as 100 Elo? 256 would certainly not suffice, and my guess is that even 512 would not be enough. It's not just that AlphaZero is stronger - it's by what margin, and I'm not sure this could be explained simply by having more powerful hardware.
What they basically showed is that alphaZero scales much better than the current alpha-beta-engines. The scaling beyond 64 cores is, frankly, quite bad.
-
- Posts: 4190
- Joined: Wed Nov 25, 2009 1:47 am
Re: Alpha Zero vs Stockfish 8 tournament conditions.
1 Google TPU is around 50W, basically you have them 4 and another Haswell to run actual MCTS on those so around 300W.Branko Radovanovic wrote:Price would, of course, be a good criterion for comparison, but from an engineering standpoint it would be interesting to compare power consumption of these two systems. If Google's TPU system does not draw (much) more power than a 64-core PC, then it might be argued that it is not (much) more powerful, both in a literal and in a metaphorical sense. Does anyone know the figures?
SF's hardware was most probably two 32 core CPUs each at 150W, so around 300W also.
However, even they were on the same wattage, problem is Alpha0 was running on specialized hardware, while SF was running on general-purpose hardware. That on itself is totally unfair point.
To make it fair, one could run SF on smaller Haswell for the search (the same one they used for Alpha0), and using 10 Xilinx UltraScale+ FPGA chips for running evaluation, each chips consuming 20W, and running move generator and 100 evaluation terms in parallel on like 300MHz in lets say 10clock cycles (including movegen). This would be DeepBlue effort but on today's cutting-edge hardware and software. In that way SF system would still consume 300W, but have 30Bnps on 10 cores performance. I can immediately tell you that it would be at least 400-500Elo stronger than current SF.
And building such a system would cost less than what was spent just on electricity to train Alpha0.
And certainly would require less working hours than what has been invested in Alpha0.
So sorry, but Alpha0 is not the holy grail or best way to make a chess machine. It is just the most hyped one atm.
Talking about the margin in B40 Sicilian difference is only 38.5Elo. 20 wins for Alpha0 vs 9 wins for SF8, outdated, 1GB hash, no TBs, no opening-books, ridiculous TC, each of these points taking away at least 15-20Elo from SF. And you still think it's a big margin???It's not just that AlphaZero is stronger - it's by what margin, and I'm not sure this could be explained simply by having more powerful hardware.
Have you seen the training diagram (Fig.1 in the paper)???
After first 4 hours of training, for next 8 hours they improved only lousy 30Elo until they totally saturated. They could continue training for months and they would most probably just get it worse not better for an inch.
If they really had a comfortable margin, they wouldn't rely on such a lousy tricks essentially crippling SF just to win. You think these ppl at Google are stupid and don't know what 1GB of hash for 64 core machine means? Or normal TC, or opening book???
-
- Posts: 1080
- Joined: Fri Sep 16, 2016 6:55 pm
- Location: USA/Minnesota
- Full name: Leo Anger
Re: Alpha Zero vs Stockfish 8 tournament conditions.
This is one of the best posts I have seen on this topic. You are never going to win without an opening book. 1 GB of RAM is a joke. 1 minute a move is totally insufficient. What engine cant beat SF 8 at this point. (I know that some still can't.) Give it Asmfish cerebellum. Like Milos said, the thing could have been saturated at 4 hours or so. Maybe that's why they didn't give it a week to learn more.Milos wrote:1 Google TPU is around 50W, basically you have them 4 and another Haswell to run actual MCTS on those so around 300W.Branko Radovanovic wrote:Price would, of course, be a good criterion for comparison, but from an engineering standpoint it would be interesting to compare power consumption of these two systems. If Google's TPU system does not draw (much) more power than a 64-core PC, then it might be argued that it is not (much) more powerful, both in a literal and in a metaphorical sense. Does anyone know the figures?
SF's hardware was most probably two 32 core CPUs each at 150W, so around 300W also.
However, even they were on the same wattage, problem is Alpha0 was running on specialized hardware, while SF was running on general-purpose hardware. That on itself is totally unfair point.
To make it fair, one could run SF on smaller Haswell for the search (the same one they used for Alpha0), and using 10 Xilinx UltraScale+ FPGA chips for running evaluation, each chips consuming 20W, and running move generator and 100 evaluation terms in parallel on like 300MHz in lets say 10clock cycles (including movegen). This would be DeepBlue effort but on today's cutting-edge hardware and software. In that way SF system would still consume 300W, but have 30Bnps on 10 cores performance. I can immediately tell you that it would be at least 400-500Elo stronger than current SF.
And building such a system would cost less than what was spent just on electricity to train Alpha0.
And certainly would require less working hours than what has been invested in Alpha0.
So sorry, but Alpha0 is not the holy grail or best way to make a chess machine. It is just the most hyped one atm.
Talking about the margin in B40 Sicilian difference is only 38.5Elo. 20 wins for Alpha0 vs 9 wins for SF8, outdated, 1GB hash, no TBs, no opening-books, ridiculous TC, each of these points taking away at least 15-20Elo from SF. And you still think it's a big margin???It's not just that AlphaZero is stronger - it's by what margin, and I'm not sure this could be explained simply by having more powerful hardware.
Have you seen the training diagram (Fig.1 in the paper)???
After first 4 hours of training, for next 8 hours they improved only lousy 30Elo until they totally saturated. They could continue training for months and they would most probably just get it worse not better for an inch.
If they really had a comfortable margin, they wouldn't rely on such a lousy tricks essentially crippling SF just to win. You think these ppl at Google are stupid and don't know what 1GB of hash for 64 core machine means? Or normal TC, or opening book???
Last edited by Leo on Thu Dec 07, 2017 6:35 pm, edited 1 time in total.
Advanced Micro Devices fan.
-
- Posts: 433
- Joined: Fri Dec 16, 2016 11:04 am
- Location: France
- Full name: Richard Delorme
Re: Alpha Zero vs Stockfish 8 tournament conditions.
What do you put on the FPGA, just the eval? the eval + qs search? the upper part of the search? with the hash table ? How would you make it parallel? Maybe you could reach 30 Bnps, but I doubt it will be with the same efficiency than in current SF search.Milos wrote:To make it fair, one could run SF on smaller Haswell for the search (the same one they used for Alpha0), and using 10 Xilinx UltraScale+ FPGA chips for running evaluation, each chips consuming 20W, and running move generator and 100 evaluation terms in parallel on like 300MHz in lets say 10clock cycles (including movegen). This would be DeepBlue effort but on today's cutting-edge hardware and software. In that way SF system would still consume 300W, but have 30Bnps on 10 cores performance. I can immediately tell you that it would be at least 400-500Elo stronger than current SF.
Richard Delorme