Alpha Zero vs Stockfish 8 tournament conditions.

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.

Alpha Zero vs Stockfish 8 tournament conditions.

Poll ended at Fri Dec 08, 2017 4:15 am

The time per move and hardware etc was fair.
27
52%
Google set it up to give Alpha Zero an edge.
25
48%
 
Total votes: 52

Milos
Posts: 2993
Joined: Wed Nov 25, 2009 12:47 am

Re: Alpha Zero vs Stockfish 8 tournament conditions.

Post by Milos » Thu Dec 07, 2017 6:12 pm

Lyudmil Tsvetkov wrote:
stavros wrote:to make it more practical: i want to know the price of 64core st hardware and the price of alpha zero hardware pls anyone? to make a more fair comparison
+ 10, most probably larger than 20/1, just as in the case with hardware speed.
And it used significantly more memory.
And an opening book, which actually decided the whole match.
I made a calculation here.

It would cost from 250k$ to half a million bucks.

CheckersGuy
Posts: 272
Joined: Wed Aug 24, 2016 7:49 pm

Re: Alpha Zero vs Stockfish 8 tournament conditions.

Post by CheckersGuy » Thu Dec 07, 2017 6:44 pm

Milos wrote:
Lyudmil Tsvetkov wrote:
stavros wrote:to make it more practical: i want to know the price of 64core st hardware and the price of alpha zero hardware pls anyone? to make a more fair comparison
+ 10, most probably larger than 20/1, just as in the case with hardware speed.
And it used significantly more memory.
And an opening book, which actually decided the whole match.
I made a calculation here.

It would cost from 250k$ to half a million bucks.
We don't know the price of a TPU so your comparision doesn't mean anything

User avatar
reflectionofpower
Posts: 1449
Joined: Fri Mar 01, 2013 4:28 pm
Location: USA

Re: Alpha Zero vs Stockfish 8 tournament conditions.

Post by reflectionofpower » Thu Dec 07, 2017 6:46 pm

shrapnel wrote:The Age of Chess Engines based on alpha-beta Search is over.
This is just nit-picking/quibbling by old men who are too set in their ways of thinking to fully grasp or accept the sudden change in the World of Computer chess.
No point arguing with them, because if the change is for real, they will have to accept the Truth sometime or the other or risk making themselves a laughing stock.
:lol:
"hodie mihi, cras tibi"

Lonnie

Milos
Posts: 2993
Joined: Wed Nov 25, 2009 12:47 am

Re: Alpha Zero vs Stockfish 8 tournament conditions.

Post by Milos » Thu Dec 07, 2017 7:05 pm

CheckersGuy wrote:
Milos wrote:
Lyudmil Tsvetkov wrote:
stavros wrote:to make it more practical: i want to know the price of 64core st hardware and the price of alpha zero hardware pls anyone? to make a more fair comparison
+ 10, most probably larger than 20/1, just as in the case with hardware speed.
And it used significantly more memory.
And an opening book, which actually decided the whole match.
I made a calculation here.

It would cost from 250k$ to half a million bucks.
We don't know the price of a TPU so your comparision doesn't mean anything
Price is infinite because no one can buy it. Comparing with equivalent V100 system is certainly much cheaper and certainly makes sense. What obviously doesn't make sense is your comment.

CheckersGuy
Posts: 272
Joined: Wed Aug 24, 2016 7:49 pm

Re: Alpha Zero vs Stockfish 8 tournament conditions.

Post by CheckersGuy » Thu Dec 07, 2017 7:17 pm

Milos wrote:
CheckersGuy wrote:
Milos wrote:
Lyudmil Tsvetkov wrote:
stavros wrote:to make it more practical: i want to know the price of 64core st hardware and the price of alpha zero hardware pls anyone? to make a more fair comparison
+ 10, most probably larger than 20/1, just as in the case with hardware speed.
And it used significantly more memory.
And an opening book, which actually decided the whole match.
I made a calculation here.

It would cost from 250k$ to half a million bucks.
We don't know the price of a TPU so your comparision doesn't mean anything
Price is infinite because no one can buy it. Comparing with equivalent V100 system is certainly much cheaper and certainly makes sense. What obviously doesn't make sense is your comment.
You shouldnt make any price comparisions then. :shock: Additionally, The V100 chips are too general purpose compared to TPU's....

Milos
Posts: 2993
Joined: Wed Nov 25, 2009 12:47 am

Re: Alpha Zero vs Stockfish 8 tournament conditions.

Post by Milos » Thu Dec 07, 2017 7:26 pm

CheckersGuy wrote:
Milos wrote:
CheckersGuy wrote:
Milos wrote:
Lyudmil Tsvetkov wrote:
stavros wrote:to make it more practical: i want to know the price of 64core st hardware and the price of alpha zero hardware pls anyone? to make a more fair comparison
+ 10, most probably larger than 20/1, just as in the case with hardware speed.
And it used significantly more memory.
And an opening book, which actually decided the whole match.
I made a calculation here.

It would cost from 250k$ to half a million bucks.
We don't know the price of a TPU so your comparision doesn't mean anything
Price is infinite because no one can buy it. Comparing with equivalent V100 system is certainly much cheaper and certainly makes sense. What obviously doesn't make sense is your comment.
You shouldnt make any price comparisions then. :shock: Additionally, The V100 chips are too general purpose compared to TPU's....
Price comparison makes perfect sense coz Google used something that would cost at least a quarter of million bucks to run on (not yet existing) top of the range GPUs to beat old, handicapped version of chess program on computer that is under 3'000$ and ppl call it fair???
I'm just offering a different view on it.
I understand if you don't like it, but again it is not meant for you.

Branko Radovanovic
Posts: 26
Joined: Sat Sep 13, 2014 2:12 pm

Re: Alpha Zero vs Stockfish 8 tournament conditions.

Post by Branko Radovanovic » Thu Dec 07, 2017 7:35 pm

Milos wrote:1 Google TPU is around 50W, basically you have them 4 and another Haswell to run actual MCTS on those so around 300W.
SF's hardware was most probably two 32 core CPUs each at 150W, so around 300W also.
However, even they were on the same wattage, problem is Alpha0 was running on specialized hardware, while SF was running on general-purpose hardware. That on itself is totally unfair point.
To make it fair, one could run SF on smaller Haswell for the search (the same one they used for Alpha0), and using 10 Xilinx UltraScale+ FPGA chips for running evaluation, each chips consuming 20W, and running move generator and 100 evaluation terms in parallel on like 300MHz in lets say 10clock cycles (including movegen). This would be DeepBlue effort but on today's cutting-edge hardware and software. In that way SF system would still consume 300W, but have 30Bnps on 10 cores performance. I can immediately tell you that it would be at least 400-500Elo stronger than current SF.
And building such a system would cost less than what was spent just on electricity to train Alpha0.
And certainly would require less working hours than what has been invested in Alpha0.
So sorry, but Alpha0 is not the holy grail or best way to make a chess machine. It is just the most hyped one atm.

Talking about the margin in B40 Sicilian difference is only 38.5Elo. 20 wins for Alpha0 vs 9 wins for SF8, outdated, 1GB hash, no TBs, no opening-books, ridiculous TC, each of these points taking away at least 15-20Elo from SF. And you still think it's a big margin???
Have you seen the training diagram (Fig.1 in the paper)???
After first 4 hours of training, for next 8 hours they improved only lousy 30Elo until they totally saturated. They could continue training for months and they would most probably just get it worse not better for an inch.
If they really had a comfortable margin, they wouldn't rely on such a lousy tricks essentially crippling SF just to win. You think these ppl at Google are stupid and don't know what 1GB of hash for 64 core machine means? Or normal TC, or opening book???
Thanks for the power consumption info - tried to find it but had no luck...

Why do you think SF should have played with an opening book and an EGTB? AlphaZero had none. As I gather, it used actual computation to play all its moves, rather than relying on instant lookup.

SF surely lost 20-30 Elo at least due to fixed move time, but given the fact that AlphaZero itself played under the same conditions, there is no reason to believe the outcome would have been any different even with dynamic time management given to both sides. (Actually, if there is any mention of AlphaZero's time management in the paper, I've missed it.)

AlphaZero's hardware is better described as novel than specialized. There is no doubt it will become commonplace, perhaps in a matter of years, while FPGAs will remain exotic, however powerful and energy-efficient they might be.

If I had to name just one reason why AlphaZero playing chess the way it plays is a monumental achievement, it is this: in the chapter title "Anatomy of a Computer Chess Program" the authors list a number of computer chess techniques (alpha-beta search, material imbalance tables, PSQT, mobility, pawn structure eval, king safety, QS, pruning, extensions, history, SEE, heuristics, TT, etc.) and conclude AlphaZero uses none of them. Not one. It appears that the only thing SF and AlphaZero have in common is the move generator. That's mind-boggling, and remains mind-boggling even if Google's results do appear to be oversold a bit.

supersharp77
Posts: 475
Joined: Sat Jul 05, 2014 5:54 am
Location: Southwest USA

Re: Alpha Zero vs Stockfish .. A Bogus Win?

Post by supersharp77 » Thu Dec 07, 2017 7:35 pm

Graham Banks wrote:The only thing I question is that AlphaZero had an opening book based on learning over millions of games, whereas it looks as though Stockfish may not have used an opening book at all.
==============================

Note:To whom it may concern...We refuse to accept the conclusions of this "so called" defeat of Stockfish 8 by 'Alpha Zero' (a program no one seems to have heard of before this so called "result was published") If Google wants the chess engine community to accept these "conclusions" follow the 'RULES'
of CHESS ENGINE TESTING..and Enter 'ALPHA ZERO" in TCEC or some other
Recognized chess engine tournament...and let the Chips fall where they mayl....until then....The "Google Teams" results are 100% meaningless!! :D :wink:

CheckersGuy
Posts: 272
Joined: Wed Aug 24, 2016 7:49 pm

Re: Alpha Zero vs Stockfish 8 tournament conditions.

Post by CheckersGuy » Thu Dec 07, 2017 8:01 pm

Milos wrote:
CheckersGuy wrote:
Milos wrote:
CheckersGuy wrote:
Milos wrote:
Lyudmil Tsvetkov wrote:
stavros wrote:to make it more practical: i want to know the price of 64core st hardware and the price of alpha zero hardware pls anyone? to make a more fair comparison
+ 10, most probably larger than 20/1, just as in the case with hardware speed.
And it used significantly more memory.
And an opening book, which actually decided the whole match.
I made a calculation here.

It would cost from 250k$ to half a million bucks.
We don't know the price of a TPU so your comparision doesn't mean anything
Price is infinite because no one can buy it. Comparing with equivalent V100 system is certainly much cheaper and certainly makes sense. What obviously doesn't make sense is your comment.
You shouldnt make any price comparisions then. :shock: Additionally, The V100 chips are too general purpose compared to TPU's....
Price comparison makes perfect sense coz Google used something that would cost at least a quarter of million bucks to run on (not yet existing) top of the range GPUs to beat old, handicapped version of chess program on computer that is under 3'000$ and ppl call it fair???
I'm just offering a different view on it.
I understand if you don't like it, but again it is not meant for you.
You dont get it. You 250$k figure is just wrong. How can you possible know that it would be at least that much ? You cant just take the price of another chip and add it all up.... thats garbage

Milos
Posts: 2993
Joined: Wed Nov 25, 2009 12:47 am

Re: Alpha Zero vs Stockfish 8 tournament conditions.

Post by Milos » Thu Dec 07, 2017 8:09 pm

CheckersGuy wrote:You dont get it. You 250$k figure is just wrong. How can you possible know that it would be at least that much ? You cant just take the price of another chip and add it all up.... thats garbage
That's just your opinion. And you know opinions are like a..es, everyone has one. Since you really don't offer anything constructive, I wouldn't hold you any more.

Post Reply