AlphaZero Chess test - SF8 16 GB vs. 512 MB

[Milos]

Conditions:
Dual AMD Opteron 6376, Cutechess-Cli, 100 games, fixed 60 seconds per move, opening book: Noomen_2-move_Testsuite.pgn, no adjudication rules, no tablebases

Why even use 2-move testsuite? To replicate original conditions one should use only chess starting position.
Also 512MB hash on Opteron 6376 is much more favourable condition than 1GB on dual 32 core modern Xeon. More fair comparison would be with only 256MB hash vs. 16GB hash.

[Vinvin]

Conditions:
Dual AMD Opteron 6376, Cutechess-Cli, 100 games, fixed 60 seconds per move, opening book: Noomen_2-move_Testsuite.pgn, no adjudication rules, no tablebases

Stockfish 8, 31 threads, 16384 MB Hash
vs
Stockfish 8, 31 threads, 512 MB Hash

Code: Select all

1   Stockfish 8 T31 16384    +7  +3/=96/-1 51.00%   51.0/100
2   Stockfish 8 T31 512      -7  +1/=96/-3 49.00%   49.0/100[/quote]
Strange numbers. 
How to explain them ?

[Ovyron]

Conditions:
Dual AMD Opteron 6376, Cutechess-Cli, 100 games, fixed 60 seconds per move, opening book: Noomen_2-move_Testsuite.pgn, no adjudication rules, no tablebases

Stockfish 8, 31 threads, 16384 MB Hash
vs
Stockfish 8, 31 threads, 512 MB Hash

Code: Select all

1   Stockfish 8 T31 16384    +7  +3/=96/-1 51.00%   51.0/100
2   Stockfish 8 T31 512      -7  +1/=96/-3 49.00%   49.0/100[/quote]
Strange numbers. 
How to explain them ?[/quote]

Divide them by 1024 to get GB:

16384 MB Hash / 1024 = 16GB
512 MB Hash / 1024 = 0.5GB

[fastgm]

Why even use 2-move testsuite?

In order to avoid that always the same first moves/openings are played.

Also 512MB hash on Opteron 6376 is much more favourable condition than 1GB on dual 32 core modern Xeon. More fair comparison would be with only 256MB hash vs. 16GB hash.

Who really knows?

[Dann Corbit]

The only one single condition that "crippled" it, was the absence of a book.

And TBs.

TBs have a bigger impact at ultra fast TCs. IIRC, this was 1min/move on good HW, so not exactly crippling.

No, it's not crippling but you don't just have 1 cause, you have to add them all up.

1GB hash + no TBs + no book + 1 year old version + weird time control = cripple.

If you separate them, the effect is not that noticeable.

The book alone is noticeable.

Did AlphaZero have a book?
If not, then it is a moot point.

[CheckersGuy]

The only one single condition that "crippled" it, was the absence of a book.

And TBs.

TBs have a bigger impact at ultra fast TCs. IIRC, this was 1min/move on good HW, so not exactly crippling.

No, it's not crippling but you don't just have 1 cause, you have to add them all up.

1GB hash + no TBs + no book + 1 year old version + weird time control = cripple.

If you separate them, the effect is not that noticeable.

The book alone is noticeable.

Did AlphaZero have a book?
If not, then it is a moot point.

AlphaZero did not have acess to an opening book even though quite a lot of ppl think the neural network is an openin book, which it is not.
As for the tb's. If u give tablebases to SF you could give them AlphaZero too. Remember, they wanted a general system that can learn any board game and they therefore did not make domain specific optimizations.
There are probably quite a few low hanging fruits if DeepMind wanted to create the strongest chess engine in the world (based on A0) and not a flexible system

[Milos]

Why even use 2-move testsuite?

In order to avoid that always the same first moves/openings are played.

That didn't seems to bother DeepMind, in their match all the first moves/openings are the same since A0 had practically 0 randomness. So actually playing SF against itself on many threads introduces much more randomness than in A0-SF match. I see no point of arbitrary starting positions when you are trying to see the effect from the match where only chess initial position was used.

Also 512MB hash on Opteron 6376 is much more favourable condition than 1GB on dual 32 core modern Xeon. More fair comparison would be with only 256MB hash vs. 16GB hash.

Who really knows?

It's rather trivial to calculate hash load factor based on nps. Nps of SF on Opteron 6376 is known (at least you should know it) as well as nps of SF from A0 match - 70Mnps probably from a bench. So relatively decent comparison can be made instead of your guesstimate of 512MB.

[Ozymandias]

Did AlphaZero have a book?

That's a good question, but to get an answer, we'd first have to agree on a definition for "opening book". That's not easy, looking at the chessprogramming entry, I see no proper definition, it just says that they're used to replace the engines' search routines, at the beginning of the game. The problem with this definition, is that it doesn't deal with new ways of doing that search, like AlphaZero's. If I had to answer, constricted by this definition, I'd have to say "no", because AlphaZero wasn't using a book per se. Such a short reply, would leave out of the debate, the fact that AlphaZero was using knowledge, previously acquired when training. SF also had indirect access to that knowledge, as it can be considered part of the hundreds of millions of games, behind most of its code (Stockfish Testing Framework). And yet, "indirect access" isn't what AlphaZero was allowed; by design, this program can't be truly run, without something that's even better than an opening book.

[Jouni]

I did some single core test 1 MB vs. 64 MB. HERT500.PGN book. With fixed time=1 sec 1 MB was better than bigger? With 60s + 0,6s it changed totally:

Code: Select all

                   
1   SF 64MB   +42  +33/=158/-9 56.00%  112.0/200
2   SF 1MB    -42  +9/=158/-33 44.00%   88.0/200

Obviously fixed time per move DESTROYS SF SEARCH BADLY!

[Vinvin]

Did AlphaZero have a book?

Yes, the learning data can compare to a book.
It's a memory filled good score for lines that give good results (and bad score for lines that gives bad results). There's a small difference because opening lines are not hardcoded (= not a numerical value for each line) but store in a neural network. But with a lot of learning NN become a book with only good moves get only good eval.
This can be compare with old book learning. Example for HIARCS here : http://www.hiarcs.com/pc_uci_options.htm

Book Learning (ON)
This setting allows HIARCS to use its experiences with the current book to make decisions about which moves to make from the book. HIARCS has clever book learning so please use it! The default is ON and clearly best.

When the engine wins with a line, the score increase for this line.