Stockfish Handicap Test

[Milos]

Stockfish can't use the Cerebellum book either.

That is a question of semantics. Ofc you can also insist that Brainfish is not SF, which would be the same kind of logic as insisting that it can't use Syzygy since you are not an original SF author...

[Ovyron]

Your insisting on irrelevant and off-topic "arguments" is really silly. Careful my friend, ppl might think you are just trolling.

I just copied and pasted a message you previously sent me and edited it slightly

It was about time control, where you told me they could easily implement it.

So, cross-posting your messages to yourself...

You really think that Google ppl are that stupid not to know how to implement a trivial time-manager with just one category of moves for different TCs? Something that would require 10mins top to implement if you ever looked at TM code of any modern engine?
And you believe actually Google ppl didn't have time to implement it, and have chosen 1min/move TC just by chance???
Gee, I really thought you were a smarter person. What you believe seems equally convincing as believing in Santa Clause...

If they are able or not to implement it is irrelevant. A0 doesn't have that option because it is explicitly said in the paper that A0 was tested at one minute per move, so they didn't need to implement a time-manager.
Or you are suggesting they are simply lying?
Your insisting on irrelevant and off-topic "arguments" is really silly. Careful my friend, ppl might think you are just trolling.

Just an experiment, I do think a Milosbot can be programmed to argue with people, and it seems you'd be stuck arguing and counter-arguing with it/yourself

[syzygy]

Stockfish can't use the Cerebellum book either.

That is a question of semantics. Ofc you can also insist that Brainfish is not SF, which would be the same kind of logic as insisting that it can't use Syzygy since you are not an original SF author...

Stockfish obviously can use TBs.

Stockfish obviously cannot use the Cerebellum book. Unless you let the GUI pick the book moves, in which case...

But whatever. If you want to make yourself believe that AlphaZero should be compared to Stockfish + Cerebellum and not to Stockfish, then just let yourself convince you.

[Milos]

Your insisting on irrelevant and off-topic "arguments" is really silly. Careful my friend, ppl might think you are just trolling.

I just copied and pasted a message you previously sent me and edited it slightly

It was about time control, where you told me they could easily implement it.

Time control is not a domain knowledge while opening books and EGBTs are. Your "logic" is as usually flawed.

[Ovyron]

Time control is not a domain knowledge while opening books and EGBTs are. Your "logic" is as usually flawed.

Yeah? Show the the code that Stockfish has for books and EGTBs, its "domain knowledge".

It has none.

Those are external parts the engine can access, not part of the engine.

Alpha Zero was tested against Stockfish, you can't claim it was crippled or something by not using a thing external to it (Stockfish + X + Y + Z), because then it's not Alpha Zero Vs. Stockfish (it's Alpha Zero Vs. Stockfish + X + Y + Z), and then why not Alpha Zero + X + Y + Z Vs. Stockfish + X + Y + Z? Because A0 does fine without such things?

In the case of Cerebellum, specifically, the point is that it may be a very strong opening book, but it sucks for testing, and anybody giving it to both sides is a fool, so why would you give it to just one side? It makes no sense unless you are testing books, and not engines, and then it should be Engine A Vs. Engine A+Cerebellum, but when you care about Engine's A strength, it's not a book you want to use. What you want is variety. You don't want to test a single position Cerebellum outputs every single time, and A0+No Book Vs. Cerebellum isn't a test against Stockfish.

They tested against the state of the art of how engines play the openings without outside interference, the best efforts by Shredder and ProDeo has engines playing it well by accessing outside source, but there's no domain knowledge about it inside the engine, just like with A0, it only has code to use an opening book outside of it, and whatever domain knowledge it has inside was tested against Alpha Zero.

I'll go again and blame Rating Lists for this, programmers don't care about making their engine play the opening well because the rating lists will skip this phase with chosen openings anyway, and users can plug in an opening book and play stronger anyway.

Alpha Zero is the best thing I have ever seen play openings well without opening books, by far.

[Dann Corbit]

It's easy to make a polyglot version of it.
I have done it, and it only takes a couple minutes.

Of course, if you are trying to find out how strong an engine is at processing opening positions, the last thing you would want to do is use an opening book.

Instead of finding out how good the engine's decision making is, you discover the quality of a database.

Which is all well and good, but it literally has nothing to do with the strength of the engine (though it clearly has something to do with the chess playing system's strength).

[Ovyron]

It's easy to make a polyglot version of it.
I have done it, and it only takes a couple minutes.

Unfortunately that's crippled Cerebellum as it'll repeat positions and allow 3-fold repetition while the original would have avoided it.

[Dann Corbit]

It's easy to make a polyglot version of it.
I have done it, and it only takes a couple minutes.

Unfortunately that's crippled Cerebellum as it'll repeat positions and allow 3-fold repetition while the original would have avoided it.

Maybe, but the engine should track that anyway.
Plus, it makes no real difference in playing strength.
See Sedat's book tournaments.

Which all has nothing to do with the point:
An opening book is nothing more than a database.
It does not show the engine strength.
It shows the strength of collected data.

If you insist on using it with SF against the neural approach, what you are doing is not showing the strength of the standard approach, you are merely hiding some of its weaknesses.

True, the Cerebellum approach adds a great deal of Elo.
But this Elo is just a DB lookup. You will admit, I hope, that there is no calculation going on whatsoever. And any engine can strap it on (though this requires code changes).
SF, itself, cannot strap it on without modification (the stock SF can NOT use the special book, that requires the modified version. Stock SF CAN use the polyglot version but that is also neither here nor there).

[Ovyron]

An opening book is nothing more than a database.
It does not show the engine strength.
It shows the strength of collected data.

If you insist on using it with SF against the neural approach, what you are doing is not showing the strength of the standard approach, you are merely hiding some of its weaknesses.

True, the Cerebellum approach adds a great deal of Elo.
But this Elo is just a DB lookup. You will admit, I hope, that there is no calculation going on whatsoever. And any engine can strap it on (though this requires code changes).

Yes, we agree on that.

I don't like how DeepMind relied on MultiProcessor randomness to play out their 100 games, though. For instance, MultiProcessor isn't as efficient as single processor with higher Hertz, so it could theoretically have been better to play 64 games at the same time, give each Stockfish a thread, and allow them 64 minutes per move (do I have my math right? It sounds like a lot - this is supposed to allow the same number of games in the same time the 64 threads Stockfish would play 64 games sequentially at one minute per move...)

But this would have lost MultiProcessor randomness, so all those games would have looked the same with SF and A0 being deterministic. How do you solve this problem? Whatever method solving this would have been a good method to use in the actual match in case SF without book played very similar openings and we only get to see a performance difference in a very small subset of the actual game.

[Milos]

Alpha Zero is the best thing I have ever seen play openings well without opening books, by far.

Openings .
You know that is plural. I know English is not your mother tongue, but calling exactly 2 distinctive openings that you could see in those 10 games "the best thing I have ever seen play openings" is just hilarious.