I can't believe that so many people don't get it!

[Rebel]

Seems to indicate there were multiple matches.

Yes, there were. They periodically took the Alpha Zero that was training, and let it play a match to measure its strength. But the games from these evaluation matches were not used for training.

You sound like the PR-man from Deepmind

It's an argument from silence, you can not know.

[corres]

[quote="hgm"]

I don't think there is anyone that doesn't believe that learning Stockfish' weaknesses will improve your results against Stockfish. The more interesting question is whether after training against Stockfish Romi would also perform better against other engines (with no further training). Or whether playing games only against itself would improve its results againsr Stockfish.

[/quote]

It is a legitimate question.
Also it is a legitimate question weather Romi would achieve against Stokfish such like results during more faster and more slower playing time.
The system of monkey see monkey do guarantee a rather restricted learning. As every other common learning system this also causes the slowing of search process and this makes weaker Romi against an engine witch Romi do not know.
Moreover Romi and any other common chess engine using centi pawn for evaluation of positions can not make distinction between positions having only small, but important difference in winning chance. This is the main difference between Romi and AlphaZero beyond the self learning.
But a Romi type Chess engine with learning capability has some important advantage against AlphaZero: We can run it on a common PC and we do not need such a giant and expensive hardware what AlphaZero claims.

[hgm]

You sound like the PR-man from Deepmind

It's an argument from silence, you can not know.

Actually, anyone can know by reading the paper.

[Michael Sherwin]

After 42 games in the first 100 games Romi only had 1 draw. After 42 games of the second 100 Romi has gotten 10 draws. SF 8 1 thread is rated 3422 on the 40/4 index. RomiChess is rated a whopping 2423 elo. So after 142 games of training on the original position Romi in the now last 45 games and 11 draws has performed at 3422 - 330 = 3092 elo. Here are the openings visited.

c15, c10, c49, c68, c66, a55, b58, c66, c48, c66, c66, c66, c66, c66, c66, c68, c65, c61, c65, c66, c66, a56, c66, c65, c66, c66, b51, c66, c66, b51, c66, c66, c65, c65, c65, c68, c65, c68, c66, c65, b51, c66, c66, c66, c65, b09, c41, c41, c66, c66, c66, c66, c63, c66, c66, c02, c66, c41, c46, e87, c66, c45, c26, c66, c65, c41, c66, c65, c65, c66, c84, c65, c41, c65, c65, c84, c41, c66, c66, c66, c41, c41, c66, a56, b51, c66, c41, c66, c66, c41, c66, c65, c65, c66, c66, c65, c66, c66, c66, c66, c66, c66, c15, c65, b51, c65, c66, c68, c66, c65, c66, b52, c65, c65, c65, b31, c84, c66, c65, c66, c66, c66, b51, c66, b51, c66, a43, c68, b44, c66, c66, b36, c65, c66, c65, c66, c66, c66, c66, c66, c65, c41, c66, c66, c61, c66, a44

(Draws ...)
c66 = Closed Berlin . . . . . . . . . .
c65 = Berlin, Anderson
c15 = Winawer, Alekhine .
c10 = French, Rubinstein
c49 = Four knights, Nimzovitch
c68 = Spanish exchange
a55 = Old Indian
b58 = Sicilian, Boleslavsky
c48 = Spanish, Classical
c61 = Spanish, Birds
a56 = Benoni, Czech
b51 = Sicilian, Bb5+ Nc6
b09 = Pirc, Austrian
c41 = Philidor, Berger Variation .
c63 = Spanish, Schliemann
c02 = French, Advance
c46 = Three Knights, Schlechter Variation
e87 = King's Indian, Samisch
c45 = Scotch, Tartakower
c26 = Vienna
c84 = Spanish, Closed Center Attack .
b52 = Sicilian, Bb5+ Bd7
b31 = Sicilian, Rossolimo 3 ... g6
a43 = Old Benoni, Schmidt
b44 = Sicilian, Taimanov
b36 = Maroczy Bind
a44 = Old Benoni, Czech

Now 157 games and 16 draws for 3422 - 315 = 3107 elo performance.

Is anyone that did not believe changing their minds yet?

Can the detractors see what 44 million games of training would do for RomiChess?

The second 100 game match is finished. Romi got 31 draws and no wins. But at only 200 games training that is to be expected against SF as SF is very hard to beat. I will add the additional eco codes at the bottom of the original list.

(Draws ...)
c66 = Closed Berlin . . . . . . . . . . . . . . . . . . . . . . . . . .
c65 = Berlin, Anderson . . .
c15 = Winawer, Alekhine .
c10 = French, Rubinstein
c49 = Four knights, Nimzovitch
c68 = Spanish exchange
a55 = Old Indian
b58 = Sicilian, Boleslavsky
c48 = Spanish, Classical
c61 = Spanish, Birds
a56 = Benoni, Czech
b51 = Sicilian, Bb5+ Nc6
b09 = Pirc, Austrian
c41 = Philidor, Berger Variation .
c63 = Spanish, Schliemann
c02 = French, Advance
c46 = Three Knights, Schlechter Variation
e87 = King's Indian, Samisch
c45 = Scotch, Tartakower
c26 = Vienna
c84 = Spanish, Closed Center Attack .
b52 = Sicilian, Bb5+ Bd7
b31 = Sicilian, Rossolimo 3 ... g6
a43 = Old Benoni, Schmidt
b44 = Sicilian, Taimanov
b36 = Maroczy Bind
a44 = Old Benoni, Czech

c00 = French, KIA reversed
a48 = Neo Kings Indian, London System
c60 = Spanish Cozio
a46 = Indian, London
a47 = Neo King's Indian
a41 = Neo Old Indian

d32 = QGD, Tarrasch
b01 = Scandinavian, Portuguese
d02 = QP, London
a28 = Four Knights, Nanarokov
d40 = Semi Tarrasch
a03 = Bird Opening, Lasker
d04 = Colle
c22 = Center Game
c42 = Russian Game
a45 = Indian
a46 = Torre Attack
d05 = Colle
a47 = Neo Queens Indian
c44 = Inverted Hanham
c28 = Vienna, Bronstein Gambit
c24 = Bishop's Opening
a08 = Reti Opening
c50 = Giuoco Pianissimo

Three draws in the first 100 games.
Twenty eight draws in the second 100 games.

Elo performance in the second 100 games = 3422 - 295 = 3127 elo. So far 33 different different eco codes and numerous sub variations within the most played eco codes. Romi has now started drawing in the c65 eco code.

From 3 draws per 100 games to 28 draws per 100 games is huge against SF which changes its play and therefore is not deterministic. Since 33 elo codes were played and there was variance by both SF and Romi in the most played lines it can be understood that Romi is benefitting by playing stronger moves and better positions even though SF varies its play!

Next 100 games starting now.

Three hundred games finished. Romi got 42 draws for a performance of 3422 - 230 = 3192 elo. Editing eco usage above. Eighteen more eco codes introduced. What is happening here is that Romi does not like draws even against superior engines (or people) and is fishing around (pun intended) for other waters. Romi lost everyone of those games and yet managed to score more points overall. Romi maybe headed for a local maxima and may suffer a setback. That may not be a bad thing though as Romi is getting more experience in more lines. Time will tell. Only 43,999,700 training games left to go, lol, just kidding. But this does bring up a valid point. In 44 million games of training Romi vs Romi would investigate more of the opening space than Romi vs SF. So my original training of Romi vs Romi vs SF would give more quantity and more quality of training. Next 100 games started.

[hgm]

The point is that you are trying to 'prove' here what everybody already believes. So no matter how long you will continue it, and how successful it will be in the end, it will have zero impact, and convinve no one.

It has been known since the dawn of humanity that a predictable opponent can be easily exploited. How exactly you achieve the required prediction accuracy (reinorcement learning, systematic search) doesn't matter very much; it might take somewhat shorter or longer, but at some point you will know everything the opponent can do, and have a perfect counter strategy for all of it. We already did that in the seventees.

The only interesting thing is how much you benefit from this in general, i.e. against opponents you did not train against.

[Michael Sherwin]

The point is that you are trying to 'prove' here what everybody already believes. So no matter how long you will continue it, and how successful it will be in the end, it will have zero impact, and convinve no one.

How exactly you achieve the required prediction accuracy (reinorcement learning, systematic search) doesn't matter very much; it might take somewhat shorter or longer, but at some point you will know everything the opponent can do, and have a perfect counter strategy for all of it. We already did that in the seventees.

The only interesting thing is how much you benefit from this in general, i.e. against opponents you did not train against.

"The only interesting thing is how much you benefit from this in general, i.e. against opponents you did not train against."

That has been demonstrated with enough test results to extrapolate an answer. And the answer is that it is a big gain.

"It has been known since the dawn of humanity that a predictable opponent can be easily exploited."

Not fully applicable because the learning is not opponent specific. And it helps the engine play into lines where it does better. But I have said this a thousand times in one form or another without being heard. And you won't stop until I'm am ground down to powder. Well I am ground down to powder so you win. Everybody, what I have done in RomiChess really sucks and you should forget all about it. I resign!

[hgm]

I am not grinding anything. I was just asking a question. Fact is that you are reporting improvements of results against Stockfish from training against Stockfish. Which is not going to score any points with the people you refer to as 'detractors'. I just point out that to strengthen your case with additional data, you'd better show them the thing that might convince them. 'Big gain' is rather vague, and you don't give any link to the tests that show it.

[duncan]

This is not what the paper describes.

Indeed. But we are not dealing with academics. We are dealing with a commercial company with a bad reputation. They have their mouth full of ethics but their actions are criminal, like the copying of books, like buying youtube while knowing its massive illegal copyrighted material and only God knows what they do with the data we publicly trust to the internet and thus to them, my name might as well be colored red after this post

Ranting aside, let's talk about what the paper doesn't reveal, the start positions of the 100 games. Those 50 start positions certainly can be learned as Mike described.

what elo would lczero have to reach so that you would think that alphazero is genuine ?

[David Xu]

This thread aged well.

[nabildanial]

This thread aged well.

It certainly is.