LCZero: Progress and Scaling. Relation to CCRL Elo

[jp]

Nah, this is self-play with fixed number of playouts. But the nets became slower. At fixed time against standard engine, there was little progress lately, even with the introduction of the larger net, and only on GPU, not on CPU (but CPU results will be anyway irrelevant). On GPU, with the introduction of ID227 larger 15x192 net, maybe there was some improvement, 20-30 Elo points or so, no more, because the nets were much slower. But since larger nets, from ID227 to ID237, in their 500,000 games, I cannot measure sensible improvement at fixed time against a standard engine, probably less than 15 Elo points.

What does the graph of net size vs. number of playouts (fixed time) look like?
(I know there've only been 3 net sizes so far.)

[Daniel Mehrmann]

I think 2950 CCRL elo is a way to high.

I tested Leela id 234 versus Naum 4.6, Deep Shredder 11 and ProDeo. After my results 10+10/game, she has a rating of around 2850 CCRL elo.

Regards
Daniel

Depends on your graphics card

Exactly and because of this, you must agree with my statement on top of course.

The point is, that no matter which is the strongest hardware for Leela, and maybe some super duper guys beat a strong(er) Engine on fast, but not biggest hardware, the "normal" middle class consumer hardware counts and nothing else!

Flighing back in history it would be pretty funny if somebody tell you, okay, my SF version 1.6.1(!!) with 43 cores beats the strongest current engine Rybka 3. Insane yes, but everybody would laught, because on "normal" hardware SF 1.6.1 is 200 Elo behind Rybka 3....

However, i think my Leela hardware with GeForce 750ti is, at the moment, still middle class hardware and these results counting.


Regards
Daniel

[Albert Silver]

I think 2950 CCRL elo is a way to high.

I tested Leela id 234 versus Naum 4.6, Deep Shredder 11 and ProDeo. After my results 10+10/game, she has a rating of around 2850 CCRL elo.

Regards
Daniel

Depends on your graphics card

Exactly and because of this, you must agree with my statement on top of course.

The point is, that no matter which is the strongest hardware for Leela, and maybe some super duper guys beat a strong(er) Engine on fast, but not biggest hardware, the "normal" middle class consumer hardware counts and nothing else!

Flighing back in history it would be pretty funny if somebody tell you, okay, my SF version 1.6.1(!!) with 43 cores beats the strongest current engine Rybka 3. Insane yes, but everybody would laught, because on "normal" hardware SF 1.6.1 is 200 Elo behind Rybka 3....

However, i think my Leela hardware with GeForce 750ti is, at the moment, still middle class hardware and these results counting.


Regards
Daniel

I consider my GTX 1060 middle-class as well, albeit more recent than your 2014 card.

[Daniel Mehrmann]

I consider my GTX 1060 middle-class as well, albeit more recent than your 2014 card.

I don't think so. Please look around beyond gamer and chess world....

But if you only look on these areas, maybe you're right.

[Albert Silver]

I consider my GTX 1060 middle-class as well, albeit more recent than your 2014 card.

I don't think so. Please look around beyond gamer and chess world....

But if you only look on these areas, maybe you're right.

There is no question that gaming and cryptocurrency mining are what drive the third-party desktop GPU market, so there is no big point looking beyond it. Those who do not game, even casually, will hardly go beyond the bare minimum. The only point is for multi-display setups otherwise. Just because there is a larger market for lower-end cards doesn't change anything.

In any case, I am basing it on the actual manufacturers as they define their cards. When NVidia release this series of cards, much like previous ones, they abided to their numbering scheme: the xx60 was their middle-class card, the xx80 was their top-of-the-line, and the xx70 was somewhere in between. A while later they release cheaper models from xx30 to xx50, and add -ti versions. None of this is new, and dates all the way to the 260, 270 and 280s.

[Laskos]

I consider my GTX 1060 middle-class as well, albeit more recent than your 2014 card.

I don't think so. Please look around beyond gamer and chess world....

But if you only look on these areas, maybe you're right.

Well, there is Go world too, as an example, and generally, the GPU will soon be as important as the CPU, if not more so, in the most frontier areas. I will soon buy GTX 1060 GPU because it is not more expensive than my i7 CPU. There will be even weirder architectures soon to become important, so you maybe will have to adapt a bit.

[Laskos]

Nice graphs Kai.
Certainly the 'accuracy' graph on tensorboard is the highest it has ever been, perhaps worryingly so, which is reflected in your results.
I guess it is better to be better if slower, since hardware tends to get better.

Just wish the 'must beat everything else now or the project is failing' brigade, would take a step back. This really is different to the usual 'copy' stockfish and produce another AB searcher. Interesting to see how far it can go. Unfortunately some around the project seem equally infected and want to install egtb so leela does better (less worse) in TCEC, which from my perspective contaminates the zero approach.

Yes, I also like the more "purist" approach. It's not only "purist", it's different and funny to play and watch. Frankly, I am a bit fed-up with AB and all sorts of pruning and reductions to depth 55.

Maybe the skepticism was unwarranted, ID239 showed a pretty significant improvement in this plot (1000 games each datapoint):



Taking into account a 3-4% slowdown of the net, I guess that the improvement since ID227 is in the range of 25 Elo points, maybe even a bit more. Have yet to test at fixed time control.

[Laskos]

Just saw in Journal Science this news article
AI researchers allege machine learning is alchemy
http://www.sciencemag.org/news/2018/05/ ... ng-alchemy

[jp]

Just saw in Journal Science this news article
AI researchers allege machine learning is alchemy
http://www.sciencemag.org/news/2018/05/ ... ng-alchemy

In the article:
"AI's reproducibility problem, in which researchers can't replicate each other's results because of inconsistent experimental and publication practices."
linked to
http://science.sciencemag.org/content/359/6377/725
"The booming field of artificial intelligence (AI) is grappling with a replication crisis, much like the ones that have afflicted psychology, medicine, and other fields over the past decade. Just because algorithms are based on code doesn't mean experiments are easily replicated. Far from it. Unpublished codes and a sensitivity to training conditions have made it difficult for AI researchers to reproduce many key results. That is leading to a new conscientiousness about research methods and publication protocols."

[Laskos]

Just saw in Journal Science this news article
AI researchers allege machine learning is alchemy
http://www.sciencemag.org/news/2018/05/ ... ng-alchemy

In the article:
"AI's reproducibility problem, in which researchers can't replicate each other's results because of inconsistent experimental and publication practices."
linked to
http://science.sciencemag.org/content/359/6377/725
"The booming field of artificial intelligence (AI) is grappling with a replication crisis, much like the ones that have afflicted psychology, medicine, and other fields over the past decade. Just because algorithms are based on code doesn't mean experiments are easily replicated. Far from it. Unpublished codes and a sensitivity to training conditions have made it difficult for AI researchers to reproduce many key results. That is leading to a new conscientiousness about research methods and publication protocols."

Reproducibility at least is not a problem here. That Leela in A0 conditions is 3300 and A0 is 3600 CCRL Elo level is not a problem to me. They both beat not only me, who am after all a human, they both beat all humans in those conditions. The same in Go. Heck, knowing only the rules of Chess and Go? This is insane!