Lc0 saturation.
Moderators: hgm, Rebel, chrisw
-
- Posts: 12542
- Joined: Wed Mar 08, 2006 8:57 pm
- Location: Redmond, WA USA
Re: Lc0 saturation.
What was the size of the alpha zero net?
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
-
- Posts: 1470
- Joined: Mon Apr 23, 2018 7:54 am
Re: Lc0 saturation.
20x256.
(So Lc0 10xxx nets are 20x256 too. Older Lc0 9xxx & 4xxx nets are 6x64.)
-
- Posts: 1797
- Joined: Thu Sep 18, 2008 10:24 pm
-
- Posts: 143
- Joined: Wed Jan 17, 2018 1:26 pm
Re: Lc0 saturation.
Why not? the 20x256 net did the job they wanted from it, which was to beat a somewhat recent Stockfish under test conditions. Why should they have expended much more resources to train an even bigger and stronger net? Google can find better uses for those TPUs, they're not that invested in board games except as a means to an end (i.e. commercially useful AI applications). They published their article and made their point, and it is now up to the Leela dev team and other chess programmers to take this as far as possible.
-
- Posts: 1797
- Joined: Thu Sep 18, 2008 10:24 pm
Re: Lc0 saturation.
The alternative, less happy, theory is that as the net gets bigger the gains are cancelled out by lower nps.
-
- Posts: 143
- Joined: Wed Jan 17, 2018 1:26 pm
Re: Lc0 saturation.
Test10 with its 256x20 architecture represented a huge improvement over the 192x15 main net. Deepmind reported that for Alphago Zero, 256x40 ended up far stronger than 256x20. So far, all available data indicates that larger net sizes work, they just increase the initial investment of training such a net by reinforcement learning.
-
- Posts: 12542
- Joined: Wed Mar 08, 2006 8:57 pm
- Location: Redmond, WA USA
Re: Lc0 saturation.
Why not go directly for 256 x 40 then?jkiliani wrote: ↑Mon Aug 20, 2018 11:45 pmTest10 with its 256x20 architecture represented a huge improvement over the 192x15 main net. Deepmind reported that for Alphago Zero, 256x40 ended up far stronger than 256x20. So far, all available data indicates that larger net sizes work, they just increase the initial investment of training such a net by reinforcement learning.
It seems that each new stage throws away the old stuff.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
-
- Posts: 143
- Joined: Wed Jan 17, 2018 1:26 pm
Re: Lc0 saturation.
That will happen at some point, but the prevailing sentiment right now is to get a bit more sure about the self-play and training parameters before "going all in". In addition, just today a bug was discovered that prevents all recent (including test10) networks from using the 50-move rule input plane. Once we are confident that everything works fine, I think a "big net" run will be in the cards.Dann Corbit wrote: ↑Mon Aug 20, 2018 11:50 pmWhy not go directly for 256 x 40 then?jkiliani wrote: ↑Mon Aug 20, 2018 11:45 pmTest10 with its 256x20 architecture represented a huge improvement over the 192x15 main net. Deepmind reported that for Alphago Zero, 256x40 ended up far stronger than 256x20. So far, all available data indicates that larger net sizes work, they just increase the initial investment of training such a net by reinforcement learning.
It seems that each new stage throws away the old stuff.
-
- Posts: 12542
- Joined: Wed Mar 08, 2006 8:57 pm
- Location: Redmond, WA USA
Re: Lc0 saturation.
Re: "I think a "big net" run will be in the cards."
Pun intentional?
Pun intentional?
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
-
- Posts: 4190
- Joined: Wed Nov 25, 2009 1:47 am
Re: Lc0 saturation.
They didn't publish the article because publishing means that it has been peer-reviewed and accepted by a journal or a conference.
That crappy PR stunt preprint was obviously never accepted for publication anywhere. It was just uploaded to arxiv.org.