This notion that neural networks are actually opening books in disguise keeps rearing its silly head. I decided to test this opening book hypothesis with one of my favorite nets. With 3 seconds per move on a 2070, I investigated a few opening lines, taking the top choice, except in the case of the first or second move as noted. So, what openings does this net like?
It likes e4 and d4. On e4, it likes e6 and c5. On e6, it likes d4. Looking a few moves deeper, we get 1. e4 e6 2. d4 d5 3. Nd2 c5 4. Ngf3 cxd4 5. Nxd4 Nf6 -- pretty main line Tarrash French. If we go down the Sicilian route, we get 1. e4 c5 2. Nf3 d6 3. Bb5+ Bd7 4. Bxd7+ Qxd7 5. O-O Nc6 -- main line Rossolimo. And on d4, 1. d4 Nf6 2. Nf3 e6 3. c4 d5 4. Nc3 Be7 5. Bg5 O-O is pretty main line QGD.
Good evidence that this net has memorized opening moves? Not so fast. This is Little Ender, an endgame net that has never seen opening positions. In fact, it has never seen a position with more than 18 pieces.
"Leela is an opening book" nonsense
Moderators: hgm, Rebel, chrisw
-
- Posts: 1631
- Joined: Tue Aug 21, 2018 7:52 pm
- Full name: Dietrich Kappe
"Leela is an opening book" nonsense
Fat Titz by Stockfish, the engine with the bodaciously big net. Remember: size matters. If you want to learn more about this engine just google for "Fat Titz".
-
- Posts: 6340
- Joined: Mon Mar 13, 2006 2:34 pm
- Location: Acworth, GA
Re: "Leela is an opening book" nonsense
Problem solved and case closed! Case dismissed due to lack of evidence.dkappe wrote: ↑Fri Dec 20, 2019 12:27 am This notion that neural networks are actually opening books in disguise keeps rearing its silly head. I decided to test this opening book hypothesis with one of my favorite nets. With 3 seconds per move on a 2070, I investigated a few opening lines, taking the top choice, except in the case of the first or second move as noted. So, what openings does this net like?
It likes e4 and d4. On e4, it likes e6 and c5. On e6, it likes d4. Looking a few moves deeper, we get 1. e4 e6 2. d4 d5 3. Nd2 c5 4. Ngf3 cxd4 5. Nxd4 Nf6 -- pretty main line Tarrash French. If we go down the Sicilian route, we get 1. e4 c5 2. Nf3 d6 3. Bb5+ Bd7 4. Bxd7+ Qxd7 5. O-O Nc6 -- main line Rossolimo. And on d4, 1. d4 Nf6 2. Nf3 e6 3. c4 d5 4. Nc3 Be7 5. Bg5 O-O is pretty main line QGD.
Good evidence that this net has memorized opening moves? Not so fast. This is Little Ender, an endgame net that has never seen opening positions. In fact, it has never seen a position with more than 18 pieces.
"Good decisions come from experience, and experience comes from bad decisions."
__________________________________________________________________
Ted Summers
__________________________________________________________________
Ted Summers
-
- Posts: 546
- Joined: Sat Aug 17, 2013 12:36 am
Re: "Leela is an opening book" nonsense
Nice strawman bro?
-
- Posts: 1631
- Joined: Tue Aug 21, 2018 7:52 pm
- Full name: Dietrich Kappe
Re: "Leela is an opening book" nonsense
Fat Titz by Stockfish, the engine with the bodaciously big net. Remember: size matters. If you want to learn more about this engine just google for "Fat Titz".
-
- Posts: 546
- Joined: Sat Aug 17, 2013 12:36 am
Re: "Leela is an opening book" nonsense
Why do you have to wish?
-
- Posts: 70
- Joined: Thu Sep 19, 2019 5:55 am
- Full name: Andy!
Re: "Leela is an opening book" nonsense
--aha. EXCELLENT IDEA!
1: Play {SF, Komodo, Ethereal, etc} vs Lc0 chess960 tournament.
2: Also play {SF, K, E etc} vs Lc0 ordinary-chess tourney as
"experimental control."
(But not just 10 games in each tourney; you will need far more games than that.)
CCC could run this test. Might be fun (although I suspect kind of pointless).
1: Play {SF, Komodo, Ethereal, etc} vs Lc0 chess960 tournament.
2: Also play {SF, K, E etc} vs Lc0 ordinary-chess tourney as
"experimental control."
(But not just 10 games in each tourney; you will need far more games than that.)
CCC could run this test. Might be fun (although I suspect kind of pointless).
-
- Posts: 4889
- Joined: Thu Mar 09, 2006 6:34 am
- Location: Pen Argyl, Pennsylvania
Re: "Leela is an opening book" nonsense
I don't think anybody who is knowledgeable thinks that now, but yea at first glance it might look like that to someone who doesn't understand. One big difference , as you illustrate quite well in your post , an opening book no longer works once it gets out of book, while a NN engine can still make an pretty decent move even if it has never seen the position before. Your post is akin to proving the earth is round, there is no dispute unless somebody just wants to be in self denial - which we cannot control. Thanks for posting.dkappe wrote: ↑Fri Dec 20, 2019 12:27 am This notion that neural networks are actually opening books in disguise keeps rearing its silly head. I decided to test this opening book hypothesis with one of my favorite nets. With 3 seconds per move on a 2070, I investigated a few opening lines, taking the top choice, except in the case of the first or second move as noted. So, what openings does this net like?
It likes e4 and d4. On e4, it likes e6 and c5. On e6, it likes d4. Looking a few moves deeper, we get 1. e4 e6 2. d4 d5 3. Nd2 c5 4. Ngf3 cxd4 5. Nxd4 Nf6 -- pretty main line Tarrash French. If we go down the Sicilian route, we get 1. e4 c5 2. Nf3 d6 3. Bb5+ Bd7 4. Bxd7+ Qxd7 5. O-O Nc6 -- main line Rossolimo. And on d4, 1. d4 Nf6 2. Nf3 e6 3. c4 d5 4. Nc3 Be7 5. Bg5 O-O is pretty main line QGD.
Good evidence that this net has memorized opening moves? Not so fast. This is Little Ender, an endgame net that has never seen opening positions. In fact, it has never seen a position with more than 18 pieces.
-
- Posts: 6995
- Joined: Thu Aug 18, 2011 12:04 pm
Re: "Leela is an opening book" nonsense
You are right, I tried, see subject Amazing Leela - http://rebel13.nl/home/whatever.html
Encouraged by the first excellent results at D=1 I tried D=2 | D=3 | D=4 | D=5 and instead of getting somewhat stronger at each deeper depth strangely enough the opposite happened, the deeper depth the weaker Leela performed. Ra ra...
90% of coding is debugging, the other 10% is writing bugs.
-
- Posts: 3657
- Joined: Wed Nov 18, 2015 11:41 am
- Location: hungary
Re: "Leela is an opening book" nonsense
I do not know your net contains policy head or not.dkappe wrote: ↑Fri Dec 20, 2019 12:27 am This notion that neural networks are actually opening books in disguise keeps rearing its silly head. I decided to test this opening book hypothesis with one of my favorite nets. With 3 seconds per move on a 2070, I investigated a few opening lines, taking the top choice, except in the case of the first or second move as noted. So, what openings does this net like?
It likes e4 and d4. On e4, it likes e6 and c5. On e6, it likes d4. Looking a few moves deeper, we get 1. e4 e6 2. d4 d5 3. Nd2 c5 4. Ngf3 cxd4 5. Nxd4 Nf6 -- pretty main line Tarrash French. If we go down the Sicilian route, we get 1. e4 c5 2. Nf3 d6 3. Bb5+ Bd7 4. Bxd7+ Qxd7 5. O-O Nc6 -- main line Rossolimo. And on d4, 1. d4 Nf6 2. Nf3 e6 3. c4 d5 4. Nc3 Be7 5. Bg5 O-O is pretty main line QGD.
Good evidence that this net has memorized opening moves? Not so fast. This is Little Ender, an endgame net that has never seen opening positions. In fact, it has never seen a position with more than 18 pieces.
But it has or not your experience proves nothing but search of Leela is rather effective even if it uses randomly generated search moves.
So if you want to prove "There is no any such a behavior of NN what we can use such a mode than how we use an opening book" (as I stated "NN is a KIND of opening book") you have to gain information without any search. Using Leela we need at least 1 node/second speed and Leela use not only her Value Head but her Policy Head too. Because the meaning of Policy Head is not unambiguous for everybody, I should state under Policy Head I mean "Those part of NN what NN engines use helping their search and it do not contain value for the searched position - or if it contains value the search does not take it into consideration." So you use Leela with 1 node/second speed AND Leela does not use Policy Head the gotten results prove what Leela knows about openings.
I think the result will show such "book" in what there are holes and as the distance from the start position is larger as the number of holes will be higher.