2 open projects

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.

stockfish vs lczero

Poll ended at Thu May 03, 2018 11:20 pm

lczero will attract more ppl, leaving the stockfish project to die
2
5%
lczero will saturate so stockfish will continue to prevail
3
8%
both projects will continue to be supported without to hurt each other
33
87%
 
Total votes: 38

MonteCarlo
Posts: 62
Joined: Sun Dec 25, 2016 3:59 pm

Re: 2 open projects

Post by MonteCarlo » Fri Apr 27, 2018 6:28 pm

Milos wrote: A0 after 10 millions training games already (allegedly because there is no proof beside that Google advertising leaflet) surpassed SF8.
I'm all for keeping a grounded view of expectations, but let's at least get the math right.

44 million games in 9 hours = ~4.888 million games/hour

From the paper:
In chess, AlphaZero outperformed Stockfish after just 4 hours (300k steps);...
4 hours * 4.888 million games/hour > 10 million games

Anyway, I wouldn't at all expect Leela to be where A0 was after the same number of games.

Silver said in that keynote talk that they did multiple runs, and I'm pretty sure they didn't publish the results of a run where they had the sorts of bugs the Leela project has had so far :)

There's very little point idly speculating about what can and can't be achieved by the project.

Time will tell.

Gian-Carlo Pascutto
Posts: 1189
Joined: Sat Dec 13, 2008 6:00 pm
Contact:

Re: 2 open projects

Post by Gian-Carlo Pascutto » Fri Apr 27, 2018 6:46 pm

MonteCarlo wrote: Anyway, I wouldn't at all expect Leela to be where A0 was after the same number of games.

Silver said in that keynote talk that they did multiple runs, and I'm pretty sure they didn't publish the results of a run where they had the sorts of bugs the Leela project has had so far :)

There's very little point idly speculating about what can and can't be achieved by the project.

Time will tell.
Well said (all of it).

One of the big open questions for me is: AZ0 ended up 100 Elo above SF8. Did DeepMind just pull the plug when they got a good enough result (likely), or did they simply not get it higher (not impossible either).

If it is the latter, I would still expect chess specific tweaks to be able to take things past SF9. But that is going to take much more experimentation than re-implementing a paper.

frankp
Posts: 216
Joined: Sun Mar 12, 2006 2:11 pm

Re: 2 open projects

Post by frankp » Fri Apr 27, 2018 9:10 pm

I thought the deepmind project was about demonstrating the generality of the application of their basic approach - in a closed system.
Beat AlphaGO - check
Beat SF - check
Beat best shogi program (?) - check

Move on.

Although as you say, I doubt it all when swimmingly on the first attempt. But who knows.

Not sure it was about finding the best approach to chess, once they were 'better' than the current best. Although it would be interesting to know how hard they had to struggle to get to this level.

jp
Posts: 839
Joined: Mon Apr 23, 2018 5:54 am

Re: 2 open projects

Post by jp » Fri Apr 27, 2018 9:33 pm

MonteCarlo wrote:...
Silver said in that keynote talk that they did multiple runs, and I'm pretty sure they didn't publish the results of a run where they had the sorts of bugs the Leela project has had so far :)
...
Yes, it's safe to assume the 9hrs & 44m games started after they were happy it was bug-free. It would not have suited the PR to do otherwise.

jp
Posts: 839
Joined: Mon Apr 23, 2018 5:54 am

Re: 2 open projects

Post by jp » Fri Apr 27, 2018 9:38 pm

Gian-Carlo Pascutto wrote:...
... Did DeepMind just pull the plug when they got a good enough result (likely), or did they simply not get it higher (not impossible either).

If it is the latter, ...
The plot in DeepMind's paper show that it had stopped improving after ~4hrs for chess. For Go, you could see its performance still climbing at the end.
So for chess, they simply could not get it higher.

Geonerd
Posts: 66
Joined: Fri Mar 10, 2017 12:44 am

Re: 2 open projects

Post by Geonerd » Sat Apr 28, 2018 1:56 am

Option 4:
Stockfish and other 'traditional' programs embrace NN evaluation techniques.
Combining the two will result in a program stronger than either.

Nay Lin Tun
Posts: 529
Joined: Mon Jan 16, 2012 5:34 am

Re: 2 open projects

Post by Nay Lin Tun » Sat Apr 28, 2018 5:45 am

NN or subjective positional guess wont be the mainstay of chess in the future ? for 10 years. Brute force ( objective proof) by minimax pruning would be the solution.
Leela has weights ( brain/ knowledge) advantage.Then SF should have similar Cerebellum and TB.
The future best chess program would be SF with
1. Improved cerebellum+TB
2. Improved pruning patches ( idea from Leela, may be)

syzygy
Posts: 4458
Joined: Tue Feb 28, 2012 10:56 pm

Re: 2 open projects

Post by syzygy » Sat Apr 28, 2018 9:47 am

Geonerd wrote:Option 4:
Stockfish and other 'traditional' programs embrace NN evaluation techniques.
Combining the two will result in a program stronger than either.
Option 5:
A neural network takes over SF development.

mirek
Posts: 42
Joined: Sat Mar 24, 2018 3:18 pm

Re: 2 open projects

Post by mirek » Sat Apr 28, 2018 10:20 am

jp wrote:
MonteCarlo wrote:...
Silver said in that keynote talk that they did multiple runs, and I'm pretty sure they didn't publish the results of a run where they had the sorts of bugs the Leela project has had so far :)
...
Yes, it's safe to assume the 9hrs & 44m games started after they were happy it was bug-free. It would not have suited the PR to do otherwise.
Also they were using big net from the beginning so if computational time is not the concern, big net most likely converges faster per number of games played.

mirek
Posts: 42
Joined: Sat Mar 24, 2018 3:18 pm

Re: 2 open projects

Post by mirek » Sat Apr 28, 2018 10:39 am

Gian-Carlo Pascutto wrote: One of the big open questions for me is: AZ0 ended up 100 Elo above SF8. Did DeepMind just pull the plug when they got a good enough result (likely), or did they simply not get it higher (not impossible either).
For me as well. I wonder by how much can the NN be increased before the the decrease in nps will cause overall negative elo gain. Even if one could run every NN in constant time (independent of NN size) there is probably diminishing elo gain by doubling the NN size, so there will be some finite limit even for infinite sum of NN doublings.

I am also wondering if there are any other NN architectures that are being considered that could go beyond what just scaling of current architecture can achieve.

Post Reply