End of Era is there: SF is finally beaten!

[corres]

I think Lc0 is just giving us an idea of proof of concept in NN as was shown with A0 with Google. The potential is there, but to be honest Lc0 would not be even on my top 10 list of engines to use if I wanted to analyze a chess game. At least not a stand alone engine. It can miss one move tactics that really are very simple to see even for a human ... and after analyzing some of A0 games, I saw one move tactical misses as well. Lc0 also has a serious thing with the endgame that some people are politely calling "trolling" ... but I think that this is just poorly written software or poorly written search code. It just misses way too many wins in games it is completely winning and endlessly prolongs games and sometimes even misses wins in clear cut endgames. A good example would be in the amazing game that it won ... I mean promote to a rook???? ... I mean come on that is just stupid! No doubt all this will be fixed, especially if some of the talented chess coders on the SF team or on Houdini team or on Komodo team get involved. I think the proof of concept of NN has been proven beyond a shadow of a doubt, and it is time to get everyone on the band wagon.

The issues of Leela are not caused by weak or bogus code - even if there are some bugs in Leela's code - these issues arises from the used technique.
As AB type engines have their own strong and weak side, NN type engines have also their strong and weak side.
It is pity but perfect chess engine does not exist.

[corres]

...

There is no any "spectacular" and there is no any "plan" on the side of Leela.
In reality the position after 19.-c4 and the good continuation is in the 32930net of Leela.
To prove this I used the weak GTX 1060 to find the 20.Bg6 move.
From net30100 to net32000 Leela want to move 20.Be4, 20.Bxc4, 20.0-0.
But using net32500 Leela finds 20.Bg6 immediately!
So anywhere between net32000 and net32500 Leela has analysed yet this position during self play
and 20.Bg6 is in the NN of Leela.
Obviously Stockfish also would be much more stronger if it could use a good book instead of analysing every each position.


I am sorry but I miss-wrote the number of moves.
The text with good move numbers is above.

[Leo]

How can one know whether the engines are playing on equal terms?
How can you compare CPU and GPU and state you've found a balance?

Nohow.
There are only subjective methods to make comparison.
Moreover AB engines are made basically for playing middle games and endgames.
An AB engines will be banished from a competition if it has a built-in opening book.
An NN type engines has definitively a built-in opening + middle game + endgame book.
AB engines have their own benefit and their own drawback and NN type engines have these too.
It is obvious the development of hardware and software gives more opportunity for enhancing
the chess power of the NN type engines than for an AB type engines.
But the problem of chess can not be solved by an NN type engine either.

Note
I think the performance of Stockfish against NN engines would be enhanced if developers of Stockfish would make test not only against the actual master of Stockfish but against the best NN engines and if they would modify in some measure the source of Stockfish according to the result of tests.
Until now to make these tests was problematic because the fast changing in the power of NN engines.
But to now the chess power of NN engines reach near a plateau so there is sense to make tests with NN type engines too.

Whoever figures out how to combine the Chess CPU power of a Stockfish like engine with the Chess GPU of an Lc0 like engine will get there first, SO many computers today have 24 , 36 or even 72 cores - take the best x moves from Lc0 and dedicate an AB search on those moves using one of those CPUs with A/B search engine like SF. Someone with some coding skills should be looking at that, Good luck!

Very true ... I think that SF even on 1000 cores would still lose to Lc0 some pretty spectacular games. I mean just look at yesterday's loss ... out of the opening and each engine plays a few moves by itself and you have this position

[d]r2q1k1r/3b1pp1/1p2p2p/p2pP2P/P1p2P2/3B4/2P1QBP1/R3K2R w KQ - 0 20

SF has reacted to Lc0 pawn attack on the Kingside by giving up a bishop temporarily to where it seemed like Lc0 would have to give it back for a couple of pawns. But here Lc0 uncorks Bg6!!! giving up the bishop without any pawn compensation.


[d]r2q1k1r/3b1pp1/1p2p1Bp/p2pP2P/P1p2P2/8/2P1QBP1/R3K2R b KQ - 0 20

It was really hard to see how this was going to work out well for Lc0 ... but the entire plan was to entomb the rook on h8 and the continuation for Lc0 was to give up a total of 3 pawn to make that happen. It somehow made it work and was even very happy to exchange queens and go for what seemed like a super slow strangulation 3 pawns down. If you want to understand why NN engines are extremely strong and very difficult for AB to handle, it is games like this. There is just no calculating through something like this.


Here is what it looked like after about 30 moves … YES … Lc0 is 3 pawns down but SF is totally helpless as the rook on h8 can't get developed!

[d]2r1b1kr/6p1/4P1Pp/1p1p1P2/p1p3p1/2P5/3K1B2/RR6 w - - 0 32


And the full game for anyone interested ... very powerful game done in A0 style !!!

1. e4 e6 2. d4 d5 3. Nc3 Nf6 4. Bg5 Bb4 5. e5 h6 6. Be3 Ne4 7. Qg4 Kf8 8. a3 Bxc3+ 9. bxc3 Nxc3 10. Bd3 b6 11. h4 Nc6 12. h5 Bd7 13. Ne2 Nxe2 14. Qxe2 Ne7 15. a4 a5 16. f4 Nf5 17. Bf2 Nxd4 18. Bxd4 c5 19. Bf2 c4 20. Bg6 fxg6 21. hxg6 Kg8 22. c3 Qf8 23. Qg4 Qf5 24. Qxf5 exf5 25. Rb1 Rb8 26. g4 fxg4 27. e6 Bxa4 28. Kd2 Be8 29. f5 b5 30. Ra1 a4 31. Rhb1 Rc8 32. Bh4 g3 33. Ke3 Bc6 34. Bxg3 Kf8 35. Bd6+ Ke8 36. Rf1 Rd8 37. Bb4 d4+ 38. cxd4 Bd5 39. f6 Bxe6 40. f7+ Kd7 41. f8=R 1-0

A 1000 core Clusterfish may not be suffering from the same horizon effect Stock and Bluefish are.
After all, it would get 800-1000 million NPS.

A 1000 core Stockfish would be like regular stockfish getting 1450 minutes + 3 minutes per move vs Leela getting only 120 minutes + 15sec per move.
It could still lose but the ELO difference between 1000fish and Leela would be around 150-200. So Leela would be in deep trouble.

People are requesting for a bonus match between 176 thread Bluefish vs the same Leela running on 2 GPU. Aloril may probably set up that !!
What is your bet?
176 Threads will win or 2 GPU will win? My bet is still on Leela.

Putting heavy hardware is not the solution of software problem. Even if you have 1,000,000 cores cpu running for 1 million years per move, horizon effect is still there(you just postpone the horizon effect from certain depth to certain depth). Beyond horizon, SF will still play like 1200 (his static positional evaluation is like 1200), and Leela may possibly win a few games.

Will Leela use a new net vs Bluefish?

And stop using those hyperboles, horizon effect would no longer be an issue with 1 million cores running for 1 million year per move. Leela only has an ELO of roughly 3450,
Not that Stockfish could ever run that many cores. Besides having a shit ton of code that would be useless if you run it for that long with that many cores.
But if we somehow create chess hardware that has 1m cores, each doing 5 million NPS. (Deep Blue 2 did 1-2 mnps per chip) If you let it run for 1 million years it most likely provides a weak solution to chess. Although it wouldn't be able to strongly solve chess.

Your right I think. I estimated LCZ to be at 3400. It could be 3450. BTW, Brainfish is at 3532.

[Leo]

How can one know whether the engines are playing on equal terms?
How can you compare CPU and GPU and state you've found a balance?

Nohow.
There are only subjective methods to make comparison.
Moreover AB engines are made basically for playing middle games and endgames.
An AB engines will be banished from a competition if it has a built-in opening book.
An NN type engines has definitively a built-in opening + middle game + endgame book.
AB engines have their own benefit and their own drawback and NN type engines have these too.
It is obvious the development of hardware and software gives more opportunity for enhancing
the chess power of the NN type engines than for an AB type engines.
But the problem of chess can not be solved by an NN type engine either.

Note
I think the performance of Stockfish against NN engines would be enhanced if developers of Stockfish would make test not only against the actual master of Stockfish but against the best NN engines and if they would modify in some measure the source of Stockfish according to the result of tests.
Until now to make these tests was problematic because the fast changing in the power of NN engines.
But to now the chess power of NN engines reach near a plateau so there is sense to make tests with NN type engines too.

does anyone observed a phenomenon about Lc0? there is not saturation! against logic
corres how you say its near plateau? i see stable rapid progress!

The curious thing is that it might be close to the level Alpha zero was at. Probably not quite. The next 6 months could show if LCZero has maxed out. I am very curious.

[Leo]

Very true ... I think that SF even on 1000 cores would still lose to Lc0 some pretty spectacular games. I mean just look at yesterday's loss ... out of the opening and each engine plays a few moves by itself and you have this position

[d]r2q1k1r/3b1pp1/1p2p2p/p2pP2P/P1p2P2/3B4/2P1QBP1/R3K2R w KQ - 0 20

Indeed that was probably the most spectacular lc0 sufi win so far. On move 23 and after calculating ~16.6 billion nodes SF gave evaluation of just 0.49 And on move 41 the game was already adjudicated as a complete destruction of SF position. That is quite incredible. SF probably completely underestimated that line that was played in the game when white can just give up pawns to completely cramp SF position. Once the Queens are off SF actually realizes it's in deep trouble, but move before that it thinks the position is OK. So I'd like to see how many nodes would be necessary for SF to realize that Qf5 is actually losing and I tend to agree that 1000 core SF might not have cut it in a practical time limits. Maybe someone with powerful HW can test it.

You want powerful hardware? Bluefish running on 192 threads and with 7-man tablebases didn't see it...

Where is Bluefish located and who owns it? Who controls it?

[Leo]

...

There is no any "spectacular" and there is no any "plan" on the side of Leela.
In reality the position after 19.-c4 and the good continuation is in the 32930net of Leela.
To prove this I used the weak GTX 1060 to find the 20.Bg6 move.
From net30100 to net32000 Leela want to move 20.Be4, 20.Bxc4, 20.0-0.
But using net32500 Leela finds 20.Bg6 immediately!
So anywhere between net32000 and net32500 Leela has analysed yet this position during self play
and 20.Bg6 is in the NN of Leela.
Obviously Stockfish also would be much more stronger if it could use a good book instead of analysing every each position.


I am sorry but I miss-wrote the number of moves.
The text with good move numbers is above.

I agree 100 %. It has to calculate anew the opening every single game. LCZ has already done that before the games starts. Of course these have been the rules for the TCEC for years and it is impressive that a new engine can do what it is doing.

[syzygy]

...

There is no any "spectacular" and there is no any "plan" on the side of Leela.
In reality the position after 19.-c4 and the good continuation is in the 32930net of Leela.
To prove this I used the weak GTX 1060 to find the 20.Bg6 move.
From net30100 to net32000 Leela want to move 20.Be4, 20.Bxc4, 20.0-0.
But using net32500 Leela finds 20.Bg6 immediately!
So anywhere between net32000 and net32500 Leela has analysed yet this position during self play
and 20.Bg6 is in the NN of Leela.
Obviously Stockfish also would be much more stronger if it could use a good book instead of analysing every each position.


I am sorry but I miss-wrote the number of moves.
The text with good move numbers is above.

I agree 100 %. It has to calculate anew the opening every single game. LCZ has already done that before the games starts. Of course these have been the rules for the TCEC for years and it is impressive that a new engine can do what it is doing.

I agree 0%. The probability that LCZ between net32000 and net32500 has seen that specific position is much smaller than 0.000001%. Between net32000 and net32500, LCZ may somehow have learned to assess that position correctly, but that is no reason to believe that LCZ has seen exactly that position.

From net32000 and net32500, LCZ has learned new patterns and one or more of those patterns is what has allowed it to find the right move in that position (and in many other positions). This has nothing to do with having a "good book".

[George Tsavdaris]

There is no any "spectacular" and there is no any "plan" on the side of Leela.
In reality the position after 20.-c4 and the good continuation is in the 32930net of Leela.
To prove this I used the weak GTX 1060 to find the 21.Bg6 move.
From net30100 to net32000 Leela want to move 21.Be4, 21.Bxc4, 21.0-0.
But using net32500 Leela finds 21.Bg6 immediately!
So anywhere between net32000 and net32500 Leela has analysed yet this position during self play
and 21.Bg6 is in the NN of Leela.
Obviously Stockfish also would be much more stronger if it could use a good book instead of analysing every each position.

You have ZERO idea about how Neural Net engines work.

[Graham Banks]

http://www.computerchess.org.uk/ccrl/40 ... 089_64-bit

The CPU version is improving rapidly too.

[corres]

There is no any "spectacular" and there is no any "plan" on the side of Leela.
In reality the position after 19.-c4 and the good continuation is in the 32930net of Leela.
To prove this I used the weak GTX 1060 to find the 20.Bg6 move.
From net30100 to net32000 Leela want to move 20.Be4, 20.Bxc4, 20.0-0.
But using net32500 Leela finds 21.Bg6 immediately!
So anywhere between net32000 and net32500 Leela has analysed yet this position during self play
and 20.Bg6 is in the NN of Leela.
Obviously Stockfish also would be much more stronger if it could use a good book instead of analysing every each position.

You have ZERO idea about how Neural Net engines work.

Thanks.
But if during self play LC0 never move in this position Bg6, from what she knows Bg6 is a good move?
For the exact answer we should look through the whole self play game between net32000 and net32500.
I am very curious to know the opinion of the main developers of LC0.
Registering the similarity between pattern is a very complex problem even in an NN system too.
If the developers of LC0 solved this problem I congratulate to they.