End of Era is there: SF is finally beaten!

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
AdminX
Posts: 5182
Joined: Mon Mar 13, 2006 1:34 pm
Location: Acworth, GA
Contact:

Re: End of Era is there: SF is finally beaten!

Post by AdminX » Sat Feb 16, 2019 11:14 am

I don't think "Father's" tricks that worked so well on AB engines would hold up to the current version of LC0. :D
Attachments
2-16-2019 6-11-13 AM.png
2-16-2019 6-11-13 AM.png (48.16 KiB) Viewed 1634 times
"Good decisions come from experience, and experience comes from bad decisions."
__________________________________________________________________
Ted Summers

mirek
Posts: 42
Joined: Sat Mar 24, 2018 3:18 pm

Re: End of Era is there: SF is finally beaten!

Post by mirek » Sat Feb 16, 2019 11:25 am

M ANSARI wrote:
Sat Feb 16, 2019 6:29 am
Very true ... I think that SF even on 1000 cores would still lose to Lc0 some pretty spectacular games. I mean just look at yesterday's loss ... out of the opening and each engine plays a few moves by itself and you have this position

[d]r2q1k1r/3b1pp1/1p2p2p/p2pP2P/P1p2P2/3B4/2P1QBP1/R3K2R w KQ - 0 20
Indeed that was probably the most spectacular lc0 sufi win so far. On move 23 and after calculating ~16.6 billion nodes SF gave evaluation of just 0.49 And on move 41 the game was already adjudicated as a complete destruction of SF position. That is quite incredible. SF probably completely underestimated that line that was played in the game when white can just give up pawns to completely cramp SF position. Once the Queens are off SF actually realizes it's in deep trouble, but move before that it thinks the position is OK. So I'd like to see how many nodes would be necessary for SF to realize that Qf5 is actually losing and I tend to agree that 1000 core SF might not have cut it in a practical time limits. Maybe someone with powerful HW can test it.

User avatar
CMCanavessi
Posts: 835
Joined: Thu Dec 28, 2017 3:06 pm
Location: Argentina

Re: End of Era is there: SF is finally beaten!

Post by CMCanavessi » Sat Feb 16, 2019 12:49 pm

mirek wrote:
Sat Feb 16, 2019 11:25 am
M ANSARI wrote:
Sat Feb 16, 2019 6:29 am
Very true ... I think that SF even on 1000 cores would still lose to Lc0 some pretty spectacular games. I mean just look at yesterday's loss ... out of the opening and each engine plays a few moves by itself and you have this position

[d]r2q1k1r/3b1pp1/1p2p2p/p2pP2P/P1p2P2/3B4/2P1QBP1/R3K2R w KQ - 0 20
Indeed that was probably the most spectacular lc0 sufi win so far. On move 23 and after calculating ~16.6 billion nodes SF gave evaluation of just 0.49 And on move 41 the game was already adjudicated as a complete destruction of SF position. That is quite incredible. SF probably completely underestimated that line that was played in the game when white can just give up pawns to completely cramp SF position. Once the Queens are off SF actually realizes it's in deep trouble, but move before that it thinks the position is OK. So I'd like to see how many nodes would be necessary for SF to realize that Qf5 is actually losing and I tend to agree that 1000 core SF might not have cut it in a practical time limits. Maybe someone with powerful HW can test it.
You want powerful hardware? Bluefish running on 192 threads and with 7-man tablebases didn't see it...
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls

Nay Lin Tun
Posts: 529
Joined: Mon Jan 16, 2012 5:34 am

Re: End of Era is there: SF is finally beaten!

Post by Nay Lin Tun » Sat Feb 16, 2019 1:04 pm

henk2 wrote:
Sat Feb 16, 2019 9:40 am
M ANSARI wrote:
Sat Feb 16, 2019 6:29 am
MikeB wrote:
Sat Feb 16, 2019 5:49 am
corres wrote:
Fri Feb 15, 2019 5:24 pm
corres wrote:
Fri Feb 15, 2019 9:50 am
Graham Banks wrote:
Fri Feb 15, 2019 9:16 am
How can one know whether the engines are playing on equal terms?
How can you compare CPU and GPU and state you've found a balance?
Nohow.
There are only subjective methods to make comparison.
Moreover AB engines are made basically for playing middle games and endgames.
An AB engines will be banished from a competition if it has a built-in opening book.
An NN type engines has definitively a built-in opening + middle game + endgame book.
AB engines have their own benefit and their own drawback and NN type engines have these too.
It is obvious the development of hardware and software gives more opportunity for enhancing
the chess power of the NN type engines than for an AB type engines.
But the problem of chess can not be solved by an NN type engine either.
Note
I think the performance of Stockfish against NN engines would be enhanced if developers of Stockfish would make test not only against the actual master of Stockfish but against the best NN engines and if they would modify in some measure the source of Stockfish according to the result of tests.
Until now to make these tests was problematic because the fast changing in the power of NN engines.
But to now the chess power of NN engines reach near a plateau so there is sense to make tests with NN type engines too.
Whoever figures out how to combine the Chess CPU power of a Stockfish like engine with the Chess GPU of an Lc0 like engine will get there first, SO many computers today have 24 , 36 or even 72 cores - take the best x moves from Lc0 and dedicate an AB search on those moves using one of those CPUs with A/B search engine like SF. Someone with some coding skills should be looking at that, Good luck!

Very true ... I think that SF even on 1000 cores would still lose to Lc0 some pretty spectacular games. I mean just look at yesterday's loss ... out of the opening and each engine plays a few moves by itself and you have this position

[d]r2q1k1r/3b1pp1/1p2p2p/p2pP2P/P1p2P2/3B4/2P1QBP1/R3K2R w KQ - 0 20

SF has reacted to Lc0 pawn attack on the Kingside by giving up a bishop temporarily to where it seemed like Lc0 would have to give it back for a couple of pawns. But here Lc0 uncorks Bg6!!! giving up the bishop without any pawn compensation.


[d]r2q1k1r/3b1pp1/1p2p1Bp/p2pP2P/P1p2P2/8/2P1QBP1/R3K2R b KQ - 0 20

It was really hard to see how this was going to work out well for Lc0 ... but the entire plan was to entomb the rook on h8 and the continuation for Lc0 was to give up a total of 3 pawn to make that happen. It somehow made it work and was even very happy to exchange queens and go for what seemed like a super slow strangulation 3 pawns down. If you want to understand why NN engines are extremely strong and very difficult for AB to handle, it is games like this. There is just no calculating through something like this.


Here is what it looked like after about 30 moves … YES … Lc0 is 3 pawns down but SF is totally helpless as the rook on h8 can't get developed!

[d]2r1b1kr/6p1/4P1Pp/1p1p1P2/p1p3p1/2P5/3K1B2/RR6 w - - 0 32


And the full game for anyone interested ... very powerful game done in A0 style !!!

A 1000 core Clusterfish may not be suffering from the same horizon effect Stock and Bluefish are.
After all, it would get 800-1000 million NPS.

A 1000 core Stockfish would be like regular stockfish getting 1450 minutes + 3 minutes per move vs Leela getting only 120 minutes + 15sec per move.
It could still lose but the ELO difference between 1000fish and Leela would be around 150-200. So Leela would be in deep trouble.
People are requesting for a bonus match between 176 thread Bluefish vs the same Leela running on 2 GPU. Aloril may probably set up that !!
What is your bet?
176 Threads will win or 2 GPU will win? My bet is still on Leela.

Putting heavy hardware is not the solution of software problem. Even if you have 1,000,000 cores cpu running for 1 million years per move, horizon effect is still there(you just postpone the horizon effect from certain depth to certain depth). Beyond horizon, SF will still play like 1200 (his static positional evaluation is like 1200), and Leela may possibly win a few games.

jmartus
Posts: 198
Joined: Sun May 16, 2010 12:50 am

Re: End of Era is there: SF is finally beaten!

Post by jmartus » Sat Feb 16, 2019 1:27 pm

Wish we could have a grandmaster comment on what he thinks of these tcec final games.

henk2
Posts: 30
Joined: Mon Jan 14, 2019 6:55 am
Full name: Henk Verbaasdonk

Re: End of Era is there: SF is finally beaten!

Post by henk2 » Sat Feb 16, 2019 2:32 pm

Nay Lin Tun wrote:
Sat Feb 16, 2019 1:04 pm
henk2 wrote:
Sat Feb 16, 2019 9:40 am
M ANSARI wrote:
Sat Feb 16, 2019 6:29 am
MikeB wrote:
Sat Feb 16, 2019 5:49 am
corres wrote:
Fri Feb 15, 2019 5:24 pm
corres wrote:
Fri Feb 15, 2019 9:50 am
Graham Banks wrote:
Fri Feb 15, 2019 9:16 am
How can one know whether the engines are playing on equal terms?
How can you compare CPU and GPU and state you've found a balance?
Nohow.
There are only subjective methods to make comparison.
Moreover AB engines are made basically for playing middle games and endgames.
An AB engines will be banished from a competition if it has a built-in opening book.
An NN type engines has definitively a built-in opening + middle game + endgame book.
AB engines have their own benefit and their own drawback and NN type engines have these too.
It is obvious the development of hardware and software gives more opportunity for enhancing
the chess power of the NN type engines than for an AB type engines.
But the problem of chess can not be solved by an NN type engine either.
Note
I think the performance of Stockfish against NN engines would be enhanced if developers of Stockfish would make test not only against the actual master of Stockfish but against the best NN engines and if they would modify in some measure the source of Stockfish according to the result of tests.
Until now to make these tests was problematic because the fast changing in the power of NN engines.
But to now the chess power of NN engines reach near a plateau so there is sense to make tests with NN type engines too.
Whoever figures out how to combine the Chess CPU power of a Stockfish like engine with the Chess GPU of an Lc0 like engine will get there first, SO many computers today have 24 , 36 or even 72 cores - take the best x moves from Lc0 and dedicate an AB search on those moves using one of those CPUs with A/B search engine like SF. Someone with some coding skills should be looking at that, Good luck!

Very true ... I think that SF even on 1000 cores would still lose to Lc0 some pretty spectacular games. I mean just look at yesterday's loss ... out of the opening and each engine plays a few moves by itself and you have this position

[d]r2q1k1r/3b1pp1/1p2p2p/p2pP2P/P1p2P2/3B4/2P1QBP1/R3K2R w KQ - 0 20

SF has reacted to Lc0 pawn attack on the Kingside by giving up a bishop temporarily to where it seemed like Lc0 would have to give it back for a couple of pawns. But here Lc0 uncorks Bg6!!! giving up the bishop without any pawn compensation.


[d]r2q1k1r/3b1pp1/1p2p1Bp/p2pP2P/P1p2P2/8/2P1QBP1/R3K2R b KQ - 0 20

It was really hard to see how this was going to work out well for Lc0 ... but the entire plan was to entomb the rook on h8 and the continuation for Lc0 was to give up a total of 3 pawn to make that happen. It somehow made it work and was even very happy to exchange queens and go for what seemed like a super slow strangulation 3 pawns down. If you want to understand why NN engines are extremely strong and very difficult for AB to handle, it is games like this. There is just no calculating through something like this.


Here is what it looked like after about 30 moves … YES … Lc0 is 3 pawns down but SF is totally helpless as the rook on h8 can't get developed!

[d]2r1b1kr/6p1/4P1Pp/1p1p1P2/p1p3p1/2P5/3K1B2/RR6 w - - 0 32


And the full game for anyone interested ... very powerful game done in A0 style !!!

A 1000 core Clusterfish may not be suffering from the same horizon effect Stock and Bluefish are.
After all, it would get 800-1000 million NPS.

A 1000 core Stockfish would be like regular stockfish getting 1450 minutes + 3 minutes per move vs Leela getting only 120 minutes + 15sec per move.
It could still lose but the ELO difference between 1000fish and Leela would be around 150-200. So Leela would be in deep trouble.
People are requesting for a bonus match between 176 thread Bluefish vs the same Leela running on 2 GPU. Aloril may probably set up that !!
What is your bet?
176 Threads will win or 2 GPU will win? My bet is still on Leela.

Putting heavy hardware is not the solution of software problem. Even if you have 1,000,000 cores cpu running for 1 million years per move, horizon effect is still there(you just postpone the horizon effect from certain depth to certain depth). Beyond horizon, SF will still play like 1200 (his static positional evaluation is like 1200), and Leela may possibly win a few games.
Will Leela use a new net vs Bluefish?

And stop using those hyperboles, horizon effect would no longer be an issue with 1 million cores running for 1 million year per move. Leela only has an ELO of roughly 3450,
Not that Stockfish could ever run that many cores. Besides having a shit ton of code that would be useless if you run it for that long with that many cores.
But if we somehow create chess hardware that has 1m cores, each doing 5 million NPS. (Deep Blue 2 did 1-2 mnps per chip) If you let it run for 1 million years it most likely provides a weak solution to chess. Although it wouldn't be able to strongly solve chess.

stavros
Posts: 138
Joined: Tue Dec 02, 2014 12:29 am

Re: End of Era is there: SF is finally beaten!

Post by stavros » Sat Feb 16, 2019 2:43 pm

corres wrote:
Fri Feb 15, 2019 5:24 pm
corres wrote:
Fri Feb 15, 2019 9:50 am
Graham Banks wrote:
Fri Feb 15, 2019 9:16 am
How can one know whether the engines are playing on equal terms?
How can you compare CPU and GPU and state you've found a balance?
Nohow.
There are only subjective methods to make comparison.
Moreover AB engines are made basically for playing middle games and endgames.
An AB engines will be banished from a competition if it has a built-in opening book.
An NN type engines has definitively a built-in opening + middle game + endgame book.
AB engines have their own benefit and their own drawback and NN type engines have these too.
It is obvious the development of hardware and software gives more opportunity for enhancing
the chess power of the NN type engines than for an AB type engines.
But the problem of chess can not be solved by an NN type engine either.
Note
I think the performance of Stockfish against NN engines would be enhanced if developers of Stockfish would make test not only against the actual master of Stockfish but against the best NN engines and if they would modify in some measure the source of Stockfish according to the result of tests.
Until now to make these tests was problematic because the fast changing in the power of NN engines.
But to now the chess power of NN engines reach near a plateau so there is sense to make tests with NN type engines too.

does anyone observed a phenomenon about Lc0? there is not saturation! against logic
corres how you say its near plateau? i see stable rapid progress!

User avatar
M ANSARI
Posts: 3411
Joined: Thu Mar 16, 2006 6:10 pm

Re: End of Era is there: SF is finally beaten!

Post by M ANSARI » Sat Feb 16, 2019 3:22 pm

I think Lc0 is just giving us an idea of proof of concept in NN as was shown with A0 with Google. The potential is there, but to be honest Lc0 would not be even on my top 10 list of engines to use if I wanted to analyze a chess game. At least not a stand alone engine. It can miss one move tactics that really are very simple to see even for a human ... and after analyzing some of A0 games, I saw one move tactical misses as well. Lc0 also has a serious thing with the endgame that some people are politely calling "trolling" ... but I think that this is just poorly written software or poorly written search code. It just misses way too many wins in games it is completely winning and endlessly prolongs games and sometimes even misses wins in clear cut endgames. A good example would be in the amazing game that it won ... I mean promote to a rook???? ... I mean come on that is just stupid! No doubt all this will be fixed, especially if some of the talented chess coders on the SF team or on Houdini team or on Komodo team get involved. I think the proof of concept of NN has been proven beyond a shadow of a doubt, and it is time to get everyone on the band wagon.

mwyoung
Posts: 1642
Joined: Wed May 12, 2010 8:00 pm

Re: End of Era is there: SF is finally beaten!

Post by mwyoung » Sat Feb 16, 2019 3:32 pm

jmartus wrote:
Sat Feb 16, 2019 1:27 pm
Wish we could have a grandmaster comment on what he thinks of these tcec final games.
There are many grandmasters that are commenting and analyzing the games from the tcec 14 super final. Just search YouTube.
Professing themselves to be wise, they became fools,
Take on me. foes 0

corres
Posts: 1757
Joined: Wed Nov 18, 2015 10:41 am
Location: hungary

Re: End of Era is there: SF is finally beaten!

Post by corres » Sat Feb 16, 2019 4:37 pm

M ANSARI wrote:
Sat Feb 16, 2019 6:29 am
...
I think that SF even on 1000 cores would still lose to Lc0 some pretty spectacular games. I mean just look at yesterday's loss ... out of the opening and each engine plays a few moves by itself and you have this position
[d]r2q1k1r/3b1pp1/1p2p2p/p2pP2P/P1p2P2/3B4/2P1QBP1/R3K2R w KQ - 0 20
SF has reacted to Lc0 pawn attack on the Kingside by giving up a bishop temporarily to where it seemed like Lc0 would have to give it back for a couple of pawns. But here Lc0 uncorks Bg6!!! giving up the bishop without any pawn compensation.
[d]r2q1k1r/3b1pp1/1p2p1Bp/p2pP2P/P1p2P2/8/2P1QBP1/R3K2R b KQ - 0 20
It was really hard to see how this was going to work out well for Lc0 ... but the entire plan was to entomb the rook on h8 and the continuation for Lc0 was to give up a total of 3 pawn to make that happen. It somehow made it work and was even very happy to exchange queens and go for what seemed like a super slow strangulation 3 pawns down. If you want to understand why NN engines are extremely strong and very difficult for AB to handle, it is games like this. There is just no calculating through something like this.
Here is what it looked like after about 30 moves … YES … Lc0 is 3 pawns down but SF is totally helpless as the rook on h8 can't get developed!
[d]2r1b1kr/6p1/4P1Pp/1p1p1P2/p1p3p1/2P5/3K1B2/RR6 w - - 0 32
And the full game for anyone interested ... very powerful game done in A0 style !!!
There is no any "spectacular" and there is no any "plan" on the side of Leela.
In reality the position after 20.-c4 and the good continuation is in the 32930net of Leela.
To prove this I used the weak GTX 1060 to find the 21.Bg6 move.
From net30100 to net32000 Leela want to move 21.Be4, 21.Bxc4, 21.0-0.
But using net32500 Leela finds 21.Bg6 immediately!
So anywhere between net32000 and net32500 Leela has analysed yet this position during self play
and 21.Bg6 is in the NN of Leela.
Obviously Stockfish also would be much more stronger if it could use a good book instead of analysing every each position.

Post Reply