"Fat fritz" stronger than Stockfish ?

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Ovyron
Posts: 2815
Joined: Tue Jul 03, 2007 2:30 am

Re: "Fat fritz" stronger than Stockfish ?

Post by Ovyron » Sun Sep 29, 2019 1:22 am

jp wrote:
Sat Sep 28, 2019 11:52 pm
And in practice SF does not play really weak chess.
Because playing decent chess +99.9% of the time is enough to hide its weaknesses. If NNs see some moves that AB engines simply don't see or won't play and they're best, in those positions SF is playing really weak chess. What happens in analysis after the game has been played doesn't matter, just like "best play" doesn't matter, and what matters is the moves that are played when a game is actually happening.

If I'm playing a game against you, jp (let's make this personal), and I play a move suggested by a NN, and you miss it because you only used Stockfish, and I beat you because of that, it doesn't matter if afterwards you analyze the position and claim that the move I played wasn't better than the others, that the actual score is 0.00, that Stockfish was right and that it's a draw with best play, because I already beat you.

That's how chess is like, it's about making the opponent blunder in a drawn position, if it's more likely than not the evaluation should reflect that, but it makes no sense to show scores near 0.00 because it's a drawn position.
Great spirits have always encountered violent opposition from mediocre minds.

jp
Posts: 836
Joined: Mon Apr 23, 2018 5:54 am

Re: "Fat fritz" stronger than Stockfish ?

Post by jp » Sun Sep 29, 2019 1:35 am

Ovyron wrote:
Sun Sep 29, 2019 1:22 am
If NNs see some moves that AB engines simply don't see or won't play and they're best, in those positions SF is playing really weak chess.
Well, that's the question: are there moves like that? So far, we've looked at many candidates in this forum and found no examples.

Lc & SF seem to agree a lot even when we humans suspect they're both wrong.
Last edited by jp on Sun Sep 29, 2019 1:39 am, edited 1 time in total.

User avatar
Ovyron
Posts: 2815
Joined: Tue Jul 03, 2007 2:30 am

Re: "Fat fritz" stronger than Stockfish ?

Post by Ovyron » Sun Sep 29, 2019 1:37 am

Because all the possible examples that could be produced would lead to a position where "best play" can still draw the game, even though in the games played nobody can guarantee "best play" :roll:
Great spirits have always encountered violent opposition from mediocre minds.

jp
Posts: 836
Joined: Mon Apr 23, 2018 5:54 am

Re: "Fat fritz" stronger than Stockfish ?

Post by jp » Sun Sep 29, 2019 1:41 am

Why would that be the case? We can use any examples we want.

Your complaint isn't really about NN vs. AB engines. You want a CC software tool that can help predict what's hardest for your computer-assisted human opponent. We could change the GUI to try to display how tricky lines are in that way, and that would work equally well or badly with both NN and AB engines.

In the example on the last page, Leela & Stockfish don't really disagree. There are examples in this forum where they have the same rather arbitrary PV for 14 or 20 plies.

User avatar
M ANSARI
Posts: 3411
Joined: Thu Mar 16, 2006 6:10 pm

Re: "Fat fritz" stronger than Stockfish ?

Post by M ANSARI » Sun Sep 29, 2019 6:45 am

jp wrote:
Sat Sep 28, 2019 9:22 am
M ANSARI wrote:
Sat Sep 28, 2019 7:08 am
I totally disagree. I think with NN engines there are some moves that AB engines simply don't see or won't play.
Don't just disagree abstractly. Please try to give examples of some best moves that NN engines see that AB engines don't. There have been many potential examples looked at in this forum, and none of them has ended up not seen or to be the single best move. There are also examples of claimed differences in evaluation of NN and AB engines, which weren't differences or were probably correctly evaluated by the AB engine, not the NN engine.

I am not sure what you mean. Are you trying to tell me that NN engines will not play or suggest a move that is stronger than SF? If so then I think you missed the zillion games already published where SF loses to Alpha Zero or Lc0 or other NN engines ... check YouTube there are hundreds if not thousands of annotated games. When I used to play my chess computer on Playchess, I used to analyze openings and try to come up with new move suggestions that "felt" good even though an engine would gives it a terrible evaluation or would have it ranked very low in choice moves. When you do that often enough, you tend to know which positions the engine will not understand and probe those moves until you find a good move ... it almost felt like mining. An experienced person who plays these engine games can actually follow a engine match and correctly predict that even though an engine is showing a slight - evaluation, the position is won simply because with experience you know that a -.3 evaluation when down a full piece in an open position with lots of piece activity will be winning. Or that when engines show 0.00 in an endgame and one side has a piece that is stuck ... you just know it is not 0.00 and within several moves things will turn. This is what NN engines do better ... they understand these positions. Now when Lc0 came along and very few people were using it ... holy crap ... it made things dramatically easier as it suggested very quickly moves that AB engines did not like ... yet turned out to be very strong. Unfortunately once everyone also started using Lc0, this fun stopped and it was back to square one. So there is absolutely no doubt that Lc0 or NN engines have brought in a new dimension in move choices. But as I mentioned before, AB engines also have their own AI and that can be tuned by learning from mistakes done with NN engines. This was never possible before as you need millions of games to be confident that an evaluation change is needed, and no engine was strong enough to point out evaluation errors.

If you want proof that Lc0 plays some moves better than SF ... just look at the huge ELO jump in SF since Lc0 came out. Progress had flat lined with SF for quite a while ... but after Lc0, there came a huge ELO gain.

User avatar
Ovyron
Posts: 2815
Joined: Tue Jul 03, 2007 2:30 am

Re: "Fat fritz" stronger than Stockfish ?

Post by Ovyron » Sun Sep 29, 2019 6:59 am

M ANSARI wrote:
Sun Sep 29, 2019 6:45 am
I am not sure what you mean. Are you trying to tell me that NN engines will not play or suggest a move that is stronger than SF? If so then I think you missed the zillion games already published where SF loses to Alpha Zero or Lc0 or other NN engines ... check YouTube there are hundreds if not thousands of annotated games.
I think jp's claim is that if you go and download those games and analyze them for a while, it turns out those positions could have been saved with "best play". So he says that a move SF comes up with is just as good as the one from NNs, because neither can beat "best play".

(I disagree with jp on the basis that if a move can beat SF under those conditions, and another cannot, the former is better, regardless of what "best play" would have played)
Great spirits have always encountered violent opposition from mediocre minds.

User avatar
M ANSARI
Posts: 3411
Joined: Thu Mar 16, 2006 6:10 pm

Re: "Fat fritz" stronger than Stockfish ?

Post by M ANSARI » Sun Sep 29, 2019 7:09 am

Ovyron wrote:
Sun Sep 29, 2019 6:59 am
M ANSARI wrote:
Sun Sep 29, 2019 6:45 am
I am not sure what you mean. Are you trying to tell me that NN engines will not play or suggest a move that is stronger than SF? If so then I think you missed the zillion games already published where SF loses to Alpha Zero or Lc0 or other NN engines ... check YouTube there are hundreds if not thousands of annotated games.
I think jp's claim is that if you go and download those games and analyze them for a while, it turns out those positions could have been saved with "best play". So he says that a move SF comes up with is just as good as the one from NNs, because neither can beat "best play".

(I disagree with jp on the basis that if a move can beat SF under those conditions, and another cannot, the former is better, regardless of what "best play" would have played)

Unfortunately that is not how chess works. Time is limited and "best play" needs to be done within a set time parameter. Given infinite time I have no doubt that even a weak AB engine will find best move every time. Of course this parameter changes as processing power increases, but even then it is finite for that period. If SF plays a move that theoretically still draws with best play ... yet it doesn't see the correct move order of the refutation ... that is just a move that it stumbled upon, and by luck it draws. You can only be lucky so many times.

USGroup1
Posts: 33
Joined: Sun Oct 14, 2018 5:01 pm
Full name: Sina Vaziri

Re: "Fat fritz" stronger than Stockfish ?

Post by USGroup1 » Sun Sep 29, 2019 10:01 am

Ovyron wrote:
Sat Sep 28, 2019 11:42 pm
jp wrote:
Sat Sep 28, 2019 10:10 am
It's likely a draw with best play

<Snip>

How can people object then to SF's evalution of 0.00?
Because an evaluation of 0.00 isn't useful. An engine that shows 0.00 for all moves that are "a draw with best play" is useless, and in practice it'd play really weak.
If the position is drawn with best play the engine evaluation MUST BE 0.00 otherwise it's absolutely useless, not the other way around!!

zullil
Posts: 5679
Joined: Mon Jan 08, 2007 11:31 pm
Location: PA USA
Full name: Louis Zulli

Re: "Fat fritz" stronger than Stockfish ?

Post by zullil » Sun Sep 29, 2019 10:30 am

USGroup1 wrote:
Sun Sep 29, 2019 10:01 am
Ovyron wrote:
Sat Sep 28, 2019 11:42 pm
jp wrote:
Sat Sep 28, 2019 10:10 am
It's likely a draw with best play

<Snip>

How can people object then to SF's evalution of 0.00?
Because an evaluation of 0.00 isn't useful. An engine that shows 0.00 for all moves that are "a draw with best play" is useless, and in practice it'd play really weak.
If the position is drawn with best play the engine evaluation MUST BE 0.00 otherwise it's absolutely useless, not the other way around!!
Then almost every position that has arisen in almost every game ever played should be 0.00. :wink:

I think folks forget that the evaluation is used by the engine to guide its search. Hard to know what choices to make if all the theoretically non-losing moves have evaluation 0.00.

jp
Posts: 836
Joined: Mon Apr 23, 2018 5:54 am

Re: "Fat fritz" stronger than Stockfish ?

Post by jp » Sun Sep 29, 2019 11:21 am

Ovyron wrote:
Sun Sep 29, 2019 6:59 am
M ANSARI wrote:
Sun Sep 29, 2019 6:45 am
I am not sure what you mean. Are you trying to tell me that NN engines will not play or suggest a move that is stronger than SF? If so then I think you missed the zillion games already published where SF loses to NN engines ...
I think jp's claim is that if you go and download those games and analyze them for a while, it turns out those positions could have been saved with "best play". So he says that a move SF comes up with is just as good as the one from NNs, because neither can beat "best play".

(I disagree with jp on the basis that if a move can beat SF under those conditions, and another cannot, the former is better, regardless of what "best play" would have played)
No, I'm not talking about engine games against other engines. Engines of course can lose games because they don't play perfectly. I don't think they're even close to playing perfectly, but that's a separate debate (which also depends on how you define "close"). As with top human games, those losses happen far more frequently at very short time controls. If you look at those, you see the losing side just played badly.

What I'm talking about is the positions we in this forum find interesting enough to want to analyze. We then don't give SF or Leela only 10 seconds to decide on a move, because why would we want it to spit out poor-quality analysis? It's not taking "infinite" time to allow the engine to calculate for half an hour or many hours. We're forced to do that anyway, because even if SF or Leela comes up with a good move and evaluation immediately, you'd be foolish to trust it without seeing whether it changes its mind.

I was wondering why Ovyron talked about all the positions having to be "a draw with best play". That sounded very strange. Now I realize it's because of the above. Obviously, the positions we look at in this forum could be draws or wins for either side. That they are interesting enough for someone to post suggests at least some initial doubt by humans about which result it is. (It's also true, for the same reason, that often we still cannot be certain after all our engine analysis of the result, because the engines are simply not good enough to give us that.) e.g. The position in this thread that Nordlandia gave is from a human GM game. That position isn't about which engine is stronger than another, because no engine got itself into that position. You can of course make engines complete the game starting from that human-game position, which is what Nordlandia did.

Post Reply