"Fat fritz" stronger than Stockfish ?

[Ovyron]

And in practice SF does not play really weak chess.

Because playing decent chess +99.9% of the time is enough to hide its weaknesses. If NNs see some moves that AB engines simply don't see or won't play and they're best, in those positions SF is playing really weak chess. What happens in analysis after the game has been played doesn't matter, just like "best play" doesn't matter, and what matters is the moves that are played when a game is actually happening.

If I'm playing a game against you, jp (let's make this personal), and I play a move suggested by a NN, and you miss it because you only used Stockfish, and I beat you because of that, it doesn't matter if afterwards you analyze the position and claim that the move I played wasn't better than the others, that the actual score is 0.00, that Stockfish was right and that it's a draw with best play, because I already beat you.

That's how chess is like, it's about making the opponent blunder in a drawn position, if it's more likely than not the evaluation should reflect that, but it makes no sense to show scores near 0.00 because it's a drawn position.

[jp]

If NNs see some moves that AB engines simply don't see or won't play and they're best, in those positions SF is playing really weak chess.

Well, that's the question: are there moves like that? So far, we've looked at many candidates in this forum and found no examples.

Lc & SF seem to agree a lot even when we humans suspect they're both wrong.

[Ovyron]

Because all the possible examples that could be produced would lead to a position where "best play" can still draw the game, even though in the games played nobody can guarantee "best play"

[jp]

Why would that be the case? We can use any examples we want.

Your complaint isn't really about NN vs. AB engines. You want a CC software tool that can help predict what's hardest for your computer-assisted human opponent. We could change the GUI to try to display how tricky lines are in that way, and that would work equally well or badly with both NN and AB engines.

In the example on the last page, Leela & Stockfish don't really disagree. There are examples in this forum where they have the same rather arbitrary PV for 14 or 20 plies.

[M ANSARI]

I totally disagree. I think with NN engines there are some moves that AB engines simply don't see or won't play.

Don't just disagree abstractly. Please try to give examples of some best moves that NN engines see that AB engines don't. There have been many potential examples looked at in this forum, and none of them has ended up not seen or to be the single best move. There are also examples of claimed differences in evaluation of NN and AB engines, which weren't differences or were probably correctly evaluated by the AB engine, not the NN engine.

I am not sure what you mean. Are you trying to tell me that NN engines will not play or suggest a move that is stronger than SF? If so then I think you missed the zillion games already published where SF loses to Alpha Zero or Lc0 or other NN engines ... check YouTube there are hundreds if not thousands of annotated games. When I used to play my chess computer on Playchess, I used to analyze openings and try to come up with new move suggestions that "felt" good even though an engine would gives it a terrible evaluation or would have it ranked very low in choice moves. When you do that often enough, you tend to know which positions the engine will not understand and probe those moves until you find a good move ... it almost felt like mining. An experienced person who plays these engine games can actually follow a engine match and correctly predict that even though an engine is showing a slight - evaluation, the position is won simply because with experience you know that a -.3 evaluation when down a full piece in an open position with lots of piece activity will be winning. Or that when engines show 0.00 in an endgame and one side has a piece that is stuck ... you just know it is not 0.00 and within several moves things will turn. This is what NN engines do better ... they understand these positions. Now when Lc0 came along and very few people were using it ... holy crap ... it made things dramatically easier as it suggested very quickly moves that AB engines did not like ... yet turned out to be very strong. Unfortunately once everyone also started using Lc0, this fun stopped and it was back to square one. So there is absolutely no doubt that Lc0 or NN engines have brought in a new dimension in move choices. But as I mentioned before, AB engines also have their own AI and that can be tuned by learning from mistakes done with NN engines. This was never possible before as you need millions of games to be confident that an evaluation change is needed, and no engine was strong enough to point out evaluation errors.

If you want proof that Lc0 plays some moves better than SF ... just look at the huge ELO jump in SF since Lc0 came out. Progress had flat lined with SF for quite a while ... but after Lc0, there came a huge ELO gain.

[Ovyron]

I am not sure what you mean. Are you trying to tell me that NN engines will not play or suggest a move that is stronger than SF? If so then I think you missed the zillion games already published where SF loses to Alpha Zero or Lc0 or other NN engines ... check YouTube there are hundreds if not thousands of annotated games.

I think jp's claim is that if you go and download those games and analyze them for a while, it turns out those positions could have been saved with "best play". So he says that a move SF comes up with is just as good as the one from NNs, because neither can beat "best play".

(I disagree with jp on the basis that if a move can beat SF under those conditions, and another cannot, the former is better, regardless of what "best play" would have played)

[M ANSARI]

I am not sure what you mean. Are you trying to tell me that NN engines will not play or suggest a move that is stronger than SF? If so then I think you missed the zillion games already published where SF loses to Alpha Zero or Lc0 or other NN engines ... check YouTube there are hundreds if not thousands of annotated games.

I think jp's claim is that if you go and download those games and analyze them for a while, it turns out those positions could have been saved with "best play". So he says that a move SF comes up with is just as good as the one from NNs, because neither can beat "best play".

(I disagree with jp on the basis that if a move can beat SF under those conditions, and another cannot, the former is better, regardless of what "best play" would have played)

Unfortunately that is not how chess works. Time is limited and "best play" needs to be done within a set time parameter. Given infinite time I have no doubt that even a weak AB engine will find best move every time. Of course this parameter changes as processing power increases, but even then it is finite for that period. If SF plays a move that theoretically still draws with best play ... yet it doesn't see the correct move order of the refutation ... that is just a move that it stumbled upon, and by luck it draws. You can only be lucky so many times.

[USGroup1]

It's likely a draw with best play

<Snip>

How can people object then to SF's evalution of 0.00?

Because an evaluation of 0.00 isn't useful. An engine that shows 0.00 for all moves that are "a draw with best play" is useless, and in practice it'd play really weak.

If the position is drawn with best play the engine evaluation MUST BE 0.00 otherwise it's absolutely useless, not the other way around!!

[zullil]

It's likely a draw with best play

<Snip>

How can people object then to SF's evalution of 0.00?

Because an evaluation of 0.00 isn't useful. An engine that shows 0.00 for all moves that are "a draw with best play" is useless, and in practice it'd play really weak.

If the position is drawn with best play the engine evaluation MUST BE 0.00 otherwise it's absolutely useless, not the other way around!!

Then almost every position that has arisen in almost every game ever played should be 0.00.

I think folks forget that the evaluation is used by the engine to guide its search. Hard to know what choices to make if all the theoretically non-losing moves have evaluation 0.00.

[jp]

I am not sure what you mean. Are you trying to tell me that NN engines will not play or suggest a move that is stronger than SF? If so then I think you missed the zillion games already published where SF loses to NN engines ...

I think jp's claim is that if you go and download those games and analyze them for a while, it turns out those positions could have been saved with "best play". So he says that a move SF comes up with is just as good as the one from NNs, because neither can beat "best play".

(I disagree with jp on the basis that if a move can beat SF under those conditions, and another cannot, the former is better, regardless of what "best play" would have played)

No, I'm not talking about engine games against other engines. Engines of course can lose games because they don't play perfectly. I don't think they're even close to playing perfectly, but that's a separate debate (which also depends on how you define "close"). As with top human games, those losses happen far more frequently at very short time controls. If you look at those, you see the losing side just played badly.

What I'm talking about is the positions we in this forum find interesting enough to want to analyze. We then don't give SF or Leela only 10 seconds to decide on a move, because why would we want it to spit out poor-quality analysis? It's not taking "infinite" time to allow the engine to calculate for half an hour or many hours. We're forced to do that anyway, because even if SF or Leela comes up with a good move and evaluation immediately, you'd be foolish to trust it without seeing whether it changes its mind.

I was wondering why Ovyron talked about all the positions having to be "a draw with best play". That sounded very strange. Now I realize it's because of the above. Obviously, the positions we look at in this forum could be draws or wins for either side. That they are interesting enough for someone to post suggests at least some initial doubt by humans about which result it is. (It's also true, for the same reason, that often we still cannot be certain after all our engine analysis of the result, because the engines are simply not good enough to give us that.) e.g. The position in this thread that Nordlandia gave is from a human GM game. That position isn't about which engine is stronger than another, because no engine got itself into that position. You can of course make engines complete the game starting from that human-game position, which is what Nordlandia did.