Stoofvlees II d2 is Killing Stockfish in TCEC

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Rebel, chrisw

Viz
Posts: 109
Joined: Tue Apr 09, 2024 6:24 am
Full name: Michael Chaly

Re: Stoofvlees II d2 is Killing Stockfish in TCEC

Post by Viz »

towforce wrote: Tue Apr 30, 2024 4:22 pm
Viz wrote: Tue Apr 30, 2024 1:43 pmBut what [larger board] has to do with "rewarding knowledge"?

Roughly speaking, the size of the game tree rises exponentially with the size of the board (this is discussed, with evidence, in the "Fruit fly races on steroids?" thread - link). In the kind of position in which you're used to getting, say, 10 ply full width and 30 ply selective search, you will now be lucky to get 3 ply full width and 10 ply selective search.

So... if you rely on search for most of your information about the position, then... oh dear - it's gone! Even having the shiniest graphics card on offer won't resolve that.

You're going to be relying on knowledge to enable you to play a good game!

...making a good NN via selfplay is mostly about how much hardware you can pour in it...
If you think that NNs are the only way to encode knowledge, then today's cool kid will be tomorrow's dinosaur!
The thing is that the bigger playing field is the more you benefit from larger NN.
More or less proven, in 3x3 tic tac toe you need to NN, in chess NNUE kinda beats deep net, in go deep nets are insane.
So yes, NNs are the only way to encode knowledge.
The only times where handcrafted stuff beats NNs are in things that can be solved fully or in search - but definitely not in evaluation of position, period.
So enlarging playing field is more or less making NNs even more important than they are in chess.
Not to mention that you make some random game instead of chess, and this is forum about you know, actual chess.
User avatar
towforce
Posts: 11751
Joined: Thu Mar 09, 2006 12:57 am
Location: Birmingham UK

Re: Stoofvlees II d2 is Killing Stockfish in TCEC

Post by towforce »

Viz wrote: Tue Apr 30, 2024 4:38 pm
towforce wrote: Tue Apr 30, 2024 4:22 pm
Viz wrote: Tue Apr 30, 2024 1:43 pmBut what [larger board] has to do with "rewarding knowledge"?

Roughly speaking, the size of the game tree rises exponentially with the size of the board (this is discussed, with evidence, in the "Fruit fly races on steroids?" thread - link). In the kind of position in which you're used to getting, say, 10 ply full width and 30 ply selective search, you will now be lucky to get 3 ply full width and 10 ply selective search.

So... if you rely on search for most of your information about the position, then... oh dear - it's gone! Even having the shiniest graphics card on offer won't resolve that.

You're going to be relying on knowledge to enable you to play a good game!

...making a good NN via selfplay is mostly about how much hardware you can pour in it...
If you think that NNs are the only way to encode knowledge, then today's cool kid will be tomorrow's dinosaur!
The thing is that the bigger playing field is the more you benefit from larger NN.
More or less proven, in 3x3 tic tac toe you need to NN, in chess NNUE kinda beats deep net, in go deep nets are insane.
So yes, NNs are the only way to encode knowledge.
The only times where handcrafted stuff beats NNs are in things that can be solved fully or in search - but definitely not in evaluation of position, period.
So enlarging playing field is more or less making NNs even more important than they are in chess.
Not to mention that you make some random game instead of chess, and this is forum about you know, actual chess.

Yes - very good points! 8-)

I am alone in this quest (like the grand computer chess men of old in the times of mainframe computers and 8-bit CPUs), but I'm hunting for deeper patterns in chess. This has been discussed in depth, but briefly:

* I believe that NNs are encoding a large number of shallow (simple) chess patterns, and very few deep (complex) ones. The strongest evidence for this was that when LC0 first came out, she was playing beautiful chess in the middle game, but was relatively weaker in the endgame

* Unlike, say SAT (the Boolean satisfiability problem), there are a lot of symmetries in chess, and a lot of dependency between the variables (in the sense that chess variables are not independent from each other)

* Therefore, there are going to be some important deep patterns

In human chess, we start by learning simple patterns. Over time, we unconsciously learn deeper patterns (I have seen, in person, GMs look at a chess position and understand it in a heartbeat). Engine NNs learn by looking at billions of positions and their evaluations (many more positions that a human will see in a lifetime). Humans learn by picking a small number of positions (in comparison to the engine NNs) and then analysing these positions in depth: a different process.

For me, the probability that GMs have encoded the best and most important patterns in their brains is close to zero. If we can find these patterns, I see the following benefits:

1. We'll be able to create chess engines that play very well and very quickly on cheap computers (or in a web browser)

2. We might be able to reverse engineer it, come up with a way to teach it, and thus enable "ordinary people" to quickly and easily learn how to play good chess

3. Once we can do that for chess, we can do it in lots of other areas as well. We can have a whole world full of great people!
The simple reveals itself after the complex has been exhausted.
Jouni
Posts: 3324
Joined: Wed Mar 08, 2006 8:15 pm

Re: Stoofvlees II d2 is Killing Stockfish in TCEC

Post by Jouni »

I didn't remember, that this Stoofvlees is current world champion: Stoofvlees wins ICGA World Computer Chess Championship 2023 :D . Stockfish has never won :( . But Is Stoofvlees Lc0 clone?
Jouni
Uri Blass
Posts: 10424
Joined: Thu Mar 09, 2006 12:37 am
Location: Tel-Aviv Israel

Re: Stoofvlees II d2 is Killing Stockfish in TCEC

Post by Uri Blass »

towforce wrote: Tue Apr 30, 2024 7:01 pm
Viz wrote: Tue Apr 30, 2024 4:38 pm
towforce wrote: Tue Apr 30, 2024 4:22 pm
Viz wrote: Tue Apr 30, 2024 1:43 pmBut what [larger board] has to do with "rewarding knowledge"?

Roughly speaking, the size of the game tree rises exponentially with the size of the board (this is discussed, with evidence, in the "Fruit fly races on steroids?" thread - link). In the kind of position in which you're used to getting, say, 10 ply full width and 30 ply selective search, you will now be lucky to get 3 ply full width and 10 ply selective search.

So... if you rely on search for most of your information about the position, then... oh dear - it's gone! Even having the shiniest graphics card on offer won't resolve that.

You're going to be relying on knowledge to enable you to play a good game!

...making a good NN via selfplay is mostly about how much hardware you can pour in it...
If you think that NNs are the only way to encode knowledge, then today's cool kid will be tomorrow's dinosaur!
The thing is that the bigger playing field is the more you benefit from larger NN.
More or less proven, in 3x3 tic tac toe you need to NN, in chess NNUE kinda beats deep net, in go deep nets are insane.
So yes, NNs are the only way to encode knowledge.
The only times where handcrafted stuff beats NNs are in things that can be solved fully or in search - but definitely not in evaluation of position, period.
So enlarging playing field is more or less making NNs even more important than they are in chess.
Not to mention that you make some random game instead of chess, and this is forum about you know, actual chess.

Yes - very good points! 8-)

I am alone in this quest (like the grand computer chess men of old in the times of mainframe computers and 8-bit CPUs), but I'm hunting for deeper patterns in chess. This has been discussed in depth, but briefly:

* I believe that NNs are encoding a large number of shallow (simple) chess patterns, and very few deep (complex) ones. The strongest evidence for this was that when LC0 first came out, she was playing beautiful chess in the middle game, but was relatively weaker in the endgame

* Unlike, say SAT (the Boolean satisfiability problem), there are a lot of symmetries in chess, and a lot of dependency between the variables (in the sense that chess variables are not independent from each other)

* Therefore, there are going to be some important deep patterns

In human chess, we start by learning simple patterns. Over time, we unconsciously learn deeper patterns (I have seen, in person, GMs look at a chess position and understand it in a heartbeat). Engine NNs learn by looking at billions of positions and their evaluations (many more positions that a human will see in a lifetime). Humans learn by picking a small number of positions (in comparison to the engine NNs) and then analysing these positions in depth: a different process.

For me, the probability that GMs have encoded the best and most important patterns in their brains is close to zero. If we can find these patterns, I see the following benefits:

1. We'll be able to create chess engines that play very well and very quickly on cheap computers (or in a web browser)

2. We might be able to reverse engineer it, come up with a way to teach it, and thus enable "ordinary people" to quickly and easily learn how to play good chess

3. Once we can do that for chess, we can do it in lots of other areas as well. We can have a whole world full of great people!
I see no reason to believe that NN encoded many simple patterns and it seems to me that they encoded in the beginning mainly complex patterns.

The simpler patterns are in the endgame and not in the middlegame so
the fact that LC0 was playing beautiful chess in the middle game, but was relatively weaker in the endgame only support my theory that she was type of genius that understood complex things in the middle game that even top GM did not understand but did not understand simple things about the endgame that even weak humans player understand.

Of course now when NN became better they learned also a lot of simple stuff that human also understand and are superior relative to humans in all patterns.
dkappe
Posts: 1632
Joined: Tue Aug 21, 2018 7:52 pm
Full name: Dietrich Kappe

Re: Stoofvlees II d2 is Killing Stockfish in TCEC

Post by dkappe »

Jouni wrote: Tue Apr 30, 2024 8:34 pm I didn't remember, that this Stoofvlees is current world champion: Stoofvlees wins ICGA World Computer Chess Championship 2023 :D . Stockfish has never won :( . But Is Stoofvlees Lc0 clone?
Are you a ChessQuack clone? Do a little reading about the author and the history of the engine before spouting such uninformed speculation.

P.S. The author Gian-Carlo Pascutto is also the author of Leela, you know, the original go program?
Fat Titz by Stockfish, the engine with the bodaciously big net. Remember: size matters. If you want to learn more about this engine just google for "Fat Titz".
Dann Corbit
Posts: 12615
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: Stoofvlees II d2 is Killing Stockfish in TCEC

Post by Dann Corbit »

I am pretty sure he was joking.

BTW, they do test the openings before the contest (or at least, I know some book-writers who had openings in TCEC and carefully tested them before-hand). The problem is analysis effort. The TCEC hardware is very high end and the time control is fairly slow. It takes months to run the contest on this high-end stuff. So there is no way to test all the combinations at the same time control and hardware strength.

In the test contests, the openings were not busted. But when you give the strong hardware a long time to think, it can find defects and break the opening beyond repair. IOW, the openings are broken by the actual tournament play.

Broken openings obviously inflate the strength of weak programs because they will get a win when normally they would not be able to. But the percentage of broken openings is low, so the contest is not affected very much. It can also soothe the psyche of the weaker engine authors, by giving them wins against SF or LC0. ;-)
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
User avatar
towforce
Posts: 11751
Joined: Thu Mar 09, 2006 12:57 am
Location: Birmingham UK

Re: Stoofvlees II d2 is Killing Stockfish in TCEC

Post by towforce »

Uri Blass wrote: Tue Apr 30, 2024 8:48 pmI see no reason to believe that NN encoded many simple patterns and it seems to me that they encoded in the beginning mainly complex patterns.

The simpler patterns are in the endgame and not in the middlegame so
the fact that LC0 was playing beautiful chess in the middle game, but was relatively weaker in the endgame only support my theory that she was type of genius that understood complex things in the middle game that even top GM did not understand but did not understand simple things about the endgame that even weak humans player understand.

Of course now when NN became better they learned also a lot of simple stuff that human also understand and are superior relative to humans in all patterns.

1. The patterns in the endgame are deep, but the positions are simple enough for humans to not get confused by interacting patterns

2. Given the way NNs are trained, how are they going to find and encode deep patterns, other than "only occasionally" and "by accident"?

3. If (1) and (2) haven't convinced you, then here's the killer evidence: everyone had assumed that Go engines are now so good that, going forward, no human would ever challenge them. We were all wrong: it turns out that they cannot see things that are simple and obvious to humans - they didn't understand the concept of a group of stones - and it doesn't get any more basic than that in Go! Suddenly, it became clear why it's such a struggle to get computers to drive cars well - something that many stupid humans have no problem doing. They have terrible blind spots because they're encoding a large number of simple patterns, but missing most of the deeper ones. Armed with the knowledge of Go engine blind spots, an amateur player was able to consistently beat a Go program that had been thought to be out of reach of any human:

https://www.vice.com/en/article/v7v5xb/ ... mple-trick

I use NNs on a daily basis (I use LLMs multiple times every day). They are phenomenal, they are very useful, and they are a lot better than anything else we have today!

But they're stochastic parrots, and achieve their effects (and, in my life, their value) by encoding large numbers of surface patterns in the knowledge area: they're missing a lot of important deeper patterns, which could add a lot of value if they could be found, encoded, and included.
The simple reveals itself after the complex has been exhausted.
Uri Blass
Posts: 10424
Joined: Thu Mar 09, 2006 12:37 am
Location: Tel-Aviv Israel

Re: Stoofvlees II d2 is Killing Stockfish in TCEC

Post by Uri Blass »

towforce wrote: Wed May 01, 2024 12:32 am
Uri Blass wrote: Tue Apr 30, 2024 8:48 pmI see no reason to believe that NN encoded many simple patterns and it seems to me that they encoded in the beginning mainly complex patterns.

The simpler patterns are in the endgame and not in the middlegame so
the fact that LC0 was playing beautiful chess in the middle game, but was relatively weaker in the endgame only support my theory that she was type of genius that understood complex things in the middle game that even top GM did not understand but did not understand simple things about the endgame that even weak humans player understand.

Of course now when NN became better they learned also a lot of simple stuff that human also understand and are superior relative to humans in all patterns.

1. The patterns in the endgame are deep, but the positions are simple enough for humans to not get confused by interacting patterns

2. Given the way NNs are trained, how are they going to find and encode deep patterns, other than "only occasionally" and "by accident"?

3. If (1) and (2) haven't convinced you, then here's the killer evidence: everyone had assumed that Go engines are now so good that, going forward, no human would ever challenge them. We were all wrong: it turns out that they cannot see things that are simple and obvious to humans - they didn't understand the concept of a group of stones - and it doesn't get any more basic than that in Go! Suddenly, it became clear why it's such a struggle to get computers to drive cars well - something that many stupid humans have no problem doing. They have terrible blind spots because they're encoding a large number of simple patterns, but missing most of the deeper ones. Armed with the knowledge of Go engine blind spots, an amateur player was able to consistently beat a Go program that had been thought to be out of reach of any human:

https://www.vice.com/en/article/v7v5xb/ ... mple-trick

I use NNs on a daily basis (I use LLMs multiple times every day). They are phenomenal, they are very useful, and they are a lot better than anything else we have today!

But they're stochastic parrots, and achieve their effects (and, in my life, their value) by encoding large numbers of surface patterns in the knowledge area: they're missing a lot of important deeper patterns, which could add a lot of value if they could be found, encoded, and included.
I find that the NN simply miss simple tactical patterns also in the middle game.
White is probably a super GM level engine but it did not help it to avoid 14.Qd2

The tactics with 14...Rh1+ is something that I expect most humans players with fide rating 2000 to notice immediately because the mate is based on a known pattern.

White is already losing at that point but at least it could play 14.Re1 that does not get a forced mate when black has a decisive material advantage after 14...Rh1+ 15.Kxh1 Nf2+

With clear hash berserk13 needs less nodes to see Rh1+ but it still need many thousands of nodes.

[pgn][Event "Computer event"]
[Site "Somewhere on Earth"]
[Date "2024.05.01"]
[Time "05:02:33"]
[Round "12"]
[Board "21"]
[White "Berserk 13 65536 nodes"]
[Black "Stockfish 16.1"]
[Result "0-1"]
[Termination "mate"]
[ECO "C45"]
[Opening "Scotch"]
[TimeControl "nodes: 65536;60+1"]
[PlyCount "34"]

1.e4 {+0.5/13 39 44664 220/749/31} e5 {-0.3/33 8415 23258381} 2.Nf3 {+0.5/13 41 60329 206/764/30} Nc6 {-0.3/27 763 2503541}
3.d4 {+0.4/13 39 35169 183/787/30} exd4 {-0.1/21 1080 2871738} 4.Nxd4 {+0.4/13 44 35950 182/791/27} Nf6 {-0.1/22 931 2557183}
5.Nxc6 {+0.7/13 43 53028 269/717/14; C45: Scotch, Schmidt variation} bxc6 {-0.1/22 1178 3210276} 6.Bd3 {+0.2/12 42 53076 96/863/41} d5 {+0.0/23 1001 2842382}
7.e5 {+0.2/12 42 25414 99/867/34} Ng4 {+0.4/23 999 2787568} 8.O-O {+0.5/14 41 43197 197/790/13} Bc5 {+0.4/23 1875 5331561}
9.Bf4 {+0.0/11 38 61869 45/898/57} g5 {+0.4/25 2016 5786227} 10.Bg3 {-0.1/12 39 23717 40/906/54} h5 {+1.1/22 1268 3295191}
11.b4 {-0.6/12 39 27316 7/770/223} h4 {+2.5/21 1024 2678885} 12.bxc5 {-0.7/14 41 29632 5/741/254} hxg3 {+2.9/21 1297 3423031}
13.hxg3 {-0.7/15 40 54647 4/746/250} Ke7 {+3.9/22 1097 3934982} 14.Qd2 {+0.5/10 39 15915 170/824/6} Rh1+ {M+4/245 282 5450494}
15.Kxh1 {M-3/69 24 65402 0/0/1000} Qh8+ {M+3/245 35 62957} 16.Bh7 {M-2/200 37 22083 0/0/1000} Qxh7+ {M+2/245 22 31493}
17.Kg1 {M-1/200 33 601 0/0/1000} Qh2# {M+1/245 77 35310} 0-1

[/pgn]
User avatar
towforce
Posts: 11751
Joined: Thu Mar 09, 2006 12:57 am
Location: Birmingham UK

Re: Stoofvlees II d2 is Killing Stockfish in TCEC

Post by towforce »

Uri Blass wrote: Wed May 01, 2024 4:24 amI find that the NN simply miss simple tactical patterns also in the middle game.
White is probably a super GM level engine but it did not help it to avoid 14.Qd2

The tactics with 14...Rh1+ is something that I expect most humans players with fide rating 2000 to notice immediately because the mate is based on a known pattern.

White is already losing at that point but at least it could play 14.Re1 that does not get a forced mate when black has a decisive material advantage after 14...Rh1+ 15.Kxh1 Nf2+

With clear hash berserk13 needs less nodes to see Rh1+ but it still need many thousands of nodes.

[pgn][Event "Computer event"]
[Site "Somewhere on Earth"]
[Date "2024.05.01"]
[Time "05:02:33"]
[Round "12"]
[Board "21"]
[White "Berserk 13 65536 nodes"]
[Black "Stockfish 16.1"]
[Result "0-1"]
[Termination "mate"]
[ECO "C45"]
[Opening "Scotch"]
[TimeControl "nodes: 65536;60+1"]
[PlyCount "34"]

1.e4 {+0.5/13 39 44664 220/749/31} e5 {-0.3/33 8415 23258381} 2.Nf3 {+0.5/13 41 60329 206/764/30} Nc6 {-0.3/27 763 2503541}
3.d4 {+0.4/13 39 35169 183/787/30} exd4 {-0.1/21 1080 2871738} 4.Nxd4 {+0.4/13 44 35950 182/791/27} Nf6 {-0.1/22 931 2557183}
5.Nxc6 {+0.7/13 43 53028 269/717/14; C45: Scotch, Schmidt variation} bxc6 {-0.1/22 1178 3210276} 6.Bd3 {+0.2/12 42 53076 96/863/41} d5 {+0.0/23 1001 2842382}
7.e5 {+0.2/12 42 25414 99/867/34} Ng4 {+0.4/23 999 2787568} 8.O-O {+0.5/14 41 43197 197/790/13} Bc5 {+0.4/23 1875 5331561}
9.Bf4 {+0.0/11 38 61869 45/898/57} g5 {+0.4/25 2016 5786227} 10.Bg3 {-0.1/12 39 23717 40/906/54} h5 {+1.1/22 1268 3295191}
11.b4 {-0.6/12 39 27316 7/770/223} h4 {+2.5/21 1024 2678885} 12.bxc5 {-0.7/14 41 29632 5/741/254} hxg3 {+2.9/21 1297 3423031}
13.hxg3 {-0.7/15 40 54647 4/746/250} Ke7 {+3.9/22 1097 3934982} 14.Qd2 {+0.5/10 39 15915 170/824/6} Rh1+ {M+4/245 282 5450494}
15.Kxh1 {M-3/69 24 65402 0/0/1000} Qh8+ {M+3/245 35 62957} 16.Bh7 {M-2/200 37 22083 0/0/1000} Qxh7+ {M+2/245 22 31493}
17.Kg1 {M-1/200 33 601 0/0/1000} Qh2# {M+1/245 77 35310} 0-1

[/pgn]

Very nice example - thank you for that! 8-)
The simple reveals itself after the complex has been exhausted.
Ras
Posts: 2523
Joined: Tue Aug 30, 2016 8:19 pm
Full name: Rasmus Althoff

Re: Stoofvlees II d2 is Killing Stockfish in TCEC

Post by Ras »

towforce wrote: Tue Apr 30, 2024 1:14 pm2. To find out which chess engines are the best. For this, you really need bigger boards (12x12 might be a good starting point).
Translation: in order to find the best chess engine, we have to play some game other than chess. Doesn't make sense.
Rasmus Althoff
https://www.ct800.net