Future of computer chess: 2 top engines?

[Madeleine Birchfield]

Top 2? Hah, no. Its going to be

1. Stockfish
2. Komodo
3. Anyone in the top 10 who uses NNUE
3. Anyone in the top 10 who uses NNUE
3. Anyone in the top 10 who uses NNUE
4. Leela

Yeah, right now Stockfish and Leela are the only projects that truely have the resources to train these Networks.
But do you think Chess.com is okay with that? Igel author himself is renting out machines to do it.
Oh, and the cheaters will obfuscate Stockfish Networks

The great Irony is that the Leela fans will be the ones to force the issue, if it is ever forced

Shortly after Leela came out, the Komodo authors tried to partner up with somebody who works in Chess.com to make a Komodo NN (possibly using the same methods as Leela); that ended not happening because that person working at chess.com was extremely busy and didn't have the time. A few weeks ago, they tried reaching out to the original author of NNUE, Yu Nasu, to talk about possibly implementing NNUE in Komodo.

I also expect a new '[adjective] Fritz' neural net bundled with the Stockfish engine to be sold at Chessbase in the same way that Fat Fritz was done with Leela, and the ratings list test it and rank it very highly in the same way that they did with Leela Fat Fritz. As well as a Fritz 18 that happens to be a NNUE based engine.

[AndrewGrant]

Top 2? Hah, no. Its going to be

1. Stockfish
2. Komodo
3. Anyone in the top 10 who uses NNUE
3. Anyone in the top 10 who uses NNUE
3. Anyone in the top 10 who uses NNUE
4. Leela

Yeah, right now Stockfish and Leela are the only projects that truely have the resources to train these Networks.
But do you think Chess.com is okay with that? Igel author himself is renting out machines to do it.
Oh, and the cheaters will obfuscate Stockfish Networks

The great Irony is that the Leela fans will be the ones to force the issue, if it is ever forced

Shortly after Leela came out, the Komodo authors tried to partner up with somebody who works in Chess.com to make a Komodo NN (possibly using the same methods as Leela); that ended not happening because that person working at chess.com was extremely busy and didn't have the time. A few weeks ago, they tried reaching out to the original author of NNUE, Yu Nasu, to talk about possibly implementing NNUE in Komodo.

I also expect a new '[adjective] Fritz' neural net bundled with the Stockfish engine to be sold at Chessbase in the same way that Fat Fritz was done with Leela, and the ratings list test it and rank it very highly in the same way that they did with Leela Fat Fritz. As well as a Fritz 18 that happens to be a NNUE based engine.

Ah, well only a moron would buy and rank Fritz

[Jouni]

There was weird net test: https://tests.stockfishchess.org/tests/ ... b8ddf83ea9.

[Ozymandias]

Komodo 11.2 is only 26 Elo points behind version 14, at CCRL 40/15. It was released on July 17, 2017. SF had been the clear number one for over a year, by the time Lc0 matured, and now it is again; this last T60 run has been rather disappointing, not to mention incredibly long.

The main reason why Leela is somewhat competitive in TCEC is because Leela has oursourced net training to jhorthos and the J92 nets are better than the T60 nets. But even then it is weaker than Stockfish.

That, and because Lc0 scales better with HW. TCEC uses high end configurations for both GPU and CPU. But not even that + long time controls are enough to close the gap.

These statements are true for millionaires but not for ordinary people.
Which is to say, it is a lot cheaper to buy a super high end GPU than to buy a super high end CPU machine.
Of course, you may also have to buy a new power supply.
For example, a 2080 super can be bought for $760

and a 1200W power supply for around $40:

so around $800 gets you extremely high compute power for chess. To get equivalent compute power through a regular computer system would be more expensive.

In the case of TCEC, it doesn't make much sense talking prices, as they always overspend, but in this case, the GPUs they're using are more expensive.

[Raphexon]

Komodo 11.2 is only 26 Elo points behind version 14, at CCRL 40/15. It was released on July 17, 2017. SF had been the clear number one for over a year, by the time Lc0 matured, and now it is again; this last T60 run has been rather disappointing, not to mention incredibly long.

The main reason why Leela is somewhat competitive in TCEC is because Leela has oursourced net training to jhorthos and the J92 nets are better than the T60 nets. But even then it is weaker than Stockfish.

That, and because Lc0 scales better with HW. TCEC uses high end configurations for both GPU and CPU. But not even that + long time controls are enough to close the gap.

These statements are true for millionaires but not for ordinary people.
Which is to say, it is a lot cheaper to buy a super high end GPU than to buy a super high end CPU machine.
Of course, you may also have to buy a new power supply.
For example, a 2080 super can be bought for $760

and a 1200W power supply for around $40:

so around $800 gets you extremely high compute power for chess. To get equivalent compute power through a regular computer system would be more expensive.

You still need a CPU for Lc0.
SF currently wins the price/performance ratio by a clear margin.

SF on an i9-9900K is roughly equal to a 3090...
If you compare equal price: The 3080 is equal in price to the R9 5950X that's coming out. (And that one's much faster than the 9900K used by Ankan)
An RTX3090 is equal in price to a 24/32 core Threadripper.

Taken from Ankan from the Lc0 Discord:

Match: TCEC SF Dev, vs TCEC lc0 (J92-190)
LC0 version: 0.26.3 dev build
LC0 options: backend=cuda-fp16, minibatch-size=240, max-out-of-order-evals-factor=2.4, max-collision-events=800, max-prefetch=120, nncache=20000000, moves-left-max-effect=0.2,moves-left-threshold=0.0,moves-left-slope=0.004,smart-pruning-minimum-batches=320,moves-left-quadratic-factor=0.0,moves-left-scaled-factor=1.0,moves-left-constant-factor=0.0
Time control: 5m+5s
Hardware: RTX 3090 for lc0, core i9 9900K 16 threads for SF
Speed: start pos after 15 seconds: 30knps for lc0, 15 MNps for Stockfish.
Book: TCEC14 openings AND some CCC12 openings
Tablebases: none
Adjudication: none
Software: Arena
Comments: still some hope for lc0 at SUFI

Code: Select all

   # PLAYER                :  RATING  ERROR  POINTS  PLAYED   (%)  CFS(%)    W    D    L  D(%)
+  1 Lc0                   :       3     17   109.5     217  50.5      64   16  187   14  86.2
-  2 Stockfish_20092822    :       0   ----   107.5     217  49.5     ---   14  187   16  86.2

White advantage = 38.91 +/- 8.60
Draw rate (equal opponents) = 91.24 % +/- 2.83

[Jouni]

Pohl results SuFi for poor: Stockfish 200928 bmi2 vs Lc0 0.26.3rc2 J92-190: 300 (+ 68,=215,- 17), 58.5 % (+60 Elo).

[voffka]

Igel author himself is renting out machines to do it.

Yes, I am renting hardware to train Igel net, and the sad part is that I need to sacrifice my CCRL testing capacity as the training is so time (and money) consuming.

I wish Andrew we can re-purpose OpenBench nodes for this

[duncan]

I fear, that soon we have only 2 top engines. Stockfish + derivates and Lc0 + derivates. I hope to be wrong .

As long as those 2 engines give us 200 elo a year, I am not going to complain.

[mwyoung]

Komodo 11.2 is only 26 Elo points behind version 14, at CCRL 40/15. It was released on July 17, 2017. SF had been the clear number one for over a year, by the time Lc0 matured, and now it is again; this last T60 run has been rather disappointing, not to mention incredibly long.

The main reason why Leela is somewhat competitive in TCEC is because Leela has oursourced net training to jhorthos and the J92 nets are better than the T60 nets. But even then it is weaker than Stockfish.

That, and because Lc0 scales better with HW. TCEC uses high end configurations for both GPU and CPU. But not even that + long time controls are enough to close the gap.

These statements are true for millionaires but not for ordinary people.
Which is to say, it is a lot cheaper to buy a super high end GPU than to buy a super high end CPU machine.
Of course, you may also have to buy a new power supply.
For example, a 2080 super can be bought for $760

and a 1200W power supply for around $40:

so around $800 gets you extremely high compute power for chess. To get equivalent compute power through a regular computer system would be more expensive.

You still need a CPU for Lc0.
SF currently wins the price/performance ratio by a clear margin.

SF on an i9-9900K is roughly equal to a 3090...
If you compare equal price: The 3080 is equal in price to the R9 5950X that's coming out. (And that one's much faster than the 9900K used by Ankan)
An RTX3090 is equal in price to a 24/32 core Threadripper.

Taken from Ankan from the Lc0 Discord:

Match: TCEC SF Dev, vs TCEC lc0 (J92-190)
LC0 version: 0.26.3 dev build
LC0 options: backend=cuda-fp16, minibatch-size=240, max-out-of-order-evals-factor=2.4, max-collision-events=800, max-prefetch=120, nncache=20000000, moves-left-max-effect=0.2,moves-left-threshold=0.0,moves-left-slope=0.004,smart-pruning-minimum-batches=320,moves-left-quadratic-factor=0.0,moves-left-scaled-factor=1.0,moves-left-constant-factor=0.0
Time control: 5m+5s
Hardware: RTX 3090 for lc0, core i9 9900K 16 threads for SF
Speed: start pos after 15 seconds: 30knps for lc0, 15 MNps for Stockfish.
Book: TCEC14 openings AND some CCC12 openings
Tablebases: none
Adjudication: none
Software: Arena
Comments: still some hope for lc0 at SUFI

Code: Select all

   # PLAYER                :  RATING  ERROR  POINTS  PLAYED   (%)  CFS(%)    W    D    L  D(%)
+  1 Lc0                   :       3     17   109.5     217  50.5      64   16  187   14  86.2
-  2 Stockfish_20092822    :       0   ----   107.5     217  49.5     ---   14  187   16  86.2

White advantage = 38.91 +/- 8.60
Draw rate (equal opponents) = 91.24 % +/- 2.83

Yes, price per performance is clearly won by Stockfish NNUE I agree. On a 4 core system or a smartphone. And If this is all I had, Stockfish is the clearly winner.

The question is if price and performance is not the only consideration, is Stockfish NNUE clearly the best. And that is still up for debate. As you can stack at bunch of RTX 3090 GPU's. For example! And Stockfish as of today has shown itself lacking in good scaling.

The answer is what is best for what you can afford. If you can not get better then a Lc0 Ratio of greater the 1. I would go with Stockfish. This is testing as of today, and could change with updates of both engines.

All I can tell you is Stockfish 12 and above, is equal to a RTX 2080 TI with Lc0 and the best NN on a 2950x CPU playing with Stockfish. Right now today. with almost a 1 to 1 Lc0 ratio. There is very little difference.

[AndrewGrant]

And Stockfish as of today has shown itself lacking in good scaling.

That is an interesting take on a dominate TCEC SuFi victory, demonstrating that Leela connote scale on GPUs in the slightest.