Future of computer chess: 2 top engines?

Discussion of anything and everything relating to chess playing software and machines.

Moderators: Harvey Williamson, Dann Corbit, hgm

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Madeleine Birchfield
Posts: 185
Joined: Tue Sep 29, 2020 2:29 pm
Location: Dublin, Ireland
Full name: Madeleine Birchfield

Re: Future of computer chess: 2 top engines?

Post by Madeleine Birchfield » Tue Oct 13, 2020 6:30 am

AndrewGrant wrote:
Tue Oct 13, 2020 5:38 am
Top 2? Hah, no. Its going to be

1. Stockfish
2. Komodo
3. Anyone in the top 10 who uses NNUE
3. Anyone in the top 10 who uses NNUE
3. Anyone in the top 10 who uses NNUE
4. Leela

Yeah, right now Stockfish and Leela are the only projects that truely have the resources to train these Networks.
But do you think Chess.com is okay with that? Igel author himself is renting out machines to do it.
Oh, and the cheaters will obfuscate Stockfish Networks :)

The great Irony is that the Leela fans will be the ones to force the issue, if it is ever forced ;)
Shortly after Leela came out, the Komodo authors tried to partner up with somebody who works in Chess.com to make a Komodo NN (possibly using the same methods as Leela); that ended not happening because that person working at chess.com was extremely busy and didn't have the time. A few weeks ago, they tried reaching out to the original author of NNUE, Yu Nasu, to talk about possibly implementing NNUE in Komodo.

I also expect a new '[adjective] Fritz' neural net bundled with the Stockfish engine to be sold at Chessbase in the same way that Fat Fritz was done with Leela, and the ratings list test it and rank it very highly in the same way that they did with Leela Fat Fritz. As well as a Fritz 18 that happens to be a NNUE based engine.

AndrewGrant
Posts: 871
Joined: Tue Apr 19, 2016 4:08 am
Location: U.S.A
Full name: Andrew Grant
Contact:

Re: Future of computer chess: 2 top engines?

Post by AndrewGrant » Tue Oct 13, 2020 6:56 am

Madeleine Birchfield wrote:
Tue Oct 13, 2020 6:30 am
AndrewGrant wrote:
Tue Oct 13, 2020 5:38 am
Top 2? Hah, no. Its going to be

1. Stockfish
2. Komodo
3. Anyone in the top 10 who uses NNUE
3. Anyone in the top 10 who uses NNUE
3. Anyone in the top 10 who uses NNUE
4. Leela

Yeah, right now Stockfish and Leela are the only projects that truely have the resources to train these Networks.
But do you think Chess.com is okay with that? Igel author himself is renting out machines to do it.
Oh, and the cheaters will obfuscate Stockfish Networks :)

The great Irony is that the Leela fans will be the ones to force the issue, if it is ever forced ;)
Shortly after Leela came out, the Komodo authors tried to partner up with somebody who works in Chess.com to make a Komodo NN (possibly using the same methods as Leela); that ended not happening because that person working at chess.com was extremely busy and didn't have the time. A few weeks ago, they tried reaching out to the original author of NNUE, Yu Nasu, to talk about possibly implementing NNUE in Komodo.

I also expect a new '[adjective] Fritz' neural net bundled with the Stockfish engine to be sold at Chessbase in the same way that Fat Fritz was done with Leela, and the ratings list test it and rank it very highly in the same way that they did with Leela Fat Fritz. As well as a Fritz 18 that happens to be a NNUE based engine.
Ah, well only a moron would buy and rank Fritz :)

Jouni
Posts: 2241
Joined: Wed Mar 08, 2006 7:15 pm

Re: Future of computer chess: 2 top engines?

Post by Jouni » Tue Oct 13, 2020 7:17 am

Jouni

User avatar
Ozymandias
Posts: 1243
Joined: Sun Oct 25, 2009 12:30 am

Re: Future of computer chess: 2 top engines?

Post by Ozymandias » Tue Oct 13, 2020 8:59 am

Dann Corbit wrote:
Mon Oct 12, 2020 10:17 pm
Ozymandias wrote:
Mon Oct 12, 2020 6:04 pm
Madeleine Birchfield wrote:
Mon Oct 12, 2020 5:42 pm
Ozymandias wrote:
Mon Oct 12, 2020 4:14 pm
Komodo 11.2 is only 26 Elo points behind version 14, at CCRL 40/15. It was released on July 17, 2017. SF had been the clear number one for over a year, by the time Lc0 matured, and now it is again; this last T60 run has been rather disappointing, not to mention incredibly long.
The main reason why Leela is somewhat competitive in TCEC is because Leela has oursourced net training to jhorthos and the J92 nets are better than the T60 nets. But even then it is weaker than Stockfish.
That, and because Lc0 scales better with HW. TCEC uses high end configurations for both GPU and CPU. But not even that + long time controls are enough to close the gap.
These statements are true for millionaires but not for ordinary people.
Which is to say, it is a lot cheaper to buy a super high end GPU than to buy a super high end CPU machine.
Of course, you may also have to buy a new power supply.
For example, a 2080 super can be bought for $760
https://www.amazon.com/Waterforce-Pre-I ... cs&sr=1-20
and a 1200W power supply for around $40:
https://www.amazon.com/437572-B21-1200W ... 25&sr=1-52
so around $800 gets you extremely high compute power for chess. To get equivalent compute power through a regular computer system would be more expensive.
In the case of TCEC, it doesn't make much sense talking prices, as they always overspend, but in this case, the GPUs they're using are more expensive.

Raphexon
Posts: 341
Joined: Sun Mar 17, 2019 11:00 am
Full name: Henk Drost

Re: Future of computer chess: 2 top engines?

Post by Raphexon » Tue Oct 13, 2020 9:53 am

Dann Corbit wrote:
Mon Oct 12, 2020 10:17 pm
Ozymandias wrote:
Mon Oct 12, 2020 6:04 pm
Madeleine Birchfield wrote:
Mon Oct 12, 2020 5:42 pm
Ozymandias wrote:
Mon Oct 12, 2020 4:14 pm
Komodo 11.2 is only 26 Elo points behind version 14, at CCRL 40/15. It was released on July 17, 2017. SF had been the clear number one for over a year, by the time Lc0 matured, and now it is again; this last T60 run has been rather disappointing, not to mention incredibly long.
The main reason why Leela is somewhat competitive in TCEC is because Leela has oursourced net training to jhorthos and the J92 nets are better than the T60 nets. But even then it is weaker than Stockfish.
That, and because Lc0 scales better with HW. TCEC uses high end configurations for both GPU and CPU. But not even that + long time controls are enough to close the gap.
These statements are true for millionaires but not for ordinary people.
Which is to say, it is a lot cheaper to buy a super high end GPU than to buy a super high end CPU machine.
Of course, you may also have to buy a new power supply.
For example, a 2080 super can be bought for $760
https://www.amazon.com/Waterforce-Pre-I ... cs&sr=1-20
and a 1200W power supply for around $40:
https://www.amazon.com/437572-B21-1200W ... 25&sr=1-52
so around $800 gets you extremely high compute power for chess. To get equivalent compute power through a regular computer system would be more expensive.
You still need a CPU for Lc0.
SF currently wins the price/performance ratio by a clear margin.

SF on an i9-9900K is roughly equal to a 3090...
If you compare equal price: The 3080 is equal in price to the R9 5950X that's coming out. (And that one's much faster than the 9900K used by Ankan)
An RTX3090 is equal in price to a 24/32 core Threadripper.

Taken from Ankan from the Lc0 Discord:
Match: TCEC SF Dev, vs TCEC lc0 (J92-190)
LC0 version: 0.26.3 dev build
LC0 options: backend=cuda-fp16, minibatch-size=240, max-out-of-order-evals-factor=2.4, max-collision-events=800, max-prefetch=120, nncache=20000000, moves-left-max-effect=0.2,moves-left-threshold=0.0,moves-left-slope=0.004,smart-pruning-minimum-batches=320,moves-left-quadratic-factor=0.0,moves-left-scaled-factor=1.0,moves-left-constant-factor=0.0
Time control: 5m+5s
Hardware: RTX 3090 for lc0, core i9 9900K 16 threads for SF
Speed: start pos after 15 seconds: 30knps for lc0, 15 MNps for Stockfish.
Book: TCEC14 openings AND some CCC12 openings
Tablebases: none
Adjudication: none
Software: Arena
Comments: still some hope for lc0 at SUFI

Code: Select all

   # PLAYER                :  RATING  ERROR  POINTS  PLAYED   (%)  CFS(%)    W    D    L  D(%)
+  1 Lc0                   :       3     17   109.5     217  50.5      64   16  187   14  86.2
-  2 Stockfish_20092822    :       0   ----   107.5     217  49.5     ---   14  187   16  86.2

White advantage = 38.91 +/- 8.60
Draw rate (equal opponents) = 91.24 % +/- 2.83

Jouni
Posts: 2241
Joined: Wed Mar 08, 2006 7:15 pm

Re: Future of computer chess: 2 top engines?

Post by Jouni » Tue Oct 13, 2020 11:35 am

Pohl results SuFi for poor: Stockfish 200928 bmi2 vs Lc0 0.26.3rc2 J92-190: 300 (+ 68,=215,- 17), 58.5 % (+60 Elo).
Jouni

voffka
Posts: 166
Joined: Sat Jun 30, 2018 8:58 pm
Location: Ukraine
Full name: Volodymyr Shcherbyna
Contact:

Re: Future of computer chess: 2 top engines?

Post by voffka » Tue Oct 13, 2020 10:03 pm

AndrewGrant wrote:
Tue Oct 13, 2020 5:38 am
Igel author himself is renting out machines to do it.
Yes, I am renting hardware to train Igel net, and the sad part is that I need to sacrifice my CCRL testing capacity as the training is so time (and money) consuming.

I wish Andrew we can re-purpose OpenBench nodes for this :)

duncan
Posts: 12029
Joined: Mon Jul 07, 2008 8:50 pm

Re: Future of computer chess: 2 top engines?

Post by duncan » Tue Oct 13, 2020 10:48 pm

Jouni wrote:
Mon Oct 12, 2020 11:40 am
I fear, that soon we have only 2 top engines. Stockfish + derivates and Lc0 + derivates. I hope to be wrong :) .
As long as those 2 engines give us 200 elo a year, I am not going to complain. :)

mwyoung
Posts: 2537
Joined: Wed May 12, 2010 8:00 pm

Re: Future of computer chess: 2 top engines?

Post by mwyoung » Tue Oct 13, 2020 11:00 pm

Raphexon wrote:
Tue Oct 13, 2020 9:53 am
Dann Corbit wrote:
Mon Oct 12, 2020 10:17 pm
Ozymandias wrote:
Mon Oct 12, 2020 6:04 pm
Madeleine Birchfield wrote:
Mon Oct 12, 2020 5:42 pm
Ozymandias wrote:
Mon Oct 12, 2020 4:14 pm
Komodo 11.2 is only 26 Elo points behind version 14, at CCRL 40/15. It was released on July 17, 2017. SF had been the clear number one for over a year, by the time Lc0 matured, and now it is again; this last T60 run has been rather disappointing, not to mention incredibly long.
The main reason why Leela is somewhat competitive in TCEC is because Leela has oursourced net training to jhorthos and the J92 nets are better than the T60 nets. But even then it is weaker than Stockfish.
That, and because Lc0 scales better with HW. TCEC uses high end configurations for both GPU and CPU. But not even that + long time controls are enough to close the gap.
These statements are true for millionaires but not for ordinary people.
Which is to say, it is a lot cheaper to buy a super high end GPU than to buy a super high end CPU machine.
Of course, you may also have to buy a new power supply.
For example, a 2080 super can be bought for $760
https://www.amazon.com/Waterforce-Pre-I ... cs&sr=1-20
and a 1200W power supply for around $40:
https://www.amazon.com/437572-B21-1200W ... 25&sr=1-52
so around $800 gets you extremely high compute power for chess. To get equivalent compute power through a regular computer system would be more expensive.
You still need a CPU for Lc0.
SF currently wins the price/performance ratio by a clear margin.

SF on an i9-9900K is roughly equal to a 3090...
If you compare equal price: The 3080 is equal in price to the R9 5950X that's coming out. (And that one's much faster than the 9900K used by Ankan)
An RTX3090 is equal in price to a 24/32 core Threadripper.

Taken from Ankan from the Lc0 Discord:
Match: TCEC SF Dev, vs TCEC lc0 (J92-190)
LC0 version: 0.26.3 dev build
LC0 options: backend=cuda-fp16, minibatch-size=240, max-out-of-order-evals-factor=2.4, max-collision-events=800, max-prefetch=120, nncache=20000000, moves-left-max-effect=0.2,moves-left-threshold=0.0,moves-left-slope=0.004,smart-pruning-minimum-batches=320,moves-left-quadratic-factor=0.0,moves-left-scaled-factor=1.0,moves-left-constant-factor=0.0
Time control: 5m+5s
Hardware: RTX 3090 for lc0, core i9 9900K 16 threads for SF
Speed: start pos after 15 seconds: 30knps for lc0, 15 MNps for Stockfish.
Book: TCEC14 openings AND some CCC12 openings
Tablebases: none
Adjudication: none
Software: Arena
Comments: still some hope for lc0 at SUFI

Code: Select all

   # PLAYER                :  RATING  ERROR  POINTS  PLAYED   (%)  CFS(%)    W    D    L  D(%)
+  1 Lc0                   :       3     17   109.5     217  50.5      64   16  187   14  86.2
-  2 Stockfish_20092822    :       0   ----   107.5     217  49.5     ---   14  187   16  86.2

White advantage = 38.91 +/- 8.60
Draw rate (equal opponents) = 91.24 % +/- 2.83
Yes, price per performance is clearly won by Stockfish NNUE I agree. On a 4 core system or a smartphone. And If this is all I had, Stockfish is the clearly winner.

The question is if price and performance is not the only consideration, is Stockfish NNUE clearly the best. And that is still up for debate. As you can stack at bunch of RTX 3090 GPU's. For example! And Stockfish as of today has shown itself lacking in good scaling.

The answer is what is best for what you can afford. If you can not get better then a Lc0 Ratio of greater the 1. I would go with Stockfish. This is testing as of today, and could change with updates of both engines.

All I can tell you is Stockfish 12 and above, is equal to a RTX 2080 TI with Lc0 and the best NN on a 2950x CPU playing with Stockfish. Right now today. with almost a 1 to 1 Lc0 ratio. There is very little difference.
Professing themselves to be wise, they became fools,
take on me. Foes 0.

AndrewGrant
Posts: 871
Joined: Tue Apr 19, 2016 4:08 am
Location: U.S.A
Full name: Andrew Grant
Contact:

Re: Future of computer chess: 2 top engines?

Post by AndrewGrant » Wed Oct 14, 2020 2:23 am

mwyoung wrote:
Tue Oct 13, 2020 11:00 pm
And Stockfish as of today has shown itself lacking in good scaling.
That is an interesting take on a dominate TCEC SuFi victory, demonstrating that Leela connote scale on GPUs in the slightest.

Post Reply