Stockfish vs. Lc0: IMHO disappointing result for Lc0

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
supersharp77
Posts: 753
Joined: Sat Jul 05, 2014 5:54 am
Location: Southwest USA

Re: Stockfish vs. Lc0: IMHO disappointing result for Lc0

Post by supersharp77 » Mon Jul 29, 2019 4:27 am

Guenther wrote:
Sun Jul 28, 2019 7:20 am
mvanthoor wrote:
Sat Jul 27, 2019 8:06 pm
Today I ran a short 20-game match between Stockfish 10 and Lc0. Specs of the match:

Stockfish 10 x64 BMI2 on Intel i7-6700K, 4 threads, 8GB hashtable
Lc0 0.21.3, w42850 on GTX 1070. 4 threads, everything else default.
Syzygy 5 men tablebase, 8 move Performance.bin opening book.
Adjucation by GUI bo overwhelming material advantage or Syzygy when win/draw/loss in endgame.

...

In some games, Leela makes exceedingly weird moves, and lost game 1 in 21 moves because of a blunder

...
Are the games available somewhere and also the other 20 games mentioned later in this thread?
(If possible old pgn format w/o gimmicks just eval/depth time)
I would like to look at them.
Wow...Took a quick look and games seemed to me to be of a very low quality...many of the evaluations seemed to be not correct...(probably due to very fast time control) reviewed games pgn here..

http://s000.tinyupload.com/?file_id=027 ... 6221434746

Here's one of the games...let me know what you think of this "beauty" :D :wink:


jp
Posts: 820
Joined: Mon Apr 23, 2018 5:54 am

Re: Stockfish vs. Lc0: IMHO disappointing result for Lc0

Post by jp » Mon Jul 29, 2019 5:45 am

supersharp77 wrote:
Mon Jul 29, 2019 4:27 am
Here's one of the games...let me know what you think of this "beauty" :D :wink:
There are some TCEC games where Leela loses to SF in a similar style.

Fulvio
Posts: 155
Joined: Fri Aug 12, 2016 6:43 pm

Re: Stockfish vs. Lc0: IMHO disappointing result for Lc0

Post by Fulvio » Mon Jul 29, 2019 6:29 am

supersharp77 wrote:
Mon Jul 29, 2019 4:27 am
Here's one of the games...let me know what you think of this "beauty" :D :wink:
I don't understand the hate.
It's a beautiful game, original, interesting and with a spectacular ending.
Lc0 just blundered with 19.. c5 due to the slow GPU (it created the d6 weakness, which was exploited with the incredible maneuver to transfer the tower to h6 with Rb6 and Rxd6).

User avatar
Ovyron
Posts: 2715
Joined: Tue Jul 03, 2007 2:30 am

Re: Stockfish vs. Lc0: IMHO disappointing result for Lc0

Post by Ovyron » Mon Jul 29, 2019 1:35 pm

mvanthoor wrote:
Sun Jul 28, 2019 10:39 pm
Then I'll run an Lc0 vs. SF10 match with 40/4 for both of them
Good. Just remember that if you want to see Leela reach/surpass Stockfish level, you'd want to play ten times less games, but with ten times more the time on the clock.

Lest, you're just going to do all that to get another disappointing Leela result, because without playing any games, everyone knows it sucks at blitz (another way to not be disappointed is knowing that Leela is going to lose in advance, because of the time control, and nothing else).
Great spirits have always encountered violent opposition from mediocre minds.

Modern Times
Posts: 2410
Joined: Thu Jun 07, 2012 9:02 pm

Re: Stockfish vs. Lc0: IMHO disappointing result for Lc0

Post by Modern Times » Mon Jul 29, 2019 2:38 pm

Lc0 is not at its best at blitz, and I've certainly seen bad moves that it simply wouldn't have played with a longer time control or on a faster GPU. Still, that statement probably applies to some A/B engines as well. As long as the testing conditions are the same for each - and that is a big "if" given that hardware equivalence is in my opinion impossible given the radically different architectures of GPU and CPU - then the results are valid.

I do believe though that Lc0 benefits hugely from stronger hardware, so if you want it to play at an extremely high level (i.e. superiority over Stockfish) you need at least an RTX 2060 from what I've seen from various postings and results.
.

Opinions expressed here are my own, and not necessarily those of the CCRL Group.

User avatar
Ovyron
Posts: 2715
Joined: Tue Jul 03, 2007 2:30 am

Re: Stockfish vs. Lc0: IMHO disappointing result for Lc0

Post by Ovyron » Mon Jul 29, 2019 2:54 pm

Modern Times wrote:
Mon Jul 29, 2019 2:38 pm
As long as the testing conditions are the same for each - and that is a big "if" given that hardware equivalence is in my opinion impossible given the radically different architectures of GPU and CPU - then the results are valid.
Yes, but mvanthoor is going to spend a lot of time and effort showing how bad lc0 is at blitz, which we already know, so it'll just be time and effort wasted.

The hypothesis is that at some time control, slow enough, Leela catches Stockfish in strength (regardless of hardware!), and if you make it a bit slower, Leela passes Stockfish. But we don't know if this is true or what is this time control (relative to CCRL time control), so if one is going to test something, this might not be a waste of time.
Great spirits have always encountered violent opposition from mediocre minds.

dragontamer5788
Posts: 130
Joined: Thu Jun 06, 2019 6:05 pm
Full name: Percival Tiglao

Re: Stockfish vs. Lc0: IMHO disappointing result for Lc0

Post by dragontamer5788 » Mon Jul 29, 2019 3:10 pm

M ANSARI wrote:
Sun Jul 28, 2019 8:05 am
You have to realize that GPU graphic cards that have the ability to do AI are in the first generation. So if you look at CPU power, this would be sort of like running SF using a single core 386 CPU. Of course a GPU card like 2080Ti is expensive today, but so was a 386 when it first came out. My guess is that GPU's that can do AI will quickly get much more powerful and much cheaper. There is no doubt that AI will transform every thing in our life and maybe it will be a transformation similar to when humanity discovered electricity. Lc0 is only competitive once you have reasonably good hardware. I don't think you need a 2080Ti card for that and most likely the new 2070 Super cards are very competitive with SF. This hardware pricing of cards that can do AI will probably change exponentially with much more powerful cards coming out at a fraction of today's prices. Also Lc0 will probably patch many of its weaknesses (tactical and endgame weakness) via software … remember Lc0 is only a little over a year old.
The "Tensor Cores" of GPUs are an interesting trick, but the fundamental SIMD-architecture of GPUs is well-researched and well-discussed by the graphics community for the last 20 years. Modern GPUs represent the sum of decades of research and development.

Case in point: a 2080 Ti has 11 Trillion-Operations/second worth of compute on 616GB/s main-memory speed. True, the "Tensor-ops" allow 100+ Trillion "16-bit floating point Multiplications" (aka: neural network operations), but that's only useful when you have an algorithm dominated by neural-nets. And frankly, I'm not convinced that the whole tensor-op methodology is working out too well.

Take a nice CPU, like the AMD Ryzen 3950x: 16 cores (with 256-bit AVX2) x 4.7 GHz. AVX2 is 8x 32-bit operations per core x 16 cores x 4.7 GHz == 0.6 Trillion operations/second, on only 50GBps main-memory speed. Actually, most chess engines avoid AVX2, and instead stick with 64-bit operations. If you're using 64-bit bitboards on traditional 64-bit operations, your CPU-algorithm only has access to 0.075 Trillion operations/second.

In any case: a CPU is operating with 10% of the main-memory bandwidth and 1% of the raw compute power. The real questions people need to be asking themselves are:

1: Why are CPUs able to play chess so well, despite the hugely deficient compute and memory resources?
2: Why do you require so many Trillions-of-ops before Neural Nets become useful? Leela-Zero is on a machine capable of performing 100-Trillion operations / second. Shouldn't we be expecting it to perform better?
3: Are there other algorithms to be discovered that can take advantage of the CPU-algorithms, and port them to use the massively improved compute and memory available to GPUs?

-------

GPU-algorithms must take advantage of parallel compute resources. Its hard to think in parallel, especially if you've been doing sequential programming for years. But I analyze the parallel-algorithms in the CPU-world and they simply are insufficient for GPU translation: YBWC, ABDADA, etc. etc. All of these are designed for low-core count machines (maybe 20 or 50 cores), and will fall apart with 4000+ cores of a 2080 TI. Every thread will visit every node in YBWC or ABDADA. In most cases, a "visit" is pining the transposition table and sharing the work done from other threads, but you will run out of main-memory bandwidth very quickly when 4000+ cores are pinging the TT so hard.

Other GPU-programmers have proven that elements of a chess engine can be ported to a GPU. With over 20-Billion nodes/second perft on an ancient 780 Ti GPU, part of the GPU-programming problem has already been solved. The only remaining part is the search algorithm.

supersharp77
Posts: 753
Joined: Sat Jul 05, 2014 5:54 am
Location: Southwest USA

Re: Stockfish vs. Lc0: IMHO disappointing result for Lc0

Post by supersharp77 » Mon Jul 29, 2019 8:37 pm

supersharp77 wrote:
Mon Jul 29, 2019 4:27 am
Guenther wrote:
Sun Jul 28, 2019 7:20 am
mvanthoor wrote:
Sat Jul 27, 2019 8:06 pm
Today I ran a short 20-game match between Stockfish 10 and Lc0. Specs of the match:

Stockfish 10 x64 BMI2 on Intel i7-6700K, 4 threads, 8GB hashtable
Lc0 0.21.3, w42850 on GTX 1070. 4 threads, everything else default.
Syzygy 5 men tablebase, 8 move Performance.bin opening book.
Adjucation by GUI bo overwhelming material advantage or Syzygy when win/draw/loss in endgame.

...

In some games, Leela makes exceedingly weird moves, and lost game 1 in 21 moves because of a blunder

...
Are the games available somewhere and also the other 20 games mentioned later in this thread?
(If possible old pgn format w/o gimmicks just eval/depth time)
I would like to look at them.
Wow...Took a quick look and games seemed to me to be of a very low quality...many of the evaluations seemed to be not correct...(probably due to very fast time control) reviewed games pgn here..

http://s000.tinyupload.com/?file_id=027 ... 6221434746

Here's one of the games...let me know what you think of this "beauty" :D :wink:





Fulvio wrote:
Mon Jul 29, 2019 6:29 am
supersharp77 wrote:
Mon Jul 29, 2019 4:27 am
Here's one of the games...let me know what you think of this "beauty" :D :wink:
It's a *beautiful* game, *original*, *interesting* and with a *spectacular ending*.
Lc0 just blundered with 19.. c5 due to the slow GPU (it created the d6 weakness, which was exploited with the incredible maneuver to transfer the tower to h6 with Rb6 and Rxd6).
Er...Excuse me..First 10+ moves were right out of the opening book..You sound like the Guy on Youtube "Hyping LC0" every day with his blog posts! LMAO... :) :wink:

Post Reply