In anticipation of my RTX 2070 GPU :)

[shrapnel]

Then take it as a rumor.

--
Srdja

LOL, good reply to that eternal Killjoy.
Anyway, from what I saw earlier Lc0 seems to like nvidia much better than AMD, so it doesn't really matter what AMD does.
But of course, any rapid progress by AMD and nVidia will be forced to up their game, so all the better for us.

[Laskos]

In at most a month I will have RTX 2070 GPU working on my desktop PC. I am just waiting to have a more diversified offer, maybe better cooling and maybe better prices. RTX 2070 gives a huge boost, a factor of 4 (!!!) compared to my GTX 1060 6GB. NVidia by allowing fp16 support, just blew out even my most optimistic expectations for AI/Machine Learning. As I am impatient to see how Leela would fare in new conditions against my top CPU engines, with a bit old i7 4790 4-core 3.8 GHz CPU, which is not a slouch, I played in varied conditions Lc0 ID11261 against SF_dev, by far the best CPU engine in 4:1 time handicap conditions with my "old" GTX 1060. So, Lc0 is at 240''+ 4'' time control on GTX 1060, SF_dev is on 4 i7 cores at 60''+ 1''.
The comparison is not fair, emulated GPU is state-of-the-art latest generation, more expensive and consumes more power than my a bit old CPU. Next step in several months will be to have an 8 core CPU AMD machine.

Leela Ratio at equal time control would be about 0.5, at this 4x time control, Leela Ratio is equivalent to ~2.0.
Lc0 is v18.1, ID11261.

Strength-wise, from 4-mover opening book of Adam Hair of regular openings, SF_dev and Lc0 are fairly equal:

• Regular openings, 4-moves:
  Score of SF_dev_Syzygy vs lc0_v18.1_Syzygy: 11 - 10 - 29 [0.510] 50
  Elo difference: 6.95 +/- 63.00
  Finished match

Having in mind the sensitivity of Lc0 (And A0) to openings, I repeated the match with GM2600.pgn up to to 6 plies. GM2600 is a set of FIDE Elo 2600+ GM games, and is of pretty high quality, at least in the openings. Also, pretty representative.

• gm2600 6 plies
  Score of SF_dev_Syzygy vs lc0_v18_Syzygy: 12 - 20 - 68 [0.460] 100
  Elo difference: -27.85 +/- 38.48
  Finished match

Against human openings it performs better than against often pretty weird openings in 4-mover opening book.

And now I wanted to see the importance of the book is similar to 100 games AlphaZero match from the standard opening position.
Leela Ratio is the same in my games, 2-2.5.

From the 1 standard opening position:

• Standard opening position
  Score of SF_dev_Syzygy vs lc0_v18_Syzygy: 3 - 32 - 65 [0.355] 100
  Elo difference: -103.73 +/- 41.20
  Finished match

The result is pretty similar to A0 result against SF8, Lc0 is 100+ Elo points stronger than SF_dev.

But if SF uses an opening book, not even a large one, BookX.bin of 15 MB, the result is completely different, and SF actually beats Lc0:

• SF enabled with BookX
  Score of SF_dev_Syzygy vs lc0_v18_Syzygy: 19 - 14 - 67 [0.525] 100
  Elo difference: 17.39 +/- 39.38
  Finished match

120 Elo points difference in outcome using a book for SF.
Lc0 excels in opening play, and once it's neutralized by the opponent's book, its performance pales. So, the lack of the book for SF8 in AlphaZero match was an important issue.

[chrisw]

In at most a month I will have RTX 2070 GPU working on my desktop PC. I am just waiting to have a more diversified offer, maybe better cooling and maybe better prices. RTX 2070 gives a huge boost, a factor of 4 (!!!) compared to my GTX 1060 6GB. NVidia by allowing fp16 support, just blew out even my most optimistic expectations for AI/Machine Learning. As I am impatient to see how Leela would fare in new conditions against my top CPU engines, with a bit old i7 4790 4-core 3.8 GHz CPU, which is not a slouch, I played in varied conditions Lc0 ID11261 against SF_dev, by far the best CPU engine in 4:1 time handicap conditions with my "old" GTX 1060. So, Lc0 is at 240''+ 4'' time control on GTX 1060, SF_dev is on 4 i7 cores at 60''+ 1''.
The comparison is not fair, emulated GPU is state-of-the-art latest generation, more expensive and consumes more power than my a bit old CPU. Next step in several months will be to have an 8 core CPU AMD machine.

Leela Ratio at equal time control would be about 0.5, at this 4x time control, Leela Ratio is equivalent to ~2.0.
Lc0 is v18.1, ID11261.

Strength-wise, from 4-mover opening book of Adam Hair of regular openings, SF_dev and Lc0 are fairly equal:

• Regular openings, 4-moves:
  Score of SF_dev_Syzygy vs lc0_v18.1_Syzygy: 11 - 10 - 29 [0.510] 50
  Elo difference: 6.95 +/- 63.00
  Finished match

Having this equal result in the general play, I was curious how they compare in different conditions.


From the balanced middlegame positions, SF_dev is much stronger:

• Middlegame balanced
  Score of SF_dev_Syzygy vs lc0_v18.1_Syzygy: 15 - 1 - 34 [0.640] 50
  Elo difference: 99.95 +/- 51.76
  Finished match

The result is very skewed, and it means that if Lc0 doesn't get a serious advantage until the middlegame, it will either draw or lose, almost never win.


Another interesting aspect would be how Lc0 behaves in unfamiliar Queenless Chess:

[d]rnb1kbnr/pppppppp/8/8/8/8/PPPPPPPP/RNB1KBNR w KQkq - 0 1

• Queenless Chess:
  Score of SF_dev_Syzygy vs lc0_v18.1_Syzygy: 7 - 1 - 42 [0.560] 50
  Elo difference: 41.89 +/- 39.83
  Finished match

The result is again skewed, SF_dev winning decisively, with Lc0 having difficulty winning any game. Very drawish too, without Queens.


But not always unfamiliar positions disfavor Lc0. In this low draw-rate variant:

[d]3rqknr/4bpp1/4bpp1/1PP1npp1/1PPN1pp1/1PPB4/1PPB4/RNKQR3 w - - 0 1
The fight already starts in the opening and usually ends in midgame, and Lc0 beats heavily SF_dev:

• Variant
  Score of SF_dev_Syzygy vs lc0_v18.1_Syzygy: 14 - 32 - 4 [0.320] 50
  Elo difference: -130.94 +/- 102.19
  Finished match

Probably, Lc0 is the strongest engine in this variant (say, with Leela Ratio of about 1 and not too short TC).


And finally, and endgame variant, which is borderline Black Win / Draw, as expected SF_dev beats heavily Lc0

[d]1nn1k1n1/4p3/8/8/8/PPPPPPPP/8/4K3 w - - 0 1

• Disbalance:
  Score of SF_dev_Syzygy vs lc0_v18.1_Syzygy: 24 - 3 - 23 [0.710] 50
  Elo difference: 155.54 +/- 71.95
  Finished match

All in all, late midgames and endgames are the main weakness of Lc0, this can explain most of the results in under- or over- performance observed.

quite a bit of the 4x factor, 2070 to 1060 will be down to software taking advantage of "other" features of the 2070, like more GPU RAM and so on, whatever.
I went from 1060 to 2080ti about a week ago, and resnets, well, the ones I am playing with, not chess, get about a 3x-ish training speed up, same software, different hardware, and I've done no fancy stuff to the basic fit/learn algorithm (as yet). So, hardware alone 2080ti, is 3x, probably a bit less with 2070. Part of your 4x comes from LC0 optimisations(Crem), I would imagine.

[Javier Ros]

In at most a month I will have RTX 2070 GPU working on my desktop PC. I am just waiting to have a more diversified offer, maybe better cooling and maybe better prices. RTX 2070 gives a huge boost, a factor of 4 (!!!) compared to my GTX 1060 6GB. NVidia by allowing fp16 support, just blew out even my most optimistic expectations for AI/Machine Learning. As I am impatient to see how Leela would fare in new conditions against my top CPU engines, with a bit old i7 4790 4-core 3.8 GHz CPU, which is not a slouch, I played in varied conditions Lc0 ID11261 against SF_dev, by far the best CPU engine in 4:1 time handicap conditions with my "old" GTX 1060. So, Lc0 is at 240''+ 4'' time control on GTX 1060, SF_dev is on 4 i7 cores at 60''+ 1''.
The comparison is not fair, emulated GPU is state-of-the-art latest generation, more expensive and consumes more power than my a bit old CPU. Next step in several months will be to have an 8 core CPU AMD machine.

Leela Ratio at equal time control would be about 0.5, at this 4x time control, Leela Ratio is equivalent to ~2.0.
Lc0 is v18.1, ID11261.

Strength-wise, from 4-mover opening book of Adam Hair of regular openings, SF_dev and Lc0 are fairly equal:

• Regular openings, 4-moves:
  Score of SF_dev_Syzygy vs lc0_v18.1_Syzygy: 11 - 10 - 29 [0.510] 50
  Elo difference: 6.95 +/- 63.00
  Finished match

Having in mind the sensitivity of Lc0 (And A0) to openings, I repeated the match with GM2600.pgn up to to 6 plies. GM2600 is a set of FIDE Elo 2600+ GM games, and is of pretty high quality, at least in the openings. Also, pretty representative.

• gm2600 6 plies
  Score of SF_dev_Syzygy vs lc0_v18_Syzygy: 12 - 20 - 68 [0.460] 100
  Elo difference: -27.85 +/- 38.48
  Finished match

Against human openings it performs better than against often pretty weird openings in 4-mover opening book.

And now I wanted to see the importance of the book is similar to 100 games AlphaZero match from the standard opening position.
Leela Ratio is the same in my games, 2-2.5.

From the 1 standard opening position:

• Standard opening position
  Score of SF_dev_Syzygy vs lc0_v18_Syzygy: 3 - 32 - 65 [0.355] 100
  Elo difference: -103.73 +/- 41.20
  Finished match

The result is pretty similar to A0 result against SF8, Lc0 is 100+ Elo points stronger than SF_dev.

But if SF uses an opening book, not even a large one, BookX.bin of 15 MB, the result is completely different, and SF actually beats Lc0:

• SF enabled with BookX
  Score of SF_dev_Syzygy vs lc0_v18_Syzygy: 19 - 14 - 67 [0.525] 100
  Elo difference: 17.39 +/- 39.38
  Finished match

120 Elo points difference in outcome using a book for SF.
Lc0 excels in opening play, and once it's neutralized by the opponent's book, its performance pales. So, the lack of the book for SF8 in AlphaZero match was an important issue.

This post is important because it makes clear the strong dependence of lc0 with the opening book and I think it deserves a whole thread by itself. With all the comparisons done between lc0 and A0 I think it's the first time that someone does the test of the 100 games from the initial position, and it is the key test!!

In several of my tests I have detected this sensitivity of lc0 with respect to the openings set, see

http://talkchess.com/forum3/viewtopic.p ... 77#p774277


Final result with Noomen Sharp Gambits

Lc01711261 - Stockfish_9_x64_bmi2 : 41,5/120 9-46-65 (=0===00==000=010==01000000=1=0=======0=0=======0=0=0=0=000=1==10=0===0=1=0=0=====1==0001=01====0===0=00=0==0=0=0===000=0) 35%
Stockfish_9_x64_bmi2 - Lc01711261 : 78,5/120 46-9-65 (=1===11==111=101==10111111=0=1=======1=1=======1=1=1=1=111=0==01=1===1=0=1=1=====0==1110=10====1===1=11=1==1=1=1===111=1) 65%

This score for Stockfish at tactical positions is better than the obtained with more quiet positions, like Nunn´s ones 58% and 12 positions from AplhaZero-Stockfish test 60%.

[Laskos]

In at most a month I will have RTX 2070 GPU working on my desktop PC. I am just waiting to have a more diversified offer, maybe better cooling and maybe better prices. RTX 2070 gives a huge boost, a factor of 4 (!!!) compared to my GTX 1060 6GB. NVidia by allowing fp16 support, just blew out even my most optimistic expectations for AI/Machine Learning. As I am impatient to see how Leela would fare in new conditions against my top CPU engines, with a bit old i7 4790 4-core 3.8 GHz CPU, which is not a slouch, I played in varied conditions Lc0 ID11261 against SF_dev, by far the best CPU engine in 4:1 time handicap conditions with my "old" GTX 1060. So, Lc0 is at 240''+ 4'' time control on GTX 1060, SF_dev is on 4 i7 cores at 60''+ 1''.
The comparison is not fair, emulated GPU is state-of-the-art latest generation, more expensive and consumes more power than my a bit old CPU. Next step in several months will be to have an 8 core CPU AMD machine.

Leela Ratio at equal time control would be about 0.5, at this 4x time control, Leela Ratio is equivalent to ~2.0.
Lc0 is v18.1, ID11261.

Strength-wise, from 4-mover opening book of Adam Hair of regular openings, SF_dev and Lc0 are fairly equal:

• Regular openings, 4-moves:
  Score of SF_dev_Syzygy vs lc0_v18.1_Syzygy: 11 - 10 - 29 [0.510] 50
  Elo difference: 6.95 +/- 63.00
  Finished match

Having in mind the sensitivity of Lc0 (And A0) to openings, I repeated the match with GM2600.pgn up to to 6 plies. GM2600 is a set of FIDE Elo 2600+ GM games, and is of pretty high quality, at least in the openings. Also, pretty representative.

• gm2600 6 plies
  Score of SF_dev_Syzygy vs lc0_v18_Syzygy: 12 - 20 - 68 [0.460] 100
  Elo difference: -27.85 +/- 38.48
  Finished match

Against human openings it performs better than against often pretty weird openings in 4-mover opening book.

And now I wanted to see the importance of the book is similar to 100 games AlphaZero match from the standard opening position.
Leela Ratio is the same in my games, 2-2.5.

From the 1 standard opening position:

• Standard opening position
  Score of SF_dev_Syzygy vs lc0_v18_Syzygy: 3 - 32 - 65 [0.355] 100
  Elo difference: -103.73 +/- 41.20
  Finished match

The result is pretty similar to A0 result against SF8, Lc0 is 100+ Elo points stronger than SF_dev.

But if SF uses an opening book, not even a large one, BookX.bin of 15 MB, the result is completely different, and SF actually beats Lc0:

• SF enabled with BookX
  Score of SF_dev_Syzygy vs lc0_v18_Syzygy: 19 - 14 - 67 [0.525] 100
  Elo difference: 17.39 +/- 39.38
  Finished match

120 Elo points difference in outcome using a book for SF.
Lc0 excels in opening play, and once it's neutralized by the opponent's book, its performance pales. So, the lack of the book for SF8 in AlphaZero match was an important issue.

This post is important because it makes clear the strong dependence of lc0 with the opening book and I think it deserves a whole thread by itself. With all the comparisons done between lc0 and A0 I think it's the first time that someone does the test of the 100 games from the initial position, and it is the key test!!

In several of my tests I have detected this sensitivity of lc0 with respect to the openings set, see

http://talkchess.com/forum3/viewtopic.p ... 77#p774277


Final result with Noomen Sharp Gambits

Lc01711261 - Stockfish_9_x64_bmi2 : 41,5/120 9-46-65 (=0===00==000=010==01000000=1=0=======0=0=======0=0=0=0=000=1==10=0===0=1=0=0=====1==0001=01====0===0=00=0==0=0=0===000=0) 35%
Stockfish_9_x64_bmi2 - Lc01711261 : 78,5/120 46-9-65 (=1===11==111=101==10111111=0=1=======1=1=======1=1=1=1=111=0==01=1===1=0=1=1=====0==1110=10====1===1=11=1==1=1=1===111=1) 65%

This score for Stockfish at tactical positions is better than the obtained with more quiet positions, like Nunn´s ones 58% and 12 positions from AplhaZero-Stockfish test 60%.

I saw your results, and they corroborated with my results, so I decided to test a book for SF against 1 starting position. Thanks for your results, they added up to the general picture that Lc0 is sensitive to the opening phase of the game.