Test position: Neural net blindness.

[OneTrickPony]

I've never seen what you describe. You should run Lc0 from a command line, to rule out GUI-related issues.

Here are five runs of the position in this thread, with Lc0 terminated after each run. No GUI was used. Default settings, except for Backend = cudnn-fp16.

It seems others are confirming unpredictability of the results. Still, I would like to run it from a command line. To ensure I don't make mistakes as I don't have experience doing so can you please paste exactly how you are running it?

[zullil]

I've never seen what you describe. You should run Lc0 from a command line, to rule out GUI-related issues.

Here are five runs of the position in this thread, with Lc0 terminated after each run. No GUI was used. Default settings, except for Backend = cudnn-fp16.

It seems others are confirming unpredictability of the results. Still, I would like to run it from a command line. To ensure I don't make mistakes as I don't have experience doing so can you please paste exactly how you are running it?

I have the weights file in the same directory as the Lc0 executable, so it will be auto-discovered. After changing to that directory, I simply invoke Lc0 from the command line, tell it to use UCI, set the desired backend, enter the fen, and tell Lc0 to search 2000000 nodes. Looks like this, using Linux:

Code: Select all

$ ./lc0
       _
|   _ | |
|_ |_ |_| v0.23.2+git.unknown built Jan 19 2020
uci
id name Lc0 v0.23.2+git.unknown
id author The LCZero Authors.
option name WeightsFile type string default <autodiscover>
option name Backend type combo default cudnn-auto var cudnn-auto var cudnn var cudnn-fp16 var random var check var roundrobin var multiplexing var demux
option name BackendOptions type string default 
option name Threads type spin default 2 min 1 max 128
option name NNCacheSize type spin default 200000 min 0 max 999999999
option name MinibatchSize type spin default 256 min 1 max 1024
option name MaxPrefetch type spin default 32 min 0 max 1024
option name LogitQ type check default false
option name CPuct type string default 3.000000
option name CPuctBase type string default 19652.000000
option name CPuctFactor type string default 2.000000
option name Temperature type string default 0.000000
option name TempDecayMoves type spin default 0 min 0 max 100
option name TempCutoffMove type spin default 0 min 0 max 1000
option name TempEndgame type string default 0.000000
option name TempValueCutoff type string default 100.000000
option name TempVisitOffset type string default 0.000000
option name DirichletNoise type check default false
option name VerboseMoveStats type check default false
option name FpuStrategy type combo default reduction var reduction var absolute
option name FpuValue type string default 1.200000
option name FpuStrategyAtRoot type combo default same var reduction var absolute var same
option name FpuValueAtRoot type string default 1.000000
option name CacheHistoryLength type spin default 0 min 0 max 7
option name PolicyTemperature type string default 2.200000
option name MaxCollisionEvents type spin default 32 min 1 max 1024
option name MaxCollisionVisits type spin default 9999 min 1 max 1000000
option name OutOfOrderEval type check default true
option name StickyEndgames type check default true
option name SyzygyFastPlay type check default true
option name MultiPV type spin default 1 min 1 max 500
option name PerPVCounters type check default false
option name ScoreType type combo default centipawn var centipawn var centipawn_2018 var win_percentage var Q
option name HistoryFill type combo default fen_only var no var fen_only var always
option name ShortSightedness type string default 0.000000
option name SyzygyPath type string default 
option name Ponder type check default true
option name UCI_Chess960 type check default false
option name UCI_ShowWDL type check default false
option name ConfigFile type string default lc0.config
option name KLDGainAverageInterval type spin default 100 min 1 max 10000000
option name MinimumKLDGainPerNode type string default 0.000000
option name SmartPruningFactor type string default 1.330000
option name RamLimitMb type spin default 0 min 0 max 100000000
option name MoveOverheadMs type spin default 200 min 0 max 100000000
option name Slowmover type string default 1.000000
option name ImmediateTimeUse type string default 1.000000
option name LogFile type string default 
uciok
setoption name backend value cudnn-fp16
position fen r7/5p1p/pR2rnp1/1p1q4/kP6/P1Q2PP1/2P4P/1K3B2 w - - 0 33
go nodes 2000000

[mwyoung]

Really nice, especially that it comes from a subvariation of a very well known human game and I am sure Kasparov would find 33.Rxe6 quickly.
One more time it looks like bug in search. I've already posted about it a few times on both Discord and Lc0 forum. There are positions where it sometimes find the solution in 3 seconds and sometimes doesn't find it even after very long time (like 10 minutes). If you run it several times with the newest T60 you will see that behavior: it's either there in like 5-10 seconds or never gets there.
Maybe it's a GUI issue or god knows what. I am running it in SCIDvsPC and usually the first run doesn't find it but then after killing the engine and restarting it it finds it instantly with +97% score. You can confirm in the task manager that it indeed kills Lc0 process in the meantime. Something is terribly wrong with the search implementation there.
It's similar with the famous Bg5 move of Alpha Zero. It sometimes finds it quickly and usually doesn't find it even after 30 minutes on 2080ti.

Very good! Yes I was testing SF, and Lc0 on one of my favorite games to test chess engine understanding. When I found this honker by Lc0! Claiming a draw in this variation. And this was not the only issue Lc0 had in this game, but it was one of the most glaring.

[pgn][Event "Hoogovens Group A"]
[Site "Wijk aan Zee NED"]
[Date "1999.01.20"]
[EventDate "1999.01.16"]
[Round "4"]
[Result "1-0"]
[White "Garry Kasparov"]
[Black "Veselin Topalov"]
[ECO "B07"]
[WhiteElo "2812"]
[BlackElo "2700"]
[PlyCount "87"]

1. e4 d6 2. d4 Nf6 3. Nc3 g6 4. Be3 Bg7 5. Qd2 c6 6. f3 b5
7. Nge2 Nbd7 8. Bh6 Bxh6 9. Qxh6 Bb7 10. a3 e5 11. O-O-O Qe7
12. Kb1 a6 13. Nc1 O-O-O 14. Nb3 exd4 15. Rxd4 c5 16. Rd1 Nb6
17. g3 Kb8 18. Na5 Ba8 19. Bh3 d5 20. Qf4+ Ka7 21. Rhe1 d4
22. Nd5 Nbxd5 23. exd5 Qd6 24. Rxd4 cxd4 25. Re7+ Kb6
26. Qxd4+ Kxa5 27. b4+ Ka4 28. Qc3 Qxd5 29. Ra7 Bb7 30. Rxb7
Qc4 31. Qxf6 Kxa3 32. Qxa6+ Kxb4 33. c3+ Kxc3 34. Qa1+ Kd2
35. Qb2+ Kd1 36. Bf1 Rd2 37. Rd7 Rxd7 38. Bxc4 bxc4 39. Qxh8
Rd3 40. Qa8 c3 41. Qa4+ Ke1 42. f4 f5 43. Kc1 Rd2 44. Qa7 1-0 [/pgn]

[mwyoung]

I filled an issue on Github about it:
https://github.com/LeelaChessZero/lc0/issues/1067

I've signaled it in the past and it seems to me it's not considered a serious problem by the devs. From my perspective it means running Lc0 with more than 1 thread just doesn't work and users who use it for analysis or position testing should be warned about random blunders occurring with more than one thread. I would never run any kind of analysis with 2+ threads if I knew about the extent of this behavior.

[d]r7/5p1p/pR2rnp1/1p1q4/kP6/P1Q2PP1/2P4P/1K3B2 w - - 0 1


One thread: Lc0 + T62255 - default

New game Line
r7/5p1p/pR2rnp1/1p1q4/kP6/P1Q2PP1/2P4P/1K3B2 w - - 0 1

Analysis by Lc0 v0.23.2+git.c8d9095:

1.Rxe6 fxe6 2.Kb2 Qe5 3.Qxe5 Nd5 4.Qxe6 h6 5.Qxd5 Rh8 6.Bxb5+ axb5 7.Qb3#
White is clearly winning: +- (6.31) Depth: 24/47 00:02:19 3242kN, tb=283
(, 09.02.2020)

Two threads: Lc0 + T62255 - default

New game Line
r7/5p1p/pR2rnp1/1p1q4/kP6/P1Q2PP1/2P4P/1K3B2 w - - 0 1

Analysis by Lc0 v0.23.2+git.c8d9095:

1.Kb2 Qe5 2.Qxe5 Rxe5 3.Rxf6 Ra7 4.Rd6 Re3 5.c3 Rxf3 6.Be2 Rf2 7.Rd2 Rxe2 8.Rxe2 Rd7 9.c4 a5 10.bxa5 b4 11.axb4 Kxb4 12.Rc2 Ra7 13.c5 Rxa5 14.c6 Ra8 15.Kc1 Kb5 16.Kd2 Rc8 17.c7 Kb6 18.Ke3 Rxc7 19.Rf2 Re7+ 20.Kf3 Re5 21.Rb2+ Rb5 22.Rd2 Rf5+
The position is equal: = (0.03) Depth: 30/55 00:09:46 14673kN, tb=26436
(, 09.02.2020)

"Houston, we've had a problem."

[Ovyron]

That's probably the worst multicore implementation in the history of computer chess.

[mmt]

"Houston, we've had a problem."

Confirmed on my 2 GPU system! Each run is a little different. I think what's going on is that LC0 with default settings latches onto the best line it sees too early and other possibilities barely get any nodes. I've seen it in my match - it would've chosen an inferior move if I didn't make it look at the better one.

That's probably the worst multicore implementation in the history of computer chess.

It does not scale well.

[Ovyron]

But even if it scaled well, what's the point if it's never going to check the best move? (while single thread would check and find it always - is not knowing if it's going to happen on the current position worth using more than 1 thread?)

[mmt]

But even if it scaled well, what's the point if it's never going to check the best move? (while single thread would check and find it always - is not knowing if it's going to happen on the current position worth using more than 1 thread?)

It does find it a bit later (it's quicker with higher cpuct settings). I'll run a test with different cpuct settings at longer time controls later.

[mwyoung]

But even if it scaled well, what's the point if it's never going to check the best move? (while single thread would check and find it always - is not knowing if it's going to happen on the current position worth using more than 1 thread?)

It does find it a bit later (it's quicker with higher cpuct settings). I'll run a test with different cpuct settings at longer time controls later.

Try this setting. Temp policy = 1.8, and FPU value = .1. All else default. It will find it quickly. For what it is worth.

[corres]

...
Of course the net is not perfectly yet.
It needs much more training.

Any nets and any AB engines will perfect never.
If somebody wants perfect play he should wait for 32 men table bases...
If an engine finds the most best move during a very short thinking time it is only a lucky case - especially if the engine is an old one.