BT5-1024x15x32h-rpe-swa-2647500.pb.gz

[Jouni]

Hi Peter. You haven't tested new Stockfish? Latest version is Stockfish dev-20251203-c109a88e.

[peter]

Hi Peter. You haven't tested new Stockfish? Latest version is Stockfish dev-20251203-c109a88e.

Hi Jouni!
No, I haven't so far, my normal download- frequency of SF dev- versions is once per month, so I'll wait for next one or two prereleases.
And I lately had much to do with Lc0's BT5- net.

But just because you're asking and the question of the MultiPV- mode was mentioned above once in a while again too, not even Lc0's scores are statistically significant within error bar between MultiPV1 and MultiPV4, you saw?
Of course not to be compared to the differences seen thus with A-B-engines, yet there are differences and they get meaningful even statistically with big enough data. The easiest way to show such differences you have of course with single positions, let's take the old Salai- study once in while again:

https://www.yacpdb.org/#search/NEsxazFc ... LzEvMA==/1

Took the older PC with the 3070ti- GPU and the BT5- net because the differences in time to solution get bigger compared to the ones with the RTX 5070. Had already pasted MultiPV=10- output of 3 runs compared to 3 with sinlge primary here. Pity with the many MultiPV- lines, amount of allowed number of signs got overdrawn, so I'd have to edit much before pasting again. Anyhow time to solution with MultiPV=10 was 19, 19 and 20", MultiPV=1 was 39, 36, 37". Problem of single positions is always, how much difference in time to solution is of how much (even statistical) meaning? Error bar of pure spreading of tts of single runs don't mean a thing compared to error bars you get with suites or even at game playing of course.
Just_to_mention_regards

[lucario6607]

You do realize that multipv does nothing for leela besides making it output the moves to uci?

That was true as for about version 0.28 and the nets actual then, from later onwards there are differences in time to solution to be seen again and again, even if (no deterministic output of Lc0 at all, not with single CPU- thread neither, unlike as for A-B-engines) it's always a matter of enough data for statistically significant proof. But take a look at the list above, when I ad two runs of older Lc0- versions and nets MultiPV=1, nr32 and 42 were not in list before but are now:

Code: Select all

    Program                                    Elo   +/-  Matches  Score   Av.Op.   S.Pos.   MST1    MST2   RIndex
  
  8 Stockfish-251112-8t-MuPV4                : 3554    2  38124    57.9 %   3498   189/256    3.4s   10.4s   0.61
 10 Stockfish17.1-8t-MuPV4                   : 3550    2  38999    57.5 %   3498   195/256    4.3s   10.4s   0.55

 13 Lc0v0.32.0-3070ti-1740-MuPV4             : 3547    2  37773    56.9 %   3499   193/256    4.8s   11.0s   0.57

 17 Lc0v0.32.1-RTX5070-6147500PT-MuPV4       : 3543    2  38481    56.4 %   3499   187/256    4.1s   11.0s   0.55

 25 PlentyChess7.0.22-8t-MuPV4               : 3540    2  37664    55.8 %   3499   181/256    3.8s   11.4s   0.54
 26 Lc0v0.32.0-1740-MuPV4-RTX5070            : 3540    2  38376    55.9 %   3499   184/256    4.0s   11.3s   0.52

 32 Lc0v0.32.0-dev-1740-MuPV1                : 3539    2  37103    55.6 %   3499   180/256    4.0s   11.7s   0.57

 42 Lc0v0.32.0-4520-MuPV1                    : 3535    2  37370    55.0 %   3500   180/256    4.6s   12.1s   0.54
 
 48 Lc0v0.31.0onnx-RTX5070-BT5-3700M         : 3532    2  38003    54.8 %   3499   185/256    5.1s   12.0s   0.49

 71 Lc0v0.31.0-dag-onnx-3070ti-BT5-3700M     : 3519    2  37659    52.9 %   3499   175/256    5.4s   13.2s   0.44

Theses two runs both were with older one GPU (3070ti), the more versions and nets I added, the less I did let MuPV1 and MuPV4 run both to see direct comparison, because the differences weren't never ever big compared to A-B MultiPV1 and MultiPV4 at all and got even smaller with newer versions and nets again, now with RTX 5070 even more like that probably, so I could as well have MuPV1 runs for Lc0 only as well as MuPV4 runs only. The reason, I don't delete the runs of little interest is, EloStatTS gets lower error bar with each and every new run in same list, computing Elo and error for each and every old and new run position- and engine- wise again, and I have A-B-engines run with MultiPV4 (if engine supports that) ho-hum. especially SF- clones with internal MultiPV- mode profit much from best of their settings in suites and hardware- TC of that kind, so I like to have Lc0 compared that way now and then too, you see? It's just some kind of list- cosmetics

Full list is 173 runs big in meantime, if pasting in fora, I just copy the parts of interest and the error bars get lower with the bigger number of compared to each other runs, so what, regards

mcts already searches those moves, multipv does nothing but output the moves to uci.

[peter]

mcts already searches those moves, multipv does nothing but output the moves to uci.

That can be true for pure MCTS, Dragon's e.g. (not "pure" neither, "MCTS" anyhow is a matter of concrete algorithmic implementation of much differences between certain engines anyhow) really doesn't have much differences in time to solution (time to same move choice, time to same eval) with MultiPV, Lc0's PUCT is something different even more.
Just try a single position with meaningful time to solution- differences, to see these differences more clearly, take single best move positions of classical meaning, gave one example in posting above of your latest answer, a study Lc0 prevails now and then even compared to SF, depending on versions and settings of both, this one from L.Salai:

https://www.yacpdb.org/#search/NEsxazFc ... LzEvMA==/1

Pity (as for the estimation of the composition, even if not as for my personal pov), there's a major dual to 1.f4 with 1.Bd1, which only matters, if engine finds it (also with clearly winning eval) as a correct one solution of the test position before 1.f4, normally it's even harder for the engines to find and choose as best move then author's solution itself is.

As for this one position MultiPV=10 is necessary to get better time to solution, try Lc0, if you want to see results of your own, if you don't see the differences between MultiPV=10 and MultiPV=1, I'll copy- paste outpout of my installation. It's a matter of at least three runs each (MultiPV1 and -10), the differences should be big enough between single primary and MultiPV10 to get out of statistical spreading even with so few runs only. As I already wrote answering to Jouni above, with the 3070ti and the BT5-3700M- net I had 19, 19 and 20" to having 1.f4 up and stable in output with MultiPV=10 and 36, 37, 39" with MultiPV=1.

Tell me then, if you need to see output of my trials with BT5- net and the onnx- compile mentioned in this thread, in that case I'll paste such, it's just a matter of much editing of the many MultiPV- output- lines to get them into a sinlge posting.

[lucario6607]

mcts already searches those moves, multipv does nothing but output the moves to uci.

That can be true for pure MCTS, Dragon's e.g. (not "pure" neither, "MCTS" anyhow is a matter of concrete algorithmic implementation of much differences between certain engines anyhow) really doesn't have much differences in time to solution (time to same move choice, time to same eval) with MultiPV, Lc0's PUCT is something different even more.
Just try a single position with meaningful time to solution- differences, to see these differences more clearly, take single best move positions of classical meaning, gave one example in posting above of your latest answer, a study Lc0 prevails now and then even compared to SF, depending on versions and settings of both, this one from L.Salai:

https://www.yacpdb.org/#search/NEsxazFc ... LzEvMA==/1

Pity (as for the estimation of the composition, even if not as for my personal pov), there's a major dual to 1.f4 with 1.Bd1, which only matters, if engine finds it (also with clearly winning eval) as a correct one solution of the test position before 1.f4, normally it's even harder for the engines to find and choose as best move then author's solution itself is.

As for this one position MultiPV=10 is necessary to get better time to solution, try Lc0, if you want to see results of your own, if you don't see the differences between MultiPV=10 and MultiPV=1, I'll copy- paste outpout of my installation. It's a matter of at least three runs each (MultiPV1 and -10), the differences should be big enough between single primary and MultiPV10 to get out of statistical spreading even with so few runs only. As I already wrote answering to Jouni above, with the 3070ti and the BT5-3700M- net I had 19, 19 and 20" to having 1.f4 up and stable in output with MultiPV=10 and 36, 37, 39" with MultiPV=1.

Tell me then, if you need to see output of my trials with BT5- net and the onnx- compile mentioned in this thread, in that case I'll paste such, it's just a matter of much editing of the many MultiPV- output- lines to get them into a sinlge posting.

Yes i would be interested to see your results. You can look at the code, multipv just makes it output more moves to uci. Dragon mcts used history as policy so even that has a policy. It still searches those moves regardless and policy returns a score for every move possible even if you use such a high pv number that maybe some moves don’t get much visits.

[peter]

Yes i would be interested to see your results. You can look at the code, multipv just makes it output more moves to uci. Dragon mcts used history as policy so even that has a policy. It still searches those moves regardless and policy returns a score for every move possible even if you use such a high pv number that maybe some moves don’t get much visits.

I don't think it's a principal difference as for the number of MultiPV- lines, the position I chose of course has an advantage as example, because 1.f4 is by search of many engines ranked as nr. 10 only, so you see the difference to single primary not until MultiPV=10 and then the better for this, because time to depth is getting bigger, time to solution yet getting lower is easier to distinct from quiet positions without such single forced lines and without such big difference in eval. And why, if you see even Dragon's policy help now and then that way, why shouldn't get Lc0- search more visits of tricky to be found and then searched deeper moves as well sooner with MultiPV? And if thats so as for MultiPV=10, why not for MultiPV=2 as well?

So here we go as for output- examples to Salai- study, 3070ti GPU, BT5 3700000- net and onnx- compile from here:

viewtopic.php?p=986146#p986146

direct link:

https://www.mediafire.com/file/wgr724b0 ... 0.exe/file

To not overdraw the number of signs possible in one posting and yet have output- lines of best and lowest ones in MultiPV=10 I pasted first and last of the 10 with ...- lines in between for the lucidity. Each one single try starts with fen of position and engine- name, directly followed by the first one output with correct best move ranked nr.1. Of course I did let run at each trial some longer to see eval getting more clearly a surely winning one and ranking staying stable, but more then one MultiPV=10- output would have become too much signs and or editing,so I hope it's clear and proving well enough like this.

Code: Select all

4K1k1/8/1p5p/1Pp3b1/8/1P3P2/P1B2P2/8 w - -

Engine: Lc0 v0.31.0onnx (0 MB)
von The LCZero Authors.
...
11/33  0:19   +1.38    1.f4 Lxf4 2.Kd7 Ld2 3.Kc6 La5 4.b4 cxb4 
                       5.Lb3+ Kg7 6.Kd5 Kf6 7.Ke4 Kg6 8.Ke5 h5 
                       9.f4 h4 10.f5+ Kh7 11.f6 h3 12.Ke6 h2 
                       13.Lc2+ Kh6 14.f7 (100.913) 4 TB:5 

11/33  0:19   +1.00    1.a3 Kg7 2.Kd7 Ld2 3.f4 Kf6 4.Ld3 h5 
                       5.Le2 h4 6.Lg4 Lc3 7.Lh3 Ld2 8.Kc6 La5 
                       9.f5 Kf7 10.f4 Ke7 11.b4 Lxb4 12.Lf1 La5 (100.913) 4 TB:5 
...
...
...
...
...
...
11/34  0:19   +1.24    1.Lf5 h5 2.Kd7 Ld2 3.Kc6 La5 4.Lh3 Kf7 
                       5.a3 h4 6.f4 Ke7 7.Kb7 Kf7 8.Lg4 Kf6 
                       9.Ka6 Ke7 10.f5 Kf6 11.Lh3 Kf7 
                       12.Kb7 Kf6 (101.289) 4 TB:5 

11/34  0:19   +1.18    1.Ld1 Ld2 2.Kd7 h5 3.f4 h4 4.Lg4 Kg7 
                       5.Kc6 La5 6.a3 Kf6 7.Kb7 Kf7 8.Ka6 Ke7 
                       9.f5 Kf6 10.Lh3 Kf7 11.Kb7 Kf6 (101.289) 4 TB:5 
    
Second run:                           
                       4K1k1/8/1p5p/1Pp3b1/8/1P3P2/P1B2P2/8 w - -
                       
Engine: Lc0 v0.31.0onnx (0 MB)
von The LCZero Authors.

...

11/40  0:19   +1.49    1.f4 Lxf4 2.Kd7 Ld2 3.Kc6 La5 4.b4 cxb4 
                       5.Lb3+ Kg7 6.Kd5 Kf6 7.Ke4 Kg6 8.Ke5 h5 
                       9.f4 h4 10.f5+ Kh7 11.Ke6 h3 12.f6 h2 
                       13.Lc2+ Kh6 14.f7 (100.344) 4 TB:5 

11/40  0:19   +1.08    1.Ld3 Kg7 2.Kd7 Ld2 3.f4 Kf6 4.Kc6 La5 
                       5.a3 h5 6.Le2 h4 7.Lg4 Ke7 8.f5 Kf7 
                       9.f3 Ke7 10.f4 Kf6 11.Lh3 Kf7 12.f6 Kxf6 
                       13.b4 c4 14.bxa5 (100.344) 4 TB:5 
...
...
...
...
...
...
...
...
11/40  0:19   +1.04    1.Lb1 Kg7 2.Kd7 Ld2 3.f4 Kf6 4.Ld3 Le1 
                       5.Kc6 La5 6.a3 h5 7.Le2 h4 8.Lg4 Ke7 
                       9.f5 Kf7 10.f3 Ke7 11.f4 Kf6 12.Lh3 Kf7 
                       13.f6 Kxf6 14.b4 (100.344) 4 TB:5 

11/41  0:21   +1.23    1.Kd7 Ld2 2.Lf5 h5 3.Lh3 h4 4.Kc6 La5 
                       5.a3 Kg7 6.f4 Kf7 7.Lg4 Ke7 8.f5 Kf7 
                       9.f3 Ke7 10.f4 Kf6 11.Lh3 Kf7 12.f6 Kxf6 
                       13.b4 c4 14.bxa5 (107.663) 3 TB:5 
                       

Third run:

4K1k1/8/1p5p/1Pp3b1/8/1P3P2/P1B2P2/8 w - -

Engine: Lc0 v0.31.0onnx (0 MB)
von The LCZero Authors.

11/37  0:20   +1.63    1.f4 Lxf4 2.Kd7 Ld2 3.Kc6 La5 4.b4 cxb4 
                       5.Lb3+ Kg7 6.Kd5 Kf6 7.Ke4 Kg6 8.Ke5 h5 
                       9.f4 h4 10.f5+ Kh7 11.Ke6 h3 12.f6 h2 
                       13.Lc2+ Kh6 14.f7 (105.483) 4 TB:5 

11/37  0:20   +1.16    1.Ld1 Ld2 2.Kd7 h5 3.f4 h4 4.Lg4 Kg7 
                       5.Kc6 La5 6.a3 Kf6 7.f5 Kg5 8.Lh3 Kf6 
                       9.f4 Kf7 10.b4 Lxb4 11.Lf1 La5 
                       12.Kd5 (105.483) 4 TB:5 
...
...
...
...
...
...
11/38  0:21   +1.23    1.Kd7 Ld2 2.Lf5 h5 3.Kc6 La5 4.Lh3 h4 
                       5.a3 Kg7 6.f4 Kf6 7.Lg4 Ke7 8.f5 Kf6 
                       9.f4 Ke7 10.b4 cxb4 11.axb4 Lxb4 
                       12.Kxb6 Ld2 13.Kb7 (107.457) 4 TB:5 

11/38  0:21   +1.22    1.Lf5 h5 2.Kd7 Ld2 3.Kc6 La5 4.Lh3 Kg7 
                       5.a3 Kf6 6.f4 h4 7.Kc7 Kf7 8.Kc6 Kf6 
                       9.f5 Kf7 10.f4 Kf6 11.Lg4 Ke7 12.b4 cxb4 
                       13.axb4 Lxb4 14.Kxb6 (107.457) 4 TB:5 

So we had 19, 19 and 20" here, and then with MultPV=1: (here I pasted each time last one output- line with wrong move up followed by first one with correct one, to show with smallest amount of lines the least difference in eval, that the engine "gets" the main points of the forced win even with too low eval already, first few moves seem to be more or less proving, e.g 4.b4. Again of course I waited some longer at each trial to see eval going on climbing and best move staingy stable in output.

Code: Select all

4K1k1/8/1p5p/1Pp3b1/8/1P3P2/P1B2P2/8 w - -

Engine: Lc0 v0.31.0onnx (0 MB)
von The LCZero Authors.

...

16/61  0:38   +1.23    1.Kd7 Ld2 2.Lf5 h5 3.f4 Kf7 4.Kc6 La5 
                       5.Lh3 h4 6.a3 Kf6 7.Kb7 Ke7 8.Ka6 Kf6 
                       9.f5 Ke7 10.Kb7 Kf6 11.Kc6 Kf7 12.b4 Lxb4 
                       13.Lf1 La5 14.Kd5 (193.686) 3 TB:5 

16/61  0:39   +1.78    1.f4 Lxf4 2.Kd7 Ld2 3.Kc6 La5 4.b4 cxb4 
                       5.Lb3+ Kg7 6.Kd5 Kf6 7.Ke4 h5 8.Kf4 Kg6 
                       9.Ke5 h4 10.Kf4 Kh5 11.f3 h3 12.Kg3 Kg5 
                       13.Kxh3 Kf4 14.Kg2 (194.817) 3 TB:5 



4K1k1/8/1p5p/1Pp3b1/8/1P3P2/P1B2P2/8 w - -

Engine: Lc0 v0.31.0onnx (0 MB)
von The LCZero Authors.

...

15/62  0:33   +1.25    1.Kd7 Ld2 2.Lf5 h5 3.Lh3 Kg7 4.Kc6 La5 
                       5.f4 Kf6 6.Kb7 h4 7.Ka7 Kf7 8.a3 Kf6 
                       9.b4 Lxb4 10.Lf1 La5 (165.717) 3 TB:5 

15/62  0:36   +1.81    1.f4 Lxf4 2.Kd7 Ld2 3.Kc6 La5 4.b4 cxb4 
                       5.Lb3+ Kg7 6.Kd5 Kf6 7.Ke4 h5 8.Kf4 Kg6 
                       9.Ke5 h4 10.Kf4 Kh5 11.f3 h3 12.Kg3 Kg5 
                       13.Kxh3 Kf4 14.Kg2 (180.291) 3 TB:5 



4K1k1/8/1p5p/1Pp3b1/8/1P3P2/P1B2P2/8 w - -

Engine: Lc0 v0.31.0onnx (0 MB)
von The LCZero Authors.

..

15/64  0:36   +1.23    1.Kd7 Ld2 2.Lf5 h5 3.f4 Kf7 4.Lh3 h4 
                       5.Lg4 Kf6 6.Kc6 La5 7.a3 Ke7 8.Kb7 Kf7 
                       9.Ka6 Kf6 10.Lh3 Kf7 11.f5 Kf6 12.b4 Lxb4 
                       13.Lf1 (178.346) 3 TB:6 

15/64  0:37   +1.81    1.f4 Lxf4 2.Kd7 Ld2 3.Kc6 La5 4.b4 cxb4 
                       5.Lb3+ Kg7 6.Kd5 Kf6 7.Ke4 h5 8.Kf4 Kg6 
                       9.Ke5 h4 10.Kf4 Kh5 11.f3 h3 12.Kg3 Kg5 
                       13.Kxh3 Kf4 14.Kg2 (186.157) 3 TB:6 

So 39, 36 and 37".
Simplified not quite but almost twice as much average time to solution single primary compared to MultiPV=10, isn't it?

[Krzysztof Grzelak]

BT5- nets (that I know of) need special Lc0- compiles to be run. Github- 0.32.0 and 0.32.1 don't work, so far I didn't find any ready to use Cuda- compile to be downloaded for free neither, regards

And what program did you use to play the version onnx-trt?

[peter]

And what program did you use to play the version onnx-trt?

Read the postings starting at least from this one:

viewtopic.php?p=986140#p986140

You have to install TensorRT in any way, the one downloading the onnx- version of Lc0 from github like told in linked posting of Jason's (and ff) is one of these ways to get all files you need, regards

[Dann Corbit]

I tried the BT5 net on a difficult test set.
I have two 2080 Super cards with 8GB VRAM.
I have an AMD Ryzen Threadripper 3970X at 4GHz with 6 man TB files.
It did not perform as well as the smaller nets.
I analyzed at one minute per position for 120 positions.
Possibly a newer machine and newer GPU cards will do better.
The BT4 net solved 93/120
The BT5 net solved 78/120
Of course, this is a measure of tactical strength and valid only for my machine and the given test conditions.

[guinhopinda]

Hello All,

As the resident dummy, I have some questions:

1) Does compelling evidence exist that this BT5 is superior to BT4-1740? I've looked at the Leela Open Bench reports, but don't understand it.

2) If it's superior, in what way is it superior? As an ICCF GM, I seek analytical quality, rather than better speed chess results.

Thanks in advance,
-Steve-

Hi.

The simple answer is "BT5 is probably far superior in postal play." A detailed answer wouldn't fit here, but I recommend the book https://www.amazon.com.br/Xadrez-2022-M ... 6500377001 (it's in Portuguese, but there's an English version). As an ICCF GM, you can get free access through www.sigmasociety.net (see the "admission criteria" page).

I'll summarize the main points to consider:
1. For "normal" engines (pre-A0/Lc0), there's generally a gain of 42 absolute rating points for each doubling in processing speed (or doubling in reflection time with "ponder" disabled).
2. For Lc0, this is different and more like humans, with about 90 to 120 rating points for each doubling. This can vary greatly; in some cases it can be 20 or 30, due to the asymptotic limit determined by excess draws, or 200+ when the time is very short (or processing is very slow) and blunder filtering has a large weight.

This is the basis of the argument. Lc0 gains strength with increased analysis time compared to other engines. Furthermore, more complex networks feel this effect more strongly, while engines with simpler networks more closely resemble "traditional" programs.

However, there are other factors to consider. Training a complex network is much more time-consuming and takes longer to reach optimized parameter weights. This can take months or years, therefore BT5 networks may not yet be as well optimized as BT4 and other smaller networks. The BT5-1024x15x32h-rpe-swa-2647500.pb.gz network, for example, might be weaker than BT4 at almost all time controls, although this might change at postal time controls. The BT5-1024x15x32h-rpe-swa-3700000.pb.gz network is on another level and is almost certainly superior to BT4 at postal time controls.

This is a sufficient summary to clarify the basics, but there are many other subtleties to consider. If you want more details, I highly recommend the book I cited. There is another book that discusses SF and Lc0 training, volume 4 of “Guia dos apodícticos”.

Cheers!