Lc0 question

[Modern Times]

OTH LC0 is very neglected in the 40/15 rating list anyways,
only cpu versions are tested there since ever. Not a single test for a version with a GPU and most of them with
not much games, so it seems no one cared for from the LC0 team, or it went unnoticed.

The Lc0 team simply don't care what CCRL do, and that is fine. They are only interested in the likes of TCEC and CCC (that is the impression I got).

Testing GPU engines is relatively easy at blitz, but totally painful at longer time controls as you can't use concurrency.

[Milos]

Testing GPU engines is relatively easy at blitz, but totally painful at longer time controls as you can't use concurrency.

What does that even mean?

[Nay Lin Tun]

For my gambit tournament (see elsewhere) I want to include Lc0-v27. Hardware RTX 1060 6Gb.

Time control one : 40moves in 2 minutes --> which net?
Time control two : 40moves in 15 minutes --> which net?

I am guessing his hardware is 1060 GTX ( RTX series start with 2060).

Assuming 1060 GTX, slow old card, latest T70 series should be the best.

[Rebel]

Tested v25.1, same bug.

The tc=60+1 format (game in 60 seconds plus 1 second Fischer bonus) does normal.

It's the way many engines test nowadays.

[Guenther]

Tested v25.1, same bug.

The tc=60+1 format (game in 60 seconds plus 1 second Fischer bonus) does normal.

It's the way many engines test nowadays.

I am getting old, I already noticed this in early 2019, but strangely I never reported it here.
The quirky time management for mps seems to be introduced since version 0.20.
I even started to conduct a test about how much LC0 loses in 40/240 vs 240+4 games vs. a variety of opponents in 2019-02-until 2019-04


In this sheet I demonstrated with stats and graphs how that time management led to a loss in a CCRL 40/40 game played 2019-02-08


I have added an entry in my details pages for LC0 now (peculiarities + bugs column) so the info will not be 'lost' again.

[Rebel]

Code: Select all

Engine         Depth       Time   Games     Moves  Average Forfeit  Book Depth     MIDG   EARLY    ENDG    LATE
Ethereal_12.75 25.85    4:41:48      60      5427    3.12     0      197  3.28    22.99 | 23.46 | 23.45 | 32.64
Lc0-v27         9.69    4:39:23      60      5409    3.10     0      197  3.28    12.13 | 12.74 |  8.75 |  7.64

Legend

Depth     : Overall average search depth
Time      : Total time engine used
Moves     : Total moves engine played
Average   : Average time per move in centi-seconds
Forfeit   : Games engine lost due to time forfeit
Book      : Moves played from opening book plus average depth

MIDG      : Average search depth during the opening and middle game
EARLY     : Average search depth during the middle game without Queens, the so called early end game
ENDG      : Average search depth during the end game
LATE      : Average search depth during the late end game

List is sorted on Average Time indicating the engine that uses the most time tops.

Trying to understand why the Lc0 search depth in the endgame drops from 12.74 -> 8.75 -> 7.64 while it should be the other way around.

[Cornfed]

OTH LC0 is very neglected in the 40/15 rating list anyways,
only cpu versions are tested there since ever. Not a single test for a version with a GPU and most of them with
not much games, so it seems no one cared for from the LC0 team, or it went unnoticed.

The Lc0 team simply don't care what CCRL do, and that is fine. They are only interested in the likes of TCEC and CCC (that is the impression I got).

Testing GPU engines is relatively easy at blitz, but totally painful at longer time controls as you can't use concurrency.

Really, CPU vs GPU...kind of apples an oranges.

[brianr]

Trying to understand why the Lc0 search depth in the endgame drops from 12.74 -> 8.75 -> 7.64 while it should be the other way around.

There is no depth with Lc0 that is comparable to depth in A/B engines.

The reported Lc0 depth is actually just an estimate and is based on total playouts, and playouts are not the same as the node count (and
nodes are also not the same). Once tablebases are reached, I think the playouts (and hence the depth) get adjusted again.
In addition, there are no transposition tables, but something sort of similar called the NNCache, which also counts "nodes" differently.

Even simply setting Lc0 to play with a fixed number of "nodes" per move is quite exacting and depends on correctly setting several other options.

In short, the only way to compare with Lc0 is based on time, where of course the search speed varies enormously with the GPU and many other set up options, even when using CPU-only backends to search (and no GPU at all).

All of this makes Lc0 very difficult for the testing sites.
The variability of Lc0 speed even against itself with different net sizes requires testing based on time.