They are both using 2 threads on an i7 laptop processor. L0 gets about 200 playouts/sec. scorpio-mcts-min could actually be 2700 elo on TCEC hardware too though one has to test. If that is the case then, all this NN work is a waste if you compare fairly.CMCanavessi wrote:Daniel Shawul wrote:It got better at 320+2 but still like -180 elosI had to shut down my lap top so this one has few games. Next one is 900+10 (15m+10) ...Code: Select all
scorpio-mcts-min : (+ 10 ,= 2 ,- 3) 1. scorpio-mcts-min 91 183 183 15 73.3% -90 13.3% 2. lczero -90 183 183 15 26.7% 91 13.3%
Is that the config I tested? How can we get so different results? What hardware config are you testing Leela on? How many threads for Scorpio?
Your setup is very unfair to the alphabeta engines because L0 was getting 1 kN/s on your GPU. You should use 1 CPU core for it too if you want to be fair. Infact AlphaZero also should have used the 64 cores used for Stockfish; the 4 TPU setup hugely benefits A0.
If leela has to run on the GPU (though there is no reason for it as examplified by TCEC results), you can measure how much nps it gets on 1 CPU core, and calculate the number of cores to use for the alphabeta engines from that. Say L0 got about 100 n/s on one cpu core, then you have to use atleast 1000/100=10 cores for the alphabeta engines if leela uses the GPU. This one still gives some edge to L0 because scalability of alphabeta engines is not that good, but it is a more fair setup than what you have. This is the reason you have been getting inflated elos for L0.
To say that L0 is not designed for the CPU is wrong; see my other post about the similarity of the situation where a speciality hardware is used to accelerate an evaluation at the same cost. What plays in L0 favour is the performance per dollar (same price GPU and CPU) or per watt; GPUs are getting cheaper and are preferred for HPC for this reason.