Help with Scorpio 3.0.8

[Modern Times]

At first I thought that it was a mistake it was launching only 10 threads, but it isn't.
The GTX 1050 has only 5 SMs, so it allocates 2x the number of threads.

I am curious though how much pps it will get with mt set to 20 or 40.
Could you set mt to these values in scoprio.ini and run scorpio.bat go quit again, and post the results here?
My estimate of using 2x number of SMs of GPU could be wrong on low end GPUs.

Thanks.
Daniel

mt 40.

Code: Select all

C:\Engines\ScorpioNN\Scorpio\bin\Windows>scorpio.bat go quit
feature done=0
Number of cores 8 of 8
treeht 83886080 X 320 = 25600.0 MB
processors [40]
ht 67108864 X 16 = 1024.0 MB
eht 262144 X 8 X 40 = 160.0 MB
pht 32768 X 24 X 40 = 30.0 MB
EgbbProbe 4.3 by Daniel Shawul
egbb_cache 4084 X 8216 = 32.0 MB
180 egbbs loaded !
nn_cache 131072 X 1568 = 196.0 MB
Loading neural network : C:\Engines\ScorpioNN\Scorpio\nets-scorpio/ens-net-20x256.uff
With "INT8" precision
Loading graph on /gpu:0
0. main_input 2048 = 32 8 810
1. policy/Reshape 1024 = 1024 1 1
2. value/BiasAdd 3 = 3 1 1
Neural network loaded !
nn_cache 131072 X 1568 = 196.0 MB
Loading neural network : C:\Engines\ScorpioNN\Scorpio\nets-maddex/ME.uff
With "INT8" precision
Loading graph on /gpu:0
0. main_input 7168 = 112 8 80
1. value_head 3 = 3 1 1
2. policy_head 1858 = 1858 1 1
Neural network loaded !
loading_time = 1401s
# rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1
# [st = 12502ms, mt = 29250ms , hply = 0 , moves_left 10]
63 32 89 2214 63 2487 0 e2-e4 e7-e5 Ng1-f3 Nb8-c6 Bf1-b5 Ng8-f6 Ke1-g1 Nf6xe4 Rf1-e1 Ne4-d6 Nf3xe5 Bf8-e7 Bb5-f1 Nc6xe5 Re1xe5 Ke8-g8 d2-d4
64 30 179 4839 64 2692 0 e2-e4 e7-e5 Ng1-f3 Nb8-c6 Bf1-b5 Ng8-f6 Ke1-g1 Nf6xe4 Rf1-e1 Ne4-d6 Nf3xe5 Bf8-e7 Bb5-f1 Nc6xe5 Re1xe5 Ke8-g8 d2-d4 Be7-f6 Re5-e1 Rf8-e8 c2-c3 Re8xe1 Qd1xe1 Nd6-e8 Bc1-f4 d7-d5 Nb1-d2 c7-c6 Bf1-d3 g7-g6
65 30 268 7336 65 2730 0 e2-e4 e7-e5 Ng1-f3 Nb8-c6 Bf1-b5 Ng8-f6 Ke1-g1 Nf6xe4 Rf1-e1 Ne4-d6 Nf3xe5 Bf8-e7 Bb5-f1 Nc6xe5 Re1xe5 Ke8-g8 d2-d4 Be7-f6 Re5-e1 Rf8-e8 c2-c3 Re8xe1 Qd1xe1 Nd6-e8 Bc1-f4 d7-d5 Nb1-d2 c7-c6 Bf1-d3 g7-g6 Nd2-f3
66 30 356 9885 66 2775 0 e2-e4 e7-e5 Ng1-f3 Nb8-c6 Bf1-b5 a7-a6 Bb5-a4 Ng8-f6 Ke1-g1 Bf8-e7 Rf1-e1 b7-b5 Ba4-b3 d7-d6 c2-c3 Ke8-g8 h2-h3 Bc8-b7 d2-d4 Rf8-e8 Nb1-d2 Be7-f8 a2-a4 h7-h6 Bb3-c2 Ra8-b8
67 29 443 12735 67 2870 0 e2-e4 e7-e5 Ng1-f3 Nb8-c6 Bf1-b5 a7-a6 Bb5-a4 Ng8-f6 Ke1-g1 Bf8-e7 Rf1-e1 b7-b5 Ba4-b3 d7-d6 c2-c3 Ke8-g8 h2-h3 Bc8-b7 d2-d4 Rf8-e8 Nb1-d2 Be7-f8 a2-a4 h7-h6 Bb3-c2 Ra8-b8 Bc2-b1
68 29 531 15536 68 2924 0 e2-e4 e7-e5 Ng1-f3 Nb8-c6 Bf1-b5 a7-a6 Bb5-a4 Ng8-f6 Ke1-g1 Bf8-e7 Rf1-e1 b7-b5 Ba4-b3 d7-d6 c2-c3 Ke8-g8 h2-h3 Bc8-b7 d2-d4 Rf8-e8 Nb1-d2 Be7-f8 a2-a4 h7-h6 Bb3-c2 Ra8-b8 Bc2-b1
69 29 619 18342 69 2959 0 e2-e4 e7-e5 Ng1-f3 Nb8-c6 Bf1-b5 a7-a6 Bb5-a4 Ng8-f6 Ke1-g1 Bf8-e7 Rf1-e1 b7-b5 Ba4-b3 d7-d6 c2-c3 Ke8-g8 h2-h3 Bc8-b7 d2-d4 Rf8-e8 Nb1-d2 Be7-f8 a2-a4 h7-h6 Bb3-c2 Ra8-b8 Bc2-b1
70 30 707 21133 70 2988 0 e2-e4 e7-e5 Ng1-f3 Nb8-c6 Bf1-b5 a7-a6 Bb5-a4 Ng8-f6 Ke1-g1 Bf8-e7 Rf1-e1 b7-b5 Ba4-b3 d7-d6 c2-c3 Ke8-g8 h2-h3 Bc8-b7 d2-d4 Rf8-e8 Nb1-d2 Be7-f8 a2-a4 h7-h6 Bb3-c2 Ra8-b8 Bc2-b1
71 30 799 24082 71 3012 0 e2-e4 e7-e5 Ng1-f3 Nb8-c6 Bf1-b5 a7-a6 Bb5-a4 Ng8-f6 Ke1-g1 Nf6xe4 d2-d4 b7-b5 Ba4-b3 d7-d5 d4xe5 Bc8-e6 c2-c3 Bf8-e7 Nb1-d2 Ne4-c5 Bb3-c2 d5-d4 Nd2-b3 d4xc3 Nb3xc5 Be7xc5 Bc2-e4 Qd8-d7 b2xc3 Ra8-d8
72 30 890 26958 72 3028 0 e2-e4 e7-e5 Ng1-f3 Nb8-c6 Bf1-b5 a7-a6 Bb5-a4 Ng8-f6 Ke1-g1 Nf6xe4 d2-d4 b7-b5 Ba4-b3 d7-d5 d4xe5 Bc8-e6 c2-c3 Bf8-e7 Nb1-d2 Ne4-c5 Bb3-c2 d5-d4 Nd2-b3 d4xc3 Nb3xc5 Be7xc5 Bc2-e4 Qd8-d7 b2xc3 Ra8-d8 Qd1xd7
73 31 940 28317 73 3012 0 e2-e4 e7-e5 Ng1-f3 Nb8-c6 Bf1-b5 a7-a6 Bb5-a4 Ng8-f6 Ke1-g1 Nf6xe4 d2-d4 b7-b5 Ba4-b3 d7-d5 d4xe5 Bc8-e6 c2-c3 Bf8-e7 Nb1-d2 Ne4-c5 Bb3-c2 d5-d4 Nd2-b3 d4xc3 Nb3xc5 Be7xc5 Bc2-e4 Qd8-d7 b2xc3 Ra8-d8 Qd1xd7
# nodes 292363 <0% qnodes> time 9525ms nps 30694 eps 0 nneps 3004 tbhits 0
# Tree: nodes 36582 depth 32 pps 3001 visits 28356 qcalls 0 scalls 0
move e2e4
Bye Bye

[Dann Corbit]

Well, it is not crashing any more. But it seems to be running much slower.
Is there some way to reduce the delay from 30 ms to 1 ms or something like that?
It is using 1% of the CPU and 1% of the GPU, and I can't imagine that is what was intended.
Right now, it is using 7.4 GB and seems to increase about 2 MB/sec. Is that normal?
I don't think it will get anywhere close to exhausting memory because I have 128 GB RAM and a lot of very fast virtual memory.
I wish I could make it go faster, because I feel like my donation of machine time is coming to nothing.

[Modern Times]

Is it best to install this as a UCI engine or a WB engine ?

[Daniel Shawul]

Is it best to install this as a UCI engine or a WB engine ?

I think they are best both tested well enough now but xboard has been with scorpio for ages.
Thanks for the tests, could you please use mt 40 with it ? I will change my formula for the case where the CPU
is relatively stronger than the GPU.

[Daniel Shawul]

Well, it is not crashing any more. But it seems to be running much slower.
Is there some way to reduce the delay from 30 ms to 1 ms or something like that?
It is using 1% of the CPU and 1% of the GPU, and I can't imagine that is what was intended.
Right now, it is using 7.4 GB and seems to increase about 2 MB/sec. Is that normal?
I don't think it will get anywhere close to exhausting memory because I have 128 GB RAM and a lot of very fast virtual memory.
I wish I could make it go faster, because I feel like my donation of machine time is coming to nothing.

Dann, how many games per min were you generating?
Using only 1% of GPU says something very wrong. I was thinking of nps halving or something close to that when using delay 1.
Could you try using only 1 GPU and setting delay back to 0 ?

Code: Select all

n_devices     1
delay             0

I used to have code that does multple instances of game generation (client) and merge the result and sends it but removed it
because I thought it is not more efficient than using both GPUs. I will take a second look.

[Daniel Shawul]

Dann, I have been experimenting on vast.ai on an instance with 2x RTX 2080s and just 5 cores.
You have way many cores than this and slightly better GPU too.
The default configuration that it produces for this setup is

Code: Select all

mt              184
n_devices     2
delay             1

This was generating 80 games per min. The GPU usage was 58% for each of them.

I then tried, launching two separate instances of the client, and each GPU worked at 43 games/min.
So using one instance of the client is not all that bad.

My guess is your GPUs should be able to generate more than 80 games per min even with delay 1.
Since you have 64 cores, it may even get better with delay = 0 if that didn't cause a problem.

[Dann Corbit]

I will try some different settings and see if I can get reasonable performance.
This morning, I saw that the batch had stalled trying to send something.

[Dann Corbit]

I decided to try a reboot. When I did that, the batch job asked me to download a new version of the client. I did that. I started a new run and we will see how it performs.

[Dann Corbit]

After the recalibration, the first batch ran at 31 games/min.
The second batch was also 31 games/min.
At a game every 2 seconds, that would give 86400/2=43,200 games per day
Which is 302,000 in a week.

Unfortunately, the program is still barely using the compute resources.
But this version is stable and not crashing or throwing errors.

[Dann Corbit]

Now I seem to be having connection problems.
I also have an engine failure message with :
Engine failure: scorpiozero.ddns.net 48555 dann corbit
Waiting for 30 sec ...
Reconnecting ...
and that message cycles