Help with Scorpio 3.0.8

Discussion of anything and everything relating to chess playing software and machines.

Moderator: Ras

Daniel Shawul
Posts: 4186
Joined: Tue Mar 14, 2006 11:34 am
Location: Ethiopia

Re: Help with Scorpio 3.0.8

Post by Daniel Shawul »

Dann Corbit wrote: Fri Jun 05, 2020 1:24 am
Daniel Shawul wrote: Thu Jun 04, 2020 11:44 pm Information is on my website http://scorpiozero.ddns.net

There is a link to the client github page on the left. Git clone (or download) that and run client.bat for windows or client.sh for linux.
The only requirement is a java runtime which most people have anyway.
It will install Scorpio for selfplay game generation and then connect to the server. Once you create a username and password,
it will keep how many games you generated for specific training runs.

That is it
Suggestion:
Have the client download the EGBB files if they are missing.
I don't have them on the machine I am running the client on, so I got a "0 egbbs loaded !" message
Since we already downloaded a pile of stuff, what is one more pile?

I need to load Java on it too
Oops, that's my bad, you told me about that and I forgot I did not do that yet
Dann, that is intentional that it doesn' download egbbs, the installer inside client.bat does "--no-egbb --no-lcnets" to reduce installation time.
At some point, I could start doing "egbb rescoring" before taining on the server-side, but originally I was doing game generation with egbbs with the client.
Dann Corbit
Posts: 12792
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: Help with Scorpio 3.0.8

Post by Dann Corbit »

I am getting more and more exceptions and I am starting to get Scorpio crashes (accessing a null pointer)
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
Daniel Shawul
Posts: 4186
Joined: Tue Mar 14, 2006 11:34 am
Location: Ethiopia

Re: Help with Scorpio 3.0.8

Post by Daniel Shawul »

Dann Corbit wrote: Fri Jun 05, 2020 3:26 am I am getting more and more exceptions and I am starting to get Scorpio crashes (accessing a null pointer)
Yes, I can see that from the server log too

Code: Select all

0: <games>
0: 0
0: Recieving cgames.pgn : 0 bytes from dann@___
0: </games>
0: <train>
0: Recieving ctrain.epd : 0 bytes from dann@___
0: </train>
Sending 0 bytes data sometimes but you did send about 5000 good games before.

Can you stop the client and run "scorpio.bat go quit" from the bin/Windows directory on the command line and post back the results?

Edit Oops I missed your pst where you did just that. Will take a look.
Last edited by Daniel Shawul on Fri Jun 05, 2020 3:39 am, edited 1 time in total.
Dann Corbit
Posts: 12792
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: Help with Scorpio 3.0.8

Post by Dann Corbit »

Something happened to my scorpio.bat file.
It is now zero sized:
Directory of C:\Users\dannc\chess\nn-dist-master\Scorpio\bin\Windows

06/04/2020 06:36 PM 0 scorpio.bat
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
Dann Corbit
Posts: 12792
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: Help with Scorpio 3.0.8

Post by Dann Corbit »

I copied scorpio.bat from the linux folder, because it looked like a Windows batch file.
When I ran it, I got this:

Code: Select all

C:\Users\dannc\chess\nn-dist-master\Scorpio\bin\Windows>scorpio.bat go quit
feature done=0
Number of cores 64 of 64
treeht 83886080 X 320 = 25600.0 MB
processors [192]
ht 67108864 X 16 = 1024.0 MB
eht 262144 X 8 X 192 = 768.0 MB
pht 32768 X 24 X 192 = 144.0 MB
EgbbProbe 4.3 by Daniel Shawul
egbb_cache 4084 X 8216 = 32.0 MB
0 egbbs loaded !
nn_cache 131072 X 1568 = 196.0 MB
Loading neural network : C:\Users\dannc\chess\nn-dist-master\Scorpio\nets-scorpio/ens-net-20x256.uff
With "INT8" precision
Loading graph on /gpu:0
Loading graph on /gpu:1
0. main_input 2048 = 32 8 8
1. policy/Reshape 1024 = 1024 1 1
2. value/BiasAdd 3 = 3 1 1
Neural network loaded !
loading_time = 4s
# rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1
# [st = 12502ms, mt = 29250ms , hply = 0 , moves_left 10]
32 28 87 48002 32 54796 0 e2-e4 e7-e5 Ng1-f3 Nb8-c6 Bf1-b5 a7-a6 Bb5-a4 Ng8-f6 Ke1-g1 Nf6xe4 d2-d4 b7-b5 Ba4-b3 d7-d5 d4xe5 Bc8-e6 c2-c3 Bf8-e7 Nb1-d2 Ne4-c5 Bb3-c2 d5-d4 Nd2-b3 d4-d3 Nb3xc5 d3xc2 Qd1xd8 Ra8xd8 Nc5xe6 f7xe6 Bc1-e3 Rd8-d5
33 30 175 95766 33 54692 0 e2-e4 e7-e5 Ng1-f3 Nb8-c6 Bf1-b5 a7-a6 Bb5-a4 Ng8-f6 Ke1-g1 Bf8-e7 Rf1-e1 b7-b5 Ba4-b3 d7-d6 c2-c3 Ke8-g8 h2-h3 Bc8-b7 d2-d4 Rf8-e8 Nb1-d2 Be7-f8 a2-a4 h7-h6 Bb3-c2 Ra8-b8 Bc2-b1 b5-b4 a4-a5 b4xc3
34 29 262 146192 34 55670 0 e2-e4 e7-e5 Ng1-f3 Nb8-c6 Bf1-b5 Ng8-f6 Ke1-g1 Nf6xe4 Rf1-e1 Ne4-d6 Nf3xe5 Bf8-e7 Bb5-f1 Nc6xe5 Re1xe5 Ke8-g8 d2-d4 Be7-f6 Re5-e1 Rf8-e8 c2-c3 Re8xe1 Qd1xe1 Nd6-e8 Bc1-f4 d7-d5 Nb1-d2 c7-c6 Bf1-d3 g7-g6 Nd2-f3 Ne8-g7
35 29 350 192225 35 54905 0 e2-e4 e7-e5 Ng1-f3 Nb8-c6 Bf1-b5 Ng8-f6 Ke1-g1 Nf6xe4 Rf1-e1 Ne4-d6 Nf3xe5 Bf8-e7 Bb5-f1 Nc6xe5 Re1xe5 Ke8-g8 d2-d4 Be7-f6 Re5-e1 Rf8-e8 c2-c3 Re8xe1 Qd1xe1 Nd6-e8 Bc1-f4 d7-d5 Nb1-d2 c7-c6 Bf1-d3 g7-g6 Nd2-f3 Ne8-g7
36 29 437 236825 36 54119 0 e2-e4 e7-e5 Ng1-f3 Nb8-c6 Bf1-b5 Ng8-f6 Ke1-g1 Nf6xe4 Rf1-e1 Ne4-d6 Nf3xe5 Bf8-e7 Bb5-f1 Nc6xe5 Re1xe5 Ke8-g8 d2-d4 Be7-f6 Re5-e1 Rf8-e8 c2-c3 Re8xe1 Qd1xe1 Nd6-e8 Bc1-f4 d7-d5 Nb1-d2 c7-c6 Bf1-d3 g7-g6 Nd2-f3 Ne8-g7
37 29 525 285559 37 54381 0 e2-e4 e7-e5 Ng1-f3 Nb8-c6 Bf1-b5 Ng8-f6 Ke1-g1 Nf6xe4 Rf1-e1 Ne4-d6 Nf3xe5 Bf8-e7 Bb5-f1 Nc6xe5 Re1xe5 Ke8-g8 d2-d4 Be7-f6 Re5-e1 Rf8-e8 c2-c3 Re8xe1 Qd1xe1 Nd6-e8 Bc1-f4 d7-d5 Nb1-d2 c7-c6 Bf1-d3 g7-g6 Nd2-f3 Ne8-g7
38 30 612 331913 38 54181 0 e2-e4 c7-c5 Ng1-f3 d7-d6 d2-d4 c5xd4 Nf3xd4 Ng8-f6 Nb1-c3 a7-a6 Bc1-e3 e7-e5 Nd4-f3 Bf8-e7 Bf1-c4 Ke8-g8 Ke1-g1 Bc8-e6 Qd1-e2 Be6xc4 Qe2xc4 b7-b5 Qc4-d3 Nb8-d7
39 30 700 377995 39 53991 0 e2-e4 c7-c5 Ng1-f3 d7-d6 d2-d4 c5xd4 Nf3xd4 Ng8-f6 Nb1-c3 a7-a6 Bc1-e3 e7-e5 Nd4-f3 Bf8-e7 Bf1-c4 Ke8-g8 Ke1-g1 Bc8-e6 Qd1-d3 Be6xc4 Qd3xc4 b7-b5 Qc4-d3 Nb8-d7 a2-a4 b5-b4
40 30 787 422607 40 53657 0 e2-e4 c7-c5 Ng1-f3 d7-d6 d2-d4 c5xd4 Nf3xd4 Ng8-f6 Nb1-c3 a7-a6 Bc1-e3 e7-e5 Nd4-f3 Bf8-e7 Bf1-c4 Ke8-g8 Ke1-g1 Bc8-e6 Qd1-d3 Be6xc4 Qd3xc4 b7-b5 Qc4-d3 Nb8-d7 a2-a4 b5-b4 Nc3-d5
41 30 875 464645 41 53096 0 e2-e4 c7-c5 Ng1-f3 d7-d6 d2-d4 c5xd4 Nf3xd4 Ng8-f6 Nb1-c3 a7-a6 Bc1-e3 e7-e5 Nd4-f3 Bf8-e7 Bf1-c4 Ke8-g8 Ke1-g1 Bc8-e6 Qd1-d3 Be6xc4 Qd3xc4 b7-b5 Qc4-d3 Nb8-d7 a2-a4 b5-b4 Nc3-d5
42 30 898 475883 42 52958 0 e2-e4 e7-e5 Ng1-f3 Nb8-c6 Bf1-b5 Ng8-f6 Ke1-g1 Nf6xe4 Rf1-e1 Ne4-d6 Nf3xe5 Bf8-e7 Bb5-f1 Nc6xe5 Re1xe5 Ke8-g8 d2-d4 Be7-f6 Re5-e1 Rf8-e8 c2-c3 Re8xe1 Qd1xe1 Nd6-e8 Bc1-f4 d7-d5 Nb1-d2 c7-c6 Bf1-d3 g7-g6 Nd2-f3 Ne8-g7
# nodes 7009122 <0% qnodes> time 9048ms nps 774659 eps 0 nneps 52683 tbhits 0
# Tree: nodes 619812 depth 31 pps 52873 visits 475914 qcalls 0 scalls 0
move e2e4
Bye Bye

C:\Users\dannc\chess\nn-dist-master\Scorpio\bin\Windows>
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
Daniel Shawul
Posts: 4186
Joined: Tue Mar 14, 2006 11:34 am
Location: Ethiopia

Re: Help with Scorpio 3.0.8

Post by Daniel Shawul »

That is odd. I guess you can copy and paste the contents of the file into it.
https://github.com/dshawul/Scorpio/blob ... corpio.bat

It seems you are generating far less games than what your powerful CPU and GPU combo implies. The nps you got with 2 GPs is just 24k.
So I recommend you run two separate instances of the client.
First edit scorpio.ini and modify the followin parameters

Code: Select all

mt                96
n_devices     1
That will make each client use 1 GPU and 96 threads. Then you can run two instances of the client in the background or on different terminals.
It would be ideal if there are two installation of the client itself but it might work also with a shared Scorpio installation so go ahead and give this a try
Last edited by Daniel Shawul on Fri Jun 05, 2020 3:50 am, edited 1 time in total.
Daniel Shawul
Posts: 4186
Joined: Tue Mar 14, 2006 11:34 am
Location: Ethiopia

Re: Help with Scorpio 3.0.8

Post by Daniel Shawul »

Dann Corbit wrote: Fri Jun 05, 2020 3:43 am I copied scorpio.bat from the linux folder, because it looked like a Windows batch file.
When I ran it, I got this:

Code: Select all

C:\Users\dannc\chess\nn-dist-master\Scorpio\bin\Windows>scorpio.bat go quit
feature done=0
Number of cores 64 of 64
treeht 83886080 X 320 = 25600.0 MB
processors [192]
ht 67108864 X 16 = 1024.0 MB
eht 262144 X 8 X 192 = 768.0 MB
pht 32768 X 24 X 192 = 144.0 MB
EgbbProbe 4.3 by Daniel Shawul
egbb_cache 4084 X 8216 = 32.0 MB
0 egbbs loaded !
nn_cache 131072 X 1568 = 196.0 MB
Loading neural network : C:\Users\dannc\chess\nn-dist-master\Scorpio\nets-scorpio/ens-net-20x256.uff
With "INT8" precision
Loading graph on /gpu:0
Loading graph on /gpu:1
0. main_input 2048 = 32 8 8
1. policy/Reshape 1024 = 1024 1 1
2. value/BiasAdd 3 = 3 1 1
Neural network loaded !
loading_time = 4s
# rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1
# [st = 12502ms, mt = 29250ms , hply = 0 , moves_left 10]
32 28 87 48002 32 54796 0 e2-e4 e7-e5 Ng1-f3 Nb8-c6 Bf1-b5 a7-a6 Bb5-a4 Ng8-f6 Ke1-g1 Nf6xe4 d2-d4 b7-b5 Ba4-b3 d7-d5 d4xe5 Bc8-e6 c2-c3 Bf8-e7 Nb1-d2 Ne4-c5 Bb3-c2 d5-d4 Nd2-b3 d4-d3 Nb3xc5 d3xc2 Qd1xd8 Ra8xd8 Nc5xe6 f7xe6 Bc1-e3 Rd8-d5
33 30 175 95766 33 54692 0 e2-e4 e7-e5 Ng1-f3 Nb8-c6 Bf1-b5 a7-a6 Bb5-a4 Ng8-f6 Ke1-g1 Bf8-e7 Rf1-e1 b7-b5 Ba4-b3 d7-d6 c2-c3 Ke8-g8 h2-h3 Bc8-b7 d2-d4 Rf8-e8 Nb1-d2 Be7-f8 a2-a4 h7-h6 Bb3-c2 Ra8-b8 Bc2-b1 b5-b4 a4-a5 b4xc3
34 29 262 146192 34 55670 0 e2-e4 e7-e5 Ng1-f3 Nb8-c6 Bf1-b5 Ng8-f6 Ke1-g1 Nf6xe4 Rf1-e1 Ne4-d6 Nf3xe5 Bf8-e7 Bb5-f1 Nc6xe5 Re1xe5 Ke8-g8 d2-d4 Be7-f6 Re5-e1 Rf8-e8 c2-c3 Re8xe1 Qd1xe1 Nd6-e8 Bc1-f4 d7-d5 Nb1-d2 c7-c6 Bf1-d3 g7-g6 Nd2-f3 Ne8-g7
35 29 350 192225 35 54905 0 e2-e4 e7-e5 Ng1-f3 Nb8-c6 Bf1-b5 Ng8-f6 Ke1-g1 Nf6xe4 Rf1-e1 Ne4-d6 Nf3xe5 Bf8-e7 Bb5-f1 Nc6xe5 Re1xe5 Ke8-g8 d2-d4 Be7-f6 Re5-e1 Rf8-e8 c2-c3 Re8xe1 Qd1xe1 Nd6-e8 Bc1-f4 d7-d5 Nb1-d2 c7-c6 Bf1-d3 g7-g6 Nd2-f3 Ne8-g7
36 29 437 236825 36 54119 0 e2-e4 e7-e5 Ng1-f3 Nb8-c6 Bf1-b5 Ng8-f6 Ke1-g1 Nf6xe4 Rf1-e1 Ne4-d6 Nf3xe5 Bf8-e7 Bb5-f1 Nc6xe5 Re1xe5 Ke8-g8 d2-d4 Be7-f6 Re5-e1 Rf8-e8 c2-c3 Re8xe1 Qd1xe1 Nd6-e8 Bc1-f4 d7-d5 Nb1-d2 c7-c6 Bf1-d3 g7-g6 Nd2-f3 Ne8-g7
37 29 525 285559 37 54381 0 e2-e4 e7-e5 Ng1-f3 Nb8-c6 Bf1-b5 Ng8-f6 Ke1-g1 Nf6xe4 Rf1-e1 Ne4-d6 Nf3xe5 Bf8-e7 Bb5-f1 Nc6xe5 Re1xe5 Ke8-g8 d2-d4 Be7-f6 Re5-e1 Rf8-e8 c2-c3 Re8xe1 Qd1xe1 Nd6-e8 Bc1-f4 d7-d5 Nb1-d2 c7-c6 Bf1-d3 g7-g6 Nd2-f3 Ne8-g7
38 30 612 331913 38 54181 0 e2-e4 c7-c5 Ng1-f3 d7-d6 d2-d4 c5xd4 Nf3xd4 Ng8-f6 Nb1-c3 a7-a6 Bc1-e3 e7-e5 Nd4-f3 Bf8-e7 Bf1-c4 Ke8-g8 Ke1-g1 Bc8-e6 Qd1-e2 Be6xc4 Qe2xc4 b7-b5 Qc4-d3 Nb8-d7
39 30 700 377995 39 53991 0 e2-e4 c7-c5 Ng1-f3 d7-d6 d2-d4 c5xd4 Nf3xd4 Ng8-f6 Nb1-c3 a7-a6 Bc1-e3 e7-e5 Nd4-f3 Bf8-e7 Bf1-c4 Ke8-g8 Ke1-g1 Bc8-e6 Qd1-d3 Be6xc4 Qd3xc4 b7-b5 Qc4-d3 Nb8-d7 a2-a4 b5-b4
40 30 787 422607 40 53657 0 e2-e4 c7-c5 Ng1-f3 d7-d6 d2-d4 c5xd4 Nf3xd4 Ng8-f6 Nb1-c3 a7-a6 Bc1-e3 e7-e5 Nd4-f3 Bf8-e7 Bf1-c4 Ke8-g8 Ke1-g1 Bc8-e6 Qd1-d3 Be6xc4 Qd3xc4 b7-b5 Qc4-d3 Nb8-d7 a2-a4 b5-b4 Nc3-d5
41 30 875 464645 41 53096 0 e2-e4 c7-c5 Ng1-f3 d7-d6 d2-d4 c5xd4 Nf3xd4 Ng8-f6 Nb1-c3 a7-a6 Bc1-e3 e7-e5 Nd4-f3 Bf8-e7 Bf1-c4 Ke8-g8 Ke1-g1 Bc8-e6 Qd1-d3 Be6xc4 Qd3xc4 b7-b5 Qc4-d3 Nb8-d7 a2-a4 b5-b4 Nc3-d5
42 30 898 475883 42 52958 0 e2-e4 e7-e5 Ng1-f3 Nb8-c6 Bf1-b5 Ng8-f6 Ke1-g1 Nf6xe4 Rf1-e1 Ne4-d6 Nf3xe5 Bf8-e7 Bb5-f1 Nc6xe5 Re1xe5 Ke8-g8 d2-d4 Be7-f6 Re5-e1 Rf8-e8 c2-c3 Re8xe1 Qd1xe1 Nd6-e8 Bc1-f4 d7-d5 Nb1-d2 c7-c6 Bf1-d3 g7-g6 Nd2-f3 Ne8-g7
# nodes 7009122 <0% qnodes> time 9048ms nps 774659 eps 0 nneps 52683 tbhits 0
# Tree: nodes 619812 depth 31 pps 52873 visits 475914 qcalls 0 scalls 0
move e2e4
Bye Bye

C:\Users\dannc\chess\nn-dist-master\Scorpio\bin\Windows>
Hmm and now you are getting a really good nps of 52k as compared to your previous run of 24k np ...
So maybe you should keep the settings then i.e. 1 client using 2 of your GPUs.
Dann Corbit
Posts: 12792
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: Help with Scorpio 3.0.8

Post by Dann Corbit »

Do we know what is causing the crashes?
I am not running out of RAM, since I have 128GB and 12 was used (it looked like perhaps 64 GB was reserved for the GPUs, but I did not see virtual memory for that).

Client crashes right away now, it seems to be accelerating.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
Daniel Shawul
Posts: 4186
Joined: Tue Mar 14, 2006 11:34 am
Location: Ethiopia

Re: Help with Scorpio 3.0.8

Post by Daniel Shawul »

Dann, also increase the number of games you generate before sending to the server.
512 games is too low for your GPUs.
Please increase it to 2048 by modifying nn-dist/scripts/job-client.bat line 7

Code: Select all

SET G=2048
For your crashes, please change to delay 1 in scorpio.ini

Code: Select all

delay   1
That will decrease the CPU load from 100% to something very small. (Threads will sleep for 30ms when idle).
This will probably decrease the number of games you generate per min, but we will talk about increasing performance once
we make your client stable.
Terje
Posts: 347
Joined: Tue Nov 19, 2019 4:34 am
Location: https://github.com/TerjeKir/weiss
Full name: Terje Kirstihagen

Re: Help with Scorpio 3.0.8

Post by Terje »

Daniel Shawul wrote: Fri Jun 05, 2020 3:03 am
Terje wrote: Fri Jun 05, 2020 2:56 am
Daniel Shawul wrote: Fri Jun 05, 2020 2:48 am
Terje wrote: Fri Jun 05, 2020 12:08 am Why does scorpio not report depths in the same way leela or allie does?
Dann, answered it that there is no clear iterative search depth in mcts search.
My odd choice of starting depth from 63 is because my AB search runs from depth = 1 to 63. Scorpio can do hybrid AB
and MCTS, running the AB first, say the first 30% of the time, so 63 is a safe option that avoids depth overlap.

Assuming an effective branching of 2 or something similar, I could calculate an average depth like leela i guess.
I see. In most parts of the engine originality is great, but just for the UCI/WB communication it would be nice if everyone was on the same page as much as possible :)
Well tell me what depth to report when the AB search first did search up to depth of 30. AFAIK lc0 depths are maximum 30.
If I want to report depths just like leela the two will overlap, and I know for instance xboard sorts your PVs based on your depth creating a mess i guess.
An example if I do AB all the time and how Scorpio avoid this. It offsets the AB depth by mcts_strategy_depth=30 when it sarts alpha-beta search.

...
If it's not possible/practical to change it then ignore what I said :)

Side note, odd for a gui to order them by anything other than when they were sent...