Help with Scorpio 3.0.8

[Daniel Shawul]

Once you get your client/server stuff sorted out we will try it again

Dann, I believe the client is stable now.
Please install it from scratch and let it reinstall Scorpio too.
We have started a brand new 20b from zero now as it turned out resuming training from an SL net
is rather complicated as it will try to rewrite what the SL net learned.

Daniel

[Dann Corbit]

I will give it a go

[Dann Corbit]

This version uses less GPU that the progenitor versions.
I was seeing up to 80% on each card at times. But mostly in the 50% range.
Now:

Code: Select all

C:\Users\dannc\chess\Stockfish-XI\src>C:\Windows\System32\DriverStore\FileRepository\nv_dispsi.inf_amd64_86e7a5db5f94798e\nvidia-smi
Wed Jun 10 12:54:01 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 442.19       Driver Version: 442.19       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 208... WDDM  | 00000000:01:00.0  On |                  N/A |
| 82%   79C    P2   109W / 250W |   1899MiB /  8192MiB |     36%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 208... WDDM  | 00000000:4E:00.0 Off |                  N/A |
| 79%   78C    P2   114W / 250W |    802MiB /  8192MiB |     37%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1596    C+G   ...osoft.LockApp_cw5n1h2txyewy\LockApp.exe N/A      |
|    0      2908    C+G   ...6.102.0_x64__kzf8qxf38zg5c\SkypeApp.exe N/A      |
|    0      3012    C+G   ...dows.Cortana_cw5n1h2txyewy\SearchUI.exe N/A      |
|    0      4348      C   ...-master\Scorpio\bin\Windows\scorpio.exe N/A      |
|    0      8268    C+G   ...6)\Google\Chrome\Application\chrome.exe N/A      |
|    0      9676    C+G   ...t_cw5n1h2txyewy\ShellExperienceHost.exe N/A      |
|    0     10268    C+G   ...x64__8wekyb3d8bbwe\Microsoft.Photos.exe N/A      |
|    0     12388    C+G   ...hell.Experiences.TextInput.InputApp.exe N/A      |
|    0     13608    C+G   Insufficient Permissions                   N/A      |
|    0     15136    C+G   C:\Windows\explorer.exe                    N/A      |
|    0     15192    C+G   ...1.93.0_x64__8wekyb3d8bbwe\YourPhone.exe N/A      |
|    1      4348      C   ...-master\Scorpio\bin\Windows\scorpio.exe N/A      |
+-----------------------------------------------------------------------------+

[Daniel Shawul]

Yes, I reverted a commit that made GPUs more independent in search but may have been the cause of hanging in some tournaments.
After we make sure things are stable now, maybe I will revisit this again.
Note that the game generation per min has plummeted now that we started from scratch ( no resignation in selfplay games) but is
rapidly improving and i expect it will be on par with what we used to have in a week.

Thanks!

[Dann Corbit]

The client was hung this morning on the login step, but I think the problem was on my end because the network was down.
I reset the network and restarted the batch

[Daniel Shawul]

There are still some quirks in the server which happens when I restart it. Sometimes one of my clients hang the only
way to let them in is to restart the server and this could affect others.
I see you are now contributing games again so all seems to be good. If in the future you find that your client is hanging,
the solution most of the time is to kill all processes and start it again.

Thanks!

[Daniel Shawul]

There was a restriction on the number of login attempts (2 or 3 times) which causes hangin on the "login:" prompt.
This should have been done on the server side, so I have removed that now and update the client.
If you are using git, do a "git pull" and hopefully you won't have this problem again.

[Dann Corbit]

Got the new client.
Giving her a try

[Dann Corbit]

Currently, it seems to be hung trying to download a new network model. It has been in the same place on the screen for several hours:

Code: Select all

0: **** Starting NTS session as dann ****
0:
0: <version>
0: 3
0: 3
0: </version>
0: <parameters> 800 125 120 25 0 80
0: </parameters>
0: <checksum>
0: 3623345478
0: http://scorpiozero.ddns.net/scorpiozero/nets-11/ID-0-model-3.uff
0: </checksum>
0: Skipping net download.
0: Sending 513 games to server
0: Executing job : C:\Users\dannc\chess\nn-dist-master\scripts\job-client.bat
[11240] Games 515: + 235 - 268 = 12
[11240] generated 515 games in 23.26 min : Rate 22.14 games/min
0: Finished executing job!
0: Sending 515 games to server
0: <checksum>
0: 1970703188
0: http://scorpiozero.ddns.net/scorpiozero/nets-11/ID-0-model-3.uff
0: </checksum>
0: Downloading new network from: http://scorpiozero.ddns.net/scorpiozero/nets-11/ID-0-model-3.uff

[Dann Corbit]

It then went into some loop where it was transmitting zero games. I restarted it, but more of the same:

Code: Select all

 <version>
0: 3
0: 3
0: </version>
0: <parameters> 800 125 120 25 0 80
0: </parameters>
0: <checksum>
0: 1970703188
0: http://scorpiozero.ddns.net/scorpiozero/nets-11/ID-0-model-3.uff
0: </checksum>
0: Skipping net download.
0: Executing job : C:\Users\dannc\chess\nn-dist-master\scripts\job-client.bat
0: Finished executing job!
0: Sending 0 games to server
0: pong
0: Executing job : C:\Users\dannc\chess\nn-dist-master\scripts\job-client.bat
0: Finished executing job!
0: Sending 0 games to server
0: pong
0: Executing job : C:\Users\dannc\chess\nn-dist-master\scripts\job-client.bat
0: Finished executing job!
0: Sending 0 games to server
0: pong
0: Executing job : C:\Users\dannc\chess\nn-dist-master\scripts\job-client.bat
0: Finished executing job!
0: Sending 0 games to server
0: pong
0: Executing job : C:\Users\dannc\chess\nn-dist-master\scripts\job-client.bat