Help with Scorpio 3.0.8

[Daniel Shawul]

Was it using 2 GPUs and only generating 31 games/min? Maybe this is a windows/linux issue related with timeslice a thread gets.
On a linux machine with 2x2080 I did 80 games/min with just a 5-cores CPU.
What were your settings for the three parameters: n_devices, mt, and delay?
If delay=1 is what makes your GPU stable and you want to use that, we can double the number of threads to improve performance.
I am certain we can tune performance to double your game generation rate, otherwise it is a waste of your resources.

Code: Select all

mt               384
n_devices      2
delay              1

This will do calibration again so will take some time.
Also, increase the number of games generated to 2304 (i.e.6x384) before sending to server in scripts/job-client.bat line 7 (G parameter)
Doing only 512 games will be a waste since it will run 384 games in parallel and usually takes some time to warm up.

Yes, the client has been updated to version 2 to fix some issues with reconnections.

Daniel

[Daniel Shawul]

Btw are you on discord ?
Me an my collaborator (GaryS) chat there often to solve these kind of issues quickly.
I do server restarts sometimes and will let you know there.
You were able to login and contribute 512 games after I just did that

[Modern Times]

Is it best to install this as a UCI engine or a WB engine ?

I think they are best both tested well enough now but xboard has been with scorpio for ages.
Thanks for the tests, could you please use mt 40 with it ? I will change my formula for the case where the CPU
is relatively stronger than the GPU.

Well I have the same issue as previously. The engine won't run in Cutechess GUI. Install as UCI, and when the match starts the GUI vanishes and there is nothing. Install as xboard, and when the match starts Scorpio brings up a crash dialogue. It has to be something to do with having the engine as a bat file surely ? Why start it from that rather than the exe ? Is it to set tale base path or something ? I'll try the exe as an experiment to see if it works, but then it may not be working as intended.

EDIT: Running the exe as a UCI engine, it doesn't crash but doesn't make a move, I get the disconnect error from cutechess.

[Daniel Shawul]

@Dann @Modern Times

I created a discord channel if you would like to join. It is more comfortable to discuss and give instant reply,
share log files etc.

https://discord.gg/BPYhwG

@Modern Times If you can show log file, I can help you with it. I only tested with xboard/arena/banksia gui/ octagon GUIs.

[Daniel Shawul]

Is it best to install this as a UCI engine or a WB engine ?

I think they are best both tested well enough now but xboard has been with scorpio for ages.
Thanks for the tests, could you please use mt 40 with it ? I will change my formula for the case where the CPU
is relatively stronger than the GPU.

Well I have the same issue as previously. The engine won't run in Cutechess GUI. Install as UCI, and when the match starts the GUI vanishes and there is nothing. Install as xboard, and when the match starts Scorpio brings up a crash dialogue. It has to be something to do with having the engine as a bat file surely ? Why start it from that rather than the exe ? Is it to set tale base path or something ? I'll try the exe as an experiment to see if it works, but then it may not be working as intended.

EDIT: Running the exe as a UCI engine, it doesn't crash but doesn't make a move, I get the disconnect error from cutechess.

I just tested it cutechess GUI on linux machine. And it works well both using uci or xboard.
Make sure you remove the previous installation of Scorpio in cutechess GUI. It saves settings of the engine
and using those could mess up things, which I suspect is the problem.

What I di go to the Tools/Settings and removed my previous scorpio installation, then browsed the Command
to be "./scorpio.sh" (in your case scorpio.bat). If you click the Advanced tab you will see settings of scorpio and if those
are different from what is in scorpio.ini it may crash.

Unless things are different on windows machine it seems to work fine with cutechess GUI to.

Daniel

[Dann Corbit]

I have never used discord.

scorpio.ini now contains:

Code: Select all

mt               384
n_devices      2
delay              1

job-client.bat now contains:

Code: Select all

SET G=2304

The GPU is running at 2%
The CPU is running at 50%, with a sawtooth pattern of activity (I guess each tooth is a game)

[Dann Corbit]

@Dann @Modern Times

I created a discord channel if you would like to join. It is more comfortable to discuss and give instant reply,
share log files etc.

https://discord.gg/BPYhwG

@Modern Times If you can show log file, I can help you with it. I only tested with xboard/arena/banksia gui/ octagon GUIs.

I can't join discord because I had a botched some time ago attempt and it saved my email address.

[Daniel Shawul]

I have never used discord.

Ok no problem, lets try to solve it here then.

scorpio.ini now contains:

Code: Select all

mt               384
n_devices      2
delay              1

job-client.bat now contains:

Code: Select all

SET G=2304

The GPU is running at 2%
The CPU is running at 50%, with a sawtooth pattern of activity (I guess each tooth is a game)

Hmm, are you sure the CPU and GPU utilizations are not reversed?
With a 2% GPU utilization, I don't think you can go 31 games/min.
What is your game generation rate after increasing mt to 384 ?

[Dann Corbit]

I think the initial numbers were during the setup phase after restarting the batch.
Currently CPU is at 50% and both GPUs are at 0%

[Daniel Shawul]

Well in that case, the best time to measure GPU utilization would be when the games generated reach the middle of the batch (around 1600 games).
The warmup step is usually very slow because we are doing 384 games in parallel.

The calibration phase wastes some time but is needed for INT8. Since your GPU is not being fully utilized anyway, you may want to try HALF precision instead

Code: Select all

float_type HALF

That will give more accurate games and avoid that extra overhead of calibration.
With 6 or more GPUs working, generating 16k games (one net) happens maybe every 15min so there will be a lot of time wasted due to calibration alone that could take upto 5 min.