STSStat.exe (v2.0)

[swami]

STS Stat v2.0:

- Added Grading system.
- Added a feature that calculates the TOTAL Scores of (STS 1+STS 2......+STS n)
- Added Average Calculations.
- Added Percentage Calculations.

http://sites.google.com/site/strategict ... atv2.0.zip

This is just wonderful! Many many Thanks, GAILHAC Phillipe!!

[El Gringo]

Hi,

A little workaround with STSStat.exe

Code: Select all

----------------------------------------------------------------------------------------------------------------------------------
                                       |  STS1  |  STS2  |  STS3  |  STS4  |  STS5  |  STS6  |  STS7  |  STS8  |  STS9  |
----------------------------------------------------------------------------------------------------------------------------------
Deep Shredder 12 x64 2CPU  |    85   |    79   |    81    |    77   |    81   |    73    |    82    |    80    |    64    |
----------------------------------------------------------------------------------------------------------------------------------
Grade                      |    S    |    A    |    A+    |    A    |    A+    |    A-    |    A+    |    A+    |    B    |
Score / 1000               |    889  |   798   |   864    |   835   |   860    |    858   |    862   |   843    |   762    |
Grade                      |    S    |    A    |    S    |    A+    |    S     |    S     |    S     |    A+    |    A    |
Meilleur coup :  702 / 900  78 %  Grade : A
Total :           7571 / 9000  84,12 %  Grade : A+
----------------------------------------------------------------------------------------------------------------------------------

Nice program !

Best
Johan

[David Dahlem]

Thanks Swami. This is indeed a wonderful tool.

I am having a problem though, not with this tool, but with the scores i'm getting from your STS suites. I tried repeating the same test with the same suite, same engine, same conditions, and everytime i get different scores. For example, on STS1.epd, i ran it three times under identical conditions with a Komodo personality and got three different scores, 806, 823, and 835. This happens with other engines also. If i can't get somewhat duplicate scores, i feel like i am wasting my time.

Have you tried repeating test runs to check if scores are consistent? Also many times different engines hang and stop analyzing and i have to restart the test. I've been using this analysis feature in Arena for years, and don't recall any engine hangups. Could Arena have a problem with the format of the STS epd files?

Anyone else had similar problems as these?

Regards
Dave

[swami]

Hi David,

I tested Umko, and repeated the same test again, It scored the same score as the first one.

I used ChessGUI and Arena for the single test and it scored the same exact score as well.

Not sure what the problem might be. Perhaps other programs took up memory at some point that caused the engine to slow down the search depth or something. Or Perhaps the engine is dynamic with learning feature and constantly changes its mind.

[yanquis1972]

could someone give a quick & dirty on how this works? i find these suites interesting but it's difficult enough for me to just figure out how to process them (before i got shredder gui anyway).

[swami]

could someone give a quick & dirty on how this works? i find these suites interesting but it's difficult enough for me to just figure out how to process them (before i got shredder gui anyway).

These are the 8 steps you need to follow to have the engine and EPD's successfully set up in Arena.

How to test EPD's in Arena?

Code: Select all

1. Download Arena 2.01 (www.playwitharena.com) 
2. Click Arena.exe 
3. Install engine (Engine -> Manage -> Details -> New -> UCI/Wb Engine) 
4. Download Test Suites and store them in a folder. (Let's say: C:\STS) 
5. Click - Engines -> Automatic Analysis -> List of EPD/PGN -> Load ALL EPD's from C:\STS 
6. In "Automatic Analysis" window - go to the next tab "Engines", select .installed engine, and specify time per move. 
7. In "Automatic Analysis" window - go to the previous tab "Source", Start the test. 
8. Once the test is finished, In "Automatic Analysis" window - go further to the next tab "Output", and click "view comprehensive output in Report file" notepad. 

How to use STS Stat?

Code: Select all

Extract the STSStat.exe and Epds and save them in a STSStats folder. 

When using automatic analysis mode in Arena, be sure to select EPDs from STSStat folder. 

Once the testing is done, open up analyses log or whatever log file(from Arena folder) with STSstat.exe 

Hope you find this useful. Good luck Testing!

Regards,
Swami

[swami]

Komodo 1.0 Sample Run with STS Stat 2.Exe

Code: Select all

File :  H:\Arena 2.01\Analyses.log

epd File :  STS2.epd   

Studied module :  Komodo-10-32-ja

Level:  10 seconds by position
Matching moves : 79 / 100
Grade : A
Total time : 12:17:56
Rated time : 253 Seconds

Score :  776 / 1000
Grade : A

--------------------------------------------------------
File :  H:\Arena 2.01\Analyses.log

epd File :  STS3.epd   

Studied module :  Komodo-10-32-ja

Level:  10 seconds by position
Matching moves : 82 / 100
Grade : A+
Total time : 12:17:56
Rated time : 239 Seconds

Score :  885 / 1000
Grade : S

--------------------------------------------------------
File :  H:\Arena 2.01\Analyses.log

epd File :  STS4.epd   

Studied module :  Komodo-10-32-ja

Level:  10 seconds by position
Matching moves : 76 / 100
Grade : A
Total time : 12:17:56
Rated time : 310 Seconds

Score :  841 / 1000
Grade : A+

--------------------------------------------------------
File :  H:\Arena 2.01\Analyses.log

epd File :  STS5.epd   

Studied module :  Komodo-10-32-ja

Level:  10 seconds by position
Matching moves : 73 / 100
Grade : A-
Total time : 12:17:56
Rated time : 283 Seconds

Score :  821 / 1000
Grade : A+

--------------------------------------------------------
File :  H:\Arena 2.01\Analyses.log

epd File :  STS6.epd   

Studied module :  Komodo-10-32-ja

Level:  10 seconds by position
Matching moves : 77 / 100
Grade : A
Total time : 12:17:57
Rated time : 282 Seconds

Score :  893 / 1000
Grade : S

--------------------------------------------------------
File :  H:\Arena 2.01\Analyses.log

epd File :  STS7.epd   

Studied module :  Komodo-10-32-ja

Level:  10 seconds by position
Matching moves : 77 / 100
Grade : A
Total time : 12:17:49
Rated time : 311 Seconds

Score :  835 / 1000
Grade : A+

--------------------------------------------------------
File :  H:\Arena 2.01\Analyses.log

epd File :  STS8.epd   

Studied module :  Komodo-10-32-ja

Level:  10 seconds by position
Matching moves : 61 / 100
Grade : B
Total time : 12:17:56
Rated time : 465 Seconds

Score :  692 / 1000
Grade : B+

--------------------------------------------------------
File :  H:\Arena 2.01\Analyses.log

epd File :  STS9.epd   

Studied module :  Komodo-10-32-ja

Level:  10 seconds by position
Matching moves : 68 / 100
Grade : B+
Total time : 12:17:56
Rated time : 386 Seconds

Score :  789 / 1000
Grade : A

--------------------------------------------------------
File :  H:\Arena 2.01\Analyses.log

epd File :  STS1.epd   

Studied module :  Komodo-10-32-ja

Level:  10 seconds by position
Matching moves : 80 / 100
Grade : A+
Total time : 12:17:58
Rated time : 237 Seconds

Score :  854 / 1000
Grade : S

----------------------------------------------------------------------------------------------------------------------------------
                |  STS1  |  STS2  |  STS3  |  STS4  |  STS5  |  STS6  |  STS7  |  STS8  |  STS9  |
----------------------------------------------------------------------------------------------------------------------------------
Komodo-10-32-ja  |    80    |    79    |    82    |    76    |    73    |    77    |    77    |    61    |    68    |
----------------------------------------------------------------------------------------------------------------------------------
Grade              |    A+    |    A    |    A+    |    A    |    A-    |    A    |    A    |    B    |    B+    |
Score / 1000     |    854    |    776    |    885    |    841    |    821    |    893    |    835    |    692    |    789    |
Grade              |    S    |    A    |    S    |    A+    |    A+    |    S    |    A+    |    B+    |    A    |
Best Move :  673 / 900  74.77 %  Grade : A-
Total :           7386 / 9000  82.06 %  Grade : A+
----------------------------------------------------------------------------------------------------------------------------------

[swami]

Wish List: (I DON'T expect it sooner, I just want to post this for "Informational" purpose)

It's not Christmas every time

1. Align results properly in table. (Should be clearly readable)
2. Port the results to Webpage (Open as HTML)
3. List the RATED Time taken for each one of the test suites.
4. Calculate the TOTAL _Rated_ time taken of ALL test suites combined.

Anyone else has feature request?

Regards,
Swami

[Jimbo I]

Komodo 1.0 Sample Run with STS Stat 2.Exe

Code: Select all

File :  H:\Arena 2.01\Analyses.log

epd File :  STS2.epd   

Studied module :  Komodo-10-32-ja

Level:  10 seconds by position
Matching moves : 79 / 100
Grade : A
Total time : 12:17:56
Rated time : 253 Seconds

Score :  776 / 1000
Grade : A

--------------------------------------------------------
File :  H:\Arena 2.01\Analyses.log

epd File :  STS3.epd   

Studied module :  Komodo-10-32-ja

Level:  10 seconds by position
Matching moves : 82 / 100
Grade : A+
Total time : 12:17:56
Rated time : 239 Seconds

Score :  885 / 1000
Grade : S

--------------------------------------------------------
File :  H:\Arena 2.01\Analyses.log

epd File :  STS4.epd   

Studied module :  Komodo-10-32-ja

Level:  10 seconds by position
Matching moves : 76 / 100
Grade : A
Total time : 12:17:56
Rated time : 310 Seconds

Score :  841 / 1000
Grade : A+

--------------------------------------------------------
File :  H:\Arena 2.01\Analyses.log

epd File :  STS5.epd   

Studied module :  Komodo-10-32-ja

Level:  10 seconds by position
Matching moves : 73 / 100
Grade : A-
Total time : 12:17:56
Rated time : 283 Seconds

Score :  821 / 1000
Grade : A+

--------------------------------------------------------
File :  H:\Arena 2.01\Analyses.log

epd File :  STS6.epd   

Studied module :  Komodo-10-32-ja

Level:  10 seconds by position
Matching moves : 77 / 100
Grade : A
Total time : 12:17:57
Rated time : 282 Seconds

Score :  893 / 1000
Grade : S

--------------------------------------------------------
File :  H:\Arena 2.01\Analyses.log

epd File :  STS7.epd   

Studied module :  Komodo-10-32-ja

Level:  10 seconds by position
Matching moves : 77 / 100
Grade : A
Total time : 12:17:49
Rated time : 311 Seconds

Score :  835 / 1000
Grade : A+

--------------------------------------------------------
File :  H:\Arena 2.01\Analyses.log

epd File :  STS8.epd   

Studied module :  Komodo-10-32-ja

Level:  10 seconds by position
Matching moves : 61 / 100
Grade : B
Total time : 12:17:56
Rated time : 465 Seconds

Score :  692 / 1000
Grade : B+

--------------------------------------------------------
File :  H:\Arena 2.01\Analyses.log

epd File :  STS9.epd   

Studied module :  Komodo-10-32-ja

Level:  10 seconds by position
Matching moves : 68 / 100
Grade : B+
Total time : 12:17:56
Rated time : 386 Seconds

Score :  789 / 1000
Grade : A

--------------------------------------------------------
File :  H:\Arena 2.01\Analyses.log

epd File :  STS1.epd   

Studied module :  Komodo-10-32-ja

Level:  10 seconds by position
Matching moves : 80 / 100
Grade : A+
Total time : 12:17:58
Rated time : 237 Seconds

Score :  854 / 1000
Grade : S

----------------------------------------------------------------------------------------------------------------------------------
                |  STS1  |  STS2  |  STS3  |  STS4  |  STS5  |  STS6  |  STS7  |  STS8  |  STS9  |
----------------------------------------------------------------------------------------------------------------------------------
Komodo-10-32-ja  |    80    |    79    |    82    |    76    |    73    |    77    |    77    |    61    |    68    |
----------------------------------------------------------------------------------------------------------------------------------
Grade              |    A+    |    A    |    A+    |    A    |    A-    |    A    |    A    |    B    |    B+    |
Score / 1000     |    854    |    776    |    885    |    841    |    821    |    893    |    835    |    692    |    789    |
Grade              |    S    |    A    |    S    |    A+    |    A+    |    S    |    A+    |    B+    |    A    |
Best Move :  673 / 900  74.77 %  Grade : A-
Total :           7386 / 9000  82.06 %  Grade : A+
----------------------------------------------------------------------------------------------------------------------------------

Hi Swami,
I like the program, but one question. I've yet to figure out what the grade S means. Can you please let me in on the secret?

[noctiferus]

Swami, is it ok what I suggested in my PM to you?

await for a new correct dataset, while I'm trying to fix different methodologies, or do you want a quick and dirty preliminary result? (maybe slightly overoptimistic?)
ciao
e.