looking for really professional statistic-tools

[Ferdy]

where is this tool to download?

I think it still on my hard disk drive.

[Laskos]

where is this tool to download?

I think it still on my hard disk drive.

Ferdi, you seem good in parsing PGN files. I had in mind for some time to write a tool for PGN files (cutechess-cli format) which does the following:

1/ Extract engine name in each game.
2/ Extract the outcome of the game.
3/ Extract its evaluation for each move.
4/ For move by move evals of that engine in the range [a,b], say [+1.20,+1.25], compute performance of that engine in the entire PGN file, which is (win+0.5*draw)/(win+draw+loss), given the window of the evaluation and the outcomes.

I wished to do that in order to plot the performance of engines (against equal opponent) as a function of their evaluations. Do you think it's easy for you to do that? I don't care about GUI, command line or a batch file would do.

[Dhanish]

i am a customer, willing to pay for a program that fullfill my needs.

How much are you willing to pay?

[Ferdy]

1/ Extract engine name in each game.
2/ Extract the outcome of the game.
3/ Extract its evaluation for each move.
4/ For move by move evals of that engine in the range [a,b], say [+1.20,+1.25], compute performance of that engine in the entire PGN file, which is (win+0.5*draw)/(win+draw+loss), given the window of the evaluation and the outcomes.

I wished to do that in order to plot the performance of engines (against equal opponent) as a function of their evaluations. Do you think it's easy for you to do that? I don't care about GUI, command line or a batch file would do.

I don't get (4), we get the game result, so we have win, loss and draw for every player in that pgn file, say we get the eval of every move of every game played by a certain engine, so what is the next procedure? Can you explain more, better if you can give an example.

[Laskos]

1/ Extract engine name in each game.
2/ Extract the outcome of the game.
3/ Extract its evaluation for each move.
4/ For move by move evals of that engine in the range [a,b], say [+1.20,+1.25], compute performance of that engine in the entire PGN file, which is (win+0.5*draw)/(win+draw+loss), given the window of the evaluation and the outcomes.

I wished to do that in order to plot the performance of engines (against equal opponent) as a function of their evaluations. Do you think it's easy for you to do that? I don't care about GUI, command line or a batch file would do.

I don't get (4), we get the game result, so we have win, loss and draw for every player in that pgn file, say we get the eval of every move of every game played by a certain engine, so what is the next procedure? Can you explain more, better if you can give an example.

Say you set a target eval window for Komodo between +1.20 and +1.25. You begin reading one game: Komodo is playing white. Outcome is 1-0. Inside this game Komodo had 3 moves which it evaluated as inside [+1.20, +1.25] window. So inside this game count "3" with outcome 1-0.

The next game inside the PGN is a draw, and Komodo had 2 moves within [+1.20, +1.25] eval window. Inside this game count "2" with outcome 1/2-1/2.

Add counts for each game inside the Komodo's eval window [+1.20, +1.25], for the whole PGN (say 300 games). Added counts for all games might be 600 wins for Komodo, 400 draws and 100 losses. The performance of Komodo in the eval window [+1.20, +1.25] is then (600+400*0.5)/1100 ~ 73%.

[Ferdy]

1/ Extract engine name in each game.
2/ Extract the outcome of the game.
3/ Extract its evaluation for each move.
4/ For move by move evals of that engine in the range [a,b], say [+1.20,+1.25], compute performance of that engine in the entire PGN file, which is (win+0.5*draw)/(win+draw+loss), given the window of the evaluation and the outcomes.

I wished to do that in order to plot the performance of engines (against equal opponent) as a function of their evaluations. Do you think it's easy for you to do that? I don't care about GUI, command line or a batch file would do.

I don't get (4), we get the game result, so we have win, loss and draw for every player in that pgn file, say we get the eval of every move of every game played by a certain engine, so what is the next procedure? Can you explain more, better if you can give an example.

Say you set a target eval window for Komodo between +1.20 and +1.25. You begin reading one game: Komodo is playing white. Outcome is 1-0. Inside this game Komodo had 3 moves which it evaluated as inside [+1.20, +1.25] window. So inside this game count "3" with outcome 1-0.

The next game inside the PGN is a draw, and Komodo had 2 moves within [+1.20, +1.25] eval window. Inside this game count "2" with outcome 1/2-1/2.

Add counts for each game inside the Komodo's eval window [+1.20, +1.25], for the whole PGN (say 300 games). Added counts for all games might be 600 wins for Komodo, 400 draws and 100 losses. The performance of Komodo in the eval window [+1.20, +1.25] is then (600+400*0.5)/1100 ~ 73%.

Now I get it. I will try to make such tool.
Perhaps asks the user to define window limits, then either asks the user what engine name or extract all engine performances according to the single window defined by the user.

[Laskos]

1/ Extract engine name in each game.
2/ Extract the outcome of the game.
3/ Extract its evaluation for each move.
4/ For move by move evals of that engine in the range [a,b], say [+1.20,+1.25], compute performance of that engine in the entire PGN file, which is (win+0.5*draw)/(win+draw+loss), given the window of the evaluation and the outcomes.

I wished to do that in order to plot the performance of engines (against equal opponent) as a function of their evaluations. Do you think it's easy for you to do that? I don't care about GUI, command line or a batch file would do.

I don't get (4), we get the game result, so we have win, loss and draw for every player in that pgn file, say we get the eval of every move of every game played by a certain engine, so what is the next procedure? Can you explain more, better if you can give an example.

Say you set a target eval window for Komodo between +1.20 and +1.25. You begin reading one game: Komodo is playing white. Outcome is 1-0. Inside this game Komodo had 3 moves which it evaluated as inside [+1.20, +1.25] window. So inside this game count "3" with outcome 1-0.

The next game inside the PGN is a draw, and Komodo had 2 moves within [+1.20, +1.25] eval window. Inside this game count "2" with outcome 1/2-1/2.

Add counts for each game inside the Komodo's eval window [+1.20, +1.25], for the whole PGN (say 300 games). Added counts for all games might be 600 wins for Komodo, 400 draws and 100 losses. The performance of Komodo in the eval window [+1.20, +1.25] is then (600+400*0.5)/1100 ~ 73%.

Now I get it. I will try to make such tool.
Perhaps asks the user to define window limits, then either asks the user what engine name or extract all engine performances according to the single window defined by the user.

Great Ferdi, thanks! I oscillated for a month writing a tool, but never quite composed myself to do it.

Be careful with the PGN output of cutechess-cli, I think by default (not opting for White POV), it's engines' POV of eval.

[Henryval]

hi ferdy, so if this is not an official tool, please can you send it to me by mail?

[Ferdy]

hi ferdy, so if this is not an official tool, please can you send it to me by mail?

This is not done yet sorry.

[Ferdy]

I have a working version now, just doing some checks on the output. I will be releasing this shortly for you to test as well.

Sample.

Code: Select all

Players:
Houdini 4.0 x64
Gull 3 x64
Sf5
Sf6
Fire_4_x64

num_games 182

input min eval window? 1.20
input max eval window? 1.30
window [1.20, 1.30]

Evaluating Houdini 4.0 x64

Reading game 1
wp Houdini 4.0 x64
bp Gull 3 x64
res 1/2-1/2
player Houdini 4.0 x64 is found
1. { book }
2. { book }
3. { book }
4. { book }
5. { book }
6. { book }
7. { book }
8. { book }
9. { +0.16 }
10. { +0.12 }
11. { +0.18 }
12. { +0.06 }
13. { +0.07 }
14. { +0.17 }

[...]

54... { +0.89 }
55... { +0.77 }
56... { +0.89 }
57... { +1.18 }
58... { +1.11 }
59... { +1.19 }
60... { +1.25 }
black score (1.25) is inside the window [1.20, 1.30]
61... { +1.22 }
black score (1.22) is inside the window [1.20, 1.30]
62... { +1.39 }
63... { +1.47 }
64... { +1.54 }
65... { +1.88 }
66... { +3.06 }
67... { +10.06 }
68... { +8.66 }
69... { +327.23 }
70... { +327.51 }
71... { +327.53 }
72... { +327.55 }
73... { +327.57 }

Reading game 38
wp Gull 3 x64
bp Sf6
res 0-1
player Gull 3 x64 is found
1. { book }
2. { book }
3. { book }

[...]

Reading game 139
wp Sf6
bp Fire_4_x64
res 1/2-1/2
player Gull 3 x64 is not found, skip this game

Reading game 140
wp Fire_4_x64
bp Sf6
res 0-1
player Gull 3 x64 is not found, skip this game

Reading game 141
wp Sf5
bp Gull 3 x64
res 1/2-1/2
player Gull 3 x64 is found
1... { book }
2... { book }
3... { book }
4... { book }
5... { book }
6... { book }
7... { book }
8... { book }
9... { -0.19 }
10... { -0.19 }
11... { -0.04 }
12... { -0.05 }

[...]

                       players      Gcnt     Wcnt     Lcnt     Dcnt     perf
               Houdini 4.0 x64        51       21        6       24   64.71%
                    Gull 3 x64        34       19        1       14   76.47%
                           Sf5        26        5        0       21   59.62%
                           Sf6        68       67        0        1   99.26%
                    Fire_4_x64         4        3        1        0   75.00%
Done!! t = 296.1s (4.9m)