I think it still on my hard disk drive.Henryval wrote:where is this tool to download?
looking for really professional statistic-tools
Moderators: hgm, Rebel, chrisw
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: looking for really professional statistic-tools
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: looking for really professional statistic-tools
Ferdi, you seem good in parsing PGN files. I had in mind for some time to write a tool for PGN files (cutechess-cli format) which does the following:Ferdy wrote:I think it still on my hard disk drive.Henryval wrote:where is this tool to download?
1/ Extract engine name in each game.
2/ Extract the outcome of the game.
3/ Extract its evaluation for each move.
4/ For move by move evals of that engine in the range [a,b], say [+1.20,+1.25], compute performance of that engine in the entire PGN file, which is (win+0.5*draw)/(win+draw+loss), given the window of the evaluation and the outcomes.
I wished to do that in order to plot the performance of engines (against equal opponent) as a function of their evaluations. Do you think it's easy for you to do that? I don't care about GUI, command line or a batch file would do.
-
- Posts: 42
- Joined: Thu Jan 09, 2014 2:55 pm
Re: looking for really professional statistic-tools
How much are you willing to pay?Henryval wrote: i am a customer, willing to pay for a program that fullfill my needs.
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: looking for really professional statistic-tools
I don't get (4), we get the game result, so we have win, loss and draw for every player in that pgn file, say we get the eval of every move of every game played by a certain engine, so what is the next procedure? Can you explain more, better if you can give an example.Laskos wrote: 1/ Extract engine name in each game.
2/ Extract the outcome of the game.
3/ Extract its evaluation for each move.
4/ For move by move evals of that engine in the range [a,b], say [+1.20,+1.25], compute performance of that engine in the entire PGN file, which is (win+0.5*draw)/(win+draw+loss), given the window of the evaluation and the outcomes.
I wished to do that in order to plot the performance of engines (against equal opponent) as a function of their evaluations. Do you think it's easy for you to do that? I don't care about GUI, command line or a batch file would do.
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: looking for really professional statistic-tools
Say you set a target eval window for Komodo between +1.20 and +1.25. You begin reading one game: Komodo is playing white. Outcome is 1-0. Inside this game Komodo had 3 moves which it evaluated as inside [+1.20, +1.25] window. So inside this game count "3" with outcome 1-0.Ferdy wrote:I don't get (4), we get the game result, so we have win, loss and draw for every player in that pgn file, say we get the eval of every move of every game played by a certain engine, so what is the next procedure? Can you explain more, better if you can give an example.Laskos wrote: 1/ Extract engine name in each game.
2/ Extract the outcome of the game.
3/ Extract its evaluation for each move.
4/ For move by move evals of that engine in the range [a,b], say [+1.20,+1.25], compute performance of that engine in the entire PGN file, which is (win+0.5*draw)/(win+draw+loss), given the window of the evaluation and the outcomes.
I wished to do that in order to plot the performance of engines (against equal opponent) as a function of their evaluations. Do you think it's easy for you to do that? I don't care about GUI, command line or a batch file would do.
The next game inside the PGN is a draw, and Komodo had 2 moves within [+1.20, +1.25] eval window. Inside this game count "2" with outcome 1/2-1/2.
Add counts for each game inside the Komodo's eval window [+1.20, +1.25], for the whole PGN (say 300 games). Added counts for all games might be 600 wins for Komodo, 400 draws and 100 losses. The performance of Komodo in the eval window [+1.20, +1.25] is then (600+400*0.5)/1100 ~ 73%.
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: looking for really professional statistic-tools
Now I get it. I will try to make such tool.Laskos wrote:Say you set a target eval window for Komodo between +1.20 and +1.25. You begin reading one game: Komodo is playing white. Outcome is 1-0. Inside this game Komodo had 3 moves which it evaluated as inside [+1.20, +1.25] window. So inside this game count "3" with outcome 1-0.Ferdy wrote:I don't get (4), we get the game result, so we have win, loss and draw for every player in that pgn file, say we get the eval of every move of every game played by a certain engine, so what is the next procedure? Can you explain more, better if you can give an example.Laskos wrote: 1/ Extract engine name in each game.
2/ Extract the outcome of the game.
3/ Extract its evaluation for each move.
4/ For move by move evals of that engine in the range [a,b], say [+1.20,+1.25], compute performance of that engine in the entire PGN file, which is (win+0.5*draw)/(win+draw+loss), given the window of the evaluation and the outcomes.
I wished to do that in order to plot the performance of engines (against equal opponent) as a function of their evaluations. Do you think it's easy for you to do that? I don't care about GUI, command line or a batch file would do.
The next game inside the PGN is a draw, and Komodo had 2 moves within [+1.20, +1.25] eval window. Inside this game count "2" with outcome 1/2-1/2.
Add counts for each game inside the Komodo's eval window [+1.20, +1.25], for the whole PGN (say 300 games). Added counts for all games might be 600 wins for Komodo, 400 draws and 100 losses. The performance of Komodo in the eval window [+1.20, +1.25] is then (600+400*0.5)/1100 ~ 73%.
Perhaps asks the user to define window limits, then either asks the user what engine name or extract all engine performances according to the single window defined by the user.
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: looking for really professional statistic-tools
Great Ferdi, thanks! I oscillated for a month writing a tool, but never quite composed myself to do it.Ferdy wrote:Now I get it. I will try to make such tool.Laskos wrote:Say you set a target eval window for Komodo between +1.20 and +1.25. You begin reading one game: Komodo is playing white. Outcome is 1-0. Inside this game Komodo had 3 moves which it evaluated as inside [+1.20, +1.25] window. So inside this game count "3" with outcome 1-0.Ferdy wrote:I don't get (4), we get the game result, so we have win, loss and draw for every player in that pgn file, say we get the eval of every move of every game played by a certain engine, so what is the next procedure? Can you explain more, better if you can give an example.Laskos wrote: 1/ Extract engine name in each game.
2/ Extract the outcome of the game.
3/ Extract its evaluation for each move.
4/ For move by move evals of that engine in the range [a,b], say [+1.20,+1.25], compute performance of that engine in the entire PGN file, which is (win+0.5*draw)/(win+draw+loss), given the window of the evaluation and the outcomes.
I wished to do that in order to plot the performance of engines (against equal opponent) as a function of their evaluations. Do you think it's easy for you to do that? I don't care about GUI, command line or a batch file would do.
The next game inside the PGN is a draw, and Komodo had 2 moves within [+1.20, +1.25] eval window. Inside this game count "2" with outcome 1/2-1/2.
Add counts for each game inside the Komodo's eval window [+1.20, +1.25], for the whole PGN (say 300 games). Added counts for all games might be 600 wins for Komodo, 400 draws and 100 losses. The performance of Komodo in the eval window [+1.20, +1.25] is then (600+400*0.5)/1100 ~ 73%.
Perhaps asks the user to define window limits, then either asks the user what engine name or extract all engine performances according to the single window defined by the user.
Be careful with the PGN output of cutechess-cli, I think by default (not opting for White POV), it's engines' POV of eval.
-
- Posts: 48
- Joined: Sun Jan 06, 2013 12:47 pm
- Location: Germany
Re: looking for really professional statistic-tools
hi ferdy, so if this is not an official tool, please can you send it to me by mail?
Don't let Your tricky mind kick You out of paradise!
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: looking for really professional statistic-tools
This is not done yet sorry.Henryval wrote:hi ferdy, so if this is not an official tool, please can you send it to me by mail?
-
- Posts: 4833
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: looking for really professional statistic-tools
I have a working version now, just doing some checks on the output. I will be releasing this shortly for you to test as well.
Sample.
Sample.
Code: Select all
Players:
Houdini 4.0 x64
Gull 3 x64
Sf5
Sf6
Fire_4_x64
num_games 182
input min eval window? 1.20
input max eval window? 1.30
window [1.20, 1.30]
Evaluating Houdini 4.0 x64
Reading game 1
wp Houdini 4.0 x64
bp Gull 3 x64
res 1/2-1/2
player Houdini 4.0 x64 is found
1. { book }
2. { book }
3. { book }
4. { book }
5. { book }
6. { book }
7. { book }
8. { book }
9. { +0.16 }
10. { +0.12 }
11. { +0.18 }
12. { +0.06 }
13. { +0.07 }
14. { +0.17 }
[...]
54... { +0.89 }
55... { +0.77 }
56... { +0.89 }
57... { +1.18 }
58... { +1.11 }
59... { +1.19 }
60... { +1.25 }
black score (1.25) is inside the window [1.20, 1.30]
61... { +1.22 }
black score (1.22) is inside the window [1.20, 1.30]
62... { +1.39 }
63... { +1.47 }
64... { +1.54 }
65... { +1.88 }
66... { +3.06 }
67... { +10.06 }
68... { +8.66 }
69... { +327.23 }
70... { +327.51 }
71... { +327.53 }
72... { +327.55 }
73... { +327.57 }
Reading game 38
wp Gull 3 x64
bp Sf6
res 0-1
player Gull 3 x64 is found
1. { book }
2. { book }
3. { book }
[...]
Reading game 139
wp Sf6
bp Fire_4_x64
res 1/2-1/2
player Gull 3 x64 is not found, skip this game
Reading game 140
wp Fire_4_x64
bp Sf6
res 0-1
player Gull 3 x64 is not found, skip this game
Reading game 141
wp Sf5
bp Gull 3 x64
res 1/2-1/2
player Gull 3 x64 is found
1... { book }
2... { book }
3... { book }
4... { book }
5... { book }
6... { book }
7... { book }
8... { book }
9... { -0.19 }
10... { -0.19 }
11... { -0.04 }
12... { -0.05 }
[...]
players Gcnt Wcnt Lcnt Dcnt perf
Houdini 4.0 x64 51 21 6 24 64.71%
Gull 3 x64 34 19 1 14 76.47%
Sf5 26 5 0 21 59.62%
Sf6 68 67 0 1 99.26%
Fire_4_x64 4 3 1 0 75.00%
Done!! t = 296.1s (4.9m)