Komodo: strongest in opening. Stockfish: middle&endgame

[Laskos]

Using adjudication rules of Cutechess-Cli, I attempted to see the swings from balanced openings/middlegames/endgames for Komodo 10 against Stockfish 20 May 2016 dev.

The conclusions are: Komodo 10 manages to swing in its favor more than twice as many games as Stockfish in the balanced early openings to at least 60cp advantage. Stockfish swings almost twice as many games in its favor in the middlegame to at least 70cp. The same in endgames, where Stockfish is twice more likely to swing a balanced endgame to a win.

Here are the results with Cutechess-Cli command line included:

Balanced early opening positions:

Code: Select all

draw movenumber=12 movecount=1 score=2000 -resign movecount=3 score=60

Score of K10 vs SF: 146 - 62 - 792  [0.542] 1000
ELO difference: 29.25 +/- 9.71
Finished match

Balanced middlegame positions:

Code: Select all

draw movenumber=12 movecount=1 score=2000 -resign movecount=3 score=70

Score of K10 vs SF: 81 - 128 - 791  [0.476] 1000
ELO difference: -16.34 +/- 9.80
Finished match

Balanced endgame positions:

Code: Select all

draw movenumber=120 movecount=10 score=20 -resign movecount=5 score=900

Score of K10 vs SF: 23 - 40 - 937  [0.491] 1000
ELO difference: -5.91 +/- 5.39
Finished match

[JJJ]

So, it seems you re saying the opposite of what we said in other thread.

[mjlef]

I do not understand how you are defining middle game. The text says:

draw movenumber=12 movecount=1 score=2000 -resign movecount=3 score=70

Which, other than a score change to 70 from 60, seems to have the same move number. Is that a typo?

Could you tell us how you defined middle game?

Interesting results. Ed's endgame tournament currently has Komodo 30 elo above SF, which appears the opposite of your findings.

Mark

[Laskos]

I do not understand how you are defining middle game. The text says:

draw movenumber=12 movecount=1 score=2000 -resign movecount=3 score=70

Which, other than a score change to 70 from 60, seems to have the same move number. Is that a typo?

Could you tell us how you defined middle game?

Interesting results. Ed's endgame tournament currently has Komodo 30 elo above SF, which appears the opposite of your findings.

Mark

I built a balanced (-0.3 to 0.3 eval) middlegame EPD test-suite, and played from these middlegame positions for 12 moves. If both engines reach 0.70 or above, the game is adjudicated as win (respectively loss). If after 12 (or 13) moves the game is not adjudicated as win/loss, it is adjudicated as draw. Therefore only middlegame swings are counted. Similarly with early openings, from the EPD of balanced early openings, the games are played for 12 moves to detect swings of eval (0.60 threshold here).

[Laskos]

So, it seems you re saying the opposite of what we said in other thread.

Just now I observed it. Would be interesting to interpret Frank's findings, simple fact that an engine wins by a certain move is not enough. About "impressions" of Anil I have reserves.

[Laskos]

Interesting results. Ed's endgame tournament currently has Komodo 30 elo above SF, which appears the opposite of your findings.

Mark

My results (for endgames too) are from balanced positions, not exactly tournament conditions. I am searching for swings from balanced to clear advantage in opening and respectively middlegame. Wins are real wins for endgames, because playing to the end is also endgame.

[shrapnel]

So, it seems you re saying the opposite of what we said in other thread.

Just now I observed it. Would be interesting to interpret Frank's findings, simple fact that an engine wins by a certain move is not enough. About "impressions" of Anil I have reserves.

Hi Kai
Maybe we are both right. My 'impressions' were from using Komodo 10 from the Black side, which I still maintain is disastrous.
But today I won a couple of difficult online games using Komodo from the White side !
So, still not sure what is going on...
Early days yet I suppose.
Anyway, please continue, I generally find your observations very useful in my online games !

[shrapnel]

My results (for endgames too) are from balanced positions, not exactly tournament conditions. I am searching for swings from balanced to clear advantage in opening and respectively middlegame. Wins are real wins for endgames, because playing to the end is also endgame.

I agree with your end-game results.
No way komodo 10 is stronger than SF in end-game.

[Norm Pollock]

It would be interesting if someone came up with a hybrid-type GUI that SWITCHES engines for each side after "x" moves, while giving the "replacement" engine sufficient time to "warm-up".

If we had that type of GUI, then a lot of experimenting regarding beginning/middle/end game would be possible.

I recently read something along similar lines concerning expertise of particular pieces. Fischer supposedly was the best in history handling the bishop. So maybe we could also find a way to experiment finding which engine is the best for each piece.

[Laskos]

It would be interesting if someone came up with a hybrid-type GUI that SWITCHES engines for each side after "x" moves, while giving the "replacement" engine sufficient time to "warm-up".

If we had that type of GUI, then a lot of experimenting regarding beginning/middle/end game would be possible.

You caught me on the scene, right now I was experimenting with "ChessCombi V1" which combines 2 UCI chess engines and is obeying UCI protocol. I tried to "build" the strongest engine by combining Komodo opening with the middlegame of Stockfish, but I am bit at odds with the config file, I cannot seem to manage well the succession of engines. Also, the NPS of the resulting hybrid is significantly lower than that of both SF and Komodo, so it's not really a solution to "Engine Room" players. A well written GUI would be nice indeed.

I recently read something along similar lines concerning expertise of particular pieces. Fischer supposedly was the best in history handling the bishop. So maybe we could also find a way to experiment finding which engine is the best for each piece.