AH_LTC Match: Komodo TCEC vs Houdini 4

[IWB]

Hello Aser,

I don't know if this is discussed in on of the other postings. If so please ignore it.

I did not check your openings in particular but their distribution is a bit one sided:


Open positions (No. 16) 11%
Half positions (No. 45) 30%
Closed positions (No. 89) 59%
(with open(C20-C99), closed(A,D,E) and half open (B + C00-C19) positions)

By chance I made some analysis a few days ago for my list and think 11% open positions is not that good (And I dont think it has to be a 1/3 for all!!!) But 11% just because the GM mainly played Sicilian the last years!?!?

Now shortening your results to just a one on one comparision it look like this:

Code: Select all

1 Stockfish DD              :   600 (+145,=336,-119), 52.2 %

Houdini 4                     : 300 (+ 82,=155,- 63), 53.2 %
Komodo TCEC                   : 300 (+ 63,=181,- 56), 51.2 %

2 Komodo TCEC               :   600 (+133,=336,-131), 50.2 %

Houdini 4                     : 300 (+ 77,=155,- 68), 51.5 %
Stockfish DD                  : 300 (+ 56,=181,- 63), 48.8 %

3 Houdini 4                 :   600 (+131,=310,-159), 47.7 %

Stockfish DD                  : 300 (+ 63,=155,- 82), 46.8 %
Komodo TCEC                   : 300 (+ 68,=155,- 77), 48.5 %

That is 4.5% between the best and the "worst" engine with a error bar of about +/- 20 Elo.

Looking at my own list and seeing that Stockfish is particular bad at open positions (I have a few more games and can make a rating list just out of those), then seeing how close your list is I hesitate to condense the good result of Stockfish "JUST" to the longer time control.

Just some thoughts.

Bye
Ingo

[Aser Huerga]

Thanks Ingo for your comments. When I started to design these testings I spend some time thinking about the best opening suite of around 150 positions, and I concluded that is VERY difficult to create a perfect opening suite, because all approaches have their faults and there are a lot of variables to take in account (databases to use, depth, draw percentages, engine evaluations ...)

I was considering to choose selected 960 positions, but I prefer to test conventional openings. So finally I rejected the idea of make a "perfect" testing opening suite and I choosed a different and original approach: I'm not looking for "balanced" openings (whatever this means, I left this almost impossible task to others ), but for the most played openings/variations from the last years. So once the database is choosed (TWIC 2400+ games) all is a matter of percentages and some tunning to avoid transpositions. I think this way, the only biasing factor is the tendencies of actual chess theory and no other controversial factor.

As the variations depth is very short in most cases, the result positions are a wide range of well known basic positions. So at least my tests will show which engines plays best the most played theory, at the less possible depth.

Let me use the occasion to tell you that your test, although pretty different, was very interesting to me, and I hope you will be back for testing in the future.

Best wishes.

[ouachita]

I did not check your openings in particular but their distribution is a bit one sided. Bye Ingo

Can you recommend a suite or book that you believe is better balanced?

[IWB]

I did not check your openings in particular but their distribution is a bit one sided. Bye Ingo

Can you recommend a suite or book that you believe is better balanced?

No, as ASER already said, there is no perfect test suit! (What is correct? 15,20,25% open positions?)

If you find something that is good for one engine it hurts another. I am just mentioned this as I know that Stockfish is worse than others in open positions and I dont want to claim the good result here "JUST" to the long time control ...

At the end it all comes down to what YOU like. I can only ask everyone to look at the openings of all lists. Just a basic classification (as I did with ASERS opening) shows interesting things for the other lists as well ... (and with the big blitz lists you can find some interesting differences).
But again, make your own classification and check the lists then.

Do not just look at time but check openings, ponder, hardware, Tbs, ... many things influence a result. Asers Test would be more conclusive if it would be more games, more opponents and repeated with the same openings at 1/10 of the time (e.g)

In short: A list is good for exactly what it tested and it is hard to be compared with others!

BYe
Ingo

[IWB]

I concluded that is VERY difficult to create a perfect opening suite,

Not difficult, impossible! It is a matter of taste ... but one has to be carefull concluding something based on ONE parameter (in this case time)

...
Let me use the occasion to tell you that your test, although pretty different, was very interesting to me, and I hope you will be back for testing in the future.

Maybe 2014.

Bye
Ingo

[ouachita]

At the end it all comes down to what YOU like.

Noomen & Nunn (a London law firm or? )

[IWB]

Looking at my own list and seeing that Stockfish is particular bad at open positions (I have a few more games and can make a rating list just out of those), then seeing how close your list is I hesitate to condense the good result of Stockfish "JUST" to the longer time control.

Just some thoughts.

I have to be more precise here: I know that SF perfomes worse tthan others in open positions at 5 + 3! I assume that this is the same at 90 + 30 but I might be completly wrong ... to definately say this it would be nice to have a couple of thousand games more to make three lists at 90 30 for closed, open and half open positions ...

Thats better!

I just wanted to seed some doubt

Bye
Ingo

[Aser Huerga]

In short: A list is good for exactly what it tested and it is hard to be compared with others!

That's the honest truth ... no more no less.

Asers Test would be more conclusive if it would be more games, more opponents and repeated with the same openings at 1/10 of the time (e.g)

I plan to eventually include 4 or 5 engines in each class. And I have repeated the games at different TCs, although I doubt I will do it again in the future: http://www.talkchess.com/forum/viewtopi ... highlight=

Maybe 2014.

(please, could someone add the popcorn emoticon? )

[ouachita]

I assume that this is the same at 90 + 30 but I might be completely wrong Ingo

Few here demonstrate any hesitation with chess match data interpolation or extrapolation, so feel free to make any assumption you like because we will likely never know for certain.

[michiguel]

In short: A list is good for exactly what it tested and it is hard to be compared with others!

That's the honest truth ... no more no less.

Asers Test would be more conclusive if it would be more games, more opponents and repeated with the same openings at 1/10 of the time (e.g)

I plan to eventually include 4 or 5 engines in each class. And I have repeated the games at different TCs, although I doubt I will do it again in the future: http://www.talkchess.com/forum/viewtopi ... highlight=

Maybe 2014.

(please, could someone add the popcorn emoticon? )

Miguel