Houdini 3 - Shredder 7.04 : who understands chess ?

[Tennison]

This post give me an idea : testing where are the real improvments in 10 years.

First, to do this, I try to eliminate these parameters :

- 32 bits / 64 bits distorsion
- nodes/second distorsion
- single / multi core(s) distorsion

So the first step is to see if Houdini 3 really understands better chess.

-------------------------------------------------------------------------------------
GUI : Fritz 11
Table bases 5 men : ON
Hash : 128 MB
Opening book : HS7movesplus.ctg
-------------------------------------------------------------------------------------

Test 1 : Fixed Depth = 8

1 Shredder 7.04 +7/-2/=11 62.50% 12.5/20
2 Houdini 3 +2/-7/=11 37.50% 7.5/20

the games


Test 2 : Fixed Depth = 12

1 Shredder 7.04 +1/-0/=3 62.50% 2.5/4
2 Houdini 3 +0/-1/=3 37.50% 1.5/4

the games

These 2 short tests seems to give a real plus to Shredder 7.04, but to reach the same depth Shredder needs a lot more time. For example, in last game (test 2) Shredder uses more than 20 minutes and Houdini less than 30 secondes.

I can't really conclude anything in so few games but I think there is two possibilities to explain Shredder success :
- Shredder better understanding
or
- to reach quickly high depth, Houdini cuts the lines to fast... It's very efficient in time control games but not in depth control games.


Anybody to discuss about this ?

Maybe I'm completely wrong but I'm really interested in hearing you ...

[Houdini]

1) Fixed depth tests are completely meaningless.
The definition of "search depth" is different for every engine.

2) Engines don't have any "chess understanding", they only have "chess strength".
Generally speaking your computer doesn't have any "understanding", it just computes bits and bytes.

Robert

[Tennison]

1) Fixed depth tests are completely meaningless.
The definition of "search depth" is different for every engine.

2) Engines don't have any "chess understanding", they only have "chess strength".
Generally speaking your computer doesn't have any "understanding", it just computes bits and bytes.

Robert

Thanks Robert for a so quick response . You never sleep ?

1) How "search depth" can be a so different thing ? Maybe I'm wrong but, for me, a fixed depth stays a fixed depth. Can you explain to me ?

2) Yes it's not a real "understanding" but a lot of knowledge are inserted in the source code ... Am I wrong too ?

[Houdini]

1) Modern engines purposely perform a lot of pruning in the final plies of the search. That means that "search depth 8" in Shredder 7 is probably equivalent to "search depth 13" or maybe even higher in Houdini 3. It doesn't make sense to say that Shredder 7 has better "knowledge" because it simply uses less pruning in the final plies of the search.

2) How can you evaluate the "knowledge in the code" other than by looking at the strength of the engine?
The chess "knowledge" resides in all the parts of the engine, not just in the evaluation. There's a lot of chess "knowledge" in the advanced move selection and pruning methods of Houdini, as well in all the other aspects like the "ponder" time management, or even inside the Tactical Mode of Houdini.

Robert

[yanquis1972]

the problem of the test is huge in that none of those 3 are in fact distortions. kn/s, depth, & even 32/64 bit are all simply part of the engine. barring a -complete- rewrite, shredder 13 will also be 32 bit...S12 had a 64 bit option with no strength improvement. HIARCS also tried a 64 bit engine some time ago with no success.

so in essence, you're just massively crippling houdini here.

i might suggest something like x seconds/move (longer being better), & on older hardware (where houdini can't take advantage of certain advanced instructions which weren't available to shredder).

[Tennison]

For me calculation time is no more a useable playground :
If an engine can play 50 moves in 5 secondes but can't reach more than depth 6 and another one play 1 move every second (due to no optimized source (no asm, no popcnt, no multicore, ...) but can reach depth 20 ... How can we compare them.

There is always bias when comparing something because you always keep some params and forget some others.

For example :

I (+/- 1900 fide) frequently beat a 2200 FIDE player in blitz games (scores are like 8-2 or 9-1). But I always lose on long games.
Am I better than him because I think quicker ? No because his real strenght is to reach deeper lines than me on long games.

It's the goal of this test ...

[yanquis1972]

this is quite silly though. depth can be faked, artificial, or misleading, depending on how you look at it. a perfect example is rybka. i think its commonly believed she underreports depth by about ~3.

as an end user, i care about time to preferred result...what do 'depth' or 'kilonodes' mean to me? those are just numbers...the reason i suggest seconds/move is because modern engines may also have significantly better time management (no foundation for this). they are also, afaik, optimized using hyperbullet these days, so it makes sense they would excel increasingly against older programs as time/move decreases.

with humans, blitz will tend to 'equalize' things, but since computers don't make mistakes as humans do, i think it works the opposite. at much longer T/C, provided equal time management, shredder would almost surely do better, as it wouldn't be so drastically outsearched. it would of course be thoroughly thrashed, & i would be very surprised if it won a game out of 100 assuming a fair book.

another good example is the original ippolit from a few years back (robbolito). it was heavily optimized for bullet chess and was able to defeat R3 routinely at very fast time controls. i think this was generally attributed to more aggressive pruning and/or better time management. as the time control slowed down however, strength was found to be about equal (iirc). to my layman's eye, the faster depth robbo was getting was negated by rybka's wider search. again, this can simply be tested by LTC or seconds/move, but depth is a horrible idea. why not equalize kn/s as well?

[overlord]

Well, it is little bit irrelevant because it is not testing under equal conditions. Reaching deep depths is essencial part of every engine. It is the same for e.g. Zappa - try to play matches for fixed depth 8 ply against other engines...and maybe it will best engine of all times (if I accept your way of thinking). Anyway thanks for posting it, it is really interesting...

[overlord]

I completly undertand you. I have also one colleague in our team who is 2100 ELO player in classical time control, but in blitz games he beats even GM and IM vere often. Anyway classical and blitz games are ganes played under equal conditions. In this test had Shredder much more time to think so the test was simply irrelevant.

[S.Taylor]

This post give me an idea : testing where are the real improvments in 10 years.

First, to do this, I try to eliminate these parameters :

- 32 bits / 64 bits distorsion
- nodes/second distorsion
- single / multi core(s) distorsion

So the first step is to see if Houdini 3 really understands better chess.

-------------------------------------------------------------------------------------
GUI : Fritz 11
Table bases 5 men : ON
Hash : 128 MB
Opening book : HS7movesplus.ctg
-------------------------------------------------------------------------------------

Test 1 : Fixed Depth = 8

1 Shredder 7.04 +7/-2/=11 62.50% 12.5/20
2 Houdini 3 +2/-7/=11 37.50% 7.5/20

the games


Test 2 : Fixed Depth = 12

1 Shredder 7.04 +1/-0/=3 62.50% 2.5/4
2 Houdini 3 +0/-1/=3 37.50% 1.5/4

the games

These 2 short tests seems to give a real plus to Shredder 7.04, but to reach the same depth Shredder needs a lot more time. For example, in last game (test 2) Shredder uses more than 20 minutes and Houdini less than 30 secondes.

I can't really conclude anything in so few games but I think there is two possibilities to explain Shredder success :
- Shredder better understanding
or
- to reach quickly high depth, Houdini cuts the lines to fast... It's very efficient in time control games but not in depth control games.


Anybody to discuss about this ?

Maybe I'm completely wrong but I'm really interested in hearing you ...

Shredder 7.04 was at the top for a long time, and was never quashed, even with immeadiate Shredder upgrades.
Then came along the first Rybka which went way above.

But by now, it is obvious, that the top Houdinis would anihalate Shredder 7.04, 100-0.

Fixed depth means nothing, which Mr. Houdart explained.

but fixed time, would be very different. I'm sure it would be 100-0 to Houdini 3, other things being equal.