Houdini 3 vs Critter 1.6a matches

[beram]

Houdini 3 vs Critter 1.6a Testmatches with own testsuite at different TC's,
50 games with 25 openings each engine plays position with black and white.
AMD Phenom II X6 1090T 3200 Mhz, each engine 4 cores, ponder off
The results show a tendency of Houdini 3 scoring better with longer time control. It wins about the same amount of games but looses less.

grts Bram

Code: Select all

Bram-PC, Blitz 2m+1s
Bram testsuite 2.0
                         
1   Houdini 3 x64        +17/=23/-10 57.00%   28.5/50
2   Critter 1.6a 64-bit  +10/=23/-17 43.00%   21.5/50

Bram-PC, Blitz 4m+2s
Bram testsuite 2.0
                         
1   Houdini 3 x64        +19/=24/-7 62.00%   31.0/50
2   Critter 1.6a 64-bit  +7/=24/-19 38.00%   19.0/50

Bram-PC, Blitz 60m
Bram testsuite 2.0
                         
1   Houdini 3 x64        +18/=28/-4 64.00%   32.0/50
2   Critter 1.6a 64-bit  +4/=28/-18 36.00%   18.0/50

[ernest]

The results show a tendency of ...

Come on Bram, this is just wishful thinking...
Do you realize what the error bars are, for 50-game matches?...

[beram]

Hi Ernest,

In general there are more draws with longer time controls, as is the case for all programs.
When you use the same openings testset and observe that program A looses lesser games with each 50 games test at LTC, while winning the same amount, you see a tendency, not more not less, but a tendency.
Having said that, I observed the same when looking at CCRL and CEGT results.
The longer the time control the better the results for Houdini 3 against Critter 1.6a
But I don't claim this scientifically or statistically 100% proven

grts Bram

[Laskos]

The results show a tendency of ...

Come on Bram, this is just wishful thinking...
Do you realize what the error bars are, for 50-game matches?...

These tests are not useless, they show with ~80% confidene that Houdini 3 performs better at longer TC.

Kai

[ernest]

These tests are not useless, they show with ~80% confidene that Houdini 3 performs better at longer TC.

OK for 80% confidence (that's not a lot...)
...that Houdini 3 performs better at longer TC... that is, better than Critter 1.6a at longer TC!

[Laskos]

These tests are not useless, they show with ~80% confidene that Houdini 3 performs better at longer TC.

OK for 80% confidence (that's not a lot...)
...that Houdini 3 performs better at longer TC... that is, better than Critter 1.6a at longer TC!

No, ~80% that Houdini 3 against Critter 1.6a at long TC performs better than Houdini 3 against Critter 1.6 at short TC. In other words, ~80% that Houdini 3 scales better than Critter 1.6a.

Kai

[ernest]

In other words, ~80% that Houdini 3 scales better than Critter 1.6a.

Yes, Kai, that's exactly what I meant to say...