Houdini 2.0 : Settings (Z, T3, Baracuda, Baracuda T3)

[Tennison]

After seeing so closed ratings on Sedat Rating List, I started new tests :

Code: Select all

Games Completed = 200 of 4000 (Avg game length = 31.702 sec)

Settings = Gauntlet/64MB/500ms+200ms/M 1000cp for 12 moves, D 150 moves/EPD:8moves.epd(3710)


 1.   Houdini 2.0b x64        	   88.5/200	62-85-53  	(L: m=28 t=0 i=0 a=57)	(D: r=9 i=10 f=11 s=1 a=22)	(tpm=183.2 d=13.3 nps=2594698)
 2.   Houdini 2.0b Z           	  25.0/50	19-19-12  	(L: m=5 t=2 i=0 a=12)	(D: r=2 i=0 f=1 s=1 a=8)	(tpm=178.3 d=13.0 nps=2581551)
 3.   Houdini 2.0b T3         	   33.0/50	26-10-14  	(L: m=3 t=0 i=0 a=7)	(D: r=3 i=5 f=4 s=0 a=2)	(tpm=177.2 d=13.0 nps=2546961)
 4.   Houdini 2.0b Baracuda      	25.5/50	18-17-15  	(L: m=5 t=0 i=0 a=12)	(D: r=3 i=2 f=5 s=0 a=5)	(tpm=185.9 d=13.8 nps=2602495)
 5.   Houdini 2.0b Baracuda T3   	28.0/50	22-16-12  	(L: m=5 t=0 i=0 a=11)	(D: r=1 i=3 f=1 s=0 a=7)	(tpm=178.2 d=13.3 nps=2590310)

I'll give partial results as often as possible.

I'll give the games when these 4000 games are ended.

[Tennison]

Code: Select all

Games Completed = 440 of 4000 (Avg game length = 31.823 sec)

Settings = Gauntlet/64MB/500ms+200ms/M 1000cp for 12 moves, D 150 moves/EPD:8moves.epd(3710)

 1.   Houdini 2.0b x64        	206.5/440	141-168-131  	(L: m=53 t=4 i=0 a=111)	(D: r=26 i=34 f=19 s=2 a=50)	(tpm=184.5 d=13.6 nps=2603972)
 2.   Houdini 2.0b Z             55.5/110	41-40-29  	(L: m=15 t=2 i=0 a=23)	(D: r=6 i=4 f=3 s=1 a=15)	(tpm=181.7 d=13.4 nps=2595078)
 3.   Houdini 2.0b T3         	64.0/110	45-27-38  	(L: m=8 t=0 i=0 a=19)	(D: r=10 i=13 f=7 s=1 a=7)	(tpm=180.9 d=13.4 nps=2579879)
 4.   Houdini 2.0b Bcuda      	57.0/110	40-36-34  	(L: m=7 t=2 i=0 a=27)	(D: r=8 i=8 f=7 s=0 a=11)	(tpm=184.9 d=13.7 nps=2601387)
 5.   Houdini 2.0b Bcuda T3   	57.0/110	42-38-30  	(L: m=12 t=2 i=0 a=24)	(D: r=2 i=9 f=2 s=0 a=17)	(tpm=183.1 d=13.7 nps=2611642)

[Tennison]

Code: Select all

Games Completed = 680 of 4000 (Avg game length = 31.867 sec)

Settings = Gauntlet/64MB/500ms+200ms/M 1000cp for 12 moves, D 150 moves/EPD:8moves.epd(3710)

 1.   Houdini 2.0b x64        	336.5/680	237-244-199  	(L: m=74 t=8 i=0 a=162)	(D: r=38 i=51 f=32 s=3 a=75)	(tpm=184.0 d=13.7 nps=2611254)
 2.   Houdini 2.0b Z           	86.5/170	63-60-47  	(L: m=21 t=3 i=0 a=36)	(D: r=10 i=7 f=5 s=1 a=24)	(tpm=182.5 d=13.6 nps=2602313)
 3.   Houdini 2.0b T3         	 92.5/170	66-51-53  	(L: m=17 t=0 i=0 a=34)	(D: r=10 i=17 f=9 s=1 a=16)	(tpm=181.7 d=13.6 nps=2593549)
 4.   Houdini 2.0b Bcuda      	 81.5/170	56-63-51  	(L: m=17 t=2 i=0 a=44)	(D: r=12 i=12 f=12 s=0 a=15)	(tpm=185.4 d=13.8 nps=2619508)
 5.   Houdini 2.0b Bcuda T3   	 83.0/170	59-63-48  	(L: m=16 t=3 i=0 a=44)	(D: r=6 i=15 f=6 s=1 a=20)	(tpm=184.5 d=13.7 nps=2616797)

[Tennison]

Code: Select all

Games Completed = 1864 of 4000 (Avg game length = 32.495 sec)

Settings = Gauntlet/64MB/500ms+200ms/M 1000cp for 12 moves, D 150 moves/EPD:8moves.epd(3710)

 1.   Houdini 2.0b x64        	944.0/1864	658-634-572  	(L: m=191 t=34 i=0 a=409)	(D: r=98 i=137 f=106 s=4 a=227)	(tpm=184.4 d=13.8 nps=2605804)
 2.   Houdini 2.0b Z           	234.5/466	162-159-145  	(L: m=48 t=17 i=0 a=94)	(D: r=25 i=32 f=34 s=1 a=53)	(tpm=183.4 d=13.7 nps=2598467)
 3.   Houdini 2.0b T3         	238.0/466	169-159-138  	(L: m=38 t=6 i=0 a=115)	(D: r=28 i=38 f=18 s=2 a=52)	(tpm=183.6 d=13.6 nps=2583979)
 4.   Houdini 2.0b Bcuda      	214.0/466	141-179-146  	(L: m=64 t=8 i=0 a=107)	(D: r=26 i=29 f=36 s=0 a=55)	(tpm=186.4 d=13.9 nps=2629058)
 5.   Houdini 2.0b Bcuda T3   	233.5/466	162-161-143  	(L: m=47 t=8 i=0 a=106)	(D: r=19 i=38 f=18 s=1 a=67)	(tpm=184.7 d=13.8 nps=2608438)

Original Baracuda settings seems less good for the time.
T3 settings are always a little bit (but not significant) ahead Z ...
But at that time control it seems there is no gain with the settings comparing to official ones.

[Tennison]

Code: Select all

Games Completed = 2348 of 4000 (Avg game length = 32.554 sec)

Settings = Gauntlet/64MB/500ms+200ms/M 1000cp for 12 moves, D 150 moves/EPD:8moves.epd(3710)

 1.   Houdini 2.0b x64        	1194.0/2348	836-796-716  	(L: m=243 t=40 i=0 a=513)	(D: r=126 i=170 f=136 s=4 a=280)	(tpm=184.3 d=13.8 nps=2598871)
 2.   Houdini 2.0b Z           	292.5/587	204-206-177  	(L: m=58 t=17 i=0 a=131)	(D: r=29 i=42 f=42 s=1 a=63)	(tpm=183.7 d=13.7 nps=2589183)
 3.   Houdini 2.0b T3         	302.5/587	215-197-175  	(L: m=51 t=7 i=0 a=139)	(D: r=34 i=43 f=30 s=2 a=66)	(tpm=183.9 d=13.7 nps=2585425)
 4.   Houdini 2.0b Bcuda      	272.0/587	179-222-186  	(L: m=79 t=10 i=0 a=133)	(D: r=34 i=39 f=43 s=0 a=70)	(tpm=185.8 d=13.9 nps=2614229)
 5.   Houdini 2.0b Bcuda T3   	287.0/587	198-211-178  	(L: m=63 t=10 i=0 a=138)	(D: r=29 i=46 f=21 s=1 a=81)	(tpm=184.9 d=13.8 nps=2606092)

[Tennison]

Code: Select all

Games Completed = 4000 of 4000 (Avg game length = 32.868 sec)

Settings = Gauntlet/64MB/500ms+200ms/M 1000cp for 12 moves, D 150 moves/EPD:8moves.epd(3710)

 1.   Houdini 2.0b x64        	2031.0/4000	1411-1349-1240  	(L: m=391 t=74 i=0 a=884)	(D: r=216 i=298 f=231 s=9 a=486)	(tpm=184.6 d=13.8 nps=2598139)
 2.   Houdini 2.0b Z           	514.5/1000	361-332-307  	(L: m=86 t=27 i=0 a=219)	(D: r=48 i=77 f=65 s=2 a=115)	(tpm=184.2 d=13.7 nps=2588062)
 3.   Houdini 2.0b T3         	506.5/1000	347-334-319  	(L: m=98 t=16 i=0 a=220)	(D: r=62 i=78 f=54 s=4 a=121)	(tpm=184.6 d=13.8 nps=2595228)
 4.   Houdini 2.0b Bcuda      	463.5/1000	304-377-319  	(L: m=117 t=18 i=0 a=242)	(D: r=58 i=73 f=68 s=2 a=118)	(tpm=185.6 d=13.8 nps=2606464)
 5.   Houdini 2.0b Bcuda T3   	484.5/1000	337-368-295  	(L: m=118 t=15 i=0 a=235)	(D: r=48 i=70 f=44 s=1 a=132)	(tpm=185.2 d=13.8 nps=2603716)

Baracuda seems clearly weaker than original Houdini in this test.
Adding T3 params to Baracuda settings seems a good idea but not enough.

T3 settings and Z settings seems (as in Sedat Rating List) a little bit better than original. A small plus to Z here but need more games to confirm because so close !

Anyway if someone want to have good settings, I think it's necessary to had the T3 params in it : each time better results than without.

[Tennison]

Here are the games :

The games

[Houdini]

T3 settings and Z settings seems (as in Sedat Rating List) a little bit better than original. A small plus to Z here but need more games to confirm because so close !

If you want to make the claims above, you need to play more games. For the moment everything is buried under the tower of statistical uncertainty.
For T3 and Z settings there is only one valid conclusion after 1000 games: they are not significantly different from the default settings.

On the Rybka Forum I published the results obtained with "z", "s" and your "T4" settings in 16´000 game matches against the default Houdini 2.0c.
Here are the results against the standard Houdini 2.0c.
- "s" scored 7797-8203 (38% draws), -8 Elo (+/- 4 Elo)
- "z" scored 7922-8078 (39% draws), -4 Elo (+/- 4 Elo)
- "T4" scored 7914-8086 (40% draws), -4 Elo (+/- 4 Elo)
Again, the only conclusion can be that these settings change very little to the objective strength of the engine.

[Tennison]

I completely agree with you Robert : it's impossible to have a huge gain with original because you have tested it a lot and choose the best you can.

But maybe it's possible to have a small gain ('til +5/+10) but not more I think.

As an example I have another test running for the moment :

Houdini 1.5a - Houdini 1.5a T3

It's a 10000 games test ... As soon as it is finished I can give you the games if you want.

And for the time the result is :

Houdini 1.5a : 2627 / 5334
Houdini 1.5a T3 : 2707 / 5334

As you can see, there is no significant plus, as above.
But each time I run a test with the Storm values to 40,50,60, I have little but better results than original.

Maybe a way to follow.

[Ajedrecista]

Hello Robert:

T3 settings and Z settings seems (as in Sedat Rating List) a little bit better than original. A small plus to Z here but need more games to confirm because so close !

If you want to make the claims above, you need to play more games. For the moment everything is buried under the tower of statistical uncertainty.
For T3 and Z settings there is only one valid conclusion after 1000 games: they are not significantly different from the default settings.

On the Rybka Forum I published the results obtained with "z", "s" and your "T4" settings in 16´000 game matches against the default Houdini 2.0c.
Here are the results against the standard Houdini 2.0c.
- "s" scored 7797-8203 (38% draws), -8 Elo (+/- 4 Elo)
- "z" scored 7922-8078 (39% draws), -4 Elo (+/- 4 Elo)
- "T4" scored 7914-8086 (40% draws), -4 Elo (+/- 4 Elo)
Again, the only conclusion can be that these settings change very little to the objective strength of the engine.

I fully agree with you in view of the results of these three 16000-game matches. Statistical uncertainties mean everything in testing, IMHO. If I take the info of your three matches that you gave in the post I quote, I get more less these results:

Code: Select all

s settings: +4757 -5163 =6080

(Error bars for 2-sigma confidence ~ 95.45% confidence) ~ -8.82 ± 4.33 Elo.
(LOS for s settings) ~ 0%.
(LOS > 95%): one engine must score 8082 points or more.

Code: Select all

z settings: +4802 -4958 =6240

(Error bars for 2-sigma confidence ~ 95.45% confidence) ~ -3.39 ± 4.29 Elo.
(LOS for z settings) ~ 5.72%.
(LOS > 95%): one engine must score 8081.5 points or more.

Code: Select all

T4 settings: +4714 -4886 =6400

(Error bars for 2-sigma confidence ~ 95.45% confidence) ~ -3.74 ± 4.26 Elo.
(LOS for T4 settings) ~ 3.96%.
(LOS > 95%): one engine must score 8081 points or more.

You may correct my results if you think they are wrong.

I guess that Houdini 3 is in the good path, only reading some of your reports. Could you give a rough estimate of the current Elo gain over Houdini 2.0c (or over Houdini 1.5a), please? Thanks in advance, and please keep up the good work!

@Ben: I wish you good luck, but remember that Robert did a lot of testing: for example, I think I read somewhere that he played circa three millions of games for tuning Houdini 1.5! Robert may correct me if I am wrong.

Regards from Spain.

Ajedrecista.