SFdev-Houdini 1.5 : 2 different time handicap

[Vinvin]

Houdini 1.5, released in December 2010 (3110 CCRL and 3020 CEGT)
Vs
Stockfish version of 2020-06-01 (around 3465 CEGT and around 3460 CCRL)


Conditions :
- No EGTB
- Openings : NoomenShortLinesTestsuite.pgn (100 positions, 200 games per match)
- Interface : BanksiaGUI 0.27 beta
- ponder = OFF
- 1 thread per engine
- CPU : i7-4930K@4Ghz (6 cores)
- 6 games simultaneously.
- No adjudication

1st match : time handicap 30x, SF made a +63 Elo Perf
---------------------------------------
3 minutes + 3 seconds for Houdini 1.5
6 seconds + 0.1 sec for SF
It was the first time I used the BanksiaGUI, I didn't set the location to save the games and I lost them.

Code: Select all

Result:
----------------------------------------------------------------------------------------
  #  name                        games    wins   draws  losses   score    los%  elo+/-
  1. Stockfish 010620 64 POPCNT    200      85      66      49   118.0    99.9    63.2
  2. Houdini 1.5 x64               200      49      66      85    82.0     0.1   -63.2

Tech:
----------------------------------------------------------------------------------------
Tech (average nodes, depths, time/m per move, others per game), counted for computing moves only, ignored moves with zero nodes:
  #  name                          nodes/m         NPS  depth/m   time/m    moves     time
  1. Stockfish 010620 64 POPCNT       295K     1859307     23.9      0.2     89.1     14.1
  2. Houdini 1.5 x64                12393K     2711959     19.0      4.6     88.8    405.8

2nd match : time handicap 60x, SF made a -37 Elo Perf
---------------------------------------
5 minutes + 5 seconds for Houdini 1.5
5 seconds + 0.083 sec for SF
Games here : http://home.scarlet.be/vincentlejeune/c ... ndicap.zip

Code: Select all

Result:
----------------------------------------------------------------------------------------
  #  name                        games    wins   draws  losses   score    los%  elo+/-
  1. Houdini 1.5 x64               200      82      57      61   110.5    96.0    36.6
  2. Stockfish 010620 64 POPCNT    200      61      57      82    89.5     4.0   -36.6

Tech:
----------------------------------------------------------------------------------------
Tech (average nodes, depths, time/m per move, others per game), counted for computing moves only, ignored moves with zero nodes:
  #  name                          nodes/m         NPS  depth/m   time/m    moves     time
  1. Houdini 1.5 x64                19647K     2556920     19.5      7.7     80.3    617.0
  2. Stockfish 010620 64 POPCNT       229K     1677791     22.9      0.1     80.4     11.0

[lkaufman]

Houdini 1.5, released in December 2010 (3110 CCRL and 3020 CEGT)
Vs
Stockfish version of 2020-06-01 (around 3465 CEGT and around 3460 CCRL)


Conditions :
- No EGTB
- Openings : NoomenShortLinesTestsuite.pgn (100 positions, 200 games per match)
- Interface : BanksiaGUI 0.27 beta
- ponder = OFF
- 1 thread per engine
- CPU : i7-4930K@4Ghz (6 cores)
- 6 games simultaneously.
- No adjudication

1st match : time handicap 30x, SF made a +63 Elo Perf
---------------------------------------
3 minutes + 3 seconds for Houdini 1.5
6 seconds + 0.1 sec for SF
It was the first time I used the BanksiaGUI, I didn't set the location to save the games and I lost them.

Code: Select all

Result:
----------------------------------------------------------------------------------------
  #  name                        games    wins   draws  losses   score    los%  elo+/-
  1. Stockfish 010620 64 POPCNT    200      85      66      49   118.0    99.9    63.2
  2. Houdini 1.5 x64               200      49      66      85    82.0     0.1   -63.2

Tech:
----------------------------------------------------------------------------------------
Tech (average nodes, depths, time/m per move, others per game), counted for computing moves only, ignored moves with zero nodes:
  #  name                          nodes/m         NPS  depth/m   time/m    moves     time
  1. Stockfish 010620 64 POPCNT       295K     1859307     23.9      0.2     89.1     14.1
  2. Houdini 1.5 x64                12393K     2711959     19.0      4.6     88.8    405.8

2nd match : time handicap 60x, SF made a -37 Elo Perf
---------------------------------------
5 minutes + 5 seconds for Houdini 1.5
5 seconds + 0.083 sec for SF
Games here : http://home.scarlet.be/vincentlejeune/c ... ndicap.zip

Code: Select all

Result:
----------------------------------------------------------------------------------------
  #  name                        games    wins   draws  losses   score    los%  elo+/-
  1. Houdini 1.5 x64               200      82      57      61   110.5    96.0    36.6
  2. Stockfish 010620 64 POPCNT    200      61      57      82    89.5     4.0   -36.6

Tech:
----------------------------------------------------------------------------------------
Tech (average nodes, depths, time/m per move, others per game), counted for computing moves only, ignored moves with zero nodes:
  #  name                          nodes/m         NPS  depth/m   time/m    moves     time
  1. Houdini 1.5 x64                19647K     2556920     19.5      7.7     80.3    617.0
  2. Stockfish 010620 64 POPCNT       229K     1677791     22.9      0.1     80.4     11.0

So it appears that 50 to 1 would be the break-even point, roughly, somewhat over 5 1/2 doublings. The rating difference according to your above numbers is 445 by Ordo, 350 by BayesElo, so in the given range doubling speed is worth about 80 elo by the actual elo formula (Ordo), about 63 by Bayeselo (closer to human ratings). These are actually lower numbers than I would have guessed given how fast the SF level was set to. But Houdini 1.5 was not known for scaling well with more time.

[Vinvin]

Some test with SF 13.
Same conditions, except time control. 1/8 time handicap for SF 13 :

Stockfish 13 : 30 sec+0.25 Sec
Houdini 1.5 : 4 min+2 sec

Code: Select all

Result:
-----------------------------------------------------------------------------
  #  name             games    wins   draws  losses   score    elo    +    -
  1. Stockfish 13       200     169      30       1   184.0    183    4    5
  2. Houdini 1.5 x64    200       1      30     169    16.0   -183    5    4

Cross table:
-----------------------------------------------------------------------------
  #  name                score   games         1         2
  1. Stockfish 13        184.0     200         x   18557.0
  2. Houdini 1.5 x64      16.0     200    1646.0         x

Tech:
-----------------------------------------------------------------------------

Tech (average nodes, depths, time/m per move, others per game), counted for computing moves only, ignored moves with zero nodes:
  #  name               nodes/m         NPS  depth/m   time/m    moves     time   #fails
  1. Stockfish 13          779K     1299779     32.1      0.6     73.8     44.3        1
  2. Houdini 1.5 x64     13637K     2775246     18.5      4.9     73.2    359.8         
     all ---              7013K     2613603     25.3      2.7     73.5    202.0        1

[Vinvin]

Games are here : https://www.dropbox.com/s/nq84ltkaf00iu ... 8.zip?dl=0

Some test with SF 13.
Same conditions, except time control. 1/8 time handicap for SF 13 :

Stockfish 13 : 30 sec+0.25 Sec
Houdini 1.5 : 4 min+2 sec
...

[Vinvin]

I see now the only lost game of SF 13, is a lost on time.
Probably due to too small time margin (0.1 sec).

Code: Select all

[Event "SF13 vs H1.5"]
[Site "Vinvin"]
[Date "2021.02.20"]
[Round "1"]
[White "Houdini 1.5 x64"]
[Black "Stockfish 13"]
[Result "1-0"]
[TimeControl "0/240+2;0/30+0.25"]
[Time "07:12:05"]
[Board "18"]
[Termination "timeout"]
[ECO "B08"]
[Opening "Pirc"]

1. e4 d6 2. d4 Nf6
3. Nc3 g6 {End of opening}  4. Nf3 {+0.46/18 7737 18277366}  Bg7 {+0.61/20 790 784766; B08: Pirc, classical (two knights) system} 
5. Bd3 {+0.37/18 15713 37997053}  O-O {+0.31/17 406 409846}  6. O-O {+0.32/18 8069 19976061}  Nc6 {+0.32/20 1125 1193097} 
7. d5 {+0.33/18 5007 12782623}  Nb4 {+0.26/18 354 367779}  8. Be2 {+0.32/18 7668 19221387}  e6 {+0.32/22 1623 1714575} 
9. a3 {+0.36/17 5587 14095080}  Na6 {+0.50/21 631 707990}  10. dxe6 {+0.40/18 7633 19520575}  Bxe6 {+0.27/20 605 647338} 
11. Be3 {+0.37/18 6056 15339685}  Nc5 {+0.16/19 454 347932}  12. Ng5 {+0.34/17 4678 12273111}  Qe7 {+0.32/21 1263 1159697} 
13. Nxe6 {+0.29/17 24518 63701966}  Nxe6 {+0.00/21 524 533891}  14. Nd5 {+0.25/18 4445 11573874}  Nxd5 {+0.00/20 489 454980} 
15. exd5 {+0.16/19 11137 28743947}  Nc5 {+0.00/21 632 676668}  16. c3 {+0.28/19 4855 13224102}  a6 {+0.00/20 552 596209} 
17. Re1 {+0.24/19 9287 25012718}  Rae8 {+0.00/19 649 724510}  18. a4 {+0.26/18 4326 11658013}  Qd7 {-0.12/22 752 805391} 
19. Bxc5 {+0.24/19 7720 21094944}  dxc5 {-0.09/21 496 540833}  20. Qb3 {+0.24/20 3913 9922546}  Rb8 {-0.13/21 594 659013} 
21. Qc4 {+0.23/19 3921 10820531}  Qd6 {-0.07/22 518 595905}  22. a5 {+0.29/19 4072 11228291}  Rfe8 {-0.04/22 599 671784} 
23. g3 {+0.29/20 12555 34087556}  Bf8 {+0.00/22 1285 1336842}  24. Bf1 {+0.28/19 5061 13896982}  Qd7 {-0.10/20 393 434961} 
25. Qb3 {+0.28/20 8729 23122138}  b5 {-0.17/23 1334 1386697}  26. axb6 {+0.30/20 9034 23885238}  Rxb6 {-0.04/22 783 900231} 
27. Qa2 {+0.30/21 5557 15323166}  Reb8 {-0.04/21 530 621491}  28. Rab1 {+0.32/20 5730 14013976}  Kg7 {-0.11/23 2302 2203309} 
29. Bc4 {+0.42/18 4771 12427705}  a5 {-0.09/23 1451 1270288}  30. Re2 {+0.37/18 3255 5537566}  Bd6 {-0.17/23 1371 1423512} 
31. Rbe1 {+0.33/17 6173 11217154}  a4 {-0.14/20 619 404788}  32. Rd2 {+0.30/18 3170 7019317}  h5 {-1.54/21 893 885737} 
33. Re4 {+0.16/18 4445 10836547}  Qf5 {-0.64/20 888 1025801}  34. Bd3 {+0.05/17 5119 13210241}  Qg5 {-1.44/20 508 578041} 
35. Ree2 {+0.00/16 4601 12015370}  a3 {-1.87/20 918 1044897}  36. bxa3 {-0.34/16 8506 21904892}  Rb3 {-1.63/25 2046 2434094} 
37. Rd1 {-0.45/18 14888 36610254}  Rxc3 {-1.98/20 1261 1484424}  38. Rc2 {-0.37/18 4847 13104284}  Rbb3 {-1.70/22 1939 2248242} 
39. Rxc3 {-0.46/18 3677 9825699}  Rxc3 {-1.94/21 251 314080}  40. a4 {-0.43/18 2758 7461791}  h4 {-1.28/25 1394 1561289} 
41. Qd2 {-0.44/19 3810 9727116}  Qf6 {-1.32/24 751 971667}  42. Rc1 {-0.38/19 4430 12595761}  Ra3 {-1.35/23 386 498251} 
43. Rc4 {-0.42/19 3381 8947001}  hxg3 {-1.30/26 658 888376}  44. hxg3 {-0.58/18 12661 34756297}  Be5 {-1.35/24 452 557800} 
45. Kg2 {-0.58/19 2718 7353990}  Bd4 {-2.13/20 428 508077}  46. Be4 {-0.69/19 12295 34552209}  Qe5 {-2.08/20 565 716489} 
47. Qe2 {-0.74/19 2992 8344979}  Qg5 {-1.84/24 1650 2115197}  48. Rc2 {-0.76/18 2313 6302969}  Rxa4 {-1.78/21 189 216345} 
49. Bf3 {-0.78/19 2221 6291070}  Rb4 {-1.78/22 243 351345}  50. Qd1 {-0.80/19 3568 10045734}  Rb8 {-1.75/20 372 497384} 
51. Ra2 {-0.88/20 6858 18803398}  Rh8 {-2.41/19 247 328294}  52. Qc2 {-0.88/20 2223 6489502}  Qh6 {-2.59/22 313 414509} 
53. Kf1 {-0.88/18 10 6501}  Re8 {-2.64/21 825 1082473}  54. Qd3 {-0.88/19 2103 6120227}  Rb8 {-2.55/24 1851 2504411} 
55. Kg2 {-0.88/19 3572 9786549}  Rh8 {-2.55/16 45 68085}  56. Kg1 {-0.88/17 10 10403}  Qh2+ {-2.50/20 289 389551} 
57. Kf1 {-0.88/1 10 2}  Rb8 {-2.96/18 178 233802}  58. Bg2 {-0.97/19 2860 8050856}  Qh6 {-2.82/19 200 186184} 
59. Qc2 {-1.07/19 6925 19429074}  Qh5 {-3.06/18 89 123294}  60. Qd3 {-1.08/20 2387 6604915}  Qg5 {-3.02/16 110 132962} 
61. Bf3 {-1.02/20 2670 7488073}  Qc1+ {-3.42/18 251 313371}  62. Bd1 {-1.02/18 133 351319}  Rb1 {-3.80/19 193 237809} 
63. Ke2 {-1.03/19 2549 7157953}  Rb4 {-4.77/17 113 135730}  64. Kf1 {-1.08/18 2530 7049603}  Bxf2 {-4.97/17 135 191125} 
65. Kxf2 {-1.13/19 1982 6010529}  Rd4 {-5.61/17 203 258291}  66. Qc2 {-1.12/18 188 516474}  Qxd1 {-6.45/19 262 400453} 
67. Qxd1 {-1.27/20 3561 11107412}  Rxd1 {-7.59/18 265 316888}  68. Ra7 {-1.07/21 2310 7393699}  Kf6 {-7.85/18 331 421835} 
69. Rxc7 {-1.11/19 2035 6669659}  Rxd5 {-8.53/17 208 288215}  70. Ke2 {-1.16/19 3013 9987759}  Kf5 {-9.53/16 179 183725} 
71. Rxf7+ {-0.94/20 2253 7760859}  Kg4 {-152.65/7 851 146}  72. Rg7 {-1.02/22 2197 7601175}  g5 {-152.65/22 494 918109} 
73. Kf2 {-1.19/22 3115 10733794}  Rd3 {-152.65/25 343 579878}  74. Ke2 {-1.16/23 1929 6262927}  Rxg3 {-152.65/24 367 662445} 
75. Rc7 {-1.24/23 5479 19860542}  {Black forfeits on time} 1-0

[Nordlandia]

Cutechess gui allow time handicap. Maybe it can be good idea to let SF 13 use ponder.

Houdini don't think during opponent's turn. SF does was my idea.

This feature is not added yet on cutechess but it is suggested as a feature.

[Ajedrecista]

Hello Vincent:

Good tests! I took a look here to have an idea on time handicaps. I did some estimates to find the handicap where both engines should perform evenly:

Code: Select all

Δ := estimated Elo difference between engines (Elo0 - Elo).
η := estimated Elo gain per doubling of time.

δ := performance in the test.

t/t0 = 2^[(Δ - δ)/η]
η = (Δ - δ)/lg2(t/t0)  // Constant.

First test:  t/t0 = 30, δ = 63.
Second test: t/t0 = 60, δ = -37.

(Δ - 63)/lg2(30) = (Δ + 37)/lg2(60)
Δ ~ 553.67  // Maybe too large, but it will serve for the interpolation of t/t0.
η ~ (553.67 - 63)/lg2(30) ~ 125.06

Try to find at what time handicap both engines will perform evenly (δ = 0):
(Δ - 63)/lg2(30) = Δ/lg2(t/t0)
t/t0 = 30^[Δ/(Δ - 63)] ~ 46.43
t/t0 ~ 325/7

Larry's guess was about 1:50, which looks good so far. Mine is 1:(46.43) or 7:325 (for example 7" for SF and 5'25" for Houdini as base time, increments aside).

For the new SF 13 and keeping constant the parameter η:

Code: Select all

t/t0 = 8, δ = 424.28
η = (Δ - 424.28)/lg2(8)
Δ ~ 799.46  // Maybe too large, but it will serve for the interpolation of t/t0.

t/t0 = 8^[Δ/(Δ - 424.48)] ~ 84.02

I get 1:84 this time (for example 5" for SF 13 and 7' for Houdini).

In first and second tests, increments were 1/60 of base time but only 1/120 in SF 13 test. Isn't it too few?

Disclaimer: my guess should not be considered an axiom, it is only a guess for fun.

Regards from Spain.

Ajedrecista.

[Nordlandia]

Can anyone figure out how much time handicap is needed if stockfish can think during the opponent's turn against Houdini 1.5 while Houdini can not.

[Vinvin]

60 times handicap (same as for SF 12) :
5 minutes + 5 seconds for Houdini 1.5
5 seconds + 0.083 sec for SF 13

I got strange results there : 41 games lost on time for SF 13, despite the fact I set 0.5 sec margin on time.
May be this come from Syzygy access I'll rerun without Syzygy.
It was my intention to not use syzygy but it was automatically set

Code: Select all

Result:
-----------------------------------------------------------------------------
  #  name             games    wins   draws  losses   score    elo    +    -
  1. Stockfish 13       200      84      54      62   111.0     24    3    4
  2. Houdini 1.5 x64    200      62      54      84    89.0    -24    4    3

Cross table:
-----------------------------------------------------------------------------
  #  name                score   games         1         2
  1. Stockfish 13        111.0     200         x   11827.5
  2. Houdini 1.5 x64      89.0     200    8907.5         x

Tech:
-----------------------------------------------------------------------------

Tech (average nodes, depths, time/m per move, others per game), counted for computing moves only, ignored moves with zero nodes:
  #  name               nodes/m         NPS  depth/m   time/m    moves     time   #fails
  1. Stockfish 13          155K     1114602     21.8      0.1     79.3     11.0       41
  2. Houdini 1.5 x64     23501K     2901925     20.3      8.1     79.1    640.2         
     all ---             11531K     2871614     21.0      4.1     79.2    325.6       41

[Vinvin]

60 times handicap (same as for SF 12) :
5 minutes + 5 seconds for Houdini 1.5
5 seconds + 0.083 sec for SF 13

I got strange results there : 41 games lost on time for SF 13, despite the fact I set 0.5 sec margin on time.
May be this come from Syzygy access I'll rerun without Syzygy.
It was my intention to not use syzygy but it was automatically set

Code: Select all

Result:
-----------------------------------------------------------------------------
  #  name             games    wins   draws  losses   score    elo    +    -
  1. Stockfish 13       200      84      54      62   111.0     24    3    4
  2. Houdini 1.5 x64    200      62      54      84    89.0    -24    4    3

Cross table:
-----------------------------------------------------------------------------
  #  name                score   games         1         2
  1. Stockfish 13        111.0     200         x   11827.5
  2. Houdini 1.5 x64      89.0     200    8907.5         x

Tech:
-----------------------------------------------------------------------------

Tech (average nodes, depths, time/m per move, others per game), counted for computing moves only, ignored moves with zero nodes:
  #  name               nodes/m         NPS  depth/m   time/m    moves     time   #fails
  1. Stockfish 13          155K     1114602     21.8      0.1     79.3     11.0       41
  2. Houdini 1.5 x64     23501K     2901925     20.3      8.1     79.1    640.2         
     all ---             11531K     2871614     21.0      4.1     79.2    325.6       41

I restart the same tournament but without syzygy.
SF 13 lost 37 games on time. Very Strange.
I set a very high time margin : 0.500 sec
Does anybody else has trouble with SF 13 lost on time ?

Code: Select all

Result:
-----------------------------------------------------------------------------
  #  name             games    wins   draws  losses   score    elo    +    -
  1. Stockfish 13       200      90      56      54   118.0     33    3    4
  2. Houdini 1.5 x64    200      54      56      90    82.0    -33    4    3

Cross table:
-----------------------------------------------------------------------------
  #  name                score   games         1         2
  1. Stockfish 13        118.0     200         x   12327.5
  2. Houdini 1.5 x64      82.0     200    8372.5         x

Tech:
-----------------------------------------------------------------------------

Tech (average nodes, depths, time/m per move, others per game), counted for computing moves only, ignored moves with zero nodes:
  #  name               nodes/m         NPS  depth/m   time/m    moves     time   #fails
  1. Stockfish 13          154K     1141635     21.3      0.1     84.4     11.4       37
  2. Houdini 1.5 x64     24221K     3036923     20.3      8.0     84.2    671.1         
     all ---             11884K     3005238     20.8      4.0     84.3    341.3       37